Lip Sync Setup

in Softimage 3D

by Adam Sale

________________________________________________________________________

Objective: To prepare your 'Actor' to perform lip sync to pre-recorded dialogue

Synopsis: Softimage can use any object as a skeleton. We will be using a primitive spline as the control for our lips. Using the Set Cluster tool, we will create a cluster for each point on the spline. We will then bring primitive squares or nulls that will act as our lip skeletons. Using the Object to Cluster constraint tool, we will constrain each primitive to a cluster point on the spline. Finally, we will envelope each square to the face geometry using the Global Envelope tool. By animating the shape of the spline, we will cause the clusters to move, which in turn updates the position of the primitive nulls or squares. Since the primitives will now control the movement of the geometry, the lip portion of the character will update, thus giving us an efficient method of achieving lip sync.

________________________________________________________________________

Softimage Tools Used:

Get> Primitive > Null

Draw> Extract

Tag > Set Cluster

Tag > Cluster List

Constraint > Object To Cluster

Skeleton > Group As Skeleton

Skin > Global Envelope

Shape > Shape Interp>Weighted

Shape> Shape List> Additive

Skin . Envelope Assignment> Modify Weights

Savekey > Object > Shape

F-Curve Select >Object > Shape Weights

Preparing the Character

This tutorial assumes that you have constructed the head of your character using the Radial Modelling Method. There are a number of Radial Modeling Methods online if you are unfamiliar with this technique. The Vault possesses a radial modelling tutorial supplied by Softimage from the Technique section of their web site found at WWW.SOFTIMAGE.COM. The name of the tutorial is 'Basket Boy'.

Extracting The Mouth Spline, & Setting Clusters with Centers

1.) You should already have constructed the head of your character in a neutral, open-mouthed position using the radial modeling method. This default mouth position will allow the envelope weighting to be evenly distributed over the lip control Nulls. A NURBS head will work most efficiently due to the added precision in extracting splines from a surface, however for our purposes, it is best to use a B-spline Patch due to our hardware limitations. Regardless, the choice is yours. I like to begin blocking my head in as a Patch object, and when I am satisfied with the overall form, convert my model to NURBS and continue from there

2.) Click on the Preferences button, and ensure that the Create Modeling Relation option has been deactivated. If this button is not deactivated, your character will not be able to perform any lip sync after it has been setup.

3.)Select the head in Single mode, and use the Draw > Extract Command in MODEL Mode(F1) to extract a spline from around the lips area. Alternatively, you may find that extracting a spline from the mouth yields a spline with too many points. If this is the case, get a primitive circle with 12 or so points. Scale, Rotate and move your circle into position so that it fits the rough shape of the opening of your character's mouth. Make sure that your circle's center is in the middle of the spline.

This extracted spline will eventually be shape animated. It will drive the lip movement of your character's head.

4.) Select the spline in single mode, and tag a single point on the curve. Click on Tag > Set Cluster to bring up the Set Cluster window. Click in the white box to the left of the Create Center option, and then click the NEW button. You have just created a cluster complete with its very own center. To verify this, you should see that a null has been created in the same Cartesian space as the point you first tagged. This null is the cluster's center. You can also look in your schematic port to see that there is now a cluster node that is connected to the original spline. It should read something like ClusterCenter01.

Select the Spline, and untag the point you first tagged.

5.) Repeat the above step for each successive point along the spline until each point has become a cluster with its own center. When this step has been completed, you will have as many nulls as there are points on your spline. Deselect the spline. In Multi mode, you can now select these nulls and hide them using Display> Hide >Toggle Desel. Hidden. These Nulls are simply centers, and will only serve to confuse you during the next step. I recommend selecting and hiding the nulls buy selecting them in schematic view.

Null Skeletons & Cluster Constraints

6.) Switch back to single mode and Get à Primitive Null. Duplicate this null so that you end up with as many Nulls as there are points on your extracted 'lip' spline. You may want to move each duplicated Null so that they are not all stacked on top of one another. The less confusion you have to deal with the better! These Nulls will become the skeletons that will drive the lip animation for your character's head. On a side note, I like to use squares instead of Nulls because they are easier for me to see in my ortho and perspective windows.

7.) Select your extracted spline. In MOTION (F2 is the hotkey) click on Shape > Cluster List to bring up the Cluster List window. Within the Cluster List window, you will see the names of all of the clusters that you have created in steps 4 & 5. Click once on the first cluster (Cluster0), and then click on OK. You will have exited the Cluster List window, and the respective cluster on your spline will now be highlighted in red.

8.) Select one of the nulls that you created in step 6. It doesn't matter which one.

In the MOTION module (F2) click on Constraint> Object To Cluster. If you look down at the mouse line at the bottom of your screen, you will see that it prompts you to 'pick a reference' using either the Left or Middle mouse button. Since the cluster has been activated on the extracted spline, click on the spline as a reference. The null will be constrained to the Cluster that you highlighted in step 7.

9.) Repeat step 7 once again, The first cluster already has a Null constrained to it, so its purpose has been served. Select the next cluster down in your cluster list. Click OK. The next cluster (Cluster1) will now be highlighted on the extracted spline.

Select the next available free Null and click on Constraint > Object To Cluster again. (Note, Softimage remembers the last command used on an object. If you look down at the mouse line when you select the new Null, you will notice that 'Pick Reference' is still activated) With the new Null selected, click on the spline. This null will snap to the next cluster that you activated in the Cluster List window.

Repeat this step until all of the Nulls have been constrained to a cluster on the 'lip' spline. Depending on the size of your character's head in Softimage Units, the Nulls that you have just constrained to the clusters may be too large. You can select each of the Nulls and scale them up or down to suit the relative sizes of each of your characters.

Transforming Nulls Into Skeletons & Enveloping Nulls to the Head

10.) The spline setup is now ready to be turned into a skeleton and enveloped to the head.

Select one of the Nulls. In your ACTOR (F3) module, choose the command Skeleton> Group As Skeleton. Your mouse line will prompt you to 'Set Skeleton'. Click on the Null that you have selected. It will temporarily turn Green. If you frame it in your Schematic viewport, you will notice that it has the word 'Skel' written in red above it, indicating that the null now has the same physical properties as a 2D or 3D chain of Joints.

Select the next null and click once on it to turn it into a skeleton object (the Set Skeleton prompt is still activated along your mouse line) Repeat until all of the constrained Nulls along the spline have been turned into skeletons.

Your Nulls should all be skeleton objects now. Next get one more null, and make it the parent of all the skeleton nulls.

11.) Select your characters head in single mode, and click on Skin >Global Envelope, and pick the parent null. Accept the default switches in the Envelopes Assignment Box.

When you have finished the enveloping process, select the spline, and parent it to the character's head joint as a child. This will allow the lips to move along with the head when the character is performing any actions. Make the Null hierarchy a child of the head joint as well.

Your characters head has now been 'boned', all that is left for us to do before beginning animation, is the weighting of the points on your character's head.

12.) Tag points on your character's head, and using the Skin> Envelope Assignment> Modify Weights command, assign and weight the points around your character's mouth to the various Nulls that make up its skeleton.

It will save you a great deal of time if you name your Nulls using the Info > Selection command. Names such as "TLLIP (top left lip) BRLIP (bottom right lip) etc. will make it much clearer when choosing which points to weight to which nulls in your weighting dialogue box.

This step requires the most patience; after all, careful weighting will determine how well the character's mouth area deforms. There is nothing worse than animating a character whose geometry 'crashes' through itself. It is also very distracting to a prospective employer or an audience viewing your work.

More on enveloping techniques in another tutorial.

Animating Your Model

We have covered a lot of ground up to now, the benefits of this setup method are ready to be harnessed in the precise control you now have over your lips animation.

Essentially what you are going to do is to create your own set of 3D sliders, and add expressions to the sliders (Nulls) which will associate them to a series of phoneme lip shapes that you have sculpted with the spline.

It will be handy for the purposes of the next few steps if you have a pictorial reference of what the mouth looks like when uttering certain phonemes. The Preston Blair books have an excellent guide to follow along with. As well, this tutorial comes with a guide to creating phoneme shapes.

13.) With your Lip Spline selected in single mode, switch to your MOTION (F2) module, and click on Shape> Shape Interp. > Weighted.

Weighted interpolation creates one F-Curve for each key shape. The term weighted comes from the fact that you can adjust each shapes function curve to emphasize any key shape or combination of key shapes.

Linear interpolation means that the physical points of the deformed object will travel in a linear direction.

14.) You all should be in the practice of starting your animations after frame 100 due to the way that the PVR orders numbers. Since we are going to be creating numerous lip phoneme shapes, and lip expressions, we can use each frame from 1 - 99 to store a shape. Potentially, we could store 99 lip shapes, although you won't need nearly that many…

Begin at frame 1. Select your lip spline in single mode, and click on Savekey > Object> Shape. Move the timeline to frame 2, and move the points on the spline to create the next shape.

Click on Savekey > Object> Shape.

Continue on until you have created all lip shapes that you will need. Remember, each lip shape should be stored on a different frame. The point to consider with shape animation is that the center of the object being shape animated should never scale or change position.

15.) With the spline selected, click on Shape > Shape List. In the window that pops up, you will see a box that contains all of the names of the shapes you have created. To the right of that there are two options Average and Additive. Make sure that Additive is selected. Additive will take the information from two shape F-curves and add their properties together instead of averaging the two. Click OK.

The result; A wider range of emotion available for animating in your characters face.

16.) To check and see if everything is working well, scroll through the frames that have a keyshape saved on them, all the while looking at your characters mouth. As each frame goes by, it will assume the shape that was saved on its respective frame.

With your lip spline selected, click on Fcurve Select > Object > Shape Weights. A series of F-curves will appear in the F-curve editor

You can stop here if you wish, and begin blocking in your animation by using the Edit Key function in the F-Curve editor. By adding keyframes on the wshp F-curves, you can manually animated the phonemes being turned on and off.

If you are technically inclined, however, and like a good challenge, proceed to the next section. This section will cover Expressions in brief, and how you can use them to animate the F-curves using simple null objects.

Set-Up for Expressions

Bring in as many NULLS as you have KEY SHAPES, and arrange them side by side in your right view window on the origin. To make matters easier, you can create a border around your nulls using a primitive square. It is a good idea to color it, so the null is easily spotted in your schematic window. This way, you can frame your slider nulls by pressing the "F" hotkey when the null is selected.

Select each of the nulls in single mode and rename them using Info à Selection. Think of names that correspond to each phoneme lip shape that you have sculpted, i.e. Ooh, Ff, Th etc.

Shape Weights & Expressions

This next step involves associating nulls to each shape that has been saved on the lip spline. Through the use of simple Expressions, we will create a slider system similar to a sound mixing board that will allow you to animate the various phoneme shapes you have created for your character.

17.) Select the lip spline in single mode. Press F2 to switch to the motion module. In the lower right of your screen, click on Expressions > Edit. The Expressions editor window will pop up. Expressions are simply a string of characters (numbers and letters) that define relationships between various elements in a scene.

For example; A = B, is a simple expression showing that A is equal or constrained to B. Expressions can be used for to create relationships for any parameter that can be animated.

You will notice that the lip spline name appears in the Affected elements box. The lip spline has all of the shapes saved onto it. When we associate the slider Nulls to each shape on the spline, it is the animation of these Nulls that are going to affect how the shapes interact with each other to create convincing lip sync.

Since the Nulls drive the shapes, the lip spline is the Affected Element!

An easy analogy would be that of a CD in a CD Player. A Compact Disc would be considered the affected element, because when placed into the tray of a CD player, it is going to rely on the CD players drive belt to rotate it.

IT IS AFFECTED DIRECTLY BY THE DRIVE BELTS ROTATION. THEREFORE PUT INTO SOFTIMAGE LANGUAGE AN EXPRESSION TO DESCRIBE THIS INTERACTION MIGHT LOOK LIKE THE FOLLOWING: CD:RotY=DRIVE BELT.RotY

18.) Make sure that the cursor is positioned after the name in the Affected Elements box. In the middle of the window is a box that reads F-Curves. Click on it. A window will pop up that contains all of the F-Curves possible for your Affected Element (Lip Spline). Your lips Shape Interpolation has been set to Weighted in step 13, so their will be a number of F-Curves in the list named Wshp0, Wshp1, Wshp2.

Each Wshp is equivalent to one of the keyshapes saved on the spline. Since you saved your shapes from frame 1 to X, the shape weight F-Curves will roughly correspond to each frame. Select the first shape weight F-Curve, and press OK.

The Wshp0 F-curve is inserted into the Affected element line

You should now have the following in your Affected Element line

LipSpline.Wshp0 (LipSpline being the name of your model)

Now click in the box below marked Expression. This is where you will associate the Lip Splines Shape Weight to the respective slider Null that describes the Phoneme shape.

Click in the Scn Element (Scene Element) box, and in the window that pops up, select the name of the null that corresponds to the Shape Weight for the Lip Spline that you have selected in your Affected Elements box.

For example; if Wshp0 is an "Oooh" phoneme shape, and LipSpline.Wshp0 is the affected element, then select the Scene Element "Oooh" as the first object on your Expressions line.

Press OK

Click in the F-curve box to bring up all possible F-curves for the Slider Null. The F-Curve that we are looking for is Etrny, (or the Nulls Explicit Translation along the Y axis) Select Etrny, and click OK.

This associates the Lip Splines Wshp0' to the Y-translation of the Null.

Shape weights F-Curves have one important feature that makes them ideal for use with a mixing board / slider approach. If you call up the F-Curve for a shape weight, you will notice that it runs in value from 0 to 1. A shape weight with a value of zero is off, or has no weight.

A shape weight of one is equal to the original shape sculpted, and is thus fully weighted. A shape weight of 0.5 then, would result in a shape that is halfway between on and off. If you had a shape weight F-curve for a smile, a weight of zero would not have any effect on the model, a weight of one, would yield the smile, and a weight of 0.5 would give a neutral emotion

The expression still isn't finished. You should have the following element and F-curve in the Expression box:

Null.Etrny (where Null is the name of the phoneme corresponding to your shape.)

The final Expression should read:

(Null.Etrny ) / 10

The brackets are important here, because they tell SOFTIMAGE to perform the math within the brackets first, before dividing the result by 10.

Here is the reasoning for the math;

As you already know, Shape Weight F-Curves run from a value of 0 to 1, 0 meaning that the particular shape is turned off, 1 meaning that the shape is turned on. The nulls that we created in the right view window are arranged on the origin if you remember, so their Y value is 0, which is equivalent to a shape weight being turned off. Dividing the expression by 10 will yield a value of 1 for the null when it is at 10 units on the Y axis.

Therefore, if you animate the Nulls Y translation from 0 to 10, it will correspond to the shape weights position from 0 to 1.

Repeat step 18 until all of the shape weights and nulls have been accounted for

19.) Your character is now ready to be animated. By moving the Null sliders, and saving Explicit Translation Keyframes, you can easily, and efficiently animate the lips to pre-recorded dialogue that has been doped out.

Keep in mind that using Shape weights allows you to mix the weight of more than one shape at once, and also allows for you to jump back and forth between any shape that has been keyed.

It takes time and patience to get this method of animation down pat, but your results will most certainly be worth it.

Lip Sync Setup in Softimage 3D was written by Adam Sale. Adam is a Technical Director and co-founder of Joncrow Entertainment. He can be reached at adamsale@home.com.