This section applies to SCI 1.1 only
To generate lip-sync data, you need three things:
- A message entry with spoke text audio associated with it
- A view/loop for that message entry’s talker, which will correspond to its mouth
- A mapping of phonemes to cels within the view/loop from (2).
Step 3 is the most time-consuming to set up, but it only needs to be done once for a mouth view/loop. Once steps 2 and 3 are done, most lip-syncing can be accomplished just by select a message entry and clicking the “Quick Lip Sync” button.
Lip Sync resources¶
Sierra’s lip-sync resources (known as sync36) are fairly straightforward. They just consist of sequence of pairs of values. The first value in each pair is the tick count (60 ticks per sound), and the second value in each pair is the cel number in the mouth view/loop that should be used at that time.
Lip Sync dialog¶
Open the Lip Sync dialog by selecting a message entry with audio and clicking on the “Advanced” button. You should see something like this:
Associating a talker number with a view/loop.¶
In the Lip Sync dialog, you can use the Set View button to associate the given talker number with a view and loop. Once that is done, the mouth loop will appear in the dialog and in the message editor when a message entry with that talker number is selected. This only needs to be done once per talker.
This information is stored inside talker_to_view.ini in your game’s lipsync directory, and is only used by SCICompanion.
For a given view loop, we need to create a mapping of phonemes to the cels corresponding to the mouth shapes for that phoneme. To get an idea of which mouth shapes correspond to which phonemes, you can try an internet search for “lip sync mouth shapes”.
From the Lip Sync dialog, click the Edit button to edit the phoneme-to-cel mapping. By default, all the phonemes will be associated with cel 0. You can drag the phonemes alongside the different cels to complete the mapping.
You can test it be having the mouth read the text at the bottom of the dialog, which contains all US English phonemes.
Once you’re done, click Ok to accept the changes.
This information will be stored in a file in the lipsync folder of your game. It’s only used by SCICompanion, not the actual game. If you prefer to edit the phonemes manually, you can open and reload the phoneme file directly from the Lip Sync dialog.
Generating the Lip Sync Data¶
Once you have a view/loop and phoeneme mapping set up for a talker, you can generate Lip Sync data directly from the message editor (Quick Lip Sync). You can do it from the Lip Sync dialog, which has more options. The dialog will draw the phonemes directly on the audio waveform, and show you the words that the Lip Sync extraction functionality guessed.
You can also save the lip sync data to a file so you can edit it manually if necessary.
The more clearly you speak, the better the data generated will be. You can also choose to provide the message text to the lip sync data generator (the Use Text checkbox). Sometimes this will produce a better result. However, it may also make things worse and try to align the words to the wrong spoken words, generating data that is not aligned with the speech. Try both to see which is better for a particular recording.
When you are generating lip-sync data from speech you recorded in SCICompanion, the audio negative will be used. The same audio processing will be run on it, but the lip sync engine is provided with a 16-bit version of the result (even if you specified the final audio resource to be 8-bit). This generates significantly better results.
Currently SCICompanion only supports US English phoneme detection. As lip-syncing does not need to be 100% accurate, this may suffice for other languages and accents too. If there is enough demand however, this could be made extensible to other languages.
Using the sample mouth¶
If you just want to play around with lip-syncing without the overhead of drawing a mouth view and setting a phoneme mapping for it, you can use the sample mouth (and its associated phoneme mapping) that comes with SCICompanion. In the Lip Sync dialog, you can check Use Sample to do this.