KR100300962B1 - Lip sync method and apparatus for speech synthesis - Google Patents

Abstract

The present invention discloses a lip sync method and apparatus for speech synthesis. The lip-sync method for speech synthesis according to the present invention, in which still image data representing a human face, shape model data, and arbitrary text are input to lip-sync the speech synthesis result from the text on the face, the (a) voice Obtaining chords and duration information for each syllable in the syllable unit at least one syllable from the synthesis result, (b) allocating durations for each phoneme within each syllable duration; (c) generating at least one video key frame representing a mouth shape for each of the phonemes within each phoneme duration, with reference to a predetermined reference table defining mouth shape information for each phoneme, (d) adjacent By interpolating between key frames, a predetermined number of consecutive intermediate frames are created to generate a video having a mouth shape synchronized with the speech synthesis result. Synthesizing the video data, and (e) characterized in that it comprises a step of outputting in synchronization with the synthesized speech resulting with a synthesized video.

Description

Lip sync method and apparatus for speech synthesis

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to lip-syncronization, and to a lip-sync method and apparatus for speech synthesis.

The research on face synthesis, which has been actively researched since the mid-1970s since the late 1980s, is now recognized as important in the field of MPEG4 standardization. In addition, in recent years, a lot of researches have been conducted on how to implement a human face through a lip sync between a face synthesized with computer graphics and a human voice.

However, many of the lip synch technologies studied in the prior art are mainly dependent on English pronunciation, and thus, there are many differences from the lip synch of Korean. At present, there is little research on this, and since the implementation environment has been implemented in a high performance workstation class computer with almost no limitation on execution speed, it is not easy to apply to a general computer.

The present invention implements a virtual agent applying a lip sync technology with TTS to a human face synthesized with computer graphics, so that news can be broadcast in Korean in the Internet or cyber virtual space.

SUMMARY OF THE INVENTION An object of the present invention is to provide a lip sync method suitable for Korean pronunciation structure, thereby enabling a virtual agent applying a lip sync technique with speech synthesis results to a human graphic synthesized face. To provide.

Another object of the present invention is to provide a lip sync device for speech synthesis that performs the lip sync method.

1 is a flowchart illustrating a lip sync method for speech synthesis according to the present invention.

2 is a block diagram of a lip sync device for speech synthesis according to the present invention.

3 is a diagram illustrating a duration allocation and synchronization method for each phoneme.

In order to achieve the above object, a lip-sync method for speech synthesis according to the present invention, which lip-syncs the speech synthesis result from the text to the face by inputting still image data, shape model data representing a human face, and arbitrary text. silver,

(a) obtaining a code and syllable duration information for at least one syllable that constitutes a syllable in syllable units from a speech synthesis result, and (b) obtaining a phoneme duration for each phoneme within a syllable duration. (C) generating at least one video key frame representing a mouth shape for each of the phonemes within each phoneme duration by referring to a predetermined reference table that defines mouth shape information for each phoneme, (d) generating a moving picture having a mouth shape synchronized with the result of voice synthesis by interpolating a predetermined number of intermediate frames by interpolation between adjacent key frames, and synthesizing it into still image data; and (e) combining the synthesized video with voice. Outputting in synchronization with the synthesis result.

In order to achieve the above object, a lip sync for speech synthesis according to the present invention, which lip-syncs the speech synthesis result from the text to the face by inputting still image data, shape model data representing a human face, and arbitrary text. The device,

An input unit for obtaining codes and syllable duration information of at least syllables in a syllable unit from the synthesized voice, and a syllable analyzer for allocating durations of each phoneme within syllable durations; Key frame generation unit for generating at least one key frame representing a mouth shape for each of the phonemes within each phoneme duration by referring to a predetermined reference table that defines mouth shape information for each phoneme, interpolation between adjacent key frames. To generate a predetermined number of consecutive intermediate frames to generate a video having a mouth shape synchronized with the speech synthesis result, and to output the video synthesizer synthesizing the still image data and to output the synthesized video in synchronization with the speech synthesis result. Equipped.

Hereinafter, with reference to the accompanying drawings, the configuration and operation of the lip sync method for speech synthesis according to the present invention, and the apparatus will be described as follows.

1 is a flowchart illustrating a lip sync method for speech synthesis according to the present invention.

For lip syncing between the face image synthesized with computer graphics and voice, first, a still image data (hereinafter, simply referred to as "face data") representing a face of a person is inputted, and arbitrary text, that is, text to be synthesized by voice is input. (Step 100). Here, the face data becomes a basic screen for realizing the speaker's face, and the present invention synthesizes a moving picture having a mouth shape on the basic screen to realize a lip sync with voice. When text is input in step 100, it is subjected to a speech synthesis process.

From the speech synthesis result of the text, code and duration information for each syllable of the syllables in one syllable unit are obtained (step 102). Here, the codes are specifically codes for initial, neutral, final, and blank, and the duration of each syllable represents the elapsed time from the start of one syllable to the end. In addition, by obtaining the start time information of the voice from the result of speech synthesis, it is applied to later synchronously outputting the shape of the face and the voice.

After step 102, the phoneme duration for each phoneme is allocated within the duration for each syllable (step 104). That is, the duration of each syllable allocated in one syllable unit is allocated to each of the phonemes constituting one syllable at a predetermined ratio. Here, the duration of each phoneme is divided into at least one syllable into a ratio of neutral> primary> species by dividing the syllables into a consonant, a neutral, and a consonant. Consonants are assigned to the ratio of lip sounds> non lip sounds to be divided into lip sounds and non lip sounds (described in detail below).

After step 104, a key frame for each of the phonemes is generated within each phoneme duration by referring to a visme (visual phoneme) table that defines mouth shape information for each phoneme. Step 106). The key frame means a scene that is the center of computer animation. In the present invention, the key frame refers to an image having a mouth shape that is the center of a phoneme. The bisms table defines, for each of the phonemes in the minimally classified Korean phoneme pattern, predetermined mouth shape information including an action unit intensity indicating at least the degree of mouth opening (described in detail below).

After step 106, intermediate frames are created within each phoneme duration given to smoothly concatenate adjacent key frames, that is, for natural mouth shape. That is, a predetermined number of consecutive intermediate frames are generated by interpolation between adjacent key frames to generate a moving picture having a mouth shape synchronized with voice based on the voice synthesis result, and synthesized into face data input in step 100 (step 108). step). Finally, the lip sync is realized by outputting the synthesized moving picture in synchronization with audio (step 110).

As described above, in order to perform the lip sync method according to the present invention, the following matters should be defined. 1. Define a mouth-shaped bismuth table for Korean pronunciation forms. 2. Define how to assign durations to the phonemes that make up a syllable. 3. Define how to synchronize between each phoneme and the assigned duration. This will be described in detail as follows.

1. Korean Bism Table Definition

A method of defining a mouth-shaped Korean visme table for Korean pronunciation forms will be described.

In the 1970s, psychologist Ekman defined the Facial Action Coding System (FACS), a classification of facial expressions. Based on this, the human face synthesized by computer graphics may express the shape of a mouth by realizing an animation for each action unit (AU), which is a basic unit of FACS. The bismuth table defines mouth shapes for each phoneme. In English, 45 phonemes are divided into 18 mouth shapes. In the present invention, Korean phonemes are preferably classified into a total of 12 patterns. (Table 1).

Specifically, a method of classifying Korean phonemes into 12 bismuth patterns will be described.

1-1. collection

The vowels are classified into a total of nine bismuth patterns. Vowels are largely classified into two types, one of which is represented by one mouth shape for one vowel, and the other is when two or more mouth shapes are successively represented by mouth shapes.

1-1-1. Basic collection

It is represented by one mouth shape per vowel, and is classified into nine kinds as follows. ㅏ, ㅓ, ㅗ, TT, ㅡ, ㅣ, ㅐ, ㅔ, ㅚ

1-1-2. Combination collection

Each vowel is represented as a series of two or more basic vowels. The vowels are as follows. These vowels are not classified as patterns because they are represented by the basic vowel. ㅑ (ㅣ + ㅏ), ㅕ (ㅣ + ㅓ), ㅛ (ㅣ + ㅗ), ㅠ (ㅣ + TT), ㅘ (ㅗ + ㅏ), ㅙ (ㅗ + ㅐ), ㅝ (TT + ㅓ), ㅞ (ㅜ + ㅔ), ㅟ (ㅜ + ㅣ), ㅢ (ㅡ + ㅣ)

1-2. Consonant

Consonants are categorized as lip-closing and non-lip-sounds, because in Korean, almost no other lip sounds have a significant effect on the shape of the mouth.

1-2-1. Lips

There are four consonants as follows, and they are classified into one bismuth pattern. ㅁ, ㅂ, ㅃ, ㅁ

1-2-2. Rain lips

As all the consonants other than the lip sounds, they are classified as one bismuth pattern.

1-3. Blank

Spaces are spaces that correspond to spaces or commas in a given sentence. In general, spaces retain the shape of the mouth of the previous phoneme. You can do it. This space is also classified as a bismuth pattern.

TABLE 1

Table 1 shows a preferred Korean bismuth table according to the present invention. For the 12 patterns, mouth shape information including the action unit (AU) combination according to FACS and the action unit strength indicating the degree of opening the mouth are defined. Since an action unit is a basic unit of FACS, as shown in Table 1, one pattern consists of a plurality of action units.

2. How to allocate duration for each phoneme

One syllable in Korean is composed of only primary and neutral, and sometimes composed of primary, neutral, and final, where the neutral vowel is divided into short vowels and middle vowels, and consonants are divided into lip and non-lip sounds. Accordingly, one syllable is preferably divided into a total of four durations.

The shape of the speaker's mouth is largely determined by vowels. Therefore, the vowel duration is assigned the most in one syllable. In the case of a middle vowel, it can be treated as two consecutive short vowels. In this case, the rear vowels dominate the shape of the mouth, so that the duration of the front vowels is short and the duration of the rear vowels is long. In the case of consonants, on the other hand, durations are assigned with the shortest last and the shortest first. At this time, the biting time of bite, bite, bite, and bite should be allotted a little longer to ensure that the bite is seen. Each case is summarized as follows.

2-1. When a syllable is composed of both primary and neutral

2-1-1. Non-lip and short vowels

Allocates a total duration of 1/3 first and 2/3 neutral.

2-1-2. Non-lip sounds and heavy vowels

The total duration is assigned to the initial 1/3, the front vowel 1/3, and the rear vowel 1/3.

2-1-3. Lip sounds and short vowels

The total duration is assigned to 1/2 initial and 1/2 neutral.

2-1-4. Lip sounds and heavy vowels

The total duration is assigned to initial 1/2, front vowel 1/6, and rear vowel 3/6.

2-2. When a syllable is composed of primary, neutral, and final

Finality is assigned to 1/5 of the total duration, and initial and neutral are assigned as follows.

2-1-1. Non-lip and short vowels

The total duration is assigned to 1/5 first and 3/5 neutral.

2-1-2. Non-lip sounds and heavy vowels

The total duration is assigned to the initial 1/5, the front vowel 1/5, and the rear vowel 2/5.

2-1-3. Lip sounds and short vowels

Allot durations are assigned to initial 1.5 / 5 and neutral 2.5 / 5.

2-1-4. Lip sounds and heavy vowels

The total duration is assigned to the initial 1.5 / 5, the front vowel 1/5, and the rear vowel 1.5 / 5.

3. Synchronization method between each phoneme and assigned duration

As described above with reference to FIG. 1, the shape of the synthesized face's mouth and the voice are basically generated by creating an intermediate frame by interpolation between key frames according to the bismuth pattern defined above within the duration of each phoneme. Implemented by creating However, it is difficult to realize accurate synchronization due to the difference in the video frame generation speed according to the computer model, the rounding error when calculating the number of frames, the generation time difference per frame, and the error of the computer timer.

Therefore, in the present invention, the following matters are considered and applied to the synchronization method.

All phonemes with the same start time as the speech resulting from the speech synthesis must produce at least one frame (ie, a keyframe). If this is not followed, it does not look like a person speaks.

-Every frame is generated, the frame rate must be calculated and reflected continuously in the number of frames per duration of the next phone.

-When interpolating between phonemes, that is, when interpolating to the next keyframe starting with the keyframe of the phoneme, it should change slowly at the beginning of the duration and change as soon as the end of the duration. Given that light is faster than sound, you should ensure that mouth-shaped keyframes appear early in the phonetic pronunciation, so that you can feel natural as people say. In other words, when it is delayed, it is clearly revealed that the mouth shape is slower than words, and it feels very awkward.

The intermediate frame may be interpolated using Equation 1 below.

[Equation 1]

In Equation 1, f _i , f _{i + 1} denotes the action unit strength of each adjacent key frame, f _j denotes the action unit strength of the intermediate frame between f _i and f _{i + 1} , and k denotes the intermediate frame index. , N respectively represent the total number of frame generations that can be generated within the duration of one phoneme. When the action unit strength of the intermediate frame is obtained according to Equation 1, the intermediate frame can be easily obtained by changing the mouth shape of the previous key frame by the obtained action unit strength.

2 is a block diagram illustrating a lip sync device for speech synthesis according to the present invention.

The face synthesizer 200 and the voice synthesizer (TTS) 260 are largely comprised, and the face synthesizer 200 includes a face data input unit 210, a face data loader 220, a video synthesizer 230, It is composed of a text input unit 250 and a synchronization control unit 270. Here, the video synthesizer 230 that actually realizes the lip sync is a syllable analyzer 232, a key frame generator 234, a face video synthesizer 236, a screen output unit 238, and a bismuth table 240. It consists of.

Referring to FIG. 2, the face data input unit 210 inputs vertices and the like necessary for representing a face of a predetermined person through computer graphics. The face data loader 220 performs a face composition initialization operation that actually expresses face data input from the face data input unit 210 on the screen using a 3D coordinate file and a connection relation file. Meanwhile, the text input unit 250 inputs a given text in accordance with the synthesized face, and the text to speech portion (TTS) 260 synthesizes the text.

The video synthesizing unit 230 inputs face data from the face data loader 220, and codes for the phonemes that constitute at least syllables in one syllable unit, that is, initial, neutral, final, and blank from the voice synthesizing unit 260. The duration information for each syllable is obtained through the synchronization controller 270. The synchronization controller 270 transmits the above-described data from the voice synthesizer 260 to the video synthesizer 230 in accordance with a predetermined synchronization in units of syllables.

In detail, the video synthesizing unit 230, the syllable analyzer 232, through the synchronization control unit 270, obtains code and duration information for each syllable of the phonemes that constitute at least one syllable in syllable units from the speech synthesis result. Get The syllable analyzer 232 allocates a phoneme duration for each phoneme within a syllable duration. The method for allocating phoneme durations is as described above.

The key frame generation unit 234 refers to at least one key frame indicating a mouth shape for each of the phonemes within the duration of each phoneme with reference to the bismuth table 240 that pre-defines mouth shape information for the Korean pronunciation form. Create The bismuth table 240 is shown in Table 1 as a preferred example.

The facial video synthesizer 236 further generates a predetermined number of consecutive intermediate frames by interpolation between adjacent key frames with the key frames generated by the key frame generator 234. The key frame and the intermediate frame thus produced eventually produce a moving picture having a mouth shape synchronized with the speech synthesis result. The facial video synthesizing unit 236 synthesizes the generated moving image into face data again, and the screen output unit 238 realizes the lip syncing by outputting the synthesized moving image in synchronization with the voice synthesizing result of the voice synthesizing unit 260. .

3 is a diagram illustrating a method of allocating and synchronizing durations of phonemes, in which the horizontal side represents the duration of one syllable and the vertical side represents the change in the intensity of the action unit (AU).

The phonemes constituting one syllable within the duration of one syllable, that is, the primary, the neutral, and the final star, are each assigned a predetermined duration. When synchronization is made between each phoneme and the assigned duration, each phoneme has an action unit intensity change within each phoneme duration. As shown in FIG. 3, within each duration of each phoneme, the action unit strength of each phoneme is initially large and gradually decreases backwards, which is determined by ρ in Equation 1.

As described above, the lip sync method and apparatus for speech synthesis according to the present invention can implement a virtual character agent that speaks like a human through the present invention by providing a lip sync suitable for the Korean pronunciation structure. Is implemented, ultimately, a human interface that seems to be a human face at the interface between a person and a computer can be realized. It can be applied to many different fields such as cyber news desk, unmanned guidance system, virtual actor, video telecommunications field.

Claims (10)

A lip sync method for speech synthesis, comprising: inputting still image data, a shape model data representing a face of a person, and arbitrary text to lip-sync the speech synthesis result from the text on the face; (a) obtaining, from the speech synthesis result, a code and duration information for each syllable for phonemes constituting at least one syllable in one syllable unit; (b) allocating phoneme durations for each of the phonemes within each syllable duration; (c) generating at least one video key frame representing a mouth shape for each of the phonemes within each phoneme duration by referring to a predetermined reference table that defines mouth shape information for each phoneme; (d) generating a moving image having a mouth shape synchronized with the speech synthesis result by generating a predetermined number of consecutive intermediate frames by interpolation between adjacent key frames, and synthesizing the still image data; And and (e) outputting the synthesized video in synchronization with the speech synthesis result. The method of claim 1, wherein the duration of each phoneme in step (b) is divided into at least one syllable into initial, neutral, and final in a ratio of neutral> initial> final, and when there is a middle vowel, The vowels in the front and the vowels in the back are divided into vowels in the rear, and the vowels in the front are separated. Lip sync method for Korean speech synthesis, characterized in that the. The method of claim 1, wherein the predetermined reference table comprises: a predetermined number of basic vowels representing one or more vowels of Korean phonemes, spaces, and consonants divided into lip sounds and non-lip sounds with which lips are closed. A lip sync for Korean speech synthesis, characterized by defining predetermined mouth shape information including an action unit strength indicating at least a mouth opening level for each of the phonemes included in the Korean phoneme pattern. Way. 4. The method of claim 3, wherein in the step (d), the intermediate frame is interpolated using the following equation, wherein f _i , f _{i + 1} are the action unit strengths of each of the adjacent key frames, and f _j is f _i. Lip sync for Korean speech synthesis, characterized in that the action unit strength of the intermediate frame between and f _{i + 1} , k is the intermediate frame index, and N is the total number of frame generations that can be generated within the duration of one phoneme. Way. [Equation 1] The method according to claim 1 or 4, wherein when generating an intermediate frame in the step (d), at least a frame generation rate is calculated every time each frame is generated and reflected in the number of frame generations per duration of the next phoneme. Lip sync method for Korean speech synthesis, characterized in that. A lip sync device for speech synthesis in which a still image data representing a human face, shape model data, and arbitrary text are input to lip-sync a speech synthesis result from the text on the face. An input unit for obtaining codes and duration information for each syllable, for at least syllables; A syllable analyzer for allocating a phoneme duration for each of the phonemes within the syllable duration; A key frame generation unit generating at least one key frame representing a mouth shape for each of the phonemes within a duration of each phoneme with reference to a predetermined reference table defining mouth shape information for each phoneme; A moving picture synthesizing unit which generates a moving picture having a mouth shape synchronized with the speech synthesis result by making a predetermined number of consecutive intermediate frames by interpolation between adjacent key frames, and synthesizing the moving picture data into the still picture data; And a screen output unit configured to output a synthesized video in synchronization with the speech synthesis result. The method of claim 6, wherein the syllable analysis unit divides at least one syllable into a consonant, neutral, and tertiary to have a ratio of neutral> primary> final, and there are two consecutive single vowels in the case of a middle vowel. The phoneme is divided into vowels, and the vowels in the rear> the vowels in the front, and the consonants are divided into lip sounds and non-lip sounds, and the lip sounds> non-lip sounds are assigned the duration of each phoneme. Lip sync device for Korean speech synthesis, characterized in that. The method of claim 6, wherein the predetermined reference table comprises: a predetermined number of basic vowels representing one or more vowels of Korean phonemes, spaces and consonants divided into lip sounds and non-lip sounds with which lips are closed. A lip sync for Korean speech synthesis, characterized by defining predetermined mouth shape information including an action unit strength indicating at least a mouth opening level for each of the phonemes included in the Korean phoneme pattern. Device. The method of claim 8, wherein the video synthesizing unit interpolates the intermediate frame by using the following equation, wherein f _i , f _{i + 1} denotes an action unit strength of each of adjacent key frames, and f _j equals f _i . lip-sync device for Korean speech synthesis, characterized in that the action unit strength of the intermediate frame between _{i + 1} , k represents the intermediate frame index, and N represents the total number of frame generations that can be generated within the duration of one phoneme. . [Equation 1] The method of claim 6 or 9, wherein the video synthesis unit, Every time the intermediate frame is generated, at least a frame generation rate is calculated and reflected in the number of frame generations per duration of the next phoneme.