KR20010038772A - Automatic and adaptive synchronization method of image frame using speech duration time in the system integrated with speech and face animation - Google Patents

Abstract

PURPOSE: An automatic application synchronism of motion picture frame using duration of voicing in sound and face animation integration system is provided to make an application and synchronism audio and motion picture of synthetic face by controlling duration carrying forward to next half-syllable if it is happened excess or deficiency time than original duration as result of frame creation when calculating average time of total frame, deciding number of frame to be created from half syllable. CONSTITUTION: A number of image frame by each syllables is calculated by dividing duration of voicing(D1) between initial sound and medium sound and duration of voicing(D2) between medium sound and final sound by a frame creating time(T0), considering parameter of mouth shape of initial sound, medium sound, final sound as parameter of key frame by each syllables when hangul text input. A parameter for each frame is calculated by selecting parameter from database of consonant, vowel mouth shape parameter according to consonant, vowel code, and database of expression change parameter, and database of head motion parameter, and by compensating(6-10) these according to the number of image frame. A face image by each frame is synthesized by mapping(6-11) texture in real time after changing shape model of individual face using the compensated parameter.

Description

Automatic and adaptive synchronization method of image frame using speech duration time in the system integrated with speech and face animation}

The present invention relates to a method of synchronizing voice and facial animation in an integrated system for synthesizing voice and facial animation at the same time, and more particularly, to automatically adjust the number of frames of facial animation according to voice duration, thereby synchronizing voice and facial animation. The present invention relates to an automatic adaptive synchronization method of a video frame using voice duration in an integrated voice and face animation system.

In general, a system integrating a synthetic voice and a synthetic facial video is constructed using a hardware configuration as shown in FIG. 1. After inputting an image or photo to the computer 1-3 using a camera 1-1 or a scanner 1-2, a synthesized voice and a synthesized facial video are generated, and the video is displayed on the display 1-4. Simultaneously display via speaker (1-5). Through this process, various facial animations are produced.

2 is a flowchart of a system integrating a synthetic voice and a synthetic facial video.

The facial video synthesizing unit 20 matches the shape model in the three-dimensional shape model DB (2-2) of the face to the face image in the face DB (2-1) (2-3), and then three-dimensionally the individual face. Obtain the shape model 2-4.

On the other hand, when the Hangul text (2-5) is input in the sentence interpretation unit 30, the sentence and word is detected through the phonological converter (2-6) (2-7), and the consonants and vowels are converted into codes for each syllable. The duration between the children and the vowel is given (2-8).

The adaptive synchronization unit 40 selects a parameter from the parameter DB (2-9) of the facial video synthesizing unit 20 according to the rule and the vowel code (2-10).

The parameter DB for facial animation constructed in the parameter DB 2-9 includes a consonant shape parameter DB, an expression change parameter DB, and a head motion parameter DB. In addition, each parameter has various types of parameters according to the movement of the facial muscles. The parameter for the mouth shape is selected by a combination of several parameters to express the shape of the mouth for the collection. The parameters for facial expressions and tofu motions are selected in a similar way, but they are selected separately from the ruler and vowels to enable facial expressions and tofu motions that match the meaning of Korean text.

The selected parameter is interpolated according to the number of video frames (2-12) to obtain a parameter for each frame. The number of frames of the image is calculated by considering the duration of the vowel / vowel of the audio and the frame generation time according to the computer model (2-11). Furthermore, the actual synthesis time is monitored for each frame, and the number of frames per syllable is adaptively calculated according to the time required. After deforming the shape model of the individual face using the interpolated parameters (2-13), the face image for each frame is synthesized and displayed by real-time texture mapping (2-14) (2-16).

On the other hand, the voice is synthesized in real time by the real-time voice synthesizer of the voice synthesizer 50 according to the self-vowel code (2-17), and then simultaneously displayed with the face video (2-18).

3A to 3C are diagrams for explaining a basic concept for synchronizing a synthesized voice and a synthesized facial video. The mouth shape for Hangeul is composed of Choseong, Neutral, Jongsung, or Choseong and Neutral. In this case, since the synchronization is performed in units of half-syllables, the two cases can be treated together, and then the basic concept of the synchronization will be explained using a syllable composed of primary, neutral, and final.

The change of mouth shape for syllable starts from the shape of mouth of the beginning, changes to the shape of the mouth of the vowel corresponding to the neutral, and ends with the shape of the mouth of the bell. That is, the mouth shape for the initial syllable, the neutral, and the final syllable for each syllable is determined by selecting parameters for the vowels and vowels from the previously established mouth shape parameter DB.

In the example of FIGS. 3A to 3C, it is assumed that the initial property is parameter 2, the neutral property is parameter 1 and parameter 2, and the final property is parameter 2 and parameter 3 selected. The strength of each parameter is stored in the mouth shape parameter DB. Frame generation time (T ₀ ) with the voice duration (D ₁ ) between the initial and the neutral and the duration (D ₂ ) between the neutral and the final with the mouth shape parameters of the initial, neutral, and final as the key frame parameters. After calculating the number of video frames by dividing by), the intensity of the parameter is linearly interpolated. The generation time (T ₀ ) of one frame depends on the computer CPU speed and the fast texture mapping method. The numbers in FIG. 3A represent the numbers of video frames.

4 is an example of a moving picture synthesized by interpolating mouth shape parameters of initial, neutral, and final.

In the fast texture mapping method, since the shape model of the face is usually composed of a set of triangles, texture mapping is performed in units of triangles. However, depending on the change in the shape of the mouth, the expression change, and the head movements, there may or may not be a triangle change compared to the previous frame. For the unchanged triangle, it is the same as the previous frame. For the sake of speed, the triangle does not perform texture mapping and uses the entire frame as it is. In this case, since the generation rate is different for each frame, it is difficult to accurately predict the generation rate of the frame. This is one of the main reasons why the voice and facial video are out of sync for long sentences.

In synchronizing audio and video, the difference in the frame rate of the video frame produced by the computer CPU, the rounding error when calculating the number of frames, the difference in the frame rate according to the shape of the mouth, the facial expression and the head, and the unit time display of the computer Because of the irregularities, in the case of long sentences, the synchronization is off. The difference in the video frame generation rate according to the computer CPU type is a phenomenon that occurs because the possibility of using a variety of different computers or processors, not limited to any one type of computer operating the system, the CPU varies depending on the type, Different CPUs produce different speeds. The rounding error in calculating the number of frames is an error that occurs because the syllable duration is not given as an integer multiple of one frame generation time. As described above, the generation time for each frame is one of the biggest reasons why the synchronization is off. Since the unit time display of the computer is not constant in microseconds and is slightly irregular, there is a problem that the current time cannot be known accurately. For this reason, there is a problem in synchronizing audio and video. Synchronization of voice and video for long sentences requires adaptive motivation to absorb these factors.

The present invention has been invented in view of the above-described circumstances, and the present invention relates to differences in frame generation time according to computer models and real-time texture mapping methods, irregularities in computer display time, and non-integral times with respect to the average frame generation speed of each syllable. It is an object of the present invention to provide an automatic adaptive synchronization method of a video frame using a voice duration in an integrated system of voice and face animation by adaptively correcting a synchronization error generated by the voice and face animation.

1 is a hardware configuration diagram of a system integrating synthesized voice and synthesized video;

2 is a flowchart of a real-time system incorporating synthesized voice and synthesized facial video;

3 is a basic conceptual diagram for synchronizing a voice and a video for one syllable;

4 is an example of changing the shape of the mouth of the video for the initial, neutral, and final of the syllables;

5 is a conceptual diagram of adaptive synchronization of voice and video,

6 is a flowchart of adaptive synchronization of video generation according to voice duration.

* Description of the symbols for the main parts of the drawings *

20-facial video synthesis, 30-sentence interpretation,

40-Adaptive Synchronizer, 50-Speech Synthesizer.

According to the present invention, when the Hangul text is input, the mouth shape parameters of the initial, neutral, and final voices are key frame parameters for each syllable, and the voice duration between the first and neutral voices (D ₁ ) and the voice duration between the neutral and the final voices ( Calculating the number of image frames for each syllable by dividing D ₂ ) by one frame generation time (T ₀ ), and selecting a parameter from a consonant shape parameter DB, a facial expression change parameter DB, and a head movement parameter DB according to a vowel / vowel code. The interpolation is performed to obtain a parameter for each frame by interpolating the number of image frames, to deform the shape model of the individual face using the interpolated parameters, and to synthesize a face image for each frame by real-time texture mapping.

Hereinafter, the configuration and operation of one embodiment of the present invention will be described in detail with reference to the accompanying drawings.

5A to 5D are conceptual views for explaining the adaptive synchronization employed in the present invention. In FIG. 5A, D ₁ denotes a duration between the supernova and neutral, and D ₂ denotes a duration between the neutral and the final. 5A divides this duration by the average frame rate T ₀ to calculate the initial frame number N ₁ . In other words,

N ₁ = [D ₁ / T ₀ ] (1)

to be. Where [] indicates rounding. T ₀ is the average generation time of precomputed frames. R ₁ means excess or short time after generation of the integer frame. This is caused by rounding the number of frames.

5B recalculates the number of frames to be generated later after generating the first frame. In other words,

N ₂ = [(D ₁ -T ₀ ') / T ₁ ] (2)

to be. Where T ₀ ′ is the time when the first frame is actually generated. T ₁ is a generation time of the first frame equal to T ₀ ′. If T ₀ 'is shorter than expected, frame N ₂ may increase, and if longer than expected, N ₂ may decrease.

5C adjusts the number of frames after generating the second frame. For adaptive synchronization, at every frame generation, the average time of all frames generated from the beginning of the system is recalculated. In other words,

(3)

N is the number of frames generated from system startup to the present. Calculate the excess short time R _N after generating an integer frame. This is not calculated in advance, but after the last frame of the half syllable is calculated as in FIG. 5D. In other words,

(4)

to be.

Thus, when frame generation is finished for the duration D ₁ between the initial and the neutral, adjust D ₂ . In other words,

(5)

to be. D ₂ ′ is the adjusted D ₂ , given by the duration of the neutral and the final. In general, if a half-syllable frame is generated as a result of excess or shortage of the original duration, the duration is slightly adjusted to carry over to the next half-syllable. As the number of frames is adaptively adjusted, the parameters are interpolated in consideration of the generation time from the beginning of the half syllable to the present. In other words,

(6)

to be. Where A _N is the parameter strength of the frame to be generated currently. A ₀ , A _l is the strength of the parameter at the beginning and end of the i-th syllable. Q _i is the synthesis time from the start of the i-th syllable to the currently generated frame. D _i 'is the adjusted duration of the i th half syllable.

This adaptive synchronization sequence is performed in the same manner as in FIG. With adaptive synchronization, the difference between the voice duration and the video frame generation time does not exceed T _N / 2 for each syllable.

(1) Start up and start the system (step 6-1).

(2) Read the average generation time T _O of the frame when the program was most recently used from the disc (step 6-2). The first time a program is ported to a computer, it is the average frame generation time of the computer before porting. T _o is automatically adapted to the frame generation period T _N of the computer in use as the program is used.

(3) Read the half syllable number M of the input text (step 6-2).

(4) Initialize the synthesized total number of frames N, the half-syllable order i, the cumulative duration D of the half-syllable, and the synthesized oversufficiency time R ₀ (step 6-3).

(5) Check synthesis start time P ₀ (steps 6-4).

(6) The duration D _i of the i th syllable is read (steps 6-5).

(7) Accumulate duration D _i (steps 6-6). to be.

(8) The duration D _i of the i-th half syllable is adjusted to the oversufficiency time in the i-th half-syllable (steps 6-7). In other words, to be.

(9) Initialize the frame order j for each syllable (steps 6-8).

(10) The start time P _i of the i-th syllable is checked (steps 6-9).

(11) Interpolate the parameters using Equation (6).

(12) The face shape model is transformed according to the interpolated parameters, and texture mapping is performed in real time (steps 6-11).

(13) Check the time P _i 'after synthesis (steps 6-12).

(14) Time required during frame synthesis of the i-th half-syllable from the beginning of synthesis (6-6-13).

(15) The cumulative duration D up to the i-th syllable is compared with the duration Q during synthesis (steps 6-14). If so, go to the next step to continue synthesis. Otherwise, go to step 6-18 for the next half-syllable synthesis.

(16) The total number of generated frames N and the number of frames j for each syllable are increased by one (steps 6-15).

(17) An average generation time T _N per frame is calculated by dividing the generation time Q of the frames so far from the system start by the total number of frames N (steps 6-16). In other words, to be. By calculating the average time per frame every frame, it automatically adapts to the speed of the computer's CPU and real-time texture mapping.

(18) Synthesis time of frame generated from the beginning of the i-th syllable syllable to the present (6-6-17).

(19) Compare the number of half syllables M of the input text with the current number of syllables i. Then continue to next step (6-6-21) to continue synthesis (steps 6-18).

(20) Remaining time of the i th syllable To calculate. Proceed to step 5 (steps 6-19).

(21) Increment the half-syllable number i by 1 (steps 6-20).

(22) The average generation time T _N of frames is stored (steps 6-21).

(23) Shut down the system (steps 6-22).

As described above, the present invention calculates the average time of the entire frame, determines the number of frames to be generated in the half-syllable, and adjusts the duration by carrying over to the next half-syllable when there is an oversufficiency time than the original duration of the half-syllable as a result of the frame generation. By doing so, it is possible to synchronize the video and voice of the synthesized face.

Claims (3)

When Hangul text is input, each syllable is composed of mouth shape parameters of initial, neutral, and final voice as key frame parameters, and voice duration between first and neutral (D ₁ ) and voice duration between neutral and final (D ₂ ). Calculating the number of image frames for each syllable by dividing the information into one frame generation time (T ₀ ), selecting a parameter from a consonant shape parameter DB, a facial expression change parameter DB, and a head movement parameter DB according to a consonant / vowel code. Integrating voice and face animation, which consists of interpolating according to the number of frames to obtain parameters for each frame, deforming the shape model of the individual face using the interpolated parameters, and then composing face images for each frame by real-time texture mapping. Automatic Adaptive Synchronization of Video Frames Using Voice Duration in System. The method of claim 1, wherein the calculating of the number of image frames for each syllable is performed by dividing the duration D ₁ between the initial and the neutral by the average generation time T ₀ of a precomputed frame, and rounding the rest. Calculating N ₁ ) and recalculating the number of frames N ₂ by dividing the remaining duration D ₁ -T ₀ ′ after generation of the first frame by the actual generation time T ₁ ′ of the first frame. After adjusting the number of frames by repeating the step of renumbering the number of frames after generating the second frame, the average time T _N of the entire frames generated from the initial system is repeated for each frame generation for adaptive synchronization. Repeating the step of reproducing the frame number and reproducing the frame number to generate the last frame of the half syllable ), After the generation of the frame for the duration (D ₁ ) between the initial and the neutral, the excess shortage time (R _N ) is added to the duration between the neutral and the final (D ₂ ) By adjusting the duration (D ₂ '), and then performing the same steps as the frame number generation step between the initial and the neutral, if a half-syllable frame is generated as a result of over-short time than the original duration, it is carried over to the next half-syllable. An automatic adaptive synchronization method of a video frame using a voice duration in an integrated voice and face animation system characterized by adjusting the duration. The method of claim 1, wherein the interpolation step of the parameters for adaptive synchronization is based on a difference between the parameter strengths A ₁ -A _o between the start parameter intensity A ₁ of the i-th half-syllable and the end parameter intensity A _o . Multiply the synthesis time (Q _i ) from the beginning of the i-sixth syllable to the currently generated frame by the sum (Q _i + T _N ) of the average time (T _N ) of all frames created from the beginning of the system, and multiply it by the i-th Voice and face characterized by dividing by the adjusted duration (D _i ') of the syllables and adding the parameter strength (A _o ) at the end of the i-th syllable to interpolate as the parameter strength of the frame to be generated now. Automatic Adaptive Synchronization of Video Frames Using Voice Duration in Animation Integration System.