KR20210094422A - Method and apparatus for generating speech animation - Google Patents

Abstract

Disclosed are a method and an apparatus for generating a speech animation including an avatar uttering an input script in consideration of the characteristic of a language and a phonological phenomenon. The embodiment includes the steps of: receiving a script and a voice signal corresponding thereto; generating grapheme sequence and timing information; generating a phoneme sequence by applying a simultaneous articulation rule determined according to adjacent final and initial consonants included in the phoneme sequence, and generating a speech animation by changing the shape of the mouth of the avatar and the shape of the tongue of the avatar based on the phoneme sequence and timing information.

Description

METHOD AND APPARATUS FOR GENERATING SPEECH ANIMATION

The embodiments below relate to a method and apparatus for generating a speech animation.

Speech animation technology relates to a technology for generating animation by deforming a shape of a character's mouth according to audio. As speech animation technology is used in the entire cultural industry such as movies, animation, games, and education, speech animation technology is required to produce the same effect as if a character is actually speaking in response to audio. To this end, it is necessary to develop a technology that generates animations so that a character's mouth moves in sync with audio, and a technology that generates animations so that the shape of a character's mouth moves similarly to the movement of a speech organ when a person speaks. do.

Embodiments may provide a technique for generating an animation including an avatar uttering Korean based on a script and a voice signal corresponding thereto.

In addition, the embodiments may provide a technique for changing the shape of the mouth and the tongue of an avatar speaking Korean by reflecting a phonological phenomenon in which the pronunciation of a grapheme is changed according to an adjacent grapheme in a phonetic phrase.

A method of generating a speech animation according to one side includes: receiving a script and a voice signal corresponding to the script; generating a grapheme sequence including at least one consonant and at least one vowel based on the script; generating timing information including timings of each of the at least one consonant and the at least one vowel included in the grapheme sequence based on the speech signal; extracting adjacent final consonants and leading consonants from the grapheme sequence to do; determining at least one co-articulation rule and an order of the at least one co-articulation rule based on the final consonant and the leading consonant; generating a phoneme sequence by updating the phoneme sequence by applying the at least one simultaneous articulation rule to the final consonant and the leading consonant according to the order; and generating a speech animation by modifying a mouth shape of the avatar and a tongue shape of the avatar based on the timing information and the phoneme sequence.

The determining of the at least one co-articulation rule and the order of the at least one co-articulation rule may include selecting at least one co-articulation rule corresponding to the final consonant when the leading consonant does not correspond to a predetermined consonant. step; and when the leading consonant corresponds to the predetermined consonant, selecting at least one simultaneous articulation rule corresponding to the leading consonant.

The step of extracting adjacent final consonants and leading consonants from the grapheme sequence may include: extracting whether the corresponding grapheme is a consonant and the position of the corresponding consonant while sequentially reading the graphes included in the corresponding phoneme phrase; and when the position of the extracted consonant is the final consonant, extracting the next consonant of the extracted consonant.

The generating of the grapheme sequence may include: separating the script into at least one semantic unit based on a dictionary in which semantic units of a language corresponding to the script are listed; The method may include sequentially arranging at least one grapheme included in the semantic unit for each of the at least one semantic unit.

The generating of the speech animation may include: mapping each of the phonemes included in the phoneme sequence to a corresponding mouth shape and a tongue shape; and transforming the avatar's mouth shape and the avatar's tongue shape into a mouth shape and a tongue shape mapped to each phoneme based on the timing information, thereby generating a speech animation.

The generating of the timing information may include: displaying a specific section as a pause section according to a predetermined criterion in the voice signal; and generating timing information corresponding to the rest period.

The generating of the speech animation may include: mapping the pause section to a corresponding mouth shape; and generating a speech animation by transforming the mouth shape of the avatar into a mouth shape mapped to the rest section based on the timing information corresponding to the pause section.

The specific section according to the predetermined criterion may correspond to a section in which a signal equal to or less than the strength of a predetermined voice signal is maintained for a predetermined time.

The generating of the grapheme sequence may further include generating the grapheme sequence by mapping at least one grapheme included in the script to a predetermined code.

The simultaneous articulation rule may correspond to a rule for changing the final consonant and the leading consonant located next to the final consonant according to pronunciation.

An apparatus for generating a speech animation according to one side receives a script and a voice signal corresponding to the script, and receives timing information including the timing of each of the grapheme sequence and at least one consonant and at least one vowel included in the grapheme sequence. generating, extracting a closing consonant and a leading consonant adjacent to each other from the grapheme sequence to determine the order of at least one coarticulation rule and the at least one coarticulation rule based on the final consonant and the leading consonant, according to the order By applying the at least one coarticulation rule to the final consonant and the leading consonant, the phoneme sequence is updated to generate a phoneme sequence, and based on the timing information and the phoneme sequence, the avatar's mouth shape and the avatar's tongue at least one processor for generating a speech animation by deforming a shape, and a memory for storing the grapheme sequence, the phoneme sequence, the timing information, and mapping information of a tongue shape and a mouth shape corresponding to the at least one phoneme; do.

In order to determine the order of the at least one co-articulation rule and the at least one co-articulation rule, the processor is configured to, when the leading consonant does not correspond to a predetermined consonant, at least one co-articulation rule corresponding to the final consonant. , and when the leading consonant corresponds to the predetermined consonant, at least one simultaneous articulation rule corresponding to the leading consonant may be selected.

The processor sequentially reads the graphes included in the corresponding phoneme phrase in order to extract the final and leading consonants adjacent to each other from the grapheme sequence, and extracts whether the corresponding grapheme is a consonant and the position of the corresponding consonant, and the extraction When the position of the consonant is the final consonant, the next consonant of the extracted consonant may be extracted.

In order to generate the grapheme sequence, the processor separates the script into at least one semantic unit based on a dictionary in which semantic units of a language corresponding to the script are listed, and the semantic unit for each of the at least one semantic unit. At least one grapheme included in may be listed in order.

The processor maps each of the phonemes included in the phoneme sequence to a corresponding mouth shape and tongue shape to generate the speech animation, and sets the avatar's mouth shape and the avatar's tongue shape based on the timing information. By transforming the shape of a mouth and a shape of a tongue mapped to each phoneme, a speech animation can be created.

The script may include a script including at least one syllable written in Korean, and the voice signal may include a voice signal uttering the script in Korean.

In order to generate the timing information, the processor marks a specific section as a pause section in the voice signal according to a predetermined criterion, generates timing information corresponding to the pause section, and generates the speech animation. In order to do this, a speech animation can be created by mapping the pause section to a corresponding mouth shape, and transforming the mouth shape of the avatar into a mouth shape mapped to the pause section based on timing information corresponding to the pause section. there is.

The processor may generate the grapheme sequence by mapping at least one grapheme included in the script to a predetermined code to generate the grapheme sequence.

1 is a diagram illustrating an overall operation flow of a speech animation generation system according to an embodiment.
2 is a diagram illustrating an example of a customized dictionary input to an acoustic model according to an embodiment.
3 is a diagram illustrating an example of a result of a forced voice alignment operation according to an embodiment;
4 is a diagram illustrating an example in which simultaneous articulation rules are determined differently according to grapheme according to an embodiment;
5 to 7 are diagrams illustrating tongue shapes and mouth shapes corresponding to phonemes of Korean according to an exemplary embodiment.
8 is a diagram illustrating a mouth shape corresponding to a pause according to an embodiment;
9 is a diagram illustrating a method of generating a speech animation according to an embodiment.

Specific structural or functional descriptions disclosed in this specification are merely illustrative for the purpose of describing embodiments according to technical concepts, and the embodiments may be embodied in various other forms and are limited to the embodiments described herein. doesn't happen

Terms such as first or second may be used to describe various elements, but these terms should be understood only for the purpose of distinguishing one element from another element. For example, a first component may be termed a second component, and similarly, a second component may also be termed a first component.

When a component is referred to as being “connected” or “connected” to another component, it is understood that the other component may be directly connected or connected to the other component, but other components may exist in between. it should be On the other hand, when it is said that a certain element is "directly connected" or "directly connected" to another element, it should be understood that no other element is present in the middle. Expressions describing the relationship between elements, for example, "between" and "between" or "neighboring to" and "directly adjacent to", etc. should be interpreted similarly.

The singular expression includes the plural expression unless the context clearly dictates otherwise. In this specification, terms such as "comprise" or "have" are intended to designate that the described feature, number, step, operation, component, part, or combination thereof exists, and includes one or more other features or numbers, It should be understood that the possibility of the presence or addition of steps, operations, components, parts or combinations thereof is not precluded in advance.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art. Terms such as those defined in a commonly used dictionary should be interpreted as having a meaning consistent with the meaning in the context of the related art, and should not be interpreted in an ideal or excessively formal meaning unless explicitly defined in the present specification. does not

1 is a diagram illustrating an overall operation flow of a speech animation generation system according to an embodiment.

Referring to FIG. 1 , the system for generating speech animation according to an embodiment may include an input step and a speech animation generating step. In the input step, the speech audio 101 and the script 102 are input to the forced speech alignment model 110 to separate the script 101 into morphemes and morphemes, and corresponding to each morpheme and grapheme based on the speech audio 102 . It may correspond to the step of extracting the timing information. The animation generation step corresponds to the step of generating the speech animation by inputting the timing information extracted in the input step into the simultaneous articulation model 120, extracting phoneme information for each grapheme, and mapping each phoneme to the mouth shape and the tongue shape. can The embodiment described below may correspond to a system for generating Korean speech animation based on Korean speech audio and Korean script.

A script according to an embodiment may be composed of letters composed of at least one consonant and at least one vowel. One letter constituting the script may correspond to one syllable, and one syllable may be divided into a leading consonant and a neutral depending on the position of the sound, and may be divided into a leading consonant, a middle consonant, and a final consonant according to the syllable. A corresponding grapheme may exist for each initial, neutral, and final consonant.

A grapheme according to an exemplary embodiment may correspond to a consonant or a vowel as a unit of a character describing a specific language. For example, the alphabet of Hangul may mean one consonant such as 'ㄱ' or 'ㄴ' or one vowel such as 'a' or 'ㅓ'. A phoneme according to an exemplary embodiment means a pronunciation corresponding to a grapheme when a phoneme phrase including graphes is actually pronounced. For example, the letters constituting the script 'Gungmul' are 'ㄱ', 'TT', 'a', 'ㅁ', 'TT', and 'ㄹ', and 'soup' is pronounced as 'Gungmul', so ' The phonemes of 'soup' can be expressed as 'ㄱ', 'TT', 'ㅇ', 'ㅁ', 'TT', and 'ㄹ'.

The pronunciation of a consonant or vowel may be determined by the shape of the mouth and/or the shape of the tongue. For example, in Korean, consonants are affected by the position and shape of the tongue except for the positive labial consonants (/b/, /ㅃ/, /p/, /ㅁ/), and vowels are mainly affected by the shape of the mouth. Looking at the structure of syllables, since only vowels in Korean form the core of syllables, vowels determine the degree of mouth opening, lips protrusion, and tongue position. In addition, up to one consonant can come in the beginning and ending, and there are also syllables that do not have a final consonant. Korean vowels are divided into short vowels and diphthongs. A short vowel is a vowel made with only one articulation movement from beginning to end, and a double vowel refers to a vowel made by combining a half vowel and a short vowel with two articulation movements. A double vowel can be pronounced by sequentially changing from a mouth shape and a tongue shape corresponding to a half vowel to a mouth shape and a tongue shape corresponding to a short vowel. There are two semivowels /j/ and /w/ in Korean, and the combined diphthongs can be called /j/ diphthongs and /w/ diphthongs, respectively. For example, /ㅢ/ can be viewed as a descending diphthong and classified as a diphthong of the /j/ series. In addition, vowels can be classified into flat vowels and round vowels according to the protrusion of the lips, legend vowels and hind tongue vowels based on where the tongue is close to the roof of the mouth, and high vowels, middle vowels, and low vowels according to the height of the tongue. .

The input step according to an embodiment may correspond to a step of pre-processing input data to generate a speech animation. The forced voice alignment model 110 according to an embodiment learns the acoustic model by receiving the speech audio 101 and the script 102, and performs a forced alignment operation by using the learned acoustic model. may correspond to the model. The forced voice alignment model according to an embodiment may include a model such as 'Montreal Forced Aligner'. The forced voice alignment operation according to an embodiment extracts voice alignment information including information on grapheme constituting the script 102 and information on which grapheme a specific frame of the speech audio 101 corresponds to, It can correspond to the task of timing the script.

The forced speech alignment model 110 according to an embodiment may train an acoustic model for a language in consideration of phonological context, phoneme-specific characteristics, speaker characteristics, etc. based on a speech recognition tool (eg, Kaldi). there is. In order to train the acoustic model according to an embodiment, the training data input to the forced speech alignment model 110 may include a large amount of corpus and dictionary. The dictionary according to an embodiment may include a customized dictionary A in which words for performing a forced voice sorting operation are listed.

The step of inputting the script into the forced speech alignment model 110 according to an embodiment may include tokenizing the script based on a word registered in advance. The tokenization step according to an embodiment may include a word tokenization step of separating a script based on spacing, and a morpheme tokenization step of separating a word separated based on spacing into a preceding sentence component, ending, and proposition. may include According to an embodiment, in addition to the customized dictionary A for forced speech alignment, a customized dictionary B for performing morpheme tokenization may be additionally input. The customized dictionary (B) for performing morpheme tokenization according to an embodiment may correspond to a dictionary built in a state in which the grapheme included in the input script is preserved without loss.

In the dictionary (B) according to an embodiment, verbs and adjectives may be registered in the form of a stem. In addition, if the shape of the stem changes according to the application, the modified stem may also be registered in advance. For example, in the case of the adjective 'difficult', the stem 'difficult' and the stem variations 'difficult', 'difficult', 'difficult', etc. may be registered in the dictionary (B).

According to an embodiment, the list of words registered in the customized dictionary B for morpheme tokenization may be processed to be input to the forced speech alignment model. In the customized dictionary (A) according to an embodiment, words registered in the customized dictionary (B) may be processed into alphabets and registered. In the step of processing the words into an alphabet, each of the words is separated into consonants and vowels, and the grapheme(s) including the consonant(s) and vowel(s) included in each word is mapped to the corresponding alphabet to form the grapheme(s). ) may correspond to the steps listed in the order of For example, the step of processing the word 'a' to be listed in the customized dictionary (A) is to separate the word 'a' into letter 'a' and 'a', and the alphabet 'K0' corresponding to 'a' and It may correspond to the step of mapping 'AA' corresponding to 'a' to 'K0 AA' in the order of the graphemes. The shape of the customized dictionary A according to an embodiment may refer to FIG. 2 . Referring to FIG. 2 , the left column shows words, and the right column shows the words translated into grapheme units.

The corpus according to an embodiment may be composed of speech audio and scripts obtained from a plurality of speakers. The script constituting the corpus according to an embodiment may be processed through the steps of tokenizing morphemes using the customized dictionary B and mapping the tokenized morphemes to alphabet symbols. The corpus including the processed script can be used as training data of the acoustic model together with the customized dictionary (A).

The forced voice alignment model 110 according to an embodiment may process the input speech audio 101 and the script 102 to perform a voice alignment operation of matching the timing of the audio and the script. A result of performing the forced voice alignment operation according to an embodiment may refer to FIG. 3 .

Referring to FIG. 3 , the result of performing the forced voice alignment operation may include a voice waveform 310 and a spectrogram 320 corresponding to audio, and the spectrogram 320 includes prosody such as pitch and intensity of a sound. It can contain information about the element. In addition, the timing information 330 in units of morpheme tokens divided by the vertical line as a boundary and timing information 340 in units of graphes divided by the vertical line as the boundary may be included. For example, referring to FIG. 3 , one morpheme unit p0yeoll may correspond to a morpheme separated by tokenizing a script, and grapheme units P0, YEO, and LL corresponding to the morpheme unit are may correspond to the elements constituting the . In addition, timing information about a pause may be included in the result of performing the forced voice alignment operation.

In the animation generation step according to an embodiment, phoneme information is extracted by applying the result obtained by performing the forced speech alignment operation in the input step to the simultaneous articulation model 120, and based on the timing information, each phoneme is converted into a corresponding mouth. It may correspond to the step of generating the speech animation by mapping to the shape and the shape of the tongue. The simultaneous articulation model 120 according to an exemplary embodiment is a model defined based on a polyphonic rule, and may include a Grapheme-to-Phoneme (G2P) model. The simultaneous articulation rule according to an embodiment is a rule regarding a method of pronouncing a specific character, and may correspond to a rule for changing a specific grapheme into a phoneme.

The simultaneous articulation rule according to an embodiment may include a plurality of rules, at least one simultaneous articulation rule is selected based on an environment of a phonological phenomenon, and the order thereof may be determined. Creating a speech animation by mapping each phoneme to a mouth shape and tongue shape corresponding to the mouth shape and tongue shape of a model including an avatar, etc., according to timing information of each phoneme, a mouth shape and tongue shape corresponding to each phoneme By changing to , it may correspond to generating an animation including a model that ignites the corresponding script 101 . Hereinafter, a mouth shape and a tongue shape corresponding to each phoneme may be expressed as a 'viseme' or a 'viseme model'.

According to an embodiment, the simultaneous articulation rule may be applied in units of phonological phrases including a plurality of graphes. A phonological phrase according to an embodiment may correspond to a minimum unit that determines intonation of a sentence including at least one syllable, a unit to which a phonological rule is applied, and a unit in which a phonological phenomenon occurs. A phonological phenomenon according to an embodiment may be defined as a phenomenon in which pronunciation is changed by adjacent graphes. That is, the phonological phenomenon according to an exemplary embodiment occurs within a phonological phrase, and even if a leading consonant is located next to a final consonant, the phonological phenomenon occurs when the corresponding final consonant and the corresponding leading consonant do not belong to the same phonological phrase. may not According to an embodiment, when the adjacent final and initial consonants do not belong to the same phonological phrase and thus a phonological phenomenon does not occur, the simultaneous articulation rule may not be applied. In other words, the simultaneous articulation model 120 according to an embodiment may be applied within each phoneme phrase. According to an exemplary embodiment, a phonophrase in a grapheme sequence may be determined based on timing information, particularly, timing information about a pause, from an output obtained through a forced speech alignment operation.

When a script is read from left to right, a phonological phenomenon can occur in a final consonant among adjacent consonants belonging to the same phonological phrase and a leading consonant corresponding to the following consonant. In other words, when the script is read from left to right, the pronunciation of the consonant corresponding to the last consonant of the letter located on the left among the two adjacent letters may be changed by the consonant corresponding to the beginning of the letter located on the right, and vice versa. Pronunciation may be changed by a consonant corresponding to the last consonant of a letter positioned on the left, and both the consonant and consonant may be influenced by each other to change the pronunciation. For example, in the case of 'soup', the pronunciation of 'ㄱ' corresponding to the last consonant of 'guk' may be changed to 'ㅇ' when uttered by 'ㅁ' corresponding to the initial consonant of 'water'. That is, the simultaneous articulation rule according to an embodiment may be applied to a final consonant and a subsequent leading consonant within a phonological phrase in which a phonological phenomenon occurs.

When a speech animation according to an embodiment is generated, one viseme is not mapped to one phoneme, but phonemes that do not have a visually significant difference in mouth shape and tongue shape when pronounced are grouped into one class and assigned to the same viseme. can be mapped. In other words, when a tongue shape and a mouth shape corresponding to a specific consonant are pronounced as the same consonant according to a phonological phenomenon, the simultaneous articulation rule may not be applied. For example, if the consonant 'f' is pronounced as 'p' according to the phonological phenomenon, the consonant is changed from the actual pronunciation according to the phonological phenomenon, but the mouth shape and the tongue shape corresponding to 'p' correspond to 'ㅅ'. Since the shape of the mouth and tongue are the same, the simultaneous articulation rule may not be applied.

The simultaneous articulation rule according to an embodiment may include at least one rule. For example, the simultaneous articulation rule may include phonological rules according to a standard pronunciation method. More specifically, among the phonological rules according to the standard pronunciation method, 9 phonological rules that should be applied without being omitted depending on the embodiment (Pronunciation of the 'H' base in Article 12 of the Standard Pronunciation method, and the rule of ending syllables in Articles 8-9 , Paragraph 10-11 Simplification of consonant groups, Paragraph 19 'd' nasalization, Paragraph 20 'ㄹ' nasalization exception, Paragraph 20 Phoneticization, Paragraph 17 Palatalization, Paragraph 29 Addition of 'b', Paragraph 12 Aspirated ( However, content that overlaps with the pronunciation of the 'ㅎ' consonant is excluded)) and three linking rules (paragraphs 13, 14, and 15) may be included. That is, according to an embodiment, not all phonological rules according to the standard pronunciation method are included, but some rules may be excluded.

4 is a diagram illustrating an example in which simultaneous articulation rules are determined differently according to each grapheme.

4, in the simultaneous articulation rule according to an embodiment, the rule is selected according to what the final consonant is based on the final consonant, and when there are a plurality of selected rules, the order in which the rules are applied can be determined, Whether or not the selected rule is applied may be determined according to an adjacent leading consonant after the final consonant. In this case, a phonological rule or a linking rule is selected according to the final consonant, and the order of application of the rules may be determined in the order indicated by an arrow. According to an embodiment, it may be determined whether the rules included in the simultaneous articulation rule selected according to the final consonant are applied according to the adjacent leading consonant. In other words, the application of the rule selected according to the leading consonant adjacent to the back may be omitted. For example, when the final consonant is 'ㄺ', it is determined whether or not to apply the aspirated rule according to the adjacent leading consonant, and then the consonant group simplification rule may be applied.

However, when the leading consonant located next to the final consonant corresponds to a specific consonant, the simultaneous articulation rule may be selected based on the leading consonant other than the final consonant, and the order may be determined.

For example, when the simultaneous articulation rule is applied as shown in FIG. 4, if the leading consonant in which the phonological phenomenon occurs corresponds to 'ㅇ' or 'ㅎ', the linking rule is selected according to the leading consonant, and the rule of the rule is selected according to the leading consonant. The order of application may be determined. In this case, the leading consonant in which the phonological phenomenon occurs in the phonological phrase 'chicki' corresponds to 'ㅇ', so the linking rule is applied according to the leading consonant and can be changed to 'dalgi'. On the other hand, in the phonological phrase 'chicken house', the rules for aspiration, consonant group simplification, and hard consonants are selected according to the final consonant 'ㄺ' in which the phonological phenomenon occurs, and the order is determined. is determined, and based on the determination of whether to apply, the abbreviation rule may be applied and the consonant group simplification rule may be applied, or the consonant group simplification rule may not be applied and the consonant group simplification rule may be applied. As a result, the 'chicken house' may be changed to 'dakjang' by applying the simultaneous articulation rule determined according to the final consonant 'ㄺ' and the initial consonant 'j'.

In Korean, vowels can be divided into short vowels and diphthongs. A short vowel is a vowel made from only one articulation from beginning to end, and a double vowel is a vowel made by combining a semivowel and a short vowel with two articulation movements. For example, the vowel ‘ㅑ’ may correspond to a diphthong made by articulating the half vowels /j/ and ‘a’. The generating of the speech animation according to an exemplary embodiment may include sequentially blending a tongue shape and a mouth shape corresponding to a half vowel of a diphthong and a tongue shape and a mouth shape corresponding to a short vowel. Unlike short vowels, half vowels have such a short duration that they cannot form one syllable, so the duration may be set shorter than that of short vowels during the blending process. According to an embodiment, the duration of the half vowel may be set to half the duration of the short vowel.

The simultaneous articulation model 120 according to an embodiment sets the mouth shape and the tongue shape of the model to the mouth shape corresponding to the phoneme and the By changing the shape of the tongue, you can create speech animations. The simultaneous articulation model 120 according to an embodiment may include a viseme model.

Phonemes according to an embodiment may be mapped to a specific mouth shape and a specific tongue shape based on their pronunciation characteristics. According to an embodiment, a plurality of corresponding phonemes may be mapped to one mouth shape, and a plurality of corresponding phonemes may be mapped to one tongue shape. In other words, when different phonemes are uttered, if it is determined that there is no visually significant difference based on the position of the tip of the tongue and the shape of the tongue in the visible region, the phonemes may be classified into the same class.

Taking Korean as an example, referring to FIG. 5 , phonemes corresponding to consonants existing as phonemes may be classified into seven classes according to the mouth shape and tongue shape to be pronounced, and referring to FIGS. 6 and 7 . In this case, phonemes corresponding to vowels may also be classified into a class according to 5 types of tongue shapes and a class according to 10 types of mouth shapes according to the mouth shape and tongue shape to be pronounced. Referring to FIG. 8 , even in the case of a shim, a mouth shape may be mapped.

9 is a diagram for explaining a method of generating a speech animation according to an embodiment.

Referring to FIG. 9 , a method of generating a speech animation according to an embodiment includes receiving a voice signal and a script ( 910 ), generating a grapheme sequence based on the script ( 920 ), and based on the voice signal generating timing information (930), extracting a closing consonant and a leading consonant adjacent to each other from a grapheme sequence (940), determining a co-articulation rule and order based on the closing consonant and the leading consonant (950); generating a phoneme sequence by sequentially applying polyarticulation rules to the final and leading consonants ( 960 ) and modifying the shape of the mouth and tongue of the avatar based on the timing information and the phoneme sequence, thereby generating a speech animation. Step 970 is included.

The step 910 of receiving the voice signal and the script according to an embodiment may correspond to the step of receiving the voice signal and the script corresponding to the voice signal. The voice signal according to an embodiment may include a voice signal in which a script is pronounced aloud. For example, when the script according to an embodiment is a script including at least one syllable written in Korean, the voice signal may correspond to a voice signal uttering the script in Korean. The voice signal according to an embodiment may correspond to the speech audio 101 that is input data of the above-described acoustic model.

The step 920 of generating a grapheme sequence based on the script according to an embodiment may correspond to the step of generating a grapheme sequence including at least one consonant and at least one vowel based on the script. A grapheme sequence according to an embodiment is a sequence in which syllables constituting a script are separated into vowels and consonant units and arranged in order, and may correspond to information on grapheme constituting the above-described script. A grapheme sequence according to an embodiment may be displayed by mapping graphes included in a script to specific codes. For example, the grapheme sequence generated based on the script 'A' is 'K0 AA' in which 'K0' mapped the grapheme 'a' constituting 'A' and 'AA' mapped 'A' are sequentially listed. may correspond to Hereinafter, a grapheme (including a consonant or a vowel) may include a sign mapped to the corresponding grapheme.

Step 920 according to an embodiment may include dividing the script into at least one semantic unit based on a dictionary in which semantic units of a language corresponding to the script are listed, and at least one semantic unit included in the semantic unit for each semantic unit. It may include listing one grapheme in order. The dictionary according to an embodiment may include the above-described customized dictionary (A) and customized dictionary (B). A semantic unit according to an embodiment may include a morpheme. When the semantic unit according to an embodiment is a morpheme, the step of dividing the script into at least one semantic unit may correspond to the step of tokenizing the script.

In the step 930 of generating timing information based on a voice signal according to an embodiment, the timing including timing of each of the at least one consonant and the at least one vowel included in the grapheme sequence, based on the voice signal It may correspond to the step of generating information. The timing information according to an embodiment may correspond to timing information included in a result of performing the above-described forced voice alignment operation. That is, the timing information according to an embodiment may include information indicating the utterance time of each grapheme by arranging the frame(s) included in the voice signal in units of graphes included in the script. The timing information according to an embodiment may include timing information in units of grapheme and timing information in units of morphemes including a plurality of grapheme units. A morpheme unit according to an embodiment may correspond to a unit in which a script is separated by the above-described morpheme tokenization step.

Step 930 of generating the timing information according to an embodiment may include displaying a specific section as a pause section according to a predetermined criterion in the voice signal and generating timing information corresponding to the pause section. can That is, the timing information according to an embodiment may include timing information about a pause. The pause or pause section according to an embodiment may correspond to a section in which a sound for uttering a script by respiration in a voice signal is temporarily cut off. For example, when the sentence "read a book" is uttered, "book" can be uttered with one breath and "read" can be uttered with the next breath, so the corresponding voice signal includes "book" and There may be a pause section in which the voice signal is temporarily cut off in a certain section between “reading”. That is, in the voice signal according to an embodiment, there may be a section in which the strength of the voice signal is weak or the voice signal is temporarily cut off, and this section may be determined as a rest section. According to an embodiment, a phoneme phrase to which the simultaneous articulation rule is applied may be determined based on the rest period.

According to an embodiment, the predetermined criterion for determining the pause period may be determined based on the strength of the voice signal and the duration of the signal of the specific strength. That is, the specific section according to the predetermined criterion may correspond to a section in which a signal equal to or less than the strength of the predetermined voice signal is maintained for a predetermined time. For example, according to a criterion for determining a section in which a signal of 10 dB or less in the voice signal lasts for 1 second as a rest section, a specific section in the voice signal is displayed as a pause section, and timing information corresponding to the grapheme in the timing information and timing information corresponding to the rest period may be included.

Steps 910 to 930 according to an embodiment may correspond to the above-described input steps.

According to an exemplary embodiment, the step of extracting the last consonant and the leading consonant adjacent to each other from the grapheme sequence 940 may correspond to the step of extracting the last consonant of a specific syllable and the leading consonant of a syllable located immediately after the syllable. In other words, the final consonant and the leading consonant adjacent to each other may correspond to the final consonant and the leading consonant located after the final consonant in the grapheme sequence in which the graphes are arranged in chronological order.

In step 940 according to an embodiment, while sequentially reading the grapheme included in the phonological phrase, extracting whether the corresponding grapheme is a consonant and the position of the corresponding consonant, and when the extracted position of the extracted consonant is the final consonant, the extracted It may include extracting the next consonant of the consonant. A phonological phrase according to an embodiment may include at least one syllable as described above and may correspond to a unit in which a phonological phenomenon occurs. In other words, the final consonant and the leading consonant extracted in step 940 may correspond to adjacent consonants existing in one phonological phrase. According to an embodiment, sequentially reading the grapheme included in the phonological phrase may correspond to reading the grapheme included in the phonemic phrase from left to right.

According to an embodiment, the step of determining the coarticulation rule and order based on the final consonant and the leading consonant ( 950 ) may include at least one coarticulation rule and at least one coarticulation rule based on the final consonant and leading consonant. It may correspond to the step of determining the order. According to an embodiment, selecting the co-articulation rule corresponding to a specific consonant means determining at least one rule(s) in response to a specific consonant among at least one rule(s) included in the co-articulation rule, and It may mean determining the order of at least one rule(s) determined in response to a consonant.

In step 950 according to an embodiment, when the leading consonant does not correspond to a predetermined consonant, selecting at least one simultaneous articulation rule corresponding to the final consonant, and when the leading consonant corresponds to a predetermined consonant, The method may include selecting at least one simultaneous articulation rule corresponding to the leading consonant. For example, if the predetermined consonants are 'ㅇ' and 'ㅎ', a simultaneous articulation rule may be selected in response to the final consonant 'ㄺ' in one phoneme phrase 'chicken', and one phonemic phrase 'chicken' ', the simultaneous articulation rule may be selected in correspondence to the leading consonant 'ㅇ' rather than the final consonant.

According to an embodiment, the step of generating a phoneme sequence 960 by applying the coarticulation rule to the final consonant and the leading consonant according to the order is by applying at least one coarticulation rule to the final consonant and the leading consonant according to the order, It may correspond to the step of generating a phoneme sequence by updating the phoneme sequence. A phoneme sequence according to an embodiment may correspond to a sequence in which phonemes included in the grapheme sequence are changed by applying a simultaneous articulation rule in units of phoneme phrases. In other words, the phoneme sequence is a list of phonemes that are phoneme codes according to actual pronunciation of the phonemes, and may correspond to the above-described phoneme information. For example, if the script includes 'soup', the grapheme sequence generated based on it corresponds to 'a', 'TT', 'a', 'ㅁ', 'TT', and 'ㄹ'. The phoneme sequence generated by applying the simultaneous articulation rule by determining that 'soup' is a single phoneme phrase may correspond to 'ㄱ', 'TT', 'ㅇ', 'ㅁ', 'TT', and 'ㄹ'.

By modifying the shape of the mouth and the tongue of the avatar based on the timing information and the phoneme sequence according to an embodiment, generating a speech animation 970 includes each of the phonemes included in the phoneme sequence generated in step 960 . mapping to corresponding mouth and tongue shapes and transforming the avatar's mouth and avatar's tongue shapes into mouth and tongue shapes mapped to respective phonemes according to timing information, thereby generating a speech animation; there is. An avatar according to an embodiment may correspond to an object including a shape of a mouth and a shape of a tongue.

When the timing information according to an embodiment includes timing information corresponding to the pause section, generating the speech animation 970 includes mapping the pause section to a mouth shape corresponding to the pause section and timing information corresponding to the pause section. The method may include generating a speech animation by transforming the avatar's mouth shape into a mouth shape mapped to the rest period based on the . That is, when the timing information according to an embodiment includes timing information corresponding to a pause section, which is a section in which the speech is temporarily stopped, by transforming the mouth shape of the avatar corresponding to the timing into a mouth shape mapped to the pause section, A speech animation may be generated.

Steps 940 to 970 according to an embodiment may correspond to the above-described speech animation generation step.

The embodiments described above may be implemented by a hardware component, a software component, and/or a combination of a hardware component and a software component. For example, the apparatus, methods and components described in the embodiments may include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate (FPGA) array), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions, may be implemented using one or more general purpose or special purpose computers. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to execution of the software. For convenience of understanding, although one processing device is sometimes described as being used, one of ordinary skill in the art will recognize that the processing device includes a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that can include For example, the processing device may include a plurality of processors or one processor and one controller. Other processing configurations are also possible, such as parallel processors.

Software may comprise a computer program, code, instructions, or a combination of one or more of these, which configures a processing device to operate as desired or is independently or collectively processed You can command the device. The software and/or data may be any kind of machine, component, physical device, virtual equipment, computer storage medium or device, to be interpreted by or to provide instructions or data to the processing device. , or may be permanently or temporarily embody in a transmitted signal wave. The software may be distributed over networked computer systems, and stored or executed in a distributed manner. Software and data may be stored in one or more computer-readable recording media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the embodiment, or may be known and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic such as floppy disks. - includes magneto-optical media, and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

As described above, although the embodiments have been described with reference to the limited drawings, those skilled in the art may apply various technical modifications and variations based on the above. For example, the described techniques are performed in a different order than the described method, and/or the described components of the system, structure, apparatus, circuit, etc. are combined or combined in a different form than the described method, or other components Or substituted or substituted by equivalents may achieve an appropriate result.

Claims (20)

receiving a script and a voice signal corresponding to the script;
generating a grapheme sequence including at least one consonant and at least one vowel based on the script;
generating timing information including timings of each of the at least one consonant and the at least one vowel included in the grapheme sequence, based on the speech signal;
extracting a final consonant and a leading consonant adjacent to each other from the grapheme sequence;
determining at least one co-articulation rule and an order of the at least one co-articulation rule based on the final consonant and the leading consonant;
generating a phoneme sequence by updating the phoneme sequence by applying the at least one simultaneous articulation rule to the final consonant and the leading consonant according to the order; and
generating a speech animation by modifying a mouth shape of an avatar and a tongue shape of the avatar based on the timing information and the phoneme sequence;
containing,
How to create speech animation. According to claim 1,
Determining the order of the at least one co-articulation rule and the at least one co-articulation rule comprises:
selecting at least one simultaneous articulation rule corresponding to the final consonant when the leading consonant does not correspond to a predetermined consonant; and
selecting at least one simultaneous articulation rule corresponding to the leading consonant when the leading consonant corresponds to the predetermined consonant;
containing,
How to create speech animation. According to claim 1,
The step of extracting a final consonant and a leading consonant adjacent to each other from the grapheme sequence comprises:
extracting whether the corresponding grapheme is a consonant and a position of the corresponding consonant while sequentially reading the grapheme included in the corresponding phoneme phrase; and
When the position of the extracted consonant is the final consonant, extracting the next consonant of the extracted consonant
containing,
How to create speech animation. According to claim 1,
The step of generating the grapheme sequence includes:
separating the script into at least one semantic unit based on a dictionary in which semantic units of a language corresponding to the script are listed;
Listing at least one grapheme included in the semantic unit in order for each of the at least one semantic unit
containing,
How to create speech animation. According to claim 1,
The step of generating the speech animation is
mapping each of the phonemes included in the phoneme sequence to a corresponding mouth shape and a tongue shape;
generating a speech animation by transforming the mouth shape of the avatar and the tongue shape of the avatar into a mouth shape and a tongue shape mapped to each phoneme based on the timing information;
containing,
How to create speech animation. According to claim 1,
The script includes a script including at least one syllable written in Korean,
The voice signal includes a voice signal uttering the script in Korean
How to create speech animation. According to claim 1,
The step of generating the timing information
displaying a specific section as a pause section in the voice signal according to a predetermined criterion; and
generating timing information corresponding to the rest period
containing,
How to create speech animation. 8. The method of claim 7,
The step of generating the speech animation is
mapping the rest period to a corresponding mouth shape; and
generating a speech animation by transforming the mouth shape of the avatar into a mouth shape mapped to the rest section based on timing information corresponding to the pause section;
containing,
How to create speech animation. 8. The method of claim 7,
The specific section according to the predetermined criteria is
A period in which a signal less than or equal to the strength of a predetermined voice signal lasts for a predetermined time.
How to create speech animation. According to claim 1,
The step of generating the grapheme sequence includes:
generating a grapheme sequence by mapping at least one grapheme included in the script to a predetermined code;
further comprising,
How to create speech animation. According to claim 1,
The simultaneous articulation rule is
A rule for changing the final consonant and the leading consonant located next to the final consonant according to the pronunciation
How to create speech animation. A computer program stored on a medium in combination with hardware to execute the method of any one of claims 1 to 11. Receive a script and a voice signal corresponding to the script to generate timing information including timings of a grapheme sequence and at least one consonant and at least one vowel included in the grapheme sequence, and a final consonant adjacent to each other from the grapheme sequence and extracting a leading consonant to determine at least one co-articulation rule based on the final consonant and the leading consonant and an order of the at least one co-articulation rule, and set the at least one co-articulation rule according to the order to the final consonant. and generating a phoneme sequence by updating the phoneme sequence by applying to the leading consonant, and generating a speech animation by modifying a mouth shape of an avatar and a tongue shape of the avatar based on the timing information and the phoneme sequence. one processor and
A memory for storing the grapheme sequence, the phoneme sequence, the timing information, and mapping information of a tongue shape and a mouth shape corresponding to the at least one phoneme
containing
A device for generating speech animations. 14. The method of claim 13,
the processor
to determine the order of the at least one polyarticulation rule and the at least one polyarticulation rule;
When the leading consonant does not correspond to a predetermined consonant, at least one simultaneous articulation rule corresponding to the final consonant is selected, and when the leading consonant corresponds to the predetermined consonant, at least one corresponding to the leading consonant is selected. to choose the simultaneous articulation rule of
A device for generating speech animations. 14. The method of claim 13,
the processor
In order to extract a final consonant and a leading consonant adjacent to each other from the grapheme sequence,
While sequentially reading the graphemes included in the corresponding phoneme phrase, whether the corresponding grapheme is a consonant and the position of the corresponding consonant are extracted, and when the extracted position of the extracted consonant is the final consonant, the next consonant of the extracted consonant is extracted.
A device for generating speech animations. 14. The method of claim 13,
the processor
To generate the grapheme sequence,
Separating the script into at least one semantic unit based on a dictionary in which semantic units of a language corresponding to the script are listed, and listing at least one grapheme included in the semantic unit for each of the at least one semantic unit in order doing
A device for generating speech animations. 14. The method of claim 13,
the processor
To create the speech animation,
Each of the phonemes included in the phoneme sequence is mapped to a corresponding mouth shape and tongue shape, and the mouth shape of the avatar and the tongue shape of the avatar are mapped to each phoneme based on the timing information into a mouth shape and a tongue shape. By transforming, to create speech animation
A device for generating speech animations. 14. The method of claim 13,
The script includes a script including at least one syllable written in Korean,
The voice signal includes a voice signal uttering the script in Korean
A device for generating speech animations. 14. The method of claim 13,
the processor
To generate the timing information,
In the voice signal, a specific section is displayed as a pause section according to a predetermined criterion, and timing information corresponding to the pause section is generated,
To create the speech animation,
Creating a speech animation by mapping the pause section to a mouth shape corresponding to the pause section, and transforming the mouth shape of the avatar into a mouth shape mapped to the pause section based on timing information corresponding to the pause section
A device for generating speech animations. 14. The method of claim 13,
the processor
To generate the grapheme sequence,
generating a grapheme sequence by mapping at least one grapheme included in the script to a predetermined code;
A device for generating speech animations.

Images