KR20210131698A - Method and apparatus for teaching foreign language pronunciation using articulator image - Google Patents

Abstract

The present invention relates to a method for teaching foreign language pronunciation using speech organ images, which increases a learner's convenience, and an apparatus thereof. According to the present invention, a method is operated by a computing device operated by at least one processor and comprises the following steps: receiving an input of a learner's pronunciation voice and image for an arbitrary text; generating a standard phoneme list of the text; generating a learner's pronunciation phoneme list by extracting phonemes from which the text is pronounced from the learner pronunciation voice and image; and inputting the standard pronunciation phoneme list into a learned image generation model to generate a standard pronunciation simulation image for pronouncing the standard pronunciation phoneme list, inputting the learner's pronunciation voice and image and the learner's pronunciation phoneme list to an image generation model to generate a learner's pronunciation simulation image for pronouncing the learner's pronunciation phoneme list, and outputting the standard pronunciation simulation image and the learner's pronunciation simulation image, wherein the standard pronunciation simulation image and the learner pronunciation simulation image are virtual pronunciation organ images simulating a pronunciation process.

Description

A method and apparatus for teaching foreign language pronunciation using an image of a pronunciation organ

The present invention relates to a technique for teaching foreign language pronunciation using an image of a pronunciation organ.

In the global era, communication skills through foreign languages are becoming very important. In particular, for smooth communication, pronunciation plays an important role. However, in the reality of foreign language education in Korea, due to the entrance exam-oriented education, emphasis is placed on grammar and vocabulary rather than pronunciation education.

Specifically, the pronunciation education of a foreign language such as traditional English is provided by providing a screen image of a native speaker's pronunciation as text or images to the learner, inducing them to follow the pronunciation, and explaining in writing.

For example, a pronunciation organ image is generated and output in 3D according to a predetermined rule, or the image is simply modified and provided by adding some audio features to an existing 3D image.

To provide a 3D visualization of the pronunciation organ, the position or shape of the tongue and the shape of the lips are provided as pictures or videos that are easy for users to understand when voice is uttered, and video information about specific pronunciation is provided to learners. can do. However, it is difficult to know how to pronounce on their own from the learner's point of view and how to correct the pronunciation of a native speaker only with the information provided in this way. This is because it is not possible to know how the pronunciation organ is represented in 3D when the learner pronounces it.

That is, there is a need for a learning method that outputs an image of a learner's pronunciation organ on the screen, directly confirms the difference from the image of a pronunciation organ of a native speaker who plays a standard role, and provides an image of how the learner should correct pronunciation.

The task to be solved is to provide a method of representing the shape and movement of a pronunciation organ when pronouncing an arbitrary sentence as a three-dimensional image through a learned deep learning model.

In addition, the task to be solved is to simultaneously provide the movement of the learner's pronunciation organ and the movement of the native speaker's pronunciation organ for the same text, showing the difference between the pronunciation organs and providing a pronunciation guide for standard pronunciation.

A method of operating a computing device operated by at least one processor according to an embodiment, the method comprising: receiving a learner pronunciation voice and an image for an arbitrary text; generating a standard pronunciation phoneme list of the text; the learner A standard pronunciation simulation of extracting phonemes from which the text is pronounced from a pronunciation voice and an image to generate a learner pronunciation phoneme list, and inputting the standard pronunciation phoneme list into a learned image generation model to pronounce the standard pronunciation phoneme list generating an image, inputting the learner pronunciation voice and image, and the learner pronunciation phoneme list into the image generation model to generate a learner pronunciation simulation image for pronouncing the learner pronunciation phoneme list, the standard pronunciation simulation image and the learner pronunciation and outputting a simulation image, wherein the standard pronunciation simulation image and the learner pronunciation simulation image are virtual pronunciation organ images simulating a pronunciation process.

The method may further include analyzing a difference in the shape or movement of each pronunciation organ included in the standard pronunciation simulation image and the learner pronunciation simulation image, and generating a pronunciation guide including the difference.

collecting a first image including a shape of the oral cavity for each phoneme and a second image including a shape of a section of a pronunciation organ for each phoneme, a phoneme corresponding to the first image or the second image, and the gender and age of the speaker generating learning data by tagging the speaker information, the shape of the oral cavity and the shape of the pronunciation organ when pronouncing the phoneme, and the shape of the oral cavity and the shape of the section of the pronunciation organ for each phoneme as the learning data. It may further include the step of learning the relationship of the image generation model.

The shape of the oral cavity may include at least one of the size of the oral cavity, the shape of the lips, whether the lower teeth and the upper teeth contact each other, or the position of the tongue.

The phoneme may be displayed as a phonetic symbol, and the phonetic symbol may be an International Phonetic Alphabet (IPA) modified according to the speaker information.

The learning data may further include a numerical value representing a difference between the phonetic symbol transformed according to the speaker information and the international phonetic symbol.

A computing device according to an embodiment, comprising a memory and at least one processor executing instructions of a program loaded in the memory, wherein the program is configured to determine the shape of the oral cavity and the shape of the section of the pronunciation organ for each phoneme. generating learning data by collecting a learning image including Learning an image generation model for outputting a virtual pronunciation organ simulation image in which standard phonemes of the phonemes are pronounced, inputting the learner's pronunciation voice and image for arbitrary text into the image generation model, and the image generation model It includes instructions described to execute the step of displaying the standard pronunciation simulation image of the text output from the screen.

The training data may further include histogram information or feature point information extracted from the training image.

The learning may include calculating a phoneme relationship that can be consecutively pronounced with a probability, and the displaying may include generating a standard phoneme list of the text and a phoneme list of pronunciation of the text based on the probability. .

The method may further include generating, by the virtual pronunciation organ, a learner pronunciation simulation image pronouncing the text.

The method may further include analyzing and providing a difference between the standard phoneme list and the phoneme list pronounced in the text.

According to the present invention, it is possible to listen to the standard pronunciation of the native speaker for the text pronounced by the learner, and the movement of the pronunciation organ for the standard pronunciation can be checked through a simulation image, which is effective for pronunciation education.

In addition, according to the present invention, not only the image of the pronunciation organ of the native speaker but also the image of the learner's pronunciation organ are provided together, so that the learner can check the part to be corrected by the image, and the pronunciation correction can be performed alone without the help of the teacher. Convenience can be increased.

1 is a block diagram of an apparatus for generating a pronunciation image according to an exemplary embodiment.
2 is an explanatory diagram of a pronunciation image model and a phoneme analysis model according to an exemplary embodiment.
3 is a flowchart of a process in which an apparatus for generating a pronunciation image according to an exemplary embodiment trains a phoneme analysis model.
4 is an explanatory diagram illustrating pronunciation positions of vowels according to an exemplary embodiment.
5 is an exemplary diagram illustrating characteristics of a pronunciation organ according to an embodiment.
6 is an exemplary diagram of a method of obtaining learning data according to an embodiment.
7 is an exemplary diagram of a phoneme list learned by a phoneme analysis model according to an embodiment.
8 is a flowchart of a method of operating an apparatus for generating a pronunciation image according to an exemplary embodiment.
9 is an exemplary diagram of a phoneme list predicted by a phoneme analysis model according to an embodiment.
10 is an explanatory diagram of a method of generating a simulation image by a pronunciation image model according to an exemplary embodiment.
11 and 12 are exemplary views of a result screen output by the apparatus for generating a pronunciation image according to an exemplary embodiment.
13 is a hardware configuration diagram of a computing device according to an embodiment.

Hereinafter, with reference to the accompanying drawings, embodiments of the present invention will be described in detail so that those of ordinary skill in the art to which the present invention pertains can easily implement them. However, the present invention may be embodied in many different forms and is not limited to the embodiments described herein. And in order to clearly explain the present invention in the drawings, parts irrelevant to the description are omitted, and similar reference numerals are attached to similar parts throughout the specification.

Throughout the specification, when a part "includes" a certain element, it means that other elements may be further included, rather than excluding other elements, unless otherwise stated. In addition, terms such as “…unit”, “…group”, and “module” described in the specification mean a unit that processes at least one function or operation, which may be implemented by hardware or software or a combination of hardware and software. have.

In this specification, a voice means a physical speech sound. Even in the same sentence, the voice may be different depending on the speaker.

International Phonetic Alphabet (IPA) refers to a system of symbols created by the International Phonetic Society as a consistent method of writing sounds to cover various sounds of languages around the world. For example, if the pronunciation of the English word 'apple' is expressed as IPA, it is ['ζpl], and if the pronunciation of the Korean word 'man' is expressed as IPA, it is [sanai].

A phoneme refers to a basic unit of speech sound that is recognized abstractly, and even the same phoneme may sound different depending on the person who pronounces it. Therefore, even when the same sentence is pronounced, the standard phoneme of the corresponding sentence and the phoneme of the sentence pronounced by the learner may be different from each other.

Therefore, the present specification aims to help the learner to learn the standard pronunciation by themselves by distinguishing the difference between the two phonemes and providing the learner's pronunciation of the phoneme and the standard pronunciation as a 3D image.

In addition, unless otherwise stated in the present specification, it is described that an utterance video includes an uttered voice.

1 is a block diagram of an apparatus for generating a pronunciation image according to an exemplary embodiment.

Referring to FIG. 1 , the pronunciation image generating apparatus 10 includes an audio and image input unit 100 for receiving an image in which a learner pronounces an arbitrary sentence, and a phoneme analysis model 210 for extracting a phoneme list constituting the input sentence. ) and the learning unit 200 for learning the pronunciation image model 220, the pronunciation image generation unit 300 for generating the standard pronunciation image and the pronunciation organ image, analyzing the difference between the standard pronunciation image and the learner's pronunciation image It includes a pronunciation guide providing unit 400 , a pronunciation image DB 500 , and a phoneme information DB 600 .

For explanation, the audio and image input unit 100, the learning unit 200, the pronunciation image generation unit 300, the pronunciation guide providing unit 400, the pronunciation image DB 500, and the phoneme information DB 600 are named. However, they are computing devices operated by at least one processor. Here, the audio and image input unit 100 , the learning unit 200 , the pronunciation image generation unit 300 , the pronunciation guide providing unit 400 , the pronunciation image DB 500 and the phoneme information DB 600 are one computing device. It may be implemented in , or distributed in a separate computing device. When distributed in a separate computing device, the voice and image input unit 100 , the learning unit 200 , the pronunciation image generator 300 , the pronunciation guide provider 400 , the pronunciation image DB 500 , and the phoneme information DB 600 may communicate with each other via a communication interface. The computing device may be any device capable of executing a software program written to carry out the present invention, and may be, for example, a server, a laptop computer, or the like.

Each of the audio and video input unit 100 , the learning unit 200 , the pronunciation image generation unit 300 , and the pronunciation guide providing unit 400 may be one AI model or may be implemented as a plurality of AI models. . Also, the phoneme analysis model 210 and the pronunciation image model 220 may be one AI model or may be implemented as a plurality of AI models. The pronunciation image generating apparatus 10 may be one artificial intelligence model or may be implemented as a plurality of artificial intelligence models. Accordingly, one or a plurality of artificial intelligence models corresponding to the above-described configurations may be implemented by one or a plurality of computing devices.

The audio and video input unit 100 receives audio and images of arbitrary sentences pronounced by the learner. Inputs can be received through the camera and microphone, and the learner's front and side images can be acquired.

The learning unit 200 learns the phoneme analysis model 210 and the pronunciation image model 220 by using the standard pronunciation phoneme information for each sentence included in the pronunciation image DB 500 and the phoneme information DB 600 . Specifically, the phoneme analysis model 210 serves to extract a list of phonemes constituting the corresponding sentence based on the voice and image in which the sentence is pronounced. When a phoneme list is input, the pronunciation image model 220 serves to generate a cross-sectional view of the movement of the articulatory organ in which each phoneme is pronounced.

The pronunciation image generator 300 generates a 3D image for pronouncing a sentence of the input learner image by using the learned pronunciation image model 220 , and generates a 3D standard pronunciation image for pronouncing the sentence.

The pronunciation guide providing unit 400 compares the difference between the learner's pronunciation image and the standard pronunciation image for each phoneme, and provides the learner with differences in pronunciation positions, mouth shapes, and tongue positions for each phoneme. A pronunciation guide may be generated based on the contents stored in the pronunciation image DB 500 and the phoneme information DB 600 .

The pronunciation guide may be provided through the interface shown in FIGS. 12 and 13 , and images or texts may be displayed for each phoneme for the convenience of the learner. In addition, the learner's 3D pronunciation image and the standard pronunciation image can be provided by overlapping.

The pronunciation image DB 500 includes an image of the movement of the pronunciation organ when pronouncing a phoneme from the outside of the body and a cross-sectional image of the inside of the body. The images stored in the pronunciation image DB 500 may be 2D or 3D. The images may be collected through web crawling or taken for model training.

Meanwhile, the image stored in the pronunciation image DB 500 may be an image pronounced by a native speaker or a person equivalent thereto. This is because it is used to generate a standard pronunciation image by transforming the sentences pronounced by the learner into the same form as those pronounced by a native speaker.

That is, the images included in the pronunciation image DB 500 may be regarded as standard pronunciation images of arbitrary phonemes.

The phoneme information DB 600 includes international standard IPA phonetic symbols representing each phoneme as a phonetic symbol. Meanwhile, the standard IPA phonetic symbols may be modified by the administrator to reflect the pronunciation characteristics of the learner. Details will be described with reference to FIG. 4 .

2 is an explanatory diagram of a pronunciation image model and a phoneme analysis model according to an exemplary embodiment.

Referring to FIG. 2 , the phoneme analysis model 210 is an image of a native speaker pronouncing a specific phoneme, a uttered sentence, a sectional image including the speaker's gender and age group, and the movement of the pronunciation organ, It is learned to output phonemes constituting an arbitrary sentence by using learning data including features and the like.

Meanwhile, information labeled on the pronunciation image and the cross-sectional image may be in the format of Table 2. Details of the training data will be described with reference to FIG. 3 .

With the learned phoneme analysis model 210 , when a learner's speech image and speech voice are input, phonemes constituting the speech may be output. In addition, the form of the input data is not limited to video and audio, but may be a text form.

In this case, when a text sentence is input to the learned phoneme analysis model 210 to generate a standard pronunciation image, a list of phonemes constituting the text sentence may be output.

The pronunciation image model 220 is trained to output a 3D pronunciation image simulating a cross-section of a human body by using the above learning data.

When the learner's speech image and speech voice are input to the learned pronunciation image model 220 , an image simulating the learner's pronunciation organ in 3D may be generated. The 3D pronunciation image may output the process of pronunciation of the sentence spoken by the learner and the movement of the pronunciation organs as an image.

Meanwhile, the pronunciation image model 220 may additionally use the phoneme list output from the phoneme analysis model 210 to generate an image. Hereinafter, the learning process of each model will be described in detail.

3 is a flowchart of a process in which the phoneme analysis model is trained by the apparatus for generating a pronunciation image according to an embodiment; FIG. 4 is an explanatory diagram illustrating pronunciation positions of vowels according to an embodiment; and FIG. It is an exemplary diagram showing characteristics of a pronunciation organ, FIG. 6 is an exemplary diagram of a method of obtaining learning data according to an embodiment, and FIG. 7 is an exemplary diagram of a phoneme list learned by a phoneme analysis model according to an embodiment.

Referring to FIG. 3 , the pronunciation image generating apparatus 10 collects a pronunciation image of a foreigner or a native speaker and a cross-sectional image of a pronunciation organ ( S110 ). As an example, images collected in the pronunciation image DB 500 may be used.

Meanwhile, in the pronunciation image DB 500 , there may be no pronunciation images of all phonemes and a cross-sectional image of the pronunciation organs. In this case, a pronunciation image of another phoneme may be generated based on the already collected pronunciation image and the cross-sectional image of the pronunciation organ and used as learning data. As an example, generative adversarial networks (GANs) may be used.

The pronunciation image generating apparatus 10 collects phoneme information in which phonemes are expressed as phonemes, and corrects them according to Korean pronunciation characteristics (S120). Phoneme information refers to the international standard IPA phonetic symbols, and it can be used as it is or modified according to the pronunciation characteristics of the learner.

The reason for changing the phoneme information is to reflect the fact that Koreans mainly pronounce Korean, and the types and locations of pronunciation organs mainly used in Korean are different from foreign languages.

For example, FIG. 4( a ) is a vowel square diagram showing IPA phonetic symbols used when pronouncing English vowels according to pronunciation positions.

Meanwhile, the pronunciation image generating apparatus 10 may use the standard IPA symbol as shown in FIG. 4A as it is, but may use a modified standard IPA according to the pronunciation characteristics of speakers of each country. As an example, (b) of FIG. 4 shows the IPA vowel symbols modified in consideration of the characteristics of Koreans pronouncing English vowels.

At this time, (b) of FIG. 4 may be modified according to the characteristics of Korean pronunciation organs, for example, Korean oral size, vocal cord size, lip shape, etc., the position of the modified IPA pronunciation symbol is The vowel may be changed according to the position of the tongue in the oral cavity when pronouncing the corresponding vowel. The modified IPA symbol may be as shown in Table 1.

Serial Number standard IPA Modified IPA One α αː' 2 æ æ 3

4

5

6

In Table 1, when indicating the phoneme pronounced by the speaker, the symbol changed to αː' can be used instead of αː. This means that, when the speaker is Korean, English, which should be pronounced as α due to the characteristics of Korean pronunciation, cannot be pronounced as α and is pronounced as αː'. Therefore, when pronouncing a text containing α, the phoneme list will contain αː', not α.

는 한국인의 발음 기관의 특성상, 표준 발음에 맞게 제대로 발음할 수 있는 음소로서,

를 그대로 사용할 수 있다.

is a phoneme that can be properly pronounced according to the standard pronunciation due to the characteristics of the Korean pronunciation system.

can be used as is.

On the other hand, Table 1 shows the case in which the pronounced phoneme is a vowel. In the case of pronunciation of a consonant, the tendency of voiced or unvoiced sounds, plosives, fricatives, and fricatives compared to standard IPA symbols along with the symbol of the phoneme pronounced by the speaker. and the like may be further included.

In addition, since an arbitrary phoneme may be sounded with multiple pronunciations according to the pronunciation characteristics of the speaker, multiple pronunciations may exist for one phoneme, and one phoneme may have a plurality of modified IPAs to distinguish and express them. . For example, the pronunciation of aI may be sounded with n different pronunciations, and n modified IPA vowel symbols ^{from aI', aI 2} ' to aI ^{n ' may be used to express this.}

The pronunciation image generating apparatus 10 generates learning data by labeling the collected pronunciation images and the cross-sectional image of the pronunciation organs with the corrected phoneme information, the speaker information, and the characteristics of the pronunciation organs ( S130 ).

The information for labeling the pronunciation image includes not only phoneme information but also speaker information and pronunciation organ characteristics. That is, the values shown in Table 2 may be labeled as the gender, age, and characteristics of a pronunciation organ when the speaker pronounces the phoneme, which is the main character of the pronunciation image.

division Contents type gender female/male F/M age Children / 10s / 20s to 30s / 40s to 60s / 70s and older 1-5 Pronunciation organ characteristics mouth/vocal cord size Small/Medium/Large 1-3 lips/tooth shape Side open mouth/round mouth shape/largely open lips/small gaping lips/closed lips/tooth touching 1-7 tongue position Legend/Medium/House tongue/Long tongue/Behind upper teeth/Inside of roof of mouth/Between upper and lower teeth/Behind lower teeth 1 to 8

In Table 2, gender is labeled with F and M, age is labeled with five types ranging from children to over 70 years old, and pronunciation organ features can be labeled according to each characteristic.

Specifically, the size of the oral cavity and the vocal cords may be divided into three categories, and the characteristics of the lips and the pronunciation organs according to the shape may be divided as shown in FIG. 5A . For example, in FIG. 5A , the lips and this shape may be a, e, i, o, u, f, or v when pronouncing a, e, i, o, u, f, or v in order from the front.

Also, the position of the tongue pronouncing the phoneme may be divided as shown in FIG. 5( b ). For example, (b) of FIG. 5 may be the position of the tongue when pronouncing legend, midheol, husserl, 'r', 'l', and 'ㄹ' in order from the front. .

Meanwhile, as described with reference to FIG. 4 , the pronunciation image generating apparatus 10 may use the standard IPA stored in the phoneme information DB 600 as it is or modify it.

Now that the information to be labeled is prepared, it is necessary to label the corresponding information on the image for each phoneme. However, in general, since a pronunciation image and a cross-sectional image of a pronunciation organ often utter a single sentence, the pronunciation image generating apparatus 10 must first divide the image into phoneme units as shown in FIG. 6 .

Referring to FIG. 6 , the pronounced sentence is 'I like an apple.' and the phoneme of 'apple' is 'aepl'. The video for pronouncing the entire sentence should be divided into videos for pronouncing the respective phonemes 'ae', 'p', and 'l'.

In order to divide the speech of the sentence unit into phonemes, an administrator may manually divide or the pronunciation image generating apparatus 10 may use a separate model for segmentation. As an example, the model may be a model that uses an image obtained by extracting acoustic features from each phoneme as learning data, learns acoustic features for each phoneme, and outputs a phoneme pronounced in a corresponding sentence when an arbitrary speech image is input. . In this case, the acoustic feature used may be a Mel Frequency Cepstral Coefficient (MFCC), and is not limited thereto.

Now, the pronunciation image generating apparatus 10 labels the phoneme information, speaker information, and pronunciation organ characteristics corrected in step S120 on the image divided into phoneme units. When labeling, the playback time of the corresponding video file and text of the uttered content may be further included. The labeled information may be expressed in a JSON (JavaScript Object Notation) format.

For example, when the phoneme ae45-1 is pronounced in a specific section of a specific video, image information of the section, information on the entire pronunciation (text), talker information, and pronunciation organ information (pronunciation organ) ) and can generate phoneme information in the data structure shown in Table 3.

“æ45-1”: {
“image”: {
“Frame length”: “2500”
“Frame data”: “1500”, “1700”
“Video resolution”: 1024", "768"
},
"text": {
"script": "I like an apple"
“phone”: “æ”
},
"voice" : {
"length": "2500"
“ipa phone”: “æ45-1”
“ipa phone difference: "-0.01", "+0.02"
},
"Talker": {
“Gender”: “female”
“Age”: “40”
},
“Pronunciation organ”: {
"timbre": "7"
“Jaw length”: “10”
“Neck length”: “12”
“Neck thickness”: “15”
“Oral cavity”: “25”
“vocal tract length”: "15"
“vocal tract diameter": "2"
“Lip size”: “5”
“Lip shape”: “type 1”
“Tongue position” (Type, Type change): “type 1”, “1, 1”
“Tongue movements” (height): “1”
“sound source strength”: “5”
}
}

In Table 3, ipa phone differenc indicates how close the pronounced vowel is to the pronunciation position of the standard IPA with reference to FIG. 4 .

For example, in Table 3, the phoneme pronounced by the speaker can be expressed as æ45-1, which is positioned 0.01 to the left in the x-axis direction and 0.02 in the y-axis direction in the vowel squareness relative to the position of the standard IPA symbol æ. It can be seen that it is located high. Meanwhile, in the vowel square diagram of FIG. 4 , the difference between the position of the phoneme pronounced by the speaker and the position of the standard IPA symbol may be determined according to the characteristics of the oral cavity of Koreans.

On the other hand, the pronunciation image generating apparatus 10 may use, as learning data, imaging features and voice features extracted from the image collected in step S110 in addition to the phoneme information, the speaker information, and the pronunciation organ information described above.

For example, features such as Histogram of Oriented Gradients (HOG) or Local Binary Pattern (LBP) may be extracted from an image and used. The extracted information can be labeled by adding it to the data structure of Table 3.

The pronunciation image generating apparatus 10 trains the phoneme analysis model 210 for extracting phonemes of the input sentence using the training data (S140). As an example, the phoneme analysis model 210 may be trained to output a network form in which phonemes are sequentially connected as shown in FIG. 7 .

As an example, when the learner pronounces the sentence 'I like an apple.' in FIG. 7 , the phoneme analysis model 210 may generate a network to determine what the phonemes constituting the sentence are as a probability.

For example, when a phoneme pronounced first is aI ⁿ ', a phoneme that can be pronounced after that may be l ⁿ ' or l ⁿ⁺¹ '. ^{AI n+3} ' may be pronounced after the phoneme l ⁿ ', and the phoneme ^{aI n+1} ' may be pronounced after the phoneme ^{l n+1 '.}

The phoneme analysis model 210 may use a pre-collected sentence corpus stored in a database (not shown) as training data, and may be implemented as an n-gram language model. Specifically, the SRI Language Modeling Toolkit (SRLIM) may be used.

The pronunciation image generating apparatus 10 learns the pronunciation image model 220 for generating a 3D image by using the learning data (S150).

The pronunciation image model 220 serves to output the pronunciation organs as a 3D video, and expresses the structures of the articulation organs such as lips, teeth, uvula, and palate. The size, shape, and position of the mouth, pharynx, lips, and tongue can change each time a phoneme is pronounced. The 3D shape of the pronunciation organ may be generated in a shape similar to the ratio of the actual human pronunciation organ, and the appearance of the eyes, nose, and ears, etc., regardless of pronunciation, may be arbitrarily set or changed.

The training data includes information in Table 3 labeled on a 2D or 3D pronunciation image and a 2D or 3D pronunciation organ cross-sectional image.

The pronunciation image model 220 may be implemented by machine learning, and may have a form in which a plurality of machine learning is combined. For example, Hidden Markov Models (HMMs), Deep Neural Networks (DNNs), Convolution Neural Networks (CNNs), etc. may be used.

Meanwhile, as described in FIG. 1 , the pronunciation image model 220 learned in step S140 and the phoneme analysis model 210 learned in step S150 may be one artificial intelligence model, and in this case, the learning step of the model is one process can proceed. In addition, after each model undergoes a separate learning process, the two models may be combined through an arbitrary artificial intelligence model, and the form of the model is not limited to any one.

Hereinafter, a process of outputting a 3D image displayed as a cross-sectional view of a pronunciation organ for a sentence uttered by a learner using the learned phoneme analysis model 210 and the pronunciation image model 220 and a standard pronunciation image of the sentence will be described.

8 is a flowchart of a method of operating an apparatus for generating a pronunciation image according to an embodiment; FIG. 9 is an exemplary diagram of a phoneme list predicted by a phoneme analysis model according to an embodiment; and FIG. 10 is a pronunciation according to an embodiment It is an explanatory diagram of a method in which an image model generates a simulation image, and FIGS. 11 and 12 are exemplary views of a result screen output by the apparatus for generating a pronunciation image according to an embodiment.

Referring to FIG. 8 , the audio and image input unit 100 of the pronunciation image generating apparatus 10 receives a learner's speech image ( S210 ). When the learner utters an arbitrary sentence, the pronunciation image generating apparatus 10 collects the shape and voice of the pronunciation organ when the learner speaks through the camera and the microphone. The speech image may include at least one of an image photographed from the front of the learner and an image photographed from the side.

The pronunciation image generating apparatus 10 inputs the learner's speech into the phoneme analysis model 210 to generate a list of phonemes constituting the learner's speech ( S220 ). In this case, the Viterbi algorithm may be used.

The Viterbi algorithm is an algorithm for finding the sequence of the most probable hidden states that caused the sequence of events observed in HMM, etc. In Speech to Text, if an acoustic signal is a sequence of observed events, a string is considered as the Hidden Cause that caused this acoustic signal. That is, the Viterbi algorithm is used to find the most probable character string for a given acoustic signal.

In detail, the pronunciation image generating apparatus 10 extracts phonemes that may constitute a pronounced sentence, calculates the probability that each phoneme corresponds to each phoneme, and calculates the probability of a next phoneme with respect to the current phoneme for each phoneme. do. Thereafter, the pronunciation image generating apparatus 10 may determine the network having the greatest probability as the phoneme list constituting the corresponding sentence.

As an example with reference to FIG. 9 , a case in which the learner utters the sentence 'I like an apple.' will be described. The pronunciation image generating apparatus 10 determines that the sentence uttered by the learner is 'I like an apple.' by using a known speech recognition technology (Speech to Text). After that, phonemes that can compose the sentence 'I like an apple.' can be configured in a network form.

, nⁿ⁺³', æⁿ⁺⁴', pⁿ⁺⁴', lⁿ⁺³'이라는 음소들이 순차적으로 발음된 것으로 분석할 수 있다. The learned phoneme analysis model 210 determines that the sentences pronounced by the learner are aI ⁿ⁺² ', l ⁿ⁺³ ', aI ⁿ⁺³ ', k ⁿ⁺² ',

, n ⁿ⁺³ ', æ ⁿ⁺⁴ ', p ⁿ⁺⁴ ', l ⁿ⁺³ ' can be analyzed as being pronounced sequentially.

Meanwhile, the pronunciation image generating apparatus 10 may additionally extract the length at which the learner pronounces each phoneme. The pronunciation image generating apparatus 10 may generate a pronunciation image by reflecting the pronunciation length for each phoneme.

The pronunciation image generating apparatus 10 generates a standard pronunciation phoneme list according to the speech recognition result of the learner's utterance (S230). The learned phoneme analysis model 210 may generate a standard phoneme list.

The pronunciation image generating apparatus 10 generates and outputs a pronunciation organ image for a learner's utterance according to the phoneme list generated in step S220 using the pronunciation image model 220 ( S240 ).

The pronunciation image generating apparatus 10 generates and outputs a standard pronunciation image for the standard pronunciation phoneme list generated in step S230 by using the pronunciation image model 220 ( S250 ).

Referring to FIG. 10 , the pronunciation image model 220 may determine the shape of a pronunciation organ, such as a mouth shape and a position of a tongue, for uttering a standard pronunciation based on a standard pronunciation image of a corresponding phoneme. A 3D pronunciation image may be generated as a standard pronunciation image by combining the determined characteristics of the pronunciation organ and the cross-sectional image of the pronunciation organ.

As an example, the pronunciation image generating apparatus 10 may output a screen as shown in FIG. 11 or FIG. 12, and the learner can confirm the difference between the standard pronunciation pronunciation and the learner’s pronunciation in detail by pressing buttons provided on the output screen. have.

The pronunciation image generating apparatus 10 provides a pronunciation guide by analyzing the difference between the learner image and the standard pronunciation image (S260). Based on the pronunciation guide, learners can correct their pronunciation by watching the standard pronunciation video and their own pronunciation video.

The provided speech guide may be obtained from a pre-stored DB (not shown), or the pronunciation image generating apparatus 10 may directly calculate the difference between the standard pronunciation and the learner's pronunciation and write it. The guide stored in the DB in advance includes instructions for the learner to act in order to correct the phoneme difference, and may be as shown in Table 4.

standard phoneme learner pronunciation phoneme pronunciation guide aI aI ⁿ⁺² ' Learners pronounce tongue position from low-medium to classical.
Must be pronounced with mouth open and tongue moving I l ⁿ⁺³ ' The mouth should be slightly open and the tongue should touch just behind the upper teeth. aI aI ⁿ⁺³ ' Mouth shape needs to be expanded k k ⁿ⁺² ' You have to drop the "U" sound and put your tongue in the classic way and pronounce it briefly with a popping sound.

The learner pronounces the position of the tongue with the middle and low tongue.
Tongue position should be pronounced with a heavy tongue n n ⁿ⁺³ ' The learner pronounces the tongue with the front part of the tongue on the upper gum.
Press the front part of the tongue firmly against the upper gums, while blowing air through the nose. æ æ ⁿ⁺⁴ ' The learner has the back part of the tongue raised upwards and the mouth is wide open.
Pronunciation with the middle part of the tongue up p p ⁿ⁺⁴ ' Learners pronounce with their mouths closed and slightly opened.
Close your mouth and open it slightly to the side l l ⁿ⁺³ ' The learner touches the tongue to the roof of the mouth
Tongue should touch behind teeth

As an example of calculating the difference in pronunciation, the similarity between a standard phoneme and a phoneme pronounced by the learner may be calculated. The degree of similarity may be calculated by representing the position of the phoneme in the vowel square as coordinates as shown in FIG. 4 .

On the other hand, if there is a difference in the length of the pronunciation, the content of the length may be added to the pronunciation guide.

13 is a hardware configuration diagram of a computing device according to an embodiment.

Referring to FIG. 13 , the audio and image input unit 100 , the learning unit 200 , the pronunciation image generation unit 300 , the pronunciation guide providing unit 400 , the pronunciation image DB 500 and the phoneme information DB 600 are In the computing device 700 operated by at least one processor, a program including instructions described to execute an operation of the present invention is executed.

The hardware of the computing device 700 may include at least one processor 710 , a memory 720 , a storage 730 , and a communication interface 740 , and may be connected through a bus. In addition, hardware such as an input device and an output device may be included. The computing device 700 may be loaded with various software including an operating system capable of driving a program.

The processor 710 is a device that controls the operation of the computing device 700 and may be a processor 710 of various types that processes instructions included in a program, for example, a central processing unit (CPU), an MPU ( It may be a micro processor unit), a micro controller unit (MCU), a graphic processing unit (GPU), or the like. The memory 720 loads the corresponding program so that the instructions described to execute the operation of the present invention are processed by the processor 710 . The memory 720 may be, for example, read only memory (ROM), random access memory (RAM), or the like. The storage 730 stores various data, programs, etc. required to execute the operation of the present invention. The communication interface 740 may be a wired/wireless communication module.

According to the present invention, it is possible to listen to the standard pronunciation of the native speaker for the text pronounced by the learner, and the movement of the pronunciation organ for the standard pronunciation can be checked through a simulation image, which is effective for pronunciation education.

In addition, according to the present invention, not only the image of the pronunciation organ of the native speaker but also the image of the learner's pronunciation organ are provided together, so that the learner can check the part to be corrected by the image, and the pronunciation correction can be performed alone without the help of the teacher. Convenience can be increased.

The embodiment of the present invention described above is not implemented only through the apparatus and method, and may be implemented through a program for realizing a function corresponding to the configuration of the embodiment of the present invention or a recording medium in which the program is recorded.

Although the embodiments of the present invention have been described in detail above, the scope of the present invention is not limited thereto, and various modifications and improved forms of the present invention are also provided by those skilled in the art using the basic concept of the present invention as defined in the following claims. is within the scope of the right.

Claims (11)

A method of operating a computing device operated by at least one processor, comprising:
A step of receiving the learner pronunciation voice and video for arbitrary text,
generating a list of standard phonetic phonemes of the text;
generating a learner pronunciation phoneme list by extracting phonemes from which the text is pronounced from the learner pronunciation voice and image; and
By inputting the standard pronunciation phoneme list into a learned image generation model, a standard pronunciation simulation image for pronouncing the standard pronunciation phoneme list is generated, and the learner pronunciation voice and image, and the learner pronunciation phoneme list are input to the image generation model. generating a learner pronunciation simulation image for pronouncing the learner pronunciation phoneme list, and outputting the standard pronunciation simulation image and the learner pronunciation simulation image,
The standard pronunciation simulation image and the learner pronunciation simulation image are virtual pronunciation organ images simulating a pronunciation process. In claim 1,
analyzing differences in the shape or movement of each pronunciation organ included in the standard pronunciation simulation image and the learner pronunciation simulation image, and generating a pronunciation guide including the differences
Further comprising, the method of operation. In claim 1,
Collecting a first image including the shape of the oral cavity for each phoneme and a second image including the shape of the section of the pronunciation organ for each phoneme;
generating learning data by tagging the phoneme corresponding to the first image or the second image, speaker information including the speaker's gender and age group, and the shape of the oral cavity and the shape of the pronunciation organ when the phoneme is pronounced; and
learning the relationship between the shape of the oral cavity and the shape of the section of the pronunciation organ for each phoneme using the learning data to the image generation model;
Further comprising, the method of operation. In claim 3,
The shape of the oral cavity includes at least one of the size of the oral cavity, the shape of the lips, whether the lower teeth and the upper teeth are in contact with each other, or the position of the tongue. In claim 3,
The phoneme is displayed as a phonetic symbol, and the phonetic symbol is an International Phonetic Alphabet (IPA) transformed according to the speaker information. In claim 5,
The learning data is
The method of claim 1, further comprising: a value representing a difference between a phonetic symbol modified according to the speaker information and the international phonetic symbol in a numerical value. A computing device comprising:
memory, and
at least one processor executing instructions of a program loaded into the memory;
the program is
Learning images including the shape of the oral cavity and the shape of the section of the pronunciation organ for each phoneme are collected, and the speaker information including the phoneme pronounced in the learning image, the standard phoneme of the pronounced phoneme, and the speaker's gender and age is taught by tagging generating data;
learning an image generation model for outputting a virtual pronunciation organ simulation image in which a standard phoneme of an input phoneme is pronounced, as the learning data;
inputting the learner's pronunciation voice and image for arbitrary text into the image generation model; and
displaying a standard pronunciation simulation image of the text output from the image generation model on a screen
A computing device comprising instructions written to execute In claim 7,
The learning data is
Computing device further comprising histogram information or feature point information extracted from the learning image. In claim 7,
The learning step is,
Calculate the probability of a phoneme relationship that can be pronounced consecutively,
The displaying step is
and generating a standard phoneme list of the text and a pronunciation phoneme list of the text based on the probability. In claim 9,
generating, by the virtual pronunciation organ, a learner pronunciation simulation image pronouncing the text
Computing device further comprising a. In claim 9,
Analyze and provide a difference between the standard phoneme list and the phoneme list pronounced in the text
Computing device further comprising a.

Images