KR20090051851A - Mobile communication terminal for outputting voice received text message to voice using avatar and method thereof - Google Patents

Abstract

The present invention relates to a mobile communication terminal and a control method for outputting a received text message as a voice by an avatar. The present invention relates to a mobile communication terminal for lip-syncing by moving a feature point that changes the mouth shape of an avatar and outputting a received text message as a voice by the avatar, so that a voice suitable for the lip-sync is output.

Therefore, the present invention uses a feature point that changes the shape of the avatar's mouth, so that the avatar's lip sync is displayed very naturally and a voice suitable for the lip sync is output, so that a person who has difficulty in recording text on the mobile communication terminal can easily receive information on the text message. There is an effect that can be learned.

Text message, Avatar, Voice, Mobile terminal, Mouth shape, Lip sync

Description

Mobile communication terminal for outputting received text message by avatar and its control method {Mobile communication terminal for outputting voice received text message to voice using avatar and Method

The present invention relates to a mobile communication terminal and a control method for outputting a received text message as a voice by an avatar. The present invention relates to a mobile communication terminal for lip-syncing by moving a feature point that changes the mouth shape of an avatar and outputting a received text message as a voice by the avatar, so that a voice suitable for the lip-sync is output.

As a conventional technology, Korean Patent Application Publication No. 10-2003-0039019 discloses a computer that implements lip sync and emotion expression by applying a voice sampled voice to a three-dimensionally scanned real face image while converting sentence text into speech in real time. The program stores the recording media, and online games, e-mails, chats, broadcasts, and foreign language learning methods using the first step of corresponding one-to-one correspondence for every character and storing the voice data in the T2S database, and predetermined emotions. A second step of storing a formula for transforming the position, size, and angle, which are the main elements of the image data when expressing the emotion, in a emotion expression database in a one-to-one correspondence with function characters representing the expression; A third to store the formula to transform the position, size, and angle of the lip-sync database And a fourth step of acquiring and storing the stereoscopic image of the member, sampling and acquiring the member's voice, and storing the same in a basic database; A fifth step of extracting and storing a main element, and comparing the main elements, such as frequency, roughness, and tone, which determine the characteristics of the voice in the voice data with a predetermined reference value, to determine an increment or decrement, and to store it as a basic database; A sixth step of storing text for output and expression of emotion; and loading the text into a memory to obtain the first character to determine whether it is a functional character, and if the character is a functional character, expressing emotion in a face image. T2S programs for branching to programs and converting text to speech if they are not function characters; and There was a seventh step of driving a lip sync program for expressing the same mouth shape as the vowel currently pronounced on the face image, and a technique configured to repeat the seventh step until the end of the text.

In the conventional technology, since the lip-sync program for expressing the same mouth shape as the vowel currently pronounced on the face image continuously extracts and displays the mouth shape data suitable for the face image, the disconnection phenomenon is unnaturally displayed when the mouth shape is changed. There was a problem.

The present invention has been made to solve the above-described problem, the present invention converts the text message received in the mobile communication terminal to the voice and naturally the lip sync through the movement of the feature point to change the shape of the avatar to suit the converted voice Its purpose is to.

Another purpose is to naturally change the shape of the avatar's mouth by using feature points that change the shape of the mouth.

Another purpose is to display the meaning of emoticons and facial expressions of a suitable avatar on a mobile communication terminal.

According to an embodiment of the present invention for achieving the above object, a mobile communication terminal for outputting a received text message to the voice by the avatar is a mobile communication terminal that outputs the text message as a voice, to receive the text message A text message receiver, a voice converter for converting the received text message into a voice, a phonological analyzer for analyzing each converted phoneme according to the shape of each phoneme, and a feature point for each independent phoneme A first storage unit storing lip sync data for each phoneme, a second storage unit for storing voice data for each independent phoneme, and receiving the lip sync data according to each independent phoneme to move a feature point position of the received lip sync data To change the shape of the avatar's mouth and to output the voices of the independent phonemes And a display unit for displaying a change in mouth shape of the avatar.

According to another embodiment of the present invention, a control method of a mobile communication terminal for outputting a received text message by voice by an avatar includes the steps of the mobile communication terminal receiving a text message and converting the received text message into a voice. Analyzing the converted voice for each independent phoneme according to the shape of each phoneme, and storing lip-sync data including a feature point for changing the mouth shape of the avatar suitable for each independent phoneme. Changing the position of the feature points of the lip sync data to change the shape of the avatar's mouth, and displaying the mouth shape of the avatar changed according to the positional movement of the feature points of the extracted lip sync data, and outputting a voice suitable for the lip sync data. Characterized in that consisting of steps.

According to the present invention, since the lip sync of the avatar is displayed very naturally and a voice suitable for the lip sync is output, the person who has difficulty in reading text can easily acquire the information of the text message. It can be effective.

Best Mode for Carrying Out the Invention Hereinafter, configurations and operations will be described with reference to the accompanying drawings.

1 is a conceptual diagram of a mobile communication terminal for outputting a received text message as a voice by an avatar according to the present invention, wherein the mobile communication terminal 10 can transmit a text message through a base station 20. The text message transmitted by using the TTS (Text-To-Speech) technology is converted into speech and the changed speech is divided into independent phonemes suitable for each phoneme type through phonological analysis. The lip sync data is extracted to change the position of the feature point so that the avatar lip syncs.

2 is a configuration diagram of a mobile communication terminal for outputting a received text message by voice according to the present invention. ), The first storage unit 400, the second storage unit 500, the control unit 600, and the display unit 700.

Furthermore, the text message receiver 100, the voice converter 200, the phonological analyzer 300, the first storage unit 400, the second storage unit 500, the control unit 600, and the display unit 700. The third storage unit 800 is further configured to store the facial expression data of the avatar so that the facial expression suitable for the emoticon is displayed on the display unit 700 by the controller 600.

In more detail, the text message receiving unit 100 receives the text message from various terminals capable of transmitting the text message.

The voice converter 200 converts a text message received by the text message receiver 100 into voice, and a known technology such as TTS (Text To Speech) is representative.

In particular, when an emoticon included in a text message that cannot be converted into a voice by TTS technology is recorded, the emoticon is transmitted to the controller 600.

The phonological analyzer 300 decomposes the voice converted by the voice converter 200 through phonological analysis and analyzes the plural independent phonemes for each phonological form. 600 is sent.

The first storage unit 400 is 46 expression data, the smallest unit that can no longer analyze the movement of the face is the data of the facial muscle movement and the expression of the facial bar through the 46 bar as described above The facial expression data has respective feature points, and the feature points include a facial definition parameter (FDP) and a facial animation feature point (FAP) defined by the MPEG-4 Synthetic / Natural Hybrid Coding (SNHC) protocol as shown in FIG. 3. By using the Facial Animation Parameter, a general and systematic feature point can be obtained, wherein a white point shown in FIG. 3 is a face definition feature point, a black point is a face animation feature point, and the face definition feature point includes an animation feature point.

Therefore, when the independent phoneme analyzed by the phoneme analyzer 300 is requested by the controller 600, the lip synch data having the feature points as described above is constructed and stored for each phoneme. 500).

The second storage unit 500 stores voice data for each of the independent phonemes, and the stored voice data are output together with the lip sync data by the controller 600.

The control unit 600 converts the lip sync data set as an initial reference value into the received lip sync data when the lip sync data of an avatar having a feature point suitable for independent phone calls is received from the first storage unit 400. The avatar's mouth is changed by moving the position of the included feature point.

As described above, as the controller 600 repeatedly receives the request from the first storage unit 400 for each independent phoneme, the avatar performs a lip sync suitable for a text message, extracts voice data suitable for the lip sync, and displays the display 700. To be displayed.

For example, the control unit 600 stores the lip synch data set as a reference value, that is, the lip synch data for lip-syncing the phonogram that the positions A and B of the feature point are 'y', as shown in (a) of FIG. 4. When received from (b), it moves to the feature point positions A1 and B1 as shown in (b) and lip-syncs the phoneme of 'yi'. Therefore, the present invention changes the shape of the avatar by moving the positions of the feature points as described above.

If the above-described process is repeated, the morphing program is provided so that the morphing technique is operated since the mouth shape of the avatar displayed on the display unit 600 is lip-synced very unnaturally.

In other words, the morphing program converts two images by properly connecting two images without moving to the position of A1 at a time. As shown in FIG. 5, the morphing program moves through A1-1, A1-2, and A1-3. Finally, it is located at A1.

In addition, the control unit 500 analyzes the emoticons of the text messages that are not converted into voices by the voice conversion unit 200 and applies the corresponding facial expressions from the third storage unit 800 to lip sync with the facial expressions suitable for the emoticons. And the requested facial expression is displayed on the display unit 700.

For example, in the case where the frequency of the symbols having fun meanings (e.g., ^, *, 0, ∇, ☞, ☜, ♡, etc.) which are various emoticons other than the emoticons shown in FIG. If the frequency is high for the symbol having a crying meaning (for example, a symbol such as>, <, T, ㅠ, etc.), the avatar's face is crying. If the frequency of the symbols (for example,-, _, M, X, etc.) is high, the avatar's face is controlled to be sad.

The third storage unit 800 stores 46 facial expression data, which is the smallest unit that can no longer analyze facial movements, and among the facial expression data, smiley expression data, crying expression data, sad expression Using only the data, the control unit 600 displays any one of the smiley facial expression data, the crying facial expression data, and the sad facial expression data suitable for the emoticon on the display unit 700.

The display unit 700 displays the lip sync data of the avatar transmitted by the controller 600, and displays any one of the smiling facial expression data, the crying facial expression data, and the sad facial expression data. ) Is further provided with a voice output unit 710 to output a voice suitable for lip synch data.

7 is a flowchart illustrating a method for controlling a mobile communication terminal for outputting a received text message by voice according to an embodiment of the present invention. The same steps take place.

Receiving, by the mobile communication terminal, a text message (S10), converting the received text message into a voice (S20), and analyzing the converted voice for each phoneme according to each phoneme type. (S30) and the lip sync data including a feature point for changing the mouth shape of the avatar suitable for each of the independent phoneme is stored in the mobile communication terminal to change the position of the feature point of the lip sync data to change the mouth shape of the avatar ( S40), and displaying the mouth shape of the avatar changed according to the positional movement of the feature point of the extracted lip sync data and outputting a voice suitable for the lip sync data (S50).

In more detail, in the step S10 of receiving a text message, the mobile communication terminal receives the text message from various terminals capable of transmitting the text message.

In the step S20 of converting the received text message into speech, the text message received in step S10 is converted into speech by a known technique such as TTS.

In step S30, the converted voice is analyzed according to each phoneme according to the form of each phoneme. In operation S20, the voice is independent so that the avatar displayed on the mobile communication terminal lip-syncs to the voice. Analyze by phoneme.

Lip sync data including a feature point for changing the mouth shape of the avatar suitable for each of the independent phonemes is stored in the mobile communication terminal to change the position of the feature point of the lip sync data to change the mouth shape of the avatar (S40). Upon requesting and receiving lip sync data of an avatar having a feature point for each of the independent phonemes analyzed in step, the lip sync data set as an initial reference value is converted into the received lip sync data. The avatar's mouth is changed by moving the position of the included feature point.

When the above process is repeated for the entire text message, the feature points are naturally moved by the morphing program since the mouth shape of the avatar displayed on the display unit 600 is very unnaturally lip-synced.

In step S50 of displaying the mouth shape of the avatar changed according to the positional change of the feature point of the extracted lip sync data and outputting a voice suitable for the lip sync data, the lip sync data of the avatar transmitted by the controller 600 is displayed. The facial expression data of the smiley face, crying facial expression data, sad expression data is displayed, and the display unit 700 further includes a voice output unit 710 to output a voice suitable for lip synch data. .

In addition, when the received text message has an emoticon, the facial expression output by the request of the controller 600 is displayed on the display unit 700 from the third storage unit 800 in which the facial expression of the avatar suitable for the emoticon is stored.

The avatar of the present invention described above is preferably an avatar programmed as a 3D image.

Embodiments of the present invention are disclosed in the drawings and the detailed description, and the specific terms used above are used only for the purpose of describing the present invention, and are used to limit the scope of the present invention as defined in the meaning or claims. It is not.

Therefore, those skilled in the art can be various modifications and other equivalent embodiments from this, and the true technical protection scope of the present invention should be determined by the technical spirit of the claims.

1 is a conceptual diagram of a mobile communication terminal for outputting a received text message as a voice by an avatar according to the present invention;

2 is a block diagram of a mobile communication terminal for outputting a received text message by voice according to the present invention;

3 is a diagram illustrating feature points of an avatar face according to the present invention.

Figure 4 is an exemplary lip sync in accordance with the present invention

5 is an exemplary lip sync by morphing program according to the present invention

6 is an exemplary emoticon according to the present invention

7 is a flowchart illustrating a control method of a mobile communication terminal for outputting a received text message as a voice by an avatar according to the present invention.

<Description of Symbols for Major Parts of Drawings>

10: mobile communication terminal 20: base station

30: terminal 100: text message receiving unit

200: voice conversion unit 300: phonological analysis unit

400: first storage unit 500: second storage unit

600: control unit 700: display unit

710: voice output unit 800: third storage unit

Claims (8)

In a mobile communication terminal that text messages are output as voice, A text message receiving unit for receiving the text message; A voice converter for converting the received text message into voice; A phonological analysis unit for analyzing the converted speech for each independent phoneme according to the form of each phonology; A first storage unit storing lip sync data having feature points for each independent phoneme for each phoneme; A second storage unit storing voice data for each independent phoneme; A control unit for receiving the lip sync data according to each independent phoneme, changing a shape of an avatar by moving the position of a feature point of the received lip sync data, and simultaneously outputting the voice for each independent phoneme; And And a display unit configured to display a change in the mouth shape of the avatar. The method according to claim 1, When the received text message has an emoticon, a third storage unit storing the facial expression of the avatar so that the facial expression of the avatar suitable for the emoticon is displayed on the display unit by the control unit. Mobile communication terminal to output voice by. The method according to claim 2, The control unit, Among the total number of emoticons, if the number of symbols having a pleasant meaning is frequent, the avatar's face is controlled to be in a smiling state. Also, if the number of symbols has a high frequency, the avatar's face is crying. And controlling the mobile device to output the received text message as a voice by the avatar, when the frequency of the symbol having a sad meaning is high so that the face of the avatar is sad. The method according to claim 1 to 3, wherein the control unit, And a morphing program for smoothly moving the position of the feature point where the mouth shape of the avatar changes according to the change of the independent phoneme. The method according to claim 4, wherein the avatar, A mobile communication terminal for outputting a text message received by the avatar, characterized in that the three-dimensional image programmed. The method of claim 2, wherein the display unit, And a voice output unit for outputting a voice of an independent phoneme suitable for the lip sync data. The mobile communication terminal outputs a received text message as a voice by an avatar. Receiving, by the mobile communication terminal, a text message; Converting the received text message into voice; Analyzing the converted speech for each independent phoneme according to the form of each phoneme; Lip-sync data including a feature point for changing the mouth shape of the avatar suitable for each of the independent phonemes is stored in the mobile communication terminal to change the position of the feature point of the lip-sync data to change the mouth shape of the avatar; And And displaying the mouth shape of the avatar that changes according to the positional movement of the feature point of the extracted lip sync data, and outputting a voice suitable for the lip sync data. Control method of communication terminal. The method of claim 7, wherein displaying the mouth shape of the avatar changed according to the movement of the feature point of the extracted lip sync data and outputting a voice suitable for the lip sync data is performed. And when the received text message has an emoticon, a facial expression of an avatar suitable for the emoticon is output by the avatar.

Images