KR100693658B1 - Poratable language study apparatus and method - Google Patents

Abstract

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a language learning apparatus and method for increasing the language learning efficiency by using a voice file and avatar information, in particular, in a portable language learning apparatus.

A language learning method of a portable language learning apparatus according to the present invention is a portable language learning apparatus that stores a sound source file and outputs it externally at the time of reproduction, wherein the subtitle and avatar data stored in synchronization with the sound source file are reproduced when the sound source file is reproduced. Searching for presence; Synchronizing the sound source file with the avatar data and outputting the same to the speaker and the display unit when the avatar data is present; And when the caption and avatar data are present, synchronizing the sound source file with the caption and avatar data and outputting the same to the speaker and the display unit.

Language learning, device, subtitles, avatar

Description

Portable language study apparatus and method

1 is a block diagram of a typical MP3 player.

2 is a block diagram showing a portable language learning device according to an embodiment of the present invention.

Figure 3 is a flow chart showing a language learning method of the portable language learning apparatus according to the present invention.

4 is a flow chart showing another example of a language learning method according to the present invention.

5 is a flowchart illustrating a method of synchronizing learning data for language learning according to the present invention.

6 is a diagram illustrating an example of inputting a caption, a sound waveform, and a pronunciation for synchronizing avatar data according to the present invention;

FIG. 7 is a diagram illustrating the voice waveform of FIG. 6 separated into voiced sound and unvoiced sound. FIG.

8 is a measurement of the time of the phonetic symbols separated in FIG.

FIG. 9 is a view showing lip shape for each phonetic sequence shown in FIG. 6; FIG.

10 is a view showing an example of storing the lip shape data of FIG. 9 in accordance with the pronunciation time measured in FIG.

11 is a view of synchronizing the lip shape of a voice, a caption, and an avatar for language learning according to the present invention.

<Description of Symbols for Main Parts of Drawings>

115 ... key input 120 ... display

125 Power supply 130 Control unit

135 ... decoder 140 ... D / A converter

145 Audio output 155 Storage means

156 Sound source storage 157 Subtitle storage

158 Avatar Storage

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a language learning apparatus and method for increasing the language learning efficiency by using a voice file and avatar information, in particular, in a portable language learning apparatus.

Moving Picture Experts Group (MPEG) is a group of video and multimedia experts that have been developed by experts in this field under the auspices of international standards bodies such as the International Organization for Standardization (ISO) and the International Electrotechnical Commission (IEC). It provides technical criteria such as compressing a signal, transmitting it, and restoring it again. MP3 stands for MPEG-1 Audio Layer-3, which is one of the audio parts of MPEG technology. By applying MPEG audio compression technology, audio sound information of a CD can be reduced to 1/12.

As such, advances in science and digital audio data processing technology have resulted in portable digital audio players, such as memory means and MP3 players, which enable the accumulation of sound data (ie, audio sound information) in addition to conventional sound storage media such as cassette tapes and the like. Enabled the development of.

MP3 players commonly used in recent years have built-in memory means, so that the desired sound data can be reproduced for a long time without deterioration of sound quality even without inserting a conventional sound storage medium.

1 is a block diagram of a typical portable audio player.

Referring to FIG. 1, the MP3 player includes a key input unit 15, a display unit 20, a power supply unit 25, a control unit 30, a decoder 35, a D / A converter 40, and an audio output unit 45. ), The earphone jack 50.

The key input unit 15 may include a play button, a menu button, a pause button, a song selection button, etc. for determining whether to play music file data stored in the MP3 storage unit 55, a playback method, a display form on the display unit 20, and the like. This includes. Of course, even if the user's key input may further include a hold (HOLD) switch for the MP3 player does not operate.

In addition, the display unit 20 displays music file recognition information of the MP3 storage unit 55, music file information (for example, song name, playing time, sequence number information of the corresponding song, etc.) being played, the operation state of the MP3 player, and the like. It is a means for. The information displayed through the display unit 20 is determined according to which function the MP3 player includes, and all information that needs to be notified to the user among the function information that can be operated by the MP3 player is displayed on the display unit ( 20).

In addition, the power supply unit 25 is a means for providing power required for performing normal operation. The power supply 25 may be, for example, a portable power supply such as a rechargeable battery, a battery, or an adapter connection for allowing driving power to be applied from an external power source.

In addition, the controller 30 may output the music data stored in the MP3 storage unit 55 through the earphone jack 50 by a user's manipulation through the key input unit 15. 20, the decoder 35, the MP3 storage unit 55, and the like are controlled.

The decoder 35 reads and decodes the information of the MP3 file stored in the MP3 storage unit 55 under the control of the controller 30 to generate audio digital data.

The D / A converter 40 is a means for converting audio digital data generated by the decoder 35 into an analog signal to generate audio analog data.

In addition, the audio output unit 45 amplifies and waveform-shapes the audio analog data received from the D / A converter 40 and amplifies the audio analog data through the earphone 2 connected to the earphone jack 50. Is the means by which sound data can be heard in the best possible condition.

However, the conventional MP3 player has a function of simply outputting only the voice through a speaker or outputting subtitle information together with the voice, so that only the listening-oriented foreign language learning function of simply listening to the voice and checking the subtitle is provided. As a result, language learning is not as efficient as any other device.

A first object of the present invention is a portable language learning apparatus and method for outputting an avatar lip data together with the voice by using the files by filing an avatar lip shape matched to a voice file made for any purpose. In providing.

A second object of the present invention is to provide a portable language learning apparatus and method for outputting avatar lip data matching the voice together with voice and subtitles.

Portable language learning apparatus according to the present invention for achieving the above object,

In the portable language learning device that stores the sound source and outputs to the outside during playback,

Storage means including a sound source storage unit for storing the sound source file and an avatar storage unit for storing avatar data corresponding to sound source information of the sound source file;

And control means for separating voiced and unvoiced sounds and measuring time of phonetic symbols.

Preferably, the storage means further comprises a subtitle storage unit for storing a subtitle corresponding to the sound source information of the sound source file.

Preferably, the atava data and the caption data are stored in synchronization with the sound source information of the sound source file.

Preferably, the control means is characterized in that for outputting the audio output and the avatar data of the sound source file in synchronization with the reproduction.

The language learning method of the portable language learning apparatus according to the present invention,

In the language learning method of the portable language learning device that stores the sound source file and outputs to the outside when playing,

Retrieving the presence or absence of subtitles and avatar data stored in synchronization with the sound source file when playing the sound source file;

Synchronizing the sound source file with the avatar data and outputting the same to the speaker and the display unit when there is only the avatar data;

And when the caption and avatar data are present, synchronizing the sound source file with the caption and avatar data and outputting the same to the speaker and the display unit.

Preferably, the voice pronunciation symbols of the sound source file are separated into voiced sounds and unvoiced sounds, respectively, and after measuring time of the separated sound symbols, the lip shape data is stored according to the pronunciation order, and the sound source file and the avatar data are synchronized. It characterized in that the storage.

As an example of a portable language learning apparatus according to the present invention, the language learning apparatus of MP3 Player is as shown in FIG. 2.

Referring to FIG. 2, a key input unit 115 for key input, a display unit 120 for displaying sound source reproduction information such as subtitles and avatar information, a power supply unit 125 for supplying power to each unit, and voice reproduction And a controller 130 for controlling screen output, a decoder 135 for decoding sound source information, a D / A converter 140 for converting an output of the decoder 135 into an analog signal, and the analog conversion. An audio output unit 145 for outputting a signal to a speaker or earphone jack, and a storage means 155 for storing sound source files, subtitles, and avatar data.

Here, the storage means 155 includes a sound source storage unit 156 storing a sound source file, a subtitle storage unit 157 storing subtitle information, and an avatar storage unit 158 storing avatar data corresponding to the sound source file. It is a constitution.

Referring to the accompanying drawings, a portable language learning apparatus and method according to an exemplary embodiment of the present invention configured as described above are as follows. For convenience of description, the present invention will be described using a portable language learning device as an example of an MP3 player.

First, referring to FIG. 2, the key input unit 115 inputs various keys for controlling playback of a sound source file, and the display unit 120 includes, for example, an LCD display unit, and includes various pieces of information of the reproduced sound source file together. Mark it. The power supply 125 supplies power to each part of the system.

The controller 130 controls each unit of the system, and controls playback of the sound source and screen output using the file information stored in the storage unit 155.

To this end, the controller 155 reads a sound source file of the storage means 155 and outputs it to the decoder 135 while reading a subtitle file and avatar data matching the sound source file and subtitle and avatar information. It is output to the screen display unit 120.

In this case, since the sound source, the subtitle, and the avatar data of the sound source file are synchronized with each other and stored as respective files, the subtitle file and the avatar data may be extracted simultaneously with the playback of the sound source.

When the sound source output to the decoder 135 is converted into an analog signal by the D / A converter 140 and then output to the outside through the audio output unit 145, the screen display unit 120 displays the audio. Subtitle information and avatar information synchronized with the output voice are output together.

The storage means 155 includes a sound source storage unit 156 storing sound source files, a subtitle storage unit 157 storing subtitle files, and an avatar storage unit 158 storing avatar data.

The sound source storage unit 156 is stored as an audio compressed file, such as an MP3 file, which is most used in a portable multimedia device. The subtitle storage unit 156 stores a character corresponding to the sound source file as a subtitle file, and is output to the screen in synchronization with the sound source. The avatar storage unit 158 stores data consisting of avatar information corresponding to the voice of each sound source file, and is output to the screen in synchronization with the sound source.

Here, avatar information generally refers to an alter ego and incarnation, and is an animated character that replaces a user's role in cyberspace. In the Internet era, two-dimensional or three-dimensional or virtual reality games or the web are used. Point to a graphic icon that represents yourself in chat, etc.

In the present invention, the avatar information may be composed of graphic information representing each sentence or voice that is a component of the sound source. For example, it is composed of a lip shape corresponding to voice.

As another example, the avatar information may be information including not only the shape of lips but also object (s) matching a person shape, an object, or a sound source. In addition, if the playback sound source is a music file, it may be avatar information consisting of a lip shape corresponding to a dance motion or a voice corresponding to a melody. have. The avatar information may be file information made of image information, graphic information, flash information, etc. that match or correspond to a sound source and a subtitle.

Here, the caption information may be included in the avatar information and displayed or may not be displayed on the screen when there is no corresponding file.

In addition, the MP3 player device as shown in FIG. 2 may further include a microphone and an encoder, and may receive a user voice or an external voice input from the microphone, encode the encoder, and then record an MP3 file. The recorded voice may be output along with the shape of the lips of the lip sync by analyzing the voice in the recorded MP3 file.

3 is a flowchart illustrating a language learning method according to the present invention. When playback of a sound source file (eg MP3) is selected (S101), the subtitle and avatar data corresponding to the playback file are searched (S103), and the search is performed. If there is no caption data, the avatar data is displayed on the screen in synchronization with the audio output (S105 and S107).

After the caption and avatar data search (S103), if there is caption information, the caption and avatar data are displayed on the screen together with the audio output (S105 and S109).

4 is another example of the language learning method of the present invention.

Referring to FIG. 4, when the sound source file is reproduced (S121), voice pronunciation of the reproduced sound source file is analyzed, voiced sounds and unvoiced sounds are separated, and time of the pronunciation symbols is measured (S123). After searching each avatar corresponding to the analyzed voice pronunciation (S125), the avatar is output to the screen in synchronization with the reproduced voice output (S127).

5 is a flowchart illustrating a method for synchronizing avatars in language learning according to the present invention.

Referring to FIG. 5, a certain sentence, a voice waveform thereof, and a pronunciation thereof are input (S131). In this case, the voice is analyzed and separated into voiced and unvoiced sounds, respectively (S133), and time of the separated phonetic symbols is measured, respectively (S135). Then, a lip shape for lip-syncing is made for each pronunciation order (S137).

Thereafter, the lip shape data is stored according to the measured pronunciation time (S139). That is, since the pronunciation (lip) shape of the voice is different according to the pronunciation time, it is calculated and stored. In addition, the specific sound source file may be stored together with the sound source, subtitles (sentence information), and avatar information, and the sound source information, subtitles, and avatar information may be separately stored as individual files.

Meanwhile, each step for avatar synchronization as described above will be described in detail as follows.

A method of synchronizing the avatar information will be described with reference to FIGS. 6 to 11 using the lip shape of the sound source as an example. Here, the sound source waveform is written arbitrarily and the pronounced sentence is "It is a cat".

First, if the "It is a cat" voice outputted to the speaker is expressed by sentence, voice waveform, and pronunciation as shown in FIG. 6, it is inputted, and the voice is divided into voiced and unvoiced sounds, respectively, as shown in FIG. 7. do. At this time, each voiced, unvoiced, and no voice section is divided into vertical lines, which can be done in software or through a voice expert.

8 measures the time of the phonetic symbols separated as shown in FIG. 7. In this case, since the pronunciation duration of each voiced and unvoiced sound is different, accurate time measurement is required. In FIG. 8, A-K is information representing a time flow.

9 creates a lip shape for each pronunciation order. The database may be databased according to the shape of the lip according to the pronunciation and then linked to the corresponding shape of the lip according to the pronunciation order. Here, since the shape of the human lips is greatly affected by the voiced sound compared to the voiceless sound, the shape of the lips is mainly made of the shape of the lips.

FIG. 10 stores lip shape data matching the voice information according to the pronunciation time measured in FIG. 8. Here, since the lip shape is to be synchronized with the voice waveform, the lip shape data should be stored at the same time as the pronunciation duration of each voiced and unvoiced sound mentioned in FIG. 8.

FIG. 11 is an example displayed on the screen, and finally confirms whether the voice, the corresponding subtitle, and the lip shape of the avatar are properly synchronized.

As described above, since the lip shape data corresponding to the voice or / and subtitle data can be obtained, the lip shape data and the subtitle can be automatically loaded and played when the corresponding voice is stored after storing the data in the MP3 player. have.

In addition, after the Ataba lip data is made into a database, an audio file is output and the corresponding sound source information is analyzed, and the avatar data (or graphic data) can be extracted and reproduced from the database.

Here, the lip shape displayed according to the capacity of the storage means may be expressed using a solid line or processed by an animation method and stored.

So far, the present invention has been described with reference to the preferred embodiments, and those skilled in the art to which the present invention pertains to the detailed description of the present invention and other forms of embodiments within the essential technical scope of the present invention. Could be implemented. Here, the essential technical scope of the present invention is shown in the claims, and all differences within the equivalent range will be construed as being included in the present invention.

As described above, according to the portable language learning apparatus and method according to the present invention, it is possible to enhance the language learning effect by outputting avatars through basic functions such as listening and viewing subtitles, and double screens.

And, by providing the avatar in the shape of a lip sync corresponding to the current voice pronunciation, there is an effect that can be learned while listening and speaking while moving.

Claims (8)

In the portable language learning device that stores the sound source and outputs to the outside during playback, Storage means including a sound source storage unit for storing the sound source file and an avatar storage unit for storing avatar data corresponding to sound source information of the sound source file; And a control means for separating voiced and unvoiced sounds and measuring time of phonetic symbols. The method of claim 1, The storage means further comprises a subtitle storage unit for storing a subtitle file corresponding to the sound source information of the sound source file. The method of claim 2, And the caption file and the atava data are synchronized with and stored in the sound source information of the sound source file. The method of claim 1, The control means is a portable language learning device, characterized in that for outputting the synchronized audio output and avatar data of the sound source file when playing.       In the portable language learning device that stores the sound source files and outputs them to the outside during playback book, Subtitles and / or avatars stored in synchronization with the sound source file when playing the sound source file Retrieving the presence or absence of data; And And synchronizing the sound source file with the avatar data and outputting the same to the speaker and the display unit when there is only the avatar data. Language learning method of a portable language learning device. The method of claim 5, The voice pronunciation symbols of the sound source file are divided into voiced sounds and unvoiced sounds, respectively, and after measuring the time of the separated phonetic symbols, storing the avatar data in accordance with the pronunciation order, and synchronizing and storing the sound source file and the avatar data. The language learning method of the portable language learning apparatus characterized by the above-mentioned. The method according to claim 5 or 6, The avatar data has a lip shape as a graphic object corresponding to each phonetic symbol of the sound source file. The portable language learning apparatus of claim 5, further comprising: synchronizing the subtitle and avatar data with the sound source file and outputting the same to the speaker and the display unit when the search result and the subtitle and avatar data are present. Language learning method.

Images