KR20190130791A - Apparatus for interactive language learning using foreign Video contents - Google Patents

Abstract

The present invention relates to a conversational foreign language learning method using foreign language video content, and a device for the same. The present invention allows a user to improve conversation ability while listening and speaking in a conversation method with a conversational learning device by using the same, and helps the user to learn gradually various expressions by adding new expressions to a video content DB according to user ability improvement. The conversational device comprises: an audio processing unit which processes a voice input inputted from a microphone and a sound of a speaker outputted from the speaker; a screen display unit which displays video scenes when foreign language video content is played, and displays the corresponding subtitle texts; and a DB unit which comprises a user function command database (DB) and a video content DB.

Description

Apparatus for interactive language learning using foreign video contents}

The present invention relates to a method and an interactive learning apparatus for a user to learn a foreign language interactively with a learning apparatus using a foreign language video content of a foreign movie subtitle or a foreign movie video file provided with subtitles of both a foreign language and a native language. In detail, an audio processing unit for processing a voice input from a microphone and a voice of a speaker output from a speaker, a screen display unit for displaying a video scene when the foreign language video content is played and displaying the corresponding caption text, and a user function command database ( The user's voice stored in the user function command DB and the video content DB using an interactive learning device having a DB including a database (hereinafter referred to as 'DB') and a video content DB. Any input stored in the above DB part If the voice recognition function that recognizes the correspondence and the user's voice input during the foreign language learning process is recognized as a user function command, the command assigned to the command is executed and the user's voice is recognized as a representation stored in the video content DB. If the user speaks the corresponding expression, and if the user's voice is not the expression stored in the DB unit, the user is treated as saying the expression that is not in the DB unit. The present invention relates to an interactive foreign language learning method using a foreign language video content and an apparatus therefor, which improves communication ability and adds new expressions to a video content DB so that various expressions can be gradually learned.

In general, a movie played on a video player such as a PC, a notebook, a tablet PC, a smartphone, and the like is AVI (Audio Video), which is a representative video format supported by all video players including the media player included in Windows. Interleaved) and the standard organization for video and audio compression, MPEG, and ASF (Advanced Streaming Format), MPEG4 V7 codec used for internet broadcasting, video lectures, etc. There are various types of video files, such as small WMVs.

In the case of a foreign language video file in which a foreign language voice is output, such as a foreign movie, a drama, or a news file, there are those that are provided with subtitles in the corresponding foreign language as well as translated local language subtitles. These files contain information on when the display of the subtitles corresponding to the scene starts and when the display of the subtitles ends, along with the information on the subtitles of the foreign language and the native language. It can be used as a great foreign language learning material.

The inventor of the present invention discloses an invention regarding a "subtitle display and subtitle audio output method when playing a video file" that allows foreign language learning by using foreign language video content having foreign and national language subtitles as described in Korean Patent Application No. 10-2012-0091110 ( The application was filed on Aug. 1, 2012).

The patent application does not allow for interactive foreign language learning because only foreign language learning with foreign language and national language subtitles has spoken language only.

On the other hand, the conventional interactive foreign language learning device has a disadvantage that it is difficult to continue the conversation naturally because the rate of recognizing the user's voice is low.

The reason for the low speech recognition rate as described above is that it requires too much contrast expression and speech data because it does not limit the content of the expression to be learned and the speech material to be contrasted with.

At present, attempts have been made to enable foreign language learning using speech recognition and text-to-speech (TTS) functions, but this also does not limit the recognition target, so the recognition rate is too low to continue conversation. .

The foreign language subtitles or subtitles of both foreign languages and native languages are provided in a video player that can play video contents such as a digital information terminal such as a PC, a notebook computer, a tablet PC, a smart phone, etc. In order to enable the user to smoothly learn foreign languages interactively with a learning device using foreign language video contents made up of foreign movie video files, the following tasks should be solved. It is provided in the form of how to make it the basic material of the interactive learning apparatus.

This is because the above foreign language video contents are delivered to the user unilaterally, and thus, the interactive learning method cannot be developed using a special method unless a special method is devised.

In order to solve the above technical problem, an interactive foreign language learning apparatus using foreign language video content according to the present invention includes an audio processor for processing a voice input from a microphone and a voice of a speaker output from a speaker, and when playing the foreign language video content. On-screen display for displaying a video scene and the corresponding subtitle text, Foreign language or national language expressions for the user function command expressions for the user to perform the functions provided by the learning apparatus for interactive foreign language learning progress Voice file recording expressions with the user's voice and the user function command DB that matches and stores the contents of functions to be performed by the learning apparatus when each expression is inputted, and scenes corresponding to foreign language voices in the video content. Information about your screen and subtitles or DB section including a video content DB storing both Korean and Korean subtitles, time information on the start time and end time of appearance of the subtitles, and a voice file recorded by the user's voice with the contents of the foreign language subtitles. And a controller which controls the input and output and the inside of the learning apparatus as a whole, wherein the controller is configured to compare the input user's voice with the voices of the user stored in the DB unit. It performs the voice recognition function to recognize which expression is stored, and if the user's voice input to the learning device is recognized as the user function command stored in the user function command DB during the foreign language learning process, the command assigned to the command is executed. When the voice of the input user is recognized as a representation stored in the video content DB, the user If the expression of the user is spoken, and if the input voice of the user is not stored in the DB, characterized in that the processing is said to say the expression not in the DB.

According to the present invention, an interactive foreign language learning method using a foreign language subtitle or a foreign language video content having subtitles of both foreign language and native language, and the interactive learning apparatus therefor are provided by a function command input by a user's voice. The foreign language learning device can execute the functions, and the user of the foreign language learning device can hear and speak in a conversational form to improve the user's communication ability and add new expressions to the video content DB according to the improvement of the ability to learn various expressions gradually. There is.

1 is an overall block diagram of an interactive foreign language learning apparatus according to the present invention.
2 is a flowchart illustrating a process of constructing a DB unit in the present invention.
3 is a view showing a process of processing according to a user's voice input in the present invention.
4 is a flowchart of a process associated with a user function command DB according to the present invention.
Figure 5 is a flow chart of the process related to the video content DB in the present invention.

Hereinafter, an interactive foreign language learning method using a foreign language video content having subtitles of a foreign language or a foreign language and a native language according to the present invention and an interactive foreign language learning apparatus therefor will be described with reference to the drawings.

The interactive foreign language learning apparatus according to the present invention is provided with foreign language subtitles or subtitles of both foreign languages and native languages in a video player capable of playing a video, such as a digital information terminal such as a PC, a notebook computer, a tablet PC, a smartphone, and the like. The present invention relates to an interactive foreign language learning apparatus in which a user interactively learns a foreign language using a foreign language video content such as a movie file. The microphone 11 and the voice of which an external voice is input as shown in FIG. An audio processing unit 10 for processing the audio output from the speaker 12 and the microphone 11 and the speaker 12 to be output to the outside, and displays the video scene when the foreign-language video content playback, and displays the subtitle text A screen display unit 20, a DB unit 40 for storing data related to a function command and a video content of a learning apparatus, and an external input signal A is composed of a receiving input unit 50, a memory unit 60 configured by an internal memory such as an external memory such as a flash memory such as a USB, the control section 30, etc. for controlling the all of the configuration and the like.

Here, the video / subtitle DB stores video contents such as movie, drama, documentary, news, and caption contents corresponding to each video contents, and the caption contents are composed of dialogue contents and corresponding time information of the video. It is.

The DB unit 40 has a user function command DB 41 for storing the user function command recorded in a foreign language or a native language voice of the user in correspondence with the corresponding functions, and the user reads and records the foreign language according to the foreign language. It consists of a video content DB (42) for storing the user's foreign language voice matching with the scene, and then updating and storing the user's foreign language voice according to the user's selection during the foreign language learning process of the user.

In addition, the interactive foreign language learning apparatus recognizes a voice when a user function command recorded in a foreign language or a native language voice and the user foreign language voice stored in the video content DB 42 are output. In addition to this, a TTS (Text to Speech) unit 80, which reads the expression stored in the form of foreign subtitles or subtitle text of both foreign languages and native languages, can be read by human-recognized voice. Can be.

The voice recognition unit 70 and the TTS unit 80 may be made of hardware, but may be performed by software.

The interactive foreign language learning apparatus compares the user function commands stored in the user function command DB (41) with the user's voice when the user speaks with respect to the phrase of the corresponding function during the foreign language learning process of the user. It considers the corresponding function as commanded and performs the function to enable interactive foreign language learning.

On the other hand, the interactive foreign language learning method using the foreign language subtitles or foreign language video content with subtitles of both foreign languages and native languages according to the present invention with reference to Figures 2 to 5 as follows.

Referring to Figure 2 the process of building the DB in the present invention as follows.

First, the foreign language or national language expressions of the user function command expressions for which the user instructs the user to perform the functions provided by the learning apparatus for the interactive foreign language learning process, and the learning apparatus must perform the respective expressions. And a user function command DB 41 for matching and storing the contents of the function to be performed (step S11), receiving a user voice for each of the user function command expressions, recording the received voice (step S12), and the corresponding text and its contents. When the representations of the text are input, the learning apparatus matches the functions to be performed and stores them in the user function command DB 41 (step S13) to build the DB unit 40 (step S30).

In addition, by matching each of the foreign language voice in the foreign language video content with information on the screen of the scene corresponding to the foreign language subtitles or both the foreign language and national language subtitles and time information about the start time and end time of the appearance of the subtitles It is provided with a video content DB 42 for storing (step S21), the learning apparatus receives the foreign language voice in the video content by the user's voice for each subtitle unit and records it (step S22), the foreign language voice of the scene And the screen information and the corresponding foreign language subtitles or foreign language and national language subtitles are matched and stored in the video content DB 42 to construct the DB unit 40 (step S30).

Next, the process according to the voice of the user input in the present invention will be described with reference to FIG.

When the user's voice is input to the learning device (step S41), the controller 30 of the learning device uses the voice recognition function of the voice recognition function unit 70 to input the user's voice into the DB unit 40. Contrast with the voices of the user stored in step (S42).

As a result, if the voice of the input user is recognized as the user voice stored in the user function command DB 41 (step S43), the function command assigned to the user voice is performed (step S44). When the voice of the user is recognized as the user voice stored in the video content DB 42 (step S45), the voice of the foreign language corresponding to the voice is recognized (step S46), and the input voice is input to the user function command DB 41. If it is recognized that neither the stored user voice nor the user voice stored in the video content DB 42, that is, the user voice stored in the DB unit 40 (step S47), the DB unit 40 The expression that does not exist is treated as said (step S48).

As a method of processing the expression that is not in the DB unit 40 as being said, since the expression of the input voice of the user is not present in the DB unit 40, it is required to speak again, or the user is asked whether he wants a hint, The contents can be configured in various ways such as showing a subtitle of the native language on the screen or asking whether the native language text is to be read by the TTS function.

Next, a process related to building a user function command DB in the present invention will be described with reference to FIG. 4.

The user function command DB is to input, record and store a user function command in a foreign language or a native language as a user's voice in order to increase the speech recognition rate in the voice recognition unit 70. The user receives a foreign language or a native language voice of the user through a microphone with respect to a command phrase in a word or sentence for each function to be performed in order to proceed with foreign language learning and records it as a user function command (step S51). In this case, a foreign language or a native language voice may be used depending on the foreign language ability of the user. However, it is preferable to use a foreign language voice rather than the native language voice for foreign language learning.

In recording the user function command by the user's voice, the user function command in text is recorded by the user's voice. In this case, the user function command may be spoken to the user by using the TTS function of the TTS unit 80 and the user may be recorded accordingly.

When a foreign language or a native language voice of the user's function command is recorded as a user function command, the user function command word is repeatedly matched with the corresponding functions and stored in the user function command DB 41 to construct a user function command DB. (Step S52).

The function of the interactive foreign language learning apparatus is controlled using the user function command stored in the user function command DB 41 (step S53).

If you want to change a specific user function command while learning a foreign language in the interactive foreign language learning device, the user inputs a new function command through a microphone for the function command and records the updated user function command (S54). ), And changes the existing function command (voice input of a word or sentence) previously recorded (step S55) and stores it in the user function command DB 41.

By doing as above. In the process of learning a foreign language, the user can change the function command according to the user's foreign language ability, thereby improving the user's conversational ability and adding new expressions to the function command expression DB as the ability is improved. Will be.

When the user speaks the command phrase of the corresponding function with a voice in the user's foreign language learning process, the voice recognition unit 70 recognizes the voice and stores the user function command words stored in the user function command DB 41. It is possible to perform the corresponding function in consideration of the corresponding function in preparation for, in this case, because the target for the voice recognition of the voice recognition unit 70 is limited to the voice data recorded by the user's voice Speech recognition rate is greatly improved.

Illustrating the things that can be included in the content of the user function command is as follows.

(1) Asking the learning device whether to start a conversation and answering the learning device if such a question is asked.

Ex) Would you like to have a conversation?

Yes, I would like that.

(2) The content of the learner's presentation to the learner again or repeatedly, or to be shown in foreign language text or in both foreign and national languages, or to be spoken in a native language.

Ex) Would you talk to me again?

Please repeat it. Would you repeat it?

Would you show me the english text of it?

Would you show me the korean text of it?

Would you tell me in korean?

(3) Contents that instruct the role play while playing the expressions of the DB unit 40 in order.

Ex) Let ’s say it in turns.

Let ’s enjoy the conversation by the roll play.

(4) Contents that instruct you to move to a specific point in time (5 minutes ago, 10 minutes ago, first time, etc.), or to the last conversation, the next conversation, the last actor (or bookmark), the next chapter, etc.

Ex) Let ’s move to previous sentence.

Let ’s move to next sentence.

Let ’s move to previous chapter.

Let ’s move to next chapter.

Let ’s move to previous bookmark.

Let ’s move to next bookmark.

(5) Contents ordered to find and move to a place containing a specific word or sentence expression

Ex) Let ’s search and move to the express “”.

(6) Command to end conversation and end

Ex) OK, That is it. Let ’s finish.

(7) whether the learning device will begin learning at a fixed time or at any time, even if the learner does not require it;

Ex) Please start talking with me after 30 minutes.

The user function command may be matched to a corresponding function in singular or plural number so that various expressions correspond to one function so that the user can use a plurality of function commands to express various expressions for one function. In this way, speaking the functional commands in various ways and listening to the functional language expressions spoken by the learning device can achieve the learning effect of foreign language conversation.

Next, the process related to the video content DB in the present invention will be described with reference to FIG. 5.

First, when a foreign language video content including a foreign language subtitle or a foreign movie video file having subtitles of both foreign language and native language is played, the foreign language subtitle or native language subtitle is displayed on the screen display. It is displayed along with the video screen (step S61), and the user sees the foreign language subtitles or the native language subtitles of the specific scene with the foreign language voice and inputs and records the foreign language voice of the user through the microphone 11 (step S62).

In this case, the learning apparatus may record the foreign language voice recorded by the user's voice and listen to the foreign language voice stored in the video content DB, and speak accordingly.

When the user's foreign language voice for the foreign language voice of the specific scene is recorded as the user's foreign language voice, the user's foreign language voice is matched with the specific scene and stored in the video content DB 42, and the video content DB 42 is repeated. (Step S63).

When the video content DB 42 is constructed primarily, when the interactive learning function is performed, when the foreign language video content including the foreign language text or the text of both the foreign language and the native language is played, the user outputs a foreign language voice. In this case, the user may hear a foreign language voice of the corresponding user already stored in the video content DB 42.

In this way, the foreign language voice recorded by the user and the current foreign language voice can be prepared so that the user can feel the progress of his / her own learning, and thus can significantly help foreign language learning.

As described above, when the user follows a foreign language voice or reads a foreign language subtitle in a scene where a foreign language voice is output and inputs it through the microphone 11, the foreign language voice of the current user is compared with the foreign language voice of the user previously recorded. If good, the user records the foreign language voice of the current user as the foreign language voice of the current user (step S64), and changes the foreign language voice of the existing user previously recorded by the user to the foreign language voice of the current user (step S65). Stored in the DB 42.

The video content DB 42 may change (update) the foreign language voice of the current user to a foreign language voice of an existing user, but may be periodically selected as the learning progresses.

A method of matching a foreign language voice of a user corresponding to a specific scene to the video content DB 42 may include a presentation time of subtitle text set in a foreign language video content including a foreign movie subtitle or a foreign movie video file including subtitles of both a foreign language and a native language. The user inputs the voice of the user corresponding to the foreign language voice using the following function, which reads and records the foreign language as the user's voice for each scene where the foreign language voice is output based on the information, and records the same. ).

As a method of conducting learning using the interactive foreign language learning apparatus, there is a method of simply telling the contents of foreign language video contents to a user in a simple order (simplely telling method), and subtitles are expressed according to predetermined time information of the foreign language video contents. As soon as the dialogue is made, the learning device and the user can talk to each other in turn (crossing method), and the learning device first speaks singular or plural expressions and the user follows it (following method). In addition, the learning apparatus and the user may decide in advance the expressions in the foreign language video content, and determine a singular or plural learning pattern such as a method of conducting a conversation according to the defined role (role play method), and learn according to this pattern. Proceed.

Next, the present invention may further use a learning device function word DB 43 for storing a learning device function word expression that the learning device needs to deliver to a user for interactive foreign language learning progress as a foreign language text or a text of both a foreign language and a native language. The following describes the process involved.

The learning device function word DB 43 is a database in which the learning device stores representations other than the representations in the video content DB in order for the learning to proceed in a fully interactive manner. These are expressions that the learning device delivers to the user for the purpose of natural interactive learning as opposed to the user function command expressions seen above.

The learning device function word DB 43 basically stores a function word representation composed of a foreign language subtitle or subtitle text of both a foreign language and a native language, and the learning device uses the TTS function of the TTS unit 80. The expressions stored in 43 are spoken to the user in a foreign language or a native language voice according to the contents set in the learning apparatus. In order to enhance the foreign language learning effect, it is preferable to speak a foreign language voice, but a native language voice may be used for beginners.

When the learning device function word expression is delivered to the user by voice can be set in advance as necessary for smooth and fun progress of interactive learning. For example, if it is recognized that the user speaks the user function command expression, the control unit of the learning device ( 30) may simply answer before performing the commanded function or retell the content of the command being executed, and the learner may express an expression that is not in the user function command DB 41 or the video content DB 42. If it is recognized that such an expression is not in the DB unit 40, it is requested to say it again, or the user is asked whether he wants a hint, and the content of the hint also shows a national subtitle on the screen or reads a foreign language text or a national language text with the TTS function. Various configurations, such as asking whether you want to hear the content Can.

Examples of what may be included in the content of the learning device function word expression stored in the learning device function word DB 43 are as follows.

(1) Asking and answering whether to start a conversation

Ex) Would you like to have a convestation?

Yes, I would like that.

(2) The content that requires the learner to speak the language again or repeatedly, to show it in a foreign language text or a native language text, or to speak a native language.

Ex) Would you talk to me again?

Please repeat it. Would you repeat it?

Would you like to show me the english text of it?

Would you to see the korean text of it?

Would you like to say it in korean?

(3) Contents that instruct the role play while playing the expressions of the DB unit 40 in order.

Ex) Let ’s say it in turns.

Let ’s enjoy the conversation by the roll play.

(4) If the command is directed to move to a specific point in time (5 minutes ago, 10 minutes ago, first time, etc.), or to the previous conversation, the next conversation, the previous chapter (or bookmark), the next chapter, etc.

Ex) O.K., I will move to the previous sentence.

O.K., I will move to the next sentence.

O.K., I will move to the previous chapter.

O.K., I will move to the next chapter.

O.K., I will move to the previous bookmark.

O.K., I will move to the next bookmark.

(5) Contents answered when ordered to find and move to a place containing a specific word or sentence expression

Yes, I will search and move to the express “”.

(6) Contents requiring presentation and selection of options if the learner needs to be selected in the course of performing each command.

Ex) Do you want one of this list?

(7) Commanded and responded to the end of a conversation

Ex) OK, I will finish.

(8) If the learning device does not match the expression that the learner says, the learner could not understand it, so that the learner speaks again by speaking the functional expression that requires the user to speak again.

Ex) I could not catch your meaning. Please tell me again.

(9) It is unclear whether the learning device is consistent with what the learner is saying, but if there are a plurality of very close expressions, the learning device presents them and asks the learner which of them.

Ex) I am not sure what do you mean. Did you tell [0115] me one of these?

Please tell me which one is your meaning.

(10) whether the learning device will start learning at a fixed time or at any time, even if the learner does not require it;

Ex) Would you like me to start talking after 30 minutes?

The learning device function word expressions can be matched with a singular or plural in the case so that various expressions can correspond to one situation so that the user can hear various expressions for one situation. Listening to the language itself can produce the learning effect of foreign language conversation.

The interactive foreign language learning apparatus further includes a basic learning expression DB 44 for storing foreign language text or texts of both foreign languages and native languages for basic learning expressions to be provided, and foreign language speech material corresponding to the foreign language text. And the basic learning expression DB 44 to allow the user to learn basic learning expressions. In the basic learning expressions, video information corresponding to the expression may be included together or only voice and text may be displayed without the video information.

When the expressions of the basic learning subjects are additionally stored as described above, the contents of the foreign language video content and the contents of the basic learning expressions may be crossed or mixed with each other.

In the above, the specific Example of this invention was described above. However, the spirit and scope of the present invention are not limited to these specific embodiments, and various modifications and variations can be made without departing from the spirit of the present invention. If you grow up, you will understand.

Therefore, since the embodiments described above are provided to completely inform the scope of the invention to those skilled in the art, it should be understood that they are exemplary in all respects and not limiting. The invention is only defined by the scope of the claims.

10: audio processor 11: microphone
12: Speaker 20: On-screen Display
30: control unit 40: DB unit
41: user function command DB 42: video content DB
43: learning device function DB 44: basic learning expression DB
50: input unit 60: memory unit
70: voice recognition unit 80: TTS unit

Claims (1)

In an interactive foreign language learning apparatus that can learn a foreign language interactively using foreign language subtitles or foreign language video content having subtitles of both foreign languages and native languages,
An audio processor for processing a voice input from a microphone and a voice of a speaker output from a speaker, a screen display for displaying a video scene when the foreign language video content is played, and displaying a corresponding subtitle text, and learning for interactive foreign language learning The learning apparatus performs foreign language or national language expressions of user function command expressions and a voice file recording the expressions with the user's voice and each expression thereof. User function command DB that matches and stores the contents of the function, information about the scene screen corresponding to the foreign language voice in the video contents, foreign subtitles or both foreign and national language subtitles, and the start time and end of the appearance of the subtitles Time information about the viewpoint and the subtitle of the foreign language And a control unit for controlling the input and output and the inside of the learning apparatus as a whole. The voice of the user is compared with the voices of the user stored in the DB unit to perform a voice recognition function to recognize which expression stored in the DB unit, and is input to the learning device during the foreign language learning process If the user's voice is recognized as the user function command stored in the user function command DB, the command assigned to the command is executed. If the input user's voice is recognized as the expression stored in the video content DB, the user speaks the expression. If the input voice of the user is not a representation stored in the DB unit The video content DB stores video contents such as movies, dramas, documentaries, and news, and subtitle contents corresponding to each video contents. Interactive foreign language learning apparatus using foreign language video content, characterized in that consisting of the dialogue content and the time information in the video.

Images