KR20190004486A - Method for training conversation using dubbing/AR - Google Patents

Abstract

The purpose of the present invention is to provide a means of enabling a learner requiring conversation learning to perform conversation training. According to an embodiment of the present invention, a participation type learning method for a conversation training using dubbing/augmented reality (AR), which enables the learner to learn the conversation training by a conversation training content application installed in a learner terminal, comprises: a conversation training content selecting process of determining the selected content as a conversation training content by allowing the learner to select a content having conversation contents and conversation training learning level to be used in the conversation training learning; a conversation learning training process of allowing the conversation training to be performed through a conversation training screen by providing a conversation training screen of the conversation training content; a conversation actual dubbing training process of allowing the conversation training to be performed through a conversation actual dubbing by providing a conversation actual dubbing screen of the conversation training content when an evaluation result obtained by performing an evaluation process after finishing the conversation learning training falls short of a preset standard; and an AR training process of allowing the conversation training to be performed through AR by providing an AR screen of the conversation training content when an evaluation result obtained by performing an evaluation process after finishing the conversation actual dubbing training falls short of a preset standard.

Description

[0001] The present invention relates to a method for training conversation using dubbing /

The present invention relates to a participatory dialog training and learning method, and more particularly, to a participatory dialog training and learning method that provides conversation training using a dubbing / augmenting reality.

In the treatment and education methods of general developmental disorders, treatment and education through treatment specialists and specialists can provide professional treatment for persons with disabilities, There are disadvantages such as lack of accessibility to use, lack of time for treatment, and waiting time for treatment. In addition, treatment and education through private treatment facilities and welfare facilities have advantages such as low cost of treatment but shortcomings such as lack of expertise of therapists, inferiority of facilities, lack of connection with professional organizations, and waiting time for treatment. In addition, the treatment and education using the treatment S / W program for PC has the advantages of reducing economic burden, improving accessibility and convenience, but also lacking the linkage with professional organizations, lack of specialized learning programs There are disadvantages.

Such conventional developmental disorder treatment and education methods are difficult to provide immediate feedback on treatment results due to lack of accessibility, delay in treatment due to increase in waiting time, lack of connection with experts, and lack of professional evaluation and advice have.

On the other hand, language education for the children of multicultural families, which are growing rapidly in recent years, is necessary, but most of them are only possible through visiting specialist institutions. In addition, although the number of people with disabilities is increasing every year, proportionately there is a lack of welfare facilities and treatment / education facilities.

As a result of the lack of such treatment / education institutions, there is a concern that the quality of the treatment / education service is deteriorated and the treatment / education time is delayed or the treatment / education period is prolonged due to the increase of treatment / education waiting time.

Thus, although language learning contents are provided in recent years, there is a problem that effectiveness in conversation training is poor. In other words, the method of producing the existing language learning contents and providing to the user was provided to the user by making only one recording script with one voice and one pronunciation speed. In addition, since the pronunciation speed is the normal conversation speed of the native language users, even if the computer and the Internet are conveniently used for the language learning, there is a problem that the effect of the conversation training can not be expected only by providing such one type of contents.

In addition, although the final goal of language therapy and education is communication, that is, communication ability, current language therapy and education methods are limited in learning and practicing various conversation skills. Currently, conversational guidance in language therapy is limited to simple or short conversation situations in which learners are often experienced, such as greeting, toys, and asking questions, or conversational topics that can be used to direct situations in a therapy scene. There are some limitations to generalization in teaching and practicing conversation skills such as scripting and play in the way of teaching conversation skills in therapy and education scenes. In the 1: 1 tutorial, it is possible to teach conversational skills using learning materials or methods considering learner characteristics, but there are limitations in providing experience with various conversations or situations. Group map is advantageous in that it is possible to practice more natural conversation than individual map, but there is a limit of dialogue training because the subjects lacking dialogue skills consist of group members and the topic of conversation is limited. It is difficult to provide appropriate learning materials and professional and instructional guidance and feedback immediately for the learner in conversation training at home. There is a limit to providing adequate conversation training materials and methods for treatment and education through professional treatment institutions and experts who can teach conversation training, lack of sufficient time for treatment due to economic burden, lack of accessibility , And waiting time for treatment. In addition, treatment and education using PC therapy S / W program has advantages of reduced economic burden, accessibility, and convenience, but currently there is a lack of therapeutic S / W program for dialogue training and lack of effectiveness. The method of producing the existing language learning contents and providing it to the user was provided to the user by making only one recording script with one voice and one pronunciation speed. In addition, since the pronunciation speed is the normal conversation speed of the native language users, even if the computer and the Internet are conveniently used for the language learning, there is a problem that the effect of the conversation training can not be expected only by providing such one type of contents. In addition, contents included in the developed therapy S / W program are lacking in the topic or conversation contents because the learner who needs conversation training needs enough conversation learning only in daily script such as ordering food, and conversation learning is presented in one direction. Considering that the conversation is two-way, it is necessary for the learner to participate in the conversation situation and to practice and practice the conversation skills practically.

Korean Patent Publication No. 10-2010-0005177

The technical object of the present invention is to provide a method and apparatus for a learner who needs conversational learning, such as a learner with slow language development, social communication disorder and stuttering, language disorder including articulatory disorder, developmental disorder including autistic disorder, People, psychologically depressed, or lack of social skills, etc.) are able to conduct conversational training.

The embodiment of the present invention is a participant-type conversation training learning method using a dubbing / AR in which a conversation training content app installed in a learner terminal performs conversation training learning for a learner, A dialog training content selection process of selecting a content from a learner and determining the selected content as a conversation training content; A conversation learning training process for providing a conversation training screen of conversation training contents to allow conversation training through a conversation training screen; A conversation practice dubbing training process for providing a conversation practice dubbing screen of the conversation training contents so as to conduct conversation training through dubbing conversation dubbing when evaluation after completion of the conversation learning training is less than a predetermined standard; And an augmented reality training process for providing conversation training through the augmented reality by providing an augmented reality screen of the conversation training content when evaluation after completion of the conversation practice derby training is less than a predetermined standard.

In the dialog training content selection process, a conversation training learning level is determined by selecting a number of conversation circles, a conversation topic, and a conversation participant number from a learner.

Wherein the conversation learning training process includes: a conversation moving picture playback process of playing a conversation moving picture of the conversation training content; A learner conversation content input process of stopping the screen at the time of appearance of a character whose dialogue is omitted during the conversation movie reproduction and inputting a conversation voice uttered by the learner as learner conversation contents; The learner's conversation content is determined to be success in the case of the conversation content meeting the predetermined conversation evaluation criterion and the learner conversation content is determined to be a conversation learning content that does not meet the conversation evaluation criteria A conversation learning evaluation process that decides that conversational learning training is failed if the content is content; A conversation execution dubbing screen movement process of performing the conversation practice dubbing training process when it is determined that the conversation learning training is successful; And a conversation learning feedback learning process for allowing the conversation learning feedback learning to be performed for the learner when it is determined that the conversation learning is failed.

The learner conversation contents input process can convert the conversation voice uttered by the learner into characters and display them on the screen.

The dialog learning learning learning process includes a dialog learning result feedback process for notifying that the conversation training has failed; A step of presenting a conversation example and selecting one of the conversation examples from among the conversation examples from the learner determined to have failed the conversation training; A step of receiving a learner voice and determining that the learner's voice is successful if the learner's voice coincides with the dialogue example within an error range; A step of causing the learner to listen to the conversation contents by reproducing the conversation contents when it is determined that the conversation example training is failed; Receiving a learner voice that imitates conversation contents and determining that conversation imitation training is successful if the input learner voice coincides with the conversation contents within an error range; A step of allowing the learner to listen to the speech conversion conversation contents by reproducing the speech conversion conversation contents in which the conversation contents are voice-converted when the conversation imitation training failure is determined; And a learner's voice imitating the conversation conversation contents. If the inputted learner's speech coincides with the conversation conversation contents reproduced within the error range, it is determined that the speech conversion training is successful. And a determining process.

The voice conversion of the conversation contents is characterized in that the voice of the character is converted by voice amplification and speed control.

Wherein the conversation execution dubbing training process comprises: a dialogue moving picture playback process of playing a conversation moving picture of the conversation training content; A learner dubbing voice input process in which a conversation voice uttered from a learner is input as a learner dubbing voice in real time when a character whose communication is omitted during the reproduction of the conversation moving picture is input; The learner's dubbing voice inputted from the learner is analyzed to determine success of the dubbing voice training when the learner's voice is a dubbing voice conforming to a predetermined conversation evaluation standard, A dubbing voice evaluation process for determining a dubbing voice training failure; An augmented reality screen movement process for performing the augmented reality training process when it is determined that the dubbing voice training is successful; And a dubbing voice feedback learning process for causing a learner to perform dubbing voice feedback learning when it is determined that the dubbing voice training is unsuccessful.

Wherein the dubbing voice feedback learning process comprises: a step of feedbacking that dubbing voice training has failed; A process of providing conversation dubbing related learning materials used for conversation learning using conversation dubbing; A dubbing voice is input from a learned learner using the conversation dubbing-related learning material, and when the inputted learner dubbing voice meets the conversation evaluation criteria, it is determined that the first learning dubbing voice training is successful, Dubbing voice training failure; Analyzing the cause of the failure of the first learning dubbing voice training and providing communication dubbing correction learning data when it is determined that the first learning dubbing voice training is failed; And a second learning dubbing voice training success is determined to be successful if the learner voice input from the learned learner using the conversation dubbing correction learning material is input and the inputted learner dubbing voice matches the conversation evaluation criteria, And determining a learning dubbing voice training failure.

The interactive dubbing-related learning material and the conversation dubbing correction learning material are learning materials that are provided differently according to the learner so as to respond to the cause of the failure of the failed learner. The learning material of the dubbing voice- And displays the dialogue speed and the dialogue between the character and the character.

The augmented reality training process includes: augmented reality regeneration process for regenerating a conversation augmented reality (AR) of the conversation training content; A gesture, and a facial expression in real time when a character to be replaced is replaced by a learner during the conversation-training augmented reality reproduction, receives the augmented reality input source in real time instead of the character, A learner augmented reality input process for displaying the learner augmented reality substitution; The input source of the augmented reality inputted from the learner is analyzed to determine success of the augmented reality training when the augmented reality input source meets a predetermined dialogue evaluation criteria, and when the augmented reality input source does not meet the conversation evaluation criteria Augmented reality training assessment process to determine augmented reality training failure; And an augmented reality feedback learning process for causing the augmented reality feedback learning to be performed to the learner if it is determined that the augmented reality training is failed.

The augmented reality feedback learning process includes: a step of informing that augmented reality training has failed; A process of providing learning materials related to augmented reality used in dialogue learning using incremental reality; The augmented reality input source including the voice, the gesture, and the facial expression from the learned learner using the augmented reality-related learning material is input, and when the augmented reality input source of the input learner meets the conversation evaluation criteria, Determine the success of the actual training and, if not, determine the failure of the first learning augmented reality training; Analyzing the cause of failure of the first learning augmented reality training and providing the augmented reality modification learning material when it is determined that the first learning augmented reality training is failed; And an augmented reality input source including a voice, a gesture, and a facial expression from a learned learner using the augmented reality modified learning material, and if the inputted augmented reality input source meets the conversation evaluation criteria, Determining that the training is successful, and, if not, determining the failure of the second learning augmented reality training.

If the learner can not gaze at the conversation partner while the learner participates in the incremental reality and the eye line is concentrated on the surrounding objects, the augmented reality modification learning data may be displayed by emphasizing the eyes of the conversation partner or tracking the learner's gaze direction, The learning data can be displayed on the screen. The dialogue evaluation standard includes a key word evaluation criterion, a dialogue expression evaluation criterion, a dialogue criterion evaluation criterion, a dialogue criterion evaluation criterion, a punctuation criterion, a linguistic criterion, a linguistic strength criterion, Criteria, and conversational accent evaluation criteria. In non-verbal conversation, one or more of the expression evaluation criteria, the gesture evaluation criteria, and the gaze evaluation criteria may be included.

According to the embodiment of the present invention, by providing the conversation training learning using the dubbing / augmenting reality, the learner who needs conversation training can efficiently conduct conversation training.

1 is a perspective view of a learner terminal capable of participatory conversation training learning according to an embodiment of the present invention;
2 is a block diagram of a learner terminal capable of participatory conversation training learning according to an embodiment of the present invention;
FIG. 3 is a flowchart illustrating a participatory dialog training / learning process using dubbing / AR according to an embodiment of the present invention.
4 is a flowchart illustrating a conversation learning training process according to an embodiment of the present invention;
FIG. 5 is a flowchart illustrating a conversation practice dubbing training process according to an embodiment of the present invention. FIG.
FIG. 6 is a flowchart illustrating an added reality training process according to an embodiment of the present invention; FIG.
FIG. 7 is a diagram illustrating dialogue evaluation criteria according to an embodiment of the present invention. FIG.
8 is a view showing learning materials according to an embodiment of the present invention.
Figure 9 is an illustration of a feedback module in accordance with an embodiment of the present invention.
10 illustrates an example of a speech conversion module according to an embodiment of the present invention.
Figure 11 is an illustration of an image conversion module according to an embodiment of the present invention.

BRIEF DESCRIPTION OF THE DRAWINGS The advantages and features of the present invention, and how to achieve them, will be apparent from the following detailed description of embodiments thereof taken in conjunction with the accompanying drawings. The present invention may, however, be embodied in many different forms and should not be construed as being limited to the exemplary embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete and will fully convey the concept of the invention to those skilled in the art. And the present invention is only defined by the scope of the claims. In the following description, well-known functions or constructions are not described in detail since they would obscure the invention in unnecessary detail.

FIG. 1 is a perspective view of a learner terminal capable of participatory conversation training learning according to an embodiment of the present invention, and FIG. 2 is a block diagram of a learner terminal capable of participatory conversation training learning according to an embodiment of the present invention.

The learner terminal 10 capable of participatory conversation training learning of the present invention is provided with an application (APP) for allowing a learner who needs conversation training such as a language disorder and a developmental disorder to train conversation, Conduct communication training for learners who need this.

The learner terminal 10 of the present invention will be described by taking a smart phone as an example in the drawing of Fig. 1, but it is possible to use not only a smart phone but also a desktop PC, a tablet PC, a slate PC A notebook computer, a digital broadcasting terminal, a PDA (Personal Digital Assistants), a PMP (Portable Multimedia Player), a navigation, and the like. Of course, the terminal to which the present invention can be applied is not limited to the above-described types, and it is of course possible to include various terminals.

2, the learner terminal 10 of the present invention includes a terminal communication unit 11, a terminal memory 12, a terminal display unit 13, a terminal input unit 14, a terminal control unit 15).

The terminal communication unit 11 is a module that performs a function of communicating through a mobile communication network. When performing mobile communication such as 3G or 4G, the terminal communication unit 11 includes an RF transmitter (not shown) for up-converting and amplifying the frequency of a radio- An RF receiver (not shown) for low-noise amplifying the received radio signal and down-converting the frequency, and the like.

The terminal memory 12 is a storage body that stores conversation training contents (conversation learning contents, dubbing learning contents, augmented reality learning contents, etc.), dialogue evaluation reference, screen graphic interface (GUI) Such a memory is a module capable of inputting and outputting information such as a flash memory, a CF card (Compact Flash Card), and an SD card (Secure Digital Card) There may be.

The terminal display unit 13 is a module for displaying a conversation training screen, a conversation practice dubbing screen, an augmented reality screen, and the like of the conversation training content selected by the learner.

The terminal input unit 14 is a module that receives input from a learner through a conversation training screen displayed through the terminal display unit 13, a conversation conversation dubbing screen, an augmented reality screen, or the like. The terminal input unit 14 and the terminal display unit 13 may be implemented as a single type of touch screen panel. A touch screen panel is a display window provided on a front surface of a terminal to provide a touch screen screen capable of simultaneously performing input and display, and includes a graphical user interface (GUI) for communicating with a learner Display.

The terminal control unit 15 is implemented by an MCU (Main Control Unit) that controls each function module of the learner terminal 10, and is a module in which the dialog training content app of the present invention is installed. Such a conversation training content app can be configured such that the conversation training content app selects a content having conversation contents and conversation training learning level to be used for conversation training learning from the learner and determines the selected content as conversation training content, And provides a conversation training dubbing screen of the conversation training contents to perform conversation training through dubbing conversation practice, and also provides an augmented reality screen of the conversation training content Let the conversation drill through the augmented reality be done.

For reference, the learner terminal 10 implemented with a smart phone or the like can install, add or delete hundreds of various applications (application programs) as desired by the user, And can implement an appropriate interface through various applications. Therefore, you can download a conversation training content app from Google Market, Apple Store, etc. and install it on smartphone.

FIG. 3 is a flowchart showing a participatory dialog training and learning process using dubbing / AR according to an embodiment of the present invention, FIG. 4 is a flowchart illustrating a dialogue learning and training process according to an embodiment of the present invention, FIG. 6 is a flowchart illustrating an actual reality training process according to an embodiment of the present invention. FIG. 7 is a flowchart illustrating a dialogue evaluation dubbing process according to an embodiment of the present invention. FIG. 8 is a view showing learning data according to an embodiment of the present invention, FIG. 9 is a view illustrating an example of a feedback module according to an embodiment of the present invention, FIG. FIG. 11 is a diagram illustrating an image conversion module according to an exemplary embodiment of the present invention. Referring to FIG.

In a participative dialog training learning method using a dubbing / AR in which a conversation training content application installed in a learner terminal 10 performs conversation training learning for a learner, A content selection process S100, a conversation learning training process S200, a chatting practice dubbing training process S300, and an augmented reality training process S400.

In the conversation training content selection process (S100), conversation contents to be used for conversation training learning are selected. And conversation training learning level of conversation training. The content having the selected dialogue content and the dialog training and learning level is determined as the dialogue training content.

Here, the choice of the level of conversational training is determined by selecting the number of conversational circles, conversation topic, and conversation participant from the learner, thereby determining the conversational training level.

Here, conversation circles are two-way communication between talkers. For example, if two people gave a conversation after watching a movie, "How was beauty and a beast movie" and "It was fun", the conversation circle number is one. The conversation continued, "Which scene did you like best?" "If I was the best when the beast and the bell were dancing," the number of conversational circles would be two.

Also, as the number of participants in the conversation increases, the amount of conversation received increases, and a high level of conversational ability is required. For example, if there are two participants in a conversation, the conversation can be exchanged and the attention can be focused only on the opponent. However, if there are four participants in the conversation, the other three persons are also remembered and responded to the conversation flow, The processing becomes more complicated.

The conversation learning training process (S200) is a process in which a conversation training content app provides a conversation training screen of conversation training contents to conduct conversation training through a conversation training screen. That is, in the dialog learning learning process (S200), a lecture in which a learner expresses a dialogue in which a learner does not have a dialog for only a representative person is presented at the time of lecture reproduction, and a learner learns a linguistic dialogue expression .

4, the conversation learning training process S200 includes a conversation moving picture reproduction process S210, a learner conversation content input process S220, a conversation learning evaluation process S230, a conversation conversation dubbing screen movement process S300 ), And a conversation learning feedback learning process (S250).

The conversation moving picture reproduction process (S210) is a process of reproducing the conversation moving picture of the conversation training content. That is, the dialogue moving picture of the dialogue training content having the conversation learning level and the dialogue contents selected by the learner is reproduced.

The learner dialogue content inputting process (S220) is a process of stopping the screen at the time of appearance of a character whose dialogue is omitted during the conversation moving picture reproduction, and receiving the conversation voice uttered by the learner as the learner conversation contents. Further, the learner conversation contents input process converts the conversation voice uttered by the learner into characters and displays them on the screen. In other words, at the time of reproducing the conversation moving picture, the learner presents a moving picture in which only the person's conversation is omitted, and the learner completes the omitted conversation.

The conversation learning evaluation process (S230) analyzes and evaluates the voice of the learner conversation content input from the learner, and determines that the conversation learning training is successful when the conversation contents of the learner meet the conversation evaluation criteria set in advance, And determining that the conversation learning training fails if the conversation contents are conversation contents that do not meet the conversation evaluation criteria.

Here, in the case of verbal conversation, as shown in Fig. 7 (a), the dialogue evaluation standard is a key word evaluation standard, a dialogue expression evaluation standard, a dialogue reaction rate evaluation standard, a dialogue rule evaluation standard, Criterion of dialogue repetition, criterion of conversation omission, criterion of conversation accentuation.

This is the criterion used for comparative analysis of the dialogue technique between the dialogue model and the learner. As described above, the conversation learning evaluation standard is a criterion for evaluating the learner's voice information.

Keyword evaluation criteria are to identify and evaluate key words in dialogue expression.

The evaluation criteria for dialogue expression evaluates whether appropriate dialogue expressions such as colloquial type, statement type, and question type are used.

The criterion for evaluating the response rate of a conversation is an evaluation of how the learner speaks at the appropriate rate to the other party's speech. If you take a long time to listen to the other person's response, it will interfere with maintaining smooth conversation, such as interruption or prompting.

Conversation rule evaluation criteria evaluate conversation skills such as answering questions, changing language expressions according to conversation partner, and responding to appropriate conversation topics.

The criterion for evaluating the speed of speech is to evaluate the speed of speech when talking, and the speed of speech is an important skill to communicate well to the other. If it is too fast or slow, it is difficult for the other party to grasp the speaker's words, or it is difficult to pay attention to the speaker's words.

The criterion for evaluating the speech strength is to evaluate the learner's speech strength during conversation. Even if the voice is too loud or too small, it becomes an obstacle to the conversation flow.

Conversation evaluation criterion evaluates repeating the same speech or word when the learner speaks. This is the case when language fluency is lacking as in stuttering or when there is a high level of tension in conversation situations.

The evaluation criteria for the dialogue omission evaluates linguistic dialogue techniques such as excluding important key words and blurring parts of words when a learner speaks.

Conversational accent evaluation criteria assess how learners use appropriate nuances according to type of dialogue expression with appropriate rhythm during conversation. For example, if you say, "Yesterday I went camping with my family," you have to express the accent at the end of the horse (eg, gone) Or the transfer of the intention of the horse becomes unclear.

On the other hand, in the case of the non-verbal conversation, as shown in Fig. 7 (b), one or more of the expression evaluation criteria, the gesture evaluation criteria, and the gaze evaluation criteria may be included. Nonverbal conversation techniques include facial expressions and gestures, except verbal expressions. Nonverbal conversation skills play a very important role in communication. For example, the word 'good' is very different when you are smiling and when you are angry. In this way, the interpretation and expressive ability of nonverbal conversation skills such as expression are more important than words if the disagreement is expressed. And the skills to keep eye contact with the opponent when talking is important in maintaining the conversation. For example, in the case of autistic disorder, it is difficult to look at the conversation, and when the social contraction is high, there is a tendency to look around the eyes or to avoid gaze, not to look at the opponent's eyes when talking, . Therefore, in order to communicate, understand and maintain conversation well, verbal conversation skills as well as nonverbal conversation skills are very important.

Therefore, in the present invention, errors are analyzed in the learner's conversation skill by comparing and analyzing the difference between the learner and the dialogue characteristic of the dialogue model.

The facial expression evaluation criterion analyzes the learner 's facial expressions through the eye, mouth, and facial muscle changes of the learner' s input, and evaluates the difference between the facial expressions between the learner and the conversational model.

The gesture evaluation criterion analyzes the gestures of the learner 's hand and foot entered as video material and evaluates the difference between the learner and the conversation model.

The gaze evaluation criterion uses eye-tracking principle to evaluate how much the learner watches the conversation partner during conversation, whether the gaze processing is appropriate during conversation, and the difference between gaze processing between learner and conversation model.

On the other hand, if it is determined that the conversation learning training is successful (S230a), success feedback is provided (S240), the conversation training learning termination or other conversation learning is performed in response to the learner request, (S300) so as to perform a conversation practice dubbing training process (S300).

On the other hand, if it is determined that the conversation learning training is failed (S230b), a conversation learning feedback learning process (S250) is performed so that conversation learning feedback learning is performed for the learner.

The dialog learning feedback learning process (S250) includes a dialog learning result feedback process (S251) for notifying that the conversation training has failed, a dialogue example is presented, and a dialogue example is selected from the conversation examples (S252). If the input learner's voice is identical with the dialogue example within the error range, the dialogue example training is determined to be successful (S253a). If the learner voice is not matched with the dialogue example, the dialogue example training is determined to be failed (S253b) (S253). If it is determined that the conversation example training is failed, a failure feedback indicating that the conversation example is failed is provided (S254), and the learner is allowed to listen to the conversation contents by reproducing the conversation contents (S255) If the learner voice is input (S256) and the inputted learner voice coincides with the conversation contents within the error range, the conversation imitation training success (S257a) (S257b). If it is determined that the conversation imitation training is failed (S257b), the failure feedback is presented (S258), and the conversation contents are subjected to speech conversion (S2591) (S2592) of causing the learner to listen to the speech conversion conversation contents by reproducing the speech conversion conversation contents, inputting the learner speech imitating the conversation conversation contents (S256), and inputting the speech conversion conversation contents The speech conversion training is determined to be successful if it is within the error range, and the speech conversion training failure is determined if the speech conversion training is not matched (S257).

For reference, this dialog learning feedback process (S200) can be processed through the feedback module shown in Fig. Dialogue evaluation feedback includes verbal dialogue feedback and nonverbal dialogue evaluation, and each evaluation result is presented graphically. Compensation feedback can provide learner with scoring or compensation for conversational assessment feedback and steady learning attitudes.

In addition, in the above, voice conversion of the conversation contents can be performed by voice-amplifying and speed-adjusting the voice of the character using the voice conversion module shown in Fig. In detail, the voice conversion module converts the voice of the conversation model so that the learner can accurately hear and follow the voice of the conversation model. Nonlinear compression allows the learner to hear the loudspeakers accurately without losing the loudness, by amplifying the loud sounds greatly and reducing the loud sounds to a small degree. Directionality makes it possible to accurately grasp the direction in which speech is heard. Feedback cancellation allows you to hear the speech well by amplifying the input speech and eliminating other noise. Noise elimination reduces ambient noise and makes speech sounds clear. Speed adjusts the speed of speech so that learners can listen well. Based on the learner 's speech analysis result, the learner speaks the speech by converting the speech model.

Meanwhile, in the conversation practice dubbing training process (S300), the conversation training content app provides the conversation execution dubbing screen of the conversation training content, so that conversation training is performed through dubbing conversation dubbing. The conversation execution dubbing training process S300 includes a conversation moving picture reproduction process S310, a learner dubbing speech input process S320, a dubbing voice evaluation process S330, an augmented reality screen movement process S400, , And a dubbing voice feedback learning process (S350).

The conversation moving picture reproduction process (S310) is a process of reproducing the conversation moving picture of the conversation training content.

The learner dubbing voice inputting process (S320) is a process in which a conversation voice presented without a screen stop at the appearance of a character whose dialogue is omitted during dialogue moving picture reproduction is input as a learner dubbing voice in real time. And watching the conversation video, the learner directly dubs the conversation of the characters whose conversation is omitted. At this time, the learner inputs voice, and the voice inputted on the screen is converted into a letter and presented.

The dubbing voice evaluation process (S330) analyzes the learner's dubbing voice input from the learner, and determines that the dubbing voice training is successful (S330a) when the learner's voice is a dubbing voice conforming to a predetermined conversation evaluation standard, In the case of a dubbing voice that does not meet this dialogue evaluation criterion, it is determined to be a dubbing voice training failure (S330b). As described above, in the case of verbal conversation, the evaluation criteria of the conversation are, in the case of verbal conversation, a keyword evaluation criterion, a conversation expression evaluation criterion, a conversation response speed evaluation criterion, a conversation rule evaluation criterion, a word speed criterion, , Dialogue elimination evaluation criteria, and conversation accent evaluation criteria, and in the case of non-dialogue, one or more of the expression evaluation criteria, the gesture evaluation criteria, and the gaze evaluation criteria may be included.

If it is determined that the dubbing voice training is successful (S330a), an augmented reality screen movement process (S400) is performed to perform an augmented reality training process (S400) if there is a learner request after success feedback presentation (S340). On the other hand, if it is determined that the dubbing voice training is failed (S330b), a dubbing voice feedback learning process (S3500) is performed so that the dubbing voice feedback learning is performed for the learner.

The dubbing voice feedback learning process (S350) includes a process (S351) of notifying that the dubbing voice training has failed, a process (S352) of providing dubbing-related learning materials used for the conversation learning using the conversation dubbing (S353) from the learned learner using the learning data, and if the input learner's voice satisfies the conversation evaluation criteria, it is determined that the first learning dubbing voice training is successful (S354a). If not, If it is determined that the first learning dubbing voice training is failed (S354b), the cause of the failure of the first learning dubbing voice training is analyzed after the failure feedback presentation (S355) (S356). The learning contents are corrected (S356) to provide conversation dubbing correction learning material (S357). The conversation dubbing correction learning material is used to learn the error portion (S358) (Step S359a). If the learner's voice is not matched with the learner's voice (step S359b), the second learning dubbing voice training failure (step S359b) ). In the error partial learning (S358), a process of providing a learning screen for a conversation technique including a conversation start timing, a speech intelligibility, a conversation speed, and an intonation may be provided.

Here, conversational dubbing-related learning materials and conversation dubbing correction training materials are learning materials that are provided differently according to the learner so as to respond to the cause of failure of the failed learner, in which the face of the character of the dubbing voice section and the video in which the voice is amplified And displays the dialogue speed and dialogue with the character.

11, the learning data is composed of a conversation moving image, an audio model, and a graphic modified by the learner-customized type, and there is a difference in contents structure according to the result of the conversation evaluation analysis of the learner. For example, a video that amplifies the face and voice of a person can be presented along with a graphic that displays the metabolism and speed to promote speech speed.

The dialogue video is a dialogue video presented by modifying the analysis results of the learner 's dialogue evaluation in the dialogue video presented above. For example, only the conversation voice of the person is amplified, or an animation graphic informing the person's conversation timing is added. The graphic is to add animated graphics to the voice spectrum and velocity profile and use it as learning aids. The error part learning is to perform the partial learning only for the error part of the dialogue technique shown in the learner's dubbing. For example, a learner who misses the start timing for participating in a conversation repeatedly, and a learner with a slow learner, presents a graphic learning device (e.g., numerical timer, arrow) prompting the start of conversation in the error partial learning, The degree of urgency is automatically adjusted according to the learner's ability to correct the error characteristics of the learner. You can practice to improve the speech speed systematically through the graphics that show the speed profile for slow speech parts. Depending on the learner's error characteristics, you can practice one by one for each part of the error, or you can practice combining two errors (eg, dialog start timing error and speech speed slow error) at the same time. This is also done systematically according to the learner 's response.

An example of the generation of the learning data is illustrated in FIG. 8 together with the learning data. The learning data includes a conversation video and learning materials, a speech model, and a graphic. The conversation video reflects the learner 's dialogue evaluation analysis result, and the face, gesture, gaze, and voice of the person are emphasized or added with animated graphic.

The voice model reflects the linguistic dialogue technique of the dialogue model and allows learners to imitate linguistic dialogue techniques including dialogue content, voice, and dialog timing.

The graphics consist of a line of sight profile, a dialogue voice spectrum, and a speed profile. It is a device that visually presents information data about conversation skills during conversation model and learner 's conversation participation, and helps learners to learn conversation skills more easily by using animation technique. The eye-gaze profile tracks the learner's eye gaze and graphically shows where the eye gazed during the conversation.

Speech spectrum can enhance the learning effect by presenting the learner 's speech voice characteristics as visual images in frequency, Hz, dB, etc., and by announcing the pitch or displaying the dialog start timing with an animation function attached.

The speed profile can enhance the learning effect by presenting the conversation speed graphically, by attaching a speech speed promotion sign to the dialogue, or by attaching an animation function to the graphic which can recognize the conversation speed.

Meanwhile, the augmented reality training process (S400) is a process for providing augmented reality screen of the conversation training content to allow conversation training through the augmented reality.

6, the augmented reality training process S400 includes augmented reality regeneration process S410, a learner augmenting reality input process S420, an augmented reality training evaluation process S430, and an augmented reality feedback learning process S450).

The augmented reality regeneration process (S410) is a process of reproducing a conversation augmented reality (AR) of the conversation training content.

The learner augmented reality inputting process S420 is a process of inputting an augmented reality input source including a learner's voice, a gesture, and a facial expression in real time at the time of appearance of a character to be replaced as a learner during the conversation augmented augmented reality reproduction, And the learner's voice, gesture, and facial expression. The participant sees the AR conversation video and participates in the video as a character whose dialogue is omitted. At this time, the learner's voice input, facial expression, and gesture information are input and processed together.

The augmented reality training evaluation process (S430) analyzes the augmented reality input source input from the learner, determines the augmented reality training success (S430a) if the augmented reality input source meets a preset dialogue evaluation criterion, If the source does not meet the conversation evaluation criteria, it is determined that the augmented reality training failed (S430b) (S430).

The augmented reality feedback learning process (S450) is a process for causing the augmented reality feedback learning to be performed for the learner when it is determined that the augmented reality training fails (S430b). In case of failure in dialogue evaluation, feedback is presented, learning material reflecting the evaluation result of the learner is presented, and the learner learns the conversation skill and engages in the video.

As shown in FIG. 6, the augmented reality feedback learning process (S350) includes a process (S451) of notifying that the augmented reality training has failed (S451), a process of providing the augmented reality related learning aids S452), inputting the augmented reality input source including voice, gesture, and expression from the learned learner using the augmented reality related learning material (S453), and if the input augmented reality input source of the input learner meets the conversation evaluation criteria If it is determined that the first learning augmented reality training is not successful (S454a), if it is determined that the first learning augmented reality training is failed (S454b), then the first learning augmented reality training failure (S454b) (S456) by analyzing the cause of the failure of the first learning augmented reality training, and then the augmented reality modified learning material is provided (S457), and the augmented reality modified learning material is used When an input augmented reality input source including voice, gesture, and facial expression is input from a learner who has learned an error part (S457), and the input augmented reality input source meets the dialogue evaluation criterion, a secondary aural augmented reality training success (S459a ), And if it does not match, it is determined to be the secondary learning augmented reality training failure (S459b) (S459). The error partial learning (S458) may provide a process of providing a learning screen for a conversation description including a conversation start timing, a speech intelligibility, a conversation speed, an intonation, a line of sight, and a gesture.

If the learner can not gaze at the conversation partner while gazing at the neighboring objects while the participant is participating in the incremental reality, the augmented reality modified learning data may be displayed by emphasizing the eyes of the conversation partner or tracking the gaze direction of the learner, May be the learning material displayed on the screen.

In other words, the learner's voice, facial expression, and gesture information data are analyzed to evaluate the conversation and provide feedback. If successful, submit success feedback and proceed to a conversation video. In case of failure, feedback is presented, and the learning data is corrected by reflecting the results of the learner 's analysis. Present modified learning materials and proceed with partial learning about learner error characteristics. For example, in the case of a learner who repeatedly misses the timing of gazing at the conversation while gazing at the surroundings during the conversation, the user can present the eye-tracking result during the conversation during the error part learning and practice the gaze processing. At this time, eye-tracking can be presented to the dialogue image while presenting the dialogue image, and the eye-tracking facilitation degree is automatically controlled according to the learner's performance rate. In addition, if the timing of continuous participation in conversation is low, a graphical learning device (eg, a numerical timer, arrow) that prompts the timing of the conversation is presented and the degree of prompting is automatically adjusted according to the learning performance rate of the learner, You can practice modifying characteristics. Depending on the error characteristics of the learner, you can practice by combining two errors at the same time by practicing one by one for the error part. This is also done systematically according to the learner 's response.

Video participation refers to the learner participating in the character as the character, and the learner's voice, facial expression, gaze, and gesture are included in the video. The learning materials consist of conversation videos, dialogue models, and graphics modified by the learner 's customized contents.

The dialogue video is a dialogue video presented by modifying the analysis results of the learner 's dialogue evaluation in the dialogue video presented above. For example, if a learner is unable to gaze at a conversation partner while watching a video and focuses on the surrounding objects, the conversation partner can highlight the conversation partner's eyes or display the eye movement with eye-tracking . The graphics are used as learning aids by adding voice spectrum, speed profile, and eye profile to animated graphics.

The error part learning is to perform partial learning only for the error part of the conversation skill shown in the participant's video participation. For example, while participating in a conversation, keep hands and feet awkward and awkward, and if conversation is small, present conversational model learning materials with emphasis on gestures so that learners can learn by imitation. Speaking can be done by presenting conversation model conversation examples and speech spectrum graphics to speak with appropriate voice tones. You can also perform gestures and voice learning separately, then combine the gestures with the voice-to-speech data to practice speaking with the appropriate-sized voice tones while maintaining proper dialog gestures.

8, the learning material includes a conversation video, a learning material, a dialogue model, and a graphic (340). The conversation video reflects the learner 's dialogue evaluation analysis result, and the face, gesture, gaze, and voice of the person are emphasized or added with animated graphic.

The dialogue model consists of a voice model, an expression model, and a gesture model. The voice model reflects the linguistic dialogue technique of the dialogue model and allows learners to imitate linguistic dialogue techniques including dialogue content, voice, and dialog timing. The facial expression model presents the eye gaze and facial expressions of the dialogue model, allowing the learner to imitate nonverbal conversation techniques related to facial expression. The gesture model allows learners to imitate nonverbal conversation skills related to behavior by explaining the learner's inappropriate gestures or presenting behaviors such as the hands, feet, and postures of the conversation model

The embodiments of the present invention described above are selected and presented in order to assist those of ordinary skill in the art from among various possible examples. The technical idea of the present invention is not necessarily limited to or limited to these embodiments Various changes, modifications, and other equivalent embodiments are possible without departing from the spirit of the present invention.

10: Learner terminal
S100: conversation content selection process
S200: Conversation Learning Course
S300: Dialogue dubbing training course
S400: Augmented Reality Training Course

Claims (13)

A participant-type conversation training learning method using a dubbing / AR in which a conversation training content app installed in a learner terminal performs conversation training learning for a learner,
A conversation training content selecting step of selecting a content having a conversation content and a conversation training learning level to be used for conversation training learning from a learner and determining the selected content as conversation training content;
A conversation learning training process for providing a conversation training screen of conversation training contents to allow conversation training through a conversation training screen;
A conversation practice dubbing training process for providing a conversation practice dubbing screen of the conversation training contents so as to conduct conversation training through dubbing conversation dubbing when evaluation after the completion of the conversation learning training is not met, And
An augmented reality training process for providing conversation training through the augmented reality by providing an augmented reality screen of the conversation training content when evaluation is made after completion of the conversation practice derby training and is below a predetermined standard;
A participatory dialog training training method using dubbing / AR.
The method according to claim 1,
Wherein the conversation training learning level is determined by selecting the number of conversation circles, conversation topic, and conversation participant from the learners.
The method according to claim 1,
A dialogue moving picture reproducing step of playing a conversation moving picture of the conversation training contents;
A learner conversation content input process of stopping the screen at the time of appearance of a character whose dialogue is omitted during the conversation movie reproduction and inputting a conversation voice uttered by the learner as learner conversation contents;
The learner's conversation content is determined to be success in the case of the conversation content meeting the predetermined conversation evaluation criterion and the learner conversation content is determined to be a conversation learning content that does not meet the conversation evaluation criteria A conversation learning evaluation process that decides that conversational learning training is failed if the content is content;
A conversation execution dubbing screen movement process of performing the conversation practice dubbing training process when it is determined that the conversation learning training is successful; And
A conversation learning feedback learning process for allowing conversation learning feedback learning to be performed for the learner when it is determined that the conversation learning is failed;
A participatory dialog training training method using dubbing / AR.
4. The method of claim 3,
And converting the conversation utterances uttered by the learner into characters and displaying them on the screen.
4. The method of claim 3,
A dialogue that informs that the conversation training failed;
A step of presenting a conversation example and selecting one of the conversation examples from among the conversation examples from the learner determined to have failed the conversation training;
A step of receiving a learner voice and determining that the learner's voice is successful if the learner's voice coincides with the dialogue example within an error range;
A step of causing the learner to listen to the conversation contents by reproducing the conversation contents when it is determined that the conversation example training is failed;
Receiving a learner voice that imitates conversation contents and determining that conversation imitation training is successful if the input learner voice coincides with the conversation contents within an error range;
A step of allowing the learner to listen to the speech conversion conversation contents by reproducing the speech conversion conversation contents in which the conversation contents are voice-converted when the conversation imitation training failure is determined; And
The learner's voice imitating the voice conversion conversation contents is input. If the input learner voice coincides with the voice conversion conversation contents to be reproduced within the error range, it is determined that the voice conversion training is successful. Process;
A participatory dialog training training method using dubbing / AR.
The method according to claim 5,
And the voice of the character is amplified and speed-adjusted to convert the voice of the character.
The method according to claim 1,
A dialogue moving picture reproducing step of playing a conversation moving picture of the conversation training contents;
A learner dubbing voice input process in which a conversation voice uttered from a learner is input as a learner dubbing voice in real time when a character whose communication is omitted during the reproduction of the conversation moving picture is input;
The learner's dubbing voice inputted from the learner is analyzed to determine success of the dubbing voice training when the learner's voice is a dubbing voice conforming to a predetermined conversation evaluation standard, A dubbing voice evaluation process for determining a dubbing voice training failure;
An augmented reality screen movement process for performing the augmented reality training process when it is determined that the dubbing voice training is successful; And
A dubbing voice feedback learning process for making a dubbing voice feedback learning to the learner when it is determined that the dubbing voice training is failed;
A participatory dialog training training method using dubbing / AR.
8. The method of claim 7,
A process of feedbacking that the dubbing voice training has failed;
A process of providing conversation dubbing related learning materials used for conversation learning using conversation dubbing;
A dubbing voice is input from a learned learner using the conversation dubbing-related learning material, and when the inputted learner dubbing voice meets the conversation evaluation criteria, it is determined that the first learning dubbing voice training is successful, Dubbing voice training failure;
Analyzing the cause of the failure of the first learning dubbing voice training and providing communication dubbing correction learning data when it is determined that the first learning dubbing voice training is failed; And
And a second learning dubbing voice training success is determined to be successful if the learner voice inputted is learned from the learned learner using the conversation dubbing correction learning material and the inputted learner dubbing voice meets the conversation evaluation criteria, Dubbing voice training failure;
A participatory dialog training training method using dubbing / AR.
9. The method of claim 8, wherein the interactive dubbing-
Dubbing voice training Provided differently for each learner to respond to the cause of failure of the failed learner, it provides the video of the face and the voice of the character of the Dubbing voice section amplified, and shows the dialogue speed and the metabolism of the character A participatory dialog training training method using dubbing / AR.
The method of claim 1, wherein the augmented reality training process comprises:
An augmented reality regeneration process for regenerating a conversation training augmented reality (AR) of the conversation training content;
A gesture, and a facial expression in real time when a character to be replaced by a learner appears during the conversation-training augmented reality reproduction, receives a learner's voice, a gesture, and a facial expression A learner augmented reality input process for displaying the learner augmented reality substitution;
The input source of the augmented reality inputted from the learner is analyzed to determine success of the augmented reality training when the augmented reality input source meets a predetermined dialogue evaluation criteria, and when the augmented reality input source does not meet the conversation evaluation criteria Augmented reality training assessment process to determine augmented reality training failure; And
An augmented reality feedback learning process for causing a learner to perform augmented reality feedback learning when it is determined that the augmented reality training is failed;
A participatory dialog training training method using dubbing / AR.
11. The method of claim 10, wherein the augmented reality feedback learning process comprises:
A process of feedbacking that augmented reality training has failed;
A process of providing learning materials related to augmented reality used in dialogue learning using incremental reality;
The augmented reality input source including the voice, the gesture, and the facial expression from the learned learner using the augmented reality-related learning material is input, and when the augmented reality input source of the input learner meets the conversation evaluation criteria, Determine the success of the actual training and, if not, determine the failure of the first learning augmented reality training;
Analyzing the cause of failure of the first learning augmented reality training and providing the augmented reality modification learning material when it is determined that the first learning augmented reality training is failed; And
The augmented reality input source including the voice, the gesture, and the facial expression from the learned learner using the augmented reality modified learning material, and if the inputted augmented reality input source meets the conversation evaluation criteria, Determine success and, if not, determine the failure of secondary augmented reality training;
A participatory dialog training training method using dubbing / AR.
[Claim 12] The method of claim 11,
If the learner is unable to gaze at the conversation partner while participating in the present reality, if the attention is focused on the surrounding objects, it is the learning material displayed by emphasizing the eyes of the conversation partner or tracking the direction of the learner ' Participatory dialogue training method using dubbing / AR.
The method according to any one of claims 1, 7, and 10,
In the case of linguistic conversation, evaluation criteria of key words, evaluation criteria of dialogue expression, evaluation criteria of dialogue reaction rate, evaluation criteria of dialogue rule, evaluation criteria of speech speed, evaluation criteria of speech strength, evaluation criteria of dialogue repetition, , ≪ / RTI >
In non-verbal conversation, participative dialogue training training method using dubbing / AR including one or more expression evaluation criteria, gesture evaluation criteria, and gaze evaluation criteria.

Images