KR20190056749A - System and method for communication training program over virtual reality and continued feedback via mobile device - Google Patents

Abstract

According to one embodiment of the present invention, a method for providing feedback may comprise the steps of: constructing a virtual reality environment for performing communication training of a speaker; obtaining speech training information including eye gaze information, behavior information, and dialog information of the speaker spoken from the speaker in the virtual reality environment; generating an analysis result related to the speaker by analyzing the obtained speech training information; and providing feedback based on the analysis result in response to speech information generated in a real situation from the speaker.

Description

TECHNICAL FIELD [0001] The present invention relates to a virtual reality-based communication training and a continuous feedback system and method using a mobile device.

The following description provides a training process for improving the user's communication ability in a virtual reality using a VR (Virtual Reality) device based on a head mounted display (HMD) The present invention relates to a system and a method for providing continuous feedback in daily life by using the system. More specifically, we analyze users' utterances and behaviors in a virtual reality that implements various situations that can be encountered in everyday life, and provide appropriate responses and feedback of the avatar. In addition, To a system and method for providing feedback in real time via a mobile device.

The virtual reality education and training enables various environments that are difficult to realize in reality to be built in virtual space, enabling education and training in various fields at low cost. Because of this efficiency, education and training using virtual reality has attracted attention in various fields. In recent years, as the importance of communication skills has been emphasized in job interviews and negotiations, various curriculums have been emerging to improve individual communication skills. In this environment, virtual reality education and training provide opportunities to effectively improve personal communication ability in real life by providing various situations similar to actual environment, away from education in theoretical and constrained situations.

There have been many methods to support communication training using digital devices so as to be able to describe various situations, but these methods were generally conducted through training with avatars. According to the registration number 10-1743230 (May 27, 2017), most of these methods are performed mainly based on voice recognition technology in order to recognize and evaluate the user's communication ability. Since communication is a process of conveying the whole meaning and intention through a combination of not only voice utterance but also a combination of a speaker's gaze and a body language, speech analysis itself is insufficient to fully understand the communication process. Therefore, in order to understand the communication process, not only natural language processing technology for understanding the meaning of speech utterance but also computer vision technology for understanding user's behavior is required.

In addition, referring to Registration No. 10-1743230 (May 27, 2017) and Published Patent No. 10-2014-0088327 (Apr. 10, 2014), training in the conventional virtual reality is performed only in a predetermined order on the scenario script Because it is static training to exchange conversation, there is a limit to apply in actual conversation situations where conversation order is not constant and various variables can occur.

In addition, there is a problem that it is difficult to track whether the communication training ability improvement of the user is improved in the communication education training method proposed in the Registration No. 10-1277313 (June 23, 2013). Since existing technologies are only intended for repetitive practice of how individuals communicate about a particular situation, they do not provide feedback on persistent communication problems to users in real conversations.

The present invention has been devised to solve the above problems, and it is an object of the present invention to not only understand a user's utterance through conversation with an avatar in a virtual environment, but also to grasp the user's gaze and upper- And to provide a communication environment similar to the actual environment by generating a dynamic response of the avatar corresponding thereto.

It is another object of the present invention to provide a method of providing feedback by analyzing a user's conversation in a real situation through a mobile device in order to continuously improve a user's communication ability.

The feedback providing method performed in the feedback providing system includes: configuring a virtual reality environment for performing communication training of a speaker; Obtaining speech training information including visual information, behavior information, and dialog information of a speaking person uttered from the speaking person in the virtual reality environment; Generating an analysis result related to the speaker by analyzing the obtained speech training information; And providing feedback based on the analysis result corresponding to the speech information generated in a real situation from the speaking person.

Wherein the step of generating the analysis result related to the speaker by analyzing the obtained speech training information comprises the steps of: analyzing the dialogue information including the speech data of the speaker or the speech data, Determining a correct answer to the answer of the speaker by measuring the degree of similarity, and selecting a next question based on whether or not the answer of the determined speaker is correct.

Wherein the step of generating the analysis result related to the speaking person by analyzing the obtained speech training information includes the steps of calculating nine direction vectors based on the three-dimensional coordinate values of the at least one body part of the speaking person, And recognizing the behavior information of the speaker based on the direction vector.

Wherein the step of generating the analysis result related to the speaking person by analyzing the obtained speech training information includes the steps of calculating the correct answer rate for the question with the communication and training device, And providing an analysis result including the behavior information that is performed a predetermined number of times or more and the meaning of the behavior information.

The step of providing feedback based on the analysis result may include the step of providing the mobile device with summary information about the conversation contents of the speaking person, whether the conversation flow of the speaking person is changed, or behavior information predicted from the speaking person as feedback information can do.

Wherein the step of configuring the virtual reality environment for performing the communication training of the speaker includes generating the avatar performing the conversation with the speaking person through the communication training apparatus that provides the virtual reality environment, And the step of communicating with the avatar is performed.

Wherein the step of configuring the virtual reality environment for performing the communication training of the speaker includes providing a scenario including at least one type for performing the communication training and generating a scenario script based on the scenario selected from the speaker Wherein the scenario script can dynamically determine the order of the questions in correspondence with the utterance training information of the speaker.

A computer program stored on a computer-readable recording medium for executing a feedback method performed in a feedback providing system comprises the steps of: constructing a virtual reality environment for performing communication training of a speaker; Obtaining speech training information including visual information, behavior information, and dialog information of a speaking person uttered from the speaking person in the virtual reality environment; Generating an analysis result related to the speaker by analyzing the obtained speech training information; And providing feedback based on the analysis result corresponding to the speech information generated in a real situation from the speaking person.

The feedback providing system comprises: a configuring unit for configuring a virtual reality environment for performing communication training of a speaker; An obtaining unit obtaining speech training information including visual information, behavior information, and dialog information of a speaking person uttered from the speaking person in the virtual reality environment; A generating unit for generating an analysis result related to the talker by analyzing the obtained speech training information; And a providing unit for providing feedback based on the analysis result corresponding to the speech information generated in the actual situation from the speaking person.

Wherein the generation unit determines whether a correct answer to the speaker's answer is obtained by measuring a good answer to the scenario script and the similarity to the answer of the speaker with respect to the conversation information including the text data or the voice data of the speaker, Based on the correct answer to the speaker's answer, the next question can be selected.

The generating unit may calculate nine direction vectors based on the three-dimensional coordinate values of at least one body part of the speaker, and may grasp the behavior information of the speaking person based on the calculated direction vector.

Wherein the generation unit comprises an analysis unit that includes a communication correcting unit that corrects a correct answer rate for a question, an incorrect answer to the question, a contact match grade of the talker, or behavior information that is performed more than a predetermined number of times from the talker, Results can be provided.

The providing unit may provide the mobile device with summary information about the conversation contents of the speaking person, whether the conversation flow of the speaking person is changed, or behavior information predicted from the speaking person, as feedback information.

The generating unit may generate an avatar to perform a conversation with the speaking person, and communicate with the generated avatar through the communication training apparatus that provides the virtual reality environment.

Wherein the configuration unit provides a scenario including at least one kind for performing the communication training, and includes setting a scenario script based on a scenario selected from the speaker, wherein the scenario script includes at least one of speech The order of the questions can be dynamically determined in response to the training information.

The present invention analyzes a user's utterance and dynamically questions a user's utterance not in a predetermined order in a scenario script but can help applications in a real conversation situation where a conversation order is not constant and various variables can occur have.

The present invention can generate feedback on user utterances and behaviors to help improve not only user utterances but also behavioral factors in communication.

The present invention can provide users with a virtual environment that describes various situations that are similar to real life, and can help improve coping ability in various situations in real life.

The present invention can help improve the user's communication ability continuously by analyzing the user's utterance in daily life.

1 is a diagram for explaining an outline operation of a feedback providing system according to an embodiment.
2 is a block diagram illustrating a configuration of a feedback providing system according to an embodiment.
3 is a flowchart illustrating a method of providing feedback of a feedback providing system according to an embodiment.
4 is a flow chart illustrating a method of analyzing a component of a dialogue in a feedback providing system according to an embodiment.
5 is a flowchart illustrating a virtual reality-based communication training method according to an exemplary embodiment of the present invention.
6 is a flow chart illustrating a method of providing feedback using a mobile device in accordance with one embodiment.
FIG. 7 is a diagram for explaining the determination of gaze information of a speaking person according to an embodiment.
8 is a diagram for explaining a process of performing communication training through a communication training apparatus according to an embodiment.
FIG. 9 is a diagram for explaining a process of selecting a scenario for performing communication training according to an embodiment.
FIG. 10 is a diagram for explaining a process of performing communication training based on a scenario in a virtual reality environment according to an embodiment.
11 is an example for illustrating that feedback information is provided to a mobile device according to an embodiment.
12 is an example for explaining that the analysis result is provided to the communication training apparatus according to the embodiment.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings.

1 is a diagram for explaining an outline operation of a feedback providing system according to an embodiment.

The feedback providing system 100 can perform communication training and can provide continuous feedback based on the speaker's utterance information that occurs in an actual situation. For example, the feedback providing system 100 may perform communication training through the communication training device 300 and may perform continuous feedback through a separate electronic device 200.

The feedback providing system 100 may configure an environment for performing communication training. The virtual reality based communication training can be performed through the communication training device 300 based on the environment for performing the communication training configured in the feedback providing system 100. [ For example, a virtual reality-based communication training environment can be provided through the display as the speaker wears the communication training device 300. [ The communication training device 300 is a device for providing a virtual reality environment and means a wearable device such as a head mounted display (HMD) in which a user wears a head like a pair of glasses and outputs an image through a display . Further, instead of mounting the display, an HMD that uses a portable electronic device as a display can be applied. Through this HMD, it is possible to provide an environment of virtual reality and augmented reality. When the user wears the HMD, the space where the user exists through the HMD can be displayed as augmented reality or virtual reality. More specifically, the feedback providing system 100 may generate scenarios for performing communication training in a virtual reality environment. The feedback providing system can generate an avatar for performing communication training in a virtual reality environment in a virtual reality environment, and can perform communication training with a speaker through the generated avatar.

Referring to FIG. 9, at least one scenario 910 may be displayed in the communication training apparatus 300 through a virtual reality environment. For example, the scenario may be displayed in list form or card form. A speaking person wearing the communication training apparatus 300 can select a scenario to perform communication training. At this time, the speaking person can indirectly select the scenario displayed in the communication training apparatus 300 through the voice data, or directly select the scenario displayed on the display.

Referring to FIG. 8, a virtual reality environment 820 for performing communication training is displayed as the speaker 810 wears the communication training apparatus 300, for example, the HMD. Communication training through the virtual reality environment 820 may be performed based on the scenario selected from the speaker 810 in the communication training apparatus 300. [ For example, an avatar suitable for the scenario may appear in the virtual reality environment displayed in the communication training device 300. [ The talker 810 can communicate with the avatar existing in the virtual reality environment 820 through the communication training device 300. [ The communication training apparatus 300 may be provided with utterance training information including visual information of the utterance uttered by the utterer 810, behavior information, and dialog information.

10, a selected scenario may be executed in the communication training apparatus 300. [ For example, the presentation scenario script may be set in the communication training apparatus 300 as the presentation scenario among the types of scenarios is selected from the speaker. At least one avatar interviewer corresponding to the presentation scenario may be displayed through the display 1010 of the communication training device 300. [ The number of avatars may be displayed differently depending on the type of the scenario selected from the speaker. The speaking person can perform communication training based on the presentation scenario executed through the display 1010 of the communication training apparatus 300. [ At this time, the gaze information of the speaker, the behavior information of the speaker, and the conversation information can be recognized through the avatar.

The feedback providing system 100 can acquire the utterance training information collected from the communication training apparatus 300. [ The feedback providing system 100 may generate analysis results related to the speaker by analyzing the speech training information. The feedback providing system can store the speech training information and the analysis result, which are uttered through the communication training from the speaker, in the database. Referring to FIG. 12, this is an example for explaining providing the analysis result 1210 generated in the feedback providing system 100. For example, the analysis result 1210 related to the speech training information of the speaker can be displayed in the communication device 300 in the virtual reality environment. The feedback providing system 100 may provide feedback based on the analysis results of the speech training information performed in the communication device 300.

Meanwhile, the feedback providing system continuously analyzes the speech information generated in the actual situation from the speaker and can provide feedback in real time through the mobile device. The feedback providing system 100 may perform continuous feedback through a separate electronic device 200. [ In this case, the separate electronic device may be a fixed type terminal implemented by a computer device or a mobile type terminal. Portable electronic devices such as a smart phone, a mobile phone, a navigation system, a computer, a notebook, a terminal for digital broadcasting, a personal digital assistant (PDA), a portable multimedia player (PMP), a tablet PC, Head mounted display). In the embodiment, a separate electronic device 200 will be described as a mobile device. The feedback providing system 100 can provide feedback by comparing the speech training information of the speaker with the speech information generated in the actual situation from the speaker. The feedback providing system can provide the feedback based on the analysis result generated by analyzing the utterance training information on the utterance information generated in the actual situation from the speaker. For example, analysis results and feedback can be displayed through the display of the mobile device 200. Accordingly, the visual information and the behavior information of the speaker can be estimated with only the talk information of the speaker.

Further, communication training may be performed through the mobile device 200 as well as the communication training device 300. At this time, communication training may be performed in the mobile device 200 in a normal environment other than a virtual reality environment or a virtual reality environment.

FIG. 2 is a block diagram for explaining a configuration of a feedback providing system according to an embodiment, and FIG. 3 is a flowchart for explaining a feedback providing method of a feedback providing system according to an embodiment.

The processor 200 of the feedback providing system 100 may include a configuration unit 210, an acquiring unit 220, a generating unit 230, and a providing unit 240. The components of such a processor 200 may be representations of different functions performed by the processor 200 in accordance with control commands provided by the program code stored in the feedback providing system 100. [ The processor 200 and the components of the processor 200 may control the feedback providing system 100 to perform the steps S310 through S340 included in the feedback providing method of FIG. At this time, the components of the processor 200 and the processor 200 may be implemented to execute an instruction according to code of an operating system and code of at least one program included in the memory.

The processor 200 may load the program code stored in the file of the program for the feedback providing method into the memory. For example, when a program is executed in the feedback providing system 100, the processor can control the server to load the program code from the file of the program into the memory under the control of the operating system. At this time, each of the configuration unit 210, the acquisition unit 220, the generation unit 230, and the provider 240 included in the processor 200 and the processor 200, And may be different functional representations of the processor 200 for executing subsequent steps S310 through S340.

In step S310, the configuration unit 210 may configure a virtual reality environment for performing communication training of the speaker. The constructing unit 210 may generate an avatar for performing a conversation of a speaking person, and communicate with a speaking person through an avatar in a communication training apparatus that provides a virtual reality environment. The configuration unit 210 provides a scenario including at least one kind for performing communication training, and a scenario script may be set based on a scenario selected from a speaker.

In step S320, the obtaining unit 220 may obtain the utterance information including the gaze information, the behavior information, and the conversation information of the utterance uttered by the utterer in the virtual reality environment.

In step S330, the generation unit 230 may generate an analysis result related to the speaker by analyzing the obtained speech training information. The generation unit 230 measures the similarity between the model answer to the scenario script and the answer of the speaker to the conversation information including the text data or the voice data of the speaker to determine whether or not the answer of the speaker is correct, The following questions can be selected based on whether or not the answer is correct. The generating unit 230 may calculate nine directional vectors based on the three-dimensional coordinate values of at least one or more body parts of the utterance, and may grasp the behavior information of the utterance based on the calculated directional vectors. The generating unit 230 provides the communication training apparatus with an analysis result indicating the percentage of correct answer to the question, the incorrect answer to the question, the contact level of the talker's gaze, or the action information and action information performed more than a predetermined number of times from the speaker .

In step S340, the providing unit 240 may provide feedback based on the analysis result corresponding to the utterance information generated in a real situation from the utterance. The providing unit 240 may provide the mobile device with the summary information about the conversation contents of the speaking person, whether the conversation flow of the speaking person is changed, or the behavior information predicted from the speaking person as feedback information.

Referring to FIG. 4, a flow chart for explaining a method of analyzing a component of a dialogue in a feedback providing system.

The feedback providing system analyzes the speech training information transmitted from the virtual reality-based communication training device 300 and the mobile device 200, and generates responses and feedbacks of the corresponding avatars and transmits them to each device.

In step S410, the feedback providing system may analyze the input signal. The feedback providing system can determine whether the input apparatus is the communication training apparatus 200 or the mobile apparatus 300. [ At this time, the apparatus to which the input signal is inputted can be determined using the difference between the input targets of different apparatuses. For example, if there is only an input as an input, it can be regarded as input from the mobile device 300.

In step S420, the feedback providing system may analyze the utterance and behavior of the speaker. The feedback providing system can generate the analysis result related to the speaker based on the utterance information including the gaze information, the behavior information, and the conversation information of the speaker received from each device. For example, when the analysis is requested from the communication training device 200, the content of the speech in the form of a text, the gaze information of the speaker (for example, whether or not the avatar is gazing) Receives the coordinate value as an input value, and determines whether or not the answer of the speaker's answer to the question of the avatar based on the received input value, the degree of contact of the avatar to be next to the next, the degree of contact with the avatar of the speaker, Content, and so on. Table 1 below shows the results of the analysis provided to the communication training device.

More specifically, the feedback providing system selects a question of the most suitable avatar according to the answer of the speaker, rather than the order of the questions set on the scenario script, so that the actual conversation sequence in which the conversation order is not constant, You can provide help to cope with the situation. At this time, the scenario script can be composed of a tree structure with the following questions related to the answer to the question.

In one example, the feedback providing system may provide at least one or more scenarios, and the speaker may select at least one of various scenarios (e.g., contract, discussion, interview, announcement, etc.). At this time, the script of each scenario may be composed of sequential questions. However, the avatar can select the dynamically appropriate question based on the speaker's answer, rather than asking the sequential questions on the scripted questions configured in the scenario, to fire the next question. One scenario script can be expressed by Equation (1) below.

Equation 1:

은 시나리오 스크립트에 구성된 m번째 질문을 의미한다. 발화자는 순서에 따라 질문을 제공받고, 제공받은 질문에 대하여 답변을 수행할 수 있다. 이때, 발화자의 답변은 질문마다 구성된 모범 답안들과 유사도를 측정하여 정답 여부를 판단할 수 있다. 모범 답안이란 질의에 대하여 정답 여부를 판단하기 위하여 설정된 답안을 의미할 수 있다. 발화자로부터 발화된 답변에 대한 정답 여부가 수학식 2에 의하여 도출될 수 있다.Here, s is a scenario script,

Means the mth question constructed in the scenario script. Speakers are provided with questions in order and can answer questions provided. At this time, the speaker 's answer can determine the correct answer by measuring the degree of similarity with the model answers constructed for each question. A model answer can mean an answer set to determine whether the question is the right answer. An answer to the answer uttered by the speaker can be derived by Equation (2).

Equation 2:

은 m번째 질문에 대한 임의의 모범 답안을 의미하며, context_sim은 두 문장간의 유사도를 context-to-vector 모델과 단어 임베딩(word embedding) 모델을 이용하여 측정하는 함수를 의미한다. 이에 따라 발화자의 답변과 임의의 모범 답안의 유사도가 일정 기준(

)보다 높거나 같은 경우 정답이라고 판단할 수 있다.At this time, answer is the answer of the speaker,

Context_sim means a function for measuring the similarity between two sentences using a context-to-vector model and a word embedding model. Thus, if the similarity between the speaker's answer and any model answer is a certain standard

), It can be judged that the answer is correct.

If the answer uttered by the speaker is the correct answer, the next question determined according to the comprehension degree of the speaker can be presented. If the answer is not correct, the question can be limited to one question before the question. In this case, if the speaker presents incorrect answers again to the questions made of material, it can be passed to the next set question in the order of the scenario script. In this way, the order of the script can be expressed by the equation (3) with respect to the answer of the speaker.

Equation (3)

는 현재 질문 q_current에 대하여 발화자가 맞춘 모범 정답

_current 이 지정한 다음 질문을 의미한다. 모범 정답은 각기 다른 질문을 다음 질문으로 지정할 수 있으며, 발화자의 답변 내용에 따라 결정될 수 있다. 아래의 예시 1과 같이 발화자의 아바타의 질문에 대하여 발화자가 답변을 할 수 있고, 발화자의 답변에 따라 아바타의 답변이 각각 다르게 제시될 수 있다. Where q _next is the _next question to be presented based on the answer to the _current question, q _current is the current question,

The current question q the _current best guess for the _current

_It means the next question specified by _current . Modal correct answers can be assigned different questions to the next question, and can be determined according to the contents of the speaker's answers. As shown in the following example 1, the speaker can answer the question of the avatar of the talker, and the answer of the avatar may be presented differently according to the speaker's answer.

Example 1:

Avatar's Question: "How much are the bananas and the price per item?"

Speaker's Answer 1: "There is no fixed order."

Avatar's answer to speaker's answer 1: "If so, at what point should I decide again?"

or,

Speaker's Answer 2: "I want a total of 10 boxes, and I will set one box for 10Kg."

Avatar's answer to user's answer 2: "Okay, when do you want a contract?"

Further, the feedback providing system can analyze the behavior information obtained from the utterance training information of the speaker. The feedback providing system can grasp the upper body movement through the position of the upper body of the speaker. At this time, the feedback providing system can calculate the direction between the adjacent positions with respect to the collected upper body positions. Left wrist - left elbow, left elbow - left shoulder, left shoulder - neck, right wrist - right elbow, right elbow - right shoulder, right shoulder - neck, head - neck, neck - Direction vectors can be calculated. The nine direction vectors thus calculated can be used to deduce the upper body motion of the speaking person, and a method of inferring the upper body motion of the speaking person can be expressed as Equation (4).

Equation 4:

부터

까지 발화자의 상체 위치로부터 일련의 과정을 통하여 계산된 9개의 3차원 방향 벡터를 의미한다.

는 계산된 9개의 3차원 방향 벡터를 순차적으로 연결한 27차원 벡터이다. 27차원 벡터에 대응되는 상체 동작은 수학식 5에 의하여 미리 정의된 상체 동작 중 가장 가까운 동작으로 계산되어 도출될 수 있다.At this time,

from

Dimensional direction vector calculated through a series of processes from the position of the upper body of the talker up to the three-dimensional direction vector.

Is a 27-dimensional vector that sequentially connects the nine calculated three-dimensional direction vectors. The upper body motion corresponding to the 27-dimensional vector can be calculated by calculating the closest motion among the upper body motion defined in advance by Equation (5).

Equation 5:

는 가장 유사도가 높은 상체 동작, sim은 두 벡터간의 코사인 유사도 측정함수,

는 임의의 사전 정의된 상체 동작을 의미한다.

는

와 같은 형식인 9개의 3차원 방향 벡터를 순차적으로 연결한 27차원의 벡터이다.At this time,

, Sim is the function of measuring the cosine similarity between two vectors,

Quot; means any predefined upper body motion.

The

Dimensional vector is a 27-dimensional vector in which nine three-dimensional direction vectors are sequentially connected.

 On the other hand, when the mobile device 200 requests the analysis, the feedback providing system receives the speech information of the text form speaker as input, and based on the received speech information, displays the conversation contents summary, conversation flow change, Can be provided as an analysis result.

The feedback providing system provides a natural language understanding program (e.g., a Watson natural language understanding application programming interface (API), a Google natural language understanding application programming interface, a Microsoft Lewis application programming interface ) Can be summarized. Furthermore, conversation contents can be summarized through various programs in addition to the programs shown in the embodiments.

The feedback providing system can derive the change of the conversation flow according to Equation (6).

Equation (6)

In this case, s _current means the current uttered contents, s _current-1 means the previous uttered contents of s _current , and if the similarity of the context between contents becomes low, it can be judged that the context has changed.

The feedback providing system can deduce the upper body movement of the speaking person from the speaking information. First, a probability model of Equation (7) can be generated on the basis of the training information and the upper body motion record at the time of training stored in the database.

Equation (7)

는 상체 동작,

는 발화된 단어를 의미하고, count(

)는 해당 단어가 발화된 횟수, count(

,

)는 해당 단어와 상체 동작이 동시에 출현한 횟수를 의미한다. 발화자에게 발화자로부터 주로 표현된 상체 동작에 대한 종합적인 피드백을 제공할 수 있다. 주로 표현된 상체 동작은 수학식 8에 의하여 선정될 수 있다.

The upper body motion,

Means the word uttered, and count (

) Is the number of times the word was spoken, count (

,

) Means the number of times the word and upper body movements appeared at the same time. It is possible to provide the speaker with comprehensive feedback on the upper body motion mainly expressed from the speaker. The upper body motion, which is mainly expressed, can be selected by Equation (8).

Equation (8)

)가 발화됐을 때, 상체 동작(

)가 발현될 확률

이 일정 기준

이상이라면, 주로 표현된 상체 동작이라고 정의될 수 있다. 피드백 제공 시스템은 모바일 장치에서 발화된 발화 정보에 기반하여 상체 동작에 대한 피드백을 제공할 수 있다. 먼저, 발화된 대화 정보를 단어 단위로 분할하고, 발화된 단어들에 대하여 수학식 7과 8에서 제시된 확률모델과 선정 기준을 토대로 표현될 확률이 높은 상체 동작들을 선정할 수 있다. 선정된 상체 동작들은 발화의 의미 분석 결과와 함께 전달될 수 있다.word(

), The upper body movement (

) Is expressed

Based on this schedule

, It can be defined as the upper body operation mainly expressed. The feedback providing system may provide feedback on upper body motion based on utterance information uttered in the mobile device. First, the utterance conversation information is divided into words, and upper-body movements having a high probability of being expressed on the basis of the probability model and the selection criterion shown in Equations (7) and (8) can be selected for uttered words. The selected upper body movements can be delivered along with the semantic analysis result of the utterance.

In step S430, the feedback providing system may generate the response and feedback of the avatar. The feedback providing system can generate the feedback of the avatar and the comprehensive feedback information on the speaker based on the analysis result generated by analyzing the speech training information or the speech information about the speaker. At this time, the feedback providing system may generate feedback differently based on the analysis result based on the communication training apparatus 300 or the mobile apparatus 200.

The feedback providing system can be expressed to the user through the avatar in the case of the communication training apparatus 300, and the contents of the avatar's utterance and behavior can be shown in Table 2.

Table 2:

The avatar can be performed until the scenario ends, and the feedback providing system can generate comprehensive feedback as the series of questions are completed. The contents of this comprehensive feedback can be shown in Table 3.

Table 3:

The feedback providing system may provide feedback to the communication training device 300 as shown in Example 2.

Example 2:

"You've often chained your chin. Be careful because the jaw can give you the impression that you are bored!"

In addition, the feedback providing system may generate feedback in the case of the mobile device 200 as shown in Table 4.

Table 4:

In addition, the feedback providing system may provide feedback to the mobile device 200 as in Example 3 or Example 4.

Example 3:

"Have you changed the topic of conversation?"

Example 4:

"Do you have a chin?"

In step S440, the feedback providing system may deliver the result to each device. In other words, the feedback providing system may deliver the generated result as the communication training apparatus 300 or the mobile apparatus 200 processes steps S410 to S430.

5 is a flowchart illustrating a virtual reality-based communication training method according to an exemplary embodiment of the present invention.

The virtual environment in the communication training device 300 can implement a background environment using a game engine (e.g., Unity 3D), and can be used to create an animation program (e.g., Autodesk 3ds Max) So that the avatar generation and the operation of the avatar can be implemented. It can be implemented through various programs in addition to the virtual environment implementation method shown in the embodiment.

In step S510, the communication training device 300 may select a communication training scenario. A situation in which a speaker wants to train from at least one or more scenarios provided by the virtual reality-based communication training device 300 can be selected.

In step S520, the communication training device 300 can query the first question for the selected scenario. At this time, the first question on the scenario script selected by the avatar from the speaker can be uttered. The voice utterance of the avatar can be expressed as voice data using a text-to-speech (TTS) program (for example, a basic text-to-speech program installed in Windows 10) have.

In step S530, the communication and training apparatus 300 may input a response of the speaker to the question of the avatar. For example, a speaker may utter an answer that he thinks is most appropriate for an avatar's question. At this time, the speaker can gaze at a specific position, such as a dialogue in daily life, and can take various upper body actions.

In step S540, the communication and training apparatus 300 may request an analysis of the utterance of the speaker and the behavior. The communication training device 300 may convert the behavior performed during the speech and the speech of the speaking person into a form that can be processed by the computer program and then transmit the communication to the feedback system 100 to request analysis.

The utterance of the speaker is recognized by the microphone built in the HMD, and can then be transferred as text data. The voice data to be uttered by the speaker can be transferred into text data using a voice recognition program (for example, Google Cloud Speech Application Programming Interface) or the like.

The behavioral information of the speaker can be obtained by collecting the user's upper body motion using the HMD (for example, HTC VIVE) using a depth camera (e.g., Kinect), and then using the computer program It can be converted to a specific format.

Among the behavior information, the gaze can be converted into information on whether or not the speaker is looking at the avatar. To do this, the user's gaze can be converted into the following three-dimensional vector value eyegaze _user by utilizing the application programming interface of the HMD. Whether or not the talker is looking at the avatar by acquiring the vector value of the sight line of the talker can be derived by Equation (9).

 Equation (9)

는 r의 최대 허용 거리로 정의한다.7, eyecontact indicates a result of judging whether or not the speaker sees the avatar, r is the shortest distance from the eyegaze _user 710 to the head position 720 of the avatar,

Is defined as the maximum allowable distance of r.

In the behavior information, the upper body motion can collect the main position of the upper body using the depth camera. Ten positions including left and right wrists, right and left elbows, left and right shoulders, head, neck, abdomen, and buttocks can be collected as three-dimensional coordinate values. The information converted into a form that can be processed by the computer program is transmitted to the feedback providing system 100.

In step S550, the communication training device 300 may output a response generated as the response of the avatar is generated. The communication training apparatus 300 can generate an avatar utterance and an action corresponding to a result received from the feedback providing apparatus 100. [ The utterance of the avatar is output through a text-to-speech program, and the behavior is output through an animation program. At this time, if the speaker's response to the utterance of the avatar is detected, step S530 may be performed.

6 is a flow chart illustrating a method of providing feedback using a mobile device in accordance with one embodiment.

The mobile device recognizes the entire conversation situation in which the user participates in daily life and analyzes the utterance of the speaker to transmit a feedback message about the utterance and behavior through the mobile device.

In step S610, the mobile device may collect the firing situation. The mobile device collects the speech information including the contents of the conversation, the sight line information or the action information of the participant through the feedback program executed in the form of an application, and converts the collected speech information into text data. For example, speech data uttered by a speaking person can be transferred as text data using a speech recognition program (e.g., Google Cloud Speech Application Programming Interface (API)) or the like.

In step S620, the mobile device may transmit the collected information from the speaker to the feedback providing system to request the speaker's utterance and behavioral analysis.

In step S630, the mobile device generates feedback based on the analysis result transmitted from the feedback providing system, and outputs the generated feedback to the mobile device. At this time, referring to FIG. 11, the analysis result of the speaking information of the speaking person and the feedback about the upper body motion predicted through speaking are outputted. The result of the analysis of the speech information can be displayed as text data at the top of the application of the mobile device. The feedback on the upper body motion expected from the speaker can be displayed at the bottom of the application of the mobile device and can notify the speaker whether or not the specific upper body operation is being performed. Speakers can be aware of and improve speech content and upper body behavior themselves through feedback from applications.

The apparatus described above may be implemented as a hardware component, a software component, and / or a combination of hardware components and software components. For example, the apparatus and components described in the embodiments may be implemented within a computer system, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA) , A programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to execution of the software. For ease of understanding, the processing apparatus may be described as being used singly, but those skilled in the art will recognize that the processing apparatus may have a plurality of processing elements and / As shown in FIG. For example, the processing unit may comprise a plurality of processors or one processor and one controller. Other processing configurations are also possible, such as a parallel processor.

The software may include a computer program, code, instructions, or a combination of one or more of the foregoing, and may be configured to configure the processing device to operate as desired or to process it collectively or collectively Device can be commanded. The software and / or data may be in the form of any type of machine, component, physical device, virtual equipment, computer storage media, or device As shown in FIG. The software may be distributed over a networked computer system and stored or executed in a distributed manner. The software and data may be stored on one or more computer readable recording media.

The method according to an embodiment may be implemented in the form of a program command that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, and the like, alone or in combination. The program instructions to be recorded on the medium may be those specially designed and configured for the embodiments or may be available to those skilled in the art of computer software. Examples of computer-readable media include magnetic media such as hard disks, floppy disks and magnetic tape; optical media such as CD-ROMs and DVDs; magnetic media such as floppy disks; Magneto-optical media, and hardware devices specifically configured to store and execute program instructions such as ROM, RAM, flash memory, and the like. Examples of program instructions include machine language code such as those produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter or the like.

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is to be understood that the invention is not limited to the disclosed exemplary embodiments. For example, it is to be understood that the techniques described may be performed in a different order than the described methods, and / or that components of the described systems, structures, devices, circuits, Lt; / RTI > or equivalents, even if it is replaced or replaced.

Therefore, other implementations, other embodiments, and equivalents to the claims are also within the scope of the following claims.

Claims (15)

In a feedback providing method performed in a feedback providing system,
Constructing a virtual reality environment for performing communication training of a speaker;
Obtaining speech training information including visual information, behavior information, and dialog information of a speaking person uttered from the speaking person in the virtual reality environment;
Generating an analysis result related to the speaker by analyzing the obtained speech training information; And
Providing feedback based on the analysis result in response to speech information generated in a real situation from the speaker
Lt; / RTI > The method according to claim 1,
Wherein the step of generating the analysis result related to the speaker by analyzing the obtained speech training information comprises:
Determining a correct answer to the answer of the speaker by measuring the best answer to the scenario script and the similarity of the answer of the speaker to the conversation information including the text data or voice data of the speaker, Selecting the next question based on whether the answer is correct
Lt; / RTI > The method according to claim 1,
Wherein the step of generating the analysis result related to the speaker by analyzing the obtained speech training information comprises:
Calculating nine directional vectors based on three-dimensional coordinate values of at least one or more body parts of the speaker, and determining behavioral information of the speaker based on the calculated directional vectors
Lt; / RTI > The method according to claim 1,
Wherein the step of generating the analysis result related to the speaker by analyzing the obtained speech training information comprises:
The communication training device provides an analysis result including the percentage of correct answer to the question, the wrong answer to the question, the answer to the question, the contact level of the talker, or the action information performed more than a predetermined number of times from the talker and the meaning of the behavior information step
Lt; / RTI > The method according to claim 1,
Wherein providing feedback based on the analysis results comprises:
Providing the mobile device with summary information on the conversation contents of the talker, whether the conversation flow of the talker is changed, or behavior information predicted from the speaker, as feedback information
Lt; / RTI > The method according to claim 1,
Wherein the step of configuring the virtual reality environment for performing the communication training of the speaker includes the steps of:
A step of communicating with the generated avatar through the communication training apparatus that provides the virtual reality environment by generating an avatar to perform conversation with the speaker,
Lt; / RTI > The method according to claim 6,
Wherein the step of configuring the virtual reality environment for performing the communication training of the speaker includes the steps of:
Providing a scenario including at least one kind for performing the communication training, and setting a scenario script based on a scenario selected from the speaker
Lt; / RTI >
The scenario script includes:
Wherein the order of questions is dynamically determined in correspondence with the utterance training information of the speaker
≪ / RTI > A computer program stored in a computer-readable medium for executing a feedback method performed in a feedback providing system,
Constructing a virtual reality environment for performing communication training of a speaker;
Obtaining speech training information including visual information, behavior information, and dialog information of a speaking person uttered from the speaking person in the virtual reality environment;
Generating an analysis result related to the speaker by analyzing the obtained speech training information; And
Providing feedback based on the analysis result in response to speech information generated in a real situation from the speaker
And a computer program stored in the computer readable recording medium. In a feedback providing system,
A constituent unit for configuring a virtual reality environment for performing communication training of a speaker;
An obtaining unit obtaining speech training information including visual information, behavior information, and dialog information of a speaking person uttered from the speaking person in the virtual reality environment;
A generating unit for generating an analysis result related to the talker by analyzing the obtained speech training information; And
And providing feedback based on the analysis result in response to the utterance information generated in the actual situation from the utterance
. 10. The method of claim 9,
Wherein the generation unit comprises:
Determining a correct answer to the answer of the speaker by measuring the best answer to the scenario script and the similarity of the answer of the speaker to the conversation information including the text data or voice data of the speaker, Based on whether the answer is correct,
The feedback system comprising: 10. The method of claim 9,
Wherein the generation unit comprises:
Calculates nine direction vectors based on three-dimensional coordinate values of at least one body part of the speaker, and grasps behavior information of the speaker based on the calculated direction vector
The feedback system comprising: 10. The method of claim 9,
Wherein the generation unit comprises:
The communication training device provides an analysis result including the percentage of correct answer to the question, the wrong answer to the question, the answer to the question, the contact level of the talker, or the action information performed more than a predetermined number of times from the talker and the meaning of the behavior information
The feedback system comprising: 10. The method of claim 9,
Wherein the providing unit comprises:
The summary information on the conversation contents of the talker, whether the conversation flow of the talker is changed, or the behavior information predicted from the talker is provided as feedback information to the mobile device
The feedback system comprising: 10. The method of claim 9,
The above-
The communication with the avatar is performed through the communication training device that provides the virtual reality environment by generating the avatar performing the conversation with the speaking person
The feedback system comprising: 15. The method of claim 14,
The above-
Providing a scenario including at least one kind for performing the communication training, and setting a scenario script based on a scenario selected from the speaker
/ RTI >
The scenario script includes:
And a query sequence is dynamically determined in response to the speech training information of the speaker.

Images