KR101743230B1 - Apparatus and method for providing realistic language learning contents based on virtual reality and voice recognition - Google Patents

Abstract

The present invention relates to an apparatus and method for providing realistic language learning contents based on virtual reality and speech recognition. The apparatus includes a storage module, a projection module, an audio module, a speech recognition module, and a content providing module. The storage module stores contents including video and audio for reproducing a virtual reality. The projection module projects an image, and the audio module receives or outputs a voice. The voice recognition module outputs a voice recognition result by voice recognition of a user's answer or question about a virtual reality object received from the audio module. The content providing module evaluates the user's answer or question according to the speech recognition result, and outputs the content including the voice and the video in which the object responds to the user's answer or question in response to the evaluation result, to provide.

Description

TECHNICAL FIELD The present invention relates to an apparatus and method for providing real-life language learning contents based on virtual reality and speech recognition,

The present invention relates to a technology for providing language learning contents, and more particularly, to an apparatus and a method for providing real learning language learning contents based on virtual reality and speech recognition.

Virtual reality (VR) refers to a specific environment or situation, or technology itself, that is created by artificial techniques using computers and the like, but is not actual. At this time, the created virtual environment or situation stimulates the user's five senses and makes the user experience the spatial and temporal experiences similar to reality, thereby freely bringing the boundary between reality and imagination free. Users can not only simply immerse themselves in virtual reality, but they can also interact with things implemented in virtual reality, such as manipulating or commanding them using real devices.

SUMMARY OF THE INVENTION An object of the present invention is to provide an apparatus and a method that can provide an experience such that a user directly interacts with a foreigner through voice recognition through a virtual reality.

In order to achieve the above object, an apparatus for providing a learning content according to an embodiment of the present invention includes a storage module for storing a content including a video and a voice for reproducing a virtual reality, a projection module A voice recognition module for outputting a voice recognition result by voice recognition of a user's answer or question about an object of a virtual reality received from the audio module; The audio module and the audio module output a content including a voice and an image in which the object responds to a user's answer or a question in response to the evaluation result, And a providing module.

According to another aspect of the present invention, there is provided an apparatus for providing a learning content, the apparatus comprising: a projection module for outputting an image; an audio module for outputting audio; A sensor module for detecting a motion of the user and outputting coordinates according to the detected movement, a user detecting a view of the user according to the coordinates, constituting a content including a video and a voice corresponding to the detected view, And a content providing module for providing the video module and the audio module to output.

According to an aspect of the present invention, there is provided a method for providing a learning content, the method comprising: receiving a user's answer or a question about an object of a virtual reality in which an audio module is received; A step of outputting a speech recognition result by voice recognition of a user's answer or a question; a step of evaluating an answer or a question of the user according to a result of speech recognition by the content providing module; Or constructing a content including a video and a voice in response to a question, and outputting the content by a video module and an audio module.

In order to achieve the above object, a method for providing a learning content according to an embodiment of the present invention includes sensing a movement of a virtual reality reproducing unit worn by a sensor module of a user and outputting coordinates according to a sensed movement Detecting a user's view according to the coordinates and constructing a content including video and audio corresponding to the detected view; and displaying the video of the configured content And outputting the voice.

According to the present invention, it is possible to maximize the effect of language learning even if the user does not go abroad by providing experiences such as meeting and interacting with foreigners directly through virtual reality and speech recognition.

1 is a diagram for explaining a configuration of a content providing apparatus according to an embodiment of the present invention.
2 is a block diagram for explaining a configuration of a content providing unit and a virtual reality reproducing unit according to an embodiment of the present invention.
3 to 6 are views for explaining an image according to an embodiment of the present invention.
7 to 12 are views for explaining an object image according to an embodiment of the present invention.
13 is a flow chart illustrating a method for providing content according to an embodiment of the present invention.
14 is a flowchart for explaining a learning content providing method according to an embodiment of the present invention.
15 is a flowchart illustrating a method for providing learning contents using speech recognition according to an embodiment of the present invention.
FIG. 16 is an example of a screen for explaining a listening object for providing learning contents according to an embodiment of the present invention.
17 is a diagram illustrating an example of a voice command according to an embodiment of the present invention.
18 is a diagram for explaining a method of providing learning contents according to an interaction according to an embodiment of the present invention.
19 is a diagram for explaining a method of providing learning contents according to an interaction according to another embodiment of the present invention.
FIG. 20 is a diagram for explaining an image configuration method according to an embodiment of the present invention.
FIG. 21 shows an example of a graph for comparing pronunciation, intensification, and intonation of a user and a native speaker according to an embodiment of the present invention.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

The present invention provides a virtual reality in which a user who wants to learn a foreign language, for example, English, can provide an experience to speak with a virtual partner in a foreign language. An apparatus for providing a virtual reality-based learning content according to an embodiment of the present invention will now be described. 1 is a diagram for explaining a configuration of a learning contents providing apparatus according to an embodiment of the present invention. Referring to FIG. 1, a learning content providing apparatus according to an embodiment of the present invention includes a content providing unit 100 and a virtual reality reproducing unit 200. The learning content providing apparatus may further include a display unit 300. [

The contents providing unit 100 provides the virtual reality reproducing unit 200 with the learning contents according to the embodiment of the present invention. Here, the content includes an image for visually expressing a virtual reality (VR) and a sound for expressing audibly. In addition, the content providing unit 100 may provide the same content to the display unit 300 as that provided to the virtual reality reproducing unit 200. [ According to one embodiment, the content providing unit 100 may be implemented as a set-top box. In this case, the content providing unit 100 may provide the virtual reality reproducing unit 200 with an HDMI distributor It is possible to provide the same content to the display unit 300. [ According to another embodiment, the virtual reality reproducing unit 200 may provide the display unit 300 with the same content as the content received from the content providing unit 100. [

The virtual reality reproducing unit 200 receives a voice of a user and delivers the input voice to the content providing unit 100, and the content providing unit 100 provides a content corresponding to the received voice. Then, the virtual reality reproducing unit 200 outputs the content so that the user can experience the virtual reality. The virtual reality reproducing unit 200 may represent an HMD (Head Mounted Display). Accordingly, the user can experience the contents as a virtual reality by wearing the virtual reality reproducing unit 200 on the head.

The display unit 300 outputs the content provided by the content providing unit 100. [ The display unit 300 may be a display device including a speaker, for example, a monitor. The display unit 300 may include a liquid crystal display (LCD), an organic light emitting diode (OLED), and an active matrix organic light emitting diode (AMOLED).

Hereinafter, the configuration of the content providing unit 100 and the virtual reality reproducing unit 200 according to the embodiment of the present invention will be described. 2 is a block diagram for explaining a configuration of a content providing unit and a virtual reality reproducing unit according to an embodiment of the present invention. 2, the content providing unit 100 includes a voice recognition module 110, a content providing module 120, and a storage module 130. [ The virtual reality reproducing unit 200 includes a projection module 210, an audio module 220, and a sensor module 230.

The voice recognition module 110 receives the user's voice from the audio module 220 of the virtual reality reproducing unit 200 and recognizes the received voice. The voice recognition module 110 extracts a feature vector from the voice and generates a voice vector when the voice is input from the audio module 220. The speech recognition module 110 then performs speech recognition on the speech vector in a search space formed on the basis of an acoustic model, a pronunciation dictionary, and a language model. This speech recognition is performed on a phoneme basis. In addition, the result of speech recognition may be a recognition result of 1-best or N-best, but a recognition result of N-best is preferable.

The content providing module 120 receives the voice of the user inputted through the microphone MIC or receives the voice of the virtual reality reproducing unit 200 sensed through the sensor module 230, And provides the virtual reality reproducing unit 200 with a content corresponding to the coordinates of the voice or movement. The content providing module 120 includes an encoder or a decoder for encoding or decoding video and audio. In particular, the encoder and decoder may be implemented in hardware or software.

The storage module 130 stores programs and data necessary for the operation of the contents providing unit 100, and can be divided into a program area and a data area. The program area may store a program for controlling an overall operation of the contents providing unit 100 and an operating system (OS) for booting the contents providing unit 100, an application program, and the like. The data area is an area where user data generated according to use of the contents providing unit 100 is stored. The storage module 130 also stores content according to an embodiment of the present invention. Such content includes video and audio. Various data stored in the storage module 130 can be deleted, changed, or added according to a user's operation.

The speech recognition module 110 and the content providing module 120 may be implemented by a central processing unit (CPU), an application processor (AP), a micro-controller, a graphics processing unit (GPU) Graphics Processing Unit), a digital signal processor (DSP), or the like. The storage module 130 may be implemented by a hard disk drive (HDD), a solid state disk drive (SDD), a read access memory (RAM), a read only memory (ROM), a flash memory, or an EEPROM.

The projection module 210 includes a projector and an optical system, although not shown. The projector emits an image through the microdisplay panel, and the optical system reflects or refracts the image and projects it onto the user's eye (pupil). Thereby, the user can view the enlarged image representing the virtual reality.

The audio module 220 includes a microphone MIC and a speaker SPK. The audio module 220 receives a user's voice through a microphone (MIC), and provides the input voice to the voice recognition module 310. Also, the audio module 220 outputs the voice input from the content providing module 320 through the speaker SPK.

The sensor module 230 is for sensing movement of the virtual reality reproducing unit 200. Since the virtual reality reproducing unit 200 is worn on the head portion of the user, the user can see his / her field of view when he / she senses the motion of the virtual reality reproducing unit 200 through the sensor module 230. For this, the sensor module 230 senses the motion of the virtual reality reproducing unit 200 and provides the sensed motion to the content providing module 320 in coordinates. The coordinates that the sensor module 230 provides to the content providing module 320 include x, y, z in the three-dimensional Cartesian coordinate system and yaw, pitch, and roll. The sensor module 230 may be implemented through one or more sensors, and the sensor may be an accelerometer, a gyroscope, a magnetometer, or the like.

Next, a background image V representing a virtual reality according to an embodiment of the present invention will be described. 3 to 6 are views for explaining an image according to an embodiment of the present invention.

In Fig. 3, reference symbol S denotes a virtual space. The virtual reality according to the embodiment of the present invention is provided such that when a user wears the virtual reality reproducing unit 200, the user can experience a virtual experience such as being in the position U on the virtual space S. To this end, the background image V is generated by photographing the virtual space S in the forward direction at the position U. Particularly, the background image V is provided only a part of the virtual space S according to the view of the user for realism. In the real world, when a user turns his or her gaze, the user's view changes. Likewise, according to the present invention, a part of the virtual space S is provided as the background image V in accordance with the changed view field of the user.

The present invention allows a user who wishes to learn a language to experience a virtual reality in a language to be learned in a virtual space (S). Depending on the subject, various virtual spaces (S) are provided, and each subject can be defined according to a specific place or situation. For example, the subject may be an airport, a restaurant, a performance hall, a hotel, a subway, a city, a playground, a scenic spot, etc., depending on a specific place. Alternatively, the subject may be a parody situation of a particular movie, a parody situation of a particular drama, or the like. That is, the learning contents according to the embodiment of the present invention has a virtual space S corresponding to each subject in order to reproduce a specific place or situation as a virtual reality according to the subject. A part of the virtual space S is provided as a background image V according to the visual field. Accordingly, the user can intuitively recognize the subject, that is, a place or a certain situation, through the background image (V). The example shown in FIG. 4 shows an example of a background image (V) used to reproduce a subway as a virtual reality. Through the vehicle, subway platform of the background video (V), the user can recognize that he will experience the virtual reality related to the subway.

According to the embodiment of the present invention, since the virtual reality reproducing unit 200 is worn on the head of the user, the virtual reality reproducing unit 200 detects the movement of the virtual reality reproducing unit 200 and detects the movement of the user's visual field Estimate the change. For this, a sensor module 230 embedded in the virtual reality reproducing unit 200 is used. 5, the sensor module 230 senses the motion of the virtual reality reproducing unit 200 and outputs the motion as coordinates (x, y, z, yaw, pitch, roll). Then, the content providing module 120 detects the direction of the line of sight from the coordinates (x, y, z, yaw, pitch, roll) and configures the background image V according to the view according to the line of sight. The background image V provided by the virtual reality reproducing unit 200 is in accordance with the viewing angle provided by the virtual reality reproducing unit 200. In other words, This viewing angle depends on the specifications of the projection module 210, that is, the specifications of the projector and the optical system. Accordingly, the content providing module 120 can obtain a direction of view (DoV) from the focus point (FP) using coordinates (x, y, z, yaw, pitch, roll) received from the sensor module 230 And extracts the background image V from the virtual space S in accordance with the field of view angle FoV provided by the projection module 210 from the center of the line of sight. 5, it is assumed that the left and right viewing angles FoV and FoV are about 115 degrees and 90 degrees, respectively, according to the specification of the projection module 210. [ The content providing module 320 derives the direction of the line of sight from the coordinates (x, y, z, yaw, pitch, roll) and extracts yaw, roll, and pitch and the virtual space S belonging to the field of view having the left and right viewing angles FoV of 115 degrees and the upper and lower viewing angles FoV and FoV of 90 degrees is extracted and constituted as a background image V according to the rolls.

After configuring the background image V, the content providing module 320 provides the background image V to the projection module 210, and the projection module 210 projects the background image V. FIG. Therefore, the user wearing the virtual reality reproducing unit 200 can see the background image V captured in the field of view corresponding to the change of the line of sight when the user changes his / her line of sight by turning his or her head. For example, as shown in FIG. 6, when the user is in the initial image (V_initial), when the user turns his / her head to the left, the sensor module 230 outputs coordinates in which the value of yaw has changed, The content providing module 320 forms a background image V_yaw with a changed yaw, and the projection module 210 projects the background image V_yaw. Similarly, when the user leans his / her head to the left, the sensor module 230 outputs the coordinate at which the value of the roll has been changed. Accordingly, the content providing module 320 outputs the background image V_roll, And the projection module 210 projects it. In addition, when the user pushes his / her head forward, the sensor module 230 outputs the coordinates at which the pitch value has been changed. Accordingly, the content providing module 320 outputs the background image V_pitch having a changed pitch And the projection module 210 projects it. Accordingly, the user can experience the virtual reality as in the virtual space S.

Meanwhile, the image of the content according to the embodiment of the present invention includes the background image V and the object image O. In other words, the image is the object image (O) matched to the background image (V). The object image O will be described below. 7 to 12 are views for explaining an object image O according to an embodiment of the present invention.

The object image O is for representing an object according to an embodiment of the present invention. 7, the object image O is photographed separately from the background image V according to the chroma key technique, and the photographed object image O is registered with the background image V to form an image of the content . In an embodiment of the present invention, an object includes a dialog object, a word object, and a moving object.

The conversation object is a virtual character existing in the virtual space S, and can talk with the user according to an embodiment of the present invention. That is, the conversation object is a virtual conversation partner. The conversation object is mainly a person, but it may be an animal or an inanimate character. For example, if the subject is a subway, a passenger, a station attendant, a police officer, etc. may be a conversation object.

8 and 9 show an example of a screen in which the background image (V) is displayed when the subject is a subway, and the object image O, which is a conversation object, is registered. The user can interact with such a conversation object according to an embodiment of the present invention. This method of conversation will be described in more detail below.

A word object is a character that describes the meaning of a word. These word objects help the user to intuitively understand the meaning of words. A word object can be an object if the word is a noun that represents the name of the object. For example, as shown in FIGS. 8 and 9, the word object may be 'signs' 53. And the word object of FIG. 10 is a 'turnstile' (55). In addition, a word object can be a subject that reproduces the meaning of the word when the word is an intangible such as a verb, adjective, adverb, etc. In Fig. 11, the word object is a character 57 that describes the meaning of the word "read ". Here, the word object adopts the imaginary person 57 reading the book to explain the meaning of "read ".

The moving object is intended to allow the user to move within the virtual space S. Such a moving object has a form of a mark indicating a direction in which the user can move. The user can move in the direction indicated by the moving object through the moving object in the virtual space S. [ For example, the moving object in Fig. 12 is an arrow 59 indicating the upward direction, and when this moving object is activated, the user can move up. This movement is not the user moving directly but moving the background image V to provide a virtual experience such as a user walking up the stairs.

Hereinafter, a method of providing learning contents according to the movement of a user wearing the virtual reality reproducing unit 200 on the head will be described based on the above description. 13 is a flowchart illustrating a method for providing learning contents according to an embodiment of the present invention.

3 to 13, the virtual reality reproducing unit 200, for example, the HMD is in a state in which the user wears it on his head. In this state, if there is a movement of the user such as turning the head, tilting, or bowing, the virtual reality reproducing unit 200 also moves. Accordingly, the sensor module 230 installed in the virtual reality reproducing unit 200 can sense the motion of the virtual reality reproducing unit 200 in step S110. The sensor module 230 transmits the movement of the virtual reality reproducing unit 200 sensed in step S120 to the content providing module 120 through the coordinates (x, y, z, yaw, pitch, roll).

Next, the content providing module 120 detects the field of view according to the coordinates (x, y, z, yaw, pitch, roll) indicating the movement of the virtual reality reproducing unit 200 in step S130. Here, it should be noted that the field of view is a field of view that can be provided by the projection module 210, not by a person's field of view. The background image V provided by the virtual reality reproducing unit 200 is in accordance with the viewing angle provided by the virtual reality reproducing unit 200. In other words, The viewing angle depends on the specifications of the projection module 210, that is, the specifications of the projector and the optical system. As shown in FIG. 5, the content providing module 320 receives the focus point (FP) from the sensor module 230 using the coordinates (x, y, z, yaw, pitch, And obtains the field of view according to the field of view angle (FoV) provided by the projection module 210 from the center of the line of sight.

Then, in step S140, the content providing module 120 composes a video including a background image (V) and an object image (O) corresponding to the previously obtained view and a content including a voice required for the image.

Then, the content providing module 120 provides the video and audio to the projection module 210 and the audio module 220 in step S150. Then, the projection module 210 and the audio module 220 output video and audio in step S160. That is, the audio module 220 outputs audio so that the user can perceive audibly, and the projection module 210 projects the image so that the user can visually recognize the image.

As described above, the image includes the background image (V) and the object image (O). According to the embodiment of the present invention, the object image O has an active mode and an inactive mode, and the object image O for each of the active mode and the inactive mode of the object is selectively provided according to the condition.

As described above, the conversation object performs a virtual conversation with the user, the word object describes the meaning of the word to the user, and the moving object enables the user to move within the virtual space S. These functions are provided in the active mode, and in the inactive mode, the basic image in which only the outline of the object is expressed is reproduced. Basically, the object image O is provided in an inactive mode, and when a predetermined condition is satisfied, it is switched to the active mode. Hereinafter, a method of providing learning contents after switching an inactive mode object to an active mode will be described. 14 is a flowchart for explaining a learning content providing method according to an embodiment of the present invention. The embodiment of FIG. 14 is performed in step S140 described above with reference to FIG.

As described above, the content providing module 120 detects the field of view from the sensor module 230 using the coordinates of the motion of the virtual reality reproducing unit 200 in step S130. Accordingly, the content providing module 120 determines whether an object exists in the center of the field of view (C) in step S210. Here, the center C of the field of view is derived from the direction of sight line DoV. 8 to 12 are images (V) including an object image O of the objects 51, 53, 55, 57 and 59 in the inactive mode. In the case of FIGS. 8 and 9, there are a conversation object police 51 and a word object 'signs' 53. In the case of FIG. 8, there is no object at the center C of the field of view, but in the case of FIG. 9, there is the object 51 which is the object at the center C of the field of view. The object image O of all the objects 51 and 53 of FIG. 8 remains in the inactive mode, and the object image O of the police 51 of FIG. 9 will be switched to the active mode. 10, 11 and 12, there are a turnstile 55, a virtual person 57 and an arrow 59 at the center C of the field of view. Accordingly, the object image O of the turnstile 55, the virtual person 57, and the arrow 59 will be switched to the active mode.

The content providing module 120 extracts the object image O corresponding to the active mode according to the type of the object from the storage module 130 in step S220, The object image (O) is matched to the background image (V) to form an image. On the other hand, if the inactive mode is maintained, the content providing module 120 extracts the object image O corresponding to the inactive mode according to the object type from the storage module 130 in step S240, Mode object image O to the background image V to construct an image.

Next, the difference between the inactive mode and the active mode for each object type will be described with reference to FIGS. 8 to 12, for example. For example, when the corresponding word of the word object of FIGS. 8, 9 and 10 among the word objects is a 'sign' 53 and a 'turnstile' 55, which are the names of objects, , And only the appearance of the object is provided in the inactive mode. On the other hand, in the active mode, the 'signs' 53 and the 'turnstiles' 55 may be spelled with the words 'signs' and 'turnstile' as text, have. In addition, the meaning of the words 'signs' and 'turnstile' can be provided in either Korean or English depending on the choice, either in voice or text. In addition, an example sentence for the words ('signs' and turnstile') can be provided in either voice or text. As another example, a hypothetical character 57, a word object of FIG. 11 of the word object, is an intangible word such as a verb 'read'. In this case, in the inactive mode, the object image O provides the appearance that the virtual person 57 reads the book. On the other hand, in the active mode, the object image O is generated in such a manner that the hypothetical person 57 directly writes the spelling of the word 'read', and the voice 57 in which the hypothetical person 57 pronounces the word ' . Optionally, in the active mode, the object image O may include a virtual person 57 describing the meaning of the word " read " directly in Korean or English, The image can be read. At this time, a voice corresponding to reading the meaning will be further included. Alternatively, the object image O may include an act of directly describing and reading the illustrative sentence of the word " read " by the virtual person 57 and a voice corresponding thereto.

As another example, in the case of the up arrow 59, which is the moving object of FIG. 12, only the outline of the up arrow 59 itself is shown in the inactive mode. When switched to the active mode, the object image O provides an image that can be viewed when the user climbs the stairs. This movement allows the user to provide a virtual experience of walking up the stairs rather than moving directly.

On the other hand, in the case of the police 51, which is the conversation object of FIGS. 8 and 9, the object image O in the non-active mode is provided only the outline of the police 51. [ On the other hand, in the active mode, the object image O of the conversation object police 51 allows the police 51, which is a virtual character, to interact with the user to enable conversation. In the active mode, an object image O prepared for various situations is prepared separately, and an image obtained by matching the object image O with the background image V according to each situation is provided. This object image O includes, for example, a dialogue object including a picture waiting for an answer during a conversation, a picture for inducing a quick answer, a picture when the user does not understand the speech, and a picture when the user understands the speech do. This dialogue is based on speech recognition. Hereinafter, a method for providing a learning content using speech recognition according to an embodiment of the present invention will be described. FIG. 15 is a flowchart illustrating a method for providing learning content using speech recognition according to an embodiment of the present invention. FIG. 16 is a flowchart illustrating a method for providing a learning object for providing learning content according to an embodiment of the present invention. Screen example.

As in Fig. 9, it is assumed that the police 51, which is a conversation object, is located at the center C of the field of view and is activated, as shown in Fig. Accordingly, the conversation object, the police 51, and the user can communicate in a language to be learned, for example, English. The conversation may be handed over by the police 51 according to a predetermined scenario, or the user may speak first. For example, the police 51 may first ask the user a question, or the user may ask the police 51 questions. Accordingly, the user can ask questions or reply.

When it is time for the user to ask or answer a question during the conversation, the content providing module 120 provides the background image (V) matching the listening object 60 as shown in FIG. 16 in step S310. Then, the projection module 210 outputs an image including the listener object 60 in step S320. The listening object 60 is represented by a microphone-shaped icon and a waveform of the voice signal.

After providing the listener object 60, a standby mode for waiting for a response from the user is performed in step S330. In the idle mode, the content providing module 120 basically provides the content including video and audio indicating that the conversation object is waiting for an answer. Accordingly, the projection module 210 and the audio module 220 may output video and audio indicating that the conversation object is waiting for an answer. Also, in the standby mode, the content providing module 120 provides a content including a video and a voice indicating that the conversation object prompts an answer in order to prompt a quick answer when there is no answer of the user for a preset time or more. Accordingly, the projection module 210 and the audio module 220 output an image and a voice indicating that the conversation object urges an answer.

If the user responds during the standby mode, the audio module 220 receives the user's voice (e.g., a question or answer) through the microphone in step S340, and transmits the user's voice to the voice recognition module 310 in step S350 do. Then, the speech recognition module 310 recognizes the user's speech in step S360 and outputs the speech recognition result in step S370.

When the output speech recognition result is transmitted to the content providing module 120, the content providing module 120 may evaluate the user's question or answer according to the speech recognition result in step S380. The speech recognition result may not be recognized due to various causes including the cause of input of a voice having a size smaller than the threshold value of the user or the like. In this case, an error may be output as a result of speech recognition. Accordingly, the content providing module 320 firstly evaluates whether or not the speech recognition result is present in step S380. If an error is output as a result of speech recognition, it can be estimated that speech recognition is not performed. If there is a speech recognition result, evaluate whether the user's question or answer is correct. The storage module 330 stores a plurality of possible questions and a plurality of answers according to various situations. Accordingly, the content providing module 120 can compare the voice recognition result with the context-based questions and answers stored in the storage module 330 to evaluate whether the user's question or answer is correct. In summary, in step S380, the content providing module 320 may evaluate that the speech recognition is not performed, evaluate that the speech recognition result is a correct question or answer, or evaluate the speech recognition result as an incorrect answer or answer have.

Then, the content providing module 320 forms a content including video and audio corresponding to the evaluation result in step S390. The content corresponding to the evaluation result includes video and audio that provide a response to the user's question or answer. For example, in the case where the speech recognition is not performed as a result of the evaluation, or the question or answer is not correct as a result of speech recognition, the content providing module 320 determines that the question or answer of the conversation object is not heard, 'Sorry', 'Sorry, I do not understand' or 'I do not understand' or 'I do not understand' (eg 'pardon me?', 'Can you say that again, (AUDIO) of 'I did not get that.', 'I did not hear what you said.', 'I'm sorry, but I could not hear you.' . As another example, if the evaluation result indicates that the user has answered a correct question or an answer, the content providing module 320 determines that the user understands the question or answer of the conversation object and the voice of the gesture and the voice For example, when a user asks a conversation object what time it is, the conversation object asks the user time for the answer, "It's two thirty.", The user answers the time, 'Is it that late already?').

Next, in step S370, the content providing module 320 provides the audio module 220 and the projection module 210 with the content including the video and audio configured in advance. Then, in step S380, the audio module 220 outputs a voice, and the projection module 210 projects an image.

As described above, according to the embodiment of the present invention, the user can communicate with the conversation object which is a virtual conversation partner through speech recognition. This enables users to maximize the effectiveness of the language learning even if they do not go abroad by providing a virtual experience such as meeting and talking to foreigners directly.

Meanwhile, according to the embodiment of the present invention, the user can perform various controls through voice. Hereinafter, a control method based on voice will be described. 17 is a diagram illustrating an example of a voice command according to an embodiment of the present invention.

As described above, the virtual space S can be classified according to the learning theme. The virtual space S can be divided into the virtual space S by selecting the subject, or reproducing the contents for virtual reality experience in the virtual space S An operation such as playing, stopping, or ending a virtual reality experience can be performed through a voice command. In addition, the present invention can provide a vocabulary word, an idiomatic vocabulary, and a lecture image according to a learning subject, and a user can input a voice command to provide a vocabulary word, an idiomatic word, and a lecture image. If there is a voice command as described above, the audio module 220 receives the voice command of the user through the microphone and transmits the voice command to the voice recognition module 310. Then, the voice recognition module 310 recognizes the voice command and outputs the voice recognition result. Then, the content providing module 120 compares the voice command with a predetermined voice command according to a voice recognition result, and executes the corresponding command in accordance with the voice command. For example, if the user inputs 'help' as a voice command, 'help' will be output as a speech recognition result, and if the 'help' is preset as a voice command to provide a lecture image, 120 may provide an image obtained by matching the image 70 of the steel to the background image V as shown in FIG.

According to the embodiment of the present invention, contents corresponding to the voice input of the user should be provided. That is, the content is provided in accordance with the interaction with the user. A method of providing contents according to such an interaction will be described. 18 is a view for explaining a method of providing contents according to an interaction according to an embodiment of the present invention.

Referring to FIG. 18, the storage module 130 combines and stores a plurality of corresponding contents into one piece of content data that is continuous in time according to an evaluation of a question or an answer.

In addition, time information indicating the temporal position of each of the plurality of contents is recorded in the metadata and stored in the storage module 130. [ For example, the first content may be a video and a voice for which a conversation object makes a first query according to a preset dialogue scenario, the second content is a video and voice corresponding to a user's answer to the first question, 3 < / RTI > content may be video and audio corresponding to an inability to understand the user's answer to the first question. In addition, the fourth content is a video and a voice asking a second question if the conversation object does not adequately answer the first question according to the scenario of the conversation, and the fifth content is a video and voice corresponding to the user's answer to the second question And the third content may be video and audio corresponding to the case where the user's answer to the second question is not understood. And the seventh content may be a video and a voice that ask a third question if the conversation object answers the second question properly according to the scenario of the conversation. In this case, the positions on the timeline of the first to seventh contents, that is, the positions on the time axis are the time information t1 To t7.

As described above, the user's question / answer is evaluated according to the speech recognition result in step S380. In step S390, when the video corresponding to the evaluation result is formed, the content providing module 120 transmits the time information And provides the corresponding content by jumping to the corresponding content. For example, the content providing module 120 first provides the first content according to the scenario. When there is a user's answer to the first question, the contents providing module 120 evaluates the user's answer to the first question according to the speech recognition result in step S380. In this case, if it is determined that the user's answer is not understood, the content providing module 120 refers to the time information stored in the storage module 130 in step S390, jumps to the third content, do.

Next, a method of providing contents according to an interaction according to another embodiment of the present invention will be described. 19 is a diagram for explaining a method of providing learning contents according to an interaction according to another embodiment of the present invention.

According to another embodiment, the storage module 130 separately stores each of a plurality of contents corresponding to the evaluation result of the question / answer. Corresponding information indicating that each of the plurality of contents produced in correspondence with the evaluation result of the question / answer is content corresponding to the evaluation result is recorded in the metadata and stored in the storage module 130. [

For example, the first content may be a video and a voice for which a conversation object makes a first query according to a preset dialogue scenario, the second content is a video and voice corresponding to a user's answer to the first question, 3 < / RTI > content may be video and audio corresponding to an inability to understand the user's answer to the first question. At this time, the corresponding information of the second content indicates that the answer corresponding to the evaluation result of the answer to the first content is appropriate, and the corresponding information of the third content corresponds to the evaluation result of the answer to the first content Indicates that the content corresponds to the case where the answer does not understand.

As described above, the user's question / answer is evaluated according to the speech recognition result in step S380. In step S390, when the content corresponding to the evaluation result is formed, the content providing module 120 transmits the corresponding information Extracts the corresponding content from the storage module 130, and provides the extracted content. For example, the content providing module 120 first provides the first content according to the scenario. When there is a user's answer to the first question, the contents providing module 120 evaluates the user's answer to the first question according to the speech recognition result in step S380. If it is determined that the user's answer is not understood, the content providing module 120 refers to the corresponding information in step S390, extracts the third content from the storage module 130, and provides the extracted third content .

Next, an image configuration method according to an embodiment of the present invention will be described. FIG. 20 is a diagram for explaining an image configuration method according to an embodiment of the present invention. 7, the object image O for representing an object is photographed separately from the background image V according to the chroma key technique, and the photographed object image O is photographed as a background image V ). 20 (a) shows a background image (V), and Fig. 20 (b) shows an object image O. Fig. 20 (c) shows an icon. According to the embodiment of the present invention, different layers are merged as shown in Figs. 20A, 20B and 20C to form one image as shown in Fig. 20D. Although three layers are shown as being merged in Fig. 20, a larger number of such layers may be used as needed. For example, the image of the lecture of Fig. 17 can be merged into an image using another layer.

Next, the present invention can provide a graph in comparison with the pronunciation, strength, and intonation of the native speaker, such as pronunciation, strength, and intonation of the user. FIG. 21 shows an example of a graph for comparing pronunciation, intensification, and intonation of a user and a native speaker according to an embodiment of the present invention.

Each time the user answers or questions are inputted from the audio module 220 in addition to voice recognition (S350), that is, every time the user's voice is input, the voice recognition module 110 detects a voltage change caused by the inputted voice signal The time envelope (b, B), which is an algebraic envelope of the temporal waveform in decibels (dB), is obtained by logarithmically processing the time waveforms (a, ), Comparing the pronunciation, strength and intonation of the user with the native speaker using the FFT spectrum (C, C) showing the power magnitude of the input signal by the Fast Fourier Transform (FFT) analysis of the selected data group , And outputs the comparison result through the graph so that the user recognizes the difference according to the comparison. Then, the content providing module 120 composes such a graph as an image, matches the background image V, and provides the output to the projection module 210 and the audio module 220. With reference to these graphs, the user can correct his pronunciation.

While the present invention has been described with reference to several preferred embodiments, these embodiments are illustrative and not restrictive. It will be understood by those skilled in the art that various changes and modifications may be made without departing from the spirit of the invention and the scope of the appended claims.

Claims (22)

An apparatus for providing learning content,
A storage module for storing a video including an object video and a background video for reproducing a virtual reality and a content including a voice required for the video;
A projection module for projecting the image;
An audio module for receiving or outputting the voice;
A voice recognition module for recognizing a user's answer or a question about a virtual reality object received from the audio module; And
And a voice recognition unit for comparing the voice recognition result with a context-based answer and a question stored in the storage module, and when the user's answer or question is voice-recognized, Or the audio module to output a content including a voice and an image in which the object responds to the user's answer or a question in response to the evaluated evaluation result A content providing module,
The content providing module
If the object does not exist in the visual center of the user, extracts an inactive mode object image in which the outline of the object is expressed by providing an inactive mode, matches the extracted inactive mode object image with a background image, In addition,
If the object exists in the center of the user's view, the active mode object image is extracted from the active mode object image providing the learning content corresponding to the object by switching the inactive mode to the active mode, To form an image. ≪ Desc / Clms Page number 19 > The method according to claim 1,
Wherein the storage module stores one piece of content data in which a plurality of contents corresponding to the evaluation result are continuously combined in time,
Wherein each of the plurality of contents has time information indicating a position in time in the one piece of contents data. 3. The method of claim 2,
The content providing module
And jumping the content corresponding to the evaluation result to a corresponding content according to the time information. The method according to claim 1,
Wherein the storage module individually stores a plurality of contents corresponding to the evaluation result,
Wherein each of the images has correspondence information indicating that the content corresponds to the evaluation result. 5. The method of claim 4,
The content providing module
And detects the content corresponding to the evaluation result in accordance with the corresponding information, and outputs the detected content. The method according to claim 1,
The speech recognition module outputs a graph comparing the user's pronunciation with respect to the input speech and the accent and intonation of the native speaker whenever the user's answer or question is input by voice,
Wherein the content providing module is configured to output the graph as an image. The method according to claim 1,
Wherein the voice recognition module outputs a voice recognition result for the voice command when receiving the voice command of the user from the audio module,
Wherein the content providing module configures a content corresponding to the voice command according to the voice recognition result and provides the content to be output by the projection module and the audio module. An apparatus for providing learning content,
A projection module mounted on a head of a user for outputting an image; And
An audio module for outputting audio;
A sensor module for detecting movement of the projection module and outputting coordinates according to the sensed movement;
Wherein the control unit detects a user's view according to the coordinates and constructs a content including an image including an object image and a background image for an object at a position corresponding to the detected view and a voice required for the image, And a content providing module for providing the output of the projection module and the audio module,
The content providing module
If the object does not exist at the center of the visual field, an inactive mode object image in which the outline of the object is expressed is extracted, and the extracted inactive mode object image is registered in the background image to form an image ,
The active mode object image providing the learning content corresponding to the object is switched by switching the inactive mode to the active mode when the object exists in the center of the visual field, And arranges the images to be matched. 9. The method of claim 8,
The content providing module
A center of the visual field of the user is detected according to the coordinates, and if an object exists in the center of the visual field, the video and audio for the object are configured, and the video and audio for the object are output, And providing audio to the projection module and the audio module. 10. The method of claim 9,
The content providing module
If the object is a dialogue object, constructs a video and a voice for questioning or answering the user of the dialogue object, and provides the composed video and audio to the projection module and the audio module, Device. 10. The method of claim 9,
The content providing module
If the object is a moving object, moving the background image so that a virtual reality in which the user moves in a direction indicated by the moving object is provided. 10. The method of claim 9,
If the object is a word object, providing a video and a voice explaining the meaning of a word corresponding to the word object. CLAIMS 1. A method for providing a learning content performed by an apparatus for representing a virtual reality by video and audio to provide learning content,
Outputting the first content to a virtual reality in which the projection module and the audio module include the object;
Receiving a user's answer or question about an object of the virtual reality in which the audio module is received;
Voice recognition module recognizing the user's answer or question;
The content providing module evaluates whether or not the user's answer or speech is recognized by the user. If the user's answer or question is recognized as speech, the content providing module compares the result of the speech recognition with the answer and question for each situation stored in the storage module, Constructing a second content including a video and a voice in which the object responds to the user's answer or question in response to the evaluated result of the evaluation; And
Wherein the projection module and the audio module output the second content,
The step of constructing the second content
If the object does not exist in the visual center of the user, extracts an inactive mode object image in which the outline of the object is expressed by providing an inactive mode, matches the extracted inactive mode object image with a background image, In addition,
If the object exists in the center of the user's view, the active mode object image is extracted from the active mode object image providing the learning content corresponding to the object by switching the inactive mode to the active mode, To form an image. ≪ Desc / Clms Page number 24 > 14. The method of claim 13,
Further comprising storing the plurality of contents corresponding to the evaluation result as one piece of content data continuously combined in time before the step of inputting the user's answer or question,
Wherein each of the plurality of contents has time information indicating a position in time in the one piece of contents data. 15. The method of claim 14,
The step of constructing the content
And jumping the content corresponding to the evaluation result to a corresponding content according to the time information. 14. The method of claim 13,
Further comprising the step of individually storing each of a plurality of contents corresponding to the evaluation result before a step of receiving a user's answer or a question,
Wherein each of the plurality of contents has correspondence information indicating that the content corresponds to the evaluation result. 17. The method of claim 16,
The step of constructing the content
And the content corresponding to the evaluation result is detected according to the corresponding information. CLAIMS 1. A method for providing a learning content performed by an apparatus for representing a virtual reality by video and audio to provide learning content,
Sensing the movement of the projection module worn by the sensor module on the user's head and outputting coordinates according to the detected movement;
A content providing module detects a user's view according to the coordinates and constructs a content including a video including an object video and a background video for an object at a position corresponding to the detected view and a voice required for the video step; And
Wherein the projection module and the audio module output video and audio of the configured content,
The step of constructing the content
If the object does not exist at the center of the visual field, an inactive mode object image in which the outline of the object is expressed is extracted, and the extracted inactive mode object image is registered in the background image to form an image ,
The active mode object image providing the learning content corresponding to the object is switched by switching the inactive mode to the active mode when the object exists in the center of the visual field, And arranging the images so as to form an image. 19. The method of claim 18,
The step of constructing the content
Detecting a center of the user's view according to the coordinates, and constructing an image and a voice for the object when the object exists in the center of the view. 20. The method of claim 19,
The step of constructing the content
And if the object is a dialog object, constructs a video and a voice for a question or an answer to the user of the dialog object. 20. The method of claim 19,
The step of constructing the content
If the object is a moving object, moving the background image so that a virtual reality in which the user moves in a direction indicated by the moving object is provided. 20. The method of claim 19,
The step of constructing the content
If the object is a word object, constitutes a content providing a video and a voice explaining the meaning of a word corresponding to the word object.

Images