KR20200079748A - Virtual reality education system and method for language training of disabled person - Google Patents

Abstract

The present invention relates to a virtual reality (VR) education system for language training of the developmentally disabled. According to the present invention, the VR education system comprises: a control device which converts a two-dimensional (2D) image into a 360° VR image, divides the VR image into a plurality of regions, and sets position coordinates in each divided region; and a head mounted display (HMD) device which receives and displays the VR image from the control device, detects the user′s voice, and changes the display of the VR image in response to the voice recognition result.

Description

VIRTUAL REALITY EDUCATION SYSTEM AND METHOD FOR LANGUAGE TRAINING OF DISABLED PERSON}

The present invention relates to a virtual reality education system and method for language training of persons with developmental disabilities, and more particularly, to a virtual reality education system and method capable of performing visual education together with language training using virtual reality images. will be.

With the spread of head mounted displays (hereinafter referred to as "HMD") devices, interest in virtual reality (hereinafter referred to as "VR") images is also increasing. The HMD device can provide the user with a VR image that can feel as if they are interacting with the actual surrounding environment.

When a 360° VR image is used, a viewpoint may be changed according to a user's movement, and thus images of various viewpoints may be provided. Attempts to apply these features to various fields such as games and education are actively underway.

However, users who lack cognitive ability, such as people with developmental disabilities, cannot expect a proper response according to VR video viewing, and have difficulty in inducing users to watch video at a specific time. Therefore, the audiovisual education method using VR images provided to these users has a problem of low efficiency.

The present invention is to solve the above-mentioned problems, by detecting the user's voice to determine what object (object) in the VR image that the user is trying to recognize, whether the user correctly recognizes the object corresponding to the voice recognition results The purpose of the present invention is to provide a virtual reality education system and method for language training of persons with developmental disabilities to judge.

In order to achieve the above object, the present invention converts a two-dimensional image into a 360° virtual reality (VR) image, divides the VR image into a plurality of regions, and sets position coordinates in each divided region. A virtual device for language training of persons with developmental disabilities, including a control device and an HMD device that receives and displays the VR image from the control device and detects a user's voice and changes the display of the VR image in response to a voice recognition result. Provide a reality education system.

According to an embodiment, the HMD device includes a display unit for displaying the VR image, a sensor unit for detecting motion of the user's head and generating motion information, and a voice for detecting the user's voice to generate a voice recognition result It may include a processor for determining the position coordinates of the central area viewed by the user based on the motion information, and controlling the image display of the display unit in response to the voice recognition result.

According to an embodiment, the control device may receive the position coordinates of the center area from the HMD device, and display a VR image of the same viewpoint as the HMD device based on the position coordinates of the center area.

According to an embodiment, the processor may compare the object information of each of a plurality of objects included in the VR image and the voice recognition result, and control the display unit to display the object corresponding to the voice recognition result with emphasis. .

According to an embodiment of the present disclosure, the processor may control the display unit such that the outline of the object corresponding to the speech recognition result is highlighted with bright luminance.

According to an embodiment, the processor may control the display unit such that the luminance of the region including the object corresponding to the speech recognition result is increased and displayed than the luminance of the other region.

According to an embodiment, the processor may control the display unit such that the luminance of an area other than the area including the object corresponding to the speech recognition result is reduced and displayed.

According to an embodiment, the object information includes positional coordinates of the object and text information indicating the object, and the object information may be generated by the control device.

According to an embodiment, the control device receives the voice recognition result from the HMD device, compares object information of each of a plurality of objects included in the VR image and the voice recognition result, and corresponds to the voice recognition result. Objects can be highlighted.

In addition, in order to achieve the above object, the present invention is a development disabled person using an HMD device that displays a virtual reality (VR) image to a user, and a control device that converts a two-dimensional image into the VR image and provides the HMD device. In the virtual reality education method for language training, the HMD device detects the user's voice to generate a speech recognition result, and the HMD device is the object among the objects included in the VR image displayed to the user Selecting an object corresponding to the result of speech recognition, and displaying, by the HMD device, highlighting the object to focus the selected object to the user, and the HMD device, the location of the user's central viewpoint Determining a coordinate, and displaying, by the HMD device, an education completion message on a display screen when the position coordinate of the user's center point of view is located in an area in which the selected object is placed for a predetermined time. It provides a virtual reality education method for language training of people with developmental disabilities.

According to an embodiment, a virtual reality education method for language training of a person with a developmental disability includes: the control device receiving a position coordinate of the center viewpoint from the HMD device, and the control device, the position of the center viewpoint The method may further include displaying a VR image identical to a VR image at a time point displayed by the HMD device based on the coordinates.

According to an embodiment, a virtual reality education method for language training of a person with developmental disabilities includes: the control device receiving the speech recognition result from the HMD device, and the control device is an object included in the VR image The method may further include a step of selecting an object corresponding to the speech recognition result, and the control device further highlighting and displaying the selected object.

According to an embodiment, the control device may further include receiving the training completion signal from the HMD device and displaying the training completion message.

According to a virtual reality education system and method for language training of a developmentally disabled person of the present invention, it is possible to detect a user's voice to determine which object the user is trying to recognize in a VR image, and to select an object corresponding to the voice recognition result. It can help users recognize objects by highlighting them.

In addition, according to the virtual reality education system and method for language training of the developmentally disabled person, it is possible to accurately determine whether the user is effectively recognizing an object by determining the position coordinates of the user's central viewpoint.

1 is a schematic block diagram of a virtual reality education system for language training of people with developmental disabilities according to an embodiment of the present invention.
2 is a schematic block diagram of an HMD device according to an embodiment of the present invention.
3A and 3B are diagrams for describing a 360° VR image according to an embodiment of the present invention.
4 is a diagram for explaining a virtual reality education method for language training of a developmentally disabled person according to an embodiment of the present invention.
5 is a flowchart illustrating a virtual reality education method for language training of a developmentally disabled person according to an embodiment of the present invention.
6 is a flowchart illustrating a virtual reality education method for language training of a person with developmental disabilities according to another embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. However, the embodiments described below are only for explaining in detail that the person skilled in the art to which the present invention pertains can easily implement the invention, thereby limiting the protection scope of the present invention. Does not mean And in describing various embodiments of the present invention, the same reference numerals will be used for components having the same technical characteristics.

1 is a diagram of a virtual reality education system for language training of a developmentally disabled person according to an embodiment of the present invention, and FIG. 2 is a schematic block diagram of an HMD device according to an embodiment of the present invention.

1 and 2, the virtual reality education system 10 for language training of a person with developmental disabilities according to an embodiment of the present invention may include a head mounted display (HMD) device 100 and a control device 200. have.

The HMD device 100 may be implemented as a wearable display device, for example, a head mounted display device, but is not limited thereto. The HMD device 100 may include a processor 110, a display unit 120, a storage unit 130, a sensor unit 140, and a voice sensing unit 150.

Processor 110 may perform certain calculations or tasks. For example, the processor 110 may be implemented as a processor capable of controlling an integrated circuit (IC), an application processor (AP), or a mobile AP, but is not limited thereto.

The processor 110 may control the display operation of the display unit 120 based on the user's voice. For example, when the user's voice is related to an object included in a specific area of the image, the processor 110 may control the display 120 to highlight and display the object.

In addition, the processor 110 may determine the position coordinates of the central region where the user views the VR image based on the user's motion information measured by the sensor unit 140.

The display 120 includes a plurality of pixels and may display a virtual reality (VR) image using the input image data. For example, the display 120 may be implemented as an organic light emitting display panel, a liquid crystal display panel, a plasma display panel, etc., but is not limited thereto. .

The storage unit 130 may store the VR image provided from the control device 200. According to an embodiment, the storage unit 130 may be implemented as a nonvolatile memory such as a flash memory or a volatile memory such as a dynamic RAM (DRAM), but is not limited thereto.

The sensor unit 140 may include a sensor that detects a user's head movement, such as a gyro sensor. The gyro sensor may generate and provide motion information of the x-axis, y-axis, and z-axis of the gyro sensor according to the user's movement.

The voice detector 150 may generate voice data by extracting only the user's voice from sounds generated around the HMD device 100, and analyze the voice data to obtain a voice recognition result including words spoken by the user. It can be provided to the processor 110.

The control device 200 may refer to a wireless communication device, a mobile device, and a computing device capable of transmitting or receiving data wirelessly or wired.

For example, the control device 200 includes a server, a PC, a smart phone, a tablet PC, a mobile Internet device (MID), an Internet tablet, an Internet of Things (IoT) device, an IoE ( internet of everything), a desktop computer, a laptop computer, a workstation computer, or a personal digital assistant (PDA), but is not limited thereto.

The control device 200 may convert a two-dimensional image to generate a 360° VR image, and provide a 360° VR image to the HMD device 100. The control device 200 may divide the VR image into a plurality of areas, and the control device 200 may set location coordinates in each of the plurality of areas.

Also, the control device 200 may display the same VR image as the VR image displayed on the HMD device 100 through wireless communication with the HMD device 100. The user wearing the HMD device 100 may not be able to properly provide feedback in response to an educator's guidance because the communication may be poor, such as a person with a developmental disability. Accordingly, the user of the control device 200 can determine whether the user of the HMD device 100 responds appropriately to the educator's guidance by viewing the same VR video displayed on the HMD device 100. .

Since the control device 200 can change the viewpoint of the VR image displayed by reflecting the change in viewpoint according to the user's movement of the HMD device 100, the educator checks the VR image displayed on the control device 200 to determine the HMD The change in the viewpoint of the user of the device 100 can be observed.

To this end, the control device 200 may receive the current center viewpoint of the user from the HMD device 100, that is, the coordinates of the position of the center area that the user is looking at in real time. The control device 200 may control the VR image displayed on the control device 200 such that the center area viewed by the user is located at the center of the screen displayed by the control device 200 based on the position coordinates of the center area. .

3A and 3B are diagrams for describing a 360° VR image according to an embodiment of the present invention.

Referring to FIG. 3A, the control device 200 may generate a spherical 360° VR image using a 2D image. After dividing the 360° VR image into a plurality of areas AR, the control device 200 may set position coordinates in each area AR. At this time, the size of each of the areas AR may be variously changed according to the administrator's setting.

Referring to FIG. 3B, a user of the HMD device 100 can watch a VR image displayed in 360° omnidirectional. The HMD device 100 may use the information measured by the gyro sensor to determine which area the user is looking at, that is, the central area (ARM).

The x-axis, y-axis, and z-axis information of the gyro sensor may be changed according to the user's movement, and the HMD device 100 views the VR image viewed by the user based on the changed x-axis, y-axis, and z-axis information. It is possible to determine the central region of (ARM). In addition, the HMD device 100 may provide the coordinates of the position of the central region ARM to the control device 200.

4 is a diagram for explaining a virtual reality education method for language training of a developmentally disabled person according to an embodiment of the present invention.

Referring to FIG. 4, the HMD device 100 may receive a VR image IM from a control device or play a VR image IM stored in the storage unit 130 and display it on the display unit 120. The VR image IM may include a plurality of objects for language education of the user.

The VR image includes a 360° omnidirectional display screen. The VR image in the HMD device 100 provides a virtual reality, and it is not possible to simultaneously watch images placed in all directions as in reality. The areas viewable by the user of the HMD device 100 are areas located in front of the user's eyeball, and the display area may be changed according to the user's movement. Objects may be disposed in each display area, and when the display screen is changed, the arrangement of recognizable objects is also changed.

For example, when the center viewpoint of the HMD device 100 looks at the first center area ARM, the first to third objects OB1 to OB3 may be displayed on the display screen, but the center viewpoint of the user is the first. If you look at an area other than the central area (ARM), the display screen will change and the arrangement of the objects will also change.

The HMD device 100 may store location coordinates of an area including an object in object information in order to distinguish the location of each of the objects. For example, the first object information for the first object OB1 may include position coordinates of the first areas ARG1, and the second object information for the second object OB2 is the second areas ARG2. ), and the third object information for the third object OB3 may include the position coordinates of the third areas ARG3.

Meanwhile, the control device 200 receives the position coordinates of the first central region ARM provided in real time or at a predetermined period from the HMD device 100, and is the same time point as the VR image IM displayed on the HMD device 100. Can display the VR video. The user of the control device 200 can determine whether the user of the HMD device 100 is performing language training normally by viewing the same screen as the user of the HMD device 100.

For example, when the center point of view of the HMD device 100 is looking at the first center area ARM, the HMD device 100 sends the position coordinate PC of the first center area ARM to the control device 200. Can provide. In addition, the control device 200 may display the VR image IM such that the first center region ARM is located at the center of the display screen based on the location coordinate PC.

The HMD device 100 may recognize a user's voice and highlight and display an image corresponding to the voice. Since the user of the HMD device 100 is a person with developmental disabilities in whom language use is inexperienced, the HMD device 100 may provide training for facilitating language development along with training to distinguish objects for the user.

First, the HMD device 100 may analyze the voice by detecting the user's voice. Since the pronunciation of those who are unskilled in language may be inaccurate, the speech detection unit 150 of the HMD device 100 detects candidate words and words to be recognized by using a speech recognition algorithm such as a HMM (Hidden Markov Model) algorithm. By comparing words, the most promising candidate word can be determined as the result of speech recognition. In addition, the HMD device 100 may provide the determined speech recognition result to the control device 200.

The HMD device 100 may select an object corresponding to the voice recognition result, and display the object by emphasizing it on the screen.

Specifically, the HMD device 100 may compare information included in the speech recognition result with object information corresponding to the speech recognition result, and select an object corresponding to the speech recognition result among the objects.

For example, when the text information of'mirror' is included in the voice recognition result, the HMD device 100 is the information corresponding to the'mirror' to the first to third object information OB1 to OB3 arranged on the display screen. You can determine if is included. If the information related to the'mirror' is included in the first object information, the HMD device 100 may determine that the first object OB1 corresponds to a voice recognition result.

The HMD device 100 may display an object of the corresponding area by emphasizing the object of the corresponding area based on the position coordinates of the area where the selected object is located.

According to an embodiment, the HMD device 100 may highlight the outline of the object corresponding to the speech recognition result.

For example, when it is determined that the first object OB1 corresponds to the speech recognition result, the HMD device 100 may display the first object OB1 with the brightness of the outline of the “mirror image” highlighted with bright brightness.

According to another embodiment, the HMD device 100 may increase the luminance of an area including an object corresponding to a voice recognition result to that of another area, thereby focusing an object displayed with high luminance to a user.

For example, when it is determined that the first object OB1 corresponds to the speech recognition result, the HMD device 100 increases only the luminance of the first areas ARG1 including the first object OB1, and the remaining areas The first object OB1 included in the first regions ARG1 may be emphasized by maintaining the luminance of the first region.

According to another embodiment, the HMD device 100 may reduce the luminance of an area other than the area including the object corresponding to the voice recognition result to focus the object displayed with high luminance to the user.

For example, when it is determined that the first object OB1 corresponds to the speech recognition result, the HMD device 100 reduces luminance of areas other than the first areas ARG1 including the first object OB1 and The first object OB1 included in the first regions ARG1 may be emphasized by maintaining the luminance of the first regions ARG1.

Thereafter, when the location coordinates of the user's center point of view are located in an area in which the selected object is placed for a predetermined time, the HMD device 100 may display a training completion message on the display screen or transmit a training completion voice to the outside. have.

For example, when the location coordinate of the user's center point of view is located in the first areas ARG1 in which the first object OB1 is disposed for 3 seconds or more, the HMD device 100 displays a training completion message on the display screen or The curriculum can be terminated by sending the training completion voice to the outside.

On the other hand, the control device 200 may receive the voice recognition result from the HMD device 100 and display the same object by highlighting the object displayed by the HMD device 100.

For example, when the voice recognition result received from the HMD device 100 by the control device 200 includes text information of'mirror', the control device 200 displays first to third object information arranged on the display screen. It may be determined whether information corresponding to the'mirror' is included in the fields OB1 to OB3. In addition, when the information related to the'mirror' is included in the first object OB1, the control device 200 determines that the first object OB1 corresponds to the speech recognition result and removes the first object OB1. The second and third objects OB2 and OB3 may be displayed with emphasis.

As described above, the virtual reality education system for language training for people with developmental disabilities according to an embodiment of the present invention senses the user's voice to enhance the user's language use and object recognition ability of the HMD device 100, and the object corresponding to the voice Can be intensively recognized by the user.

5 is a flowchart illustrating a virtual reality education method for language training of a developmentally disabled person according to an embodiment of the present invention.

Referring to FIG. 5, the HMD device 100 may detect a user's voice and generate a voice recognition result (S100).

In addition, the HMD device 100 may select an object corresponding to a voice recognition result from among objects included in the VR image IM displayed to the user (S110).

In addition, the HMD device 100 may highlight and display the object in order to focus the selected object on the user (S120).

In addition, the HMD device 100 may determine the location coordinate PC of the user's center point of view (S130).

In addition, when the location coordinate (PC) of the user's center point of view is located in an area in which the selected object is placed for a predetermined time, the HMD device 100 may end the language education by displaying a training completion message on the display screen. (S140).

6 is a flowchart illustrating a virtual reality education method for language training of a person with developmental disabilities according to another embodiment of the present invention.

Referring to FIG. 6, the control device 200 may receive the position coordinate PC of the center view point from the HMD device 100 (S200).

Then, the control device 200 may display the same VR image as the VR image IM at the time point displayed by the HMD device 100 based on the position coordinate PC of the center point of view (S210).

Then, the control device 200 may receive a voice recognition result from the HMD device 100 (S220).

In addition, the control device 200 may select an object corresponding to a voice recognition result from among objects included in the VR image (S230).

In addition, the control device 200 may display the selected object by highlighting the object (S240).

Then, the control device 200 may receive the training completion signal from the HMD device 100 and display the training completion message (S250).

Although the embodiments of the present invention have been described above, it is understood that those skilled in the art to which the present invention pertains can perform various modifications without departing from the claims of the present invention.

100: HMD device
110: processor
120: display unit
130: storage unit
140: sensor unit
150: voice detection unit
200: control device

Claims (13)

A control device for converting a 2D image into a 360° virtual reality (VR) image, dividing the VR image into a plurality of regions, and setting position coordinates in each divided region; And
And a HMD device that receives and displays the VR image from the control device and detects a user's voice to change the display of the VR image in response to a voice recognition result. According to claim 1,
The HMD device includes a display unit for displaying the VR image, a sensor unit for detecting motion of the user's head and generating motion information, a voice sensor unit for detecting the user's voice to generate a voice recognition result, and A virtual reality education system for language training of persons with developmental disabilities, including a processor that determines a position coordinate of a central area viewed by the user based on motion information and controls an image display of the display unit in response to the voice recognition result. According to claim 2,
The control device receives the position coordinates of the center area from the HMD device, and displays a VR image of the same viewpoint as the HMD device based on the position coordinates of the center area. According to claim 2,
The processor compares the object information of each of a plurality of objects included in the VR image with the speech recognition result, and controls the display unit so that an object corresponding to the speech recognition result is highlighted and displayed for language training of the disabled person. Virtual reality education system. According to claim 4,
The processor is a virtual reality education system for language training of persons with developmental disabilities to control the display unit such that the outline of the object corresponding to the speech recognition result is highlighted with bright luminance. According to claim 4,
The processor is a virtual reality education system for language training of persons with developmental disabilities to control the display unit such that the luminance of the region including the object corresponding to the speech recognition result is increased and displayed than the luminance of other regions. According to claim 4,
The processor is a virtual reality education system for language training of persons with developmental disabilities to control the display unit such that the luminance of an area other than the area including the object corresponding to the speech recognition result is reduced and displayed. According to claim 4,
The object information includes text information indicating the position coordinates of the object and the object, and the object information is a virtual reality education system for language training of a developmentally disabled person generated by the control device. According to claim 2,
The control device receives the voice recognition result from the HMD device, compares object information of each of a plurality of objects included in the VR image and the voice recognition result, and highlights and displays the object corresponding to the voice recognition result Virtual reality education system for language training of people with developmental disabilities. In the virtual reality training method for language training of people with developmental disabilities using a HMD device that displays a virtual reality (VR) image to a user, and a control device that converts a two-dimensional image into the VR image and provides it to the HMD device,
Generating, by the HMD device, a voice recognition result by detecting a user's voice;
Selecting, by the HMD device, an object corresponding to the voice recognition result among objects included in the VR image displayed to the user;
The HMD device highlighting and displaying the selected object to focus the selected object on the user;
Determining, by the HMD device, a position coordinate of the user's center point of view; And
The HMD device, if the location coordinates of the center point of the user is located in the area in which the selected object is placed for a predetermined time, displaying an education completion message on a display screen for language training of a person with a developmental disability. Virtual reality education method. The method of claim 10,
The control device receiving the coordinates of the position of the center point of view from the HMD device, and the control device, based on the position coordinates of the center point of view, the same VR image as the VR image at the time point displayed by the HMD device Virtual reality education method for language training of people with developmental disabilities further comprising the step of displaying. The method of claim 11,
The control device receiving the speech recognition result from the HMD device, and the control device selecting an object corresponding to the speech recognition result among objects included in the VR image, and the control device A, virtual reality education method for language training of persons with developmental disabilities further comprising the step of highlighting and displaying the selected object. The method of claim 12,
And the control device receiving the education completion signal from the HMD device and displaying the education completion message.

Images