KR102330218B1 - Virtual reality education system and method for language training of disabled person - Google Patents

Abstract

The present invention relates to a virtual reality education system for language training of persons with developmental disabilities, which converts a two-dimensional image into a 360° VR image, divides the VR image into a plurality of regions, and sets position coordinates in each of the divided regions. a control device; and an HMD device that receives and displays a VR image from the control device, detects a user's voice, and changes the display of the VR image in response to a voice recognition result.

Description

Virtual reality education system and method for language training of persons with developmental disabilities

The present invention relates to a virtual reality education system and method for language training of persons with developmental disabilities, and more particularly, to a virtual reality education system and method capable of performing visual education together with language training using virtual reality images will be.

As head mounted display (hereinafter referred to as "HMD") devices become popular, interest in virtual reality (hereinafter referred to as "VR") images is also increasing. The HMD device can provide the user with a VR image that can feel as if they are interacting with the real surrounding environment.

If a 360° VR image is used, the viewpoint changes according to the movement of the user, so that images from various viewpoints can be provided. Attempts to apply these characteristics to various fields such as games and education are actively underway.

However, users who lack cognitive ability, such as those with developmental disabilities, cannot expect an appropriate response to VR video viewing, and have difficulty in guiding them to view a video at a specific point in time. Therefore, the audiovisual education method using the VR image provided to these users has a problem in that the efficiency is lowered.

Prior Document 1: Republic of Korea Patent Publication No. 10-2018-0098936 (published on September 5, 2018) Prior Document 2: Republic of Korea Patent Publication No. 10-1696558 (published on Jan. 13, 2017)

The present invention is to solve the above-mentioned problems, by detecting the user's voice, determining which object the user is trying to recognize in the VR image, and whether the user accurately recognizes the object corresponding to the voice recognition result. The purpose of this is to provide a virtual reality education system and method for language training of people with developmental disabilities.

In order to achieve the above object, the present invention converts a two-dimensional image into a 360° virtual reality (VR) image, divides the VR image into a plurality of regions, and sets position coordinates in each of the divided regions. Virtual for language training for people with developmental disabilities, comprising: a control device; and an HMD device that receives and displays the VR image from the control device, detects a user's voice, and changes the display of the VR image in response to a voice recognition result Provides a real education system.

According to an embodiment, the HMD device may include a display unit displaying the VR image, a sensor unit generating motion information by detecting a movement of the user's head, and a voice sensing unit detecting the user's voice and generating a voice recognition result. It may include a sensor and a processor for determining the coordinates of the location of the central region viewed by the user based on the motion information, and controlling the display of the image on the display unit in response to the voice recognition result.

According to an embodiment, the control device may receive the position coordinates of the central region from the HMD device, and display a VR image of the same viewpoint as the HMD device based on the position coordinates of the central region.

According to an embodiment, the processor may compare the voice recognition result with object information of each of a plurality of objects included in the VR image, and control the display unit to highlight and display an object corresponding to the voice recognition result .

According to an embodiment, the processor may control the display unit so that an outline of an object corresponding to the voice recognition result is highlighted and displayed with a bright luminance.

According to an embodiment, the processor may control the display unit so that the luminance of the region including the object corresponding to the voice recognition result is displayed with increased luminance compared to the luminance of other regions.

According to an embodiment, the processor may control the display unit to display the reduced luminance of a region other than the region including the object corresponding to the voice recognition result.

According to an embodiment, the object information may include position coordinates of the object and text information indicating the object, and the object information may be generated by the control device.

According to an embodiment, the control device receives the voice recognition result from the HMD device, compares the voice recognition result with object information of each of a plurality of objects included in the VR image, and uses the voice recognition result corresponding to the voice recognition result. You can highlight an object to make it visible.

In addition, in order to achieve the above object, the present invention is a developmentally disabled using an HMD device that displays a virtual reality (VR) image to a user, and a control device that converts a two-dimensional image into the VR image and provides it to the HMD device A virtual reality education method for language training, the method comprising: generating, by the HMD device, a voice recognition result by detecting a user's voice; Selecting an object corresponding to the voice recognition result, the HMD apparatus highlighting and displaying the object to focus the selected object on the user; Determining the coordinates and displaying, by the HMD device, an education completion message on a display screen when the location coordinates of the user's central viewpoint are located in the area where the selected object is disposed for a preset time Provides a virtual reality education method for language training for people with developmental disabilities.

According to an embodiment, the virtual reality education method for language training of the developmentally disabled includes the steps of: receiving, by the control device, the coordinates of the position of the central viewpoint from the HMD device; The method may further include displaying the same VR image as the VR image of the viewpoint displayed by the HMD device based on the coordinates.

According to an embodiment, the virtual reality education method for language training of the developmentally disabled includes the steps of: receiving, by the control device, the voice recognition result from the HMD device; The method may further include selecting an object corresponding to the voice recognition result from among them, and displaying, by the control device, the selected object by emphasizing the object.

The method may further include, by the control device, receiving a training completion signal from the HMD device and displaying the training completion message.

According to the virtual reality education system and method for language training of persons with developmental disabilities of the present invention, it is possible to determine which object the user is trying to recognize in the VR image by detecting the user's voice, and to select the object corresponding to the voice recognition result. By highlighting the display, it is possible to help the user recognize the object.

In addition, according to the virtual reality education system and method for language training of persons with developmental disabilities of the present invention, it is possible to accurately determine whether the user effectively recognizes the object by determining the coordinates of the location of the user's central viewpoint.

1 is a schematic block diagram of a virtual reality education system for language training of persons with developmental disabilities according to an embodiment of the present invention.
2 is a schematic block diagram of an HMD device according to an embodiment of the present invention.
3A and 3B are diagrams for explaining a 360° VR image according to an embodiment of the present invention.
4 is a view for explaining a virtual reality education method for language training for persons with developmental disabilities according to an embodiment of the present invention.
5 is a flowchart illustrating a virtual reality education method for language training for persons with developmental disabilities according to an embodiment of the present invention.
6 is a flowchart illustrating a virtual reality education method for language training for persons with developmental disabilities according to another embodiment of the present invention.

Hereinafter, an embodiment of the present invention will be described in detail with reference to the accompanying drawings. However, the embodiments described below are merely for explaining in detail enough that a person of ordinary skill in the art to which the present invention pertains can easily implement the invention, thereby limiting the scope of protection of the present invention. doesn't mean And in describing various embodiments of the present invention, the same reference numerals will be used for components having the same technical characteristics.

1 is a diagram of a virtual reality education system for language training of persons with developmental disabilities according to an embodiment of the present invention, and FIG. 2 is a schematic block diagram of an HMD device according to an embodiment of the present invention.

1 and 2 , the virtual reality education system 10 for language training for the developmentally disabled according to an embodiment of the present invention may include a head mounted display (HMD) device 100 and a control device 200 . have.

The HMD device 100 may be implemented as a wearable display device, for example, a head mounted display device, but is not limited thereto. The HMD device 100 may include a processor 110 , a display unit 120 , a storage unit 130 , a sensor unit 140 , and a voice sensing unit 150 .

The processor 110 may perform certain calculations or tasks. For example, the processor 110 may be implemented as a processor capable of controlling an integrated circuit (IC), an application processor (AP), and a mobile AP, but is not limited thereto.

The processor 110 may control a display operation of the display unit 120 based on the user's voice. For example, when the user's voice is related to an object included in a specific region of the image, the processor 110 may control the display unit 120 to highlight and display the corresponding object.

Also, the processor 110 may determine the coordinates of the location of the central region where the user views the VR image based on the user's motion information measured by the sensor unit 140 .

The display unit 120 includes a plurality of pixels, and may display a virtual reality (VR) image using input image data. For example, the display unit 120 may be implemented as an organic light emitting display panel, a liquid crystal display panel, a plasma display panel, etc., but is not limited thereto. .

The storage unit 130 may store the VR image provided from the control device 200 . According to an embodiment, the storage unit 130 may be implemented as a non-volatile memory such as a flash memory or a volatile memory such as a dynamic RAM (DRAM), but is not limited thereto.

The sensor unit 140 may include a sensor that detects a user's head movement, for example, a gyro sensor. The gyro sensor may generate motion information on the x-axis, y-axis, and z-axis of the gyro sensor according to a user's movement and provide it to the processor 110 .

The voice sensor 150 may generate voice data by extracting only the user's voice from the sounds generated around the HMD device 100, and analyze the voice data to obtain a voice recognition result including the words spoken by the user. It may be provided to the processor 110 .

The control device 200 may refer to a wireless communication device, a mobile device, and a computing device capable of sending or receiving data wirelessly or by wire.

For example, the control device 200 is a server, PC, smart phone (smart phone), tablet (tablet) PC, mobile internet device (mobile internet device (MID)), Internet tablet, IoT (internet of things) device, IoE ( internet of everything) device, a desktop computer, a laptop computer, a workstation computer, or a personal digital assistant (PDA), but is not limited thereto.

The control device 200 may generate a 360° VR image by converting the 2D image, and may provide the 360° VR image to the HMD device 100 . The control device 200 may divide the VR image into a plurality of regions, and the control device 200 may set position coordinates in each of the plurality of regions.

Also, the control device 200 may display the same VR image as the VR image displayed on the HMD device 100 through wireless communication with the HMD device 100 . A user wearing the HMD device 100 may not be able to properly provide feedback in response to the instruction of the educator because it may be a person with poor communication, such as a developmentally disabled person. Accordingly, the user of the control device 200 may determine whether the user of the HMD device 100 properly responds to the instruction of the educator by viewing the same VR image as the VR image displayed on the HMD device 100 . .

Since the control device 200 may change the viewpoint of the displayed VR image by reflecting the viewpoint change according to the user's movement of the HMD device 100 , the educator checks the VR image displayed on the control device 200 to determine the HMD A change in the viewpoint of the user of the device 100 may be observed.

To this end, the control device 200 may receive from the HMD device 100 the current center point of view of the user, that is, the coordinates of the location of the center area that the user is looking at in real time. The control device 200 may control the VR image displayed on the control device 200 so that the central region viewed by the user is located at the center of the screen displayed by the control device 200 based on the position coordinates of the central region. .

3A and 3B are diagrams for explaining a 360° VR image according to an embodiment of the present invention.

Referring to FIG. 3A , the control device 200 may generate a spherical 360° VR image using a 2D image. After dividing the 360° VR image into a plurality of regions AR, the control device 200 may set position coordinates in each region AR. In this case, the size of each of the regions AR may be variously changed according to a setting of an administrator.

Referring to FIG. 3B , a user of the HMD device 100 may view a VR image displayed in 360° omnidirectional directions. The HMD device 100 may determine which area the user is looking at, ie, the center area ARM, by using information measured by the gyro sensor.

Information on the x-axis, y-axis, and z-axis of the gyro sensor may be changed according to the user's movement, and the HMD device 100 displays a VR image viewed by the user based on the changed information on the x-axis, y-axis, and z-axis. It is possible to determine the center area ARM of . In addition, the HMD device 100 may provide the location coordinates of the center region ARM to the control device 200 .

4 is a view for explaining a virtual reality education method for language training for persons with developmental disabilities according to an embodiment of the present invention.

Referring to FIG. 4 , the HMD device 100 may receive a VR image IM from the control device or reproduce the VR image IM stored in the storage unit 130 and display it on the display unit 120 . The VR image IM may include a plurality of objects for language education of a user.

VR video includes a 360° omnidirectional display screen. In the HMD device 100 , the VR image provides virtual reality, and it is not possible to simultaneously view images arranged in all directions as in reality. The user-viewable area of the HMD device 100 is the area located in front of the user's eyeball, and the display area may be changed according to the user's movement. Objects may be arranged in each display area, and when the display screen is changed, the arrangement of recognizable objects is also changed.

For example, when the central viewpoint of the HMD device 100 looks at the first central region ARM, the first to third objects OB1 to OB3 may be displayed on the display screen, but the user's central viewpoint is the first When looking at an area other than the center area ARM, the display screen is changed and the arrangement of objects is also changed.

The HMD device 100 may store the position coordinates of a region including the object in object information in order to distinguish the positions of each of the objects. For example, the first object information on the first object OB1 may include position coordinates of the first regions ARG1 , and the second object information on the second object OB2 includes the second regions ARG2 . ), and the third object information on the third object OB3 may include position coordinates of the third regions ARG3.

Meanwhile, the control device 200 receives the position coordinates of the center region ARM provided in real time or every preset period from the HMD device 100 , and the same viewpoint as the VR image IM displayed on the HMD device 100 . of VR images can be displayed. The user of the control device 200 may determine whether the user of the HMD device 100 is normally performing language training by viewing the same screen as the user of the HMD device 100 .

For example, when the central viewpoint of the HMD device 100 faces the first central region ARM, the HMD device 100 transmits the position coordinate PC of the first central region ARM to the control device 200 . can provide In addition, the control device 200 may display the VR image IM so that the first central region ARM is positioned at the exact center of the display screen based on the position coordinate PC.

The HMD device 100 may recognize a user's voice and display an image corresponding to the corresponding voice by emphasizing it. Since the user of the HMD device 100 is a person with a developmental disability in which language use is immature, the HMD device 100 may provide training for facilitating language development as well as training for classifying objects for the user.

First, the HMD device 100 may detect a user's voice and analyze the voice. Since the pronunciation of those who are inexperienced in language may be inaccurate, the voice sensing unit 150 of the HMD device 100 uses a voice recognition algorithm such as a Hidden Markov Model (HMM) algorithm to identify candidate words and detected words. By comparing the words, the most likely candidate word may be determined as the speech recognition result. In addition, the HMD device 100 may provide the determined voice recognition result to the control device 200 .

The HMD device 100 may select an object corresponding to the voice recognition result, and display the object by highlighting it on the screen.

Specifically, the HMD device 100 may compare information included in the voice recognition result with the corresponding object information to select an object corresponding to the voice recognition result from among the objects.

For example, when text information 'mirror' is included in the voice recognition result, the HMD device 100 may display information corresponding to 'mirror' in the first to third object information OB1 to OB3 disposed on the display screen. It can be determined whether or not If the first object information includes information related to 'mirror', the HMD device 100 may determine that the first object OB1 corresponds to the voice recognition result.

The HMD apparatus 100 may display the object of the corresponding area with more emphasis than the object of the other area based on the coordinates of the location of the area where the selected object is located.

According to an embodiment, the HMD device 100 may highlight and display an outline of an object corresponding to the voice recognition result.

For example, when it is determined that the first object OB1 corresponds to the voice recognition result, the HMD device 100 may highlight and display the outline of the 'mirror image' that is the first object OB1 with bright luminance.

According to another embodiment, the HMD apparatus 100 may increase the luminance of a region including an object corresponding to the voice recognition result from that of other regions to focus an object displayed with a high luminance on the user.

For example, when it is determined that the first object OB1 corresponds to the voice recognition result, the HMD apparatus 100 increases only the luminance of the first areas ARG1 including the first object OB1, and the remaining area The first object OB1 included in the first regions ARG1 may be emphasized by maintaining the luminance.

According to another embodiment, the HMD apparatus 100 may reduce the luminance of a region other than the region including the object corresponding to the voice recognition result to focus the object displayed with high luminance on the user.

For example, when it is determined that the first object OB1 corresponds to the voice recognition result, the HMD apparatus 100 reduces the luminance of regions other than the first regions ARG1 including the first object OB1 and , the first object OB1 included in the first regions ARG1 may be emphasized by maintaining the luminance of the first regions ARG1 .

Thereafter, when the location coordinates of the user's central viewpoint are located in the area where the selected object is disposed for a preset time, the HMD device 100 may display a training completion message on the display screen or transmit a training completion voice to the outside. have.

For example, when the position coordinates of the user's central viewpoint are located on the first regions ARG1 in which the first object OB1 is disposed for 3 seconds or longer, the HMD device 100 displays an education completion message on the display screen or The training course can be ended by transmitting the training completion voice to the outside.

Meanwhile, the control device 200 may receive the voice recognition result from the HMD device 100 and display the object highlighted and displayed by the HMD device 100 with the same emphasis.

For example, when the text information 'mirror' is included in the voice recognition result received by the control device 200 from the HMD device 100, the control device 200 provides information on the first to third objects disposed on the display screen. It may be determined whether information corresponding to the 'mirror' is included in the fields OB1 to OB3. And when the first object OB1 includes information related to 'mirror', the control device 200 determines that the first object OB1 corresponds to the voice recognition result and removes the first object OB1. The second and third objects OB2 and OB3 may be displayed with greater emphasis.

As such, the virtual reality education system for language training for the developmentally disabled according to an embodiment of the present invention detects the user's voice in order to strengthen the user's language use and object recognition ability of the HMD device 100, and an object corresponding to the voice can be focused on the user.

5 is a flowchart illustrating a virtual reality education method for language training for persons with developmental disabilities according to an embodiment of the present invention.

Referring to FIG. 5 , the HMD device 100 may detect the user's voice and generate a voice recognition result (S100).

In addition, the HMD device 100 may select an object corresponding to the voice recognition result from among the objects included in the VR image IM displayed to the user (S110).

In addition, the HMD device 100 may highlight and display the selected object in order to focus the selected object on the user ( S120 ).

In addition, the HMD device 100 may determine the location coordinates (PC) of the user's central viewpoint ( S130 ).

And when the location coordinates (PC) of the user's central viewpoint are located in the area where the selected object is disposed for a preset time, the HMD device 100 may display a training completion message on the display screen to end the language training. (S140).

6 is a flowchart illustrating a virtual reality education method for language training for persons with developmental disabilities according to another embodiment of the present invention.

Referring to FIG. 6 , the control device 200 may receive the location coordinates PC of the central viewpoint from the HMD device 100 ( S200 ).

In addition, the control device 200 may display the same VR image as the VR image IM of the viewpoint displayed by the HMD device 100 based on the position coordinate PC of the central viewpoint ( S210 ).

And the control device 200 may receive a voice recognition result from the HMD device 100 (S220).

In addition, the control device 200 may select an object corresponding to the voice recognition result from among the objects included in the VR image (S230).

In addition, the control device 200 may display the selected object by emphasizing the object ( S240 ).

In addition, the control device 200 may receive the training completion signal from the HMD device 100 and display the training completion message (S250).

Although the embodiments of the present invention have been described above, it is understood that those of ordinary skill in the art to which the present invention pertains may make various modifications without departing from the scope of the claims of the present invention.

100: HMD device
110: processor
120: display unit
130: storage
140: sensor unit
150: voice detection unit
200: control device

Claims (1)

An HMD device that is mounted on a person with developmental disabilities to display a virtual reality (VR) image, converts a two-dimensional image into the VR image and provides it to the HMD device. In a virtual reality education method for language training of persons with developmental disabilities using a control device that changes the central point of view of an image,
generating, by the control device, a spherical 360° VR image using the two-dimensional image;
dividing, by the control device, the VR image into a plurality of regions and setting position coordinates in each of the plurality of regions;
displaying, by the HMD device, the VR image to the developmentally disabled;
generating, by the HMD device, a voice recognition result by detecting an educator's voice;
selecting, by the HMD device, an object corresponding to the voice recognition result from among the objects in the VR image;
highlighting and displaying, by the HMD device, the selected object to focus the selected object on the developmentally disabled;
The HMD device determines the position coordinates of the central viewpoint in the VR image viewed by the developmentally disabled based on the x-axis, y-axis, and z-axis information that changes according to the movement of the developmentally disabled measured by the gyro sensor to do;
receiving, by the control device, the coordinates of the position of the central viewpoint from the HMD device;
displaying, by the control device, a VR image of the same viewpoint as the HMD device based on the location coordinates of the central viewpoint in order to recognize the change in viewpoint of the developmentally disabled to the educator;
The HMD device displays an education completion message on the display screen when the position coordinates of the central viewpoint are located at the position coordinates of the area in which the selected object is disposed for a preset time using the object information and the position coordinates of the central viewpoint to mark on; and
and receiving, by the control device, a training completion signal from the HMD device and displaying the training completion message,
receiving, by the control device, the voice recognition result from the HMD device; selecting, by the control device, an object corresponding to the voice recognition result from among the objects included in the VR image; A, further comprising the step of displaying the selected object by highlighting the object,
The HMD device stores the location coordinates of the area including the object in the object information in order to distinguish the location of each of the objects in the VR image.

Images