WO2021162207A1

WO2021162207A1 - Virtual-reality system and method for rehabilitating exotropia patients on basis of artificial intelligence, and computer-readable medium

Info

Publication number: WO2021162207A1
Application number: PCT/KR2020/016128
Authority: WO
Inventors: 오석희; 양희경; 황정민; 황보택근; 김제현
Original assignee: 가천대학교 산학협력단; 서울대학교산학협력단
Priority date: 2020-02-11
Filing date: 2020-11-17
Publication date: 2021-08-19
Also published as: KR102120112B1

Abstract

The present invention relates to a virtual-reality system and method for rehabilitating exotropia patients on basis of artificial intelligence, and a computer-readable medium, and, more specifically, to a virtual-reality system and method for rehabilitating exotropia patients on basis of artificial intelligence, and a computer-readable medium, the system and the method providing virtual-reality content that enables exotropia patients to train for eye convergence, so as alleviate the boredom of repetitive training for the exotropia patients, and collecting training progress data, so as to provide strabismus diagnosis information about exotropia patients who have trained with the virtual-reality content.

Description

Artificial intelligence-based virtual reality system, method and computer-readable medium for rehabilitation of exotropia patients

The present invention relates to a virtual reality system, method, and computer-readable medium for rehabilitation training for exotropia patients based on artificial intelligence. Artificial intelligence-based exotropia rehabilitation virtual reality system, method, and computer- which can improve boredom for repetitive training and provide strabismus diagnosis information for exotropia patients who performed virtual reality contents by collecting training progress data It relates to a readable medium.

Virtual reality is a technology that enables interaction between a user and a three-dimensional virtual space created by a computer system. It is a convergence technology that provides a sense of reality as if it were felt and actually existed in the space. In the global virtual reality market, head mounted displays occupy most of the market, and various types of virtual reality devices are being distributed recently. The influence of platforms that distribute virtual reality contents and contents is expected to expand.

On the other hand, since the head mounted display is hardware based on the user's vision, it is expected that vision therapy can be utilized in various healthcare fields. Vision therapy, also called vision training, is a clinical approach that corrects eye movement disorders, abnormalities of binocular function such as amblyopia, and control disorders such as strabismus and improves related symptoms. This includes various methods of improving visual function through visual training in a non-surgical way. Existing patients with amblyopia and strabismus have been undergoing rehabilitation training such as traditional eye collection training for treatment.

In order to solve this problem, we provide virtual reality content for eyesight training that users can experience anytime and anywhere through virtual reality experience that can be provided without restrictions in space and time with only a simple device, and users can feel interest By doing so, research for the purpose of improving visual function has been progressed.

The present invention improves the boredom of repetitive training of exotropia patients by providing virtual reality contents for eye collection training to exotropia patients, and collects training progress data to perform virtual reality contents for strabismus patients. An object of the present invention is to provide a virtual reality system, method and computer-readable medium for rehabilitation training for exotropia patients based on artificial intelligence capable of providing diagnostic information.

In order to solve the above problems, as an exotropia rehabilitation system including an HMD module and a service server, the virtual reality content can be executed in the HMD module, and the virtual reality content is virtual according to the user's controller operation. a triggering step of moving the first object in a preset area of the central part of the real screen from the starting position to the user's side; A targeting step of moving the virtual reality screen according to the direction manipulation of the HMD module according to the movement of the user's head; a release step of emitting the first object from a virtual reality screen according to a user's controller manipulation; and a score calculation step of calculating a score by determining whether the first object fired in the release step is in contact with one or more second objects existing in the virtual reality screen; including a HMD module and a service server Provides a rehabilitation training system for exotropia patients.

In an embodiment of the present invention, the one or more second objects may be set to have respective preset sizes, and the coordinates of the second objects may have respective distances from the user's coordinates.

In one embodiment of the present invention, the HMD module collects the coordinate information of the user's gaze and the pupil position information on the virtual reality screen, and the triggering step includes: A release level of the first object is derived based on the coordinate information of the gaze and the user's pupil position information, and the release step includes the first object in the virtual reality screen based on the release level derived in the trigger step. You can determine the firing intensity or firing distance of

In one embodiment of the present invention, the HMD module collects the user's pupil position information on the virtual reality screen, and the triggering step is, when the user's pupil position is out of a preset range, moves to the user's side The position of the first object may be reset to the start position.

In one embodiment of the present invention, the HMD module collects the coordinate information of the user's gaze on the virtual reality screen, and the releasing step is performed on the coordinate information of the user's left eye gaze and the right eye gaze coordinate information. A release area may be derived based on a preset criterion, and the first object may be launched within a range of the derived release area.

In an embodiment of the present invention, the virtual reality content generates a gaze heat map based on coordinate information of the user's gaze in the trigger step, targeting step, and release step, and uses the generated gaze heat map to the service server The strabismus diagnosis step of transmitting the strabismus diagnosis step to; further comprising, the service server may derive the strabismus diagnosis information for the gaze heat map received by the diagnosis model learned from the learning gaze heat map data.

In order to solve the above problems, as a rehabilitation training method for an exotropia patient using a rehabilitation training system for an exotropia patient including an HMD module and a service server, by the HMD module, the central portion of the virtual reality screen according to the user's controller operation a trigger step of moving the first object in the set area from the start position to the user side; A targeting step of moving the virtual reality screen according to the direction manipulation of the HMD module according to the movement of the user's head by the HMD module; a release step of emitting the first object from the virtual reality screen according to the user's controller manipulation by the HMD module; And by the HMD module, a score calculation step of calculating a score by determining whether the first object fired in the release step is in contact with one or more second objects existing in the virtual reality screen; provide training methods.

In order to solve the above problems, an embodiment of the present invention is a computer-readable medium for implementing a method for rehabilitation of an exotropia patient using an exotropia rehabilitation system including an HMD module and a service server, wherein the The computer-readable medium stores instructions for causing the components of the exotropia rehabilitation system to perform the following steps, wherein the steps are: by the HMD module, the center of the virtual reality screen according to the user's controller operation a trigger step of moving a first object in a preset area of a part from a start position to a user side; A targeting step of moving the virtual reality screen according to the direction manipulation of the HMD module according to the movement of the user's head by the HMD module; a release step of emitting the first object from the virtual reality screen according to the user's controller manipulation by the HMD module; and a score calculation step of calculating a score by determining whether the first object fired in the release step is in contact with one or more second objects existing in the virtual reality screen by the HMD module; Provide available media.

According to an embodiment of the present invention, by providing virtual reality content for eye collection training, it is possible to reduce boredom caused by repetitive training of exotropia patients and to be more immersed in rehabilitation training.

According to an embodiment of the present invention, the difficulty of the game is adjusted by varying the release level of the first object implemented in the game according to the degree of the user's eye collection, thereby providing an effect of providing customized virtual reality content. can perform

According to an embodiment of the present invention, the user can check his/her eye consolidation level through a virtual reality screen on which information related to the coordinate information of his/her gaze is displayed, and perform a game and perform eye converging training can perform

According to an embodiment of the present invention, by analyzing the content performance information of the user who has executed the virtual reality content based on artificial intelligence, it is possible to exhibit the effect of deriving strabismus diagnosis information on the degree of strabismus.

According to an embodiment of the present invention, the strabismus diagnosis information of the user who executed the virtual reality content can be provided to the user, the user's guardian or a specialist, and the strabismus diagnosis information can be used as the user's treatment, treatment and consultation data. can perform

1 schematically shows the overall form of a rehabilitation training system according to an embodiment of the present invention.

2 schematically shows a state in which a user using the rehabilitation training system according to an embodiment of the present invention wears an HMD module.

3 schematically illustrates a step of performing virtual reality content according to an embodiment of the present invention.

4 schematically shows a display screen in the HMD module provided in the trigger stage, the targeting stage and the release stage, and the score calculation stage according to an embodiment of the present invention.

5 schematically shows a display screen in the HMD module provided in the trigger stage according to an embodiment of the present invention.

6 schematically shows a display screen in the HMD module provided in the release step according to an embodiment of the present invention.

7 schematically shows a display screen in the HMD module provided in the targeting step, the release step, and the score calculation step according to an embodiment of the present invention.

7 schematically shows a display screen in the HMD module provided in the release step according to an embodiment of the present invention.

8 schematically shows a display screen in the HMD module provided in the release step according to an embodiment of the present invention.

9 schematically shows an internal configuration of a service server according to an embodiment of the present invention.

10 schematically illustrates a gaze heat map generated based on coordinate information of a user's gaze according to an embodiment of the present invention.

11 schematically shows the execution steps of the HMD module and the service server according to an embodiment of the present invention.

12 schematically shows the operation of the diagnostic model learning unit of the service server according to an embodiment of the present invention.

13 exemplarily shows a computing device according to an embodiment of the present invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those of ordinary skill in the art can easily implement them. However, the present invention may be embodied in many different forms and is not limited to the embodiments described herein. And in order to clearly explain the present invention in the drawings, parts irrelevant to the description are omitted, and similar reference numerals are attached to similar parts throughout the specification.

Throughout the specification, when a part is "connected" with another part, this includes not only the case of being "directly connected" but also the case of being "electrically connected" with another element interposed therebetween. . In addition, when a part "includes" a certain component, this means that other components may be further included rather than excluding other components unless otherwise stated.

Also, terms including an ordinal number, such as first, second, etc., may be used to describe various elements, but the elements are not limited by the terms. The above terms are used only for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, a first component may be referred to as a second component, and similarly, a second component may also be referred to as a first component. and/or includes a combination of a plurality of related listed items or any of a plurality of related listed items.

In this specification, a "part" includes a unit realized by hardware, a unit realized by software, and a unit realized using both. In addition, one unit may be implemented using two or more hardware, and two or more units may be implemented by one hardware. Meanwhile, '~ unit' is not limited to software or hardware, and '~ unit' may be configured to be in an addressable storage medium or may be configured to reproduce one or more processors. Thus, as an example, '~' denotes components such as software components, object-oriented software components, class components, and task components, and processes, functions, properties, and procedures. , subroutines, segments of program code, drivers, firmware, microcode, circuitry, data, databases, data structures, tables, arrays and variables. The functions provided in the components and '~ units' may be combined into a smaller number of components and '~ units' or further separated into additional components and '~ units'. In addition, components and '~ units' may be implemented to play one or more CPUs in a device or secure multimedia card.

The "user terminal" referred to below may be implemented as a computer or portable terminal that can access a server or other terminal through a network. Here, the computer includes, for example, a laptop, a desktop, and a laptop equipped with a web browser (WEB Browser), and the portable terminal is, for example, a wireless communication device that ensures portability and mobility. , PCS (Personal Communication System), GSM (Global System for Mobile communications), PDC (Personal Digital Cellular), PHS (Personal Handyphone System), PDA (Personal Digital Assistant), IMT (International Mobile Telecommunication)-2000, CDMA (Code) Division Multiple Access)-2000, W-Code Division Multiple Access (W-CDMA), Wireless Broadband Internet (Wibro) terminals, and the like may include all types of handheld-based wireless communication devices. In addition, "network" refers to a wired network such as a local area network (LAN), a wide area network (WAN), or a value added network (VAN), or a mobile radio communication network or satellite. It may be implemented as any kind of wireless network, such as a communication network.

The rehabilitation training system according to an embodiment of the present invention includes an HMD module 1000 and a service server 2000 . The service server 2000 and the HMD module 1000 correspond to a computing device including one or more processors and one or more memories, and a user can perform an eye collection training game provided through the rehabilitation training system of the present invention. And, through such performance in a virtual space, it is possible to maximize the training continuity of exotropia patients by alleviating symptoms and generating interest through game elements. The service server 2000 may analyze the patient's training data collected through the HMD module 1000 through the learned diagnosis model and transmit the analyzed strabismus diagnosis information to the outside. The HMD module 1000 and the service server 2000 may communicate through a network. In addition, the HMD module 1000 may receive a user's input and operation through a controller, and the received user's input and operation are transmitted to the HMD module 1000 and reflected in the virtual reality content.

The HMD module 1000 includes a display unit 1100 , a speaker unit 1200 , an eye tracker unit 1300 , a content execution unit 1400 , and a strabismus diagnosis unit 1500 .

Specifically, the display unit 1100 displays a display screen provided to a user wearing the HMD module 1000 . Rehabilitation training virtual reality system of the present invention, including the trigger step (S1000), the targeting step (S1100), the release step (S1200) and the score calculation step (S1300) according to the user's controller operation, the first object (O1) Provides virtual reality content to be launched to any one of one or more second objects O2, and the display unit 1100 displays a virtual reality screen displayed in providing such virtual reality content.

The speaker unit 1200 may provide sound information to a user by outputting sound information of the virtual reality content.

The eye tracker unit 1300 may detect the eye movement of the user wearing the HMD module 1000, and collects coordinate information of the user's gaze and pupil position information from the sensed eye movement. The present invention is a rehabilitation training system for exotropia patients, and implements a content that fires a first object (O1) to any one of one or more second objects (O2), and the eye tracker unit 1300 is a virtual reality content It detects the user's eye movement and collects the coordinate information of the user's gaze and the pupil position information while the virtual reality content is running. Thereafter, the collected information related to the coordinate information of the gaze and the pupil position information may be utilized to derive the user's strabismus diagnosis information.

The content execution unit 1400 may execute virtual reality content through the HMD module 1000, receive user input and motion through a controller in the provided virtual reality screen, and execute a game for eye collection training. let it be In one embodiment of the present invention, the content execution unit 1400 virtual reality including a trigger step (S1000), a targeting step, a release step (S1200), a score calculation step (S1300), and a strabismus diagnosis step (S1400) Run the content.

The strabismus diagnosis unit 1500 generates a gaze heat map based on the coordinate information of the user's gaze in the trigger step (S1000), the targeting step (S1100), and the release step (S1200) of the virtual reality content. The generated gaze heat map is transmitted to the service server 2000 .

Meanwhile, the service server 2000 receives the gaze heat map generated by the HMD module 1000 and derives strabismus diagnosis information for the received gaze heat map by the learned diagnosis model. Preferably, the service server 2000 includes a strabismus diagnosis information derivation unit 2100 and a diagnosis model learning unit 2200 .

The rehabilitation training system including the HMD module 1000 and the service server 2000 shown in FIG. 1 may further include elements other than the illustrated components, but for convenience, the rehabilitation training system according to embodiments of the present invention and Only relevant components are shown.

2 schematically shows a state in which a user using the rehabilitation training system according to an embodiment of the present invention wears the HMD module 1000 .

As shown in FIG. 2 , an embodiment of the HMD module 1000 worn by the user is shown. In an embodiment of the present invention, a virtual reality device that provides virtual reality content for rehabilitation to a user may include an HMD module 1000 and a controller, and the user may use the HMD module 1000 as shown in FIG. 2 . Wearing and holding a controller, the user's motion and input can be received by the HMD module 1000 through the controller, and image information and sound information in virtual reality can be provided to the user through the HMD module 1000 have.

3 schematically shows the execution steps of the content execution unit 1400 according to an embodiment of the present invention, and FIG. 4 is a trigger step (S1000), a targeting step (S1100) and a release according to an embodiment of the present invention. A display screen in the HMD module 1000 provided in the step S1200 and the score calculation step S1300 is schematically shown.

The present invention is an exotropia patient rehabilitation system including an HMD module 1000 and a service server 2000. In the HMD module 1000, the virtual reality content can be executed by the content execution unit 1400, and the virtual The real content includes: a trigger step of moving the first object O1 in a preset area of the central portion of the virtual reality screen from the start position to the user side according to the user's controller manipulation (S1000); A targeting step of moving the virtual reality screen according to the direction manipulation of the HMD module 1000 according to the movement of the user's head (S1100); A release step (S1200) of firing the first object (O1) on the virtual reality screen according to the user's controller manipulation; A score calculation step (S1300) of calculating a score by determining whether the first object (O1) fired in the release step (S1200) is in contact with one or more second objects (O2) existing in the virtual reality screen (S1300); And generating a gaze heat map based on the coordinate information of the user's gaze in the trigger step (S1000), the targeting step (S1100), and the release step (S1200), and the generated gaze heat map to the service server (2000) It includes a strabismus diagnosis step (S1400) of transmitting.

Specifically, in the trigger step S1000, the first object O1 in the preset area of the central part of the virtual reality screen is moved from the start position to the user's side according to the user's manipulation of the controller. As shown in FIG. 4A , a first object O1 and one or more second objects O2 are displayed. Preferably, the one or more second objects O2 are set to have respective preset sizes, and the coordinates of the second objects O2 have respective distances from the user's coordinates. As shown in (a) of FIG. 4 , the second objects O2 have respective distances from the user's coordinates, and are displayed in a form having a predetermined size and shape, respectively. The user provided with such a virtual reality screen operates the controller, and according to the user's controller manipulation, the first object O1 in the preset area of the central part of the virtual reality screen is moved from the preset start position to the user's side. As shown in (b) and (c) of FIG. 4 , the coordinates of the first object O1 are moved in such a way that the distance from the coordinates of the user becomes closer and closer to the user side. In this way, in the trigger step S1000, the first object O1 in the preset area of the central part of the virtual reality screen is moved from the start position to the user's side according to the user's manipulation of the controller. The user can naturally perform eye collection training by focusing on the moving first object O1.

In the targeting step (S1100), the virtual reality screen moves according to the direction manipulation of the HMD module 1000 according to the movement of the user's head. After the first object O1 is all moved toward the user in the trigger step S1000, the user looks at the second object O2, which is the target to fire the first object O1, and the user's head According to the direction manipulation of the HMD module 1000 according to the movement, the virtual reality screen moves as shown in (d) of FIG. 4 .

In the release step (S1200), the first object O1 is launched from the virtual reality screen according to the user's manipulation of the controller. When the user performs a controller operation for firing the first object O1 on the virtual reality screen moved in the targeting steps (S1100) (S1300), as shown in (e) of FIG. 4, in the virtual reality screen The first object O1 is launched, and the launched first object O1 comes into contact with the target second object O2.

In the score calculation step (S1300), the score is calculated by determining whether the first object (O1) fired in the release step (S1200) comes into contact with one or more second objects (O2) existing in the virtual reality screen. . As shown in Figs. 4 (d) and 4 (e), the first object O1 released by the user is fired and contacted by the target second object O2, and the scoring step (S1300) The score is calculated by determining whether the fired first object O1 is in contact with the second object O2.

On the other hand, in the strabismus diagnosis step (S1400), a gaze heat map is generated based on the coordinate information of the user's gaze in the trigger step (S1000), the targeting step (S1100) and the release step (S1200), and the generated gaze heat map is transmitted to the service server 2000 . The eye tracker unit 1300 of the HMD module 1000 collects the coordinate information of the user's gaze in the trigger step (S1000), the targeting step (S1100) and the release step (S1200) in real time, The strabismus diagnosis unit 1500 of the HMD module 1000 generates a gaze heat map based on the coordinate information of the user's gaze collected from the eye tracker unit 1300 . In the gaze heat map, the user's gaze in the virtual reality screen is virtual based on the user's gaze information that changes according to the movement of the first object O1 as the user performs the virtual reality content, or according to the movement of the user's head. It is information that visually displays information about the length of time spent on the real screen. In the strabismus diagnosis step S1400 , such a gaze heat map may be generated, and the generated gaze heat map may be transmitted to the service server 2000 .

In this way, by providing a virtual reality screen, which is an environment in which the user can gather their eyes, through the first object O1 that the user moves, the rehabilitation training of the exotropia patient can be performed, and the result of firing the object By providing the content from which the score is derived according to the content, it is possible to exert the effect of improving the user's content immersion.

5 schematically shows a display screen in the HMD module 1000 provided in the trigger step S1000 according to an embodiment of the present invention.

Specifically, FIG. 5 shows a display screen in the HMD module 1000 provided in the above-described trigger step (S1000). The display screen as shown in FIG. 5 is displayed by the display unit 1100 of the HMD module 1000 . As shown in (a) of FIG. 5 , the first object O1 is at the start position of the first object O1 in a preset area in the center of the virtual reality screen displayed in the HMD module 1000 . is displayed. In addition, it is shown that one or more second objects O2 set to have respective preset sizes are displayed, and the coordinates of one or more second objects O2 are displayed to have respective distances from the user's coordinates. Then, in the trigger step (S1000), as shown in FIGS. 5 (b) and 5 (c), the position of the first object O1 is moved from the starting position to the user's position. The one or more second objects O2 are not moved, and only the first objects O1 are moved toward the user according to the user's manipulation of the controller.

On the other hand, the HMD module 1000 collects the user's pupil position information on the virtual reality screen, and in the triggering step (S1000), when the user's pupil position is out of a preset range, it moves to the user side. The position of the first object O1 is reset to the starting position. The eye tracker unit 1300 of the HMD module 1000 collects the user's pupil position information in the trigger step S1000, and the trigger step ( In S1000), based on the collected pupil position information, it is determined whether the pupil position of the user meets a preset criterion. For example, when it is determined that the angle of deviation according to the user's pupil position is out of a preset range due to the difficulty of collecting the eyes, or when strabismus develops and the angle of deviation is greater than or equal to a preset standard, the angle of deviation is set When the user's pupil position does not meet the preset criteria, such as when the time out of the standard is greater than or equal to the preset time, etc., the position of the first object O1 moved toward the user reset to the starting position. When the pupil position of the user provided with the screen as shown in (c) of FIG. 5 does not meet the preset criteria, the position of the first object O1 is reset to the starting position and the virtual reality screen provided to the user is shown in FIG. 5 A screen as shown in (a) of FIG. 5 may be provided again instead of the screen as shown in (d) of FIG.

In addition, in the triggering step (S1000), the release level of the first object O1 is derived based on the coordinate information of the user's gaze and the user's pupil position information while the first object O1 is moving. . The eye tracker unit 1300 of the HMD module 1000 collects the coordinate information of the gaze and the pupil position information in the trigger step S1000, and the content execution unit 1400 of the HMD module 1000, the trigger step The release level of the first object O1 may be derived according to a preset criterion based on the coordinate information of the user's gaze in S1000 and the pupil position information. For example, whether the angle of view according to the user's pupil position is within a preset range, whether the movement information of the coordinate information of the user's gaze coincides with the movement information of the first object O1, etc. Accordingly, it is possible to derive a release level that is a reference for emitting the first object O1 in the release step (S1200).

The virtual reality content executed in this way moves the first object O1 displayed on the virtual reality screen from a preset start position to the user side, so that the user can focus while looking at the first object O1. It can exert the effect that patients with disabilities can do eye collection training with more interest, and while the user experiences the virtual reality content, the coordinate information of the user's gaze and the pupil position information are collected and reflected in the virtual reality content By doing so, it is possible to exert the effect of improving the user's immersion level.

The main purpose of the rehabilitation training system of the present invention is to assist in training the eyes of a user with an exotropia disorder in which the angle of deviation according to eye movement, such as exotropia or intermittent exotropia, is shifted to the outside. In this case, the coordinate information of the gaze collected by the eye tracker of the HMD module 1000 may be displayed differently according to each angle of view of the exotropia patients.

6 shows a display screen in the HMD module 1000 in which the release area derived in the targeting step (S1100) is different according to each different collected pupil position information. The display screen provided by the user's HMD module 1000 in a state in which the oblique angle is not shifted is shown. In this way, the content execution unit 1400 of the HMD module 1000, in the targeting step (S1100), coordinate information of the gaze of the left eye (Left point in (a) of FIG. 6) and coordinate information of the gaze of the right eye (FIG. 6) A release area can be derived according to a preset criterion based on the Right point of (a) of (a)).

6 (b) shows that the coordinate information of the gaze on the virtual reality screen is displayed differently according to the deviation angle of the right eye. In (b) of FIG. 6, the user's right eye looking at the virtual reality screen has an outward oblique angle, and therefore, coordinate information of the right eye's gaze in the virtual reality screen (Right point in FIG. 6 (a)) It is shown that is displayed at a farther distance from the first object O1 than the coordinate information of the left eye's gaze (Left point in FIG. 6(a)). 6(c) also shows that the coordinate information of the gaze in the virtual reality screen is displayed differently according to the distorted perspective angle of the left eye, as opposed to FIG. 6(b). In this way, the coordinate information of the gaze of the left eye and the gaze of the right eye may be displayed differently on the virtual reality screen, and the release area is derived according to a preset criterion based on the coordinate information of the gaze of the left eye and the coordinate information of the gaze of the right eye. According to the coordinate information of the gaze, it is shown that the release area derived in FIGS. 6 (b) and 6 (c) has a wider range than the release area in FIG. 6 (a).

Thereafter, the first object O1 is emitted within the range of the release area derived according to the user's manipulation of the controller. The first object 01 may be randomly launched within the range of the release area. Therefore, the user focuses his/her gaze on the first object O1 and aligns the two eyes symmetrically so that the coordinate information of the gaze on the first object O1 does not deviate from the area of the first object O1. As the eyes are aligned, the release area may be set within a range that does not deviate from the area of the first object O1. Thereafter, in the release step (S1200), the first object O1 is randomly fired within the range of the release area, and the smaller the range of the release area is from the area with the first object O1, the more accurately it is. The first object O1 may be launched toward the second object O2.

Preferably, the release area may be displayed for a preset time according to a preset criterion based on an average value of the user's pupil position while the first object O1 moves in the triggering step S1000. In one embodiment of the present invention, when the user's pupil position is symmetrically aligned while the first object moves in the trigger step (S1000) and is maintained without deviation of the oblique angle, the release area is maintained for a long time. When the user's pupil position is changed at an oblique angle while the first object O1 is moving and the pupil position is changed, the release region may be displayed for a shorter time according to a preset criterion. In addition, the first object O1 in the trigger step ( S1000 ) may be displayed for a preset time according to a preset criterion based on an average value of coordinate information of the user's gaze while moving.

7 schematically shows a display screen in the HMD module 1000 provided in the targeting step (S1100), the release step (S1200), and the score calculation step (S1300) according to an embodiment of the present invention.

On the other hand, when the first object O1 moves to the final position preset by the user, a command for an operation to be performed by the user may be displayed on the virtual reality screen as shown in FIG. 5D . . Alternatively, a voice for an operation to be performed may be output and transmitted through the speaker unit 1200 of the HMD module 1000 . The user, who has received the command for the operation to be performed on the virtual reality screen, looks around to perform the command, and in the targeting step (S1100), the direction manipulation in the HMD module 1000 is performed according to the movement of the user's head, and , the virtual reality screen is moved as shown in (a) of FIG. 7 according to the direction manipulation of the HMD module 1000 . After the virtual reality screen is moved, in the virtual reality content, a release step (S1200) of emitting the first object O1 from the virtual reality screen according to the user's controller operation is performed. The user manipulates the controller (eg, pushes a button) to fire the first object O1, and according to the user's input, the first object O1 is As shown, the second object O2 is emitted in the displayed direction so that the first object O1 and the second object O2 come into contact with each other.

On the other hand, in the release step (S1200), based on the release level derived in the trigger step (S1000), the firing intensity or firing distance of the first object O1 in the virtual reality screen is determined. In the above-described trigger step S1000, the release level of the first object O1 is derived based on the coordinate information of the user's gaze and the user's pupil position information while the first object O1 is moving. In the trigger step (S1000), a release level is derived according to a preset criterion, and in the release step (S1200), the first object in the virtual reality screen is based on the release level derived in the trigger step (S1000). Determines the firing intensity or firing distance of (O1). In an embodiment of the present invention, the firing intensity or firing distance may be derived based on a release level derived based on a preset criterion as shown in Table 1 below.

릴리즈레벨release level	발사거리firing range	발사세기launch century
1One	10m10m	약approximately
22	15m 15m		약중medicine
33	20m20m	중middle
44	25m25m	중강Zhonggang
55	30m30m	강River

In such an embodiment, whether the angle of view according to the pupil position of the user is within a preset range, whether the movement information of the coordinate information of the user's gaze coincides with the movement information of the first object O1, etc. A release level may be derived according to a preset criterion for deriving a level. When the angle of view according to the pupil position of the user is within a preset range, the pupil position of the user using the virtual reality content, such as when the movement information of the coordinate information of the user's gaze coincides with the movement information of the first object O1 It is determined that exotropia is not developed and is maintained, and it is determined that the user concentrates on the virtual reality content and the gaze according to the movement of the first object O1 stays along the first object O1, the release level is shown in Table 1 above. It can be derived with a large value of the release level. As described above, as the line of sight according to the movement of the first object O1 is maintained, the firing intensity and firing distance from which a better score can be derived can be determined.

Thereafter, the content execution unit 1400 of the HMD module 1000 performs a score calculation step (S1300) of calculating a score by determining whether the first object O1 and the second object O2 are in contact. As shown in (b) of FIG. 7 , when the first object O1 fired according to the user's manipulation comes into contact with the target second object O2, the content execution unit 1400 of the HMD module 1000 is , a contact is determined to calculate a score, and the calculated score may be reflected in real time while the virtual reality content is being executed and displayed on the virtual reality screen as shown in (c) of FIG. 7 .

In this way, the virtual reality content collects the coordinate information of the user's gaze and the pupil position information while the user experiences the virtual reality content and reflects it in the virtual reality content, thereby improving the user's immersion. can perform

8 schematically shows a display screen in the HMD module 1000 provided in the release step S1200 according to an embodiment of the present invention.

Specifically, FIGS. 8 (a) and 8 (b) show a display screen of the HMD module 1000 in which a score calculated by performing the score calculation step S1300 is displayed. As shown in FIG. 8(a), the first object O1 of FIGS. 8(a) and 8(b) is the second object O1 that is targeted according to the user's manipulation of the controller. It is shown being fired and coming into contact with the object O2. Referring to each of these screens, it is shown that each of the first objects O1 is launched and touched the same second object O2, but each score is calculated and displayed differently. As such, even if the second object O2 from which the first object O1 is launched is the same, the score calculated by the score calculation step S1300 may be different. Preferably, in the second object O2, the score calculated in the score calculation step S1300 is set differently according to a preset criterion for the area in contact with the first object O1. FIG. 8(c) shows the second object O2 displayed in FIGS. 8(a) and (b). As shown in (c) of FIG. 8, the second object O2 displayed on the virtual reality screen is calculated according to a preset criterion for the area in which the first object O1 is in contact in the score calculation step (S1300). The calculated score may be set differently.

More preferably, in the targeting step (S1100), when the coordinate information of the user's gaze and the pupil position information meet the preset criteria, a score for each area set according to the preset criteria is applied to the second object O2. Display. In addition, although it is shown in FIG. 8(c) that a higher score is given as the center position of the second object O2 is increased, a score may be assigned based on other preset criteria according to settings in virtual reality content.

The user may be provided with a virtual reality screen in which a score given differently for each area of the second object O2 is displayed by gathering more eyes and concentrating his/her eyes in order to obtain a higher score, and based on the displayed score information By concentrating on the target area of the second object O2 and obtaining a high score, it is possible to exert the effect of improving the immersion of the virtual reality content.

9 schematically shows an internal configuration of a service server 2000 according to an embodiment of the present invention.

The service server 2000 of the present invention receives the gaze heat map generated by the strabismus diagnosis unit 1500 of the HMD module 1000, and the gaze heat map received by the diagnosis model learned from the learning gaze heat map data. strabismus diagnosis information for As shown in FIG. 10 , the service server 2000 includes a strabismus diagnosis information extracting unit 2100 and a diagnosis model learning unit 2200 .

The strabismus diagnosis information derivation unit 2100 derives strabismus diagnosis information for the gaze heat map received from the HMD module 1000 through a diagnosis model using machine learning. After receiving the gaze heat map, the strabismus diagnosis information derivation unit 2100 automatically performs a diagnosis using the diagnostic model, and derives strabismus diagnosis information for the gaze heat map.

The diagnosis model learning unit 2200 may learn a diagnosis model for deriving strabismus diagnosis information using the learning gaze heat map data. The strabismus diagnosis information for the gaze heat map received by the diagnostic model learned based on the learning gaze heat map data is derived.

The service server 2000 in FIG. 9 may further include elements other than the illustrated elements, but only elements related to the rehabilitation training system according to embodiments of the present invention are displayed for convenience.

Specifically, the strabismus diagnosis unit 1500 of the HMD module 1000 of the present invention is based on the coordinate information of the user's gaze in the trigger step (S1000), the targeting step (S1100), and the release step (S1200). A strabismus diagnosis step (S1400) of generating a gaze heat map and transmitting the generated gaze heat map to the service server 2000; is performed. The strabismus diagnosis unit 1500 of the HMD module 1000 generates a gaze heat map based on coordinate information of the user's gaze, and such a gaze heat map is based on the coordinate information of the user's gaze, as shown in FIG. Based on the information about the length of time the user's gaze stayed on the virtual reality screen is information displayed as an image. Through this gaze heat map, when comparing the viewpoint of the exotropia patient with the coordinates of the first object (O1) and the second object (O2), it is possible to grasp information about where the patient stayed and how long he stayed there have. Such a gaze heat map may be displayed as two-dimensional image information as shown in FIG. 10 .

11 schematically shows the steps of performing the HMD module 1000 and the service server 2000 according to an embodiment of the present invention.

Rehabilitation training system including the HMD module 1000 and the service server 2000 of the present invention, generating a gaze heat map based on the coordinate information of the user's gaze (S200); transmitting the generated heat map to the service server 2000 (S210); deriving strabismus diagnosis information for the gaze heat map received by the diagnostic model learned based on the learning gaze heat map data (S220); Transmitting the derived strabismus diagnosis information (S230) is performed.

Specifically, in step S200, the strabismus diagnosis unit 1500 of the HMD module 1000 performs a trigger step (S1000), a targeting step (S1100), and a release step (S1200) by the content execution unit 1400. A gaze heat map is generated based on the coordinate information of the user's gaze during the period. The gaze heat map is information that displays information about the length of time the user's gaze stays on the virtual reality screen as an image based on the coordinate information of the user's gaze.

In step S210 , the strabismus diagnosis unit 1500 of the HMD module 1000 transmits the generated gaze heat map to the service server 2000 .

In step S220, the service server 2000 derives strabismus diagnosis information for the gaze heat map received by the diagnostic model learned from the learning gaze heat map data. The diagnosis model is learned by gaze heat map data of a plurality of exotropia patients who have performed virtual reality contents in the past to derive strabismus diagnosis information for the gaze heat map received from the HMD module 1000 . Preferably, the strabismus diagnosis information may include information on the user's strabismus, information on the angle of strabismus, and the frequency of occurrence of exotropia. The diagnostic model can analyze the gaze heat map using artificial neural network technology that includes temporal concepts such as RNN, LSTM, and GRU, and the diagnostic model includes one or more deep learning-based trained artificial neural network modules. can do.

In step S230, the service server 2000 transmits the derived strabismus diagnosis information. The derived strabismus diagnosis information may be transmitted to the HMD module 1000 to display the strabismus diagnosis information in the HMD module 1000, or may be transmitted to a user's terminal or a user's guardian or a specialist's terminal to be utilized for exotropia diagnosis. .

In this way, the HMD module 1000 generates a gaze heat map based on the coordinate information and pupil position information of the user's gaze and transmits it to the service server 2000, and the service server 2000 transmits the gaze heat map. By deriving strabismus diagnosis information for the strabismus and transmitting it to the HMD module 1000 or an external terminal, it is possible to exert the effect of utilizing the strabismus diagnosis information as the user's treatment, treatment and consultation data.

12 schematically illustrates the operation of the diagnostic model learning unit 2200 of the service server 2000 according to an embodiment of the present invention.

The service server 2000 of the present invention receives the gaze heat map generated by the strabismus diagnosis unit 1500 of the HMD module 1000, and the gaze heat map received by the diagnosis model learned from the learning gaze heat map data. strabismus diagnosis information for As shown in (a) of FIG. 12 , the service server 2000 includes a diagnosis model learning unit 2200, and the diagnosis model learning unit 2200 is configured to learn a diagnosis model based on the learning gaze heat map data. can As described above, the learning gaze heat map data for learning the diagnostic model may be gaze heat map data of a plurality of exotropia patients using the rehabilitation training system of the present invention as shown in FIG. 12( b ). A plurality of exotropia patients with different strabismus information including the strabismus developed eye and strabismus angle in the past performed virtual reality content in the past, and a plurality of gaze heatmaps are used as learning gaze heatmap data to learn the diagnostic model. can be Preferably, the learning gaze heat map data is gaze heat map data of a plurality of exotropia patients using the past rehabilitation system stored in the service server 2000 . Alternatively, a diagnosis result for the gaze heat map received from the HMD module 1000 may be received, and the gaze heat map including the diagnosis result may be utilized as learning gaze heat map data for learning the diagnostic model.

13 exemplarily shows an internal configuration of a computing device according to an embodiment of the present invention.

13, the computing device 11000 includes at least one processor 11100, a memory 11200, a peripheral interface 11300, an input/output subsystem ( I/O subsystem) 11400 , a power circuit 11500 , and a communication circuit 11600 may be included at least. In this case, the computing device 11000 may correspond to the service server 2000 or the HMD module 1000 .

The memory 11200 may include, for example, a high-speed random access memory, a magnetic disk, an SRAM, a DRAM, a ROM, a flash memory, or a non-volatile memory. have. The memory 11200 may include a software module required for the operation of the computing device 11000 , an instruction set, or other various data included in the learned embedding model.

In this case, access to the memory 11200 from other components such as the processor 11100 or the peripheral device interface 11300 may be controlled by the processor 11100 .

Peripheral interface 11300 may couple input and/or output peripherals of computing device 11000 to processor 11100 and memory 11200 . The processor 11100 may execute a software module or an instruction set stored in the memory 11200 to perform various functions for the computing device 11000 and process data.

The input/output subsystem 11400 may couple various input/output peripherals to the peripheral interface 11300 . For example, the input/output subsystem 11400 may include a controller for coupling a peripheral device such as a monitor, keyboard, mouse, printer, or a touch screen or sensor as required to the peripheral interface 11300 . According to another aspect, input/output peripherals may be coupled to peripheral interface 11300 without going through input/output subsystem 11400 .

The power circuit 11500 may supply power to all or some of the components of the terminal. For example, the power circuit 11500 may include a power management system, one or more power sources such as batteries or alternating current (AC), a charging system, a power failure detection circuit, a power converter or inverter, a power status indicator, or a power source. It may include any other components for creation, management, and distribution.

The communication circuit 11600 may enable communication with another computing device using at least one external port.

Alternatively, as described above, if necessary, the communication circuit 11600 may include an RF circuit to transmit and receive an RF signal, also known as an electromagnetic signal, to enable communication with other computing devices.

This embodiment of FIG. 13 is only an example of the computing device 11000 , and the computing device 11000 may omit some components shown in FIG. 13 , or further include additional components not shown in FIG. 13 , or 2 It may have a configuration or arrangement that combines two or more components. For example, a computing device for a communication terminal in a mobile environment may further include a touch screen or a sensor in addition to the components shown in FIG. 13 , and may include various communication methods (WiFi, 3G, LTE) in the communication circuit 1160 . , Bluetooth, NFC, Zigbee, etc.) may include a circuit for RF communication. Components that may be included in the computing device 11000 may be implemented in hardware, software, or a combination of both hardware and software including an integrated circuit specialized for one or more signal processing or applications.

Methods according to an embodiment of the present invention may be implemented in the form of program instructions that can be executed through various computing devices and recorded in a computer-readable medium. In particular, the program according to the present embodiment may be configured as a PC-based program or an application dedicated to a mobile terminal. The application to which the present invention is applied may be installed in the user terminal through a file provided by the file distribution system. As an example, the file distribution system may include a file transmission unit (not shown) that transmits the file according to a request of the user terminal.

The device described above may be implemented as a hardware component, a software component, and/or a combination of the hardware component and the software component. For example, devices and components described in the embodiments may include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA). , a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions, may be implemented using one or more general purpose or special purpose computers. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to execution of the software. For convenience of understanding, although one processing device is sometimes described as being used, one of ordinary skill in the art will recognize that the processing device includes a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that can include For example, the processing device may include a plurality of processors or one processor and one controller. Other processing configurations are also possible, such as parallel processors.

Software may comprise a computer program, code, instructions, or a combination of one or more thereof, which configures a processing device to operate as desired or is independently or collectively processed You can command the device. The software and/or data may be any kind of machine, component, physical device, virtual equipment, computer storage medium or device, to be interpreted by or to provide instructions or data to the processing device. , or may be permanently or temporarily embody in a transmitted signal wave. The software may be distributed over networked computing devices, and stored or executed in a distributed manner. Software and data may be stored in one or more computer-readable recording media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the embodiment, or may be known and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic such as floppy disks. - includes magneto-optical media, and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

As described above, although the embodiments have been described with reference to the limited embodiments and drawings, various modifications and variations are possible from the above description by those skilled in the art. For example, the described techniques are performed in a different order than the described method, and/or the described components of the system, structure, apparatus, circuit, etc. are combined or combined in a different form than the described method, or other components Or substituted or substituted by equivalents may achieve an appropriate result.

Therefore, other implementations, other embodiments, and equivalents to the claims are also within the scope of the following claims.

Claims

As an exotropia patient rehabilitation system comprising an HMD module and a service server,

In the HMD module, virtual reality content can be executed,

The virtual reality content,

a triggering step of moving the first object in a preset area of the central portion of the virtual reality screen from the start position to the user side according to the user's manipulation of the controller;

A targeting step of moving the virtual reality screen according to the direction manipulation of the HMD module according to the movement of the user's head;

a release step of emitting the first object from a virtual reality screen according to a user's controller manipulation; and

Exotropia including HMD module and service server; Patient Rehabilitation Training System.
The method according to claim 1,

The one or more second objects are set to have respective preset sizes,

The coordinates of the second object have respective distances from the coordinates of the user, an exotropia patient rehabilitation system comprising an HMD module and a service server.
The method according to claim 1,

The HMD module,

Collecting coordinate information of the user's gaze on the virtual reality screen, and pupil position information,

The trigger step is

Deriving the release level of the first object based on the coordinate information of the user's gaze and the user's pupil position information while the first object is moving,

The release step is

Exotropia patient rehabilitation system comprising an HMD module and a service server, which determines the firing intensity or firing distance of the first object in the virtual reality screen based on the release level derived in the trigger step.
The method according to claim 1,

The HMD module,

Collecting the user's pupil location information on the virtual reality screen,

The trigger step is

Exotropia patient rehabilitation system comprising an HMD module and a service server for resetting the position of the first object moved to the user side to the starting position when the user's pupil position does not meet the preset criteria.
The method according to claim 1,

The HMD module,

Collecting the coordinate information of the user's gaze on the virtual reality screen,

The targeting step is

Based on the coordinate information of the user's left eye gaze and the right eye gaze coordinate information, a release area is derived according to a preset criterion, and the first object is launched within the derived release area range, HMD module and service Exotropia rehabilitation system including a server.
The method according to claim 1,

The virtual reality content,

Further comprising; a strabismus diagnosis step of generating a gaze heat map based on the coordinate information of the user's gaze in the trigger step, the targeting step, and the release step, and transmitting the generated gaze heat map to the service server;

The service server is an HMD module and a service server for deriving strabismus diagnosis information for the gaze heat map received by the diagnostic model learned based on the learning gaze heat map data.
As a rehabilitation training method for an exotropia patient using an exotropia rehabilitation system including an HMD module and a service server,

a triggering step of moving, by the HMD module, a first object in a preset area of the central portion of the virtual reality screen from the start position to the user side according to the user's controller manipulation;

A targeting step of moving the virtual reality screen according to the direction manipulation of the HMD module according to the movement of the user's head by the HMD module;

a release step of emitting the first object from the virtual reality screen according to the user's controller manipulation by the HMD module; and

A score calculation step of calculating a score by determining whether the first object fired in the release step is in contact with one or more second objects existing in the virtual reality screen by the HMD module; Way.
As a computer-readable medium for implementing a method for rehabilitation of an exotropia patient using an exotropia rehabilitation system including an HMD module and a service server,

The computer-readable medium stores instructions for causing a component of the exotropia rehabilitation system to perform the following steps:

a triggering step of moving, by the HMD module, a first object in a preset area of the central portion of the virtual reality screen from the start position to the user side according to the user's controller manipulation;

A targeting step of moving the virtual reality screen according to the direction manipulation of the HMD module according to the movement of the user's head by the HMD module;

a release step of emitting the first object from the virtual reality screen according to the user's controller manipulation by the HMD module; and

Computer-readable, including; a score calculation step of calculating a score by determining whether the first object fired in the release step is in contact with one or more second objects existing in the virtual reality screen by the HMD module media.