KR102165590B1 - Social Virtual Reality System for the Treatment of Mild Cognitive Impairment and Social Virtual Reality Providing Method - Google Patents

Abstract

The present invention relates to a social virtual reality system for mild cognitive impairment treatment and to a method for providing social virtual reality. More specifically, provided are the social virtual reality system for mild cognitive impairment treatment in which clinical therapists and patients with mild cognitive impairment conduct mutual activities in social virtual reality through VR devices, and at the same time perform treatment through voice and eye movements of patients with mild cognitive impairment, so as to perform diagnosis of patients with mild cognitive impairment, and the method for providing social virtual reality.

Description

Social Virtual Reality System for the Treatment of Mild Cognitive Impairment and Social Virtual Reality Providing Method}

The present invention relates to a method of providing a social virtual reality system and a social virtual reality for the treatment of mild cognitive impairment, and more particularly, a clinical therapist and a patient with mild cognitive impairment conduct mutual activities in social virtual reality through a VR device. It relates to a social virtual reality system for the treatment of mild cognitive impairment and a method of providing a social virtual reality for the treatment of mild cognitive impairment to perform a diagnosis of a patient with mild cognitive impairment through voice and eye movement of a patient with mild cognitive impairment while performing treatment.

Currently, virtual reality contents are being developed in various fields such as video, games, education, and medical care, mainly overseas. As a device for using such virtual reality contents, various devices are spreading, from a cardboard-type HMD to devices that provide high-quality VR environments such as Gear VR, Oculus Rift, and HTC Vive, and VR contents are also increasing. .

Social virtual reality is a social network platform that allows multiple users to perform various activities such as games in a virtual reality space. In the game industry where virtual reality contents are actively being developed, the center of gravity is gradually shifting from VR games to social-based virtual reality games, and game services that emphasize storytelling, interaction, and fun elements are provided. In such a social-based virtual reality, the user does not exist alone, but exists in a common space, so physical interaction as well as social interaction occurs.

Such social virtual reality allows users who are not in the same space to be gathered in a virtual space to perform mutual activities, so that people in a distance can conduct joint actions without meeting them in person.

The present invention allows a clinical therapist and a patient with mild cognitive impairment to perform mutual activities in social virtual reality through a VR device to perform treatment and at the same time perform a diagnosis of a patient with mild cognitive impairment through voice and eye movements of the patient with mild cognitive impairment. Its purpose is to provide a social virtual reality system for the treatment of mild cognitive impairment and a method of providing social virtual reality.

In order to solve the above problems, the present invention, as a social virtual reality system for the treatment of mild cognitive impairment, communicates with a therapist VR device worn by a clinical therapist and a patient VR device worn by a mild cognitive impairment patient, and the A therapist VR device and a patient VR device include a service server that transmits information on social virtual reality, and the patient VR device provides images and sounds for social virtual reality to patients with mild cognitive impairment, and the patient with mild cognitive impairment The therapist's voice and behavior are sensed, and the therapist VR device provides images and sounds for social virtual reality to the clinical therapist, detects the voice and behavior of the clinical therapist, and the patient with mild cognitive impairment and the clinical therapist Provides a social virtual reality system that enables interactive activities through dialogue and movement through avatars implemented in social virtual reality.

In the present invention, the service server may perform diagnosis of the mild cognitive impairment patient based on the voice of the mild cognitive impairment patient.

In the present invention, the service server may perform diagnosis of the mild cognitive impairment patient based on the amplitude, length and pattern of the voice signal of the patient with mild cognitive impairment received from the patient VR device.

In the present invention, the patient VR device, the eye tracker for detecting the eye movement of the patient with mild cognitive impairment; Including, the service server, on the basis of the voice and eye movement of the patient with mild cognitive impairment received from the patient VR device may perform diagnosis of the patient with mild cognitive impairment.

In the present invention, the service server may perform diagnosis of the mild cognitive impairment patient based on the time coordinate of the focus according to the eye movement of the mild cognitive impairment patient and the movement pattern of the time coordinate.

In the present invention, the service server may perform diagnosis of the mild cognitive impairment patient by analyzing the eye movement of the mild cognitive impairment patient through an artificial neural network.

In order to solve the above problems, the present invention communicates with a therapist VR device worn by a clinical therapist and a patient VR device worn by a patient with mild cognitive impairment, and the therapist VR device and the patient VR device are used in social virtual reality. A method of providing a social virtual reality for the treatment of mild cognitive disorders executed by a service server that transmits information about the patient, comprising: transmitting information on social virtual reality to the patient VR device and the therapist VR device; Receiving the voice and behavior of the patient with mild cognitive impairment detected by the patient VR device; Receiving the voice and behavior of the clinical therapist detected by the therapist VR device; Controlling an avatar implemented in the social virtual reality based on the voices and actions of the patient with the mild cognitive impairment and the clinical therapist's voices and performing mutual activities through dialogue and movement; It provides a method of providing a social virtual reality comprising a.

In the present invention, the method of providing the social virtual reality includes the steps of performing a diagnosis of the mild cognitive impairment patient based on the voice of the mild cognitive impairment patient; It may further include.

According to an embodiment of the present invention, clinical treatment can be easily performed by accessing a social virtual reality environment without the need for a patient with mild cognitive impairment and a clinical therapist to meet directly.

According to an embodiment of the present invention, a plurality of patients with mild cognitive impairment and a clinical therapist may perform various learning and training such as memory training by performing mutual activities in a social virtual reality environment.

According to an embodiment of the present invention, personalized clinical treatment can be performed by diagnosing a patient with mild cognitive impairment undergoing clinical treatment.

According to an embodiment of the present invention, the voice and eye movements of a patient with mild cognitive impairment can be diagnosed based on an artificial neural network, diagnosed more accurately, and clinical treatment can be performed.

According to an embodiment of the present invention, it is possible to increase the accuracy of diagnosis based on an artificial neural network by inputting a diagnosis result by a specialist.

1 is a diagram schematically illustrating a state in which a social virtual reality system according to an embodiment of the present invention provides a social virtual reality.
2 is a diagram schematically illustrating a VR device worn by a user using social virtual reality according to an embodiment of the present invention.
3 is a diagram schematically showing an inner surface of a VR device according to an embodiment of the present invention.
4 is a diagram schematically showing an internal configuration of a service server according to an embodiment of the present invention.
5 is a diagram schematically showing the operation of a service server according to an embodiment of the present invention.
6 is a diagram schematically showing the operation of a social virtual reality implementation unit according to an embodiment of the present invention.
7 is a diagram schematically showing the operation of a patient diagnosis unit according to an embodiment of the present invention.
8 is a diagram schematically illustrating an operation of analyzing voice information by a patient diagnosis unit according to an embodiment of the present invention.
9 is a diagram schematically showing a detailed configuration of a patient diagnosis unit according to an embodiment of the present invention.
10 is a diagram schematically illustrating an operation of analyzing eye movement information by a patient diagnosis unit according to an embodiment of the present invention.
11 is a diagram schematically showing heat map information and movement trajectory information generated by analyzing eye movement information according to an embodiment of the present invention.
12 is a diagram schematically showing each step of a method of providing a social virtual reality according to an embodiment of the present invention.
13 is a block diagram illustrating an example of an internal configuration of a computing device according to an embodiment of the present invention.

In the following, various embodiments and/or aspects are now disclosed with reference to the drawings. In the following description, for purposes of explanation, a number of specific details are disclosed to aid in an overall understanding of one or more aspects. However, it will also be appreciated by those of ordinary skill in the art that this aspect(s) may be practiced without these specific details. The following description and the annexed drawings set forth in detail certain illustrative aspects of one or more aspects. However, these aspects are exemplary and some of the various methods in the principles of the various aspects may be used, and the descriptions described are intended to include all such aspects and their equivalents.

Further, various aspects and features will be presented by a system that may include multiple devices, components and/or modules, and the like. It is also noted that various systems may include additional devices, components and/or modules, and/or may not include all of the devices, components, modules, etc. discussed in connection with the figures. It must be understood and recognized.

As used herein, “an embodiment,” “example,” “aspect,” “example,” and the like may not be construed as having any aspect or design described as being better or advantageous than other aspects or designs. . The terms'~part','component','module','system', and'interface' used below generally mean a computer-related entity, for example, hardware, hardware It can mean a combination of software and software, or software.

In addition, the terms "comprising" and/or "comprising" mean that the corresponding feature and/or element is present, but excludes the presence or addition of one or more other features, elements, and/or groups thereof. It should be understood as not.

In addition, terms including ordinal numbers such as first and second may be used to describe various elements, but the elements are not limited by the terms. These terms are used only for the purpose of distinguishing one component from another component. For example, without departing from the scope of the present invention, a first element may be referred to as a second element, and similarly, a second element may be referred to as a first element. The term and/or includes a combination of a plurality of related listed items or any of a plurality of related listed items.

In addition, in the embodiments of the present invention, unless otherwise defined, all terms used herein including technical or scientific terms are commonly understood by those of ordinary skill in the art to which the present invention belongs. It has the same meaning as. Terms as defined in a commonly used dictionary should be interpreted as having a meaning consistent with the meaning in the context of the related technology, and unless explicitly defined in the embodiments of the present invention, an ideal or excessively formal meaning Is not interpreted as.

1 is a diagram schematically illustrating a state in which a social virtual reality system according to an embodiment of the present invention provides a social virtual reality.

Referring to FIG. 1, the social virtual reality system according to an embodiment of the present invention includes a therapist VR device 100 worn by a clinical therapist 20 and a patient VR device 100 worn by a mild cognitive impairment patient 10. A service server 3000 for performing communication and transmitting information on the social virtual reality 5000 to the therapist VR device 100 and the patient VR device 100; It may include.

Such a service server 3000 implements a social virtual reality 5000 and provides information on the social virtual reality 5000 to the clinical therapist 20 and the VR device 100 of the mild cognitive impairment patient 10 The clinical therapist 20 and the mild cognitive impairment patient 10 can access the social virtual reality 5000 to experience virtual reality.

As shown in FIG. 1, the mild cognitive impairment patient 10 and the clinical therapist 20 engage in mutual activities through dialogue and movement through the avatars 50 and 60 implemented in the social virtual reality 5000. You can proceed.

To this end, the patient VR device 100 provides images and sounds for the social virtual reality 5000 to the mild cognitive impairment patient 10, detects the voice and behavior of the mild cognitive impairment patient 10, and The therapist VR device 100 provides images and sounds for the social virtual reality 5000 to the clinical therapist 20 and detects the voice and behavior of the clinical therapist 20. According to the voices and actions of the mild cognitive impairment patient 10 and the clinical therapist 20 sensed in this way, the avatars 50 and 60 implemented in the social virtual reality 5000 speak or make movements. Take it and proceed with mutual activities.

As shown in FIG. 1, a clinical therapist 20 and a plurality of patients with mild cognitive impairment 10 may simultaneously access the social virtual reality 5000 to perform clinical treatment in the social virtual reality 500. On the other hand, unlike the one shown in Figure 1, it will be apparent that a plurality of clinical therapists 20 can access the social virtual reality 5000 to perform clinical treatment.

Through such a social virtual reality system, patients with mild cognitive impairment 10 and clinical therapists 20 can perform clinical treatment by accessing social virtual reality through respective VR devices without having to meet directly in reality.

2 is a diagram schematically illustrating a VR device worn by a user using social virtual reality according to an embodiment of the present invention.

Referring to FIG. 2, an embodiment of a patient VR device 100 and a therapist VR device 100 worn by the mild cognitive impairment patient 10 and the clinical therapist 20 can be confirmed. In an embodiment of the present invention, the patient VR device 100 and the therapist VR device 100 may have the same configuration. Hereinafter, both the patient VR device 100 and the therapist VR device 100 are referred to as a VR device 100. The configuration of the VR device 100 described below may be included in the patient VR device 100 or the therapist VR device 100.

Referring to FIG. 2, the VR device 100 according to an embodiment of the present invention inputs an HMD device 1000 providing information on virtual reality to a user (a patient with mild cognitive impairment or a clinical therapist) and the user's movement. Receiving controller 2000; Including, the HMD device 1000 is a time providing unit 1100 for providing visual information on virtual reality to the user; A hearing providing unit 1300 providing auditory information on virtual reality to a user; A voice input unit 1500 for detecting a user's voice; And a fixing part 1200 fixing the HMD device 1000 to a user's body. It may include.

The VR device 100 according to an embodiment of the present invention provides information on the social virtual reality 5000 to the user through the time providing unit 1100 and the hearing providing unit 1300 so that the user can The social virtual reality 5000 allows the user to experience the same experience as existing in the virtual space of the social virtual reality 5000, and receives the user's voice and behavior through the voice input unit 1500 and the controller 2000, 5000), the avatar of the user implemented in the virtual space may move or speak.

The controller 2000 may detect the movement of the hand by moving while the user is holding it in his hand, or may receive a command through an provided button or a joystick. In an embodiment of the present invention, the avatar implemented in the social virtual reality 5000 may be moved or the direction may be changed by a button provided in the controller 2000. By moving or changing the direction of the avatar in this way, the user can not only experience a visual experience as if looking around the virtual space of the social virtual reality 5000, but also move the space as if it were actually located in the virtual space. You can experience it.

Alternatively, the VR device 100 according to an embodiment of the present invention may include a motion detector (not shown) to directly detect movement of the user's arm, torso, and legs. Such a motion detector recognizes the user, the user's arm, and the user's leg by using information such as an image, and detects the movement of the user, the user's arm, and the user's leg to determine the user's avatar. By controlling, the user can experience the virtual space of the social virtual reality 5000 through direct movement.

3 is a diagram schematically showing an inner surface of a VR device according to an embodiment of the present invention.

Referring to FIG. 3, a screen unit 1110 for providing visual information to the inside of the HMD device 1000 to give a three-dimensional effect to both eyes of the user; And an eye tracker 1600 for detecting eye movement of the user. It may further include.

In an embodiment of the present invention, the eye movement of a user, particularly a patient with mild cognitive impairment, can be detected and used for diagnosis of the user. It is possible to check the gaze of the mild cognitive impairment patient 10 according to the eye movement of the mild cognitive impairment patient 10, and analyze the position of the focal point and the movement of the focus of the mild cognitive impairment patient 10 It can be used as an index to diagnose.

The eye tracker 1600 may be positioned at an edge of the screen unit 1110 so that the screen unit 1110 does not interfere with providing visual information to the user.

4 is a diagram schematically showing an internal configuration of a service server according to an embodiment of the present invention.

4, the service server 3000 according to an embodiment of the present invention

A social virtual reality realization unit that implements a social virtual reality 5000 based on the information received from the patient VR device 100 and the therapist VR device 100 and provides information on the social virtual reality 5000 (3100); A VR device communication unit 3200 that communicates with the patient VR device 100 and the therapist VR device 100 to exchange information; And a patient diagnosis unit 3300 for performing diagnosis on the patient with mild cognitive impairment using the social virtual reality 5000. It may include.

The social virtual reality realization unit 3100 implements a social virtual reality 5000 and implements an avatar of a user (a patient with mild cognitive impairment or a clinical therapist) who accesses the virtual space of the social virtual reality 5000, The avatar enables the user to experience the virtual space, and allows the avatar to interact with other avatars, thereby allowing a plurality of users to experience the same space as being in the same space.

In one embodiment of the present invention, the mild cognitive impairment patient 10 and the clinical therapist 20 have conversations and various activities in the virtual space of the social virtual reality 5000 implemented by the social virtual reality realization unit 3100. Clinical treatment can be performed through (drawing pictures, simple exercises, memory training, etc.).

The VR device communication unit 3200 may communicate with the patient VR device 100 and the therapist VR device 100 to exchange information. In this way, the VR device communication unit 3200 communicates with each of the patients with mild cognitive impairment 10 and the VR devices 100 of the clinical therapist 20 to perform clinical treatment in the social virtual reality 5000. To be.

Preferably, the VR device communication unit 3200 may communicate with the patient VR device 100 and the therapist VR device 100 through a 5G wireless network to exchange information. In one embodiment of the present invention, a plurality of mild cognitive impairment patients 10 and clinical therapists 20 are connected to the social virtual reality 5000 to perform various mutual activities. That is, by receiving information from the patient VR device 100 and the therapist VR device 100 of each mild cognitive impairment patient 10 and clinical therapist 20, the social virtual reality 5000 is implemented, and the implemented The information on the social virtual reality 5000 is transmitted to the patient VR device 100 and the therapist VR device 100 of the mild cognitive impairment patient 10 and the clinical therapist 20 again to use the social virtual reality 5000 In order to be able to do so, it is necessary to transmit large amounts of information quickly. In an embodiment of the present invention, the delay of information processing is prevented by performing communication through a 5G wireless network, and the social virtual reality (5000) allows the mild cognitive impairment patient 10 and the clinical therapist 20 to easily ) Can be used.

The patient diagnosis unit 3300 performs diagnosis on the mild cognitive impairment patient 10 using the social virtual reality 5000. Preferably, the patient diagnosis unit 3300 performs diagnosis based on the information received from the patient VR device 100 of the patient with mild cognitive impairment 10. In this way, the diagnosis result of the mild cognitive impairment patient 10 performed by the patient diagnosis unit 3300 is transmitted to the therapist VR device 100 so that the clinical therapist 20 can transmit the mild cognitive impairment patient 10 It can be possible to provide customized clinical treatment to patients.

For example, the clinical therapist 20 provides clinical treatment to the mild cognitive impairment patient 10 through the social virtual reality 5000, and the patient diagnosis unit 3300 of the service server 3000 provides clinical treatment. The patient 10 with mild cognitive impairment is diagnosed in real time and the diagnosis result is provided to the clinical therapist 20. The clinical therapist 20 may check the effect of clinical treatment provided to the mild cognitive impairment patient 10 in real time by looking at the diagnosis result, and provide clinical treatment according to the diagnosis result of the mild cognitive impairment patient 10 Treatment can be changed or the level of clinical treatment can be adjusted.

In one embodiment of the present invention, the patient diagnosis unit 3300 is based on one or more of the voice, eye movement, and behavior of the mild cognitive impairment patient 10 received from the patient VR device 100. Diagnosis of patients can be performed. Preferably, the patient diagnosis unit 3300 performs diagnosis of the mild cognitive impairment patient based on the voice and eye movement of the mild cognitive impairment patient 10 received from the patient VR device 100.

5 is a diagram schematically showing the operation of a service server according to an embodiment of the present invention.

Referring to Figure 5, the service server 3000 according to an embodiment of the present invention from the patient VR device (100a) and the therapist VR device (100b), respectively, mild cognitive impairment patient 10 and clinical therapist 20 Each of the voice information and behavior information is received, and information on the social virtual reality 5000 is provided to the patient VR device 100a and the therapist VR device 100b. Meanwhile, eye movement information of the patient with mild cognitive impairment 10 is transmitted from the patient VR device 100b, and diagnostic information of the mild cognitive impairment patient 10 is provided to the therapist VR device 100b.

By taking such a configuration, the service server 3000 implements the avatars of the mild cognitive impairment patient 10 and the clinical therapist 20 in the virtual space of the implemented social virtual reality 5000, respectively, and the avatars The mild cognitive impairment patient 10 and the clinical therapist 20 can interact with each other in the virtual space by controlling based on the voice information and behavior information of each of the patients with mild cognitive impairment (10) and the clinical therapist (20). Make it possible.

6 is a diagram schematically showing the operation of a social virtual reality implementation unit according to an embodiment of the present invention.

In the configuration as shown in FIG. 5, in more detail, the VR device communication unit 3200 of the service server 3000 provides information on the social virtual reality 5000 implemented by the social virtual reality realization unit 3100 to the patient. Transmitted to the VR device 100a and the therapist VR device 100b, respectively, and the mild cognitive impairment patient 10 and the clinical therapist 20 respectively from the patient VR device 100a and the therapist VR device 100b By receiving the voice information and behavior information of the social virtual reality realization unit 3100 to control the avatars of the mild cognitive impairment patient 10 and the clinical therapist 20 implemented.

7 is a diagram schematically showing the operation of a patient diagnosis unit according to an embodiment of the present invention.

On the other hand, in the configuration as in FIG. 5, in more detail, the VR device communication unit 3300 of the service server 3000 provides eye movement information and voice of the mild cognitive impairment patient 10 from the patient VR device 100a. Receives at least one of information and behavior information and provides it to the patient diagnosis unit 3300, and the patient diagnosis unit 3300 includes at least one of eye movement information, voice information, and behavior information of the mild cognitive impairment patient 10 Based on the diagnosis of the mild cognitive impairment patient 10, patient diagnosis information is generated. Thereafter, the patient diagnosis information is transmitted to the therapist VR device 100b through the VR device communication unit 3200, and the clinical therapist 20 to the mild cognitive impairment patient 10 based on the patient diagnosis information. It will provide customized clinical treatment.

At this time, in one embodiment of the present invention, the patient diagnosis unit 3300 is based on the time-wise coordinates of the focus according to the eye movement of the mildly cognitively impaired patient 10 and the movement pattern of the time-wise coordinates Diagnosis of (10) can be performed.

Meanwhile, in an embodiment of the present invention, the specialist 30 may input a result of a professional diagnosis on the patient 10 with mild cognitive impairment in the patient diagnosis unit 3300. The specialist 30 may directly diagnose the mild cognitive impairment patient 10 and then input the diagnosis result, or the information on the mild cognitive impairment patient 10 input to the service server 3000 After the diagnosis is made based on the diagnosis, the diagnosis result may be input. The diagnosis result input as described above may be used as information for the patient diagnosis unit 3300 to make a more accurate diagnosis for the mild cognitive impairment patient 10.

In an embodiment of the present invention, the patient diagnosis unit 3300 may perform diagnosis of the mild cognitive impairment patient 10 based on the behavior information of the mild cognitive impairment patient 10.

In an embodiment of the present invention, the patient VR device 100a may receive the behavior of the mild cognitive impairment patient 10 through the controller 2000, and the patient diagnosis unit 3300 inputs the same. The diagnosis of the mild cognitive impairment patient 10 may be performed based on the behavior of the mild cognitive impairment patient 10. In an embodiment of the present invention, the mild cognitive impairment patient 10 may move the avatar 50 implemented in the social virtual reality 5000 through the controller 2000, and the patient diagnosis unit 3300 May perform diagnosis of the mild cognitive impairment patient 10 based on the movement history of the avatar 50.

8 is a diagram schematically illustrating an operation of analyzing voice information by a patient diagnosis unit according to an embodiment of the present invention.

As shown in FIG. 8, the service server 3000 according to an embodiment of the present invention extracts information on the amplitude, length, and pattern of silver from the input voice of the patient 10 with mild cognitive impairment. Thereafter, the patient diagnosis unit 3300 is based on the amplitude, length and pattern of the voice signal of the mild cognitive impairment patient 10 received from the patient VR device 100. You will perform a diagnosis.

In an embodiment of the present invention, the clinical therapist 20 may perform a preset speech test through mutual activity with the mild cognitive impairment patient 10 in the social virtual reality 5000. In the speech test, the mild cognitive impairment patient 10 reads a preset sentence, and the service server 3000 analyzes the voice of the mild cognitive impairment patient 10 reading the preset sentence to recognize the mild cognitive impairment. Diagnosis of the disabled patient 10 can be performed.

9 is a diagram schematically showing a detailed configuration of a patient diagnosis unit according to an embodiment of the present invention.

Referring to FIG. 9, the patient diagnosis unit 3300 according to an embodiment of the present invention may perform diagnosis based on at least one of eye movement information, voice information, and behavior information of the mild cognitive impairment patient 10. have. At this time, the patient diagnosis unit 3300 may perform diagnosis of the mild cognitive impairment patient by analyzing the eye movement, voice, and behavior of the mild cognitive impairment patient 10 through an artificial neural network.

To this end, the patient diagnosis unit 3300 according to an embodiment of the present invention may perform analysis including each artificial neural network for analyzing the eye movement information, the voice information, and the behavior information.

In the embodiment shown in Fig. 9, the patient diagnosis unit 3300 includes a first artificial neural network 3310 that receives and analyzes the eye movement information; A second artificial neural network 3320 that receives and analyzes the voice information; A third artificial neural network (3330) that receives and analyzes the behavior information; And a comprehensive diagnosis module for performing diagnosis on the mild cognitive impairment patient 10 based on the analysis results of the first artificial neural network 3310, the second artificial neural network 3320 and the third artificial neural network 3330. (3340) may be included.

As shown, the first artificial neural network 3310 to the third artificial neural network 3330 respectively receive the eye movement information, voice information, and behavior information of the mild cognitive impairment patient 10 and perform analysis, and the analysis result Is transmitted to the comprehensive diagnosis module, and the comprehensive diagnosis module 3340 receives the analysis result, performs diagnosis on the mild cognitive impairment patient 10, and generates and outputs patient diagnosis information. The patient diagnosis information output as described above is transmitted to the therapist VR device 100 and used as information for the clinical therapist 20 to perform customized clinical treatment.

10 is a diagram schematically illustrating an operation of analyzing eye movement information by a patient diagnosis unit according to an embodiment of the present invention.

In an embodiment of the present invention, the focus according to the gaze of the mild cognitive impairment patient 10 may be determined based on the eye movement of the mild cognitive impairment patient 10 detected by the eye tracker 1600, and thus On the basis of the mild cognitive impairment patient 10 can be diagnosed.

In an embodiment of the present invention, as illustrated in FIG. 9, the eye movement may be analyzed based on an artificial neural network. To this end, it is necessary to process the eye movement information in a form that is easy to analyze through the artificial neural network.

In an embodiment of the present invention, diagnosis may be performed based on the coordinates of the focus according to the time of the focus of the patient 10 with mild cognitive impairment. That is, the coordinate information of the focus and time information of each of the coordinate information can be derived and used.

Heat map information and movement trajectory information can be derived based on the coordinate information and time information of the focus.

The heat map information is information displaying information on the length of time that the focus stays at each coordinate in the visual field of the mild cognitive impairment patient 10 as an image. Through such heat map information, it is possible to grasp information on where the focus of the mild cognitive impairment patient 10 stays for a long time. Such heat map information may appear as 2D image information.

The movement trajectory information is information that summarizes the movement order of the focus over time. That is, such movement trajectory information is information in which the coordinates of the focus at each time are arranged in order, and may be displayed as continuous list information.

In an embodiment of the present invention, the patient diagnosis unit 3300 may analyze the derived heat map information and the movement trajectory information, respectively, based on an artificial neural network, and synthesize the analysis results to derive a final analysis result. In this case, each artificial neural network for analysis may have a different structure. In an embodiment of the present invention, the heat map information may be analyzed based on a convolutional neural network (CNN), and the moving trajectory information may be analyzed based on a recurrent neural network (RNN). have. CNN is an artificial neural network structure that is advantageous for the analysis of two-dimensional images, and RNN is an artificial neural network structure that is advantageous for the analysis of a continuous list. As described above, in an embodiment of the present invention, it is possible to derive more accurate analysis results by converting the eye movement information into focus heat map information and movement trajectory information, and analyzing each of them based on an artificial neural network.

11 is a diagram schematically showing heat map information and movement trajectory information generated by analyzing eye movement information according to an embodiment of the present invention.

Referring to FIG. 11, hardness viewing an image in which three different users (a patient with mild cognitive impairment or a clinical therapist) are located in a social virtual reality 5000 implemented by a service server 3000 according to an embodiment of the present invention. It is possible to check the heat map and movement trajectory according to the gaze movement of the cognitive impairment patient 10.

As shown in FIG. 11, the heat map is a color that indicates the time the focal point of the patient with mild cognitive impairment 10 has stayed at the corresponding coordinates, and the movement trajectory is the movement of the gaze according to the time sequence (1->2->3) Is indicated by the lines connected in order.

Such heat map information and movement trajectory information may be input to artificial neural networks such as CNN and RNN, respectively, and may be used to diagnose the mild cognitive impairment patient 10.

12 is a diagram schematically showing each step of a method of providing a social virtual reality according to an embodiment of the present invention.

Referring to FIG. 12, a method of providing a social virtual reality executed by the service server 3000 according to an embodiment of the present invention includes the patient VR device 100 and the therapist VR device 100 using social virtual reality ( 5000) transmitting information (S100); Receiving the voice and behavior of the mild cognitive impairment patient 10 sensed by the patient VR device 100 (S200); Receiving the voice and behavior of the clinical therapist 20 sensed by the therapist VR device 100 (S300); Based on the voices and actions of the patient with mild cognitive impairment (10) and the clinical therapist (20) received, the avatars (50) (60) implemented in the social virtual reality (5000) are controlled to interact with each other through dialogue and movement. Step of proceeding the activity (S400); And performing a diagnosis of the mild cognitive impairment patient 10 based on the voice of the mild cognitive impairment patient 10 (S500). It may include.

Each of these steps may be performed by each component of the social virtual reality system as described above.

As described above, in step S200, not only the voice and behavior of the patient 10 with the mild cognitive impairment detected by the patient VR device 100 is transmitted, but also the mild cognitive impairment detected by the patient VR device 100 The eye movement of the patient 10 may be transmitted, and in step S500, the mild cognitive impairment patient 10 based on the voice of the mild cognitive impairment patient 10 and the eye movement of the mild cognitive impairment patient 10 Can perform a diagnosis of.

In an embodiment of the present invention, the clinical therapist 20 further comprises a step (not shown) of transmitting the diagnosis result of the mild cognitive impairment patient 10 to the therapist VR device 100 after performing step S500. It can be used as data for performing clinical treatment so that a customized clinical treatment for the mild cognitive impairment patient 10 can be performed.

13 is a block diagram illustrating an example of an internal configuration of a computing device according to an embodiment of the present invention.

13, the computing device 11000 includes at least one processor 11100, a memory 11200, a peripheral interface 11300, and an input/output subsystem ( I/O subsystem) 11400, a power circuit 11500, and a communication circuit 11600. In this case, the computing device 11000 may correspond to a VR device or a service server.

The memory 11200 may include, for example, high-speed random access memory, magnetic disk, SRAM, DRAM, ROM, flash memory, or nonvolatile memory. have. The memory 11200 may include a software module, an instruction set, or other various data necessary for the operation of the computing device 11000.

In this case, accessing the memory 11200 from another component such as the processor 11100 or the peripheral device interface 11300 may be controlled by the processor 11100.

The peripheral device interface 11300 may couple input and/or output peripheral devices of the computing device 11000 to the processor 11100 and the memory 11200. The processor 11100 may execute various functions for the computing device 11000 and process data by executing a software module or instruction set stored in the memory 11200.

The input/output subsystem 11400 may couple various input/output peripherals to the peripherals interface 11300. For example, the input/output subsystem 11400 may include a monitor, a keyboard, a mouse, a printer, or a controller for coupling a peripheral device such as a touch screen or a sensor to the peripheral device interface 11300 as needed. According to another aspect, the input/output peripheral devices may be coupled to the peripheral device interface 11300 without going through the input/output subsystem 11400.

The power circuit 11500 may supply power to all or part of the components of the terminal. For example, the power circuit 11500 may include a power management system, one or more power sources such as batteries or alternating current (AC), a charging system, a power failure detection circuit, a power converter or inverter, a power status indicator or power. It may contain any other components for creation, management, and distribution.

The communication circuit 11600 may enable communication with another computing device using at least one external port.

Alternatively, as described above, the communication circuit 11600 may enable communication with other computing devices by transmitting and receiving an RF signal, also known as an electromagnetic signal, including an RF circuit, if necessary.

The embodiment of FIG. 13 is only an example of the computing device 11000, and the computing device 11000 omits some of the components shown in FIG. 13, further includes an additional component not shown in FIG. 13, or 2 It can have a configuration or arrangement that combines two or more components. For example, a computing device for a communication terminal in a mobile environment may further include a touch screen or a sensor in addition to the components shown in FIG. 13, and various communication methods (WiFi, 3G, LTE) in the communication circuit 1160 , Bluetooth, NFC, Zigbee, etc.) may include a circuit for RF communication. Components that can be included in the computing device 11000 may be implemented as hardware, software, or a combination of hardware and software including one or more signal processing or application-specific integrated circuits.

Methods according to an embodiment of the present invention may be implemented in the form of program instructions that can be executed through various computing devices and recorded in a computer-readable medium. In particular, the program according to the present embodiment may be configured as a PC-based program or an application dedicated to a mobile terminal. An application to which the present invention is applied may be installed on a user terminal through a file provided by the file distribution system. For example, the file distribution system may include a file transmission unit (not shown) that transmits the file according to the request of the user terminal.

According to an embodiment of the present invention, clinical treatment can be easily performed by accessing a social virtual reality environment without the need for a patient with mild cognitive impairment and a clinical therapist to meet directly.

According to an embodiment of the present invention, a plurality of patients with mild cognitive impairment and a clinical therapist may perform various learning and training such as memory training by performing mutual activities in a social virtual reality environment.

According to an embodiment of the present invention, personalized clinical treatment can be performed by diagnosing a patient with mild cognitive impairment undergoing clinical treatment.

According to an embodiment of the present invention, the voice and eye movements of a patient with mild cognitive impairment can be diagnosed based on an artificial neural network, diagnosed more accurately, and clinical treatment can be performed.

According to an embodiment of the present invention, it is possible to increase the accuracy of diagnosis based on an artificial neural network by inputting a diagnosis result by a specialist.

As described above, although the embodiments have been described by the limited embodiments and drawings, various modifications and variations are possible from the above description by those of ordinary skill in the art. For example, the described techniques are performed in a different order from the described method, and/or components such as a system, structure, device, circuit, etc. described are combined or combined in a form different from the described method, or other components Alternatively, even if substituted or substituted by an equivalent, an appropriate result can be achieved. Therefore, other implementations, other embodiments, and claims and equivalents fall within the scope of the claims to be described later.

Claims (8)

As a social virtual reality system for the treatment of mild cognitive disorder,
A service server that communicates with a therapist VR device worn by a clinical therapist and a patient VR device worn by a patient with mild cognitive impairment, and transmits information on social virtual reality to the therapist VR device and the patient VR device; Including,
The patient VR device,
Provides images and sounds for social virtual reality to patients with mild cognitive impairment, and detects the voice and behavior of patients with mild cognitive impairment,
The therapist VR device,
Provides images and sounds for social virtual reality to clinical therapists, detects voices and actions of the clinical therapists
The mild cognitive impairment patient and the clinical therapist may conduct mutual activities through dialogue and movement through an avatar implemented in the social virtual reality,
The patient VR device,
Including; an eye tracker for detecting the eye movement of the patient with mild cognitive impairment,
The service server,
A social virtual reality realization unit that implements a social virtual reality based on the information received from the patient VR device and the therapist VR device, and provides information on the social virtual reality;
A VR device communication unit that communicates with the patient VR device and the therapist VR device to exchange information; And
Including; a patient diagnosis unit for performing diagnosis on the mild cognitive impairment patient using the social virtual reality,
The patient diagnosis unit,
Diagnosing the mild cognitive impairment patient based on the time coordinates of the focus according to the voice and eye movement of the patient with mild cognitive impairment received from the patient VR device and the movement pattern of the time coordinates,
The patient diagnosis unit,
A first artificial neural network for receiving and analyzing information on the eye movement of the patient with mild cognitive impairment;
A second artificial neural network for receiving and analyzing information on the voice of the patient with mild cognitive impairment;
A third artificial neural network for receiving and analyzing information on the behavior of the patient with mild cognitive impairment; And
Comprehensive diagnosis module for performing diagnosis on the patient with mild cognitive impairment based on analysis results of the first artificial neural network, the second artificial neural network, and the third artificial neural network; and
Heat map information that displays information on the length of time that the focus stays at each coordinate according to the eye movement as an image and movement trajectory information, which is information that summarizes the movement order of the focus according to time, is derived,
A social virtual reality system for analyzing the heat map information and the movement trajectory information by the first artificial neural network.
delete The method according to claim 1,
The service server,
A social virtual reality system for diagnosing the mild cognitive impairment patient based on the amplitude, length and pattern of the voice signal of the mild cognitive impairment patient received from the patient VR device.
delete delete delete Hardness executed by the service server that communicates with the therapist VR device worn by the clinical therapist and the patient VR device worn by the patient with mild cognitive impairment, and transmits information about social virtual reality to the therapist VR device and the patient VR device. As a method of providing social virtual reality for the treatment of cognitive disorders,
Transmitting information on social virtual reality to the patient VR device and the therapist VR device;
Receiving the voice and behavior of the patient with mild cognitive impairment detected by the patient VR device;
Receiving the voice and behavior of the clinical therapist detected by the therapist VR device; And
Controlling an avatar implemented in the social virtual reality based on the voices and actions of the patient with the mild cognitive impairment and the clinical therapist's voices and performing mutual activities through dialogue and movement; Including,
The patient VR device,
Including; an eye tracker for detecting the eye movement of the patient with mild cognitive impairment,
The service server,
Performing a patient diagnosis step of performing diagnosis on the mild cognitive impairment patient using the social virtual reality, and
The patient diagnosis step,
Diagnosing the mild cognitive impairment patient based on the time coordinates of the focus according to the voice and eye movement of the patient with mild cognitive impairment received from the patient VR device and the movement pattern of the time coordinates,
The patient diagnosis step,
A first artificial neural network for receiving and analyzing information on the eye movement of the patient with mild cognitive impairment;
A second artificial neural network for receiving and analyzing information on the voice of the patient with mild cognitive impairment;
A third artificial neural network for receiving and analyzing information on the behavior of the patient with mild cognitive impairment; And
Diagnosing a patient with mild cognitive impairment by a comprehensive diagnosis module for performing diagnosis on the mild cognitive impairment patient based on the analysis results of the first artificial neural network, the second artificial neural network, and the third artificial neural network,
Heat map information that displays information on the length of time that the focus stays at each coordinate according to the eye movement as an image and movement trajectory information, which is information that summarizes the movement order of the focus according to time, is derived,
A method of providing a social virtual reality, analyzing the heat map information and the movement trajectory information by the first artificial neural network.
delete

Images