KR20200071647A - Biofeedback method based on virtual/augmented reality contents and bio-signal for diagnosis and healing of mental illness - Google Patents

Abstract

For early diagnosis and treatment for mental illness for all classes including children, youth, and seniors, the present invention suggests a biofeedback method through extraction of user valid biodata reacting to experiential content using data measured through a virtual/augmented reality device capable of bio signal measurement. According to the present invention by analyzing a bio signal such as heartbeat and a gaze motion obtained through a virtual/augmented reality device in which a bio signal measurement sensor is mounted and providing augmented reality content with high sense of reality, the method capable of increasing accessibility of a mental patient and effective early diagnosis and treatment through interaction between a user and content is provided.

Description

Biofeedback method based on virtual/augmented reality contents and bio-signal for diagnosis and healing of mental illness}

The present invention relates to a technique for the diagnosis and treatment of mental disorders such as depression, ADHD, mild cognitive impairment, from children, adolescents to senior generations. More specifically, the present invention relates to a biofeedback method for early diagnosis and training healing of mental illness based on tangible virtual/augmented reality content and biosignals using a virtual/augmented reality device (HMD, etc.).

Diagnosis and treatment related to existing mental illnesses are diagnosed and treated through one-on-one face-to-face meetings with specialists by visiting specialized hospitals or counseling institutions. In addition, due to negative perception and prejudice of mental illness, time and economic problems Therefore, early diagnosis and treatment have not been made.

As a solution to this problem, a healing technique based on Internet diagnosis with high accessibility and a virtual reality system based on bio-signals have been developed. For example, the existing bio-signal-based virtual/augmented reality device is developed in the form of a biosignal display system using a worker's safety work helmet or simple augmentation technology based on location information and bio-signals. In addition, there is an existing mental health evaluation system using biosignal analysis, and after providing an audiovisual image, a mental health evaluation is performed using a separate biosignal measurement sensor and analysis system in a wired form. However, they have low realism, need to wear equipment, and require additional measuring devices, which makes them less useful.

Republic of Korea Patent Publication 10-2001-0037127 (published on May 7, 2001) Korea Registered Patent 10-1793426 (announcement date 2017. 11. 3.)

Accordingly, the present invention is an experiential virtual / using data measured through a virtual / augmented reality device capable of measuring bio-signals for the purpose of early diagnosis and healing of mental disorders for children, adolescents, seniors, etc. We propose a method of extracting effective biometric data from a user responding to augmented reality content and feeding the virtual/augmented reality content back to the subject.

According to the biofeedback method for diagnosing and healing mental disorders based on virtual/augmented reality contents and biosignals of the present invention for solving the above problems, a heartbeat acquired through a virtual/augmented reality device including a biosignal measurement sensor, It provides a virtual/augmented reality content with high-sensitivity by analyzing bio signals such as eye movements, and provides a method for effective early diagnosis and healing through interaction between the user and the content.

In addition, according to the present invention, depression based on bio-signals (biometric data) measured using a virtual/augmented reality device including a bio-signal measurement sensor to utilize depression, bio-signal analysis, pupil data analysis, etc. It provides (outputs) experiences, healing, and training contents based on biofeedback suitable for the characteristics and purposes of mental disorders such as ADHD and mild cognitive impairment.

The configuration and operation of the present invention introduced above will be further clarified through specific embodiments described with reference to the drawings.

Treatment solutions using virtual reality content (image + sound + sensational devices) exist in the past, but most are anxiety disorders and simple meditation treatment techniques, and virtual/augmented reality mental health treatment solutions are insufficient, according to the present invention. It is possible to receive the sensational interaction (motion recognition, motion analysis) for mental illness, diagnosis technology using bio signals and pupil data, and biofeedback-based healing technology through virtual/augmented reality services.

Applicability and usability can be expanded to patients who are the core target group by solving the inconvenience of wearing, such as the need to attach multiple measurement sensors and flowing down the device, and physiological augmented reality content and biofeedback-based healing training and history It becomes possible to provide life-cycle personalized management services through connection with hospitals and institutions such as initial prevention through management and prevention of recurrence.

Furthermore, it is expected that accurate diagnosis and clinical diagnosis data of mental symptoms and psychological states can be used as objective evaluation data during interviews at hospitals and institutions by analyzing and analyzing biosignal and gaze movement patterns.

1 is a conceptual explanatory diagram of a biofeedback method for diagnosing and healing mental disorders based on virtual/augmented reality content and biosignals of the present invention.
FIG. 2 is a minimum environmental configuration diagram for implementing the biofeedback method according to the concept of FIG. 1.
FIG. 3 is a configuration diagram including additional elements necessary to actually implement the biofeedback method according to the concept of FIG. 1.
4 is a detailed explanatory diagram of the processing apparatus 30.
5 is a detailed configuration diagram of the content generation unit 33 of FIG. 4
6 is a process flow diagram of the attention determining unit 34 of FIG. 4
FIG. 7 is a flow chart of the extraction process of the pupil size required to perform the process shown in FIG. 6;
8 is a configuration example of a system implementing the spirit of the present invention with hardware and/or software and other computing elements.

Advantages and features of the present invention, and a method of achieving them will become apparent by referring to embodiments described in detail with reference to the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but will be implemented in various different forms, and only the present embodiments make the disclosure of the present invention complete, and have ordinary knowledge in the technical field to which the present invention pertains. It is provided to fully inform the person of the scope of the invention, and the invention is defined by the description of the claims.

On the other hand, the terms used in this specification are for describing the embodiments and are not intended to limit the present invention. In the present specification, the singular form also includes the plural form unless otherwise specified in the phrase. As used herein, "comprises" or "comprising" means the presence of one or more other components, steps, operations and/or elements other than the components, steps, operations and/or elements mentioned, or Addition is not excluded.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. In adding reference numerals to the components of each drawing, the same components are assigned the same reference numerals as possible, even though they are shown on different drawings, and in describing the present invention, detailed descriptions of related well-known components or functions When the subject matter of the present invention can be obscured, the detailed description thereof will be omitted.

A typical embodiment for carrying out the method of the present invention will be described with reference to the drawings.

1 shows the concept of the present invention in a scenario format. The biometric module 10 is integrated in the VR/AR HMD 20 worn by the patient (user, subject) 5, and the VR/AR HMD 20 includes natural from the VR/AR stimulation unit 21. Stimulus signals such as sounds (wind, leaves, laughter, etc.) and positive visual data (pets, avatars, etc.) (stimulus signals are also generated in the form of content and will be described again) are given and the patient (5) The body responds. Accordingly, the biometric module 10 integrated into the VR/AR HMD 20-EEG (electroencephalogram), ECG (electrocardiogram), EOG (eye conduction), EMG (electromyography) sensor, etc. In 30), the biosignal is analyzed (22) and the content to be fed back to the patient 5 is fed back in VR/AR (23) (in the present invention, such feedback is referred to as a'biofeedback'). Biofeedback can induce psychological/physiological stability of the patient. Meanwhile, in order to generate biofeedback content, effective biodata generation 24 is performed through effective biovariable mapping for each patient from the analyzed biosignals. Biofeedback contents such as healing, meditation, and positive emotion induction are determined (25) from the generated effective biodata, and the contents are biofeedback by VR/AR (23). This biofeedback content is also delivered to the VR/AR stimulation unit 21 and provided to the VR/AR HMD 20 as personalized VR/AR content of the patient 5 (26). In other applications, this biofeedback content is not only provided to the patient 5, but is provided to the hospital system (e.g., a monitoring system or a diagnostic system) in the same VR/AR format or in a different signal format, such as a doctor, diagnostician, or therapist. You can make use of this.

FIG. 2 is a basic conceptual diagram of a representative embodiment for implementing the present invention, and shows a minimum environment configuration for implementing the biofeedback method according to the concept of FIG. 1. Referring to FIG. 1, a patient or a subject 5 receives virtual/augmented reality stimulation through a virtual/augmented reality (VR/AR) device 20 in which a biosignal measurement module 10 is integrated. The stimulation is activated by the VR/AR stimulation section 25. This stimulus analyzes the bio signals such as physical changes occurring in the patient 5, such as heart rate, electromyography, and gaze movements with the processing device 30, thereby generating realistic, virtual/augmented reality experiences, healing, and training contents. The first virtual/augmented reality content is provided to the virtual/augmented reality device 20 worn by the patient 5. The virtual/augmented reality device 20 in which the biosignal measurement module 10 is integrated may be implemented as a VR/AR head mounted display (HMD). On the other hand, the processing device 30 determines the second virtual/augmented reality content from the generated valid biometric data by generating valid biometric data through the effective biometric variable mapping for each subject from the biosignal, and this content is second virtual/enhanced. It is provided to the virtual/augmented reality stimulation section 25 as real content. The processing device 30 is basically an electronic computing device (computer) for performing the task of the method according to the present invention by means of a software program (and suitably associated hardware), a fixed computer device, a mobile computer device, a mobile device, It can be implemented in the form of other terminals.

FIG. 3 is a configuration diagram in which additional elements necessary for actually implementing the method according to the present invention using the basic configuration of FIG. 2 are included in the processing device 30.

The biosignal measurement module 10 includes a plurality of sensors for measuring biosignals and pupil data (pupil movement information) related to EEG (electroencephalogram), ECG (electrocardiogram), EOG (eye conduction), and EMG (electromyography), It is linked or integrated with the virtual/augmented reality device 20 such as HMD.

An interface 40 is required to interact (interact) with the biosignals and pupil data obtained from the measured EEG, ECG, EMG, and EOG, and to interact with virtual/augmented reality content for diagnosis and healing.

The processing device 30 includes an analysis unit 31 for performing sensation interaction (motion, motion recognition and analysis), biosignal analysis, pupil data analysis, and mental diseases such as depression, ADHD, and mild cognitive impairment from the analyzed biodata. An effective biometric data digitization unit 32 for quantifying effective biometric data for generating biofeedback content suitable for the characteristics and purpose of the content, and content for generating experience, healing, and training content based on biofeedback with the digitized valid biometric data Includes part 33.

Here, the effective bio-data generated by the effective bio-data digitization unit 32 includes attention, stress, happiness/depression, left-right brain ratio, and cognitive function information obtained from bio signals measured by the EEG sensor; Heart rate, SDNN, LF, HF information obtained from bio signals measured by the ECG sensor; RMS information obtained from bio signals measured by the EMG sensor; There are 12 types (including but not limited to blinking information) of blinking information obtained from bio signals measured by the EOG sensor. The effective biometric data digitization unit 32 digitizes these valid biometric data, and the content generation unit 33 generates content to which state information and character animation are applied.

The above-described analysis unit 31, effective biometric data digitization unit 32, and content generation unit 33 are implemented as software programs (and appropriately associated hardware) in the processing unit 30. The processing device 30 will be described in more detail with reference to FIG. 4.

Referring to FIG. 4, the processing unit 30 includes, in addition to the aforementioned analysis unit 31, effective bio-data digitization unit 32, and content generation unit 33, the attention determination unit 34 through user pupil size analysis And, further includes an abnormal behavior detection unit 35 using the user bio-signal and the pupil size change rate. These two parts (34, 35), in particular, by providing the results to a hospital system (for example, a monitoring system or a diagnostic system, etc.) can be used for use by doctors, diagnosticians, therapists, etc. It is not.

5 illustrates components that may be included in the content generation unit 33 shown in FIGS. 3 and 4. In order to induce psychological stability and physiological stability, a biosignal data content generator 331 capable of expressing physiological stability based on the valid biometric data generated by the effective biometric data digitization unit 32, and when the abnormal behavior is detected Safety content generator 332 for providing to the user by including warning information on it, and current attention concentration based on effective biometric data to determine attention to improve the concentration of users with ADHD, mild cognitive impairment, etc. Illustrates to include a content generator 333, including.

6 shows the process flow of the attention determining unit 34 through the analysis of the user pupil size shown in FIG. 4.

First, by analyzing the bio-signal from the EOG sensor, the average μ1 of the pupil size during the initial 10 seconds is calculated (s10).

In parallel or subsequently, the standard deviation σ1 of the pupil size for the initial 10 seconds is calculated (s20).

Then, the average μ2 of the pupil size for 5 seconds is calculated (s30).

The calculated data determines μ1±σ1≤μ2 (s40).

As a result of this judgment, the results of'attention reduction' (s50),'no change of attention' (s60), and'attention increase' (s70) are output.

7 shows a process for extracting the pupil size required for the process of FIG. 6.

First, the pupil image is taken from the module 10 included in the VR/AR HMD 20 or other means (eg, a camera). The size of the region occupied by the pupil is detected from this image. Then, a median filter is applied and an average filter is applied to calculate the slope of each frame of the pupil image. Thereafter, a zero-crossing candidate is detected to group the detected candidates, and a zero-crossing point within each group is selected. Thereby, the size of the pupil can be extracted.

8 shows an example of a system composed of specific hardware and/or software and other computing elements to implement the method of the present invention described above.

A hospital server 100 that can be implemented by a PC, a research server 200, and a mobile-based home terminal 300 are installed at each site (ie, hospital, laboratory, home). The operating system (O/S) of the hospital server 100 and the research server 200 includes a monitoring/analysis program and VR/AR content, and in particular, the hospital server 100 interoperates with a big data server platform. The user-side hardware includes a PC-based wired/wireless VR HMD 400, a wireless VR/AR HMD 500, and a mobile-based HMD 600, and there is a wireless biosignal module (sensor) 700 integrated with them. . The wireless biosignal module 700 may include a biosignal AD converter, a camera module, a noise filter, a pre-processing/signal processing module, and a wireless communication module.

These user-side hardware and servers are connected through middleware. Specifically, the PC-based HMD (400, 500) communicates via the PC-based hospital server 100 and the research server 200 and the API for the PC via the wireless bio-signal module 700, and the home mobile 300 ) Communicates with the API for Android. And the PC-based HMD (400, 500) receives content from the PC-based hospital server 100 and the research server 200.

The function or process of each component of the processing apparatus implementing the method of the present invention described above is a digital signal processor (DSP), processor, controller, application-specific IC (ASIC), programmable logic device ( FPGA, etc.), at least one of other electronic devices, and a combination of hardware elements. It can also be implemented as software in combination with hardware elements or independently, which can be stored on a record carrier.

The configuration of the present invention has been described in detail through preferred embodiments of the present invention, but those skilled in the art to which the present invention pertains are disclosed herein without changing the technical spirit or essential features of the present invention. It will be understood that the lesson may be implemented in other specific forms. It should be understood that the embodiments described above are illustrative in all respects and not restrictive. The protection scope of the present invention is defined by the claims below, rather than the detailed description, and all changes or modifications derived from the claims and equivalent concepts should be interpreted to be included in the technical scope of the present invention. .

Claims (7)

A virtual/augmented reality stimulation step of providing a stimulus signal to the subject through virtual/augmented reality;
Measuring a biosignal of the subject by measuring the biosignal of the physical change occurring in the subject by the stimulus signal;
An analysis step of performing motion recognition and motion analysis, biosignal analysis, and pupil data analysis of the subject from the biosignal;
An effective biological data digitization step of digitizing effective biological data from the analyzed biological data;
A biofeedback method for diagnosing and healing mental disorders based on virtual/augmented reality content and biosignals, comprising a content generation step of generating virtual/augmented reality content to be provided to the subject with digitized effective biodata. The biosignal measured in the biosignal measurement step is based on virtual/augmented reality content and biosignals that are at least one of EEG (electroencephalogram), ECG (electrocardiogram), EOG (eye conduction), and EMG (electromyography). Biofeedback method for diagnosis and treatment of mental illness. The method of claim 1, wherein the generated virtual/augmented reality content is a biofeedback method for diagnosing and healing mental diseases based on virtual/augmented reality content and biosignals provided by the subject and the virtual/augmented reality stimulation step. The method of claim 3, wherein the virtual/augmented reality content is a biofeedback method for diagnosing and healing mental diseases based on virtual/augmented reality content and bio signals provided by a system built in a hospital. In claim 1, wherein the analysis step
Attention determination step through the pupil size analysis of the subject,
A biofeedback method for diagnosing and healing mental disorders based on virtual/augmented reality content and biosignals, which further includes detecting abnormal behavior using the subject's biosignal and pupil size change rate. In claim 1, The content creation step
In order to induce psychological stability and physiological stability, generating biosignal data content capable of expressing physiological stability based on the effective biodata generated by the effective biodata digitization unit,
Generating safety content for providing to the user by including warning information on the abnormal behavior when it is detected,
Diagnosis of mental illness based on virtual/augmented reality content and bio-signals, including the step of generating content that includes the current concentration of attention based on the effective biological data to determine attention to improve concentration for subjects with mental illness and Biofeedback method for healing. A virtual/augmented reality stimulator that provides a stimulus signal including a content type to be provided as a virtual/augmented reality to the subject, and receives a stimulus signal from the virtual/augmented reality stimulator and provides it to the subject. As a method performed in a system including a processing device connected to the virtual / augmented reality device worn by the test subject with a module for measuring,
Analyzing a biosignal of a physical change occurring in the subject receiving the stimulus signal;
Generating first virtual/augmented reality content to be fed back to the subject based on the analyzed biosignal and providing the virtual/augmented reality device to induce psychological/physiological stability of the subject;
Determining second virtual/augmented reality content from the generated valid biological data by generating effective biological data through effective biological variable mapping for each subject from the analyzed biological signal;
A biofeedback method for diagnosing and healing mental disease based on a bio-signal sensor performing the step of providing the second virtual/augmented reality content to the virtual/augmented reality device worn by a subject and the virtual/augmented reality stimulator.

Images