KR20200117089A - The virtual reality contents control system and the virtual reality contents control method operating by the same - Google Patents

Abstract

According to an embodiment of the present invention, a virtual reality content control system can comprise: a content providing part that provides virtual reality content to a user through a head mounted display (HMD); a motion chair part that is synchronized with the virtual reality content to be driven; a bio-signal measurement part that measures a bio-signal of the user that changes in response to the virtual reality content provided to the user and the driving of the motion chair part; a bio-signal analysis part that analyzes the mental state of the user based on the measured bio-signal; and a control part that controls at least one of the content providing part and the motion chair part according to the analyzed mental state of the user. Therefore, the sense of reality can be increased by allowing the user to ride a motion chair synchronized with the virtual reality content when experiencing virtual reality.

Description

Virtual reality content control system and virtual reality content control method using the same {THE VIRTUAL REALITY CONTENTS CONTROL SYSTEM AND THE VIRTUAL REALITY CONTENTS CONTROL METHOD OPERATING BY THE SAME}

The present invention relates to a virtual reality content control system and a virtual reality content control method using the same, and more particularly, a motion chair that performs a task through virtual reality content to a user and is synchronized with the virtual reality content By measuring the bio-signals of the user while boarding in the vehicle, analyzing the measured results to determine what the user's mental state is, and according to the identified user's mental state, the type of virtual reality content, the degree of stimulation, and the motion chair The present invention relates to a virtual reality content control system in which user psychotherapy is performed by controlling the operation of a user and a virtual reality content control method using the same.

There are many people living in the modern world who are so busy that they do not feel depressed, and sometimes even though they or their family are depressed, they do not know whether they are depressed, or they do not know what symptoms the depression complains of, so they are neglected.

Various mental disorders such as depression are most often caused by a combination of psychological or social factors as well as biological factors, and various methods for treating such complex mental disorders have been proposed.

Among them, virtual reality (VR) is a trend being used for the treatment of a person suffering from phobias such as acrophobia or claustrophobia among the mental disorders. In other words, it is training to try to get out of the situation on your own by exposing people to situations or objects that create a sense of fear.

However, in using virtual reality for mental illness treatment, it is difficult to judge the progress of treatment, so it is difficult to find appropriate responses and side effects such as revising future treatment plans or changing treatment methods.

Accordingly, in order to increase the user's sense of reality with virtual reality contents, the user's motion is synchronized with the virtual reality contents and provided with motion by analyzing changes in various bio signals such as the user's blood pressure and body temperature. There is a further demand for the development of techniques for determining mental state.

1. Republic of Korea Patent Publication No. 10-2007-0012977 (published on January 30, 2007)

The present invention has been conceived in accordance with the above-described technical development demand, and in detail, it is an object of the present invention to increase the sense of reality by riding a motion chair synchronized with virtual reality content in experiencing virtual reality.

In addition, the purpose is to determine the psychological state of the user according to the bio-signal by measuring the aspect of the user's bio-signal in real time.

The technical objectives to be achieved in the present invention are not limited to the above-mentioned matters, and other technical problems that are not mentioned are to those of ordinary skill in the art from the embodiments of the present invention to be described below. Can be considered by

As an embodiment of the present invention, a virtual reality content control system may be provided.

The virtual reality content control system according to an embodiment of the present invention includes a content providing unit that provides virtual reality content to a user through a head mounted display (HMD), a motion chair unit that is driven in synchronization with the virtual reality content, A bio-signal measurement unit that measures the user’s bio-signals that change in response to the virtual reality content provided to the user and the motion of the chair unit, and a bio-signal analysis unit that analyzes the user’s mental state based on the measured bio-signals and analysis A control unit for controlling at least one of a content providing unit and a motion chair unit may be included according to the mental state of the user.

In the biosignal measurement unit of the virtual reality content control system according to an embodiment of the present invention, a first sensor module for measuring a user's blood flow and heart rate variability, a second sensor module for measuring a user's body temperature, humidity, and impedance, and A third sensor module for detecting user's motion information may be included.

In the virtual reality content control system according to an embodiment of the present invention, a first sensor module, a second sensor module, and a third sensor module are provided in at least one of a leg portion, an armrest portion, and a belt portion of the motion chair. Can be arranged in shape.

In the biosignal analysis unit of the virtual reality content control system according to an embodiment of the present invention, a significance index for each mental state may be calculated based on at least one of an average, a standard deviation, and a correlation coefficient for the measured biosignal.

In the biosignal analysis unit of the virtual reality content control system according to an embodiment of the present invention, preprocessing is performed on the measured biosignal, and noise is filtered using a component analysis algorithm on the preprocessed biosignal. The significance index may be calculated based on a probability position (z-score) on a normal distribution extracted based on at least one of an average, a standard deviation, and a correlation coefficient for the obtained biosignal.

In the virtual reality content control system according to an embodiment of the present invention, the biological signal is measured using an electroencephalogram (EEG) signal generated from the user's brain and a functional near-infrared spectroscopy (fNIRS). The user's brain activation signal is further included, and the head-mounted display (HMD) further includes a wearing unit to be worn on the user's head, and the wearing unit includes an electroencephalogram sensing module for measuring an EEG signal, and a brain activation signal. A near-infrared emission module and a near-infrared detection module may be disposed at a predetermined position.

In the controller of the virtual reality content control system according to an embodiment of the present invention, the content providing unit may be controlled based on a control model previously learned according to a deep learning algorithm for the virtual reality content corresponding to the analyzed user's mental state. .

As an embodiment of the present invention, a virtual reality content control method using a virtual reality content control system may be provided.

A virtual reality content control method according to an embodiment of the present invention includes providing virtual reality content to a user through a head mounted display (HMD), synchronizing with the virtual reality content and driving the motion chair, the user The step of measuring the user's bio-signals that change in response to the virtual reality content provided to the user and the motion of the motion chair, the step of analyzing the user's psychological state based on the measured bio-signal, and the analyzed user's psychological state. The step of controlling at least one of providing virtual reality content and driving the motion chair may be included.

In the step of analyzing the user's mental state of the virtual reality content control method according to an embodiment of the present invention, a significance index for each mental state is calculated based on at least one of the average, standard deviation, and correlation coefficient of the measured bio-signal. Can be.

In the step of analyzing the user's psychological state of the virtual reality content control method according to an embodiment of the present invention, the pre-processing of the measured bio-signals is performed, and the pre-processed bio-signals are subjected to noise using a component analysis algorithm. Is filtered, a step of calculating a significance index based on a probability position (z-score) on a normal distribution extracted based on at least one of an average, a standard deviation, and a correlation coefficient for the filtered biosignal. .

In the virtual reality content control method according to an embodiment of the present invention, in the step of controlling at least one of providing virtual reality content and driving the motion chair according to the analyzed user's mental state, the analyzed user's mental state The provision of contents may be controlled based on a control model previously learned according to a deep learning algorithm with respect to the virtual reality contents corresponding to.

As an embodiment of the present invention, a computer-readable recording medium in which a program for implementing the above-described method is recorded may be provided.

According to the virtual reality content control system provided as an embodiment of the present invention, in experiencing virtual reality, it is possible to increase the sense of reality by allowing the user to ride a motion chair synchronized with the virtual reality content.

In addition, by measuring the aspect of the user's bio-signal in real time and accurately determining the user's psychological state according to the bio-signal, virtual reality content and motion chair control for reducing the user's depression may be performed.

1 is an exemplary view showing a virtual reality content control system according to an embodiment of the present invention.
2 is an exemplary view showing a motion chair part of a virtual reality content control system according to an embodiment of the present invention.
3 is an exemplary view showing a form in which sensor modules are arranged in a virtual reality content control system according to an embodiment of the present invention.
4 is an exemplary diagram showing a state in which a user's brain activation signal is measured through functional near-infrared spectroscopy (fNIRS) in a virtual reality content control system according to an embodiment of the present invention.
5 is an exemplary view showing a state in which an electroencephalogram sensing module, a near-infrared emission module, and a near-infrared detection module are disposed on a head-mounted display (HMD) in a virtual reality content control system according to an embodiment of the present invention.
6 is an exemplary view showing virtual reality content provided by a content provider of a virtual reality content control system according to an embodiment of the present invention.
7 and 8 are flowcharts illustrating a virtual reality content control method according to an embodiment of the present invention.

The terms used in the present specification will be briefly described, and the present invention will be described in detail.

The terms used in the present invention have been selected from general terms that are currently widely used while considering functions in the present invention, but this may vary depending on the intention or precedent of a technician working in the field, the emergence of new technologies, and the like. In addition, in certain cases, there are terms arbitrarily selected by the applicant, and in this case, the meaning of the terms will be described in detail in the description of the corresponding invention. Therefore, the terms used in the present invention should be defined based on the meaning of the term and the overall contents of the present invention, not a simple name of the term.

When a part of the specification is said to "include" a certain component, it means that other components may be further included rather than excluding other components unless otherwise stated. In addition, terms such as "... unit" and "module" described in the specification mean units that process at least one function or operation, which may be implemented as hardware or software, or as a combination of hardware and software. In addition, when a part of the specification is said to be "connected" with another part, this includes not only the case that it is "directly connected", but also the case that it is connected "with another configuration in the middle." .

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those of ordinary skill in the art can easily implement the present invention. However, the present invention may be implemented in various forms and is not limited to the embodiments described herein. In the drawings, parts irrelevant to the description are omitted in order to clearly describe the present invention, and similar reference numerals are assigned to similar parts throughout the specification.

Hereinafter, the present invention will be described in detail with reference to the accompanying drawings.

As an embodiment of the present invention, a virtual reality content control system may be provided.

1 is an exemplary view showing a virtual reality content control system according to an embodiment of the present invention, Figure 2 is an exemplary view showing a motion chair 200 of a virtual reality content control system according to an embodiment of the present invention .

1 and 2, a virtual reality content control system according to an embodiment of the present invention includes a content providing unit 100 that provides virtual reality content to a user through a head mounted display (HMD), The motion chair unit 200 that is driven in synchronization with the virtual reality content, the virtual reality content provided to the user, and the bio-signal measurement unit 300 that measures the user's bio-signals that change in response to the driving of the motion chair unit 200 , Controls at least one of the biosignal analysis unit 400 for analyzing the user's psychological state based on the measured biosignal and the content providing unit 100 and the motion chair 200 according to the analyzed user's psychological state A control unit 500 may be included.

In the present invention, the term'user' includes not only patients complaining of mental disorders such as psychological anxiety or depression, but also general people who do not suffer from the above mental disorders but are suffering from mental pressure such as stress or who want to classify their psychological state. I can. That is, the biosignal analysis system of the present invention is not a medical system or treatment system for a patient suffering from mental illness, but a health management system for measuring and analyzing an individual's biological phenomena or providing medical information for daily health management. Or it may correspond to a wellness (wellness) system.

In the content providing unit 100 according to an embodiment of the present invention, virtual reality content is provided to a user through a head mounted display (HMD), and the head mounted display (HMD) is worn like goggles to display an image. It refers to a monitor that can be displayed and is not limited to the shape of the head mounted display 20 shown in the drawings of the present invention, and may be formed in various shapes. Virtual reality content can refer to content that enables a person to experience a real life in a virtual world created with a computer, and the virtual reality content is a single image by superimposing a 3D virtual image on a real image or background. Augmented Reality (AR) content shown as may also be included.

In the virtual reality content control system according to an embodiment of the present invention, by providing a virtual scenario based on the user's questionnaire data in the category of the virtual reality content, diagnostic content to identify the user's mental state factor, the user's mental state In addition to the stability content for stabilizing and guiding content for allowing the user to recognize the psychological state factor on their own, integrated content in which all of the diagnostic content, stability content, and induction content are combined may be further included.

The diagnostic content may include content for allowing the user to experience events or environmental factors that may cause stress to the user based on, for example,'Holmes-Rahe Life Stress Inventory' through virtual reality content. .

6 is an exemplary view showing virtual reality content provided by the content providing unit 100 of the virtual reality content control system according to an embodiment of the present invention.

Referring to FIG. 6, the stability content may include all contents that can be expressed as virtual reality content such as nature appreciation content, performance appreciation content, leisure experience content, etc. to relieve the user's stress or stabilize the user's mental state. There is no limit to the type. For example, the nature appreciation content may be provided with nature appreciation content for allowing a user to appreciate a natural landscape such as sea, river, forest, and grassland. The nature appreciation content is based on virtual reality, and allows the user to feel a three-dimensional effect and a sense of reality as if they actually fly over the sea or in a forest. In addition, the nature appreciation content may be provided with natural sound such as wind sound and wave sound in accordance with the virtual reality image. Through the performance appreciation content, a user's tension may be relieved or a user's psychological stability may be experienced through virtual reality. The performance appreciation content may include virtual reality content for appreciating various performances such as an orchestra, concert, concert, and magic performance. In addition, the leisure experience contents may include various virtual reality leisure contents such as paragliding, scuba diving, and snorkeling.

The content for induction may include content that can be induced so that the user can perceive a factor of depression or anxiety and see the user objectively. In other words, content to encourage users to actively participate in recognizing factors that cause depression or anxiety of users, whether physical or psychological, as well as internal environmental factors such as external environmental factors, genetics, aging, and trauma are included. I can.

The motion chair part 200 of the virtual reality content control system according to an embodiment of the present invention supports and supports the user's body, but is synchronized with the virtual reality content so that the user can be in the virtual reality space at the actual site (Ex. It may refer to a device or device including an actuator module for various movements in order to deliver experiences that you see and feel in nature, performance halls, etc.). The actuator part is a roll for rotating the motion chair part 200 forward and backward (x-axis), a pitch for rotating a horizontal axis (y-axis), and a vertical rotation (z-axis). A yaw to perform, a surge for an anteroposterior (x-axis) forward and backward reciprocating motion, a sway for a horizontal axis (y-axis) left and right reciprocating motion, and a heave for up and down reciprocating motions on the vertical axis (z-axis) It can be driven by (heave) operation. In addition, the motion chair unit 200 may be vibrated according to virtual reality content. That is, in the motion chair part 200 according to an embodiment of the present invention, the motion chair part 200 is rolled, pitched, yaw, and surged according to each scene of virtual reality content. ), sway, heave, and vibration can be synchronized with the content and driven.

The biosignal measurement unit 300 of the virtual reality content control system according to an embodiment of the present invention includes a first sensor module for measuring a user's blood flow and heart rate variability, and a second sensor module for measuring a user's body temperature, humidity, and impedance. A sensor module and a third sensor module for detecting user motion information may be included. In addition, the virtual reality content control system may further include a database (not shown) for storing data measured through the first sensor module, the second sensor module, and the third sensor module.

The first sensor module refers to a photoplethysmography sensor module (PPG sensor module) for measuring not only blood flow and heart rate variability, but also heart rate output, oxygen carrying capacity, blood pressure, blood vessel elasticity, blood flow rate and oxygen saturation. can do. In the second sensor module, not only the temperature, humidity and skin impedance of the user's skin, but also skin conductivity may be measured. The third sensor module is a sensor module for measuring a change in the moving speed and acceleration of the user's body according to the drive of the motion chair 200, and the third sensor module is an acceleration sensing module, a gyroscope, and It may refer to an inertial measurement unit (IMU) including a geomagnetic sensing module.

That is, in the present specification, the biosignal may refer to the user's optical blood flow (PPG), skin temperature, skin impedance, body motion information, etc. measured by the aforementioned first to third sensor modules. In addition, as will be described later, brain waves (EEG) and brain activity signals may be further included in the bio signals.

In the virtual reality content control system according to an embodiment of the present invention, at least one of the leg portion, the armrest portion 50, and the belt portion 40 of the motion chair portion 200 includes a first sensor module and a second sensor. The module and the third sensor module may be arranged in a predetermined shape.

That is, as shown in FIG. 2, sensor modules for measuring a user's bio-signals are provided at a predetermined position of the motion chair 200. In particular, sensor modules arranged in a predetermined shape are provided in a structure (leg part, armrest part, belt part, etc.) of the motion chair part 200 in contact with the user's body.

Specifically, the leg portion is a portion that supports the user's foot by providing the footrest 60 when the user is seated in the motion chair 200, and the footrest 60 in contact with the user's foot includes the aforementioned first sensor module and the first sensor module. The 2 sensor module and the third sensor module may be disposed in a predetermined shape and embedded. Also, in the armrest part 50 in contact with the user's arm, the first sensor module, the second sensor module, and the third sensor module may be disposed in a predetermined shape and embedded. A third sensor module for measuring a change in a moving speed and acceleration of the user's body according to the driving of the motion chair 200 may be attached to the belt unit 40 in contact with the user's abdomen.

3 is an exemplary view showing a form in which sensor modules are arranged in a virtual reality content control system according to an embodiment of the present invention. As shown in (a) and (b) of Fig. 3, the first sensor module, the second sensor module, and the third sensor module have a structure such as a leg part, an armrest part 50, and a belt part of the motion chair part 200. They can be arranged in different patterns depending on the situation. For example, a first sensor module, a second sensor module, and a third sensor module may be arranged in five rows as shown in FIG. 3 (a) on the footrest 60 of the leg portion of the motion chair part 200. Unlike this, the armrest part 50 of the motion chair part 200 may be provided in a form in which a first sensor module, a second sensor module, and a third sensor module are arranged in three rows as shown in FIG. 3(b). .

Hereinafter, a description will be given of a content of analyzing a user's psychological state by the bio-signal analysis unit 400 of the present virtual reality content control system. Through the analysis process described below, the user's psychological state is not only normal, depression, and anxiety, but also agoraphobia, flight phobia, obstructive phobia, acrophobia, panic disorder, schizophrenia, post-traumatic stress disorder, and addiction. , Insomnia, anorexia, bulimia, and obsessive-compulsive disorder can be classified into various psychological states.

In the biosignal analysis unit 400 of the virtual reality content control system according to an embodiment of the present invention, a significance index for each psychological state may be calculated based on at least one of an average, a standard deviation, and a correlation coefficient for the measured biosignal. I can.

In addition, in the biosignal analysis unit 400 of the virtual reality content control system according to an embodiment of the present invention, a preprocessing unit that performs preprocessing on the measured biosignals, and a component analysis algorithm on the biosignals subjected to preprocessing are used. Significance in which a significance index is calculated based on a probability position (z-score) on the normal distribution extracted based on at least one of the noise filtering unit, the average of the filtered bio-signals, the standard deviation, and the correlation coefficient. An index calculation unit may be further included.

In the preprocessing unit, components other than the biosignal frequency are performed through frequency filtering processes such as a low pass filter (LPF) and a high pass filter (HPF) in the frequency domain by using Fourier transform for each biosignal measured by the user. (Ex. other physiological phenomena, signals for sensor module operation, etc.) can be filtered.

In the noise filtering unit, noise that may be additionally included in the biosignal may be filtered using a component analysis algorithm such as an independent component analysis (ICA) on the preprocessed biosignal.

In the significance index calculation unit, the significance index for each situation can be calculated by calculating the jet score (z-score) for the biosignals for each time-series situation (ex. the first situation, the second situation ??the nth situation, etc.). However, as the value of the calculated significance index for each situation is smaller, it may be determined that the degree of significance with the corresponding situation is higher.

Hereinafter, a process in which a significance index is calculated for a user biosignal that is preprocessed and filtered by the preprocessing unit and the noise filtering unit and transmitted to the significance index calculation unit will be described as an example.

(i= 1, 2, … , n)와 같이 표현할 수 있다.

는 사용자가 가상현실 컨텐츠를 체험하는 시간을 지칭하며,

는 상기

시간 동안 사용자의 생체신호가 본 발명의 생체신호측정부(300)에서의 기록될 때의 시간을 지칭할 수 있고, 아래와 같이 [수학식 1]로 표현할 수 있으며, k= 1, 2, …, m 이다.The biosignal of the user for each of the time series situations (Ex. the first situation, the second situation ??the nth situation, etc.)

It can be expressed as (i = 1, 2,…, n).

Refers to the time when a user experiences virtual reality content,

Is reminded

During the time, it may refer to the time when the user's biosignal is recorded in the biosignal measurement unit 300 of the present invention, and can be expressed as [Equation 1] as follows, k = 1, 2, ... , m.

[Equation 1]

)은 다음의 [수학식 2]와 같이 나타낼 수 있다.Here, the average of the biosignals in the first to nth situations (

) Can be expressed as the following [Equation 2].

[Equation 2]

) 및 제 1 상황 내지 제 n 상황에서의 생체신호 간 상관계수(

)는 각각 다음의 [수학식 3] 및 [수학식 4]와 같이 나타낼 수 있다.In addition, the standard deviation of the biosignal in the first to nth situations (

) And the correlation coefficient between the biosignals in the first to nth situations (

) Can be expressed as the following [Equation 3] and [Equation 4], respectively.

[Equation 3]

[Equation 4]

Hereinafter, a jet score (z-score) calculated differently according to the average, standard deviation, and correlation coefficient of the biological signal will be described.

First, a first jet score calculated from the average is derived.

)에 대한 평균은

와 같이 나타낼 수 있고, 데이터베이스에 미리 저장되어 있는 기준 상황 별(제 1 상황 내지 제 p 상황) 생체신호의 평균(

)에 대한 표준편차는

와 같이 나타낼 수 있다. 상기 평균 및 표준편차를 이용하면 제 1 제트스코어는 아래 [수학식 5]와 같이 나타낼 수 있다.By reference situation stored in the database in advance (the first situation to the p situation) Average of biosignals (

) Is the average

And the average of the biosignals for each reference situation (the first situation to the pth situation) stored in the database in advance (

The standard deviation for) is

It can be expressed as Using the mean and standard deviation, the first jet score can be expressed as [Equation 5] below.

[Equation 5]

)는 전술한 바와 같이 본 발명의 생체신호측정부(300)에서 측정된 제 1 상황 내지 제 n 상황 별 생체신호(

)의 평균에 기초하여 도출된 기준 상황 별 표준점수를 나타낸다.In the above [Equation 5], the first jet score (

) Is a biosignal for each of the first to nth situations measured by the biosignal measurement unit 300 of the present invention as described above (

Shows the standard score for each standard situation derived based on the average of ).

Next, a second jet score calculated from the standard deviation is derived.

)에 대한 평균은

와 같이 나타낼 수 있고, 데이터베이스에 미리 저장되어 있는 기준 상황 별(제 1 상황 내지 제 p 상황) 생체신호의 표준편차(

)에 대한 표준편차는

와 같이 나타낼 수 있다. 상기 평균 및 표준편차를 이용하면 제 2 제트스코어는 아래 [수학식 6]와 같이 나타낼 수 있다.The standard deviation of the biosignal for each reference situation (the first situation to the pth situation) stored in the database in advance (

) Is the average

The standard deviation of the biosignal for each reference situation (the first situation to the pth situation) and stored in the database in advance (

The standard deviation for) is

It can be expressed as Using the mean and standard deviation, the second jet score can be expressed as [Equation 6] below.

[Equation 6]

)는 전술한 바와 같이 본 발명의 생체신호측정부(300)에서 측정된 제 1 상황 내지 제 n 상황 별 생체신호(

)의 표준편차에 기초하여 도출된 기준 상황 별 표준점수를 나타낸다.In the above [Equation 6], the second jet score (

) Is a biosignal for each of the first to nth situations measured by the biosignal measurement unit 300 of the present invention as described above (

Shows the standard score for each standard situation derived based on the standard deviation of ).

Next, a third jet score calculated from the correlation coefficient is derived.

)에 대한 평균은

와 같이 나타낼 수 있고, 데이터베이스에 미리 저장되어 있는 기준 상황 별(제 1 상황 내지 제 p 상황) 생체신호의 상관계수(

)에 대한 표준편차는

와 같이 나타낼 수 있다. 상기 평균 및 표준편차를 이용하면 제 3 제트스코어는 아래 [수학식 7]와 같이 나타낼 수 있다.Correlation coefficient of bio-signals for each reference situation (1st situation to pth situation) stored in the database in advance (

) Is the average

Correlation coefficient of bio-signals for each reference situation (the first situation to the p-th situation) that can be expressed as and stored in the database in advance (

The standard deviation for) is

It can be expressed as Using the mean and standard deviation, the third jet score can be expressed as shown in [Equation 7] below.

[Equation 7]

)는 전술한 바와 같이 본 발명의 생체신호측정부(300)에서 측정된 제 1 상황 내지 제 n 상황 별 생체신호(

)의 상관계수에 기초하여 도출된 기준 상황 별 표준점수를 나타낸다.In the above [Equation 7], the third jet score (

) Is a biosignal for each of the first to nth situations measured by the biosignal measurement unit 300 of the present invention as described above (

Shows the standard score for each standard situation derived based on the correlation coefficient of ).

), 제 2 제트스코어(

) 및 제 3 제트스코어(

)에 기초하여 상황(제 1 상황 내지 제 p 상황) 별 유의성 지수 값이 다음의 [수학식 8]과 같이 도출될 수 있다.The first jet score calculated above (

), 2nd Jet Score (

) And the third jet score (

), the significance index value for each situation (the first situation to the pth situation) may be derived as shown in [Equation 8] below.

[Equation 8]

)는 상기 [수학식 8]과 같이 제 1 제트스코어(

), 제 2 제트스코어(

) 및 제 3 제트스코어(

) 합의 평균을 상황 별로 도출한 값에 해당할 수 있다. 전술한 바와 같이 유의성 지수는 그 값이 작을수록 해당 상황과의 유의성 정도가 높다고 할 수 있으며, 유의성 지수를 통해 사용자의 심리상태가 판단될 수 있다.That is, the significance index (

) Is the first jet score as shown in [Equation 8] above (

), 2nd Jet Score (

) And the third jet score (

) It may correspond to the value derived from the average of the sum for each situation. As described above, the smaller the value of the significance index is, the higher the degree of significance with the corresponding situation is, and the user's psychological state can be determined through the significance index.

In addition, as described below, the biosignal may further include an electroencephalogram (EEG) signal and a brain activation signal, and the biosignal is a user's optical blood flow (PPG) measured by the above-described first to third sensor modules, It can be divided into a first biosignal representing skin temperature, skin impedance, body motion information, etc., and an EEG signal and a second biosignal representing brain activation signal, which will be described later, and similarly to the second biosignal, the significance index as described above is Procedures to obtain (Ex. pre-processing, noise filtering, significance index calculation, etc.) can be applied.

) 및 제 2 생체신호에 기초하여 도출된 제 2 유의성 지수(

,

)를 지칭할 수 있다. 상기 제 1 유의성 지수(

) 및 제 2 유의성 지수(

)에 따르면, 사용자의 심리상태가 제 2 생체신호(뇌전도 및 뇌 활성 신호)와 유의한지 여부가 판단될 수 있다. 다시 말하면, 현재 사용자의 심리상태와 뇌 활성 신호와 같은 뇌와 관련하여 측정된 신호와 어떠한 연관 관계가 있는지 판단될 수 있다.That is, the significance index is the first significance index derived by the above-described process based on the first biosignal (

) And a second significance index derived based on the second biosignal (

,

) May be referred to. The first significance index (

) And the second significance index (

According to ), it may be determined whether the user's psychological state is significant with the second biological signal (electroencephalogram and brain activation signal). In other words, it may be determined whether there is a relationship between the current user's psychological state and a signal measured in relation to the brain, such as a brain activation signal.

According to the significance index value for each situation derived through the above process, the controller 500 may change the type, level, time, and period of the virtual reality content or control the motion of the motion chair 200.

In addition, the control unit 500 controls the operation of the virtual reality content and motion chair to be provided as described above, so that the biosignal analysis unit 400 provides the user's State change can be determined.

That is, when the feedback biosignal or significance index changes by a predetermined reference ratio compared to the biosignal or significance index at the previous time, it can be further confirmed that the user's condition has improved or deteriorated. For example, if it is determined that the biosignal measured at the time t2 changes by about 15% compared to the biosignal of the user measured at the time t1, it may be determined that the user's state has changed compared to the time t1. However, the above 15% ratio is only exemplary and is not limited thereto.

5 is an exemplary view showing a state in which an electroencephalogram sensing module, a near-infrared emission module, and a near-infrared detection module are disposed on a head-mounted display (HMD) in a virtual reality content control system according to an embodiment of the present invention.

Referring to FIG. 5, in the virtual reality content control system according to an embodiment of the present invention, bio-signals include an electroencephalogram (EEG) signal generated from a user's brain and a functional near-infrared spectroscopy (fNIRS). ), the user's brain activation signal measured using) is further included, the head-mounted display (HMD) further includes a wearing part 21 for being worn on the user's head, and the wearing part for measuring an EEG signal An EEG sensing module, a near-infrared emission module for measuring a brain activation signal, and a near-infrared detection module may be disposed at predetermined positions. That is, the biosignal measurement unit 300 of the present invention may further include an EEG sensing module, a near-infrared emission module, and a near-infrared detection module.

As described above, the head mounted display HMD may be formed in various forms. Similarly, the wearing part 21 of the head-mounted display (HMD) for being worn on the user's head and fixing the head-mounted display 20 on the head may also be formed in various shapes such as a band or a helmet shape. Referring to FIG. 5, the wearing part 21 is a band part surrounding the user's back head and a display part that provides virtual reality content to a user of a head mounted display (HMD) through a portion of the band part through the top of the user's head. It can be formed to extend to the part.

At least one near-infrared emission module 221, a near-infrared detection module 222, and an electroencephalogram sensing module may be disposed in a predetermined position in the wearing part 21 in contact with the user's head. The predetermined location may include a location for allowing the near-infrared emission module 221 and the near-infrared detection module 222 of the user's head to extract a brain activation signal from the user's frontal lobe region.

In the electroencephalogram sensing module, the electroencephalogram (EEG) may correspond to an electrical record recorded by inducing a potential change occurring in a person's cerebrum or a brain current caused by the potential change on the scalp (頭皮). The EEG sensing module may further include a plurality of EEG channels 225 and a signal amplifier and a filter for minimizing the influence of noise.

5, a near-infrared emission module 221, a near-infrared detection module 222 and an electroencephalogram of the biosignal measurement unit 300 of the present invention in a part of the wearing part 21 of the head mounted display 20 225 is arranged in a predetermined shape.

Specifically, the near-infrared emission module 221 and the near-infrared detection module 222 are disposed to cross each other to form two lines. That is, there are two lines in which the near-infrared emission module 221 and the near-infrared detection module 222 are alternately arranged one by one, and the near-infrared emission module 221 and the near-infrared detection module 222 are alternately arranged from each of the two lines. Has been. In other words, a rectangular arrangement module 25 formed by a dotted line in FIG. 4 is shown. On the illustrated arrangement module 25, the near-infrared emission module 221 and the near-infrared detection module 222 are alternately arranged to be the same module. They are arranged so that they are located diagonally. In addition, the electroencephalogram channel 225 is disposed at a predetermined interval between the two lines.

1) 및 근적외선 파장(

2) 모두 방출될 수 있다. 상기 근적외선 방출 모듈(221)에서는 적외선 파장(

1)이 방출되고 근적외선 파장(

2)이 방출될 수 있다. 예를 들면, 도 4의 배치모듈(25) 상에서 근적외선 방출 모듈(L1, L2)로부터 빛이 조사되는 순서는 'L1(

1) -> L1(

2) -> L2(

1) -> L2(

2) -> Dark -> L1(

1) -> ...' 또는 'L1(

1) -> L1(

2) -> L2(

1) -> L2(

2) -> L1(

1) -> L1(

2) -> L2(

1) -> L2(

2) -> Dark -> L1(

1) -> ...'와 같이 방출될 수 있다. 근적외선 검출 모듈(D1, D2)는 'Dark'인 경우를 제외하고 근적외선 방출 모듈(L1 및 L2)로부터 방출된 빛을 검출할 수 있다. 즉, 근적외선 방출 모듈(221)로부터 빛이 방출되지 않은 경우(Dark) 근적외선 검출 모듈(222)에서는 빛을 검출할 수 없다. 본 발명의 생체신호측정부(300)에서 근적외선 방출 모듈(221), 근적외선 검출 모듈(222) 및 뇌전도 채널(225)의 배치 형태는 도 5에 제한되지 않으며, 배치 간격도 다양하게 정해질 수 있다.A process in which light is irradiated based on the above-described arrangement module 25 will be described. One arrangement module 25 includes two near-infrared emission modules 221 and two near-infrared detection modules 222. Each near-infrared emission module 221 may emit not only near-infrared wavelengths, but also dual wavelengths in a wavelength region including infrared rays. That is, in the near-infrared emission module 221, the infrared wavelength (

1) and near-infrared wavelength (

2) All can be released. In the near-infrared emission module 221, the infrared wavelength (

1) is emitted and the near-infrared wavelength (

2) can be released. For example, the order in which light is irradiated from the near-infrared emission modules L1 and L2 on the arrangement module 25 of FIG. 4 is'L1 (

1) -> L1(

2) -> L2(

1) -> L2(

2) -> Dark -> L1(

1) ->...'or'L1(

1) -> L1(

2) -> L2(

1) -> L2(

2) -> L1(

1) -> L1(

2) -> L2(

1) -> L2(

2) -> Dark -> L1(

1) ->...' can be emitted. The near-infrared detection modules D1 and D2 may detect light emitted from the near-infrared emission modules L1 and L2 except for the case of'Dark'. That is, when light is not emitted from the near-infrared emission module 221 (Dark), the near-infrared detection module 222 cannot detect light. The arrangement of the near-infrared emission module 221, the near-infrared detection module 222, and the EEG channel 225 in the biosignal measurement unit 300 of the present invention is not limited to FIG. .

4 is an exemplary diagram showing a state in which a user's brain activation signal is measured through functional near-infrared spectroscopy (fNIRS) in a virtual reality content control system according to an embodiment of the present invention.

Referring to FIG. 4, FIG. 4 shows a functional near-infrared spectroscopy (fNIRS) for measuring a user's brain activity signal. It may refer to a biosignal measurement method in which a local brain activation signal can be extracted using the difference in absorption rate. In other words, it is possible to obtain hemodynamic information on neuronal activity and brain activity in the brain through blood oxygen saturation according to the near-infrared absorption rate through functional near-infrared spectroscopy (fNIRS). As shown in FIG. 4, the near-infrared ray emission module 221 can emit near-infrared rays to the user's brain, and the near-infrared ray detection module 222 detects near-infrared rays that are emitted from the near-infrared emission module 221 and penetrated the user's brain. can do.

In addition, the above-described bio-signal analysis unit 400 analyzes the user's psychological state may be equally applied to a bio-signal further including a brain activation signal and an EEG signal.

In the control unit 500 of the virtual reality content control system according to an embodiment of the present invention, a content providing unit based on a control model previously learned according to a deep learning algorithm for the virtual reality content corresponding to the analyzed user's mental state. 100) can be controlled.

That is, in order to determine the most suitable virtual reality content for the user's mental state, the control unit 500 learns a control model using a deep learning algorithm for the virtual reality content that causes a change in the mental state, and based on the learned control model. Thus, optimal virtual reality content can be provided.

As an embodiment of the present invention, a virtual reality content control method using a virtual reality content control system may be provided. In connection with the virtual reality content control method described below, the same contents as those of the virtual reality content control system described above are omitted.

7 is a flowchart illustrating a method of controlling virtual reality content according to an embodiment of the present invention.

Referring to FIG. 7, in a virtual reality content control method according to an embodiment of the present invention, a virtual reality content is provided to a user through a head mounted display (HMD), and motion is synchronized with the virtual reality content. The step of driving the magnetic part 200, the step of measuring the user's bio-signals changed in response to the operation of the virtual reality content and motion chair part 200 provided to the user, and the user's psychological state based on the measured bio-signals It may include a step of controlling at least one of providing virtual reality content and driving the motion chair 200 according to the analyzing step and the analyzed mental state of the user.

In the step of analyzing the user's mental state of the virtual reality content control method according to an embodiment of the present invention, a significance index for each mental state is calculated based on at least one of the average, standard deviation, and correlation coefficient of the measured bio-signal. Can be.

8 is a flowchart illustrating a method of controlling virtual reality content according to an embodiment of the present invention.

Referring to FIG. 8, in the step of analyzing the user's mental state of the virtual reality content control method according to an embodiment of the present invention, a step of performing pre-processing on a measured bio-signal, and a component of the pre-processed bio-signal The noise is filtered using an analysis algorithm, and a significance index is calculated based on a probability position (z-score) on the normal distribution extracted based on at least one of the average, standard deviation, and correlation coefficient of the filtered biosignal. May include steps to become.

In the virtual reality content control method according to an embodiment of the present invention, in the step of controlling at least one of providing virtual reality content and driving the motion chair 200 according to the analyzed user's mental state, the analyzed user The provision of contents may be controlled based on a control model learned in advance according to a deep learning algorithm for virtual reality contents corresponding to the mental state of.

Meanwhile, as an embodiment of the present invention, a computer-readable recording medium in which a program for implementing the above-described method is recorded may be provided.

In addition, the above-described method can be written as a program that can be executed on a computer, and can be implemented in a general-purpose digital computer that operates the program using a computer-readable medium. Further, the structure of the data used in the above-described method may be recorded on a computer-readable medium through various means. A recording medium for recording executable computer programs or codes for performing various methods of the present invention should not be understood as including temporary objects such as carrier waves or signals. The computer-readable medium may include a storage medium such as a magnetic storage medium (eg, ROM, floppy disk, hard disk, etc.), and an optical reading medium (eg, CD-ROM, DVD, etc.).

The above description of the present invention is for illustrative purposes only, and those of ordinary skill in the art to which the present invention pertains will be able to understand that it can be easily modified into other specific forms without changing the technical spirit or essential features of the present invention. will be. Therefore, it should be understood that the embodiments described above are illustrative and non-limiting in all respects. For example, each component described as a single type may be implemented in a distributed manner, and similarly, components described as being distributed may also be implemented in a combined form.

The scope of the present invention is indicated by the claims to be described later rather than the detailed description, and all changes or modified forms derived from the meaning and scope of the claims and their equivalent concepts should be interpreted as being included in the scope of the present invention. .

20: head mounted display 40: belt portion
50: armrest part 60: footrest
100: content providing unit 200: motion chair unit
300: biosignal measurement unit 400: biosignal analysis unit
500: control unit

Claims (12)

In the virtual reality content control system,
A content providing unit that provides virtual reality content to a user through a head mounted display (HMD);
A motion chair unit driven in synchronization with the virtual reality content;
A bio-signal measuring unit for measuring a bio-signal of a user that is changed in response to the virtual reality content provided to the user and driving the motion chair;
A biosignal analysis unit for analyzing the user's psychological state based on the measured biosignal; And
A virtual reality content control system comprising: a control unit for controlling at least one of the content providing unit and the motion chair unit according to the analyzed psychological state of the user.
The method of claim 1,
The bio-signal measuring unit includes: a first sensor module for measuring the user's blood flow and heart rate variability;
A second sensor module for measuring temperature, humidity, and impedance of the user's body; And
Virtual reality content control system including; a third sensor module for detecting the user's motion information.
The method of claim 2,
A virtual reality content control system in which the first sensor module, the second sensor module, and the third sensor module are arranged in a predetermined shape in at least one of the leg portion, the armrest portion, and the belt portion of the motion chair portion.
The method of claim 1,
The bio-signal analysis unit calculates a significance index for each psychological state based on at least one of an average, a standard deviation, and a correlation coefficient for the measured bio-signal.
The method of claim 4,
In the biosignal analysis unit, preprocessing is performed on the measured biosignal, noise is filtered using a component analysis algorithm for the biosignal subjected to the preprocessing, and the average, standard deviation and correlation of the filtered biosignal A virtual reality content control system in which the significance index is calculated based on a probability position (z-score) on a normal distribution extracted based on at least one of the coefficients.
The method of claim 1,
The biosignal further includes an electroencephalogram (EEG) signal generated from the user's brain and a brain activity signal of the user measured using a functional near-infrared spectroscopy (fNIRS),
The head mounted display (HMD) further includes a wearing part for being worn on the user's head, and the wearing part includes an electroencephalogram sensing module for measuring the EEG signal, a near-infrared emission module for measuring the brain activation signal, and Virtual reality content control system in which the near-infrared ray detection module is disposed at a predetermined position.
The method of claim 1,
The virtual reality content control system in which the content provider controls the virtual reality content corresponding to the analyzed user's mental state based on a control model previously learned according to a deep learning algorithm.
In the virtual reality content control method using a virtual reality content control system,
Providing virtual reality content to a user through a head mounted display (HMD);
Driving the motion chair unit in synchronization with the virtual reality content;
Measuring a bio-signal of a user that changes in response to the virtual reality content provided to the user and driving of the motion chair;
Analyzing the psychological state of the user based on the measured biometric signal; And
Controlling at least one of providing the virtual reality content and driving the motion chair according to the analyzed psychological state of the user.
The method of claim 8,
In the step of analyzing the user's mental state, a significance index for each mental state is calculated based on at least one of an average, a standard deviation, and a correlation coefficient for the measured biological signal.
The method of claim 9,
In the step of analyzing the user's psychological state, pre-processing is performed on the measured bio-signal, noise is filtered using a component analysis algorithm for the pre-processed bio-signal, and the filtered bio-signal is A virtual reality content control method comprising the step of calculating a significance index based on a probability position (z-score) on a normal distribution extracted based on at least one of a mean, a standard deviation, and a correlation coefficient.
The method of claim 8,
In the step of controlling at least one of the provision of the virtual reality content and the driving of the motion chair according to the analyzed user's mental state, a deep learning algorithm is used for the virtual reality content corresponding to the analyzed user's mental state. Accordingly, a virtual reality content control method in which control for provision of the content is performed based on a control model learned in advance.
A computer-readable recording medium on which a program for implementing the method of any one of claims 8 to 11 is recorded.

Images