KR102113634B1 - Virtual reality head mounted display for showing user's status and user status display method and content control method using the system - Google Patents

Abstract

Disclosed is a virtual reality head mounted display showing a user's status, a user status display method using the same, and a content management method. According to an embodiment of the present invention, in a virtual reality head mounted display (VR HMD) that allows a user to watch audio-visual content played through an internal display installed inside a frame body by wearing it on the head, a rear portion of the frame body A camera unit installed and acquiring a user facial image including the user's eyes; A signal processor generating a user state image processed by processing the user's face image according to a predetermined image processing algorithm; And it is installed on the front portion of the frame body, a virtual reality head-mounted display is provided that includes a front display for displaying the user-processed image image.

Description

{Virtual reality head mounted display for showing user's status and user status display method and content control method using the system}

The present invention relates to a virtual reality head mounted display (VR HMD) showing a user's status and a user status display and content management method using the same.

Virtual Reality (VR) is expanding its application to education, medical care, and nursing care. You can also treat alcoholism through VR, cure soldiers' war trauma, perform VR-based care for dementia patients, and conduct classes using VR for students, such as children.

However, for this virtual reality experience, a head mounted display (HMD) is required. When wearing a VR HMD, the user's eyes and facial expression cannot be seen. As a result, when a user is a child or a patient, it is difficult for guardians, therapists, and caregivers to grasp the user's status in real time, which is a problem in use in education, medical care, and care. For example, if the user is a child, the teacher or parent may be uneasy as to whether the user is using VR content properly. For elderly patients, it is difficult to detect signs of motion sickness or seizures caused by VR content in advance.

In addition, in the case of VR contents used for the treatment of psychosis, contents that the treatment subject wants to avoid are often used as treatment elements. For example, to show a scene of veterans suffering for post-traumatic stress disorder (PTSD) treatment, to show high-rise buildings, high-rise bridges, etc. for VR, or to treat alcoholism. After a hangover, you can show a vomiting scene. In this case, the treatment subjects often close their eyes, and thus the treatment purpose is often not achieved.

Korean Patent Publication No. 10-2017-0139185 (released on December 19, 2017)-Foldable virtual reality HMD with cooling fan

According to the present invention, a manager (guardian, healer, nursery, caregiver, etc.) can intuitively check the user status by showing an image of the user's eyes and / or facial expression through the front surface, thereby showing a user's status capable of immediate response It is for providing a real head mounted display and a user status display and content management method using the same.

The present invention is a virtual reality head mounted display showing a user's viewing state that allows an administrator to intuitively check whether a VR content experience is properly performed by conceptually showing information about a user's eyes and / or facial expression through the front surface. And to provide a user status display and content management method using the same.

Objects other than the present invention will be easily understood through the following description.

According to an aspect of the present invention, in a virtual reality head mounted display (VR HMD) that allows a user to watch audio-visual content played through an internal display installed inside a frame body by wearing it on the head, installed on the rear surface of the frame body A camera unit for acquiring a user facial image including the user's eyes; A signal processor generating a user state image processed by processing the user's face image according to a predetermined image processing algorithm; And it is installed on the front portion of the frame body, a virtual reality head-mounted display is provided that includes a front display for displaying the user-processed image image.

The signal processing unit may generate the user state image by scaling up or down the user's face image to fit the size of the front display unit.

The signal processing unit may generate the user state image with an animation conceptualizing it based on information obtained from the user's face image.

The front display unit is implemented in a small size on one side of the front unit, and the signal processing unit displays an emoticon corresponding to whether the user has an eye open or the user's emotional state determination result according to a result of analyzing the user's face image. It can be created as a user status image.

The user's brain wave (EGG), heart rate (HR), heart rate variability (HRV), electrocardiogram (ECG), pupil size, further comprising a sensor unit for obtaining a biosignal of at least one of the gaze direction, the signal processing unit is the living body The signal may be processed as user additional information so that it is superimposed on the user status image displayed through the front display unit.

Further comprising a status output unit for outputting a user state according to the analysis result of the bio-signal, the status output unit may include one or more of a plurality of color capable LED and sound output speaker.

The controller may further include a control unit for determining the emotional state of the user according to the analysis result of the biosignal or the analysis result of the user's face image, and automatically switching or stopping the audio-visual content currently being played according to the determination result.

Further comprising an operation unit capable of content-related manipulation including playback, stop, and conversion of the audio-visual content, the manipulation unit may be implemented as a hardware button on the front panel or a software button on the front display.

On the other hand, according to another aspect of the present invention, as a user state display method performed on a virtual reality head-mounted display to allow a user to watch audio-visual content played through an internal display installed inside a frame body by wearing it on the head, the audio-visual content Acquiring a user facial image of the user's face through a camera unit installed on the rear surface of the frame body when being reproduced; Generating a user state image by processing the user's face image according to a predetermined image processing algorithm; And displaying the user status image on a front display unit installed on a front portion of the frame body.

Obtaining at least one biosignal of the user's EEG, heart rate (HR), heart rate variability (HRV), electrocardiogram (ECG), pupil size, and gaze direction; And processing the biosignal as user additional information so as to be superimposed on the user state image displayed through the front display unit.

On the other hand, according to another aspect of the present invention, as a content management method performed on a virtual reality head-mounted display that allows a user to watch audio-visual content played through an internal display installed inside a frame body by wearing it on the head, the audio-visual content Acquiring at least one biosignal of the user's brainwave (EGG), heart rate (HR), heart rate variability (HRV), electrocardiogram (ECG), pupil size, and gaze direction when reproduced; Analyzing the bio-signal and determining the user response to the audio-visual content as one of a plurality of emotional states according to a predetermined criterion; And determining maintenance or conversion of the audio-visual content according to the determined determined emotional state.

Further comprising the step of storing the preference and aversion to the audio-visual content based on the determined determined emotional state, to recommend or avoid any audio-visual content based on the preference and aversion when switching the audio-visual content can do.

According to another aspect of the present invention, as a virtual reality head mounted display system showing the user's state, the user is worn on the head to watch the audio-visual content played through the internal display installed inside the frame body, the frame body A virtual reality head mounted display (VR HMD) for acquiring a user's face image including the user's eye through a camera unit installed on the rear portion of the camera; When connected to the virtual reality head-mounted display through a network, the user's face image is transmitted or a user state image corresponding to the user's face image is received, and when the user's face image is received, according to a predetermined image processing algorithm A virtual reality head mounted display system is provided that includes an administrator terminal that converts the user state image to the user state image and outputs the user state image to the screen when the user state image is transmitted.

The administrator terminal may receive an administrator's treatment instruction according to the user status image, and the virtual reality head mounted display may receive the treatment instruction through the network and output the treatment instruction to the user.

The virtual reality head mounted display or the manager terminal further includes an analysis module for analyzing the user's face image or the user's state image, and the analysis result in the analysis module determines whether the user is being evaded or not. If it is determined that the warning sound or direction sound can be output through the virtual reality head mounted display worn by the user.

Other aspects, features, and advantages other than those described above will become apparent from the following drawings, claims, and detailed description of the invention.

According to an embodiment of the present invention, by displaying an image of the user's eyes and / or facial expression through the front surface, an administrator (guardian, therapist, nursery, caregiver, etc.) can intuitively check the user status, thereby enabling immediate response There is.

In addition, by conceptualizing and showing information about the user's eyes and / or facial expressions through the front, there is also an effect that allows the administrator to intuitively check whether the VR content experience is properly performed.

1 is a schematic block diagram of a VR HMD showing a user's state according to an embodiment of the present invention;
Figure 2 is a flow chart of a user status display method using a VR HMD showing the user's status according to an embodiment of the present invention,
3 and 4 are views of the VR HMD showing the user's state according to an embodiment of the present invention, as viewed from the rear and the front,
5 is a view showing another example of the front display unit of the VR HMD,
6 is a view showing another example of the front display unit of the VR HMD,
7 is an exemplary view showing a user state in a VR HMD showing a user state,
8 is a flowchart of a content management method using VR HMD according to another embodiment of the present invention,
9 is a schematic configuration diagram of a virtual reality head mounted display system showing a user's state according to another embodiment of the present invention.

The present invention can be applied to various changes and can have various embodiments, and specific embodiments will be illustrated in the drawings and described in detail. However, this is not intended to limit the present invention to specific embodiments, and should be understood to include all modifications, equivalents, and substitutes included in the spirit and scope of the present invention.

When an element is said to be "connected" or "connected" to another component, it is understood that other components may be directly connected to or connected to the other component, but there may be other components in between. It should be. On the other hand, when a component is said to be "directly connected" or "directly connected" to another component, it should be understood that no other component exists in the middle.

Terms such as first and second may be used to describe various components, but the components should not be limited by the terms. The terms are used only for the purpose of distinguishing one component from other components.

The terms used herein are only used to describe specific embodiments, and are not intended to limit the present invention. Singular expressions include plural expressions unless the context clearly indicates otherwise. In this specification, terms such as “include” or “have” are intended to indicate that a feature, number, step, operation, component, part, or combination thereof described in the specification exists, and that one or more other features are present. It should be understood that the existence or addition possibilities of fields or numbers, steps, operations, components, parts or combinations thereof are not excluded in advance.

In addition, the components of the embodiments described with reference to the drawings are not limited to those embodiments, and may be implemented to be included in other embodiments within the scope of maintaining the technical spirit of the present invention. Although the description is omitted, it is natural that a plurality of embodiments may be reimplemented as one integrated embodiment.

In addition, in the description with reference to the accompanying drawings, the same components, regardless of reference numerals, are assigned the same or related reference numerals, and redundant description thereof will be omitted. In the description of the present invention, when it is determined that detailed descriptions of related known technologies may unnecessarily obscure the subject matter of the present invention, detailed descriptions thereof will be omitted.

In addition, terms such as “… unit”, “… unit”, “… module”, and “… group” described in the specification mean a unit that processes at least one function or operation, which is hardware or software or hardware and software. It can be implemented as a combination of.

1 is a schematic block diagram of a VR HMD showing a user's state according to an embodiment of the present invention, and FIG. 2 is a user state displaying method using a VR HMD showing a user's state according to an embodiment of the present invention 3 and 4 are views of the VR HMD showing the user's state according to an embodiment of the present invention, as viewed from the rear and front, and FIG. 5 is a view showing another example of the front display unit of the VR HMD. , FIG. 6 is a diagram illustrating another example of the front display unit of the VR HMD, and FIG. 7 is an exemplary diagram of displaying the user status in the VR HMD showing the user's status.

VR HMD (Virtual Reality Head Mounted Display) showing the state of a user (wearer) according to an embodiment of the present invention is an animation corresponding to an image obtained by analyzing a user's eyes and / or facial expression through a front display or an analysis result thereof / Display emoticons so administrators can easily check user status.

The VR HMD 100 showing the user's state according to the present embodiment includes a frame body 105, an internal display unit 110, a control unit 120, a camera unit 130, a front display unit 140, and a signal processing unit ( 150). Also, according to an embodiment, one or more of the sensor unit 160, the status output unit 170, and the operation unit 180 may be further included.

The frame body 105 is a body in which components such as an internal display unit 110, a control unit 120, a camera unit 130, a front display unit 140, and a signal processing unit 150 are installed, and a user may use a head ( Face).

The rear surface 106 of the frame body 105 may cover a user's face around the eyes, and may have a shape corresponding to the user's facial curvature. An opening corresponding to a user's eye may be formed in the rear portion 106, and an optical system including a lens may be installed in the opening.

The inside of the frame body 105 is installed so that the inner display unit 110 faces the rear (ie, the rear portion) corresponding to the optical system. Accordingly, when worn, the user can watch audio-visual content such as VR content played on the internal display unit 110 through the optical system of the rear unit.

The internal display 110 reproduces and displays audio-visual content such as VR content (step S210). For example, the internal display unit 110 may be a flat panel display such as LCD, LED, OLED, etc. that reproduces the transmitted image signal and displays it on the screen. In addition, the internal display 110 may also include a speaker for audio output.

The control unit 120 controls reproduction of audio-visual content through the internal display unit 110. The control unit 120 determines the type of audio-visual content to be played back, or stops playback or playback of the determined audio-visual content, by a user or administrator input through an operation unit 180 or a communication unit, which will be described later, or according to a preset control algorithm.

In this embodiment, the internal display unit 110 and the control unit 120 may be replaced with a smart phone, and in this case, the smart phone functioning as the internal display unit 110 and the control unit 120 may be detachably configured. . In this case, the frame body 105 may be designed as a detachable structure of a smartphone, and the detachable smartphone configuration in the head mounted display is a matter obvious to a person skilled in the art to which the present invention pertains. Omitted.

Audio-visual content may be stored in advance in a separate storage (not shown). Alternatively, it may be transmitted in real time from a remote management server (not shown) through a communication unit (not shown).

The camera unit 130 is installed on the rear surface of the frame body 105 (ie, the eyepiece). The camera unit 130 acquires a user's facial image by photographing a user's face including the user's eyes that have encountered audio-visual content played through the internal display unit 110 (step S220).

One camera unit 130 may be installed in the center of the rear portion. Alternatively, two can be installed, one for each position corresponding to the optical system of the rear portion, so that each of the user's eyes can be photographed. Or three or more may be installed for more sophisticated image acquisition.

The front display unit 140 is a display device that is installed on the front portion of the frame body 105, and processes the user's face image captured by the camera unit 130 by using an image processing algorithm set in advance to generate a user state image. It can be displayed (step S230).

Through the user status image displayed on the front display unit 140, the administrator can intuitively check whether the user is currently viewing the audio-visual content or the user's reaction (user emotion state) to the currently playing audio-visual content.

The signal processing unit 150 generates a user state image by processing a user's face image photographed by the camera unit 130 in a format displayable by the front display unit 140 according to a predetermined image processing algorithm.

The front display unit 140 may consist of one display as shown in FIG. 4 or may be composed of two displays 140a and 140b as shown in FIG. 5.

When the number of camera units 130 and the number of front display units 140 are the same, the signal processing unit 150 may display an image captured by an arbitrary camera unit 130 corresponding to the corresponding camera unit 130. It can be displayed directly through (140).

Here, the user state image may be simply an image that is scaled up or down or noise-reduced to fit the user's face image to fit the size of the front display 140.

Alternatively, the signal processing unit 150 may generate a user state image with an animation conceptualizing it based on information obtained from the user's face image. For example, if the user opens his eyes, it can be replaced with an animation that opens his eyes, or if the user closes his eyes, the animation that closes his eyes, or an animation that expresses an expression corresponding to the determined user's emotional state. It is. This is an image processing algorithm considering that the user may be reluctant to directly display the face including his / her eyes on the front display unit 140.

Alternatively, the front display unit may be implemented in a small size (see 140c) on one side of the front unit as illustrated in FIG. 6. This is implemented to be usable in situations where the user is reluctant to expose all of his or her face.

In this case, the small front display unit 140c does not show the image taken by the camera unit 130 but determines whether the eye is opened or the emotional state is determined according to the result of analyzing the image by the signal processing unit 150. Corresponding emoticons (see (b) and (c) of FIG. 6) may be displayed.

Referring back to FIG. 1, the VR HMD 100 according to the present embodiment may further include a sensor unit 160 that measures additional information about the user.

The sensor unit 160 is an EEG sensor that measures the user's EEG, an pupil sensor that measures the pupil size of the user, an eye tracker that tracks the user's gaze (eye movement), and the user's heart rate (HR ), Heart rate variability (HRV), heart sensor measuring electrocardiogram (ECG), skin sensor measuring user's skin resistance (GSR).

The sensor unit 160 may obtain a biosignal such as an EEG, heart rate (HR), heart rate variability (HRV), electrocardiogram (ECG), pupil size, and gaze direction (step S240).

The bio-signal obtained by the sensor unit 160 is processed as additional user information in the signal processing unit 150 (step S250), so that it can be superimposed on the user state image displayed through the front display unit 140 ( Step S260).

Referring to (a) of FIG. 7, an example in which patient information (eg, age, etc.), pupil size, pupil size change, and the like are displayed in addition to the user state image through the front display unit 140 is superimposed.

Referring to (b) of FIG. 7, an example in which patient emotion information, an EEG spectrum, and the like are displayed in addition to a user state image through the front display unit 140 is illustrated.

Also, the sensor unit 160 may measure an eye blink signal through an EEG sensor. The iBlink signal is a signal related to blinking of the eyes.

The control unit 120 may determine that the user is closing his eyes when the i-link signal is not continuously detected. In this case, the state of the current user (especially, whether or not the eyes are closed) may be displayed through the front display unit 140 that controls the signal processing unit 150 and displays animation or emoticons. While the i-link signal is detected, the animation or emoticon with the eyes open may be displayed. If the i-link signal is not detected for a certain time or more, the animation or emoticon with the eyes closed may be displayed.

In addition, the VR HMD 100 according to the present embodiment may further include a status output unit 170 that outputs a user state according to a biosignal analysis result.

The status output unit 170 may be installed on the front surface of the frame body 105, and may include one or more of a plurality of color-expressing LEDs and a speaker capable of sound output.

When the analysis result of the bio-signal obtained from the sensor unit 160 indicates an abnormal state different from the expected state to be obtained through the content (for example, fear, anxiety, etc.), the LED is turned on to warn the administrator to recognize it. You can express it or output a warning sound through the speaker. The manager may stop the content currently being played or release the VR HMD 100 through the warning light or the warning sound.

Alternatively, the controller 120 may automatically switch or stop the currently playing content.

The result of the analysis of the bio-signal can be determined to be in an abnormal state by the user in the following changes.

EEG-High beta increase, Gamma increase

Heart rate (HR)-increased

Heart Rate Variability (HRV)-RR, LF, HF, FL / HF increase

Skin resistance (GSR)-increase

Pupil Size-Increase

Eye movement-increased irregularity

The increase in the bio-signal can be regarded as an increase in interest up to a certain range, and can be interpreted as surprise or anxiety when the threshold is exceeded. The threshold may have individual differences for each user, and may be corrected in consideration of personal status information such as the user's disease and normal condition.

In this embodiment, the control unit 120 analyzes the user's facial image obtained through the camera unit 130 or the analysis result of the bio-signal obtained through the sensor unit 160, a user preset time (for example, about 15 seconds) ) If the eyes are continuously closed above, the manager may recognize this by outputting a warning signal (warning light, warning sound) through the status output unit 170.

If the user closes his or her eyes for a certain period of time, the playback of the current audio-visual content will not cause the intended effect to the user, so that the user can take appropriate action by notifying the administrator. In this case, it is possible to enable a small number of managers to care for multiple users at the same time by making it easy to recognize even if the manager does not continuously monitor through the warning signal output.

In addition, the VR HMD 100 according to the present embodiment may further include an operation unit 180 capable of content-related manipulation, such as reproduction, stop, conversion, etc. of audio-visual content from the outside.

The operation unit 180 may be implemented as a hardware (HW) button as shown in FIG. 7 (a), or implemented as software (SW) as shown in FIG. 7 (b) to display the front display unit ( 140).

The administrator simply presses a hardware button or a software button suitable for content-related operations, such as switching audio-visual content to be played, stopping audio-visual content being played, and playing audio-visual content, depending on the user state identified through the front display 140. You can care for the user by manipulation.

This is because, in the situation where an abnormal state such as psychological instability occurs to the user while viewing the audio-visual content, the user of the physically uncomfortable may not be able to autonomously release the VR HMD 100. It can be applied, so that the administrator can help smoothly from the outside.

According to the present embodiment, whether the user continuously closes his or her eyes through the front display unit 140, what is the emotional state (for example, fear, surprise, fear, etc.) felt about the currently playing VR content, and The administrator can intuitively grasp the emotion level through the user state image captured by the camera unit 130 or through an animation or emoticon corresponding to the user state, so that an appropriate response can be immediately performed according to the user state. To help.

In particular, the scope of VR content is expanding due to the high degree of realism and immersion in the treatment of mental disorders and the care of patients with long-term care or dementia. According to this embodiment, it is possible to observe the facial expression, eyes, etc. of the patient wearing the VR HMD 100, so that if the patient's psychological instability occurs, immediate response is possible, and intensive care is possible even for patients with physical discomfort. Therefore, there is an advantage that a small number of caregivers can simultaneously care for a large number of patients.

8 is a flowchart of a content management method using VR HMD according to another embodiment of the present invention.

The content is played through the internal display 110 (step S310).

The user wearing the VR HMD 100 watches the played content, and the user's content response is acquired by the camera 130 as an image (step S320). In addition, the sensor unit 160 may acquire a user's content response as a biosignal.

The acquired image and / or bio-signal may be stored in association with content reproduced by time (step S330). The acquired image and / or biosignal may be displayed in real time through the front display unit 140 (step S335).

Then, the obtained image and / or bio-signal is analyzed to analyze the user's reaction and corresponding emotion for the corresponding content (step S340).

The result of the emotion analysis on the user's content is determined according to a predetermined criterion (step S350). For example, the emotional state can be divided into interest, boring, restful, and fear.

The content is maintained or converted according to the determined emotional state (step S360).

In addition, the user's preference and aversion to the corresponding content may be stored based on the determined emotional state, and may be recommended or avoided when switching the content later (step S370).

For example, it will be described on the assumption that a pupil size and additional user biometric information are used.

The camera unit 130 measures the pupil size. In addition, you can measure the part of attention by grasping the line of sight.

The sensor unit 160 may measure pulse waves (PPG), electrocardiogram (ECG), and brain waves (EGG).

The signal processing unit 150 determines significant bioinformation changes, including pupil size. Here, comprehensive judgment can be performed using pulse waves, electrocardiograms, and brain waves. In addition, movement of the gaze can also be used.

The control unit 120 may record a time point or section of the corresponding content corresponding to a point in time of a significant change in biological information or create a snapshot.

For example, in the case of PTSD treatment for firefighters, a point in time at which the pupil is particularly enlarged (in addition, the pulse wave response also fluctuates) in a fire situation is measured, and a snapshot of the image or a section before and after that point is saved as a video clip, and additionally, the point of view Alternatively, the part where the subject's gaze is concentrated in the section (for example, a lighted curtain) may be tracked and only the relevant part may be marked.

Alternatively, in the case of caring for dementia patients using the content, it can be used for art therapy education after viewing the content with a snapshot of the part where the pupil size occurred or a video clip in the front and rear sections.

Alternatively, in the case of drug addiction treatment, anger disorder treatment, etc., the dialogue content at the time can be extracted to find out the cause of anger about the sound through counseling, and the direction of additional treatment can be determined.

Or, in aversion treatment and reinforcement treatment using methods such as systematic desensitization, various types of stimuli and sensory conditions such as smell, sound, vision, and tactile sense of the relevant section are utilized to maximize the effect by selecting the type of sensation to be strengthened in future treatment. Can pursue.

Through a separate storage unit, the corresponding snapshot corresponding to the point of change of the significant bio-information can be filed, output, and DB, and used as it is used for post-treatment or education.

The source of the sensory stimuli such as snapshots or video, sound, odor drugs, and tactile stimuli at the point of view can be filed, output, and DB-by-application to support utilization.

For example, in the case of dementia, output of a snapshot to be used for art therapy after content care can be performed. In the case of PTSD, the video clip of the section with strong response and the marking of the focusing point can be stored in the video. In the case of addiction treatment and anger disorder treatment, the information of stimulus provision can be DB analyzed and used for further treatment maximization.

According to the present exemplary embodiment, an image and / or a bio-signal for a user may be analyzed in real time to switch to other content when the response to the content is not pleasant (eg, fear, boredom, etc.).

In addition, for a specific user, the content type preferred by the user may be determined and similar content may be recommended.

By automatically converting and recommending such content, it is possible to continuously enjoy the content desired by the user without any additional manipulation, so that a small number of administrators can manage a large number of users.

The user state display method and / or content management method using the VR HMD according to the above-described embodiments can be implemented as computer-readable codes on a computer-readable recording medium. The computer-readable recording medium includes any kind of recording medium that stores data that can be read by a computer system. For example, there may be a read only memory (ROM), a random access memory (RAM), a magnetic tape, a magnetic disk, a flash memory, and an optical data storage device. In addition, the computer-readable recording medium may be distributed over computer systems connected by a computer communication network, and stored and executed as code readable in a distributed manner.

9 is a schematic configuration diagram of a virtual reality head mounted display system showing a user's state according to another embodiment of the present invention.

Referring to FIG. 9, a plurality of patients are wearing a virtual reality head mounted display, and a manager terminal 400 is remotely located to be connected to a plurality of virtual reality head mounted displays through a network such as the Internet.

The manager terminal 400 includes a terminal screen 420 through which a therapist (eg, doctor, nurse, caregiver, etc.) in charge of treatment for a patient can check the patient's condition.

The virtual reality head mounted display acquires a user's face image of each user's (patient's) face (including the eyes). In addition, sensor information may be acquired through various attached sensors. The sensor information may include at least one biosignal among brain waves, heart rate, heart rate variability, electrocardiogram, pupil size, and gaze direction.

Here, the analysis of the image signal for the user's face image or the analysis of the biosignal may be performed in the analysis module (not shown) of the virtual reality head mounted display as the client end as described above. Alternatively, image signal analysis and / or biosignal analysis may be performed through the analysis module 410 in the server terminal, the administrator terminal 400.

On the terminal screen 420, the state of each patient (the subjects to be treated) can be clearly expressed. Referring to FIG. 9, it can be seen that patients 1 and 3 have their eyes open, and patients 2 and 4 have their eyes closed.

When the current user's condition, such as closing his eyes or turning his head, is determined to be a treatment avoidance behavior, the manager (therapist) may give a micro-treatment instruction to the user (patient). If the user selects the microphone icon 430 corresponding to the user by clicking or touching, and the administrator speaks through the microphone provided in the administrator terminal 400, the treatment instruction can be transmitted to the user only by voice.

Alternatively, the user's treatment avoidance behavior is automatically detected at the client end (virtual reality head mounted display) or the server end (administrator terminal 400), and a warning sound is output or directed through the virtual reality head mounted display for the treatment avoidance behavior. You can also output sound (eg, don't turn your head, don't close your eyes, etc.).

Information about the user's eyes may be transmitted as an image and expressed as an image on the terminal screen 420 (see patients 1 and 3). Alternatively, the information transmitted as an image may be displayed in the form of an icon or animation through the analysis module 410 (see patients 2 and 4). Or, it can be interpreted at the client side and delivered to the server side only in the form of an icon instead of an image. Thereby, problems such as privacy can be solved.

Although described above with reference to embodiments of the present invention, those skilled in the art variously modify the present invention without departing from the spirit and scope of the present invention as set forth in the claims below. And can be changed.

100: VR HMD 105: Frame body
110: internal display unit 120: control unit
130: camera unit 140: front display unit
150: signal processing unit 160: sensor unit
170: status output unit 180: operation unit

Claims (15)

In the virtual reality head mounted display (VR HMD) that allows the user to watch the audio-visual content played through the internal display installed inside the frame body worn on the head,
A camera unit installed on the rear surface of the frame body to acquire a user facial image including the user's eyes;
A signal processor generating a user state image processed by processing the user's face image according to a predetermined image processing algorithm; And
A display device installed on the front portion of the frame body, including a front display unit for displaying the user-processed image of the image processing,
The user state image is generated by scaling up or down the user's face image to fit the size of the front display,
The user's brain wave (EGG), heart rate (HR), heart rate variability (HRV), electrocardiogram (ECG), pupil size, further includes a sensor unit for obtaining a biosignal of at least one of the gaze direction,
The signal processing unit processes the biosignal as user additional information so that it is superimposed on the user state image displayed through the front display unit,
The sensor unit measures the eye blink signal related to the blink of the eye through the EEG sensor,
Further comprising a control unit for determining that the user is closing his eyes when the iBlink signal is not continuously detected for a certain period of time,
The control unit controls the signal processing unit to display whether or not the user is closing his or her eyes through the front display unit.
delete delete delete delete According to claim 1,
Further comprising a status output unit for outputting a user state according to the analysis result of the bio-signal,
The status output unit is a virtual reality head-mounted display, characterized in that it comprises at least one of a plurality of color capable LED and a speaker capable of sound output.
According to claim 1,
Virtual reality head mount further comprising a control unit for determining the emotional state of the user according to the analysis result of the bio-signal or the analysis result of the user's face image, and automatically switching or stopping the audio-visual content currently playing according to the determination result display.
According to claim 1,
Further comprising a control unit capable of content-related manipulation, including playback, stop, conversion of the audio-visual content,
The operation unit is implemented as a hardware button on the front portion or a virtual reality head mounted display, characterized in that implemented as a software button on the front display portion.
A user state display method performed on a virtual reality head mounted display that allows a user to watch audio-visual content played through an internal display installed inside the frame body by wearing it on the head,
Acquiring a user's face image of the user's face through a camera unit installed on a rear portion of the frame body when the audio-visual content is reproduced;
Generating, by the signal processing unit, the user's face image by processing the image according to a predetermined image processing algorithm; And
Displaying the user status image on the front display unit installed on the front portion of the frame body,
The user state image is generated by scaling up or down the user's face image to fit the size of the front display,
The sensor unit further comprises the step of acquiring at least one bio-signal of the user's brain wave (EGG), heart rate (HR), heart rate variability (HRV), electrocardiogram (ECG), pupil size, gaze direction,
The signal processing unit processes the biosignal as user additional information so that it is superimposed on the user state image displayed through the front display unit,
The sensor unit measures an eye-blink signal related to blinking of an eye through an EEG sensor;
Determining, by the control unit, that the user is closing his eyes when the iBlink signal is not continuously detected for a predetermined time or longer;
And controlling the signal processing unit in the control unit to display whether the user is closing his or her eyes through the front display unit.
delete delete delete delete delete delete

Images