WO2022070415A1 - Head-mounted display device - Google Patents

Abstract

Provided is a feature related to a head-mounted display device (HMD) that enables the psychological/emotional state of a user to be inferred/ascertained quickly and more profoundly, and, on the basis thereof, enables output content to be suitably adjusted. A HMD according to the present invention is provided with devices 2, 4 that acquire electromyogram signals from facial expression muscles of a user and uses the electromyogram signals as input information. The HMD adjusts output content in response to an inferred psychological state on the basis of the electromyogram signals. The HMD adjusts, as the adjustment output content, the displayed GUI image type or information quantity.

Description

Head-mounted display device

The present invention relates to a technique of a head-mounted display device (HMD).

HMDs such as eyeglasses and goggles that are worn on the user's head have been realized. The HMD displays an image on a display surface of a display method such as a transparent type or a non-transparent type. The image can be displayed in an form such as augmented reality (AR) or virtual reality (VR). For example, the HMD can superimpose and display a virtual image on a real image viewed by a user through a display surface based on an OS, an application program, or the like.

On the other hand, in electronic devices, a technique has been developed in which a biological signal such as a user's electro-oculography signal is acquired, the psychological / emotional state of the user is estimated based on the acquired biological signal, and some control is performed.

Examples of the prior art include JP-A-2017-70602 (Patent Document 1) and JP-A-2019-118448 (Patent Document 2). Patent Document 1 describes, as an "information processing method" or the like, "to intuitively grasp what kind of state the user's own psychological state is" and the following. Eyeglasses (eyewear) as shown in FIG. 1 have a bioelectrode mounted on a nose pad and a bridge portion, and acquire an electrooculogram signal from the bioelectrode. This eyeglass processing device transmits a sensor signal, an electro-oculography signal, and the like to an external device or a server. The external device displays a predetermined figure formed from a plurality of objects representing the psychological state of the user based on the sensor signal, the electrooculogram signal, and the like.

Patent Document 2 describes, as a "mental state estimation system" or the like, that "not only can the mental state be estimated quickly and accurately, but also switching or instantaneous change of the mental state is detected" and the following. There is. This system estimates mental state using features related to muscle activity in facial muscles. This system uses surface electrodes to acquire myoelectric signals and analyze time-series data of myoelectric signals to acquire features.

JP-A-2017-70602 Japanese Unexamined Patent Publication No. 2019-118448

For example, Patent Document 1 describes that a system including eyeglasses calculates psychological parameters from a user's electro-oculography signal, and measures the user's blinks, eye movements, body movements, etc., and the user's vitality and concentration. , There is a description to acquire a psychological state such as calmness.

In electronic devices, etc., in technology that attempts to infer and acquire psychological and emotional states from the electrooculographic signal based on the movement of the user's eyes, in general, the unconscious psychology that is difficult to appear in facial expressions and movements from the electrooculographic signal.・ It is difficult to guess and grasp the emotional state. When a state such as discomfort or tiredness appears in the movement of the eye, it can be inferred from the electro-oculography signal, but an unconscious state is unlikely to appear in the electro-oculography signal.

On the other hand, as in the example of Patent Document 2, there is also a technique for inferring / acquiring a psychological / emotional state from a user's EMG signal. Psychological / emotional states include comfort / discomfort, fatigue, relaxation, concentration, and stress. The technique using the EMG signal is easier to guess and grasp the unconscious psychological / emotional state of the user than the technique using the electro-oculography signal.

In the HMD, it is preferable that the image such as the graphical user interface (GUI), which is the output content for the user, is displayed in a suitable manner such that the user does not get tired, does not feel uncomfortable, and is easy to understand. Therefore, in an HMD worn on the user's head, a technique of detecting a biological signal such as an electro-oculography signal from facial facial muscles to estimate the user's psychological / emotional state can be considered. In particular, in HMD, a technique of estimating the influence of an image of an OS or an application on a user's psychological / emotional state from a biological signal and adjusting the output content based on the guess is conceivable.

However, in HMD, when trying to estimate the psychological / emotional state from the electrooculographic signal based on the movement of the eye, the influence of the image such as GUI on the psychological / emotional state does not always appear in the movement of the eye. It is difficult to guess the psychology and emotions. In such a technology, when the output content such as a GUI image is inappropriate for the user, the user actually becomes uncomfortable or tired due to the influence of the output content and appears as a facial expression or movement. It is difficult to guess and grasp such a psychological / emotional state (that is, the influence that the output content makes the user tired, etc.) unless later.

An object of the present invention is to provide a technique for quickly and deeply guessing and grasping a user's psychological / emotional state with respect to the HMD technique, and a technique for appropriately adjusting the output content based on the technique. That is.

A typical embodiment of the present invention has the following configuration. The head-mounted display device of the embodiment includes a device that acquires an electromyographic signal from a user's facial muscle, and uses the electromyographic signal as input information. Further, the head-mounted display device of the embodiment adjusts the output content according to the psychological state estimated based on the electromyographic signal. Further, the head-mounted display device of the embodiment adjusts the type or amount of information of the GUI image to be displayed as the adjustment of the output content.

According to a typical embodiment of the present invention, the psychological / emotional state of the user can be quickly and deeply estimated / grasped with respect to the HMD technology, and the output content is appropriately adjusted based on the guess / deeper. Can be done. Issues, configurations, effects, and the like other than those described above will be described in the embodiment for carrying out the invention.

The configuration outline of the HMD of Embodiment 1 of this invention is shown. The configuration of the HMD of the first embodiment is shown. The configuration which looked at the front surface of the HMD of Embodiment 1 is shown. An example of the configuration of the biological signal acquisition unit and the electrode of the HMD of the first embodiment is shown. An example of the functional block configuration of the HMD of the first embodiment is shown. In the HMD of the first embodiment, the processing flow at the time of a test is shown. In the HMD of the first embodiment, an output example at the time of a test is shown. In the HMD of the first embodiment, an output example at the time of evaluation input is shown. In the HMD of the first embodiment, the processing flow at the time of user processing at the time of using an application is shown. In the first embodiment, an example of the correspondence between the output content and the EMG signal is shown. The first embodiment shows an example of control information. In the first embodiment, an example of a target facial muscle is shown. The configuration of the HMD of the modification of the first embodiment is shown. The processing flow in the HMD of Embodiment 2 of this invention is shown. In the second embodiment, an example of user setting information regarding a voluntary input operation is shown. In the second embodiment, an example of a voluntary input operation and a display image is shown. The processing flow in the HMD of Embodiment 3 of this invention is shown. A control example is shown in the HMD of the third embodiment.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. In the drawings, the same parts are designated by the same reference numerals in principle, and repeated description thereof will be omitted. In the description, when explaining the processing by the program, the program, the function, the processing unit, etc. may be mainly described, but the main body as the hardware for them is a processor or a controller composed of the processor or the like. Devices, processors, systems, etc. The computer executes processing according to the program read out on the memory by the processor while appropriately using resources such as the memory and the communication interface. As a result, a predetermined function, a processing unit, and the like are realized. The processor is composed of, for example, a semiconductor device such as a CPU or a GPU. A processor is composed of a device or a circuit capable of performing a predetermined operation. The processing is not limited to software program processing, but can be implemented by a dedicated circuit. FPGA, ASIC, etc. can be applied to the dedicated circuit. Further, the program may be installed in the target computer as data in advance, or may be distributed and installed as data from the program source to the target computer. The program source may be a program distribution server on a communication network or a non-transient computer-readable storage medium. The program may be composed of a plurality of program modules. Further, for the sake of explanation, various data and information may be described by expressions such as tables and lists, but the structure and format are not limited to these. Further, data and information for identifying various elements may be described by expressions such as identification information, an identifier, an ID, a name, and a number, but these expressions can be replaced.

<Embodiment 1>
The HMD1 which is the head-mounted display apparatus of Embodiment 1 of this invention will be described with reference to FIGS. 1 to 13. The HMD1 of the first embodiment has a function capable of acquiring and inputting information representing a user's unconscious psychological / emotional state based on the detection of an electromyographic signal from a facial muscle, and a GUI image based on the information. It has a function to adjust the output contents such as display. The HMD 1 of the first embodiment shown in FIG. 1 and the like includes a function and devices 2 and 4 for acquiring an EMG signal from a user's facial muscle, and uses the acquired EMG signal as input information. Further, the HMD 1 of the first embodiment preferably adjusts the output content such as a GUI image according to the inferred psychological state based on the electromyographic signal. Further, the HMD 1 of the first embodiment adjusts the type of GUI image to be displayed, the amount of information, and the like as the adjustment of the output content.

In the first embodiment, the estimation / grasp of the psychological state is realized by controlling the correspondence between the EMG signal and the output content (for example, the image displayed on the display surface 11), and the parameter value representing the psychological state or the like is realized. It is not necessary to handle the information of. It should be noted that the HMD of the modified example is controlled by the correspondence between the three elements of the EMG signal, the psychological state, and the output content, and even if it is handled by reading and writing information such as a parameter value indicating the psychological state. good.

[the term]
A supplementary explanation will be given for terms. The user's psychological / emotional state (sometimes collectively referred to as psychological state) includes states such as comfort / discomfort, fatigue, stress, vitality, relaxation, concentration, arousal, and anger.

The electromyogram signal may be described as a muscle potential signal. The electromyogram may be described as EMG. The electromyogram is a diagram of the weak changes in electric potential generated in the muscle on the axis of electric potential and time. EMG is used in medical treatment for auxiliary diagnosis of involuntary movements.

The electrooculogram signal may be described as an electrooculogram signal. The electrocardiogram may be described as EOG. The electro-oculography signal is related to the movement of the eye and records the potential due to the movement of the eyeball. The electro-oculography signal also appears as a signal change during blinking or eye movement. The electro-oculography signal is also mixed with the influence of biological phenomena (for example, smile) other than eye movement.

[HMD (1)]
FIG. 1 shows a schematic view of how the HMD 1 of the first embodiment is attached to the user's head as an outline of the configuration. (A) is a case of a glasses-type HMD, and (B) is a case of a goggle-type HMD. The HMD 1 is not limited to a single configuration, and may be provided with a remote controller, or may be connected to an external device such as a smartphone, a PC, or a server by communication. The remote controller is an operation device that can input the operation by the user's hand. The HMD 1 performs, for example, short-range wireless communication with the controller. By operating the manipulator, the user can input instructions to the HMD 1 and move the cursor on the display surface 11.

An external device such as a smartphone has, for example, an application program or data. Those programs and data may be provided to HMD1 from an external device. Data may be stored and displayed from the HMD 1 to an external device.

The application may be, for example, an application corresponding to AR or VR functions. Examples of the application include those that guide the user in the space and those that provide work support. For example, the HMD 1 generates a virtual image for guidance and work support based on the program processing of the OS or the application, and displays it on the display surface 11. The images also include various GUI images such as cursors, buttons, commands, icons, menus, windows and the like.

The HMD1 of the first embodiment includes unique devices 2 and 4 for handling an electromyographic signal. The devices 2 and 4 are an electromyogram signal detection device, in other words, an electrode mounting portion, which is arranged so as to be in contact with a portion of facial facial muscles. The device 2 is arranged so as to be in contact with the vicinity of the cheek in particular, and detects an electromyographic signal from the facial muscles near the cheek. The device 4 is arranged so as to be in contact with the vicinity of the eyebrows in particular, and detects an electromyographic signal from the facial muscles near the eyebrows. The controller of HMD1 performs control processing using those EMG signals.

The HMD1 of the first embodiment has an output content adjusting function using an EMG signal as a unique function. Based on the EMG signals acquired using the devices 2 and 4, the HMD1 automatically outputs the output contents such as GUI images by the OS, application, user interface function, etc. according to the estimated unconscious psychological state. adjust.

[HMD (2)]
FIG. 2 shows the detailed configuration of the HMD1 of FIG. 1, particularly the spectacle-shaped HMD1 of (A), as a perspective view. For the sake of explanation, X, Y, and Z are shown as directions. In the coordinate system with respect to the user's head and HMD1, the X direction is the front-back direction, the Y direction is the left-right direction, and the Z direction is the up-down direction. The HMD 1 includes a spectacle-shaped housing 10, a display device including a display surface 11, a device 2 related to an EMG signal, a device 4, and the like. A controller, a display device, a camera 12, a distance measuring sensor 13, a sensor unit 14, a biological signal acquisition unit 20, a voice output device 19, and the like are mounted on the housing 10.

The housing 10 has, as each part, a binocular portion 10a provided with a display surface 11, a bridge portion connecting between the left eye portion and the right eye portion of the binocular portion 10a, and temples on the left and right outside the binocular portion 10a. It has a part 10c and the like. The binocular portion 10a is provided with a display surface 11, a camera 12, a distance measuring sensor 13, a device 4, an electrode 5, and the like. An electrode 5 for detecting an electro-oculography signal is provided in the vicinity of the nose pad of the bridge portion 10b. A controller, a biological signal acquisition unit 20, a sensor unit 14, an audio output device 19 including a speaker and an earphone terminal, and the like are mounted on the temple unit 10c.

A real image of the outside world is transmitted through the display surface 11, and the image is superimposed and displayed on the real image. In this example, the display device including the display surface 11 is a transparent display method, but the display device is not limited to this, and a non-transparent type (in other words, VR type) display method may be used. For example, in the case of a goggle-type housing as shown in FIG. 1B, the display device may be a non-transparent display method.

The camera 12 has, for example, two cameras arranged on the left and right sides of the housing 10, and captures a range including the front of the HMD 1 to acquire an image. The distance measuring sensor 13 is a sensor that measures the distance between the HMD 1 and an object in the outside world. As the distance measuring sensor 13, a TOF (Time Of Flight) type sensor may be used, or a stereo camera or another type may be used. The sensor unit 14 is a portion provided with various sensors other than the camera 12 and the distance measuring sensor 13. The sensor unit 14 includes a group of sensors for detecting the position and orientation of the HMD1. The position where the camera 12, the distance measuring sensor 13, and the sensor unit 14 are arranged is not limited to the position shown in the figure.

Device 2 is located near the cheek. The device 4 is arranged near the eyebrows. The device 2 is configured as one set on the left and right of the device 2R on the right eye side and the device 2L on the left eye side, and has a symmetrical shape with respect to the XZ plane. The device 4 is configured as one set on the left and right of the device 4R on the right eye side and the device 4L on the left eye side, and has a symmetrical shape with respect to the XZ plane. The devices 2 and 4 are provided as an extension to the spectacle-shaped housing 10. Devices 2 (2R, 2L) are connected to the left and right temple portions 10c so as to extend forward (in the V direction in the figure) on the lower side. The housing 25 of the device 2 has a shape extending in the V direction, for example, substantially like an arm, but is not limited thereto. The left and right devices 4 (4R, 4L) are connected to the upper side of the binocular portion 10a so as to be exposed to the eyebrow side. In the case of the goggle type as shown in FIG. 1B, the device 4 is arranged in the goggle type housing. The devices 2 and 4 may be detachably connected to the housing 10, respectively, or one of the housings 10 so as to continuously extend from the housing 10 (temple portion 10c or binocular portion 10a). It may be fixedly provided as a portion.

Each of the device 2R and the device 2L is provided with an electrode 3 on the surface of the housing 25 facing inward in the Y direction. Each of the device 4R and the device 4L is provided with an electrode 3 on a surface facing the eyebrow side in the X direction. The electrode 3 is a sensor electrode for detecting an electromyographic signal. In the device 2, an electrode 3 and a voice input device 18 including a microphone are arranged in the V direction, which is the longitudinal direction. The devices 2 and 4 also include a circuit and the like connected to the electrode 3 and the biological signal acquisition unit 20 of the temple unit 10c.

The biological signal acquisition unit 20 is built in the left and right temple units 10c. The biological signal acquisition unit 20 includes an electromyogram signal sensor 201 and an electro-oculography signal sensor 202, as shown in FIG. 4 described later. The electromyogram signal sensor 201 of the biological signal acquisition unit 20 detects the electromyogram signal from the electrode 3 of the device 2 and the electrode 3 of the device 4, and the electrooculogram signal sensor 202 detects the electromyogram signal from the electrode 5. .. The biological signal acquisition unit 20 acquires these electromyographic signals and electro-oculography signals as biological signals, and performs predetermined processing. The biological signal acquisition unit 20 cooperates with the controller 101 (FIG. 4) of the HMD1. The biological signal acquisition unit 20 may be mounted as a part of the controller 101.

In the first embodiment, the body part to detect the electromyographic signal, that is, the part of the facial muscle to which the electrode 3 which is the sensor electrode is in contact is included, in particular, the vicinity of the cheek and the vicinity of the eyebrows. And said. The target is not limited to this, and at least one facial muscle may be targeted.

In the first embodiment, the HMD 1 arbitrarily uses the electro-oculography signal from the electrode 5 as information independent of the electromyogram signal from the electrode 3. For example, the HMD 1 may determine a state such as blinking or eye movement from the electro-oculography signal. The HMD of the modified example may be configured not to include the electrode 5 for the electro-oculography signal.

[Devices and electrodes]
FIG. 3 shows a configuration in which the front surface of the HMD 1 in FIG. 2 is viewed, and in particular, shows a configuration example of the electrodes 3 of the devices 2 and 4 for the electromyographic signal, the electrodes 5 for the electrooculogram signal, and the like. Electrodes 5a and 5b are arranged as electrodes 5 in the vicinity of the nose pad. The electrode 5 comes into contact with the vicinity of the user's nose (FIG. 12).

In the device 2R, two electrodes 3 of the electrode 3a and the electrode 3b are arranged as one pair p1. In the device 2L, two electrodes 3 of the electrode 3c and the electrode 3d are arranged as one pair p2. The housing 25 of the device 2 (2R, 2L) is arranged so as to be inclined to the left and right outward in the Z direction, for example. The electrode 3 of the device 2 protrudes slightly outward from the inner surface in the Y direction of the housing 25, and easily comes into contact with the skin near the cheek.

On the back side of the device 4 (4R, 4L), two electrodes 3 are arranged on the left and right, respectively. In the device 4R, two electrodes 3 of the electrode 3e and the electrode 3f are arranged as one pair p3. In the device 4L, two electrodes 3 of the electrode 3g and the electrode 3h are arranged as one pair p4. The electrode 3 of the device 4 protrudes slightly from the back surface to facilitate contact with the skin near the eyebrows.

The electrode 3 group (3a to 3d) of the device 2 provided on the buccal side is referred to as the electrode 3A, and the electrode 3 group (3e to 3h) of the device 4 provided on the eyebrow side is referred to as the electrode 3B. The electrode 3A is a sensor electrode for detecting an electromyographic signal from the facial muscle near the cheek. The electrode 3B is a sensor electrode for detecting an electromyographic signal from the facial muscle near the eyebrows. The electrode 3A contacts the skin near the cheek. The electrode 3B contacts the skin near the eyebrows. The shape of the electrode 3 may be a disk or the like.

As for the plurality of electrodes 3 for detecting the EMG signal, the two electrodes 3 are used as one pair. The electromyographic signal is detected as a potential difference between the pair of electrodes 3. The potential difference between the two electrodes 3 is used as a 1-channel EMG signal. The electro-oculography signal is detected for each electrode 5. The potential difference between one electrode 5 and the ground electrode 21 is used as a one-channel ocular potential signal.

The ground electrode 21 is a ground electrode for the electrode 3 and the electrode 5 of the biological signal acquisition unit 20, and is common to a plurality of sensor electrodes. The ground electrode 21 is arranged on a part of the housing 10, for example, as a surface facing the head side, on a surface near the ear hook at the rear of the temple portion 10c, and comes into contact with the vicinity of the ear.

The type, number, position, shape, and the like of the electrodes and devices for acquiring the biological signal mounted on the HMD are not limited to the example of the first embodiment. The HMD may be configured to be provided with a device including electrodes so that an electromyographic signal of at least one facial muscle can be acquired.

[About load]
The band 26 of FIG. 2 connects between the rear ends of the ear hooks of the left and right temple portions 10c of the housing 10. As a result, the housing 10 is stably fixed to the user's head, and the contact between the electrodes 3 and 5 and the skin is more reliable.

When the user wears the HMD1 on the head, the device 2 (2R, 2L) is arranged so as to be in contact with the vicinity of the cheekbone, and the device 4 (4R, 4L) is arranged so as to be in contact with the vicinity of the eyebrows. ing. In particular, the device 2 (2R, 2L) is arranged in such a state that a part of the portion including the electrode 3 comes into contact with and rides on the skin of the facial muscle near the cheekbone. As a result, the device 2 (particularly the electrode 3) becomes a fulcrum that supports a part of the load of the HMD 1. In a conventional general spectacle-type HMD, the load of the entire HMD is supported by the nose and ears where the housing mainly contacts. In the HMD 1 of the first embodiment, the load of the entire HMD 1 is supported not only at the nose and the ear, but also at the device 2, and the load is distributed at the plurality of the devices. The shape and the like of the device 2 are designed so that the contact between the electrode 3 and the portion of the facial muscle and the state of the load on the skin are suitable.

Thereby, the HMD 1 of the first embodiment can reduce the weight of the housing 10 or the like applied to the user's ears or nose as compared with the conventional HMD without the device 2 or the like. As a result, the user's feeling of wearing the HMD can be improved, and the discomfort due to the weight of the HMD can be prevented. Further, while the load is supported by the device 2, the contact between the electrode 3 and the portion of the facial muscle becomes better, and the detection of the EMG signal becomes more stable.

The inner surface of the housing 25 of the device 2 may be arranged diagonally with respect to the XZ surface or may be formed as a curved surface so as to fit well with the cheek surface, for example. The device 2 may have an elliptical shape or another shape in a cross section perpendicular to the V direction. The device 2 may be bent in the V direction. The device 2 is not limited to rigidity and may be composed of an elastic body.

It is more preferable that there is a mechanism that can adjust the state such as the arrangement and shape of the device 2 so that it is easy to deal with individual differences of users. One example is: The HMD 1 has a mechanism that can adjust the position of the device 2 from the temple portion 10c of the housing 10 by sliding in each direction, bending, stretching, or the like. The device 2 may be expanded and contracted in the V direction, for example. Further, in the device 2, the tip of the device 2 may be movable inward or outward in the Y direction or in another direction with the connection point with the temple portion 10c as a fulcrum. By incorporating an elastic body in the temple portion 10c or the device 2, the device 2 may be pressed against the inner face. As a result, the device 2 is stably held, and the electrode 3 of the device 2 and the portion of the facial muscle are in constant contact with each other. Similarly, it is more preferable that the device 4 has a mechanism capable of adjusting the state such as arrangement and shape.

Further, in this example, a voice input device 18 including a microphone is also mounted near the tip of the device 2. Since this microphone is placed near the mouth, it is easy to collect sound. Not limited to this, other components may be mounted on the devices 2 and 4.

Further, the HMD 1 may be provided with a mechanism that can be adjusted by the user so that the devices 2 and 4 do not come into contact with the face when the biological signal is not used, that is, when the function is off. The mechanism may be such that the devices 2 and 4 can be attached to and detached from the housing 10. A mechanism or the like that allows the devices 2 and 4 to be spread outward with respect to the face may be used.

[Biomedical signal acquisition unit]
FIG. 4 shows a configuration example of the biological signal acquisition unit 20 and the sensor electrode. The biological information acquisition unit 20 includes an electromyogram signal sensor 201 and an electro-oculography signal sensor 202. In FIG. 4, the EMG signal sensor 201 acquires two channels of EMG signals (signals sg1, sg2) from the electrodes 3A (3a to 3d) of the device 2, and the electrodes 3B (3e to 3h) of the device 4. Two-channel EMG signals (signals sg3, sg4) are acquired from. The electro-oculography signal sensor 202 acquires a 2-channel electro-oculography signal (signals sg5, sg6) from the electrodes 5 (5a, 5b). The biological information acquisition unit 20 acquires biological signals from the user's head and face by using these sensors.

The electromyographic signal from the pair of two electrodes 3 is used as a one-channel signal. For example, in the device 2R, the potential difference between the detection signal of the electrode 3a (difference from the ground electrode 21) and the detection signal of the electrode 3b (difference from the ground electrode 21) is the signal sg1 as one myocardial signal. This signal sg1 is an electromyographic signal of the facial muscle near the right cheek. Similarly, the signal sg2 from the electrodes 3c and 3d of the device 2L is an electromyographic signal of the facial muscle near the left cheek. The signal sg3 from the electrodes 3e and 3f of the device 4R is an electromyographic signal of the facial muscle near the right eyebrow. The signal sg4 from the electrodes 3g and 3h of the device 4L is an electromyographic signal of the facial muscle near the left eyebrow.

Although not shown, the EMG signal sensor 201 and the electrooculogram signal sensor 202 are provided with an amplifier circuit, a noise removal circuit, a rectifier circuit, and the like, and the detected signal is processed by these circuits. The biometric signal acquisition unit 20 has four (4 channels) EMG signals (signals sg1 to sg4) from the EMG signal sensor 201 and two (2 channels) electromyograms from the electrooculogram signal sensor 202. The signal (signals sg5, sg6) and the signal are acquired. The biological signal acquisition unit 20 processes those biological signals and stores them in a memory as data. The controller 101 performs a predetermined control process using the data of those biological signals.

The process performed by the biological signal acquisition unit 20 may be a process of calculating a predetermined feature amount as a process of calculating an electromyogram signal. As an example of the feature amount of the EMG signal, the period, frequency, etc. may be used in addition to the level such as intensity and amplitude.

[HMD-Functional block]
FIG. 5 shows an example of a functional block configuration of HMD1. The HMD 1 includes a processor 101, a memory 102, a camera 12, a distance measuring sensor 13, a sensor unit 14, a display device 103 including a display surface 11, a communication device 104, a voice input device 18 including a microphone, and a voice output device 19 including a speaker. , Operation input unit 105, battery 106, biometric signal acquisition unit 20, and the like. These elements are connected to each other through a bus or the like.

The processor 101 is composed of a CPU, ROM, RAM, etc., and constitutes an HMD1 controller. The processor 101 realizes functions such as an OS, middleware, and applications and other functions by executing processing according to the control program 31 and the application program 32 of the memory 102. The memory 102 is composed of a non-volatile storage device or the like, and stores various data and information handled by the processor 101 and the like. The memory 102 also stores images, detection information, and the like acquired by the camera 12 and the like as temporary information.

The camera 12 acquires an image by converting the light incident from the lens into an electric signal by an image sensor. When a TOF sensor is used, for example, the distance measuring sensor 13 calculates the distance to the object from the time until the light emitted to the outside hits the object and returns. The sensor unit 14 includes, for example, an acceleration sensor 141, a gyro sensor (in other words, an angular velocity sensor) 142, a geomagnetic sensor 143, and a GPS receiver 144. The sensor unit 14 uses the detection information of these sensors to detect states such as the position, orientation, and movement of the HMD1. The HMD is not limited to this, and may include an illuminance sensor, a proximity sensor, a barometric pressure sensor, and the like.

The display device 103 includes a display drive circuit and a display surface 11, and displays an image on the display surface 11 based on the display information 34. The communication device 104 includes a communication processing circuit, an antenna, and the like corresponding to various predetermined communication interfaces. Examples of communication interfaces include mobile networks, Wi-Fi (registered trademark), Bluetooth (registered trademark), infrared rays and the like. The communication device 104 performs wireless communication processing and the like with an external device. The communication device 104 also performs short-range communication processing with the actuator.

The voice input device 18 converts the input voice from the microphone into voice data. The voice output device 19 outputs voice from a speaker or the like based on the voice data. The voice input device may include a voice recognition function. The voice output device may include a voice synthesis function. The operation input unit 107 is a part that receives operation inputs to the HMD 1, such as power on / off and volume adjustment, and is composed of a hardware button, a touch sensor, and the like. The battery 108 supplies electric power to each part.

The memory 102 stores a control program 31, an application program 32, setting information 33, display information 34, biological signal data 35, control information 36, and the like. The control program 31 is a program for realizing the unique function in the first embodiment. The application program 32 is a program that realizes predetermined functions such as user guidance and work support by, for example, AR. The setting information 33 includes system setting information and user setting information related to each function. The display information 34 is information or data for displaying an image on the display surface 11. The biological signal data 35 is the data of the biological signal acquired by the biological signal acquisition unit 20 and the processed data thereof. The control information 36 is information for management / control related to a function peculiar to the first embodiment (output content adjusting function using an EMG signal). The control information 36, which will be described later, is information / data for controlling output contents based on information / data such as tests, evaluations, and learnings, and estimation of a psychological state from biological signals.

The controller by the processor 101 has a communication control unit 101A, a display control unit 101B, a data processing unit 101C, and a data acquisition unit 101D as a configuration example of a functional block realized by processing. The communication control unit 101A controls communication processing using the communication device 104 when communicating with an external device or the like. The display control unit 101B controls the image display on the display surface 11 of the display device 103 by using the display information 34. The data processing unit 101C performs processing related to a unique function while reading and writing the biological signal data 35 and the control information 36. The data acquisition unit 101D acquires various information / data necessary for processing. The data acquisition unit 101D acquires biological signal data from the biological signal acquisition unit 20, and acquires various data from the camera 12, the distance measuring sensor 13, the sensor unit 14, and the like.

[Facial muscle]
FIG. 12 schematically shows an example of facial facial muscles. Examples of facial muscles capable of acquiring an electromyographic signal by the electrode 3 provided in the HMD 1 of the first embodiment include the following. That is, the facial muscles include the frontalis muscle, the corrugator supercilii muscle, the orbicularis oculi muscle, the upper lip levitation muscle, the laughing muscle, the zygomaticus minor muscle, and the zygomaticus major muscle as shown in FIGS.

Among these facial muscles, the frontalis muscle, the corrugator supercilii muscle, the orbicularis oculi muscle, and the levator labii muscle are provided with electrodes 3 for detecting signals from at least one of them. In the example of the first embodiment, the electrode 3B is arranged on the device 4 arranged on the upper side of the binocular portion 10a in the vicinity of the binocular portion 10a. With this electrode 3B, it is possible to efficiently detect the EMG signal from those facial muscles, particularly the corrugator supercilii muscle.

For the risorius muscle, the zygomaticus minor muscle, and the zygomaticus major muscle, an electrode 3 for detecting a signal from at least one of them is arranged. In the example of the first embodiment, the electrode 3A is arranged on the device 2 arranged so as to extend from the temple portion 10c along the cheek. With this electrode 3A, it is possible to efficiently detect the electromyographic signal from those facial muscles, particularly the portion from the zygomaticus minor muscle to the zygomaticus major muscle.

Electrodes 5 for acquiring electrooculogram signals are arranged near the nasal muscles and the orbicularis oculi muscles. Further, in the HMD1 of the first embodiment, as described above, a part of the load of the HMD1 is supported by the electrodes 3A arranged for the facial muscles near the cheekbones located below the wearer's eyes. The target facial muscles are, for example, the orbicularis oculi muscle, the levator labii superior muscle, the risorius muscle, the zygomaticus minor muscle, or the zygomaticus major muscle.

[Psychological state]
The psychological state can be inferred from the EMG signal of the facial muscles. A known technique may be applied to a method of estimating and evaluating a psychological state from an electromyographic signal of a facial muscle. As described above, the psychological states that can be inferred from the EMG signal based on the known technique include states such as comfort / discomfort, relaxation, concentration, stress, frustration, fatigue, arousal, and anger. In addition, the degree of these various psychology and emotions can be inferred. For example, the degree of comfort may be quantified in multiple stages.

In the first embodiment, the psychological state estimated from the EMG signal is associated with the state of the user's evaluation of the GUI image which is the output content, and simply put, the user feels the output content. Corresponds to the state.

The HMD 1 displays an image on the display surface 11 and acquires an electromyogram signal through the device 2 or the like, and adjusts a GUI image or the like as an output content according to a psychological state estimated by the electromyogram signal. Thereby, the HMD1 of the first embodiment is adjusted so as to have a more suitable output content before the physical movement of the user is affected, for example, before it appears as a large stress. This makes it possible to provide the user with a more suitable HMD usage environment. For example, when the HMD1 displays a GUI image as an output by a guidance or work support application, the GUI image can be adjusted while observing the degree of comfort or discomfort of the user from the EMG signal. The HMD1 can adjust, for example, the type, amount, size, fineness, speed, and the like of GUI images for work support instructions and explanations.

[Test output and evaluation input]
The HMD1 of the first embodiment outputs the output content as a test in advance in order to estimate and evaluate the psychological state of the user. At the time of this test output, the HMD 1 acquires the user's EMG signal and the user's subjective evaluation value. Based on the acquired information / data, the HMD1 learns about the correspondence between the output content, the psychological state, and the EMG signal. Based on learning, HMD1 creates and updates control information (control information 36 in FIG. 5) for controlling adjustment of output contents. This process such as test output also corresponds to calibration for adapting to individual user differences. The HMD1 creates and holds a dictionary for each user as control information based on the processing such as this test, evaluation, and learning.

FIG. 6 shows a flow of processing such as test output by HMD1 and has steps S601 to S605. The flow of FIG. 6 is similarly repeated for each test unit. The user wears the HMD1 on the head. In the HMD1, the on / off state of the function related to the biological signal can be set by the user. The HMD1 performs the following processing when the function is on.

First, in step S601, the HMD1 outputs to initialize the psychological state of the user. This initialization is to make the psychological state as neutral as possible and to relax it. The output of this initialization includes, for example, a playback display of video or music that allows the user to relax, an output of a message to relax, and the like. This output may be performed by the HMD1 or an external device (display or the like) different from the HMD1 may be used.

Next, in step S602, the HMD1 outputs for a test that allows the user to experience a psychological state. During this test output, the HMD 1 acquires the user's EMG signal through the device 2 and the like. This test output includes playback display of various images and music for causing various psychology and emotions such as pleasure and discomfort, and work instructions through GUI images for work using HMD1. The HMD1 performs this test output as a repetition of a plurality of test units while changing the type of video and music and the type of work every hour. The HMD 1 changes the image including the GUI displayed on the display surface 11 at the time of the work instruction. The HMD1 stores the acquired EMG signal data together with the output content information for each test.

In step S603, the HMD1 asks the user to input the degree of psychology / emotion such as comfort / discomfort for the above test as a subjective evaluation. For example, the HMD 1 displays a GUI image for subjective evaluation input on the display surface 11, and asks the user to select and input an evaluation value for each test unit. For example, the degree of comfort / discomfort can be selectively input for the GUI image of the work instruction in a certain test unit. This subjective evaluation corresponds to the user's psychological state with respect to the output during the test.

In step S604, the HMD1 stores the test output content information, the EMG signal, and the subjective evaluation input information in association with each other as data.

In step S605, the HMD1 learns about the relationship between the output content (for example, GUI image), the psychological state, and the EMG signal using the data accumulated by the processing of the above test, and records the learning result. For this learning, machine learning using a neural network or the like may be applied. Then, the HMD 1 sets and customizes the correspondence relationship between the EMG signal and the output content as control information (control information 36 in FIG. 5) based on the learning result. This control information is information for defining and controlling what kind of output content is to be obtained in what kind of EMG signal state. This control information may be, for example, a table or the like that defines the correspondence. This control information may be a correspondence between the EMG signal and the output content, and it is possible to omit interposing a value representing a psychological state between them. This control information is updated according to the test.

After that, when the function is on, the HMD1 can control the adjustment of the output content according to the state of the biological signal based on the above control information.

[Test output example]
FIG. 7 shows an example of the test output, and shows an example in which a GUI image for a test is displayed on the display surface 11 of the HMD1. In the test of this example, the work task for the user is displayed as a work instruction or explanation by a GUI image. The three examples (A), (B), and (C) in FIG. 7 are examples of GUI images in which the amount of explanation at that time is changed. The amount of explanation is large in the order of the images (A), (B), and (C). For example, in the image 701 of (A), an image 710 (indicated by a square or a circle) of a virtual object is displayed on the right side in an area of a predetermined size corresponding to the size of the display surface 11. There is. The object 710 is user operable for work. The operation corresponds to the function of the main body of the HMD1, and examples of known techniques include operation by an operating device, recognition of hand gestures, voice recognition, operation by line-of-sight detection, and the like. In the area of the image 701, the image 720 of the work explanation is displayed on the left side. In the case of the transparent display method, when viewed from the user, an image (image 710 or image 720) as shown in FIG. 7 is superimposed and displayed on the actual object on the display surface 11.

The image 720 such as the work explanation is the image 720A having a relatively large amount of explanation in the image 701 of (A), the image 720B having a medium explanation amount in the image 702 of (B), and the explanation amount in the image 703 of (C). It is said that the image is 720C with few images. For each image of these tests, the user inputs a subjective evaluation as to whether the amount of explanation by the image 720 or the like was appropriate for him / her, whether it was comfortable, easy to understand, or the like (step S603). What kind of explanatory amount or the like is suitable as the output content depends on the individual user. In some cases, an image with a large amount of explanation is more suitable, and in other cases, an image with a small amount of explanation is more suitable. The amount of information of suitable instructions / explanations required by the user changes depending on the skill level of the user for the work of a certain application.

[Evaluation input example]
FIG. 8 shows an example of the output at the time of the subjective evaluation input (step S603). The image 801 of FIG. 8A is an example in which the HMD 1 displays a GUI image for subjective evaluation input on the display surface 11. The image 801 has an image of a message prompting a subjective evaluation input such as "How was the amount of explanation?" And an image 810 by a GUI component such as a scale for inputting the subjective evaluation value. The scale of the image 810 shows the degree of appropriateness and satisfaction, such as insufficient, appropriate, and excessive, regarding how to feel the amount of explanation. The user can input the value selected on the scale by the operation as the subjective evaluation value. In this example, in the image 810 of the scale, "appropriate" is set as the center value, and "insufficient" and "excessive" are arranged as values that increase the inadequacy on the left and right. The evaluation input format is not limited to this.

Image 802 of FIG. 8B shows another display example of the evaluation input. In this example, the image 820 for evaluation input is a scale for inputting an evaluation value regarding how to feel the explanatory amount, as in (A), but it is "comfortable" and "unpleasant" depending on the operation of the bar. It is a format in which the degree of comfort and discomfort can be input as an evaluation value.

The HMD1 sets the control information for adjusting the output contents for each user and each application in steps S604 and S605 of FIG. 6 based on the learning of the test and the evaluation input as described above. The HMD1 reflects the user's evaluation value in the user-specific dictionary of control information. The HMD1 determines a GUI image or the like as a candidate for output content adjustment according to the classification of the correspondence between the output content and the user's psychological state (corresponding biological signal state), and sets it as control information. back. After that, the HMD1 appropriately adjusts the output contents such as the GUI image based on the control information when the user actually uses the application of the HMD1. As the dictionary is updated for each user, the sensitivity of the output content adjustment function gradually increases, and more suitable adjustment becomes possible.

The output of the GUI image or the like when actually using the application and the output at the time of the test output as shown in FIG. 7 may be basically in the same format, or the test output may be the actual output. It may be simplified as compared with.

[Transformation example-evaluation input]
The image 803 of FIG. 8C shows another display example regarding the subjective evaluation input in the modified example of the first embodiment. The HMD 1 is an image 830 for evaluation input so that the user can input a subjective evaluation value for the output content at that time while displaying the GUI image on the display surface 11 as a user process when actually using the application. Is displayed. In this example, it is assumed that the image of the test output and the image when actually using the application are similar. The HMD 1 displays, for example, an image 803 similar to the image 702 of FIG. 7 (B) on the display surface 11 during user processing when using the application. The image 803 has an image 710 of the object and an image 720 (702B) of the description. Further, at that time, the HMD 1 displays the image 830 for evaluation input in a part of the area of the display surface 11 together with those images. The HMD 1 may display the image 830 at the edge of the region or the like so as not to interfere with the main image. The user can immediately input a subjective evaluation value for the output content (for example, image 702B) at that time to the image 830 and reflect it in the evaluation.

According to this modification, the HMD1 can use such an evaluation input at the time of actual use as a complement or an alternative to the preliminary test for collecting data related to the output content adjustment control. When the image 830 for evaluation input is displayed at the time of actual use as shown in FIG. 8C, the user does not have to ignore the image 830 and input the evaluation at any time. Just do. The image 830 for evaluation input may be displayed according to a predetermined user operation, or may be displayed at regular time intervals.

As another modification, in the middle of the test output as shown in FIG. 7, the evaluation target image (for example, image 720) is displayed for each test unit, and the evaluation input image is shown as shown in FIG. 8 (C). 830 may be displayed. When the screen and time are divided between the evaluation target and the evaluation input as shown in FIGS. 7 and 8 (A), the evaluation target becomes clearer. When the evaluation target and the evaluation input are mixed in the screen as shown in FIG. 8C, the efficiency of the evaluation can be improved.

[Output content adjustment]
The adjustment of the output content according to the psychological state of the user (the state of the corresponding EMG signal) is not limited to the change of the information amount of the GUI image, and various elements such as the following are possible. The target of the output content adjustment is not limited to the amount and fineness of the explanatory text of the image 720 of FIG. 7, and the following is also possible. It is also possible to change the type, size, color, brightness, character type (font), etc. of the GUI image. Examples of the type of GUI image include various GUI parts (also called Widgets and controls). Examples of GUI components include windows, text boxes, dialog boxes, buttons, icons, bars, lists, menus, and the like. For example, even if the same explanation is used, the effect on psychology differs depending on what GUI parts, colors, character types, etc. are used for presentation. The effect is different depending on the size (area, etc.) and display position of the explanatory image. For example, adjustment may be made to increase or decrease the ratio of the explanatory image to the area of the display surface 11. The effect depends on whether the description is placed in a fixed position or near the object. The effect will differ depending on whether the explanation is in text or in pictures, figures, or icons.

The output of HMD1 is not limited to image display, but it is also possible to output voice, vibration, light emission, etc. according to the function of HMD1. When vibration and light emission are possible, the HMD 1 includes a vibrator and a light emitting device in the housing 10 and the like. Regarding the control of the adjustment of the output contents, it is possible to control the various outputs thereof and their combinations. In the case of voice, light emission, and vibration control, it is possible to adjust the presence / absence, type, amount, timing, etc. of them. The use of sound, light emission, and vibration with or instead of images also has different psychological effects. The effect is different depending on whether the explanation is read aloud, BGM is played, or the type of voice or BGM is changed. The effect is different depending on the type of viewing content such as output video or BGM. For example, if the user is presumed to be bored, the type of viewing content may be changed.

The output content may be adjusted as the output speed. For example, if there are a plurality of sentences or a plurality of images for work explanation and they are automatically sent, the speed can be adjusted.

Further, in the case of a game whose difficulty level can be changed as an application provided by HMD1 to a user, the difficulty level of the game can be set as an adjustment target. The HMD1 adjusts in real time from the occasional EMG signals during the game so that the difficulty level is appropriate for the user. Alternatively, in the case of content in which the story can be branched and the purpose is to make the user feel thrill, the adjustment target may be the story based on the branch. The HMD1 changes the story by selecting a branch from the EMG signal so that the user feels an appropriate thrill in the progress of the story.

[Processing flow]
FIG. 9 shows a processing flow in the HMD 1 of the first embodiment when the user processes the user when the user actually uses the application and adjusts the output contents in real time. This process is a process after the correspondence between the EMG signal and the output content is once set as control information based on the test and evaluation as shown in FIG. FIG. 9 has steps S901 to S906. As an example, a process in which an application such as work support provided for HMD1 gives an instruction or explanation regarding work to a user by displaying a GUI image on the display surface 11 will be described.

In step S901, the HMD1 executes the processing of the application used by the user, determines, for example, a work instruction / explanation, and first outputs the first output based on the above-mentioned control information related to the output content adjustment or the initial setting information. Determine the content and output. For example, the HMD 1 displays a GUI image similar to that shown in FIG. 7 (B).

In step S902, the HMD1 acquires the electromyographic signal as time-series data through the devices 2 and 4 and the biological signal acquisition unit 20 together with the above output. The HMD1 may have acquired the EMG signal before the output.

In step S903, the HMD 1 determines whether or not the current output content is appropriate, for example, whether or not the amount of explanation by the GUI image is large or small, based on the psychological state associated with the acquired EMG signal of the user. Guess based on. With HMD1, for example, when the feature amount of the EMG signal at that time represents discomfort, high stress, etc., it can be inferred that the user feels that the amount of explanation is too much or too little, which is pleasant. If it indicates low stress or the like, it can be inferred that the user feels that the amount of explanation is appropriate.

In step S904, based on the above estimation, the HMD1 determines whether the state of comfort / discomfort, stress, satisfaction, etc. as the psychological state of the user is within a predetermined allowable range based on the control information. The permissible range can be defined by a threshold value or the like. For example, the state of the feature amount of a certain EMG signal represents the degree of comfort, and when the degree of comfort is equal to or higher than the threshold value, the user feels comfortable or satisfied with the output content at that time. I can guess that there is. Alternatively, the state of the feature amount of a certain EMG signal represents the degree of discomfort, and when the degree of discomfort is equal to or higher than the threshold value, the user feels discomfort or dissatisfaction with the output content at that time. I can guess that there is.

If it is within the permissible range (Y) in step S904, step S905 is skipped and the process proceeds to step S906, and if it is out of the permissible range (N), the process proceeds to step S905.

In step S905, the HMD1 adjusts the output content based on the control information. The HMD1 determines, for example, to adjust the output content so as to reduce the amount of explanation when it is estimated that the user feels that the comfort is low and the amount of explanation is too large. As an adjustment of the output content, the HMD 1 changes, for example, the image 720B as shown in FIG. 7 (B) to the image 720C as shown in FIG. 7 (C) so as to provide a more concise explanation. On the contrary, when it is estimated that the user feels that the comfort is low and the explanation amount is small, the HMD1 determines the adjustment of the output content so as to increase the explanation amount. HMD1 changes, for example, the image 720B as shown in FIG. 7 (B) to the image 720A as shown in FIG. 7 (A) so as to include more information.

The HMD1 determines from the EMG signal that the output content (eg, image 720) remains unchanged when the user infers that the current output content is within the permissible range of satisfaction. .. This corresponds to the case of proceeding from step S904 to step S906. In step S906, HMD1 confirms whether the user processing is completed or continued, and if it is terminated (Y), the flow ends, and if it is continued (N), it returns to step S901 and the same is performed every hour. It is a repetition of control.

[Correspondence]
As a supplement, an example of the correspondence between the EMG signal, the psychological state, and the output content will be described. As the facial expression muscles, the corrugator supercilii muscle and the zygomaticus major muscle as shown in FIG. 12 will be described as an example. The corrugator supercilii is a muscle that wrinkles the eyebrows, and the tension of this muscle means generally negative emotions such as frustration, stress, concentration, tension, and discomfort. The zygomaticus major muscle is a muscle that makes a smile, and the tension of this muscle means generally positive emotions such as a sense of security and pleasure. The zygomaticus minor muscle is also a muscle that makes a smile, but I will explain it with the zygomaticus major muscle as a representative.

In the first embodiment, the electromyographic signal that can be detected from the zygomaticus major muscle corresponds to the signals sg1 and sg2 that can be detected from the electrode 3A of the device 2 of FIG. The electromyographic signal that can be detected from the corrugator supercilii corresponds to the signals sg3 and sg4 that can be detected from the electrode 3B of the device 4 in FIG. In the first embodiment, the EMG signal from the zygomaticus major muscle is used as an index of a positive state, and the EMG signal from the corrugator supercilii muscle is used as an index of a negative state, and the psychological state is estimated by a combination thereof. do.

A classic example of how the EMG signals from these two types of muscles change depending on the amount of descriptive text in image 720, such as Figure 7, in the output of a work support application, for example. It is shown below. First, as in the example of image 701 in (A), there is a case where the explanation is long and the amount of information is larger than necessary for a certain user. In this case, the user does not have any trouble in performing the work because there are many explanations, so that the user is not particularly nervous and the level of the EMG signal from the corrugator supercilii is weak. On the other hand, since the explanation is redundant, it does not feel very pleasant, and the level of the EMG signal from the zygomaticus major muscle is also weakened.

Next, as in the example of image 702 in (B), when the length of the explanation is appropriate for the user and there is a necessary amount of information. In this case as well, the user does not have any trouble in performing the work, so that the user is not particularly nervous and the level of the EMG signal from the corrugator supercilii is weak. On the other hand, since the work can be carried out comfortably without the redundancy of the explanation, the level of the EMG signal from the zygomaticus major muscle becomes strong.

Finally, as in the example of image 703 in (C), when the length of the explanation is short for the user and the necessary information is insufficient. In this case, the user feels frustrated because it is difficult to perform the work, and the level of the EMG signal from the corrugator supercilii becomes stronger. On the other hand, the level of the EMG signal from the zygomaticus major muscle is weakened because it is difficult to perform the work and the situation is unpleasant.

FIG. 10 is a table summarizing an example of the correspondence between the output content and the level of the EMG signal as described above. As shown in this table, the combination of the levels of the EMG signals of the two types of facial muscles generally appears as, for example, three types of patterns, depending on the amount of the description of the image 720 which is the output content. .. Pattern 1 corresponds to the example of image 701 of (A) above, pattern 2 corresponds to the example of image 702 of (B) above, and pattern 3 corresponds to the example of image 703 of (C) above. ing. As described above, the state of the EMG signal from the facial muscles, particularly the state of the level such as the strength in the combination of the plurality of EMG signals from the plurality of facial muscles, reflects the psychological state of the user. On the contrary, by grasping the state of the EMG signal from the facial muscle, it is possible to infer how the user feels about the output content.

HMD1 collects, learns, and grasps data on the pattern of the correspondence between such output contents and EMG signals based on the above-mentioned tests and evaluations. The HMD1 can control the adjustment of the output content according to the EMG signal by using the control information based on such a correspondence. For example, when the levels of the two types of EMG signals are both weak, it can be inferred that the user feels that the amount of explanatory text is large for the GUI image at that time based on the pattern 1. Therefore, in this case, the HMD 1 may attempt to adjust the output content by using the various means described above so as to reduce the amount of information in the GUI image.

[Control information]
FIG. 11 shows a configuration example of control information (control information 36 in FIG. 5) related to the function of adjusting the output content according to the EMG signal. This control information is created as a setting for each user and each application. This table of control information has an electromyographic signal, a psychological state, and output contents as columns. The "electromyographic signal" column stores, for example, a pattern or classification of a feature amount calculated from the electromyographic signal, for example, a range of intensity. Although the "psychological state" column can be omitted, a value representing the user's psychological state associated with the feature amount of the EMG signal, for example, the degree of comfort / discomfort or satisfaction, etc. is stored. The "output content" column stores information that defines output content that is a candidate for output when adjusting the output content, for example, information such as the type of GUI image to be displayed and the amount of explanation.

[Effects (1)]
As described above, according to the HMD 1 of the first embodiment, the psychological / emotional state of the user can be quickly and deeply estimated by using the electromyographic signal acquired through the device 2 or the like, and the output content is based on the estimation. Can be suitably adjusted. According to the first embodiment, the GUI image or the like for each application can be adjusted to a suitable state according to the state of the EMG signal of the individual user. According to the first embodiment, it is possible to quickly adjust the output content to a suitable one before the physical movement of the user is affected, for example, before it appears as a large stress.

The following is also possible as a modification of the first embodiment. The HMD1 processor is not limited to the mode in which the control process such as the output content adjustment described above is performed. As a modified example system, the HMD may communicate with an external device such as a server, and the external device may perform the control process thereof.

[Modification 1]
FIG. 13 shows the configuration of HMD1 of the first modification of the first embodiment. Not limited to the example of FIG. 2 of the first embodiment, various arrangements and shapes of the above-mentioned devices 2 and 4 and the sensor electrodes are possible. FIG. 13 is a modified example of the attachment structure of the device 2 and the electrode 3 to the housing 10, and other structural parts are the same as those in FIG. The HMD 1 of the first modification is provided with the device 2 so as to be extended and connected to the lower side of the binocular portion 10a which is the front surface portion of the housing 10. Similar to the first embodiment, the device 2 is provided as left and right devices 2R and 2L in a symmetrical shape. Similar to the first embodiment, the housing 28 of the device 2 (2R, 2L) is for detecting an electrocardiogram signal from the facial muscles near the cheeks (the portion from the zygomaticus major muscle to the zygomaticus minor muscle). Electrodes 3i to 3l are provided as the electrodes 3A. The device 2R on the right eye side is provided with electrodes 3i and 3j on the lower surface side. The device 2L on the left eye side is provided with electrodes 3k and 3l on the lower surface side. A microphone is provided near the tip of the device 2 as in the first embodiment.

This device 2 functions as a cheekbone pad, and is arranged so that a part of the device 2 including the electrode 3 comes into contact with the skin near the cheekbone and rides on the skin. In other words, the binocular portion 10a is placed on the upper side of the cheekbone via the device 2. As a result, as in the first embodiment, the device 2 supports a part of the load of the HMD 1 and distributes the load. As a result, in particular, the load of the HMD 1 applied to the vicinity of the nose pad of the nose can be dispersed by the device 2, and the user's wearing sensation can be improved.

As another modification related to the mounting of the sensor electrode, in the case of the goggle type housing as shown in FIG. 1B, the housing that comes into contact with the skin without providing the expansion portion as in the device 2 The electrode 3 may be partially provided.

<Embodiment 2>
The HMD of the second embodiment will be described with reference to FIG. 14 and the like. The basic configuration of the second embodiment and the like is the same as that of the first embodiment, and the components different from the first embodiment of the second embodiment and the like will be mainly described below. The configuration of the electrode 3 and the like of the device 2 in the HMD 1 of the second embodiment is the same as that of FIG. The HMD of the second embodiment also includes the electrode 3 for detecting the electromyogram signal for reading the unconscious psychological state of the user described in the first embodiment. In the second embodiment, the electrode 3 is also used to detect a user's conscious operation of facial muscle movement (also referred to as voluntary input). The HMD 1 detects the user's conscious operation of the facial muscle movement as a voluntary input from the electromyographic signal of the electrode 3. The user makes a voluntary input such as consciously raising the right cheek. The HMD1 uses the detected information of the voluntary input as one of the inputs to the HMD1, for example, the input information for the GUI image of the application.

The HMD 1 describes an electromyographic signal regarding the operation of the facial muscle movement according to the unconscious psychological state of the user described in the first embodiment and the operation of the user's conscious facial muscle movement described in the second embodiment. Judgment and switching according to the level of strength etc. When the EMG signal is also used as a conscious input signal, the HMD1 uses the EMG signal after rectifying it. The HMD1 compares the level of the rectified EMG signal with a predetermined threshold value, and when the level is equal to or higher than the threshold value, sometimes treats the signal as a conscious voluntary input. The HMD1 treats the signal as a signal representing an unconscious psychological state when and sometimes the level is below the threshold. The HMD1 stops estimating the unconscious psychological state at the time when the EMG signal is treated as a conscious voluntary input. The HMD1 can set a threshold value according to the user for the threshold value of the level of the EMG signal.

The HMD 1 of the second embodiment includes a circuit or the like that rectifies the electromyographic signal acquired from the electrode 3. The electromyogram signal sensor 201 of the biological signal acquisition unit 20 of FIG. 4 is provided with a circuit or the like for rectifying the signal sg1 or the like. The circuit of the EMG signal sensor 201 cuts off a predetermined high frequency band component from the EMG signal in order to remove noise, and removes the DC component due to the polarization potential generated between the electrode 3 and the skin. Therefore, a predetermined low frequency band component is also cut off. The EMG signal sensor 201 is provided with a filter circuit or the like for such processing. As an example, the HMD1 of Embodiment 2 uses a bandwidth of 1 Hz to 500 Hz in an electromyographic signal.

[Processing flow]
FIG. 14 shows the processing flow of HMD1 of the second embodiment. The flow of FIG. 14 is similar to the flow of FIG. 9 of the first embodiment for many steps, and mainly has steps S1403 and S1408 as different parts. In step S1403, the HMD1 compares the level of the strength of the EMG signal after rectification or the like with the threshold value of the EMG signal acquired during the user processing, and determines whether the EMG signal is equal to or higher than the threshold value. If it is equal to or more than the threshold value (Y), the process proceeds to step S1408, and if it is less than the threshold value (N), the process proceeds to step S1404. The processing of steps S1404 to S1407 is the same as that of the first embodiment.

In step S1408, the HMD1 determines that it is a voluntary input and processes the voluntary input. The HMD1 determines what kind of operation is intended as a voluntary input from the state of the levels of the plurality of electromyographic signals (signals sg1 to sg4) at that time. For example, when the level of the signal sg1 corresponding to the facial muscle of the right cheek is the highest, the HMD1 can be determined to be a voluntary input associated with the right cheek, and corresponds to the voluntary input based on the setting information. The attached operation (for example, affirmative button operation) can be identified. The HMD1 gives the OS or the application the information of the operation associated with the discriminated voluntary input. The OS or the application executes a predetermined process (for example, affirmative button input process) according to the voluntary input operation. After step S1408, the process proceeds to step 1407.

[Optional input]
The above voluntary input is realized as an operation in which the user consciously moves, for example, the facial muscles near the cheeks and the facial muscles near the eyebrows in FIG. This voluntary input operation can be used as a predetermined input associated with, for example, pressing a button of hardware or software for the HMD 1, cursor movement control, or the like. The HMD 1 of the second embodiment allows the user to set what kind of input operation is assigned to the voluntary input operation using the facial muscle, for example, for each user and each application.

FIG. 15 shows an example of user setting information regarding voluntary input. The HMD 1 displays user setting information on the display surface 11, for example, and enables the user to set the information. The user setting information regarding this voluntary input has "voluntary input" and "operation" as columns. In the "voluntary input" column, a plurality of voluntary inputs can be set by combining the electromyographic signals of each facial muscle. For example, "voluntary input 1" corresponds to an operation of moving the facial muscle of the right cheek, and corresponds to the level of the signal sg1 being equal to or higher than the threshold value. An operation selected by the user from the candidates, for example, an "affirmative button" can be assigned to this "voluntary input 1". Similarly, the "voluntary input 2" corresponds to the operation of moving the facial muscle of the left cheek and the signal sg2, and is assigned, for example, the operation of the "negative button". For example, when an application having HMD1 displays a GUI image of work support as in FIG. 7, the user selects an operation such as an "affirmative button" by "voluntary input 1" of this application or one of them. It can be given to an image 710 or the like of an object. The HMD1 application receives such a voluntary input operation through the OS and performs the prescribed processing.

As another example, in the case of cursor movement control, for example, "voluntary input 1" on the right cheek can be used as the right movement of the cursor, and "voluntary input 2" on the left cheek can be used as the left movement of the cursor. It is also possible to assign that the "voluntary input 3" of the right eyebrow is moved up the cursor and the "voluntary input 4" of the left eyebrow is moved down the cursor. In addition, a voluntary input operation may be set by combining the movements of a plurality of facial muscles. For example, when both the "voluntary input 3" of the right eyebrow and the "voluntary input 4" of the left eyebrow are turned on (at or above the threshold value) at the same time, the operation is accepted as a predetermined operation. It should be noted that the design item of the HMD 1 is not limited to the user setting, and may be fixedly set so that a predetermined voluntary input can be associated with a predetermined input operation in advance.

FIG. 16 shows a case where the cursor movement control by voluntary input is accepted in the image 1601 displayed on the display surface 11 by the HMD1. Here, the left-right direction in the display surface 11 is shown as the x direction, and the up-down direction is shown as the y direction. The HMD1 displays a GUI image 1602 such as an object depending on the OS or an application, and also displays a cursor 1603 for indicating a position.

The cursor 1603 is, for example, a cross-shaped GUI image. The user can operate the cursor 1603 by using the function of the main body of the HMD1 or the voluntary input of the facial muscles. The user, for example, moves the cursor 1603 to position it on the GUI image 1602 and then presses the affirmative or negative button. The user can perform operations such as cursor movement and affirmative button at this time by the above-mentioned optional input.

The following are detailed examples of cursor movement control by voluntary input. HMD1 acquires two channels of electromyographic signals (for example, signals sg1 and sg2 in FIG. 4) from two facial muscle locations, for example, the right cheek and the left cheek. The HMD1 controls to change the movement position (direction, movement amount, etc.) of the cursor 1603 according to the state of the combination of the acquired two-channel EMG signal levels. For example, the left and right signals sg1 and sg2 are used to control the position coordinates of the cursor 1603 in the left-right direction (x direction) in the display surface 11.

Let the intensity levels of the two EMG signals (eg, signals sg1, sg2) be M ₁ and M ₂ . Let the X coordinate value of the cursor be C _X. Let ΔC _X be the change in C _X in the unit control cycle Δt. Let k be the constant of proportionality. ΔC _X can be controlled by the following equation 1.
Equation 1: ΔC _X = k (M ₁ -M ₂ )

When moving the cursor in two dimensions, two channels of EMG signals (for example, right eyebrow and left eyebrow signals sg3 and sg4) are added for the y direction, for a total of four channels of EMG. It can be controlled in the same way by combining the signals shown in the figure.

[Effects (2)]
As described above, according to the HMD of the second embodiment, in addition to the effect that the output content can be adjusted according to the unconscious psychological state of the user as in the first embodiment, the user is conscious of the facial muscles. It is possible to perform a voluntary input operation by movement. This makes it possible to increase and diversify input means such as instructions to the HMD OS and applications.

The following is also possible as a modification of the second embodiment. In the modified example, the above-mentioned optional input may be used for the test output and the evaluation input as shown in FIG. For example, when the test image 702 as shown in FIG. 7B is displayed, the user can easily input the subjective evaluation value by using the voluntary input. For example, the voluntary input of the signal sg1 on the right cheek is an operation representing an evaluation of affirmation or comfort, and the voluntary input of the signal sg2 on the left cheek is an operation representing an evaluation of negation or discomfort. The HMD1 processes in response to an operation such as "affirmation" / "denial" by voluntary input so as to reflect it as a user's evaluation of the output content at that time.

<Embodiment 3>
The HMD of the third embodiment will be described with reference to FIG. 17 and the like. In the third embodiment, the electrooculogram signal that can be detected from the electrode 5 of FIG. 2 is used in combination with the electromyogram signal that can be detected from the electrode 3. In the HMD of the third embodiment, when performing a conscious voluntary input operation using the electromyographic signal of the facial muscle as in the second embodiment, the electrooculogram signal from the electrode 5 (signal sg5 in FIG. 4). , Sg6) are also used for control. As in FIG. 15, the user can be set as to what kind of voluntary input operation each signal of the EMG signal and the electrooculogram signal is assigned. In the third embodiment, for example, the cursor movement is controlled by the electromyographic signal and the button is pressed by the electromyographic signal, or the cursor movement is controlled by the electromyographic signal and the button is pressed by the electromyographic signal. It is possible to do.

There is a mutual influence between the movement of the facial muscles and the movement of the eyes, and there is a mutual influence between the EMG signal and the electrooculographic signal. Even if no voluntary input operation is performed during control by the EMG signal, the movement of the eyeball related to the electrooculogram signal is relative to the level of the EMG signal for detecting the unconscious psychological state. , Can be a hindrance. Therefore, in that case, the HMD of the third embodiment may stop the estimation / evaluation of the unconscious psychological state. For example, in the HMD, when the level of the electrooculogram signal acquired in response to the voluntary input operation using the user's conscious eye movement is equal to or higher than the threshold value, and sometimes, the psychological state using the EMG signal. Guessing and adjusting the output contents may be suspended. That is, the HMD controls while switching between the voluntary input and the psychological state estimation according to the level of the electro-oculography signal and the electromyogram signal on the time axis.

[Processing flow]
FIG. 17 shows a flow at the time of user processing of HMD1 in the third embodiment. The flow of FIG. 17 has steps S1702 and S1703 as main differences from the flow of FIG. 14 of the second embodiment. In step S1702, the HMD1 acquires both the electromyographic signal and the electrooculographic signal. In FIG. 4, the HMD 1 acquires an electromyographic signal (signals sg1 to sg4) from the electrodes 3 of the devices 2 and 4 and an electro-oculography signal (signals sg5, sg6) from the electrodes 5. In step S1703, in HMD1, the level of the EMG signal is equal to or higher than a predetermined threshold value (referred to as the threshold value A for the EMG signal), or the level of the electrooculogram signal is a predetermined threshold value (for the electrooculogram signal). It is determined whether or not it is equal to or higher than the threshold value B of). Each of these thresholds is user configurable. If any of the signals is less than the threshold value (N), the process proceeds to step S1704. If any of the signals is equal to or greater than the threshold value (Y), the process proceeds to step S1708. In steps S1704 to S1706, the HMD 1 adjusts the output content according to the EMG signal, as in the second embodiment. In step S1708, HMD1 performs a voluntary input process using at least one of the electromyogram signal and the electrooculogram signal which was equal to or higher than the threshold value in step S1703.

[Optional input]
FIG. 18 shows an example of a voluntary input operation when both an electromyogram signal and an electrooculogram signal are used in the third embodiment. The horizontal axis of FIG. 18 is time. It has the above-mentioned 4-channel electromyographic signals (signals sg1 to sg4) and 2-channel electro-oculography signals (signals sg5, sg6). The HMD1 performs an arbitrary input process according to the state of the combination of these signals. In this example, as the setting of the voluntary input, the 4-channel EMG signal is used for cursor movement control (referred to as voluntary input A), and the 2-channel electrooculogram signal is used for button pressing (referred to as voluntary input B). Is shown.

For example, at time T1, the level of any signal is below the threshold value. At this time T1, the output content is adjusted by guessing the psychological state using the EMG signal as needed. For example, as in FIG. 7, the amount of description of image 720 is adjusted. Next, at time T2, the level of the electromyographic signal (at least one of the signals sg1 to sg4) is equal to or higher than the threshold value. At this time T2, the cursor movement control is performed as the voluntary input A according to the states of the signals sg1 to sg4. For example, as in FIG. 16, the cursor 1603 is moved. Next, at time T3, the level of the electro-oculography signal (at least one of the signals sg5 and sg6) is equal to or higher than the threshold value. At this time T3, the button is pressed as the voluntary input B according to the states of the signals sg5 and sg6. For example, as in FIG. 16, the affirmative button or the negative button is pressed while the cursor 1603 is on the image 1602 of the object. At the time of the voluntary input operation of the times T2 and T3, the adjustment of the output content by the psychological state estimation is temporarily stopped.

Similarly, as another setting example, it is also possible to assign a button press operation to a voluntary input using an EMG signal and a cursor movement operation to a voluntary input using an electrooculogram signal. .. It is also possible to assign different types of operations using the electromyographic and electrooculographic signals of each channel.

Further, when the HMD1 simultaneously receives both the voluntary input by the electromyographic signal and the voluntary input by the electrooculogram signal, the HMD1 may be processed as follows. When the level of the EMG signal is equal to or higher than the threshold value and the level of the electrooculographic signal is equal to or higher than the threshold value, the HMD 1 may be processed as one predetermined voluntary input operation. Alternatively, the HMD 1 may be processed as a voluntary input operation associated therewith by using a signal that is relatively larger than the level of the EMG signal and the level of the electrooculogram signal.

[Effects (3)]
According to the third embodiment, in addition to adjusting the output content according to the estimation of the user's unconscious psychological state, the user can also perform a conscious voluntary input operation using the electromyogram signal and the electrooculogram signal. , It is possible to increase and diversify the input means for the HMD.

The following is also possible as a modification of the third embodiment. In the modified HMD, the EMG signal of the facial muscle is used only for acquiring information on the unconscious psychological state, and only the electrooculogram signal is used for the voluntary input.

Although the present invention has been specifically described above based on the embodiments, the present invention is not limited to the above-described embodiments and can be variously modified without departing from the gist.

1 ... HMD, 2 (2R, 2L) ... device, 3 (3a, 3b, 3c, 3d) ... electrode, 4 (4R, 4L) ... device, 5 ... electrode, 11 ... display surface, 20 ... biological signal acquisition unit ..

Claims (13)

1. Equipped with a device to acquire EMG signals from the user's facial muscles,
   The EMG signal is used as input information.
   Head-mounted display device.
2. In the head-mounted display device according to claim 1,
   The output content is adjusted according to the psychological state estimated based on the EMG signal.
   Head-mounted display device.
3. In the head-mounted display device according to claim 2,
   As the adjustment of the output content, the type or amount of information of the GUI image to be displayed is adjusted.
   Head-mounted display device.
4. In the head-mounted display device according to claim 1,
   The device has electrodes that come into contact with parts of the facial muscles other than the user's nose and ears and detect the electromyographic signal from the facial muscles.
   A part of the load of the head-mounted display device is supported by the contact point of the device including the electrode.
   Head-mounted display device.
5. In the head-mounted display device according to claim 1,
   As the EMG signal, a first EMG signal generated according to the state of the facial muscle unconsciously by the user and a second EMG signal generated according to the conscious operation of the movement of the facial muscle by the user. Detects the figure signal and
   The second EMG signal is used as an input operation associated with the setting.
   Head-mounted display device.
6. In the head-mounted display device according to claim 1,
   Further, an electrode for acquiring an electro-oculography signal from the user's facial muscle is provided.
   The electro-oculography signal generated in response to the user's conscious operation of movement near the eye is detected.
   The electro-oculography signal is used as an input operation associated with the setting.
   Head-mounted display device.
7. In the head-mounted display device according to claim 2,
   The output content as a test is output, and the EMG signal at this time is acquired.
   A subjective evaluation value regarding the psychological state of the user with respect to the output content as the test is input based on the operation of the user.
   By learning the correspondence between the output content as the test, the EMG signal, and the psychological state corresponding to the subjective evaluation value,
   Controlling the adjustment of the output content based on the correspondence.
   Head-mounted display device.
8. In the head-mounted display device according to claim 2,
   A subjective evaluation value regarding the psychological state of the user with respect to the output content when using the OS or the application is input based on the operation of the user.
   By learning the correspondence between the output content, the EMG signal, and the psychological state corresponding to the subjective evaluation value,
   Controlling the adjustment of the output content based on the correspondence.
   Head-mounted display device.
9. In the head-mounted display device according to claim 1,
   As the device, there is a first device equipped with a first electrode arranged so as to be in contact with a portion of the facial muscle near the cheekbone of the user.
   The first EMG signal acquired from the first electrode of the first device is used as input information.
   Head-mounted display device.
10. In the head-mounted display device according to claim 9,
    As the device, there is a second device equipped with a second electrode arranged so as to be in contact with a portion of the facial muscle near the user's eyebrows.
    The second EMG signal acquired from the second electrode of the second device is used as input information.
    Head-mounted display device.
11. In the head-mounted display device according to claim 10,
    Based on the combination of the first EMG signal and the second EMG signal, the output content is adjusted according to the inferred psychological state.
    Head-mounted display device.
12. In the head-mounted display device according to claim 9,
    The first device has a shape that extends downward and forward from the temple portion of the housing of the head-mounted display device.
    Head-mounted display device.
13. In the head-mounted display device according to claim 1,
    The first device has a shape so as to extend downward from a binocular portion in which a display surface of a housing of the head-mounted display device is arranged.
    Head-mounted display device.

Images