JP6396351B2 - Psychosomatic state estimation device, psychosomatic state estimation method, and eyewear - Google Patents

Description

  Embodiments described herein relate generally to a psychosomatic state estimation apparatus, a psychosomatic state estimation method, and eyewear to which an electrooculogram sensing technique is applied.

  Regarding electrooculogram sensing technology, there is a known example 1 (see Patent Document 1). In this known example 1, an EOG (Electro-Oculography) electrode is used for electrooculogram sensing, and a user's eye movement is detected from electroocular data acquired from the EOG electrode (paragraph 0023). In the embodiment of the known example 1, the EOG electrode is provided in the goggles (paragraph 0061), but there is also the known example 2 in which the electrode for EOG is provided in the nose pad portion of the glasses (see Patent Document 2, paragraph 0013). Goggles and glasses are a type of eyewear that can be easily used by ordinary people.

JP 2009-288529 A JP 2013-244370 A

  EOG has good compatibility with eyewear, and the application of eyewear with EOG has been studied in various ways. However, the range of application has not been exhausted, and new possibilities remain for the application of eyewear with EOG.

  For example, in order to observe the audience reaction of a screened work in a movie theater, a large number of people must be observed in the dark without interfering with movie viewing. However, such observation is difficult to carry out, so in reality it becomes an exit survey after watching a movie. In this case, for example, a story with a screening time of 2 hours is recollected to each audience in tens of seconds. However, since a considerable amount of time has passed since the audience was impressed by multiple scenes and recalled their impressions, the results of the exit survey for the screening work were limited to a large frame and the amount of targeted information Will be less. The contents of the survey results that we want here are, for example, what kind of emotions were induced by the audience in various scenes of the screening work, whether the audience concentrated, or whether the audience was fatigued or stressed, The content is useful for estimating the state of mind and body, such as having fallen asleep. (This psychosomatic state is related to how people responded or evaluated to external stimuli.) However, it is difficult to obtain such survey contents through exit surveys.

  For example, it has not been known how to use eyewear with EOG for the above-mentioned problems (it is difficult to estimate the psychosomatic state of people who can change depending on the situation).

  One of the problems to be solved by the embodiment of the present invention is to make it possible to estimate a person's mind and body state using eyewear with EOG.

  The psychosomatic state estimation apparatus according to one embodiment is used in the form of an eyewear device. Based on the electro-oculogram (EOG in FIG. 4) of the user wearing the eyewear, characteristic eye movements (such as blinking, eye meditation, up / down / left / right movement or rotation of the eyeball) are detected. Then, characteristic eye movements and external stimuli when the eye movements are detected (for example, stimulations of the human senses in various scenes of movies and videos) are recorded in association with time codes. And to be able to compare the two (characteristic eye movements and external stimuli).

  If the correspondence between the characteristic eye movement pattern and the user's mental and physical condition (emotional expression such as crying, laughing, and anger, motivation, mental concentration, fatigue, stress, sleep, etc.) is examined in advance, specific external stimuli This makes it possible to estimate what kind of mental and physical state the user has entered.

The figure explaining the eyeglasses type eyewear with which the information processing part 11 for the mind and body state estimation apparatus which concerns on one Embodiment was integrated (example of AR glasses by which the EOG electrode is arrange | positioned at the nose pad). The figure explaining the example of mounting of the EOG electrode in the glasses-type eyewear which concerns on one embodiment. The figure explaining the relationship between the information processing part 11 which can be attached to various embodiment, and its peripheral device. Various eye movements of the user wearing eyewear (blink, eye meditation, eye movement, eyeball rotation, etc.) and detection signal levels (Ch0, Ch1) obtained from the three analog / digital converters (ADC) shown in FIG. , Ch2), an electrooculogram (EOG) illustrating the relationship. The flowchart explaining an example of the mind-and-body state estimation method which concerns on 1st Embodiment. The flowchart explaining an example of the mind-and-body state estimation method which concerns on 2nd Embodiment. The flowchart explaining an example of the mind-and-body state estimation method which concerns on 3rd Embodiment. The flowchart explaining an example of the mind-and-body state estimation method which concerns on 4th Embodiment. The figure explaining the example of mounting of the EOG electrode in other embodiments. The figure explaining the mounting example of the EOG electrode in the glasses-type eyewear which concerns on other embodiment (The example of AR glasses by which the EOG electrode is arrange | positioned around the eyeball). FIG. 6 is a diagram for explaining an example of mounting an EOG electrode in goggle-type eyewear according to another embodiment (a type in which the eye frames of both left and right eyes are continuous) (another example of AR glasses in which EOG electrodes are arranged around the eyeball). . FIG. 6 is a view for explaining an example of mounting an EOG electrode in goggle-type eyewear according to another embodiment (a type in which the eye cups of the left and right eyes are separated) (Still another example of AR glasses in which the EOG electrode is arranged around the eyeball) ).

  In the following, basic information will be provided first, and then various embodiments will be described with reference to the drawings.

<Basic information>
The diameter of an adult eyeball is about 25 mm. After birth, it is about 17 mm and grows with growth.
The distance between the pupils of an adult male is about 65 mm. (Many commercially available stereo cameras are made at intervals of 65 mm.)
The distance between the pupils of adult women is several mm shorter than that of men.
The electrooculogram is several tens of mV.
The eyeball has a positive potential on the cornea side and a negative potential on the retina side. When this is measured on the surface of the skin, it appears as a potential difference of several hundred μV.

Examples of types of eye movements and eye movement ranges related to eye movement detection include the following:
<Types of eye movement (eye movement)>
(01) Compensatory eye movement Involuntary eye movement developed to stabilize the image of the outside world on the retina regardless of the movement of the head or body.

(02) Voluntary eye movement An eye movement developed to bring the visual image to the center of the retina and can be controlled arbitrarily.

(03) Impact eye movement (saccade)
Eye movement that occurs when the point of sight is changed to see an object (easy to detect).

(04) Sliding eye movement Smooth eye movement that occurs when tracking slowly moving objects (difficult to detect).

<Eyeball movement range (for general adults)>
(11) Horizontal direction Left direction: 50 ° or less Right direction: 50 ° or less (12) Vertical direction Down direction: 50 ° or less Up direction: 30 ° or less (The vertical angle range that can be moved by one's intention is only upward. Narrow (Because there is a “bell phenomenon” in which the eyeball moves upward when the eye is closed, the eye movement range in the vertical direction shifts upward when the eye is closed.)
(13) Others Angle of convergence: 20 ° or less.

<Targets to be estimated from blinks> About the estimation of the degree of concentration, the blink occurrence timing on the time axis with respect to the story is examined from simple blink frequency changes.

・ In tension / stress estimation, examine changes in blink frequency.

・ For sleepiness estimation, examine changes in blink cluster, blink time / blink speed.

・ Estimation of emotions, physical condition, and other psychosomatic states is often left to future research, but in this embodiment, there is a technical idea to estimate emotions, physical condition, and other psychosomatic states. doing.

  FIG. 1 is a diagram for explaining a glasses-type eyewear 100 in which an information processing unit 11 for a mind and body state estimation apparatus according to an embodiment is incorporated (an example in which an EOG electrode is arranged on a nose pad). In this embodiment, a right eye frame (right rim) 101 and a left eye frame (left rim) 102 are connected to a bridge 103. The left and right eye frames 102 and 101 and the bridge 103 can be made of, for example, an aluminum alloy or titanium. The left outer side of the left eye frame 102 is connected to the left temple bar 106 via the left hinge 104, and a left modern (left ear pad) 108 is provided at the tip of the left temple bar 106. Similarly, the right outer side of the right eye frame 101 is connected to the right temple bar 107 via the right hinge 105, and a right modern (right ear pad) 109 is provided at the tip of the right temple bar 107.

  In the right temple bar 107 (or the left temple bar 106), an information processing unit 11 (several millimeter square integrated circuit) is attached. This information processing unit 11 is a communication processing unit using a microcomputer having a processing capability of several tens of MIPS or more, a flash memory (EEPROM) having a free area of several tens of MB or more, BlueTooth (registered trademark), USB, etc. The above ADC (Analog-to-Digital Converter) and the like can be configured by an LSI (details of the information processing unit 11 will be described later with reference to FIG. 3).

  A small battery (BAT) such as a lithium ion battery is embedded in, for example, the left temple bar 106 (or the right temple bar 107 or the modern 108 or 109), and serves as a power source necessary for the operation of the eyeglass-type eyewear 100. ing.

  A left camera 13L is attached to the end of the left eye frame 102 near the left hinge 104, and a right camera 13R is attached to the end of the right eye frame 101 near the right hinge 105. These cameras can be configured using ultra-compact CCD image sensors.

  These cameras (13L, 13R) may constitute a stereo camera. Alternatively, an infrared camera (13R) and a laser (13L) may be arranged at the positions of these cameras to form a distance sensor using an infrared camera + laser. This distance sensor can also be composed of a small semiconductor microphone (13R) that collects ultrasonic waves and a small piezoelectric speaker (13L) that emits ultrasonic waves.

  By the distance sensor, the distance between the visual object at the tip of the user's line of sight and the left and right frames 101/102 is known. Although depending on the design of the eyeglass frame, the user's eyeball position is shifted from the left and right frames 101/102 to a position deeper to the back of the head about 40 to 50 mm, and the distance between the pupils of an adult is about 65 mm. From this 65 mm interpupillary distance and the distance corrected by the distance from the frame to the eyeball position measured by the distance sensor, the convergence angle (more eye angle) when the user is looking at the object to be viewed, It can be calculated. When the distance from the frame to the visual object is much larger than the shift amount from the frame to the eyeball position, the convergence angle can be roughly calculated even if this shift amount is ignored.

In addition, instead of the left and right cameras 13L / 13R, or in addition to the left and right cameras 13L / 13R, an embodiment in which a central camera (not shown) is provided in the bridge 103 can be considered. Conversely, there may be embodiments in which no camera is provided. (These cameras are shown as the camera 13 in FIG. 3. These camera and external line camera + laser distance sensor can be treated as an option, and provided as appropriate according to the use of the eyewear 100. Good.)
A left display 12L is fitted in the left eye frame 102, and a right display 12R is fitted in the right eye frame 101. This display is provided on at least one of the left and right eye frames, and can be configured using a film liquid crystal or the like. Specifically, one or both of the left and right displays 12L and 12R can be configured using a film liquid crystal display device that employs a polymer-dispersed liquid crystal (PDLC) that does not use a polarizing plate. (This display is shown as display 12 in FIG. 3.)
The transparent left and right displays 12L / 12R using film liquid crystal can be used as means for providing augmented reality (AR) that adds image information such as numbers and letters to the real world seen through glasses.

  The film liquid crystal of the left and right displays 12L / 12R can also be used as a liquid crystal shutter for separating the left and right images when watching a 3D movie (or 3D video). In configuring 3D glasses, in addition to the liquid crystal shutter system, a polarization system can be used.

  When the distance between the user's eyeball position and the visual object is known in advance, the above-described convergence angle can be calculated without a distance sensor. For example, the vergence angle of a spectator (65mm pupil distance) sitting in a seat 10m away from the screen (located at the seat number) in the center of a movie theater seat is about tan-1 [65/10000]. Become. When displaying 3D subtitles in AR on the left and right displays 12L / 12R, the convergence angle when viewing the AR display is set to be the same as the convergence angle when viewing the 3D image projected on the screen 10 meters ahead. It is possible to control the left and right display positions of 3D subtitles for AR display. This can reduce eyestrain when continuing to view 3D subtitles displayed in AR and a 3D screen image 10 meters ahead.

  Generally speaking, not only for viewing 3D movies, if the displays are on the left and right, the images (IM1 and IM2 in FIG. 1) displayed on the left and right displays correspond to the desired convergence angle. It can be shifted in the opposite direction. As a result, it is possible to reduce the burden on the eyes when alternately viewing the object that can be seen in the real world and the AR display.

  A nose pad portion is provided between the left and right eye frames 102 and 101 and below the bridge 103. This nose pad portion is composed of a pair of a left nose pad 150L and a right nose pad 150R. The right nose pad 150R is provided with right nose pad electrodes 151a and 151b, and the left nose pad 150L is provided with left nose pad electrodes 152a and 152b.

These electrodes 151a, 151b, 152a, and 152b are electrically separated from each other, and are connected to three AD converters (ADCs 1510, 1520, and 1512) through insulated wiring members (not shown). The outputs from these ADCs have different signal waveforms according to the movement of the eye of the user wearing the eyewear 100, and are supplied to the information processing unit 11 as digital data according to the eye movement of the user. The electrodes 151a, 151b, 152a, and 152b are used as line-of-sight detection sensors and are components of the eye movement detection unit 15 in FIG. 3 together with the three ADCs. The three ADCs can be incorporated into one LSI together with the information processing unit 11. The number of ADCs to be used can be appropriately increased or decreased depending on the purpose, and the number of three is an example (for example, when the purpose is only to detect an EOG waveform due to a blink on the right side, the number of ADCs to be used is equal to that of the ADC 1510. Depending on the purpose, the number of ADCs may be one or two, or four or more may be used.)
The right nose pad 150R is provided with a right microphone (or right vibration pickup) 16R, and the left nose pad 150L is provided with a left microphone (or left vibration pickup) 16L. The left and right microphones 16L / 16R can be composed of, for example, ultra-small electret condenser microphones. Further, the left and right vibration pickups 16L / 16R can be constituted by, for example, piezoelectric elements. Sounds and movements induced by large emotional expressions such as laughter, screams, angers, and reflexive body movements when surprised can be detected using a microphone (or vibration pickup) 16L / 16R (these microphones or vibrations). The pickup is shown as a microphone (or vibration pickup) 16 in FIG. 3). Small sobbing with a nose can be detected by the left and right microphones 16L / 16R. It should be noted that the user's body movement can also be detected by an acceleration sensor or gyro built in the sensor unit 11e of FIG.

  The eyewear 100 is fixed to a user's head (not shown) by left and right nose pads (150L, 150R), left and right temple bars (106, 107), and left and right moderns (108, 109). In this embodiment, only the left and right nose pads (150L, 150R), the left and right temple bars (106, 107), and the left and right modern (108, 109) need to touch the user's head (or face) directly. However, there may be an embodiment in which a part other than the ADC (1510, 1520, 1512) and the user's body touches the user other than those (nose pad, temple bar, modern).

  On the film liquid crystal of the right display 12R in FIG. 1, for example, a right display image IM1 including Japanese characters (Hiragana Katakana Kanji), numerals, alphabets, marks of a predetermined shape, and other icon groups can be displayed. The left display image IM2 having the same contents as IM1 can also be displayed on the film liquid crystal of the left display 12L.

  The display content of the displays 12L and 12R may be anything. For example, the bright spot used for glaucoma visual field inspection can be displayed on the display 12L or 12R. When the display is not performed, the display state can be set to black display, and the film liquid crystal can be used as a blind or a light shielding sheet. The numeric keys and alphabets displayed on the right display 12R (or the left display 12L) can be used when inputting numbers and characters. Character strings, marks, icons, etc. displayed on the right display 12R (or the left display 12L) can be used when searching for specific information items, selecting / determining target items, and alerting the user. .

  The display images IM1 and IM2 can be used as augmented reality (AR) display means for adding information such as numbers, letters and marks to the real world seen through the glasses, and the AR display can be turned on and off as appropriate. The content of the display image IM1 and the display image IM2 may be the same content (IM1 = IM2) or different content (IM1 ≠ IM2) depending on the embodiment. The display image IM1 (or IM2) can be displayed on the right display 12R and / or the left display 12. When the content of the AR display is a 3D image that overlaps the real world (with depth) that can be seen through the glasses (with depth), IM1 and IM2 can be used as separate left and right 3D images.

  Further, when the displays (12R, 12L) are present on the left and right, for example, by adjusting the convergence angle, the images of the left and right display images (IM1, IM2) can be shifted in the opposite directions on the left and right. As a result, it can be considered that the burden on the eyes when the object and the AR display that are visible in the real world are alternately viewed is reduced. However, normally, images with the same contents are displayed on the left and right displays (12R, 12L).

  Display control on the displays 12L and 12R can be performed by the information processing unit 11 embedded in the right temple bar 107. (Technology for displaying characters, marks, icons, etc. on the display is well known.) The power supply necessary for the information control unit 11 and other operations can be obtained from the battery BAT embedded in the left temple bar 106.

  Note that there is a possibility that the designer or designer wears a prototype of the eyewear 100 corresponding to the embodiment and feels that the weight balance is bad. If the main cause is the BAT in the left temple bar 106, a “weight (or another same weight battery)” corresponding to the BAT in the left temple bar 106 can be placed in the right temple bar 107.

  FIG. 2 is a diagram for explaining an example of mounting the EOG electrode in the eyeglass-type eyewear 100 according to the embodiment. Right nose pad electrodes 151a and 151b are provided above and below the right nose pad 150R, and left nose pad electrodes 152a and 152b are provided above and below the left nose pad 150L. The outputs of the right nose pad electrodes 151a and 151b are supplied to the ADC 1510, the outputs of the left nose pad electrodes 152a and 152b are supplied to the ADC 1520, and the outputs of the lower electrodes 151b and 152b (or the upper electrodes 151a and 152a) of the left and right nose pads. Is provided to the ADC 1512.

From the ADC 1510, a Ch1 signal that changes in response to the right / left eye movement of the user is obtained. From the ADC 1520, a Ch2 signal that changes in accordance with the left and right eye movements of the user is obtained. The ADC 1512 obtains a Ch0 signal that changes in accordance with the left and right eye movements of the user. The vertical movement of both the left and right eyes may be evaluated by averaging the outputs of the ADC 1510 and the ADC 1520. (The relationship between the signal waveforms of Ch0, Ch1, and Ch2 and eye movement will be described later with reference to FIG. 4.)
<Supplementary explanation about the electrooculogram sensor shown in FIGS. 1 and 2>
* EOG electrode provided on the nose pad At least one pair of measurement electrodes (a pair of 151a and 151b; or a pair of 152a and 152b) is provided for detection of eye movement in the vertical direction. In that case, electrodes for measuring electrooculogram are provided above and below the left or right (either one) nose pads (the number of electrode terminals is two).

  A set of measurement electrodes is added to improve the detection accuracy of eye movement in the vertical direction. In that case, electrodes for electrooculogram measurement (151a and 151b; and 152a and 152b) are provided above and below the right and left (both) nose pads 150L / 150R (the number of electrode terminals is four).

* ADC (Analog-to-Digital Converter) At least one ADC (for Ch1 or Ch2) is provided for detecting eye movement in the vertical direction.

One ADC (for Ch2 or Ch1) is added to improve the vertical eye movement detection accuracy (as a result, one ADC is provided for each of Ch1 and Ch2). (Option for improving accuracy)
One more ADC (for Ch0) is added for detection of eye movement in the left-right direction. (Function improvement option)
FIG. 3 is a diagram illustrating the relationship between the information processing unit 11 that can be attached to various embodiments and the peripheral devices. In the example of FIG. 3, the information processing unit 11 includes a processor 11a, a nonvolatile memory (flash memory / EEPROM) 11b, a main memory 11c, a communication processing unit 11d, a sensor unit 11e, a timer (frame counter) 11f, a device ID 11g, and the like. It is configured. The processor 11a can be constituted by a microcomputer having a processing capability according to product specifications. Various programs executed by the microcomputer and various parameters used when the programs are executed can be stored in the nonvolatile memory 11b. The nonvolatile memory 11b can store various detection data together with a time stamp (and / or a frame number counted by a frame counter) using the timer 11f. The main memory 11c provides a work area for executing the program.

  The sensor unit 11e includes a sensor group for detecting the position and / or orientation of the eyewear 100 (or the head of the user wearing the eyewear). Specific examples of these sensor groups include an acceleration sensor that detects movement in three axial directions (xyz directions), a gyro that detects rotation in three axial directions, and a geomagnetic sensor that detects absolute orientation (compass function). There are beacon sensors that receive radio waves, infrared rays, and the like to obtain position information and the like. IBeacon (registered trademark) or Bluetooth (registered trademark) 4.0 can be used to acquire the position information and others.

  LSIs that can be used for the information processing unit 11 have been commercialized. One example is the “TZ1000 Series for Wearable Terminals” by Toshiba Semiconductor & Storage. In this series, the product name “TZ1011MBG” includes CPU (11a, 11c), flash memory (11b), Bluetooth Low Energy (registered trademark) (11d), sensor group (acceleration sensor, gyroscope, geomagnetic sensor) (11e). , 24-bit delta-sigma ADC, I / O (USB etc.).

  What to do with the processor 11a can be commanded from the local server (or personal computer) 1000 via the communication processing unit 11d. The communication processing unit 11d can use existing communication methods such as ZigBee (registered trademark), Bluetooth (registered trademark), and Wi-Fi (registered trademark). The processing result in the processor 11a can be sent to the local server 1000 in real time via the communication processing unit 11d. In addition, information stored in the memory 11b can be provided to the local server 1000 via the communication processing unit 11d. The local server 1000 can record and store information sent from the information processing unit 11 of one or more eyewear 100 in the recorder / local database 1002. Information recorded / stored in the recorder / local database 1002 can be sent to the main server 10000 via a network such as the Internet. Various information is sent from the one or more local servers 1000 to the main server 10000, and a big data database can be constructed by integrating the information.

  A display 12 (12L and 12R), a camera 13 (13L and 13R), an eye movement detection unit 15, a microphone (or vibration pickup) 16, and the like are connected to the system bus of the information processing unit 11. Each device (11 to 16) in FIG. 3 is powered by the battery BAT.

  The eye movement detection unit 15 in FIG. 3 includes four eye movement detection electrodes (151a, 151b, 152a, and 152b) that constitute a line-of-sight detection sensor, and three ADCs that extract digital signals corresponding to the eye movements from these electrodes ( 1510, 1520, 1512) and a circuit for outputting the output data from these ADCs to the processor 11a side. The processor 11a interprets a command or state corresponding to the type of eye movement from various eye movements (vertical movement, left and right movement, blinking, eye meditation, etc.) of the user, and performs processing corresponding to the instruction or state. Can be executed.

  As a specific example of a command corresponding to the type of eye movement, if the eye movement is, for example, eye meditation, select an information item at the tip of the line of sight (similar to one click of a computer mouse), and a blink of a predetermined number of consecutive eyes If it is (or blink / wink for one eye), there is a command for starting the processing for the selected information item (similar to double clicking with a computer mouse).

  Specific examples of the state corresponding to the type of eye movement include a predetermined number of blinks, eye meditation for a predetermined time, and vertical / horizontal movement of the line of sight. For example, the eye movement is one or two blinks in a lecture section. If so, it can be estimated that the user concentrated on the contents of the lecture. It can be estimated that the user has fallen asleep during the reproduction of the examination video if the eye movement is an eye meditation of several seconds or more during the reproduction of the examination video. If the eye movement is up / down / left / right movement during sleep, it can be estimated that the user was in REM sleep.

  The command or state corresponding to the type of eye movement is an example of information input B using the eye movement detection unit 15.

<Supplementary explanation about the components shown in FIG. 3>
* About the operation of the microcomputer unit constituting the processor 11a By configuring the detection unit such as blinks by programming the microcomputer unit, and operating this detection engine, not the real-time ADC raw data but the time stamped Blink events can be recorded. (For example, if there is a blinking event that occurs once or twice consecutively 5 minutes and 10 seconds after starting the timer, instead of recording raw data obtained by AD conversion of the EOG waveform of the blink, Instead, the occurrence of such a blink is recorded with a time stamp of 5 minutes and 10 seconds.) In this way, the memory capacity required for recording can be greatly saved compared to recording raw data as it is. Note that when the memory capacity has a sufficient margin, the ADC raw data may be recorded.

* Flash memory / EEPROM composing the non-volatile memory 11b For example, consider the case where blink information is collected from a large number of spectators watching a movie in a movie theater. In that case, rather than continuously collecting blink information in real time from all the audiences, the blink information of each audience is temporarily stored in the local memory (11b) of each audience, and after the movie ends, It is possible to collect information more efficiently by collecting the blink information stored from the local memory (11b).

* Communication processing unit 11d For example, if the movie ends and the blink information is temporarily stored in the local memory (11b) of all the audiences, the temporarily stored blink information (with a time stamp) is stored in the local movie theater. It can be collected by the server (1000). This collection can be performed via, for example, BlueTooth (wireless) or USB (wired) provided at a seat in a movie theater or at a specific location in the hall.

* Power supply (battery / battery) BAT A small battery is used to maintain power supply for the following operations.

  ・ During movie screening, the “Blink” detection engine is operated to detect “Blink” in real time from the AD conversion result of the EOG waveform, and the information of the detected “Blink” event together with the time stamp The data is recorded in the memory 11b.

  After the movie has ended, the “blink” event information recorded in the memory 11b can be collected.

  FIG. 4 shows various eye movements of the user wearing the eyewear 100 (blink, eye meditation, eyeball rotation, etc .; especially blink) and detection obtained from the three analog / digital converters (ADC) shown in FIG. It is an electrooculogram (EOG) which illustrates the relationship with signal level (Ch0, Ch1, Ch2). The vertical axis represents the sample value of each ADC (corresponding to the EOG signal level), and the horizontal axis represents time.

  In the ADC sample value of the Ch0 EOG waveform shown in the upper part of FIG. 4, the combination waveform Pr (or Pr *) of the concave waveform when the line of sight is directed to the left and the convex waveform when the line of sight is directed to the right is used. Eyeball rotation is detected.

  In the ADC sample value of the Ch1 EOG waveform shown in the middle part of FIG. 4 or the ADC sample value of the Ch2 EOG waveform shown in the lower part of the figure, the convex waveform when the line of sight faces upward and the line of sight when the line of sight faces downward The eyeball rotation in the vertical direction is detected by the combination waveform Pt (or Pt *) of the concave waveform.

  In the ADC sample value of the Ch1 EOG waveform shown in the middle part of FIG. 4 or the ADC sample value of the Ch2 EOG waveform shown in the lower part of FIG. It is detected by the convex pulse waveform Pc.

In the ADC sample value of the Ch1 EOG waveform shown in the middle part of FIG. 4 or the ADC sample value of the Ch2 EOG waveform shown in the lower part of FIG. It is detected by the waveform Ps. (Although not shown, four or more blinks can be detected as in the case of three.)
In the ADC sample value of the Ch1 EOG waveform shown in the middle part of FIG. 4 or the ADC sample value of the Ch2 EOG waveform shown in the lower part of the figure, eye meditation that occurs for a longer period than the blink is detected by the wide convex pulse waveform Pf. The (Although not shown, longer eye meditation is detected with a wider pulse width waveform.)
<Relationship between EOG Waveform and Eye Movement in FIG. 4>
(01) When the line of sight is moving from side to side, the waveform Pr (or Pr *) appears. When the line of sight is moving up and down, the waveform Pt (or Pt *) appears. When the line of sight is moving left and right and up and down, the waveform Pr (or Pr *) and the waveform Pt (or Pt *) appear continuously. In other words, the “shrinking” type eye movement can be detected from the EOG waveform Pr and / or Pt (Pr * and / or Pt *).

  (02) When blinking is performed, pulse waveforms Pc, Ps (or Pc *, Ps *) corresponding to the number of blinks appear. In other words, various “blink” type eye movements can be detected from the EOG waveforms Pc, Ps (or Pc *, Ps *). Note that the ADC of Ch0 is not necessary if it is sufficient to detect “blink” type eye movement. For the purpose of detecting “blink” type eye movement, at least one ADC of Ch1 or Ch2 is sufficient. However, the signal-to-noise ratio (S / N) can be improved by using two ADCs Ch1 and Ch2 and combining the outputs of both. The reason is that the signal components (Pc, Ps, Pf, Pt, etc.) correlated between Ch1 and Ch2 increase by +6 dB by combining the two ADC outputs, but the random noise component without correlation increases only by +3 dB. Because there is no.

  (03) When eye meditation is performed, a pulse waveform Pf (or Pf *) corresponding to the length of eye meditation appears. That is, “eye meditation” type eye movements of various lengths can be detected from the EOG waveform Pf (or Pf *). Although “eye meditation” type eye movement appears in the EOG waveform of Ch0 a little, for the purpose of detecting “eye meditation” type eye movement, the ADC of Ch 0 is not necessary.

<Relationship between eye movement and psychosomatic state>
(11) If a “shocking” type of eye movement is detected during an examination using EOG (for example, during a visual field examination of glaucoma), it is assumed that the subject's concentration is lacking and rests. Can be considered.

  (12) During examination using EOG (for example, during a lecture or class comprehension survey), a “blink” type that is a small number of times (for example, 1 to 2 times) at key points (separation points of the meaning of the lecture content, etc.) When eye movement is detected, it can be estimated that the subject is concentrated and understanding is advanced.

  (13) During inspection using EOG (for example, during aptitude inspection of customer service staff), the number of important points (such as the location of the inspection contents such as “the subject receives various orders quickly from multiple customers in a crowded store”) When “blink” type eye movements are detected more than once (for example, three times or more), it may be presumed that the subject is tense and mentally stressed.

  (14) When a short “eye meditation” type of eye movement is detected multiple times during an examination using EOG (for example, in the examination using a drive simulator, while outputting a monotonous running screen and monotonous continuous vibration sound) It can be estimated that the subject is tired and cannot concentrate on the examination items.

  (15) If a long-term “eye meditation” type eye movement is detected during an examination using EOG (eg during sleep investigation), the subject is sleeping (nap) and is in non-REM sleep or REM sleep state It can be estimated that

  (16) In a test using EOG (for example, a sleep state test), when a “shrinking” type of eye movement is detected while a long-time “eye meditation” type of eye movement is detected, It can be estimated that the subject is in a state of REM sleep.

  (17) In a test using EOG (for example, a test for awakening from sleep), after long-term "eye meditation" type eye movement is detected (it is estimated that the patient is in a state of non-REM sleep or REM sleep). When a “blink” type eye movement is detected, it may be estimated that the user has awakened from sleep by detecting the “blink” type eye movement.

<Emotional (mental and physical state) estimation / recording points for theater 3D glasses using electrooculography sensing technology>
* The main detection target is blinks.

  A pair of EOG electrodes (a minimum of one ADC to be used) is attached to the user's face at least in the vertical direction (for example, up and down on one side of the left and right nose pads).

  Based on the detection results from one or more sets of EOG electrodes attached to the user's face, the user's stress, concentration, sleepiness, sleep state (REM sleep / non-REM sleep), etc. can be estimated.

* Add the following to collect screen reactions and audience reactions to 4D theater gimmicks. In other words, a minimum of one set (two sets to increase accuracy) of EOG electrodes (two or more ADCs in total) are additionally mounted in the horizontal direction on the face of the user (individual audience). (For example, a total of three or four EOG electrodes are provided above and below one side of the left and right nose pads and above or below the other side, or above and below each of the left and right nose pads.)
With the above additional configuration, it is possible to estimate the rotation direction of the eyeball including the left-right direction (gaze movement or eyeball rotation in an arbitrary direction).

  FIG. 5 is a flowchart for explaining an example of the psychosomatic state estimation method according to the first embodiment. Although 3D video playback has not been widespread in ordinary households, 3D screening in movie theaters is ongoing. Figure 5 summarizes and records the concentration, fatigue, and stress of the audience in a 3D movie using 3D glasses (or a 4D movie that adds effects on the five senses other than audiovisual such as vibration to a 3D movie using 3D glasses). An example of processing to be performed is shown. Further, FIG. 5 shows the general evaluation of a large number of spectators by aggregating unspecified majority spectator data on works screened over a certain period (not only 3D / 4D works but also 2D works). It also illustrates the process of feeding back to the next production.

  Currently, there is a 3D / 4D movie theater equipped with a local server 1000 as shown in FIG. 3 in a projection room (or office room) (not shown), and a 3D (or 4D) movie is shown in this movie theater. Let's assume the case. Before the screening of the 3D work begins, a staff member of the cinema distributes 3D glasses 100 having an eyewear form as shown in FIG. 1 to the visitors (audience) (ST10).

  The 3D glasses 100 to be distributed are color-coded for men and women, and sizes are divided for adults and children. In other words, there are a total of four types of 3D glasses 100, two types for adult men and women and two types for children men and women. These four types can be classified by, for example, the first bit of the device ID 11g in FIG. 4 types with 2 bits and 16 types with 4 bits are possible). In addition, the individual identification of the 3D glasses 100 (identification for the number of people to be distributed) can be classified by the subsequent bits of the device ID 11g (12 bits for 4096 people and 14 bits for 16384 people). The number of bits of the device ID 11g may be appropriately determined according to the number of classification categories of the glasses 100 to be distributed and the maximum number of glasses 100 to be distributed.

  Note that the glasses 100 as shown in FIG. 1 can perform AR display so that captions can be turned on / off in the images IM1 and / or IM2. At that time, if the device ID indicates for adults, the default displayed subtitle can be a small “Kana-Kanji mixed Japanese or original language (English)”, and if the device ID indicates for children, the default is displayed. The displayed subtitle can be the larger “Japanese only in Hiragana or Katakana”.

  After the audience wearing the glasses 100 sits in the seats of the movie theater, when the screening start time comes, the lighting in the hall is turned off and the screening starts. Simultaneously with the start of the screening, a signal to start the screening is sent from the local server 1000 of FIG. When this signal is received by the communication processing unit 11d, the timer (or frame counter) 11f of each glasses 100 starts automatically (that is, even if the glasses user does nothing). Then, every time an event corresponding to eye movements (such as blinking) of the eyeglass user (individual audience) is detected by the eye movement detection unit 15, or the voice or body movement of the eyeglass user (laughing voice or reflection when surprised) Each time an event corresponding to the movement) is detected by the microphone 16 or the sensor unit 15, detection data obtained by adding an information bit of a time stamp (or a frame count value) to an information bit indicating the event is detected by each eyeglass user. Are continuously recorded in the memory 11b (ST12). Note that when the storage capacity of the memory 11b is sufficiently large, the EOG-converted EOG raw data may be recorded in the memory 11b as EOG data of each glasses user.

  The detection data includes information bits indicating the type of detected event (type of eye movement, type of voice, type of body movement, etc.) and information on a time stamp (or frame count value) in the information bits of the device ID 11g. A bit can be added. Although it depends on the application, the number of bits of detection data is considered to be sufficiently practical if it is about 80 to 160 bits (10 to 20 bytes).

  The display format of the video that causes the occurrence of the event is not limited to 3D, and may be 2D or 4D (4D is equivalent to 3D in terms of only the video portion). In addition, the sound / sound effect that can cause an event (in 4D, vibration and other bodily effects are also added) may be monaural, front 2-3 channel stereo, or front / rear surround stereo of 4 channels or more. These sound / sound effects may be appropriately switched depending on the movie scene. The sound / sound effect can be specified from the time stamp (or frame count value) obtained from the timer (or frame counter) 11f.

  The frame count value indicates one video frame. For example, when one or several subliminal images are inserted in a continuous frame of a moving image, the subliminal image frame can be specified by the frame count value. . For example, if the frame rate is 60 frames per second, the time stamp corresponding to the frame count value can be calculated by dividing the frame count value by 60. Therefore, the frame counter can have the same function as the timer.

  Data recorded in the memory 11b is automatically transferred from the communication processing unit 11d to the local server 1000 using Bluetooth or the like at a predetermined timing (for example, every 5 to 10 minutes) including its own device ID 11g. (ST14). The local server 1000 continuously records the transferred data including the device ID 11g of the transmission source in the digital recorder 1002 and / or cumulatively records it in the local database 1002. Even if various glasses user data are recorded in the recording area of the recorder / local database 1002 in a discontinuous manner, the data of individual glasses users can be selected and collected by using the device ID 11g as a selection key. Can do.

  The data recording to the memory 11b (ST12) and the divided transfer of the recorded data to the local server 1000 (ST14) are repeated until the screening work ends (NO in ST16).

  When the screening work ends and the end telop starts (YES in ST16), the untransferred data in the memory 11b is transferred to the local server 1000. As a result, the data of all glasses (all user glasses) distributed to the audience are collected by the local server 1000, and the recording to the recorder / local database 1002 is completed (ST18). If the data amount becomes too large when data of all glasses users is collected, a fixed number of device IDs may be randomly determined, and only user data of the determined device ID may be collected (or, for example, an odd number) Alternatively, the amount of data can be halved by collecting only user data with even device IDs.

  When the necessary user data is collected and the recording is completed, the 3D glasses 100 distributed by the staff are collected from the user (audience) who leaves (ST20). (Or have the used 3D glasses be put in the collection box.) The communication function of the collected glasses 100 is immediately turned off. Then, since the communication circuit between the uncollected glasses 100 and the local server 1000 is alive, when the glasses 100 are taken out, a staff member at the exit of the theater determines the owner of the glasses 100 by the beacon sensor 11e or the like. It is possible to identify and prevent the glasses 100 from being lost.

  When the audience is switched and re-screened (YES in ST22), the processes of ST10 to ST20 are repeated to collect data from a new audience.

  When all the screenings on that day are completed (NO in ST22), the collected data is totaled and the records are organized (ST24). For example, based on the collected data, a characteristic EOG waveform change (for example, corresponding to an event corresponding to blinking) of one or more users (audience) and a time stamp (or frame number) when the change occurred Correspondences between the contents to be shown (4D effects such as scenes, sound effects, vibrations, etc.) and user characteristics (such as gender / age group) are summarized in a spreadsheet format. Then, for example, adult women often scream surprisedly with certain scenes and sound effects, but men rarely scream. For example, the degree of audience reaction (mind and body state) for each scene and sound effect is estimated. become able to.

  Data collected by the local server 1000 during a certain period (for example, one month) in one or more movie theaters can be continuously sent to the main server 10000 via a communication line such as the Internet (ST26).

  For example, if the detection data is composed of 10 bytes and 2 bytes are used for the device ID, 2 bytes can be assigned to the type of detection event, and 6 bytes can be assigned to the time stamp (or frame count value) or the like. . Assuming that an average of 1000 events per person is detected in one movie screening, 10 Mbytes of information is collected from 1,000 audiences in one movie screening. If there are 100 screenings in a month, the amount of detected data is 1 Gbyte per month per movie theater. When similar data from 100 movie theaters is collected in the main server 10000 in FIG. 3, it becomes a total of 100 Gbytes of data, and big data of 1 Tbyte or more per year.

  This big data gives hints for selecting the genre of the next movie work, considering the social situation of the data collection. For example, in a social situation where suicide bombings occur frequently, if it can be inferred from a set of EOG data that the audience is highly concentrated in the battle scene with terrorists in a 3D movie, a 3D movie of a hero that confronts terrorism Next, it will be possible for a film production company to make a production.

<Summary of First Embodiment (FIG. 5)>
Combined with 3D glasses for movie theater 3D screening (either shutter method or polarization method), concentration, fatigue, and stress estimation using electroocular sensing. This makes it possible to aggregate the audience's reactions (concentration / fatigue / stress for each scene) to the movie that is currently showing, re-editing the main part according to the screening time, and future new movies Can be fed back as a voice of the customer (VOC).

  By detecting, estimating, and recording (along the time axis) the audience response to the movie being screened, it is possible to collect the response results of all the people in the movie theater.

  FIG. 6 is a flowchart for explaining an example of the psychosomatic state estimation method according to the second embodiment. Here, the AR eyewear (AR glasses) 100 is provided with an electrooculogram sensing function so that blinks (blinks) can be detected, and a test video (training video) through the AR glasses (or on the AR glasses). The concentration, fatigue, and stress are estimated based on “blinks” and the like.

  First, a subject (a user of eyewear) wears an AR eyewear (glasses type or goggles type) 100 with an EOG electrode (ST30). At the same time as the screening of the test video (driver simulator video for driver aptitude test, etc.), the timer (or frame counter) 11f is started, and EOG data of the subject (long-distance bus driver, elderly car driver, etc.) Continuous recording (for example, event information corresponding to the EOG waveform in FIG. 4) is started using the memory 11b (ST32). The test video may be computer graphics (CG), a live-action video, or an animation. The video display method may be any of 2D, 3D, and 4D. The sound (including vibrations in 4D) may be stereo, monaural, or surround. The video display method and audio presentation method can be switched as appropriate according to the test contents.

  The test video incorporates video content (various stress video) related to the EOG signal waveform pattern for estimating the concentration level, fatigue level, stress, etc. of the subject at predetermined check points. For example, a video that incorporates a landscape in which the lane suddenly turns, a ball or an animal suddenly pops out from the side, or a monotonous sleepy landscape continues. EOG data (data corresponding to an event such as blinking) obtained from the eyewear 100 attached by the subject at such a predetermined check location is temporarily recorded in the memory 11b together with a time stamp or a frame number. The The content of the data recorded in the memory 11b is transferred to the local server 1000 at a predetermined timing (for example, a scene break in the test video) and recorded in the recorder / local database 1002 (ST34).

When the subject's data is recorded using test images including various stress images (YES in ST36), the data recorded in the recorder / local database 1002 is evaluated (ST38). In the evaluation of ST38, it is checked whether or not the EOG data obtained in this test is successfully taken. (Normally, if there is a suspicion such as no blink being recognized in a video part where there is a blink reaction, it is rechecked (ST40 YES), otherwise, the process proceeds to the next process.)
If the subject's EOG data seems to be well in this test (NO in ST40), the subject's characteristic EOG waveform change (corresponding to blinking, eye meditation, etc.) and the time when the change occurred Correspondence between test video contents (scene, sound effect, 4D effects such as vibration and temperature, etc.) indicated by the stamp (or frame number) and characteristics (gender / age / past history, etc.) of the subject (glasses user) The relation is totalized in the local server 1000 for the subject, and the total result is recorded in the recorder / local database 1002 (ST42).

  The recorder / local database 1002 stores past EOG data for the same test video when the same subject (long-distance bus driver, elderly car driver, etc.) is healthy, or a healthy average in the same industry. Comparative EOG data for the same test video of a person has already been recorded. The subject's EOG data obtained in this test is compared with the previously recorded comparative EOG data. The locations of both data to be compared are locations of the same time stamp (or the same frame count value). As a result of comparison, from the test data of this time, doze (such as Pf in FIG. 4) is observed in a monotonous running image, or there is no occurrence of blink reaction (Pc, Ps, etc. in FIG. 4) in an unexpected pop-up image location. When a spot is scattered, it is determined whether the subject (long-distance bus driver, elderly car driver, etc.) can normally perform future tasks (such as long-duration bus driving) according to the situation. (ST44).

  Data collected by the local server 1000 over a certain period (for example, one year) is continuously sent to the main server 10000 via the Internet or the like. Aggregated data continuously sent from the local server 1000 is accumulated in the database of the main server and becomes big data (ST46). By analyzing this big data, it is possible to estimate what kind of EOG data is normally obtained from the test video of ST32 for a healthy person. Knowledge based on such estimation is useful for determining the work aptitude of fatigued workers and elderly people.

  By creating test videos with appropriate content, the test in Fig. 6 can also be used to estimate concentration, fatigue, stress, etc. in work in industries other than automobile driving (such as infrastructure maintenance work and warehouse picking work). Is available. For example, during work / training, it is possible to estimate in real time that a concentration reduction, stress increase, or nerve abnormality has occurred. Phenomena in which the number of “blinks” increases may be psychogenic, such as anxiety, tension, mental conflict, and stress, and cases of abnormal nerves. In the work, any state is not preferable. By detecting / recording “blinks”, it is possible to examine and take measures to create an environment and improve work procedures to reduce the burden on workers.

  In addition, the understanding level can be estimated in real time from the blink (blink) occurrence pattern during training. When listening to explanations, it is well understood if there is blinking at the breaks of meaning, and if blinking at a place that is not a break, it can be assumed that the user cannot concentrate on listening to the explanation.

  By detecting / recording “blinks”, it is possible to evaluate the training effect and improve the training content.

  The dozing can be estimated in real time from the occurrence pattern of “blinks” and “eye meditation” during work.

  There is a clear difference in the absolute value of the electrooculogram in the vertical direction when the line of sight is directed straight up and when the eye is intentionally closed for a certain period of time (when the line of sight is directed straight up << A certain period of time, for example, 0.5 seconds or more of closed eyes).

  Since the eyes are not kept closed for more than a certain period of time (several seconds) on a daily basis, such movements occurred during work (for example, while driving a car) by detecting “blinks” and “eye meditation” In some cases, not only can the occurrence be recorded, but an alert can be issued to the worker.

  In addition, by providing the AR eyewear 100 with an acceleration sensor / gyro sensor (11e in FIG. 3), the detection accuracy of the body movement (such as a head sag caused by a nap) of the user wearing the AR eyewear 100 is improved (eye Compared to the case where only the motion detection unit 15 is used, this can be improved.

  FIG. 7 is a flowchart for explaining an example of the psychosomatic state estimation method according to the third embodiment. Here, the degree of concentration (whether the line of sight is not moving), fatigue level, stress, etc. at the time of glaucoma visual field examination are totalized and recorded. First, the facial skin around the eye of the subject is brought into contact with the EOG electrode of the eye frame of the visual field inspection apparatus (not shown) (ST50).

  Next, a timer 11f that provides a time stamp is started, and a bright spot (not shown) for visual field inspection is generated in accordance with a predetermined program procedure in front of the right eye (or left eye) of the subject. When the subject sees the bright spot, he presses an inspection button (not shown) to notify the apparatus that the bright spot has been seen. The event that the check button is pressed is recorded in the memory 11b as detection data together with the time stamp when the button is pressed and the EOG data (FIG. 4) at that time (ST52).

  When the bright spot generation program is completed (YES in ST54), the reliability of the detection data recorded in the memory 11b is evaluated (ST56). In this evaluation, if the EOG data when the button is pressed is a waveform corresponding to “Koromotsu” (Pr, Pt, etc.) or a waveform corresponding to “eye meditation” (Pf, etc.) It is estimated that is pressed. If the inspection engineer or ophthalmologist determines that “there are too many button misoperations”, the test is re-examined (YES in ST58).

  If the examination engineer or ophthalmologist determines that “the frequency of erroneous button operation is low enough to cause no problem in diagnosis”, the examination result in the memory 11b is transferred to the local server (personal computer) 1000. The transferred examination result is recorded in the recorder / local database 1002 as a part of the subject's medical record including the past examination result of the subject and the diagnosis view of the doctor (ST60).

  In addition, the process of ST50-ST60 has the case with respect to a test subject's one eye (right eye or left eye), and the case with respect to a test subject's both eyes.

  FIG. 8 is a flowchart for explaining an example of the psychosomatic state estimation method according to the fourth embodiment. Here, the subject's REM sleep and non-REM sleep are detected from EOG of eye movements, and what kind of video the subject was watching when moving from REM to non-REM and / or what kind of voice / music Collect and record information such as whether you were listening. In addition, the REM sleep and non-REM sleep of the subject are detected from EOG of eye movement, and the subject is forcibly awakened at the timing of transition from the REM to the non-REM, and the information of the dream that was seen before awakening is collected and recorded.

  First, the subject (eye wear user) wears the AR eyewear (glasses type, goggles type or eye mask type) 100 with an EOG electrode (ST70). When the eyewear 100 is a glasses-type or goggles-type, a video image projected on an external monitor (not shown) can be seen through the AR display screen. When the eyewear 100 is an eye mask type, a plate-like image display unit (not shown) using organic EL or the like is provided inside the eye mask, and the image display unit can be configured to display video.

  The subject wearing the eyewear 100 is placed in an environment where sleep is likely to be induced. As an example of an environment that easily induces sleep, an environment in which a reclining seat is covered with a soft sheet in a dimly lit room at an appropriate temperature (about 25 ° C.) and an appropriate humidity (about 50 to 60%) can be considered. Alternatively, it may be an environment where a person is lying in a cold room.

  In such an environment where sleep is likely to be induced, playback of the sleep induction video is started. Simultaneously with the start of the video reproduction, the timer (or frame counter) 11f is started, and continuous recording of the subject's EOG data is started using the memory 11b (ST72). The video of the sleep guiding video may be computer graphics (CG), a live-action video, or an animation. This video is not limited to the purpose only for sleep induction, and may include content intended for sleep learning.

  The video display method may be any of 2D, 3D, and 4D. The sound (including vibrations in 4D) may be stereo, monaural, or surround. The video display method and the audio presentation method can be switched as appropriate according to the playback content. The content of the sleep-inducing video is selected from works of images, sounds, and stories that have been confirmed to have a sleep-inducing effect for many people in past experiments (in 4D works, vibration, smell, etc. are added as appropriate). After the subject has fallen asleep (the eye meditation continues), the video will not be seen, so sound (music, etc.) and 4D effects (vibration, etc.) will have a significant meaning. The sound after the subject falls asleep and the contents of the 4D effect can be specified by the time stamp (or frame number).

  Note that physiological data (respiration rate, pulse, blood pressure, etc.) of the subject may be added to the EOG data to be recorded. The respiration rate per minute can be detected by, for example, the timer 11f and the microphone 16 (detecting a breathing sound) in FIG. The microphone 16 can also be used to detect a subject's sleep and snoring. Physiological data such as blood pressure can be obtained using, for example, a wristband type small blood pressure monitor (not shown). If the physiological data is appropriately included in the EOG data and examined, the estimation accuracy of the subject's psychosomatic state can be further increased.

  The subject's sleep (start of sleep) is detected from long-term eye meditation (and the occurrence of sleep and snoring, or a decrease in pulse and blood pressure). Generally speaking, when you fall asleep, you will first see non-REM sleep, then REM sleep, and before you wake up, you will have non-REM sleep (the brain is sleeping) and REM sleep (the brain is active) Appear alternately. For example, REM sleep appears every 90 minutes and lasts about 20 to 30 minutes. Since there is eye movement during REM sleep (the eye moves slightly), REM sleep can be distinguished from non-REM sleep by detecting this eye movement from the EOG waveform. Thereby, REM sleep and non-REM sleep are detected from the presence or absence of eye movement during sleep, and those detection events are recorded in the memory 11b together with the time stamp (ST74). I often dream because my brain is active during REM sleep.

  When switching from REM sleep to non-REM sleep, a change occurs that eye movement during sleep is not detected. When such a change occurs (ST76 YES), if the subject is forcibly awakened, there is a high possibility of recalling the dream he was looking at during the last REM sleep.

  Therefore, when investigating a dream, the subject is forcibly awakened (ST78 YES), and the subject is recorded with the contents of the dream (regardless of the means for recording the dream, even with voice recording using the microphone 16, Handwritten notes may be used). The contents of the dream are recorded in the memory 11b together with the time stamp at the time of awakening (ST80). EOG data before awakening is recorded separately (ST72), and the content of the video played during REM sleep immediately before awakening (especially information appealing to the five senses other than vision: music, sound effects, or multiplication table) Simple, rhythmic speech, etc.) can be identified from the time stamp.

  In order to collect another dream data, the subject is caused to sleep again (YES in ST82), and the processes of ST72 to ST80 are repeated. Otherwise (NO in ST82), all the data recorded for each subject (time stamp, EOG data, dream content memo, etc.) is sent to the local server 1000, organized and recorded in the recorder / local database 1002 ( ST84). In particular, when the content of the dream is a handwritten memo, ask the subject after awakening to rewrite the document in an easy-to-understand meaning (for example, a person who came out of the dream said “Ningarashi, Nigarashi” The document data may be recorded in the recorder / local database 1002. The data recorded in the local server 1000 is appropriately sent to the main server 10000 and recorded in that server.

  Data recorded in the local server 1000 and / or the main server 10000 can be used for the following investigation, for example. That is, in REM sleep and non-REM sleep that occur once or more during one sleep time (about 3 to 9 hours), it is possible to investigate how the video content reproduced during sleep is reflected in the dream (ST88). ). The findings can be used to develop sleep learning.

  In addition, the time required for sleep induction from the start of reproduction of the sleep induction video to the start of sleep (or until the first REM sleep or non-REM sleep is detected) can be measured using the timer 11f. Various sleep induction videos V1, V2, V3,... Are shown to a plurality of subjects, and individual sleep induction time data in the reproduction of each video V1, V2, V3,. Analyzing the time data for sleep guidance can be used to produce “videos that can sleep well / sleep” (movies).

  FIG. 9 is a diagram for explaining an example of mounting the EOG electrode in another embodiment. This embodiment can be used in, for example, gaze, concentration / stress checks in visual field examinations of glaucoma patients. In the electrode mounting example of FIG. 9, the electrode arrangement different from that of FIG. 1 is performed on the cushion surface of the goggle-type eyewear (the surface that contacts the user's facial skin).

  That is, for example, a silicone cushion is provided on the goggle frame 110, and required electrooculogram detection electrodes (EOG electrodes 151a, 151b, 152a, 152b) are attached to the illustrated positions of the cushions. The silicone cushion is softly deformed to fit the unevenness of the user's face, so that all EOG electrodes are in stable contact with the user's skin surface.

  On the goggle frame 110, four LEDs (M01, M02, M11, M12) as calibration markers are attached (optionally). Furthermore, a center marker (optionally) is formed directly in front of the left and right eyes (positions P1 and P2) as a guide for the user to look straight ahead. The center marker can be formed, for example, by scoring at predetermined locations (positions P1 and P2) of a transparent plastic plate fitted into the frame 110. When side light from the LEDs attached to the peripheral frame 110 hits the center marker, a part of the light is diffusely reflected at the center marker portion, so that a bright spot can be seen by the user.

  When a display using film liquid crystal or the like is provided on the transparent plate in the frame 110, AR display or VR display can be performed by the display. (AR stands for Augmented Reality and refers to technology that adds various information to the real world that can be seen through goggles. VR stands for Virtual Reality and is separated from the real world. Arbitrary information can be displayed.The luminance pattern display for visual field inspection can also be regarded as a kind of VR display.) This AR display (or VR display) is used to make markers (M01, M02, M11, M12, P1, P2) can also be presented to the user.

  In addition, 12L and 12R are made with an organic EL panel. If the entire front surface of the eye frame is covered with a light shielding sheet, an eye mask with a video display (or VR display) function is obtained.

  The structure shown in FIG. 9 can be modified to provide an EOG electrode on a hat with a chin strap (head mount device such as a conductor cap). Although not shown in the figure, a cap with a chin strap often worn by a train conductor (user) wearing a uniform has a forehead that abuts the user's forehead and a chin strap that fixes the worn hat to the user's head. attached. Although not shown, the EOG electrodes 151a and 152a shown in FIG. 9 can be provided at the inner portion of the frame. Although not shown, the EOG electrode 151b of FIG. 9 is slidably attached to the right cheek portion of the chin strap, and the EOG electrode 12b of FIG. 9 is slidably attached to the left cheek portion of the chin strap. The user of the conductor cap adjusts the length of the chin strap and fixes the hat after wearing the conductor cap so that the hat does not fall off even if the user moves his body or receives wind. After this fixing, the user slides the EOG electrode 151b on the right side of the chin strap near the right ear so that the center of the right eye (P2 in FIG. 9) is on the line connecting the electrodes 151a and 151g. Similarly, the user slides the EOG electrode 152b on the left side of the chin strap near the left ear so that the center of the left eye (P1 in FIG. 9) is on the line connecting the electrodes 152a and 152g.

  In this way, the electrooculogram of the user's left and right eyeballs is detected by the EOG electrodes 151a, 151b, 152a, 152b provided on the forehead and chin strap of the conductor cap.

In summary, when applying the EOG electrode to the conductor cap,
The upper electrodes (151a and 152a) are placed inside the forehead of the conductor cap;
-Electrodes (151b and 152b) that are movable in the vertical direction are arranged on the parotid masseter part of the jaw string. Here, the point is that the electrode on the chin strap side is movable in the vertical direction. The electrode positions on the chin strap are aligned so that the line connecting the left and right upper and lower electrodes passes near the center of each eyeball. This makes it possible to obtain an EOG detection signal with sufficient amplitude to detect blinks and line of sight.

  FIG. 10 is a diagram for explaining an example of mounting the EOG electrode in the eyeglass-type eyewear according to another embodiment (an example of AR glasses in which the EOG electrode is arranged around the eyeball). The eyewear 100 in FIG. 10 is similar to the eyewear 100 in FIG. 9 with respect to the EOG electrode arrangement, but differs from the eyewear 100 in FIG. 1 in the following points.

  That is, the EOG electrodes 151a and 151b of the right nose pad 150R are moved to the right eye frame 101 side, and the EOG electrodes 152a and 152b of the left nose pad 150L are moved to the left eye frame 102 side. The EOG electrodes 151a and 151b are arranged at positions that are substantially point-symmetric with respect to the user's right eye center position (not shown), and the EOG electrodes 152a and 152b are with respect to the user's left eye center position (not shown). It is arranged at a position that is substantially point-symmetric. Further, the right oblique line connecting each of the EOG electrodes 151a and 151b and the left oblique line connecting each of the EOG electrodes 152a and 152b are substantially line symmetrical with respect to a vertical line (not shown) along the user's nose. The electrodes 151a, 151b, 152a, and 152b can be formed of a conductive member (metal, conductive polymer, or the like) provided at the tip of an elastic body (sponge, silicone cushion, or the like). Each EOG electrode is lightly pressed against the skin surface of the face of the user wearing the eyewear 100 by the elastic repulsive force of the elastic body.

  When the EOG electrode arrangement structure as shown in FIG. 10 is adopted, the electric field generated around the charged eyeball can be detected more easily than the structure in which the EOG electrode is arranged in the nose pad portion. Therefore, the embodiment of FIG. 10 can detect an EOG signal having a larger amplitude than the embodiment of FIG.

  FIG. 11 is a view for explaining an example of mounting EOG electrodes in goggle-type eyewear (a type in which the eye frames of both left and right eyes are continuous) according to still another embodiment (AR glasses in which EOG electrodes are arranged around the eyeball). Other example). When the structure as shown in FIG. 11 is adopted, the skin contact stability of the EOG electrode is higher than that in the structure of FIG. Therefore, the EOG signal can be detected with higher accuracy. In the structure of FIG. 11, the information processing unit 11 and other devices that have been increased in size are easily incorporated. Therefore, in the structure of FIG. 11, the function of the smartphone with GPS can be loaded without worrying about design. Screen display in that case can be performed by AR display on the left and right displays 12L / 12R using film liquid crystal or the like. Although not shown, a small speaker can be attached near the left and right ears of the goggle frame, and a small microphone can be attached near the nose cushion. In addition, command input to the smartphone can be performed by eye movement such as eye movement, blinking, eye meditation, and winking. The goggle-type eyewear 100 as shown in FIG. 11 can be used for concentration / stress check of skiers and snowboarders. Further, the goggle-type eyewear 100 can support not only AR display but also VR display.

  FIG. 12 is a view for explaining an example of mounting EOG electrodes in goggle-type eyewear (a type in which the eye cups of the left and right eyes are separated) according to still another embodiment (AR glasses in which EOG electrodes are arranged around the eyeball). Is yet another example). In the embodiment of FIG. 12, the same EOG electrode arrangement structure and the arrangement structure of the sensor unit 11e are adopted as in the case of FIG. However, the goggles structure of FIG. 12 can withstand use of underwater diving (or use of sky diving). The goggles in FIG. 12 can be used for concentration / stress checks during training of marine divers and skydivers.

The application target of the VR display function is not limited to goggles type eyewear as protective glasses. For example, a VR display function can be added to the goggles 100 as shown in FIG. With this VR goggles 100, training videos of underwater diving and sky diving (or game) can be shown to the user. (This training video can include live action, computer graphics, animation, etc. of training content that is difficult for beginners to handle.)
It can be detected using the EOG electrodes (two or more of 151a, 151b, 152a, and 152b) of the VR goggles 100 what kind of eye movement reaction the user showed in a specific scene of the training video. Further, the specific scene when the eye movement reaction is detected can be specified by the time stamp (or frame number) from the timer / frame counter 11f in FIG. The detection result of the eye movement reaction is temporarily stored in the memory 11b of FIG. 3 together with the time stamp (or frame number) as detection data. The temporarily stored detection data can be taken into the local server (personal computer) 1000 after the training video ends.

  The trainer (training instructor) can analyze the degree of concentration of the subject (VR goggles user) in the video scene serving as the checkpoint from the occurrence pattern of blinks and the change thereof. The personal computer of the local server 1000 can be used for this analysis.

  As a checkpoint, the video content when a large number of subjects have the same eye movement reaction (similar blink generation pattern and change timing thereof) can be used. The video content at the checkpoint can be confirmed by reproducing the corresponding scene based on the time stamp (or frame number). For example, in a skydiving training video, the scene where the parachute rope gets entangled with the body becomes a checkpoint, and how the subject's blinking appears at that time compared to the case of many subjects (or experts) (One or two blink reactions: calm or focused, eye meditation or many blink reactions: panicked, etc.).

  For the blinks that are detected when the training video scene is a monotonous aerial scene (such as blinks that the subject unconsciously performs), the occurrence timing varies from subject to subject. Can be removed.

  The use of the VR goggles 100 as described above can provide the same effect as the application of 3D glasses for movie theaters.

  Applications other than those described above are also conceivable for the AR / VR eyewear of the present application. For example, a camera 13L / 13R is incorporated in the eyewear 100 of FIG. 1 or FIG. During live-action recording using this camera, various eye movement reactions (blink occurrence patterns and their change timing) can be recorded in the memory 11b together with a time stamp (or frame number) (in this case, the memory 11b). It is desirable to use a flash memory with as large a capacity as possible).

  Specific scenes of this live-action recording (for example, people suddenly collided from behind when waiting for a train at the station platform) and eye movements that occurred at that time (for example, short eye meditation and many blinks) Can be recorded together with a time stamp (or frame number), and the recorded contents can be transferred to the local server 1000 as appropriate. The data transferred to the local server 1000 can be used to analyze what tension / stress has occurred depending on the situation where the user of the eyewear 100 is placed.

<Summary of Embodiment>
(A) For example, electrooculogram sensing can be performed using a minimum of one ADC by disposing at least one pair of electrodes on the upper and lower nose pads. As a result, low power consumption in 3D glasses (the power required for electrooculogram sensing using ADC is equivalent to an electrocardiograph operating wirelessly) and low cost (feature MIPS for feature value detection for “blink” detection) By performing at about several tens of MIPS, it is possible to detect “blink” (instantaneous) and “eye meditation” (continuous for a certain period of time) in a wirelessly operated electrocardiograph.

  (B) With the technology of the embodiment, electrooculogram sensing can be applied to 3D glasses for movie theater 3D screening (supporting both shutter and polarization methods) and combining it with concentration, fatigue, and stress estimation. Can do. This makes it possible to tabulate the audience's response (concentration / fatigue / stress along the time axis or for each scene) to the movie currently being shown. Furthermore, it can be fed back to the production of new movies as a VOC (Voice of the Customer).

(C) Furthermore, by adding an acceleration sensor or a gyro sensor to the 3D glasses, the range of VOC collection in an experience type / attraction type theater called 4D can be expanded. (In the experience type / attraction type theater, in addition to moving the seat, you can experience Air, Water mist, fragrance, foam, fog, wind, flash, snow, rain. 4DX, There is TOHO CINEMAS MediaMation MX4D etc.)
Here, in electrooculogram sensing, it is possible to estimate the direction of the line of sight (2 to 3 ADCs) from blinking or eye meditation (at least one ADC), but an acceleration sensor or gyro sensor is added to this. Thus, it is possible to collect VOC information including the direction in which the 3D glasses are facing (the direction in which the glasses user's head is facing).

(D)
In addition, there is a voice saying that it is difficult to read subtitles in theater 3D glasses (subtitles are also 3D and displayed in the foreground, so if you reciprocate the line of sight with other images), there is congestion based on AR glasses It is also effective to display subtitles including angle control. (Effectiveness of AR subtitle display including convergence angle control can be confirmed by estimating stress using electrooculogram sensing while using known methods. ).

<Example of Correspondence Between Contents of Claims Initially Applied and Embodiments>
[1] A psychosomatic state estimation apparatus according to an embodiment adopts a device form of eyewear (100), and based on the electrooculogram of the user wearing the eyewear (EOG in FIG. 4), An eye movement detection unit (15) for detecting the eye movement, a timer (11f) for measuring timing when the eye movement is detected, and an information processing unit (11) for performing information processing based on the eye movement. . In this psychosomatic state estimating apparatus, external stimuli (for example, stimulation to the five senses in various entertainment scenes; stimulation to the five senses in various stress test images; visual and mental tension in visual field examinations) are given to the user wearing the eyewear. Characteristic eye movements of the user (for example, blink Pc / Ps, eye meditation Pf, up / down / left / right movement or rotation Pr / of eyeball in FIG. 4) The eye movement detection unit (15) detects EOG corresponding to Pt or the like.

  Also, the timing when the characteristic eye movement is detected is measured by the timer (11f), and the content of the external stimulus at the measurement timing and the characteristic eye movement at the measurement timing (the eye The user's psychosomatic state (related to personal evaluation with respect to external stimuli) can be estimated from the corresponding relationship with the EOG waveform corresponding to movement).

  [2] The device of [1] has a memory for temporarily storing information collected by each eyewear. That is, this device stores information (output of a timer or a frame counter) for specifying the content of the external stimulus (such as various movie scenes) and information about the characteristic eye movement (such as blinks) ( 11b).

  [3] The device of [1] is configured so that an external local server and / or main server can collect information. That is, this apparatus mutually corresponds information (timer or frame counter output) specifying the content of the external stimulus (various movie scenes, etc.) and characteristic eye movement (blinks, etc.) information. Means (11a to 11d) for providing the attached survey information to the external server (1000 and / or 10000) can be further provided.

  [4] The eyewear (100) in which the device of [1] is incorporated is configured so as to be able to specify the provider of the survey information. That is, this apparatus has a device ID (11g) that can identify itself, and the eyewear that is the individual information provider can be identified by this device ID.

  [5] A psychosomatic state estimation method according to an embodiment detects an eye movement of the user based on an electrooculogram (EOG in FIG. 4) of the user wearing the eyewear (100), and the eye movement is detected. The structure (FIG. 3) which measures the timing when being performed and performs the information processing based on the said eye movement is used. In this psychosomatic state estimation method, external stimuli (for example, stimulation to the five senses in various entertainment scenes; stimulation to the five senses in various stress test images; visual and mental tension in visual field examinations) are applied to the user wearing the eyewear. The user's characteristic eye movements (for example, blink Pc / Ps in FIG. 4, eye meditation Pf, up / down / left / right of FIG. 4) EOG corresponding to motion or rotation Pr / Pt) is detected (ST12; ST32: ST52; T72). Further, the timing when the characteristic eye movement is detected is measured (ST12; ST32; ST52; ST72). Then, from the correspondence between the content of the external stimulus at the measurement timing and the content of the characteristic eye movement at the measurement timing (an EOG waveform corresponding to the eye movement), the user's psychosomatic state (individual to the external stimulus) (Related to evaluation) is estimated (ST24; ST42-ST44; ST56; ST74-ST88).

  [6] The method of [5] can be used, for example, to investigate the reaction of the audience to a 3D / 4D screening work in a movie theater. In this method, if the characteristic eye movement detected is a blink (Pc in FIG. 4) that is less than a predetermined number of times (1 to 2 times), the user's psychosomatic state is that of the external stimulus at the measurement timing. It is presumed that the content is concentrated on the content (the exciting scene of the movie, etc.) (ST24 in FIG. 5). Alternatively, if the detected characteristic eye movement continues for a predetermined number of times or more (three or more times) blink (Ps in FIG. 4), the user's psychosomatic state is the content of the external stimulus at the measurement timing ( It is presumed that the subject is stressed by a scary scene of a movie) (ST24).

  [7] The method of [5] can be used, for example, to investigate the fatigue level of a long-distance bus driver. In this method, the detected characteristic eye movement corresponds to eye meditation for less than a predetermined time (for example, 3 seconds or more and less than 5 minutes) (until the first blink is detected after eye meditation is detected). Is less than a predetermined time), it is estimated that the user's state of mind and body is tired or doze (nap) at the measurement timing (ST42 in FIG. 6). Alternatively, the detected characteristic eye movement corresponds to eye meditation for a predetermined time or more (for example, 5 minutes or more) (the time interval from when eye meditation is detected until the first blink is detected) If it is equal to or longer than the time, the user's mind and body state is estimated to be a nap (for example, 15 to 30 minutes of sleep) or a sleep (for example, 90 or more sleeps) after the measurement timing (ST42).

  In a fatigue level survey using this method, when a dozing state is observed during a short period (for example, 30 minutes or less), or a nap or sleep state is observed during a long period (for example, two hours or longer) Can stop the subject from working on long distance bus driving.

  [8] The method of [5] can be used, for example, for visual field survey of glaucoma patients. In this method, the presence or absence of a bright spot or the movement of a bright spot is used as an external stimulus for examination with respect to a subject. In this visual field inspection, if the detected characteristic eye movement is a blink (Pc in FIG. 4) that is less than a predetermined number of times (1 to 2 times) (Pc in FIG. 4), the psychosomatic state of the user is the measurement timing (under visual inspection It is presumed that the state is concentrated on the content of the external stimulus when the examination button is pressed (ST56 in FIG. 7). Alternatively, if the detected characteristic eye movement is an electrooculogram waveform (for example, Pr / Pt in FIG. 4) indicating the vertical movement or rotation (shrinking) of the eyeball, the psychosomatic state of the user is the measurement. It is estimated that the content of the external stimulus at the timing is stressed (ST56).

  [9] The method of [5] can be used, for example, for sleep surveys. In this method, the presence or absence of rapid eye motion is used to detect a change in the state of the subject during sleep. In this sleep test, the detected characteristic eye movement corresponds to sleep for a specific time or longer (for example, 90 minutes or longer) (after a long eye meditation is detected, the first blink is detected by awakening If an electrooculogram showing rapid eye motion (eg, Pr / Pt in FIG. 4) is detected during the sleep period, the user's psychosomatic state is REM sleep. The state is estimated (ST74 in FIG. 8). Alternatively, the detected characteristic eye movement corresponds to sleep of a specific time or longer (for example, 90 minutes or longer) (from the detection of long eye meditation until the first blink is detected by awakening) If the electrooculogram showing rapid eye motion (eg, Pr / Pt in FIG. 4) is not detected during the sleep period, the user's psychosomatic state is a non-REM sleep state. (ST74).

  For example, when a change from REM sleep to non-REM sleep is detected from the EOG waveform of the subject's eye movement, the subject is forcibly awakened. Then, since there is a high possibility of remembering the dream that was seen at the time of REM sleep just before waking up, ask the subject to record the contents of the dream (sound, text, picture, etc.). And the contents of the video image that was played right before the awakening from the time stamp at the time of awakening (the video and sound before going to sleep, the sound after going to sleep) and the content of the dream that the subject had recorded Compare. From this comparison, you can check how the content of the video you were playing during REM sleep was reflected in your dream. This check result can be used for sleep learning.

  Although several embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, replacements, and changes can be made without departing from the scope of the invention. These embodiments and modifications thereof are included in the scope and gist of the invention, and are included in the invention described in the claims and the equivalents thereof. Note that it is possible to combine a part or all of one embodiment of a plurality of disclosed embodiments with a part or all of another embodiment of the plurality of disclosed embodiments. Or included in the abstract.

  DESCRIPTION OF SYMBOLS 100 ... Eyewear (goggles type or glasses type); 110 ... Eye frame; 11 ... Information processing part (processor 11a, non-volatile memory 11b, main memory 11c, communication processing part (GPS processing Part) 11d, integrated circuit including sensor part 11e, etc .; 11e ... sensor part including acceleration sensor, gyro sensor, etc .; BAT ... power source (lithium ion battery etc.); 12 ... display part (right display 12R and left display 12L: IM1 ... Right display image (numeric keypad, alphabet, character string, mark, icon, etc.); IM2 ... Left display image (numeric keypad, alphabet, character string, mark, icon, etc.); 13 ... Camera (right camera 13R) And attached to the left camera 13L or bridge 103 15... Eye movement detector (line-of-sight detection sensor); 16. Microphone (or vibration pickup); 1510... Right side (Ch1) AD converter; 1520... Left side (Ch2) AD converter; Between (Ch0) AD converter; 1514 ... Between left and right (Ch3) AD converter.

Claims (19)

An eye movement detection unit that adopts an eyewear device form, and detects an eye movement including at least vertical movement, left-right movement, blinking, and eye movement of the user based on the electrooculogram of the user wearing the eyewear A motion detector;
A timer for measuring timing when the eye movement is detected;
An information processing unit for performing information processing based on the eye movement;
In the psychosomatic state estimation device comprising:
The eye movement detecting unit detects a characteristic eye movement of the user when an external stimulus including an image related to an eye movement pattern for estimating a predetermined mental and physical state is given to the user wearing the eyewear. Detected by
The timer measures the timing when the characteristic eye movement is detected,
A psychosomatic state estimation apparatus configured to be able to estimate the user's psychosomatic state from a correspondence relationship between the content of the external stimulus at the measured timing and the characteristic eye movement content at the measured timing.   The apparatus according to claim 1, further comprising a memory that stores information for specifying contents of the external stimulus and information about the characteristic eye movement.   The apparatus according to claim 1, further comprising means for providing, to an external server, survey information in which information specifying the content of the external stimulus and the characteristic eye movement information are associated with each other. Eyewear that can be worn on the user's head,
An eye movement detection unit that detects eye movement including at least vertical movement, left-right movement, blinking, and eye meditation based on the user's electrooculogram;
A timer for measuring timing when the eye movement is detected;
An information processing unit for performing information processing based on the eye movement;
With
It has a device ID that can identify itself,
The eye movement detection unit detects characteristic eye movement of the user when an external stimulus including an image related to an eye movement pattern for estimating a predetermined psychosomatic state is given to the user,
The timer measures the timing when the characteristic eye movement is detected,
Eyewear configured to be able to estimate the state of mind of the user from the correspondence between the content of the external stimulus at the measured timing and the content of the characteristic eye movement at the measured timing. Detecting eye movement including at least vertical movement, left-right movement, blinking, eye meditation of the user based on the electrooculogram of the user wearing the eyewear, measuring timing when the eye movement is detected, In a psychosomatic state estimation method using a configuration for performing information processing based on eye movements,
Detecting a characteristic eye movement of the user when an external stimulus including an image related to an eye movement pattern for estimating a predetermined psychosomatic state is given to the user wearing the eyewear;
Measure the timing when the characteristic eye movement is detected,
A psychosomatic state estimation method configured to estimate the psychosomatic state of the user from a correspondence relationship between the content of the external stimulus at the measured timing and the characteristic eye movement content at the measured timing. If the detected characteristic eye movement is blinking less than a predetermined number of times, it is estimated that the state of mind of the user is concentrated on the content of the external stimulus at the measured timing, or
If the detected characteristic eye movement is a blink that continues for a predetermined number of times or more, it is estimated that the state of mind of the user is stressed by the content of the external stimulus at the measured timing. Method 5.   The method according to claim 5, wherein the state of mind and body of the user is estimated according to a time interval from when eye meditation is detected until the first blink is detected. In the visual field test using the presence or absence of bright spots or the movement of bright spots as an external stimulus for the test,
If the detected characteristic eye movement is blinking less than a predetermined number of times, it is estimated that the state of mind of the user is concentrated on the content of the external stimulus at the measured timing, or
If the detected characteristic eye movement is an electrooculogram that indicates the vertical movement or rotation of the eyeball, the state of mind of the user is stressed by the content of the external stimulus at the measured timing. The method of claim 5 for estimating. In a sleep test that uses the presence or absence of rapid eye motion to detect changes in the state of the subject during sleep,
If the detected characteristic eye movement corresponds to sleep of a specific time or more and an electrooculogram showing rapid eye motion is detected during the sleep period, the psychosomatic state of the user is REM sleep. Presumed to be in a state, or
If the detected characteristic eye movement corresponds to sleep of a specific time or more and an electrooculogram showing rapid eye motion is not detected during the sleep period, the psychosomatic state of the user is non-REM sleep. 6. The method of claim 5, wherein the method is estimated to be a condition. An eye movement detection unit that adopts a device form of eyewear, and an eye movement detection unit that detects the eye movement of the user based on the electrooculogram of the user wearing the eyewear,
A timer for measuring timing when the eye movement is detected;
An information processing unit for performing information processing based on the eye movement;
In the psychosomatic state estimation device comprising:
Detecting the eye movement characteristic of the user when an external stimulus is given to the user wearing the eyewear by the eye movement detector;
The timer measures the timing when the characteristic eye movement is detected,
From the correspondence between the content of the external stimulus at the measured timing and the content of the characteristic eye movement at the measured timing, the psychosomatic state of the user can be estimated,
A psychosomatic state estimating apparatus further comprising means for providing, to an external server, survey information in which information specifying the content of the external stimulus and the characteristic eye movement information are associated with each other.   The apparatus according to claim 10, further comprising a memory that stores information for specifying contents of the external stimulus and information about the characteristic eye movement.   The apparatus according to claim 10, wherein the eye movement detection unit has a device ID capable of identifying itself.   The apparatus according to claim 10, wherein the eye movement detection unit detects eye movement including at least vertical movement, left-right movement, blinking, and eye meditation of the user. Psychosomatic state estimation using a configuration that detects eye movement of the user based on the electrooculogram of the user wearing eyewear, measures timing when the eye movement is detected, and performs information processing based on the eye movement In the method
Detecting a characteristic eye movement of the user when an external stimulus is given to the user wearing the eyewear;
Measure the timing when the characteristic eye movement is detected,
From the correspondence between the content of the external stimulus at the measured timing and the characteristic eye movement content at the measured timing, the psychosomatic state of the user is estimated,
A psychosomatic state estimation method configured to provide, to an external server, survey information in which information specifying the content of the external stimulus and the characteristic eye movement information are associated with each other. If the detected characteristic eye movement is blinking less than a predetermined number of times, it is estimated that the state of mind of the user is concentrated on the content of the external stimulus at the measured timing, or
If the detected characteristic eye movement is a blink that continues for a predetermined number of times or more, it is estimated that the state of mind of the user is stressed by the content of the external stimulus at the measured timing. 14. The method according to 14.   The method according to claim 14, wherein the state of mind and body of the user is estimated according to a time interval from when eye meditation is detected until the first blink is detected. In the visual field test using the presence or absence of bright spots or the movement of bright spots as an external stimulus for the test,
If the detected characteristic eye movement is blinking less than a predetermined number of times, it is estimated that the state of mind of the user is concentrated on the content of the external stimulus at the measured timing, or
If the detected characteristic eye movement is an electrooculogram that indicates the vertical movement or rotation of the eyeball, the state of mind of the user is stressed by the content of the external stimulus at the measured timing. The method of claim 14 to estimate. In a sleep test that uses the presence or absence of rapid eye motion to detect changes in the state of the subject during sleep,
If the detected characteristic eye movement corresponds to sleep of a specific time or more and an electrooculogram showing rapid eye motion is detected during the sleep period, the psychosomatic state of the user is REM sleep. Presumed to be in a state, or
If the detected characteristic eye movement corresponds to sleep of a specific time or more and an electrooculogram showing rapid eye motion is not detected during the sleep period, the psychosomatic state of the user is non-REM sleep. The method of claim 14, wherein the method is estimated to be a condition.   The method according to claim 15, wherein the eye movement of the user includes at least vertical movement, left-right movement, blinking, and eye meditation of the user.

Images