JP2021035499A - Eyewear, data collection system and data collection method - Google Patents

Abstract

To estimate a psychosomatic state of a person using an eyewear with an EOG.SOLUTION: A psychosomatic state estimation device in one embodiment is used in an eyewear device mode. Based on an eye potential of a user who wears the above mentioned eyewear, characteristics eye motion (nictation, shutting one's eyes, vertical and lateral motion and rotation of eyeballs) is detected. Then, characteristic eye motion and external stimulation (for example, stimulation to five senses of a human being in various scenes in a movie or video) are associated and recorded by a time code, and both of them can be compared and examined (characteristic eye motion and external stimulation). By examining in advance, correspondence between a pattern of characteristic eye motion and a psychosomatic state of a user (expression of emotion such as crying, laughing and becoming angry, motivation, gathering one's thoughts, fatigue, stress, sleep), it is possible to estimate which psychosomatic state the user is in, by a specific external stimulation.SELECTED DRAWING: Figure 4

Description

Embodiments of the present invention relate to eyewear, data collection systems, and data collection methods to which electro-oculography sensing technology is applied.

Regarding the electro-oculography sensing technique, there is a known example 1 (see Patent Document 1). In this known example 1, an EOG (Electro-Oculografy) electrode is used for electro-oculography sensing, and the user's eye movement is detected from the electrooculographic data acquired from the EOG electrode (paragraph 0023). In the embodiment of Known Example 1, the EOG electrode is provided on the goggles (paragraph 0061), but there is also Known Example 2 in which the electrode for EOG is provided on the nose pad portion of the glasses (see Patent Document 2, paragraph 0013). Goggles and glasses are a type of eyewear that can be easily used by the general public.

Japanese Unexamined Patent Publication No. 2009-288529 Japanese Unexamined Patent Publication No. 2013-244370

EOG is compatible with eyewear, and various applications of eyewear with EOG have been studied. However, its application range has not been completely researched, and new possibilities remain for the application of eyewear with EOG.

For example, when observing the audience reaction of a screening work in a movie theater, it is necessary to observe a large number of people in the dark without interfering with watching the movie. However, since such an observation is difficult to carry out, it is actually an exit poll after watching a movie. In this case, for example, a story with a screening time of 2 hours will be recollected by each audience in several tens of seconds. However, since a considerable amount of time has passed since the audience was impressed by multiple scenes and recalled their impressions, the results of exit polls for the screenings were limited to a broad outline or a targeted amount of information. Will be less. The contents of the survey results I want here are, for example, what kind of emotions were induced in the audience in various scenes of the screening work, whether the audience concentrated on watching, and whether the audience was tired or stressed. It is useful for estimating the mental and physical condition such as whether you have fallen asleep. (This state of mind and body is related to how people responded to or evaluated external stimuli.) However, it is difficult to obtain such findings from exit polls.

For example, it has not been known how to use eyewear with EOG for the above-mentioned problems (it is difficult to estimate the mental and physical condition of people who can change depending on the situation).

One of the problems to be solved by the embodiment of the present invention is to collect data by using eyewear with EOG.

The eyewear according to the embodiment includes a detection unit, a measurement unit, an information processing unit, and a transmission unit.

The detection unit detects a specific eye movement based on the user's electro-oculography when a specific external stimulus is given. The measuring unit measures the timing when the detecting unit detects the specific eye movement. The information processing unit is based on the correspondence between the content of the specific external stimulus at the detection timing of the specific eye movement measured by the measurement unit and the content of the specific eye movement at the detection timing. Estimate the mental and physical condition. The transmission unit includes detection data including information on the content of the specific eye movement detected by the detection unit, information on the detection timing of the specific eye movement measured by the measurement unit, and device identification information. To the server.

The figure explaining the glasses-type eyewear which incorporated the information processing unit 11 for the mind-body state estimation apparatus which concerns on one Embodiment (an example of AR glasses which arranged the EOG electrode on the nose pad). The figure explaining the mounting example of the EOG electrode in the eyeglass type eyewear which concerns on one Embodiment. The figure explaining the relationship between the information processing unit 11 which can be attached to various embodiments, and the peripheral device thereof. Various eye movements (blinking, eye closure, eye movement, eye rotation, etc.) of the user wearing eyewear and detection signal levels (Ch0, Ch1) obtained from the three analog-to-digital converters (ADCs) shown in FIG. , Ch2), an electrooculogram (EOG) illustrating the relationship. The flowchart explaining an example of the mind-body state estimation method which concerns on 1st Embodiment. The flowchart explaining an example of the mind-body state estimation method which concerns on 2nd Embodiment. The flowchart explaining an example of the mind-body state estimation method which concerns on 3rd Embodiment. The flowchart explaining an example of the mind-body state estimation method which concerns on 4th Embodiment. The figure explaining the mounting example of the EOG electrode in another embodiment. The figure explaining the mounting example of the EOG electrode in the eyeglass type eyewear which concerns on another embodiment (the example of AR glasses in which the EOG electrode is arranged around the eyeball). The figure explaining the mounting example of the EOG electrode in the goggle type eyewear (the type in which the eye frames of both left and right eyes are continuous) according to still another embodiment (another example of AR glasses in which the EOG electrode is arranged around the eyeball). .. The figure explaining the mounting example of the EOG electrode in the goggle type eyewear (the type in which the eye cups of both left and right eyes are separated) according to still another embodiment (the other example of AR glasses in which the EOG electrode is arranged around the eyeball). ).

Hereinafter, basic information will be provided first, and then various embodiments will be described with reference to the drawings.

<Basic information>
The diameter of an adult eyeball is about 25 mm. It is about 17 mm after birth and grows as it grows.
The interpupillary distance of an adult male is about 65 mm. (Many commercially available stereo cameras are made at intervals of 65 mm.)
The interpupillary distance of adult females is several millimeters shorter than that of males.
The electrooculogram is several tens of mV.
The eyeball has a positive potential on the corneal side and a negative potential on the retina side. When this is measured on the surface of the skin, it appears as a potential difference of several hundred μV.

The types of eye movements and the range of eye movements related to eye movement detection include, for example: <Types of eye movements (eye movements)> (01) Compensatory eye movements For head and body movements Regardless, involuntary eye movements developed to stabilize the image of the outside world on the retina.

(02) Voluntary eye movement This is an eye movement developed to bring the visual pair image to the center on the retina, and is a movement that can be controlled voluntarily.

(03) Impact eye movement (saccade)
Eye movement (easy to detect) that occurs when changing the gazing point to see an object.

(04) Sliding eye movement Smooth eye movement (difficult to detect) that occurs when tracking a slowly moving object.

<Eye movement range (for general adults)> (11) Horizontal direction Left direction: 50 ° or less Right direction: 50 ° or less (12) Vertical direction Down direction: 50 ° or less Up direction: 30 ° or less (self The vertical angle range that can be moved by the intention of is narrow only in the upward direction. (Because there is a "bell phenomenon" in which the eyeball rolls up when the eye is closed, the vertical eyeball movement range shifts upward when the eye is closed.) (13 ) Others Convergence angle: 20 ° or less.

<For the target to be estimated from the blink> -In the estimation of the degree of concentration, the timing of the blink occurrence on the time axis with respect to the story is investigated from the simple change in the blink frequency.

・ For tension / stress estimation, examine changes in the frequency of blinks.

・ In estimating drowsiness, the change in blinking group and blinking time / blinking speed is examined.

-The estimation of emotions, physical condition, and other mental and physical conditions is largely left to future research, but in the embodiment of the present application, it is illustrated that there is a technical idea for estimating emotions, physical condition, and other mental and physical conditions. doing.

FIG. 1 is a diagram illustrating a glasses-type eyewear 100 in which an information processing unit 11 for a mental / physical state estimation device according to an embodiment is incorporated (an example in which an EOG electrode is arranged on a nose pad). In this embodiment, the right eye frame (right rim) 101 and the left eye frame (left rim) 102 are connected to the bridge 103. The left and right eye frames 102, 101 and the bridge 103 can be made of, for example, an aluminum alloy or titanium. The left outer side of the left eye frame 102 is connected to the left temple bar 106 via a left hinge 104, and a left modern (left ear pad) 108 is provided at the tip of the left temple bar 106. Similarly, the right outer side of the right eye frame 101 is connected to the right temple bar 107 via the right hinge 105, and the right modern (right ear pad) 109 is provided at the tip of the right temple bar 107.

An information processing unit 11 (integrated circuit of several millimeters square) is mounted in the right temple bar 107 (or the left temple bar 106). The information processing unit 11 is a communication processing unit using a microcomputer having a processing capacity of several tens of MIPS or more, a flash memory (EEPROM) having a free area of several tens of MB or more, BlueTooth (registered trademark), USB, or the like. It can be configured by an LSI in which the above-mentioned ADC (Analog-to-Digital Computer) and others are integrated (details of the information processing unit 11 will be described later with reference to FIG. 3).

A small battery (BAT) such as a lithium-ion battery is embedded in, for example, the left temple bar 106 (or the right temple bar 107, or the modern 108 or 109) to serve as a power source necessary for the operation of the eyeglass-type eyewear 100. ing.

A left camera 13L is attached to the end of the left eye frame 102 near the left hinge 104, and a right camera 13R is attached to the end of the right eye frame 101 near the right hinge 105. These cameras can be configured using an ultra-small CCD image sensor.

These cameras (13L, 13R) may constitute a stereo camera. Alternatively, an infrared camera (13R) and a laser (13L) may be arranged at the positions of these cameras to form a distance sensor using an infrared camera and a laser. This distance sensor can also be composed of a small semiconductor microphone (13R) that collects ultrasonic waves and a small piezoelectric speaker (13L) that radiates ultrasonic waves.

With the distance sensor, the distance between the visual object in front of the user's line of sight and the left and right frames 101/102 can be known. Although it depends on the design of the spectacle frame, the position of the user's eyeball is shifted from the left and right frames 101/102 to a position recessed about 40 to 50 mm toward the back of the head, and the interpupillary distance of an adult is about 65 mm. From the interpupillary distance of 65 mm and the distance measured by the distance sensor corrected by the deviation from the frame to the eyeball position, the convergence angle (more eye angle) when the user is looking at the visual object is determined. Can be calculated. When the distance from the frame to the visual object is much larger than the deviation amount from the frame to the eyeball position, the convergence angle can be roughly calculated even if this deviation amount is ignored.

In addition, instead of the left and right cameras 13L / 13R, or in addition to the left and right cameras 13L / 13R, an embodiment in which a central camera (not shown) is provided on the bridge 103 portion is also conceivable. On the contrary, there may be an embodiment in which the camera is not equipped at all. (These cameras are shown as the camera 13 in FIG. 3. These cameras and the distance sensor by the outside line camera + laser can be treated as an option, and can be appropriately provided according to the application of the eyewear 100. Good.)
The left display 12L is fitted in the left eye frame 102, and the right display 12R is fitted in the right eye frame 101. This display is provided on at least one of the left and right eye frames, and can be configured by using a film liquid crystal or the like. Specifically, one or both of the left and right displays 12L and 12R can be configured by using a film liquid crystal display device that employs a polymer-dispersed liquid crystal (PDLC) that does not use a polarizing plate. (This display is shown as display 12 in FIG. 3).
The transparent left and right displays 12L / 12R using a film liquid crystal can be used as a means for providing augmented reality (AR) that adds image information such as numbers and characters to the real world seen through glasses.

The film liquid crystal of the left and right displays 12L / 12R can also be used as a liquid crystal shutter that separates the left and right images when watching a 3D movie (or 3D video). In addition to the liquid crystal shutter method, a polarization method can also be used to configure the 3D glasses.

When the distance between the user's eyeball position and the visual object is known in advance, the above-mentioned convergence angle can be calculated without the distance sensor. For example, the convergence angle of a spectator (interpupillary distance 65 mm) sitting in a seat 10 m away from the screen (at a position that can be identified by the seat number) in the center of the audience seat of a movie theater is about tan-1 [65/10000]. Become. When displaying 3D subtitles in AR on the left and right displays 12L / 12R, the convergence angle when viewing the AR display should be about the same as the convergence angle when viewing the 3D image projected on the screen 10 m away. , It is possible to control the left and right display positions of 3D subtitles to be displayed in AR. By doing so, it is possible to reduce eye strain when continuing to view the AR-displayed 3D subtitles and the 3D screen image 10 m away.

Generally speaking, not limited to watching 3D movies, when the displays are on the left and right, the images (IM1, IM2 in FIG. 1) to be displayed on the left and right displays are displayed on the left and right according to the desired convergence angle. It can be shifted in the opposite direction. As a result, it is possible to reduce the burden on the eyes when alternately viewing the object visible in the real world and the AR display.

A nose pad portion is provided between the left and right eye frames 102 and 101 and below the bridge 103. This nose pad portion is composed of a pair of a left nose pad 150L and a right nose pad 150R. The right nose pad 150R is provided with right nose pad electrodes 151a and 151b, and the left nose pad 150L is provided with left nose pad electrodes 152a and 152b.

These electrodes 151a, 151b, 152a, 152b are electrically separated from each other and connected to three AD converters (ADC 1510, 1520, 1512) via an insulated wiring material (not shown). The outputs from these ADCs have different signal waveforms according to the movement of the eyes of the user wearing the eyewear 100, and are supplied to the information processing unit 11 as digital data according to the movements of the eyes of the user. The electrodes 151a, 151b, 152a, 152b are used as eye-gaze detection sensors, and together with the three ADCs, are components of the eye movement detection unit 15 in FIG. The three ADCs can be incorporated into one LSI together with the information processing unit 11. The number of ADCs used can be increased or decreased as appropriate depending on the purpose, and the number of 3 is an example (for example, when the purpose is only to detect the EOG waveform due to the blink on the right side, the number of ADCs used is ADC1510. Only one is required. Depending on the purpose, the number of ADCs may be one or two, or four or more may be used.)
The right nose pad 150R is provided with a right microphone (or right vibration pickup) 16R, and the left nose pad 150L is provided with a left microphone (or left vibration pickup) 16L. The left and right microphones 16L / 16R can be configured by, for example, an ultra-small electret condenser microphone. Further, the left and right vibration pickups 16L / 16R can be configured by, for example, a piezoelectric element. Sounds and movements induced by large emotional expressions such as laughter, screams, angry waves, and reflexive body movements when surprised can be detected using a microphone (or vibration pickup) 16L / 16R (these microphones or vibrations). The pickup is shown as a microphone (or vibration pickup) 16 in FIG. A small sobbing nose can be detected by the left and right microphones 16L / 16R. The user's body movement can also be detected by an acceleration sensor or a gyro built in the sensor unit 11e of FIG.

The eyewear 100 is fixed to the user's head (not shown) by the left and right nose pads (150L, 150R), the left and right temple bars (106, 107), and the left and right modern (108, 109). In this embodiment, only the left and right nose pads (150L, 150R), the left and right temple bars (106, 107), and the left and right modern (108, 109) need to directly touch the user's head (or face). However, there may be an embodiment in which parts other than those (nose pad, temple bar, modern) touch the user in order to adjust the voltage between the ADC (1510, 1520, 1512) and the user's body.

The film liquid crystal of the right display 12R of FIG. 1 can display, for example, a right display image IM1 including Japanese characters (hiragana, katakana, and kanji), numbers, alphabets, marks of a predetermined shape, and other icon groups. The left display image IM2 having the same contents as IM1 can be displayed on the film liquid crystal of the left display 12L.

The display contents of the displays 12L and 12R may be anything. For example, the bright spot used for the visual field test of glaucoma can be displayed on the display 12L or 12R. When the display is not performed, the display state can be set to black on the entire surface, and the film liquid crystal can be used as a blindfold or a light-shielding sheet. The numeric keypad and alphabets displayed on the right display 12R (or left display 12L) can be used when inputting numbers and letters. Character strings, marks, icons, etc. displayed on the right display 12R (or left display 12L) can be used when searching for a specific information item, selecting / determining a target item, or alerting the user. ..

The display images IM1 and IM2 can be used as augmented reality (AR) display means for adding information such as numbers, characters, and marks to the real world seen through glasses, and the AR display can be turned on and off as appropriate. The contents of the display image IM1 and the display image IM2 may have the same contents (IM1 = IM2) or different contents (IM1 ≠ IM2) depending on the embodiment. Further, the display image IM1 (or IM2) can be displayed on the right display 12R and / or the left display 12. If you want the content of the AR display to be a 3D image that overlaps (with depth) in the real world that you can see through the glasses, you can make IM1 and IM2 separate left and right 3D images.

Further, when the displays (12R, 12L) are present on the left and right, for example, the convergence angle can be adjusted to shift the images of the left and right display images (IM1, IM2) in the left and right directions in opposite directions. As a result, it is possible to reduce the burden on the eyes when alternately viewing the object visible in the real world and the AR display. However, normally, the left and right displays (12R, 12L) display the same image.

The display control on the displays 12L and 12R can be performed by the information processing unit 11 embedded in the right temple bar 107. (Technology for displaying characters, marks, icons, etc. on a display is well known.) The power supply required for the information control unit 11 and other operations can be obtained from the battery BAT embedded in the left temple bar 106.

In addition, there is a possibility that the designer or the designer may feel that the weight balance is poor when the prototype of the eyewear 100 corresponding to the embodiment is attached. If the main cause is the BAT in the left temple bar 106, a "weight (or another same weight battery)" commensurate with the BAT in the left temple bar 106 can be placed in the right temple bar 107.

FIG. 2 is a diagram illustrating a mounting example of an EOG electrode in the eyeglass-type eyewear 100 according to the embodiment. Right nose pad electrodes 151a and 151b are provided above and below the right nose pad 150R, and left nose pad electrodes 152a and 152b are provided above and below the left nose pad 150L. The output of the right nose pad electrodes 151a, 151b is given to the ADC 1510, the output of the left nose pad electrodes 152a, 152b is given to the ADC 1520, and the output of the lower electrodes 151b, 152b (or upper electrodes 151a, 152a) of the left and right nose pads. Is given to ADC 1512.

From the ADC 1510, a Ch1 signal that changes according to the user's right upper and lower eye movements can be obtained. From the ADC 1520, a Ch2 signal that changes according to the user's left upper and lower eye movements can be obtained. From the ADC 1512, a Ch0 signal that changes according to the left and right eye movements of the user can be obtained. The vertical movement of the left and right eyes may be evaluated by averaging the outputs of the ADC 1510 and the ADC 1520. (The relationship between the signal waveforms of Ch0, Ch1, and Ch2 and the eye movement will be described later with reference to FIG. 4.) <Supplementary explanation of the electro-oculography sensor shown in FIGS. 1 and 2> * Provided on the nose pad. About EOG electrodes A minimum of one set (a set of 151a and 151b; or a set of 152a and 152b) of measurement electrodes is provided for detecting eye movement in the vertical direction. In that case, electrodes for measuring the electrooculogram are provided above and below the left or right (either one) nose pad (the number of electrode terminals is two).

A set of measuring electrodes is added to improve the accuracy of detecting eye movement in the vertical direction. In that case, electrodes (151a and 151b; and 152a and 152b) for measuring the electrooculogram are provided above and below the left and right (both) nose pads 150L / 150R (the number of electrode terminals is four).

* About ADC (Analog-to-Digital Converter) At least one ADC (for Ch1 or Ch2) is provided for detecting eye movement in the vertical direction.

One ADC (for Ch2 or Ch1) is added to improve the accuracy of detecting eye movement in the vertical direction (as a result, one ADC is provided for both Ch1 and Ch2). (Option for improving accuracy)
One more ADC (for Ch0) is added for detecting eye movement in the left-right direction. (Option for function improvement)
FIG. 3 is a diagram illustrating a relationship between the information processing unit 11 that can be attached to various embodiments and peripheral devices thereof. In the example of FIG. 3, the information processing unit 11 includes a processor 11a, a non-volatile memory (flash memory / EEPROM) 11b, a main memory 11c, a communication processing unit 11d, a sensor unit 11e, a timer (frame counter) 11f, a device ID 11g, and the like. It is configured. The processor 11a can be configured by a microcomputer having a processing capacity according to the product specifications. Various programs executed by this microprocessor and various parameters used at the time of program execution can be stored in the non-volatile memory 11b. Various detection data can be stored in the non-volatile memory 11b together with a time stamp (and / or a frame number counted by the frame counter) using the timer 11f. The main memory 11c provides a work area for executing the program.

The sensor unit 11e includes a group of sensors for detecting the position and / or the orientation of the eyewear 100 (or the head of the user wearing the eyewear). Specific examples of these sensors include an acceleration sensor that detects movement in the three-axis direction (xyz direction), a gyro that detects rotation in the three-axis direction, and a geomagnetic sensor that detects absolute orientation (compass function). , There is a beacon sensor that receives radio waves, infrared rays, etc. to obtain position information and the like. IBeacon (registered trademark) or Bluetooth (registered trademark) 4.0 can be used to acquire this location information and others.

The LSI that can be used in the information processing unit 11 has been commercialized. One example is Toshiba Semiconductor & Storage's "TZ1000 series for wearable terminals". Among this series, the product name "TZ1011MBG" includes CPU (11a, 11c), flash memory (11b), Bluetooth Low Energy (registered trademark) (11d), sensor group (accelerometer, gyro, geomagnetic sensor) (11e). , 24-bit delta-sigma ADC, I / O (USB, etc.).

What to do with the processor 11a can be instructed from the local server (or personal computer) 1000 via the communication processing unit 11d. The communication processing unit 11d can use existing communication methods such as ZigBee (registered trademark), Bluetooth (registered trademark), and Wi-Fi (registered trademark). The processing result in the processor 11a can be sent to the local server 1000 in real time via the communication processing unit 11d. Further, the information stored in the memory 11b can also be provided to the local server 1000 via the communication processing unit 11d. The local server 1000 can record and store the information sent from the information processing unit 11 of one or more eyewear 100s in the recorder / local database 1002. The information recorded / stored in the recorder / local database 1002 can be sent to the main server 10000 via a network such as the Internet. Various information is sent to the main server 10000 from one or more local servers 1000, and it is possible to construct a big data database by integrating the information.

A display 12 (12L and 12R), a camera 13 (13L and 13R), an eye movement detection unit 15, a microphone (or vibration pickup) 16, and the like are connected to the system bus of the information processing unit 11. Each device (11-16) in FIG. 3 is powered by a battery BAT.

The eye movement detection unit 15 of FIG. 3 has four eye movement detection electrodes (151a, 151b, 152a, 152b) constituting the line-of-sight detection sensor, and three ADCs (151a, 151b, 152a, 152b) that extract digital signals corresponding to eye movements from these electrodes. It includes 1510, 1520, 1512) and a circuit that outputs the output data from these ADCs to the processor 11a side. The processor 11a interprets a command or state corresponding to the type of eye movement from various eye movements (vertical movement, left-right movement, blinking, eye closing, etc.) of the user, and performs processing corresponding to the command or state. Can be executed.

As a specific example of the command corresponding to the type of eye movement, if the eye movement is, for example, eye meditating, select the information item at the tip of the line of sight (similar to one click of a computer mouse), and a predetermined number of consecutive blinks of both eyes. (Or one eye blink / wink), there is a command (similar to double-clicking a computer mouse) to start executing processing for the selected information item.

Specific examples of the state corresponding to the type of eye movement include a predetermined number of blinks, eye closure for a predetermined time, and up / down / left / right movement of the line of sight. Then, it can be estimated that the user concentrated on the content of the lecture and understood it. If the eye movement is a few seconds or more during the reproduction of the examination image, it can be estimated that the user has fallen asleep during the reproduction of the examination image. If the eye movement is up / down / left / right during sleep, it can be estimated that the user was in REM sleep.

The command or state corresponding to the type of eye movement is an example of information input B using the eye movement detection unit 15.

<Supplementary explanation of the components shown in FIG. 3> * Operation of the microcomputer unit constituting the processor 11a By configuring the detection engine such as blinking by programming the microcomputer unit and operating this detection engine, It is possible to record a blinking event with a time stamp instead of real-time raw ADC data. (For example, if there is an event in which a blink occurs once or twice in succession 5 minutes and 10 seconds after the timer is started, the raw data obtained by AD-converting the EOG waveform of that blink is not recorded. Instead, the occurrence of such a blink is recorded with a time stamp of 5 minutes and 10 seconds.) In this way, the memory capacity required for recording can be significantly saved as compared with recording the raw data as it is. If there is sufficient memory capacity, ADC raw data may be recorded.

* About the flash memory / EEPROM that constitutes the non-volatile memory 11b For example, consider the case of collecting momentary information from a large number of spectators who are watching a movie in a movie theater. In that case, rather than continuing to collect blink information from all spectators in real time, the blink information of each spectator is temporarily stored in the local memory (11b) of each spectator, and after the movie is over, all spectators It is possible to collect information more efficiently by collecting the instantaneous information stored from the local memory (11b).

* About communication processing unit 11d For example, when the movie ends and the temporary storage of the blink information is completed in the local memory (11b) of all the spectators, the temporarily stored blink information (with time stamp) is stored locally in the movie theater. It can be collected by the server (1000). This collection can be performed, for example, via Bluetooth (wireless) or USB (wired) provided in a movie theater seat or a specific location in the theater.

* Power supply (battery / battery) BAT A small battery is used to maintain the power supply to perform the following operations.

-During the movie screening, the "blink" detection engine is operated to detect the "blink" in real time from the AD conversion result of the EOG waveform, and the detected "blink" event information is displayed together with the time stamp. Recording is performed in the memory 11b.

-After the movie is shown, the "blink" event information recorded in the memory 11b can be collected.

FIG. 4 shows various eye movements (blinks, eye closures, eye rotations, etc.; especially blinks) of the user wearing the eyewear 100 and detections obtained from the three analog-to-digital converters (ADCs) shown in FIG. It is an electrocardiogram (EOG) exemplifying the relationship with the signal level (Ch0, Ch1, Ch2). The vertical axis shows the sample value of each ADC (corresponding to the EOG signal level), and the horizontal axis shows the time.

In the ADC sample value of the Ch0 EOG waveform shown in the upper part of FIG. 4, the combination waveform Pr (or Pr *) of the concave waveform when the line of sight is turned to the left and the convex waveform when the line of sight is turned to the right is used in the left-right direction. Eye rotation is detected.

In the ADC sample value of the Ch1 EOG waveform shown in the middle of FIG. 4 or the ADC sample value of the Ch2 EOG waveform shown in the lower part of the figure, the convex waveform when the line of sight is directed upward and the convex waveform when the line of sight is directed downward. The vertical eye rotation is detected by the combination waveform Pt (or Pt *) of the concave waveform.

In the ADC sample value of the Ch1 EOG waveform shown in the middle of FIG. 4 or the ADC sample value of the Ch2 EOG waveform shown in the lower part of the figure, the level of the 1st and 2nd blinks momentarily rises and returns to the original value. It is detected by the convex pulse waveform Pc.

In the ADC sample value of the Ch1 EOG waveform shown in the middle of FIG. 4 or the ADC sample value of the Ch2 EOG waveform shown in the lower part of the figure, the third blink momentarily raises the level and returns to the original convex pulse. It is detected by the waveform Ps. (Although not shown, 4 or more blinks can be detected in the same way as 3 times.)
In the ADC sample value of the Ch1 EOG waveform shown in the middle of FIG. 4 or the ADC sample value of the Ch2 EOG waveform shown in the lower part of the figure, the eye closure that occurs for a longer period than the blink is detected by the wide convex pulse waveform Pf. To. (Not shown, longer eye closures are detected in wider pulse width waveforms.)
<Relationship between EOG waveform and eye movement in Fig. 4>
(01) When the line of sight is moving from side to side, the waveform Pr (or Pr *) appears. When the line of sight is moving up and down, the waveform Pt (or Pt *) appears. When the line of sight is moving from side to side and up and down, the waveform Pr (or Pr *) and the waveform Pt (or Pt *) appear continuously. That is, "staggering" type eye movements can be detected from the EOG waveforms Pr and / or Pt (Pr * and / or Pt *).

(02) When the blink is made, pulse waveforms Pc and Ps (or Pc *, Ps *) corresponding to the number of blinks appear. That is, various times of "blinking" type eye movements can be detected from the EOG waveforms Pc and Ps (or Pc *, Ps *). The Ch0 ADC is unnecessary as long as it can detect "blinking" type eye movements. For the purpose of detecting "blinking" type eye movements, at least one ADC of Ch1 or Ch2 is required. However, if two ADCs, Ch1 and Ch2, are used and the outputs of both are combined, the signal-to-noise ratio (S / N) can be improved. The reason is that the signal components (Pc, Ps, Pf, Pt, etc.) that are correlated between Ch1 and Ch2 increase by +6 dB due to the combination of the two ADC outputs, but the random noise components that do not correlate increase by only +3 dB. Because there isn't.

(03) When the eye closure is performed, a pulse waveform Pf (or Pf *) corresponding to the length of the eye closure appears. That is, from the EOG waveform Pf (or Pf *), "eye closure" type eye movements of various lengths can be detected. Although the "eye-closing" type eye movement appears a little in the Ch0 EOG waveform, the Ch0 ADC may not be present for the purpose of detecting the "eye-closing" type eye movement.

<Relationship between eye movement and mental and physical condition>
(11) When "staggering" type eye movements are detected during an EOG-based examination (for example, during a visual field test for glaucoma), it is presumed that the subject is in a state of restlessness due to lack of concentration. Can be considered.

(12) During an inspection using EOG (for example, during a lecture or class comprehension survey), a small number of times (for example, 1-2 times) at key points (such as the breaks in the meaning of the lecture content) When eye movements are detected, it is presumed that the subject is in a state of concentration and understanding.

(13) During the inspection using EOG (for example, during the aptitude test of the customer service clerk), the number of inspection contents (such as "the subject receives various orders from multiple customers in a crowded store") When "blinking" type eye movements are detected more than once (for example, three times or more), it is considered that the subject is in a tense and mentally stressed state.

(14) When short "eye closing" type eye movements are detected multiple times during an inspection using EOG (for example, during an inspection using a drive simulator, during a monotonous running screen and monotonous continuous vibration sound output). Is presumed that the subject is tired and unable to concentrate on the test items.

(15) When long-term "eye closure" type eye movement is detected during an EOG-based examination (for example, during a sleep survey), the subject is sleeping (nap) and is in non-REM sleep or REM sleep. It is conceivable to presume that it is in.

(16) When a "sloppy" type eye movement is detected while a long-time "eye closure" type eye movement is detected in an EOG-based test (for example, a sleep state test), It is possible to presume that the subject is in REM sleep.

(17) After a long period of "eye closure" type eye movement is detected in an EOG-based test (for example, a test of awakening from sleep) (presumed to be in non-REM sleep or REM sleep). When "blinking" type eye movements are detected, it is considered that the detection of the "blinking" type eye movements is presumed to have awakened from sleep.

<Emotion (mental and physical state) estimation / recording points for theater 3D glasses using electro-oculography sensing technology>
* The main detection target is blinking.

A set of EOG electrodes (at least one ADC is used) is attached to the user's face at least in the vertical direction (for example, up and down on one side of the left and right nose pads).

Based on the detection results from one or more sets of EOG electrodes attached to the user's face, the user's stress, concentration, drowsiness, sleep state (REM sleep / non-REM sleep), and the like can be estimated.

* By adding the following to this, it will be possible to collect the reaction of the audience to the screen image and the gimmick of the 4D theater. That is, a minimum of one set (two sets to improve accuracy) of EOG electrodes (a total of two or more ADCs are used) is additionally attached to the face of the user (individual spectator) in the left-right direction. (For example, a total of 3 or 4 EOG electrodes are provided above and below one side of the left and right nose pads and above or below the other side, or above and below each of the left and right nose pads.)
With the above additional configuration, the rotation direction of the eyeball including the left-right direction (eye movement or eyeball rotation in an arbitrary direction) can be estimated.

FIG. 5 is a flowchart illustrating an example of the mental and physical state estimation method according to the first embodiment. Although 3D video playback has not become widespread in ordinary households, 3D screening in movie theaters is continuing. FIG. 5 aggregates and records the degree of concentration, fatigue, stress, etc. of the audience in a 3D movie using 3D glasses (or a 4D movie in which effects on five senses other than audiovisual effects such as vibration are added to a 3D movie using 3D glasses). An example of the processing to be performed is shown. Further, FIG. 5 aggregates unspecified majority spectator data for works screened over a certain period (not only 3D / 4D works but also 2D works), and determines the general evaluation of a large number of spectators. It also exemplifies the process of feeding back to the next production of the work.

Now, there is a 3D / 4D compatible movie theater equipped with a projection room (or office) not shown with a local server 1000 as shown in FIG. 3, and a certain 3D (or 4D) work is screened in this movie theater. Imagine a case. Before the screening of the 3D work begins, the movie theater staff distributes the 3D glasses 100 having the eyewear form as shown in FIG. 1 to the visitors (audience) (ST10).

The 3D glasses 100 to be distributed are color-coded for men and women, and are sized for adults and children. That is, there are a total of four types of the 3D glasses 100, two types for adult men and women and two types for children men and women. These four types can be classified by, for example, the first bit of the device ID 11g in FIG. It is possible to classify 4 types with 2 bits and 16 types with 4 bits). In addition, the identification of each of the 3D glasses 100 (identification for the number of distributed people) can be classified by the subsequent bits of the device ID 11g (12 bits can be used for 4096 people, and 14 bits can be used for 16384 people). The number of bits of the device ID 11g may be appropriately determined according to the number of classified classifications of the glasses 100 to be distributed and the maximum number of glasses 100 to be distributed.

In the glasses 100 as shown in FIG. 1, the subtitles can be AR-displayed on / off in the images IM1 and / or IM2. At that time, if the device ID indicates that it is for adults, the subtitles that are displayed by default can be set to a smaller "Japanese or original language mixed with katakana (English, etc.)", and if the device ID indicates that it is for children, it is the default. The displayed subtitles can be large "Japanese only in hiragana or katakana".

After the spectators wearing the glasses 100 are seated in the seats of the movie theater, when the screening start time comes, the lights in the hall are turned off and the screening starts. At the same time as the start of the screening, a signal to start the screening is sent from the local server 1000 in FIG. 3 to each of the glasses 100. When this signal is received by the communication processing unit 11d, the timer (or frame counter) 11f of each eyeglass 100 starts automatically (that is, even if the eyeglass user does nothing). Then, every time an event corresponding to the eye movement (blink, etc.) of the eyeglass user (individual spectator) is detected by the eye movement detection unit 15, or the voice or body movement of the eyeglass user (laughter or reflection when surprised). Every time an event corresponding to the event is detected by the microphone 16 or the sensor unit 15, the detection data obtained by adding the information bit of the time stamp (or frame count value) to the information bit indicating the event is generated by each eyeglass user. The EOG data is continuously recorded in the memory 11b (ST12). When the storage capacity of the memory 11b is sufficiently large, the raw data of the AD-converted EOG may be recorded in the memory 11b as the EOG data of each eyeglass user.

The detection data includes an information bit indicating the type of detected event (type of eye movement, type of voice, type of body movement, etc.) and information of a time stamp (or frame count value) in the information bit of the device ID 11g. It is also possible to add a bit. Although it depends on the application, it is considered that 80 to 160 bits (10 to 20 bytes) of the detected data can be sufficiently practical.

The display format of the video that causes the above-mentioned event is not limited to 3D, and may be 2D or 4D (4D is equivalent to 3D in terms of the video portion alone). In addition, the audio / acoustic effect that can cause an event (vibration and other sensational effects are added in 4D) may be monaural, front 2-3 channel stereo, or front / rear surround stereo of 4 channels or more. These audio / sound effects may be appropriately switched depending on the scene of the movie. What kind of audio / acoustic effect was produced can be specified from the time stamp (or frame count value) obtained from the timer (or frame counter) 11f.

The frame count value indicates each video frame. For example, when one or several subliminal images are inserted in a continuous frame of a moving image, the subliminal image frame can be specified by the frame count value. .. Further, if the frame rate is, for example, 60 frames per second, the time stamp corresponding to the frame count value can be calculated by dividing the frame count value by 60. Therefore, the frame counter can have a function similar to that of a timer.

The data recorded in the memory 11b, including its own device ID 11g, is automatically transferred from the communication processing unit 11d to the local server 1000 at a predetermined timing (for example, every 5 to 10 minutes) using Bluetooth or the like. Can be done (ST14). The local server 1000 continuously records the transferred data (subdivided) in the digital recorder 1002 including the source device ID 11g, and / or accumulates the transferred data in the local database 1002. Even if the data of various eyeglass users are discontinuously divided and recorded on the recording area of the recorder / local database 1002, the data of individual eyeglass users can be selected and collectively retrieved by using the device ID 11g as the selection key. Can be done.

The data recording to the memory 11b (ST12) and the subdivision transfer of the recorded data to the local server 1000 (ST14) are repeated until the screening work is completed (ST16 no).

When the screening work ends and the end telop starts (ST16 yes), the untransferred data in the memory 11b is transferred to the local server 1000. As a result, the data of all the glasses (all user glasses) distributed to the spectators is collected by the local server 1000, and the recording to the recorder / local database 1002 is completed (ST18). If the amount of data becomes too large when the data of all the glasses users is collected, a certain number of device IDs may be randomly determined and only the user data of the determined device IDs may be collected (or, for example, an odd number). Alternatively, the amount of data can be halved by collecting only the user data of the device ID equivalent to an even number).

When the collection of necessary user data is completed and the recording is completed, the 3D glasses 100 distributed by the staff are collected from the leaving user (audience) (ST20). (Or have the used 3D glasses put in the collection box.) The communication function of the collected glasses 100 is immediately turned off. Then, since the communication circuit between the uncollected glasses 100 and the local server 1000 is alive, when the glasses 100 are taken out, the staff at the exit of the theater uses the beacon sensor 11e or the like to determine the owner of the glasses 100. It can be specified and the loss of the glasses 100 can be prevented.

When the audience is replaced and re-screened (ST22 yes), the processes of ST10 to ST20 are repeated and data from a new audience is collected.

When all the screenings on that day are over (ST22 no), the collected data will be aggregated and the records will be organized (ST24). For example, based on the collected data, the characteristic EOG waveform change (for example, equivalent to the event corresponding to the blink of an eye) of one or more users (audience) and the time stamp (or frame number) at the time of the change are The correspondence between the screening contents (scenes, sound effects, 4D effects such as vibration, etc.) and the characteristics of the user (gender / age group, etc.) is summarized in a spreadsheet format. Then, for example, adult women often scream surprised at a certain scene and sound effect, but men often scream, but the frequency is low for men, so the degree of audience reaction (mental and physical condition) for each scene and sound effect of the screening work is estimated. become able to.

The data aggregated by the local server 1000 in one or more movie theaters for a certain period (for example, one month) can be continuously sent to the main server 10000 via a communication line such as the Internet (ST26).

For example, if the detection data is composed of 10 bytes and 2 bytes of them are used for the device ID, 2 bytes can be assigned to the type of detection event and 6 bytes can be assigned to the time stamp (or frame count value) or the like. .. Assuming that an average of 1000 events are detected per person in one movie screening, 10 Mbytes of information will be collected from 1000 spectators in one movie screening. If there are 100 screenings in a month, the amount of detected data will be 1 GB per movie theater per month. When similar data from 100 movie theaters are collected on the main server 10000 in FIG. 3, the total amount of data is 100 GB, and the annual unit is 1 Tbyte or more of big data.

This big data gives hints for selecting the genre of the next movie work, taking into consideration the social situation of the data collection party. For example, in a social situation where suicide bombing occurs frequently, if it can be inferred from the collection of EOG data that the audience is highly concentrated on the battle scene with terrorists in a certain 3D movie, a 3D movie of a hero who confronts terrorism It will be possible to plan a movie production company to produce next.

<Summary of the first embodiment (FIG. 5)>
3D glasses for 3D screening in movie theaters (either shutter method or polarization method) are combined with concentration / fatigue / stress estimation using electro-oculography sensing. This makes it possible to aggregate the reaction of the audience to the movie currently being shown (concentration, fatigue, stress for each scene), re-editing the main part according to the screening time, and further, future new movies. The voice of the customer (VOC) can be fed back to the production of the movie.

By detecting, estimating, and recording (along the time axis) the audience's reaction to the work being screened with each pair of glasses, it is possible to collect the reaction results for everyone in the movie theater.

FIG. 6 is a flowchart illustrating an example of the mental and physical state estimation method according to the second embodiment. Here, the AR eyewear (AR glasses) 100 is provided with an electro-oculography sensing function so that blinks and the like can be detected, and a test image (training video) is provided through the AR glasses (or on the AR glasses). While displaying, the degree of concentration, fatigue, and stress are estimated based on "blinks" and the like.

First, a subject (user of eyewear) wears AR eyewear (glasses type or goggles type) 100 with an EOG electrode (ST30). EOG data of the subject (long-distance bus driver, elderly car driver, etc.) by starting the timer (or frame counter) 11f at the same time as the screening of the test video (drive simulator video, etc. for driver aptitude test) starts. Continuous recording (for example, event information corresponding to the EOG waveform of FIG. 4) is started using the memory 11b (ST32). The test video may be computer graphics (CG), live-action video, or animation. The image display method may be any of 2D, 3D, and 4D. The audio (vibration, etc. is added in 4D) may be stereo, monaural, or surround. The video display method and the audio presentation method can be appropriately switched according to the test content.

The test video incorporates video content (various stress videos) related to the EOG signal waveform pattern for estimating the concentration level, fatigue level, stress, etc. of the subject at a predetermined check location. For example, images are incorporated that make the lane suddenly turn, suddenly a ball or animal pops out from the side, or a monotonous and sleepy landscape continues. The EOG data (data corresponding to an event such as blinking) obtained from the eyewear 100 attached by the subject at such a predetermined check location is temporarily recorded in the memory 11b together with the time stamp or the frame number. To. The contents of the data recorded in the memory 11b are transferred to the local server 1000 at a predetermined timing (for example, division of scenes in the test video) and recorded in the recorder / local database 1002 (ST34).

After recording the subject's data using test footage including various stress footage (ST36 yes), the data recorded in the recorder / local database 1002 is evaluated (ST38). In the evaluation of ST38, it is checked whether the EOG data obtained in this test is taken well. (Usually, if there is any doubt that the blink is not recognized in the video part where there is a blink reaction, recheck (ST40 yes), otherwise proceed to the next process.)
If the subject's EOG data is well obtained in this test (ST40 no), the subject's characteristic EOG waveform change (corresponding to blinking, eye closure, etc.) and the time when the change occurred Correspondence between the content of the test video (scene, sound effect, 4D effect such as vibration and temperature, etc.) indicated by the stamp (or frame number) and the characteristics (gender / age / past history, etc.) of the subject (glasses user) The relationship is aggregated for the subject on the local server 1000, and the aggregation result is recorded in the recorder / local database 1002 (ST42).

The recorder / local database 1002 contains past comparative EOG data for the same test footage when the same subject (such as a long-distance bus driver or an elderly car driver) was healthy, or a healthy average in the same industry. Comparative EOG data of a person for the same test video has already been recorded. The subject's EOG data obtained in this study will be compared with the already recorded comparative EOG data. The parts of both data to be compared are the parts with the same time stamp (or the same frame count value). As a result of comparison, from the test data of this time, doze (Pf, etc. in Fig. 4) was observed in the monotonous driving image, and there was no blinking reaction (Pc, Ps, etc. in Fig. 4) at the unexpected pop-out image part. When some parts are found, it is judged whether the subject (long-distance bus driver, elderly car driver, etc.) can normally perform future work (long-term bus driving, etc.) according to the occurrence situation. (ST44).

The data aggregated by the local server 1000 over a certain period of time (for example, one year) is continuously sent to the main server 10000 via the Internet or the like. The aggregated data continuously sent from the local servers 1000 here and there is accumulated in the database of the main server and becomes big data (ST46). By analyzing this big data, it is possible to estimate what kind of EOG data can be obtained from the test video of ST32, which is normal for a healthy person. Findings from such estimates are useful in determining the employment aptitude of tired workers and the elderly.

By creating a test video with appropriate content, the test in Fig. 6 can also be used to estimate the degree of concentration, fatigue, stress, etc. in work in industries other than automobile driving (infrastructure maintenance work, warehouse picking work, etc.). It is available. For example, it is possible to estimate in real time that a decrease in concentration, an increase in stress, or a neurological abnormality has occurred during work / training. Phenomena in which the number of "blinks" increases are considered to be psychogenic such as anxiety, tension, mental conflict, and stress, and cases of nerve abnormalities. None of the conditions are favorable in the work. By detecting / recording the "blink", it is possible to consider and take measures such as creating an environment for reducing the burden on the worker and improving the work procedure.

In addition, the degree of comprehension can be estimated in real time from the blink generation pattern during training. When listening to the explanation, it is well understood if you blink at the break of meaning, and if you blink at a place that is not a break, you can presume that you cannot concentrate on listening to the explanation.

By detecting / recording "blinks", it is possible to evaluate the training effect and improve the training content.

Inemuri can be estimated in real time from the occurrence patterns of "blinking eyes" and "eye closing" during work.

There is a clear difference in the absolute value of the electrooculogram in the vertical direction between the case where the line of sight is directed directly upward and the case where the eyes are intentionally closed for a certain period of time or longer (when the line of sight is directed directly upward << Closed eyes for a certain period of time, for example, 0.5 seconds or longer).

Since the eyes are not closed for more than a certain period of time (several seconds) on a daily basis, such movements occur during work (for example, while driving a car) by detecting "blinks" and "eye closures". In that case, it is possible not only to record the occurrence but also to issue an alert to the worker.

In addition, by providing the AR eyewear 100 with an acceleration sensor / gyro sensor (11e in FIG. 3), the detection accuracy of the body movements of the user wearing the AR eyewear 100 (such as the head squeezing due to falling asleep) can be detected (eyes). This can be improved (compared to the case where only the motion detection unit 15 is used).

FIG. 7 is a flowchart illustrating an example of the mental and physical state estimation method according to the third embodiment. Here, the degree of concentration (whether the line of sight is moving), the degree of fatigue, stress, etc. of the subject at the time of the glaucoma visual field test are aggregated and recorded. First, the facial skin around the eyes of the subject is brought into contact with the EOG electrode of the eye frame of a visual field test device (not shown) (ST50).

Next, the timer 11f that provides the time stamp is started, and a bright spot (not shown) for visual field test is generated in front of the right eye (or left eye) of the subject by a predetermined program procedure. When the subject sees the bright spot, he presses the inspection button (not shown) to notify the device that the bright spot is visible. The event that the inspection button is pressed is recorded in the memory 11b as detection data together with the time stamp when the button is pressed and the EOG data (FIG. 4) at that time (ST52).

When the program for generating bright spots is completed (ST54 yes), the reliability of the detection data recorded in the memory 11b is evaluated (ST56). In this evaluation, if the EOG data when the button is pressed is a waveform corresponding to "Kyorotsuki" (Pr, Pt, etc.) or a waveform corresponding to "eye closure" (Pf, etc.), the button is erroneously operated. Is presumed to have been pressed. If the laboratory technician or ophthalmologist determines that "there are too many button errors", the examination will be re-examined (ST58 yes).

When the examination engineer or the ophthalmologist determines that "the frequency of button erroneous operation is low enough to cause no problem in diagnosis", the examination result in the memory 11b is transferred to the local server (personal computer) 1000. The transferred test results, including the subject's past test results and the physician's diagnostic opinion, are recorded in the recorder / local database 1002 as part of the subject's chart (ST60).

The processing of ST50 to ST60 may be applied to only one eye (right eye or left eye) of the subject, or to both eyes of the subject.

FIG. 8 is a flowchart illustrating an example of the mental and physical state estimation method according to the fourth embodiment. Here, the subject's REM sleep and non-REM sleep are detected from the EOG of the eye movement, and what kind of image the subject was watching when transitioning from REM to non-REM, and / or what kind of voice / music Collect and record information such as whether you were listening. In addition, the subject's REM sleep and non-REM sleep are detected from the EOG of the eye movement, and the subject is forcibly awakened at the timing of transition from REM to non-REM, and information on the dream that was seen before awakening is collected and recorded.

First, a subject (user of eyewear) wears AR eyewear (glasses type, goggles type or eye mask type) 100 with an EOG electrode (ST70). When the eyewear 100 is a glasses type or goggles type, a video image projected on an external monitor (not shown) can be viewed through the AR display screen. When the eyewear 100 is an eye mask type, a plate-shaped image display unit (not shown) using an organic EL or the like is provided inside the eye mask, and the image display unit can be configured to display a video.

The subject wearing the eyewear 100 is placed in an environment that easily induces sleep. As an example of an environment that easily induces sleep, an environment in which a reclining seat is covered with soft sheets in a dimly lit room at an appropriate temperature (about 25 ° C.) and an appropriate humidity (about 50 to 60%) can be considered. Alternatively, it may be an environment in which the kotatsu is laid down in a cold room.

In such an environment that easily induces sleep, playback of the sleep induction video is started. At the same time as the start of video playback, the timer (or frame counter) 11f is started to start continuous recording of the subject's EOG data using the memory 11b (ST72). The image of the sleep-inducing video may be computer graphics (CG), live-action image, or animation. This video is not limited to those intended only for sleep induction, and may include content intended for sleep learning.

The image display method may be any of 2D, 3D, and 4D. The audio (vibration, etc. is added in 4D) may be stereo, monaural, or surround. The video display method and the audio presentation method can be appropriately switched according to the playback content. The content of the sleep-inducing video is selected from the images, sounds, and stories that have been found to have a sleep-inducing effect on a large number of people in past experiments (vibrations and odors are added as appropriate in 4D works). After the subject falls asleep (eyes are closed), the image is no longer seen, so the sound (music, etc.) and 4D effect (vibration, etc.) become significant. The sound after the subject falls asleep and the contents of the 4D effect can be identified by the time stamp (or frame number).

In addition, physiological data (respiratory rate, pulse, blood pressure, etc.) of the subject may be added to the recorded EOG data. The respiratory rate per minute can be detected by, for example, the timer 11f and the microphone 16 (which detects the breath sounds) in FIG. The microphone 16 can also be used to detect the subject's sleeping breath and snoring. Physiological data such as blood pressure can be obtained using, for example, a wristband type small sphygmomanometer (not shown). The accuracy of estimating the physical and mental condition of the subject can be further improved by appropriately including the physiological data in the EOG data.

Detects subject falling asleep (beginning of sleep) from long-eyed meditation (and occurrence of snoring or snoring, or decreased pulse or blood pressure). Generally speaking, when you fall asleep, non-REM sleep first appears, then REM sleep, and then between non-REM sleep (brain is sleeping) and REM sleep (brain is active) before waking up. Appears alternately. REM sleep appears, for example, every 90 minutes and lasts about 20 to 30 minutes. Since there is eye movement during REM sleep (eyes move steadily), REM sleep can be distinguished from non-REM sleep by detecting this eye movement from the EOG waveform. Thereby, REM sleep and non-REM sleep are detected from the presence or absence of eye movement during sleep, and the detection events thereof are recorded in the memory 11b together with the time stamp (ST74). Since the brain is active during REM sleep, I often dream.

When switching from REM sleep to non-REM sleep, there is a change in which eye movements during sleep are not detected. When such a change occurs (ST76 yes), forcibly awakening the subject is likely to remind us of the dream we had during the last REM sleep.

Therefore, when investigating a dream, the subject is forcibly awakened (ST78 yes) and the subject is asked to record the contents of the dream (regardless of the means of recording the dream, voice recording using the microphone 16 is also possible. Handwritten notes are also acceptable). The content of the dream is recorded in the memory 11b together with the time stamp at the time of awakening (ST80). EOG data before awakening is recorded separately (ST72), and the content of the video played during REM sleep immediately before awakening (especially information that appeals to the five senses other than vision: music, sound effects, or multiplication tables Such as simple and rhythmic sounds) can be identified from the time stamp.

To collect data for another dream, the subject is asked to sleep again (ST82 yes), and the processes of ST72 to ST80 are repeated. Otherwise (ST82 no), all the data recorded for each subject (time stamp, EOG data, dream content memo, etc.) is sent to the local server 1000, organized, and then recorded in the recorder / local database 1002 (ST82 no). ST84). In particular, when the content of the dream is a handwritten memo, ask the subject after awakening to rewrite it in a document that is easy to understand (for example, the person who appears in the dream says "Ningashi, Nisangaroku". It is advisable to record the document data in the recorder / local database 1002. The data recorded in the local server 1000 is appropriately sent to the main server 10000 and recorded in that server.

The data recorded on the local server 1000 and / or the main server 10000 can be used for the following investigations, for example. That is, it is possible to investigate how the video content played during sleep is reflected in the dream in REM sleep and non-REM sleep that occur once or more during one sleep time (about 3 to 9 hours) (ST88). ). The findings can be used to develop sleep-learning.

In addition, the sleep induction station time from the start of playback of the sleep induction video to the start of sleep (or until the first REM sleep or non-REM sleep is detected) can be measured using the timer 11f. Various sleep-inducing videos V1, V2, V3, ... Can be shown to a plurality of subjects, and individual sleep-inducing time data in the reproduction of each video V1, V2, V3, ... Can be collected from a plurality of subjects. Analyzing these sleep induction station time data can be useful for producing "videos (movies) that allow you to sleep well / sleep well".

FIG. 9 is a diagram illustrating a mounting example of the EOG electrode in another embodiment. This embodiment can be used, for example, in a visual field test of a glaucoma patient, for staggering, concentration / stress check. In the electrode mounting example of FIG. 9, the electrode arrangement is different from that of FIG. 1 on the cushion surface (the surface in contact with the user's facial skin) of the goggle type eyewear.

That is, for example, a cushion made of silicone is provided on the frame 110 of the goggles, and the required electro-oculography detection electrodes (EOG electrodes 151a, 151b, 152a, 152b) are attached to the illustrated positions of the cushion. The silicone cushion is softly deformed to fit the unevenness of the user's face and provides stable contact of all EOG electrodes with the user's skin surface.

Further, four LEDs (M01, M02, M11, M12) serving as calibration markers are mounted (optionally) on the frame 110 of the goggles. Further, as a guide when the user looks straight ahead, a center marker (optionally) is formed directly in front of both the left and right eyes (positions of P1 and P2). The center marker can be formed, for example, by marking the predetermined positions (positions of P1 and P2) of the transparent plastic plate to be fitted in the frame 110. When the center marker is exposed to the side light from the LED attached to the surrounding frame 110, a part of the light is diffusely reflected at the center marker portion to become a bright spot that can be visually recognized by the user.

When a display using a film liquid crystal or the like is provided on the transparent plate in the frame 110, AR display or VR display is possible by the display. (AR is an abbreviation for Augmented Reality, which refers to a technology that adds various information to the real world that can be seen through goggles. VR is an abbreviation for Virtual Reality, which is separated from the real world. Arbitrary information can be displayed. The brightness pattern display for visual field inspection can also be regarded as a kind of VR display.) Markers (M01, M02, M11, M12, P1, P1) can be displayed using this AR display (or VR display). P2) can also be presented to the user.

In addition, 12L and 12R are made with an organic EL panel. If the entire front surface of the eye frame is covered with a light-shielding sheet, it becomes an eye mask with a video display (or VR display) function.

The structure of FIG. 9 can be modified to provide an EOG electrode on a hat with a chin strap (a head mount device such as a conductor cap). Although not shown, the conductor cap with a chin strap that is often worn by train conductors (users) in uniform has a forehead brim that contacts the user's forehead and a chin strap that secures the worn hat to the user's head. attached. Although not shown, the EOG electrodes 151a and 152a of FIG. 9 can be provided inside the forehead collar. Although not shown, the EOG electrode 151b of FIG. 9 is slidably attached to the right cheek-side portion of the chin strap, and the EOG electrode 12b of FIG. 9 is slidably attached to the left cheek-side portion of the chin strap. The user of the conductor cap adjusts the length of the chin strap after wearing the conductor cap to fix the hat so that the hat does not come off even if the body is moved or the wind is received. After this fixation, the user slides the EOG electrode 151b attached to the right side of the chin strap near the right ear so that the center of the right eye (P2 in FIG. 9) is on the line connecting the electrodes 151a and 151g. Similarly, the user slides the EOG electrode 152b attached to the left side of the chin strap near the left ear so that the center of the left eye (P1 in FIG. 9) is on the line connecting the electrodes 152a and 152g.

In this way, the electro-oculography of the left and right eyeballs of the user is detected by the EOG electrodes 151a, 151b, 152a, 152b provided on the forehead collar and the chin strap of the conductor cap.

To summarize the above, when applying the EOG electrode to the conductor cap,
-Place the upper electrodes (151a and 152a) inside the forehead of the conductor cap;
-Place electrodes (151b and 152b) that can move in the vertical direction on the masseter muscle of the parotid gland of the jaw cord. Here, the point is that the electrodes on the jaw strap side are movable in the vertical direction. Align the electrodes on the chin strap so that the line connecting the upper and lower electrodes on the left and right passes near the center of each eyeball on the left and right. As a result, it is possible to obtain an EOG detection signal having a sufficient amplitude for detecting the blink of an eye or the line of sight.

FIG. 10 is a diagram illustrating an example of mounting an EOG electrode in eyeglass-type eyewear according to another embodiment (an example of AR glasses in which the EOG electrode is arranged around the eyeball). The eyewear 100 of FIG. 10 is similar to the eyewear 100 of FIG. 9 in terms of the EOG electrode arrangement, but differs from the eyewear 100 of FIG. 1 in the following points.

That is, the EOG electrodes 151a and 151b of the right nose pad 150R are moved to the right eye frame 101 side, and the EOG electrodes 152a and 152b of the left nose pad 150L are moved to the left eye frame 102 side. The EOG electrodes 151a and 151b are arranged at positions that are substantially point-symmetrical with respect to the user's right eye center position (not shown), and the EOG electrodes 152a and 152b are arranged with respect to the user's left eye center position (not shown). It is placed at a position that is approximately point-symmetrical. Further, the right diagonal line connecting the EOG electrodes 151a and 151b and the left diagonal line connecting the EOG electrodes 152a and 152b are substantially axisymmetric with respect to the vertical line (not shown) along the user's nose. The electrodes 151a, 151b, 152a, 152b can be composed of a conductive member (metal, conductive polymer, etc.) provided at the tip of an elastic body (sponge, silicone cushion, etc.). Each EOG electrode is lightly pressed against the skin surface of the user's face wearing the eyewear 100 by the elastic rebound force of the elastic body.

When the EOG electrode arrangement structure as shown in FIG. 10 is adopted, it becomes easier to detect the electric field generated around the charged eyeball as compared with the structure in which the EOG electrode is arranged on the nose pad portion. Therefore, the EOG signal having a larger amplitude can be detected in the embodiment shown in FIG. 10 than in the embodiment shown in FIG.

FIG. 11 is a diagram for explaining an implementation example of the EOG electrode in the goggle type eyewear (type in which the eye frames of the left and right eyes are continuous) 100 according to still another embodiment (AR glasses in which the EOG electrode is arranged around the eyeball). Another example). When the structure shown in FIG. 11 is adopted, the skin contact stability of the EOG electrode is higher than that of the structure shown in FIG. Therefore, the EOG signal can be detected with higher accuracy. Further, in the structure of FIG. 11, it is easy to incorporate the enlarged information processing unit 11 and other devices. Therefore, in the structure of FIG. 11, the function of the GPS-equipped smartphone can be incorporated without worrying about the design. In that case, the screen display can be performed by AR display on the left and right displays 12L / 12R using a film liquid crystal or the like. Although not shown, small speakers can be attached near the left and right ears of the goggle frame, and small microphones can be attached near the nose cushion. In addition, command input to the smartphone can be performed by eye movements such as eye movement, blinking, eye closing, and winking. The goggle type eyewear 100 as shown in FIG. 11 can be used for checking the concentration and stress of skiers and snowboarders. Further, the goggle type eyewear 100 can support not only AR display but also VR display.

FIG. 12 is a diagram illustrating an implementation example of the EOG electrode in the goggle type eyewear (type in which the eye cups of the left and right eyes are separated) 100 according to still another embodiment (AR glasses in which the EOG electrode is arranged around the eyeball). Yet another example). In the embodiment of FIG. 12, the EOG electrode arrangement structure and the arrangement structure of the sensor unit 11e are adopted as in the case of FIG. However, the goggle structure of FIG. 12 can withstand underwater diving use (or skydiving use). The goggles of FIG. 12 can be used for concentration / stress check during training of marine divers and skydivers.

The application target of the VR display function is not limited to goggle-type eyewear as protective glasses. For example, a VR display function can be added to the goggles 100 as shown in FIG. With this VR goggles 100, training videos of underwater diving and skydiving (or games) can be shown to the user. (This training video can include live-action footage, computer graphics, or animation of training content that is difficult for beginners to handle.)
What kind of eye motion reaction the user showed in a specific scene of the training video can be detected by using the EOG electrodes (two or more of 151a, 151b, 152a, 152b) of the VR goggles 100. Further, the specific scene when the eye movement reaction is detected can be specified by the time stamp (or frame number) from the timer / frame counter 11f of FIG. The detection result of the eye movement reaction is temporarily stored in the memory 11b of FIG. 3 as detection data together with the time stamp (or frame number). The temporarily stored detection data can be taken into the local server (personal computer) 1000 after the training video ends.

The trainer (training instructor) can analyze how the subject (VR goggles user) was concentrated in the video scene, which is a checkpoint, from the occurrence pattern of blinks and their changes. A personal computer of the local server 1000 can be used for this analysis.

As the checkpoint, it is possible to use the video content when a large number of subjects all have similar eye movement reactions (similar eye movement patterns and their change timings). You can check what the video content of the checkpoint was by playing the corresponding scene based on the time stamp (or frame number). For example, in a skydiving training video, the checkpoint is a scene where the parachute rope is entangled in the body and cannot be opened, and how does the subject's blinking appearance at that time compare to the case of many subjects (or experts)? It can be investigated whether it was (1 or 2 flash reactions: calm and focused, or eye closure or multiple flash reactions: panicked, etc.).

For blinks detected when the training video scene is a monotonous aerial scene (such as blinks that the subject unconsciously makes), the timing of occurrence varies from subject to subject, so such scenes can be viewed from the checkpoint. Can be removed.

The use of VR goggles 100 as described above may provide the same effect as the application of 3D glasses for movie theaters.

Applications other than the above can be considered for the AR / VR eyewear of the present application. For example, the camera 13L / 13R is incorporated in the eyewear 100 of FIG. 1 or FIG. During live-action recording using this camera, various eye movement reactions (blink generation pattern and its change timing) can be recorded in the memory 11b together with a time stamp (or frame number) (in this case, the memory 11b). It is desirable to use a flash memory with as large a capacity as possible).

A specific scene of this live-action recording (for example, a person suddenly hits from behind while waiting for a train at the platform of a station) and the eye movement reaction that occurred at that time (for example, a short eye close and many blinks). Can record together with a time stamp (or frame number) and transfer the recorded contents to the local server 1000 as appropriate. The data transferred to the local server 1000 can be useful for analyzing what kind of tension / stress has occurred depending on the situation in which the user of the eyewear 100 is placed.

<Summary of embodiments>
(A) For example, at least one set of electrodes can be arranged above and below the nose pad on either the left or right side, and electro-oculography sensing can be performed using at least one ADC. As a result, the power consumption of 3D glasses is low (the power required for electro-oculography sensing using ADC is equivalent to that of an electrocardiograph that operates wirelessly), and the cost is low (feature calculation for "blink" detection is performed by several MIPS. By performing at about several tens of MIPS, it is possible to detect "instantaneous eyes" (instantaneous) and "eye closure" (lasting for a certain period of time) with an electrocardiograph operating wirelessly (the cost is about the same).

(B) By the technology of the embodiment, the electro-oculography sensing can be applied to 3D glasses for 3D screening in a movie theater (both shutter method and polarization method can be supported), and the degree of concentration, fatigue, and stress estimation are combined with the application. Can be done. This makes it possible to aggregate the reaction of the audience to the movie currently being shown (concentration, fatigue, stress along the time axis or for each scene), and re-editing the main part according to the screening time. Furthermore, it can be fed back as a VOC (Voice of the Customer) to the production of a new movie in the future.

(C) Furthermore, by adding an acceleration sensor or a gyro sensor to the 3D glasses, it is possible to expand the range of VOC collection in an experience-based / attraction-type theater called 4D. (In the experience-based / attraction-type theater, in addition to moving the seat, you can experience Air, Water mist, scent, bubbles, fog, wind, flash, snow, and rain. As such experience, 4DX, There is MediaMation MX4D of TOHO CINEMAS etc.)
Here, in the electro-oculography sensing, it is possible to estimate the line-of-sight direction (2 to 3 ADCs) from the blinking eyes and eye closure (minimum 1 ADC), and an acceleration sensor or a gyro sensor is added to this. This makes it possible to collect VOC information, including the direction in which the 3D glasses face (the direction in which the head of the eyeglass user faces).

(D) In addition, there are voices saying that subtitles are difficult to read with theater 3D glasses (subtitles are also converted to 3D and displayed in the foreground, so it gets tired if you move your eyes back and forth with other images), so wear AR glasses. It is also effective to display subtitles including congestion angle control as a base (effectiveness of AR subtitle display including congestion angle control by estimating stress using electro-oculography sensing while using a known method. Can be confirmed).

<Example of correspondence between the content of the initial claim and the embodiment>
[1] The mental and physical state estimation device according to the embodiment adopts the device form of the eyewear (100), and the eye movement of the user based on the electro-oculography (EOG of FIG. 4) of the user wearing the eyewear. It is provided with an eye movement detection unit (15) for detecting the eye movement, a timer (11f) for measuring the timing when the eye movement is detected, and an information processing unit (11) for performing information processing based on the eye movement. .. In this mental and physical state estimation device, an external stimulus (for example, a stimulus to the five senses in various entertainment scenes; a stimulus to the five senses in various stress test images; a visual and mental tension in a visual field test) is given to the user wearing the eyewear. Stimulation of the user; characteristic eye movements of the user when given (stimulation of consciousness in various dreams) (for example, blink Pc / Ps in FIG. 4, eye closure Pf, eyeball up / down / left / right movement or rotation Pr / The EOG) corresponding to Pt or the like is detected by the eye movement detection unit (15).

Further, the timing when the characteristic eye movement is detected is measured by the timer (11f), and the content of the external stimulus at the measurement timing and the content of the characteristic eye movement at the measurement timing (the eye). It is configured so that the mental and physical state of the user (related to individual evaluation for external stimuli) can be estimated from the correspondence with the EOG waveform corresponding to the motion.

[2] The device of the above [1] has a memory for temporarily storing the information collected by each eyewear. That is, this device stores information (output of a timer or frame counter) that identifies the content of the external stimulus (various movie scenes, etc.) and information on the characteristic eye movement (blink, etc.) (memory that stores the information. 11b) can be further provided.

[3] The device of the above [1] is configured so that an external local server and / or a main server can collect information. That is, this device mutually corresponds information that identifies the content of the external stimulus (various movie scenes, etc.) (output of a timer or frame counter) and information on the characteristic eye movement (blink, etc.). Means (11a to 11d) for providing the attached survey information to an external server (1000 and / or 10000) can be further provided.

[4] The eyewear (100) in which the device of the above [1] is incorporated is configured so that the source of the survey information can be identified. That is, this device has a device ID (11 g) that can identify itself, and the device ID can identify the eyewear that is the individual information provider.

[5] The mental and physical state estimation method according to the embodiment detects the eye movement of the user based on the electro-oculography (EOG of FIG. 4) of the user wearing the eyewear (100), and the eye movement is detected. A configuration (FIG. 3) is used in which the timing at which the information is processed is measured and information processing is performed based on the eye movement. In this method of estimating the physical and mental state, an external stimulus (for example, a stimulus to the five senses in various entertainment scenes; a stimulus to the five senses in various stress test images; a visual and mental tension in a visual field test) is given to the user wearing the eyewear. (Stimulation of consciousness in various dreams), and the characteristic eye movements of the user when an external stimulus is given (for example, blinking Pc / Ps in FIG. 4, eye closing Pf, eyeball up / down / left / right). EOG) corresponding to motion or rotation Pr / Pt or the like is detected (ST12; ST32: ST52; T72). In addition, the timing when the characteristic eye movement is detected is measured (ST12; ST32; ST52; ST72). Then, from the correspondence relationship between the content of the external stimulus at the measurement timing and the content of the characteristic eye movement at the measurement timing (EOG waveform corresponding to the eye movement), the physical and mental state of the user (individual to the external stimulus). (Relevant to evaluation) is estimated (ST24; ST42-ST44; ST56; ST74-ST88).

[6] The method of [5] above can be used, for example, to investigate the reaction of an audience to a 3D / 4D screening work in a movie theater. In this method, if the detected characteristic eye movement is less than a predetermined number of times (1-2 times) of blinks (Pc in FIG. 4), the physical and mental state of the user is the external stimulus at the measurement timing. It is presumed that the state is concentrated on the content (exciting scenes of the movie, etc.) (ST24 in FIG. 5). Alternatively, if the detected characteristic eye movements are continuous (three times or more) blinks (Ps in FIG. 4) a predetermined number of times or more, the mental and physical state of the user is the content of the external stimulus at the measurement timing (Ps in FIG. 4). It is presumed that he is in a state of being stressed by (such as a scary scene in a movie) (ST24).

[7] The method of [5] can be used, for example, for investigating the degree of fatigue of a long-distance bus driver. In this method, the detected characteristic eye movement corresponds to eye closure for less than a predetermined time (for example, 3 seconds or more and less than 5 minutes) (after eye closure is detected until the first blink is detected. If the time is less than a predetermined time), it is estimated that the physical and mental state of the user is a tired state or a dozing (nap) state at the measurement timing (ST42 in FIG. 6). Alternatively, the detected characteristic eye movement corresponds to eye closure for a predetermined time or longer (for example, 5 minutes or longer) (the time interval from the detection of eye closure to the detection of the first blink is predetermined. If it is (time or more), it is estimated that the physical and mental state of the user is a nap (for example, 15 to 30 minutes of sleep) or a sleep (for example, 90 or more sleep) after the measurement timing (ST42).

In a fatigue survey using this method, when a dozing state is observed during a short-term (for example, 30 minutes or less) survey, or a nap or sleeping state is observed during a long-term (for example, 2 hours or more) survey. Can stop the subject from now on the job of driving a long-distance bus.

[8] The method of [5] can be used, for example, for a visual field survey of a glaucoma patient. In this method, the presence or absence of bright spots or the movement of bright spots is used as an external stimulus for examination of the subject. In this visual field test, if the characteristic eye movement detected is less than a predetermined number of times (1-2 times) of blinks (Pc in FIG. 4), the physical and mental state of the user is the measurement timing (during the visual field test). It is presumed that the state is concentrated on the content of the external stimulus when the test button is pressed (ST56 in FIG. 7). Alternatively, if the detected characteristic eye movement is an electrooculographic waveform (for example, Pr / Pt in FIG. 4) indicating vertical movement or rotation (staggering) of the eyeball, the mental and physical condition of the user is the measurement. It is presumed that the state is stressed by the content of the external stimulus at the timing (ST56).

[9] The method of [5] above can be used, for example, for a sleep survey. In this method, the presence or absence of rapid eye motion is used to detect a change of state during sleep of a subject. In this sleep test, the characteristic eye movements detected correspond to sleep for a specific time or longer (for example, 90 minutes or longer) (long eye closure is detected, and then the first blink is detected by awakening. (The time interval until the sleep is 90 minutes or more), and if an electroocular waveform (for example, Pr / Pt in FIG. 4) indicating rapid eye motion is detected during the sleep period, the physical and mental state of the user is REM sleep. It is presumed to be in a state (ST74 in FIG. 8). Alternatively, the detected characteristic eye movement corresponds to sleep for a specific time or longer (for example, 90 minutes or longer) (from the detection of long eye closure to the detection of the first blink by awakening). (The time interval is 90 minutes or more), and if the electrooculogram indicating rapid eye motion (for example, Pr / Pt in FIG. 4) is not detected during the sleep period, the user's mental and physical state is a non-REM sleep state. Is estimated (ST74).

For example, when the change from REM sleep to non-REM sleep is detected from the EOG waveform of the subject's eye movement, the subject is forcibly awakened. Then, since there is a high possibility that the dream that was dreamed during REM sleep just before awakening can be remembered, the subject is asked to record the content of the dream (voice, text, picture, etc.). Then, from the time stamp at the time of awakening, the content of the video image that was being played immediately before awakening (image and sound before going to sleep, sound after going to sleep) and the content of the dream recorded by the subject Compare. From this comparison, you can check how the content of the video you were playing during REM sleep was reflected in your dreams. This check result can be used for sleep learning.

Although some embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other embodiments, and various omissions, replacements, and changes can be made without departing from the gist of the invention. These embodiments and modifications thereof are included in the scope and gist of the invention, and are also included in the scope of the invention described in the claims and the equivalent scope thereof. It should be noted that it is also within the scope of the invention to combine a part or all of one embodiment of the disclosed embodiments and a part or all of another embodiment of the disclosed embodiments. And included in the abstract.

100 ... Eyewear (goggles type or glasses type); 110 ... Eye frame; 11 ... Information processing unit (processor 11a, non-volatile memory 11b, main memory 11c, communication processing unit (GPS processing), which is the main part of the mental and physical state estimation device. (Unit) 11d, integrated circuit including sensor unit 11e, etc.); 11e ... Sensor unit including acceleration sensor, gyro sensor, etc .; BAT ... Power supply (lithium ion battery, etc.); 12 ... Display unit (right display 12R and left display 12L: Film liquid crystal, etc.); IM1 ... Right display image (tenkey, alphabet, character string, mark, icon, etc.); IM2 ... Left display image (tenkey, alphabet, character string, mark, icon, etc.); 13 ... Camera (right camera 13R) And left camera 13L, or center camera (not shown) attached to the bridge 103 part); 15 ... eye movement detection unit (line-of-sight detection sensor); 16 ... microphone (or vibration pickup); 1510 ... right side (Ch1) AD converter; 1520 ... Left side (Ch2) AD converter; 1512 ... Left and right (Ch0) AD converter; 1514 ... Left and right (Ch3) AD converter.

Claims (15)

A detector that detects specific eye movements based on the user's electro-oculography when a specific external stimulus is given, and
A measuring unit that measures the timing when the detecting unit detects the specific eye movement, and a measuring unit.
Information for estimating the physical and mental state of the user from the correspondence between the content of the specific external stimulus at the detection timing of the specific eye movement measured by the measuring unit and the content of the specific eye movement at the detection timing. Processing unit and
A transmission unit that transmits detection data including information on the content of the specific eye movement detected by the detection unit, information on the detection timing, and device identification information to the server.
Eyewear equipped with. Further provided with a memory for storing the detected data,
The eyewear according to claim 1, wherein the transmission unit transmits the detection data stored in the memory to the server. The eyewear according to claim 1 or 2, wherein the device identification information indicates the type of user who wears the eyewear. The eyewear according to any one of claims 1 to 3, wherein the external stimulus includes stimulation to the five senses and includes contents related to eye movement for estimating the physical and mental state of the user. The eyewear according to any one of claims 1 to 3, wherein the external stimulus includes a test image related to eye movement for estimating a user's mental and physical condition. A data collection system that includes eyewear and a server.
The eyewear
A detector that detects specific eye movements based on the user's electro-oculography when a specific external stimulus is given, and
A measuring unit that measures the timing when the detecting unit detects a specific eye movement,
An information processing unit that estimates the physical and mental state of the user from the correspondence between the content of the external stimulus at the detection timing of the specific eye movement measured by the measurement unit and the content of the specific eye movement at the detection timing. When,
A data collection system including a transmission unit that transmits detection data including information on the content of the specific eye movement detected by the detection unit, information on the detection timing, and device identification information to the server. The server
A plurality of local servers that collect the detection data from the plurality of eyewear, and
A main server that collects the detection data from the plurality of local servers, and
6. The data collection system according to claim 6. Further provided with a memory for storing the detected data,
The data collection system according to claim 6 or 7, wherein the transmission unit transmits the detection data stored in the memory to the server. The data collection system according to claim 6, claim 7, or claim 8, wherein the device identification information indicates the type of user who wears the eyewear. The data collection system according to any one of claims 6 to 9, wherein the external stimulus includes a stimulus to the five senses and includes contents related to eye movement for estimating the physical and mental state of the user. The data collection system according to any one of claims 6 to 9, wherein the external stimulus includes a test image related to eye movement for estimating a user's mental and physical condition. A specific eye movement is detected based on the user's electro-oculography when a specific external stimulus is given, and the mind and body of the user are based on the correspondence between the content of the specific external stimulus and the content of the specific eye movement. It is a data collection method in which the server collects the detection data detected by the eyewear that estimates the state.
The detection data is a data collection method including information on the content of the specific eye movement, information on the detection timing of the specific eye movement, and device identification information. The data collection method according to claim 12, wherein the device identification information indicates the type of user who wears the eyewear. The data collection method according to claim 12 or 13, wherein the external stimulus includes stimulation to the five senses and includes contents related to eye movement for estimating the physical and mental state of the user. The data collection method according to claim 12 or 13, wherein the external stimulus includes a test image related to eye movement for estimating the physical and mental state of the user.

Images