JP6342440B2 - Eye movement detection device, eye movement detection method, and eyewear - Google Patents

Description

  Embodiments described herein relate generally to an eye movement detection device, an eye movement detection method, and eyewear to which an electrooculogram sensing technique is applied.

  Normally, when a person walks while looking forward, the person's head shakes up and down, left and right, but the person can see the surrounding scenery clearly. This is because the movement of the head during walking is suppressed to a range smaller than other parts of the body (amplitude is about 2 to 3 cm regardless of the distance of the line of sight). This is because they move in a coordinated manner so as to maintain (see Non-Patent Document 1). This Non-Patent Document 1 teaches that "the head is shaken up and down and left and right during walking, and the rotation of the head and the compensated rotation of the eyeball are generated as a line-of-sight stabilization operation against this shake".

  Regarding electrooculogram sensing technology, there is a known example 1 of detecting a person's line-of-sight position based on an electrooculogram generated by a person's eye movement (see Patent Document 1). In this known example 1, an EOG (Electro-Oculography) electrode is used for electrooculogram detection (electrooculogram sensing), and a user's line-of-sight position is detected from electrooculogram data acquired from the EOG electrode (paragraph 0025). In the embodiment of the known example 1, the EOG electrode is provided in the goggles (paragraph 0061), but there is also the known example 2 in which the EOG electrode is provided in the glasses (see Patent Document 2). Goggles and glasses are a type of eyewear worn near the human eyeball. When the user wearing the eyewear walks, the eyewear also shakes with the head.

JP 2009-288529 A JP 2013-244370 A

Papers on Head Movements and Eye Movements to Maintain Gaze Stability During Walking 2000-03 Yazaki, Sharp Arrow Journal of Graduate School of Human Sciences, Osaka University. 26 P.177-P.193 http: //ir.library.osaka-u .ac.jp / dspace / bitstream / 11094/5672/1 / hs26-177.pdf

  In a stationary seating state (a state in which no disturbing movement is mixed into the head of the user wearing the eyewear with EOG electrodes), the gaze / eyeball rotation can be stably detected by electrooculogram sensing. However, when looking at a fixed object in the real world in a walking state (a state in which the user's head wearing EOG electrode-equipped eyewear shakes up and down, left and right), the target line of sight / eyeball is compared to the sitting state. It is difficult to detect rotation. The reason is that eyeball rotation is unconsciously performed as a complementary operation for stabilizing the gaze of the human body, and the signal component of this unconscious eyeball rotation is superimposed (as noise or disturbance) on the electrooculogram sensing result. It is.

  That is, when the user wearing the eyewear with EOG electrode walks, the electrooculogram data acquired from the EOG electrode also fluctuates due to shaking of the head accompanying walking even if the user fixes his / her line of sight to a specific object ahead. . The fluctuation of the electrooculogram data becomes noise (disturbance) with respect to the electrooculogram data corresponding to the actual line of sight.

  One of the problems to be solved by the embodiment of the present invention is to suppress a noise component mixed in the electrooculogram data due to the shaking of the head.

  An eye movement detection device according to an embodiment is used in an eyewear (for example, eyeglass type) device form, and includes eye movement (eye line of sight) including eyeball rotation based on an electrooculogram (EOG) of a user wearing the eyewear. An eye movement detector for detecting movement, blinking, etc., a triaxial acceleration sensor for detecting movement of the eyewear, and a triaxial gyro sensor for detecting rotation of the eyewear.

  When the user wearing the eyewear is walking while gazing at a fixed object in the real world, the movement and rotation of the eyewear (that is, the user's head) is detected simultaneously with the detection of the eyeball rotation. And noise that mixes in the electro-oculogram when gazing at the real world fixed object while the user's head is moving (the electro-oculogram added by the compensatory eye rotation that works in the direction to cancel the movement of the head) Component) is canceled (reduced or eliminated) by synthesizing a signal component corresponding to the head movement detected by the acceleration sensor and / or gyro sensor (a component having a phase opposite to the mixed noise).

  Note that the phase of the signal to be combined for noise cancellation (whether the phase is normal or negative with respect to the noise signal component) cannot be determined by the acceleration sensor / gyro sensor alone. The determination can be made from the result of adding and combining the detection results of the acceleration sensor / gyro sensor and the detection result of the electrooculogram. That is, a determination may be made that “the synthesized signal phase with the smaller change in electrooculogram signal amplitude at the walking pitch (or walking cycle) detected by the acceleration sensor / gyro sensor during walking is selected”. it can.

The figure explaining head shaking (position change by up-and-down and right-and-left movement) of a person walking while looking forward. The figure explaining the head shaking (angle change by forward bending / back bending) of the person who is gazing at the fixed object ahead while walking. The figure explaining the relationship between a target distance (distance from an eye to a gaze object) and eye movement when the head has bent back and forth in the vertical direction. The figure which illustrates the phase relationship between the angle change / velocity change of head bending, and the angle change / velocity change of eye movement. The figure explaining the eyeglasses type eyewear with which the eye movement detection device concerning one embodiment was incorporated (example in which an EOG electrode is arranged on a nose pad). The figure explaining the example of mounting of the EOG electrode in the glasses-type eyewear which concerns on one embodiment. The figure explaining the relationship between the information processing part 11 which can be attached to various embodiment, and its peripheral device. The figure which shows an example of an EOG waveform when it begins to walk from the right foot and stops at the right foot of the 13th step while gazing at a fixed object in the real world, for example, 25 meters away. The figure which shows the other example (confirmation of reproducibility) of an EOG waveform at the time of starting on the right foot | foot again on the same conditions as FIG. 8, and stopping with the 13th right foot | foot. When the noise caused by the shaking of the head is canceled, the eye movements from the front to the upper side and the detection signal levels (Ch0, Ch1, Ch2) obtained from the three analog / digital converters (ADC) shown in FIG. And an electrooculogram (EOG) illustrating the relationship between the average level Ch1 + 2) of Ch1 and Ch2. When the noise caused by the motion of the head is canceled, the eye movement from the front to the bottom and the detection signal levels (Ch0, Ch1, Ch2, and Ch1 + 2) obtained from the three ADCs shown in FIG. An electrooculogram illustrating the relationship. When the noise caused by the motion of the head is canceled, the eye movement from left to right and the detection signal levels (Ch0, Ch1, Ch2, and Ch1 + 2) obtained from the three ADCs shown in FIG. An electrooculogram illustrating the relationship. In the case where the noise caused by the motion of the head is canceled, when the line of sight is facing the front, the blinking (both eyes) is repeated five times at intervals of 5 seconds, and the three ADCs shown in FIG. FIG. 6 is an electrooculogram illustrating a relationship with detection signal levels (Ch0, Ch1, and Ch2) obtained from FIG. When the noise caused by the motion of the head is canceled, when the line of sight is facing the front, the eye movement that repeats 1 second of eye meditation (both eyes) and 4 seconds of eye opening (both eyes) 5 times FIG. 7 is an electrooculogram illustrating the relationship between detection signal levels (Ch0, Ch1, and Ch2) obtained from the three ADCs shown in FIG. When the noise caused by the shaking of the head has been canceled, the eye that has repeated the blink of the left eye (the blink of the left eye) five times immediately after the blink of both eyes has been repeated five times when the line of sight is facing the front FIG. 7 is an electrooculogram illustrating the relationship between movement and detection signal levels (Ch0, Ch1, and Ch2) obtained from the three ADCs shown in FIG. When the noise caused by the shaking of the head is canceled, the eye that has repeated the blink of the right eye (the blink of the right eye) 5 times immediately after the blink of both eyes 5 times when the line of sight is facing the front FIG. 7 is an electrooculogram illustrating the relationship between movement and detection signal levels (Ch0, Ch1, and Ch2) obtained from the three ADCs shown in FIG. The flowchart explaining an example of the process (noise cancellation process) which minimizes the noise resulting from the shaking of a head. The figure explaining the mounting example of the EOG electrode in the spectacles type eyewear which concerns on other embodiment (example in which the EOG electrode is arrange | positioned around the eyeball). Furthermore, the figure explaining the example of mounting of the EOG electrode in the goggles type eyewear (type in which the eye frames of both right and left eyes are continuous) according to another embodiment (an other example in which the EOG electrode is arranged around the eyeball). Furthermore, the figure explaining the example of mounting of the EOG electrode in the goggles type eyewear concerning another embodiment (the type which the eyecup of both right and left eyes separated) (the other example in which the EOG electrode was arranged around the eyeball).

  In the following, basic information will be provided first, and then various embodiments will be described with reference to the drawings.

<Basic information>
The diameter of an adult eyeball is about 25 mm. After birth, it is about 17 mm and grows with growth.
The distance between the pupils of an adult male is about 65 mm. (Many commercially available stereo cameras are made at intervals of 65 mm.)
The distance between the pupils of adult women is several mm shorter than that of men.
The electrooculogram is several tens of mV.
The eyeball has a positive potential on the cornea side and a negative potential on the retina side. When this is measured on the surface of the skin, it appears as a potential difference of several hundred μV.

Examples of types of eye movements and eye movement ranges related to eye movement detection include the following:
<Types of eye movement (eye movement)>
(01) Compensatory eye movement Involuntary eye movement developed to stabilize the image of the outside world on the retina regardless of the movement of the head or body.

(02) Voluntary eye movement An eye movement developed to bring the visual image to the center of the retina and can be controlled arbitrarily.

(03) Impact eye movement (saccade)
Eye movement that occurs when the point of sight is changed to see an object (easy to detect).

(04) Sliding eye movement Smooth eye movement that occurs when tracking slowly moving objects (difficult to detect).

<Eyeball movement range (for general adults)>
(11) Horizontal direction Left direction: 50 ° or less Right direction: 50 ° or less (12) Vertical direction Downward direction: 50 ° or less Upward direction: 30 ° or less Narrow (Because there is a “bell phenomenon” in which the eyeball moves upward when the eye is closed, the eye movement range in the vertical direction shifts upward when the eye is closed.)
(13) Others Angle of convergence: 20 ° or less.

<About the movement of the head accompanying walking (Sources of FIGS. 1 and 2 are non-patent documents 1)>
FIG. 1 is a diagram for explaining head fluctuation (position change due to up / down / left / right movement) of a person walking while looking forward. Due to the movement of the lower limbs during walking, the position of the human head changes vertically (FIG. 1A). A typical adult walking cycle (one walking cycle) is, for example, about 0.6 seconds. At the same time, the position of the person's head also changes in the left and right during this walking (FIG. 1B). The cycle of this left-right position change is about 1.2 seconds, which is twice the walking cycle. The vertical position change width and the horizontal position change width are about 2 to 3 cm, respectively. This up / down / left / right movement is substantially linear and regular, and has a cycle closely related to the walking cycle.

  FIG. 2 is a diagram for explaining a head sway (angle change due to forward / backward bending) of a person who is gazing at a fixed object in front during walking. When a person who is gazing at the front visual target (target) starts walking, the head position changes vertically with walking (FIG. 2A). While gazing at the target, when the head is in the upper position, the head bends forward (pitch down), and when the head is in the lower position, the head bends backward (pitch up). (FIG. 2 (b)). Such back and forth bending (vertical rotational movement of the head) occurs in response to (in synchronization with) the vertical movement of the head during one walking cycle. Similarly, when the position of the head changes from side to side (FIG. 1B), a horizontal rotational movement (synchronized with the vertical movement of the head) occurs in the head.

  That is, while walking, the human head performs a rotational motion that compensates for vertical and horizontal movements, and the head moves back and forth at high and low positions so as to help the line of sight to catch the front visual target (target). Bow. The same applies to the left and right. When the body swings from side to side and the head moves to the left and right during walking, the head rotates to the left and right (reciprocating). This rotation of the head is called “Compensatory Rotation” and is thought to help stabilize the line of sight.

  In addition, since the human trunk rotates back and forth and right and left during walking, the head needs to compensate for these. The angular position of the head is kept horizontal by rotating the head in the opposite direction with respect to the change in the trunk.

<Regarding the movement of the eyeball accompanying walking (Sources of FIGS. 3 and 4 are non-patent documents 1)>
FIG. 3 is a diagram for explaining the relationship between the target distance (the distance from the eye to the gaze target) and the eye movement when the head is bent back and forth in the vertical direction. Here, the target position when the head is fixed is HFP, and when the actual target is closer (Near Target) and far (Far Target) than the HFP, the anteroposterior rotation angle (φh) of the head And the compensatory rotation angle (φen, φef) of the eyeball.

As described above, the head rotates at a cost, but it is not possible to achieve line-of-sight stabilization completely by itself. When the head rotates at the expense of vertical movement, it still causes over and under. In order to compensate for the deviation due to the excessive or insufficient compensation rotation, periodic compensation rotation of the eyeball is required during walking. For example, for a near target 30 cm away, the eyeball performs a compensatory rotation (φen) in the same direction as the rotation / rotation (φh) of the head. Thereby, even if the rotation / rotation of the head is insufficient, the eyeball rotates in cooperation with the head in the moving direction to compensate for the insufficient rotation of the head. On the other hand, for example, for a far target (Far Target) 100 m ahead, the eyeball performs a compensatory rotation (φef) in the opposite direction to the rotation / rotation (φh) of the head. As a result, even if the rotation / rotation of the head becomes excessive, the eyeball rotates in the opposite direction to cancel the excessive rotation of the head. (Note that there is no substantial difference in the amount of head movement when walking, whether the target is near or far.)
FIG. 4 is a diagram illustrating the phase relationship between the angle change / velocity change of head anteroposterior bending and the angle change / velocity change of eye movement. The head bending angle (φh) shown in FIG. 3 can be detected by a three-axis gyro (11e in FIG. 7) attached to the eyewear. The eyeball rotation angles (φen, φef) shown in FIG. 3 can be detected by an eye movement detection unit (15 in FIG. 7) using EOG of the eyeball.

  The head vertical movement change (vertical forward / backward bending change) accompanying walking changes in phase with the eyeball rotation when the target is near (Near Target) (FIG. 4A). On the other hand, when the target is far (Far Target), the vertical movement change of the head (vertical forward / backward bending change) changes in opposite phase to the change of the eyeball rotation (FIG. 4B). When the visual target is at the fixed position HFP in FIG. 3, the eyeball is not rotated, and the waveform change of the broken line shown in FIGS. 4 (a) and 4 (b) is minimal (substantially horizontal in the figure) (this minimal point is a boundary). Thus, the phase between the longitudinal anteroposterior bending change and the eye rotation change is reversed). Similar phase inversion is also observed between the speed change of the head vertical movement and the speed change of the eyeball rotation (FIGS. 4C and 4D).

  Now, the signal change that occurs in the EOG due to the above-mentioned “change in the vertical movement of the head accompanying walking” will be referred to as “body motion noise”. The walking action can be detected by an acceleration sensor or a gyro sensor, and the eye movement action can be detected from the EOG. Then, when “walking” and “no eye movement” are detected, if body motion noise does not appear in the EOG, “the eyewear user may invert the body motion noise depending on the distance of the target. It is indicated that the point (corresponding to the position HFP in FIG. 3) is being watched.

  FIG. 5 is a diagram illustrating an eyewear-type eyewear 100 in which an eye movement detection device according to an embodiment is incorporated (an example in which EOG electrodes are arranged on a nose pad). In this embodiment, a right eye frame (right rim) 101 and a left eye frame (left rim) 102 are connected to a bridge 103. The left and right eye frames 102 and 101 and the bridge 103 can be made of, for example, an aluminum alloy or titanium. The left outer side of the left eye frame 102 is connected to the left temple bar 106 via the left hinge 104, and a left modern (left ear pad) 108 is provided at the tip of the left temple bar 106. Similarly, the right outer side of the right eye frame 101 is connected to the right temple bar 107 via the right hinge 105, and a right modern (right ear pad) 109 is provided at the tip of the right temple bar 107.

  An information processing unit 11 (an integrated circuit of several millimeters square) that is a main part of the eye movement detection device is attached to the bridge 103. The information processing unit 11 can be configured by an LSI in which a microcomputer, a memory, a communication processing unit, and the like are integrated (details of the information processing unit 11 will be described later with reference to FIG. 7).

  A small battery (BAT) such as a lithium ion battery is embedded in, for example, the left temple bar 106 (or the right temple bar 107 or the modern 108 or 109), and serves as a power source necessary for the operation of the eyeglass-type eyewear 100. ing.

  A left camera 13L is attached to the end of the left eye frame 102 near the left hinge 104, and a right camera 13R is attached to the end of the right eye frame 101 near the right hinge 105. These cameras can be configured using ultra-compact CCD image sensors.

  These cameras (13L, 13R) may constitute a stereo camera. Alternatively, an infrared camera (13R) and a laser (13L) may be arranged at the positions of these cameras to form a distance sensor using an infrared camera + laser. This distance sensor can also be composed of a small semiconductor microphone (13R) that collects ultrasonic waves and a small piezoelectric speaker (13L) that emits ultrasonic waves.

  In addition, instead of the left and right cameras 13L / 13R, or in addition to the left and right cameras 13L / 13R, an embodiment in which a central camera (not shown) is provided in the bridge 103 can be considered. Conversely, there may be embodiments that are not equipped with cameras at all (these cameras are shown as camera 13 in FIG. 7).

  A left display 12L is fitted in the left eye frame 102, and a right display 12R is fitted in the right eye frame 101. This display is provided on at least one of the left and right eye frames, and can be configured using a film liquid crystal or the like. Specifically, one or both of the left and right displays 12L and 12R can be configured using a film liquid crystal display device that employs a polymer-dispersed liquid crystal (PDLC) that does not use a polarizing plate. 12).

  The transparent left and right displays 12L / 12R using film liquid crystal can be used as means for providing augmented reality (AR) that adds image information such as numbers and letters to the real world seen through glasses.

  A nose pad portion is provided between the left and right eye frames 102 and 101 and below the bridge 103. This nose pad portion is composed of a pair of a left nose pad 150L and a right nose pad 150R. The right nose pad 150R is provided with right nose pad electrodes 151a and 151b, and the left nose pad 150L is provided with left nose pad electrodes 152a and 152b.

  These electrodes 151a, 151b, 152a, and 152b are electrically separated from each other, and are connected to three AD converters (ADCs 1510, 1520, and 1512) through insulated wiring members (not shown). The outputs from these ADCs have different signal waveforms according to the movement of the eye of the user wearing the eyewear 100, and are supplied to the information processing unit 11 as digital data according to the eye movement of the user. The electrodes 151a, 151b, 152a, and 152b are used as line-of-sight detection sensors and are components of the eye movement detection unit 15 in FIG. 7 together with the three AD converters.

  The eyewear 100 is fixed to a user's head (not shown) by left and right nose pads (150L, 150R), left and right temple bars (106, 107), and left and right moderns (108, 109). In this embodiment, only the left and right nose pads (150L, 150R), the left and right temple bars (106, 107), and the left and right modern (108, 109) need to touch the user's head (or face) directly. However, there may be an embodiment in which a part other than the ADC (1510, 1520, 1512) and the user's body touches the user other than those (nose pad, temple bar, modern).

  On the film liquid crystal of the right display 12R in FIG. 5, a right display image IM1 including, for example, a numeric keypad (numbers, operators, Enter, etc.), alphabets, marks of a predetermined shape, and other icon groups can be displayed. Further, the left display image IM2 including, for example, an arbitrary character string, an arbitrarily shaped mark, various icons, and the like can be displayed on the film liquid crystal of the left display 12L (the display contents of the displays 12L and 12R may be anything). The numeric keys and alphabets displayed on the right display 12R (or the left display 12L) can be used when inputting numbers and characters. Character strings, marks, icons, etc. displayed on the right display 12R (or the left display 12L) can be used when searching for specific information items, selecting / determining target items, and alerting the user. .

  The display images IM1 and IM2 can be used as augmented reality (AR) display means for adding information such as numbers, letters and marks to the real world seen through the glasses, and the AR display can be turned on and off as appropriate. The content of the display image IM1 and the display image IM2 may be the same content (IM1 = IM2) or different content (IM1 ≠ IM2) depending on the embodiment. The display image IM1 (or IM2) can be displayed on the right display 12R and / or the left display 12. When the content of the AR display is a 3D image that overlaps the real world (with depth) that can be seen through the glasses (with depth), IM1 and IM2 can be used as separate left and right 3D images.

  Further, when the displays (12R, 12L) are present on the left and right, for example, by adjusting the convergence angle, the images of the left and right display images (IM1, IM2) can be shifted in the opposite direction from side to side. As a result, it can be considered that the burden on the eyes when the object and the AR display that are visible in the real world are alternately viewed is reduced. However, normally, images with the same contents are displayed on the left and right displays (12R, 12L).

  Display control on the displays 12L and 12R can be performed by the information processing unit 11 embedded in the right temple bar 107. (Technology for displaying characters, marks, icons, etc. on the display is well known.) The power supply necessary for the information control unit 11 and other operations can be obtained from the battery BAT embedded in the left temple bar 106.

  Note that there is a possibility that the designer or designer wears a prototype of the eyewear 100 corresponding to the embodiment and feels that the weight balance is bad. If the main cause is the BAT in the left temple bar 106, a “weight (or another same weight battery)” corresponding to the BAT in the left temple bar 106 can be placed in the right temple bar 107.

  FIG. 6 is a diagram illustrating an example of mounting the EOG electrode in the eyeglass-type eyewear 100 according to the embodiment. Right nose pad electrodes 151a and 151b are provided above and below the right nose pad 150R, and left nose pad electrodes 152a and 152b are provided above and below the left nose pad 150L. The outputs of the right nose pad electrodes 151a and 151b are supplied to the ADC 1510, the outputs of the left nose pad electrodes 152a and 152b are supplied to the ADC 1520, and the outputs of the lower electrodes 151b and 152b (or the upper electrodes 151a and 152a) of the left and right nose pads. Is provided to the ADC 1512.

From the ADC 1510, a Ch1 signal that changes in response to the right / left eye movement of the user is obtained. From the ADC 1520, a Ch2 signal that changes in accordance with the left and right eye movements of the user is obtained. The ADC 1512 obtains a Ch0 signal that changes in accordance with the left and right eye movements of the user. The vertical movement of both the left and right eyes can be evaluated by a Ch1 + 2 signal corresponding to the average of the outputs of the ADC 1510 and the ADC 1520. (The relationship between the signal waveform of Ch0, Ch1, Ch2, and Ch1 + 2 and eye movement will be described later.)
FIG. 7 is a diagram for explaining the relationship between the information processing unit 11 that can be attached to various embodiments and its peripheral devices. In the example of FIG. 7, the information processing unit 11 includes a processor 11a, a nonvolatile memory 11b, a main memory 11c, a communication processing unit 11d, a sensor unit 11e, and the like. The processor 11a can be constituted by a microcomputer having a processing capability according to product specifications. Various programs executed by the microcomputer and various parameters used when the programs are executed can be stored in the nonvolatile memory 11b. The main memory 11c provides a work area for executing the program.

  The sensor unit 11e includes a sensor group for detecting the position and / or orientation of the eyewear 100 (or the head of the user wearing the eyewear). Specific examples of these sensor groups include an acceleration sensor that detects movement in three axial directions (xyz directions), a gyro that detects rotation in three axial directions, and a geomagnetic sensor that detects absolute orientation (compass function). There are beacon sensors that receive radio waves, infrared rays, and the like to obtain position information and the like. IBeacon (registered trademark) or Bluetooth (registered trademark) 4.0 can be used to acquire the position information and others.

  LSIs that can be used for the information processing unit 11 have been commercialized. One example is the “TZ1000 Series for Wearable Terminals” by Toshiba Semiconductor & Storage. In this series, the product name “TZ1011MBG” includes CPU (11a, 11c), flash memory (11b), Bluetooth Low Energy (registered trademark) (11d), sensor group (acceleration sensor, gyroscope, geomagnetic sensor) (11e). , 24-bit delta-sigma ADC, I / O (USB etc.).

  The information processing unit 11 is attached to the bridge 103 as an example of a place where the relative positional relationship between the sensor unit 11e such as an acceleration sensor and a gyro sensor and the EOG electrodes (151a, 151b, 152a, 152b) does not collapse.

  What to do with the processor 11a can be commanded from an external server (or personal computer) (not shown) via the communication processing unit 11d. The communication processing unit 11d can use existing communication methods such as ZigBee (registered trademark), Bluetooth (registered trademark), and Wi-Fi (registered trademark). The processing result in the processor 11a can be sent to a server (not shown) or the like via the communication processing unit 11d.

  A display 12 (12L and 12R), a camera 13 (13L and 13R), an eye movement detection unit 15 and the like are connected to the system bus of the information processing unit 11. Each device (11 to 15) in FIG. 7 is powered by the battery BAT.

The communication processing unit 11d may incorporate a well-known GPS (Global Positioning System) function using a mobile phone, a smartphone, or the like (or the GPS mobile phone function may be installed somewhere in the eyewear 100). If the processor 11a executes the Internet map service application using the GPS function, the user who wears the eyewear 100 is currently walking and looking at which direction and in which direction the eyeball compensated rotation during walking. Noise can be detected and suppressed. (In the conventional mobile phone GPS, even if the current position of the user can be displayed on the map screen, it cannot be displayed until the user's line of sight is directed.)
The eye movement detection unit 15 in FIG. 7 includes four eye movement detection electrodes (151a, 151b, 152a, and 152b) that constitute a line-of-sight detection sensor, and three ADCs that extract digital signals corresponding to the eye movements from these electrodes ( 1510, 1520, 1512) and a circuit for outputting the output data from these ADCs to the processor 11a side. The processor 11a can interpret a command corresponding to the type of eye movement from various eye movements of the user (vertical movement, left-right movement, blink, eye-meditation, etc.) and execute the instruction.

  As a specific example of the command corresponding to the type of eye movement, if the eye movement is, for example, eye meditation, an information item at the tip of the line of sight is selected (similar to one click of a computer mouse), and multiple blinks or winks are consecutive. If so, there is a command (similar to double-clicking a computer mouse) to start execution of processing on the selected information item. This command is an example of information input B using the eye movement detection unit 15.

  FIG. 8 is a diagram illustrating an example of an EOG waveform in a case where, for example, a person walks from the right foot and stops at the 13th right foot while gazing at a fixed object in the real world that is 25 meters ahead. Compensated rotation of the eyeball corresponding to the vertical movement of the head accompanying walking appears in the ADC detection signal level change (EOG change) of Ch1 and Ch2. The head vertical movement change synchronized with the EOG change can be detected by a triaxial acceleration sensor of the sensor unit 11e of the information processing unit 11. Further, the compensated rotation of the eyeball corresponding to the left-right rotation of the head accompanying walking is shown in the ADC detection signal level change (EOG change) of Ch0. The left-right rotation change of the head synchronized with the EOG change can be detected by a three-axis gyro sensor included in the sensor unit 11e of the information processing unit 11.

  In the example of FIG. 8, the EOG fluctuation period of Ch1 and Ch2 (about 0.6 seconds) is synchronized with the walking pitch, and the EOG fluctuation period of Ch0 (about 1.2 seconds) is synchronized with twice the walking pitch. ing. Since the EOG fluctuation cycle corresponding to the vertical movement of the head and the left-right rotation accompanying walking is clear, it is a cycle synchronized with the walking pitch (in this example, about 0.6 seconds and twice that) and is caused by walking EOG variation can be detected.

  FIG. 9 is a diagram showing another example of the EOG waveform (confirmation of repetitive reproducibility) when starting to walk again from the right foot and stopping at the 13th right foot under the same conditions as FIG. Also in the example of FIG. 9, the EOG fluctuation period of Ch1 and Ch2 (about 0.6 seconds) is synchronized with the walking pitch, and the EOG fluctuation period of Ch0 (about 1.2 seconds) is synchronized with twice the walking pitch. ing. The EOG fluctuation cycle corresponding to the head vertical movement and left-right rotation accompanying walking is clearly recognized, and it is due to walking with a cycle synchronized with the walking pitch (in this example, about 0.6 seconds and twice that). It can be seen that EOG fluctuations can be detected.

The following features are commonly observed in FIGS. 8 and 9, confirming the reproducibility of the event:
(1) The eyeball rotational movement in the vertical direction can be confirmed in accordance with the walking cycle (about 0.6 seconds) (see the EOG detection signal levels of Ch1 and Ch2).

  (2) The eyeball rotational movement in the left-right direction can be confirmed in accordance with a period twice as long as the walking period (about 1.2 seconds) (see the EOG detection signal level of Ch0).

  (3) There is a place where the amplitude of the EOG detection signal resulting from the movement of the head accompanying walking reaches 200 μV to 400 μV. Since the EOG signal amplitude measured with the head still is usually about 1 mV, 200 μV to 400 μV is a noise that cannot be ignored when detecting any eyeball rotation. This noise can be canceled (reduced or eliminated) as described below.

  That is, the vertical movement of the head accompanying walking can be detected by the acceleration sensor in a cycle synchronized with the walking pitch (in this example, about 0.6 seconds), and the left-right rotation of the head accompanying walking is synchronized with the walking pitch. It can be detected by the gyro sensor at a period (about 1.2 seconds in this example). Then, the EOG fluctuation (considered as noise) caused by the vertical movement (swing) of the head accompanying walking can be canceled (reduced or eliminated) by the detection result of the acceleration sensor. The EOG fluctuation (considered as noise) caused by (sway) can be canceled (reduced or eliminated) by the detection result of the gyro sensor (a specific example of this cancellation will be described later with reference to FIG. 17).

  FIG. 10 shows a case where eye movement from the front and upward, and detection signal levels (Ch0, Ch0) obtained from the three analog / digital converters (ADC) shown in FIG. It is an electrooculogram (EOG) illustrating the relationship between Ch1, Ch2, and the average level of Ch1 and Ch2 (Ch1 + 2). Eye movement detection is performed based on the detection signal waveform in the broken line frame in the figure. The detection criterion is that the user wearing the eyewear 100 is looking directly in front and there is no eye movement (if the noise caused by the shaking of the head is canceled, the three ADCs shown in FIG. The output signal waveforms Ch0 to Ch2 are substantially flat and hardly change with the passage of time in a section where the head is viewed in front without blinking). With the eyes of the left and right eyes of the user facing the front, the eyes of both eyes are moved up instantaneously, the state of facing up is maintained for 1 second, and then the eyes are immediately returned to the front. The change in the detection signal level when this is repeated five times is illustrated in FIG.

  FIG. 11 illustrates eye movement detection similar to that in FIG. 10 in the case where the line of sight moves downward from the front. If the noise caused by the shaking of the head is canceled, the line of sight is on the basis of the case where the line of sight is facing the front from the waveform change in FIGS. 10 and 11 even when walking. It can be detected whether it is below or below.

  FIG. 12 shows the detection signal levels (Ch0, Ch1, Ch2, and the eye movements from the left to right eye movements and the three ADCs shown in FIG. 6 when the noise caused by the head motion is canceled. It is an electrooculogram which illustrates the relationship with Ch1 + 2). If there is eye movement from left to right, the change over time in the detection signal waveform of Ch0 will rise to the right (not shown, but if there is eye movement from right to left, the change over time in the detection signal waveform of Ch0 Is going down.) From such a change in the waveform of Ch0, it is possible to detect whether the line of sight is on the right or left with reference to the case where the line of sight is facing the front.

  The following can be understood by combining the detection results of FIGS. In other words, if the noise caused by the motion of the head is canceled, it can be seen whether the line of sight is facing up, down, left, or right, even when walking, based on the case where the line of sight is facing the front. .

  FIG. 13 shows eye movements in which blinking / blinks (both eyes) are repeated five times at 5 second intervals when the line of sight is facing the front in the case where noise due to head shaking is canceled. 6 is an electrooculogram illustrating a relationship with detection signal levels (Ch0, Ch1, and Ch2) obtained from three ADCs shown in FIG. Blink / blink of both eyes can be detected by pulses appearing in Ch1 and Ch2. In many cases, the user's unconscious blink does not have periodicity. Therefore, as shown in FIG. 13, the intentional blink / blink of the user can be detected by detecting a plurality of pulses at regular intervals.

  It should be noted that the time for human “blink / blink” operation or the time for which the visual field is blocked by “blink / blink” is about 300 msec according to one theory (about 100 msec to 150 msec according to another theory). In either case, the signal component due to “blink / blink” can be removed by the high-pass filter (or band-pass filter) that is shorter than the walking pitch period. The blink / blink interval (for example, about 300 msec) is different from the normal walk pitch (about 600 msec), and the unintentional blink / blink interval is not synchronized with the normal walk pitch. For this reason, when canceling EOG noise caused by head fluctuations associated with normal walking, all EOG waveform changes due to blinking / blinking will not be canceled.

  Fig. 14 shows the repetition of 1-second eye-meditation (both eyes) and 4-second eye-opening (both eyes) five times when the line of sight is looking forward when the noise caused by head movement is canceled FIG. 7 is an electrooculogram illustrating the relationship between the observed eye movement and the detection signal levels (Ch0, Ch1, and Ch2) obtained from the three ADCs illustrated in FIG. 6. Eye meditation can be detected by a wide pulse appearing in Ch1 and Ch2 (the time during which the eye is intentionally meditated is longer than the time to close the eye with blink / blink, so the detected pulse width becomes wider) . By detecting the wide pulse of Ch1 and Ch2 as exemplified in FIG. 14, it is possible to detect the user's intentional eye meditation.

  Although not shown, a wide pulse with a large amplitude appears on Ch1 when the user meditates only on the right eye, and a wide pulse with a large amplitude appears on Ch2 when the user meditates only on the left eye. This makes it possible to detect eye meditation separately on the left and right.

  If you meditate with only one eye, a small wave appears on Ch0 that is out of phase on the left and right. When viewed in Ch0, the following small uneven waveform appears.

• Convex waveform when the right eye is meditated, because the positive potential of the electrode decreases;
・ When the left eye is meditated, the positive potential of the + electrode decreases, resulting in a concave waveform.

  FIG. 15 shows the case where the left eye wink (the blink of the left eye) is 5 immediately after the blinking of both eyes is repeated five times when the line of sight is facing the front when the noise due to the shaking of the head is canceled. FIG. 7 is an electrooculogram illustrating the relationship between eye movements repeated twice and detection signal levels (Ch0, Ch1, and Ch2) obtained from the three ADCs shown in FIG. 6.

  As illustrated in FIG. 6, the position of the ADC 1512 of Ch0 is offset downward from the eyeball center line of the left and right eyes. Due to this offset, a negative potential change appears at both the + and − inputs of the ADC 1512 of FIG. At this time, if the potential change (amount and direction) of both the + input and the − input is substantially the same, the change is almost canceled, and the value of the signal level output from the ADC 1512 of Ch0 becomes substantially constant ( (Refer to the Ch0 level in the broken line on the left side of FIG. 15). On the other hand, in the blink of one eye (left eye), there is almost no potential change on the negative input side of the ADC 1512 and a relatively large negative potential change appears on the positive input side of the ADC 1512. Then, the amount of cancellation of the potential change between the + input and the − input of the ADC 1512 becomes small, and a small pulse (a small wave of the signal level) appears in the negative direction in the signal level output from the ADC 1512 of Ch0 (FIG. 15). (Refer to the Ch0 level in the broken line on the right). It is possible to detect that the left eye wink has been made from the polarity of the small wave (pulse in the negative direction) of this signal level (when the EOG noise caused by the shaking of the head is canceled, the left wink detection using Ch0 is performed. Example).

  Note that if the change in potential of the + input and -input of the ADC 1512 is not equal due to distortion of the user's face, skin condition, etc., the output of Ch0ADC when the eyewear 100 is worn by the user and the eyes blink at the same time is minimal ( A calibration that maximizes the amount of cancellation between the + input component and the − input component may be performed in advance.

  Further, when the peak ratio SL1a / SL2a of the detection signal Ch1 / Ch2 when the blinking / blinking of both eyes is performed is used as a reference, the peak ratio SL1b / SL2b when the left-eye wink is performed changes (SL1b / SL2b Is not equal to SL1a / SL2a). From this, the left wink can be detected.

  FIG. 16 shows the right eye wink (right one eye blink) immediately after the blinking of both eyes is repeated five times when the line of sight is facing the front when the noise caused by the shaking of the head is canceled. FIG. 7 is an electrooculogram illustrating the relationship between eye movements repeated twice and detection signal levels (Ch0, Ch1, and Ch2) obtained from the three ADCs shown in FIG. 6.

  As described above, since the position of the ADC 1512 in FIG. 6 is offset downward from the eyeball center line of the left and right eyes, a negative potential change appears in both the + input and the − input of the ADC 1512 in the blink of both eyes simultaneously. However, similar potential changes at the + input and the-input are almost canceled, and the value of the signal level output from the ADC 1512 of Ch0 is substantially constant (see the Ch0 level in the broken line on the left side of FIG. 16). On the other hand, in the blink of one eye (right eye), there is almost no potential change on the + input side of the ADC 1512 and a relatively large negative direction potential change appears on the −input side of the ADC 1512. Then, the amount of potential change cancellation between the − input and the + input of the ADC 1512 becomes small, and a small pulse (a small wave of the signal level) appears in the positive direction in the signal level output from the ADC 1512 of Ch0 (FIG. 16). (Refer to the Ch0 level in the broken line on the right). It is possible to detect that the right eye wink has been made from the polarity of this signal level wavelet (pulse in the positive direction) (when the EOG noise caused by the shaking of the head is canceled, the right wink detection using Ch0 is detected. Example).

  When the peak ratio SR1a / SR2a of the detection signal Ch1 / Ch2 when the blinking of both eyes is performed as a reference, the peak ratio SR1b / SR2b when the right eye wink is performed changes (SR1b / SR2b is SR1a / SR2b). SR2a and not equal). Further, the peak ratio SL1b / SL2b at the time of the left wink has a value different from the peak ratio SR1b / SR2b at the time of the right wink (how much it can be confirmed by experiment). Therefore, the left wink can be detected separately from the right wink (an example of left and right wink detection using Ch1 and Ch2).

  Whether the Ch0 or Ch1 / Ch2 is used for the left and right wink detection may be determined as appropriate by the device designer. The left and right wink detection results using Ch0 to Ch2 can be used as operation commands.

  FIG. 17 is a flowchart for explaining an example of a process (noise canceling process) for minimizing noise caused by head shaking. A computer program corresponding to this flowchart is stored in, for example, the nonvolatile memory 11b of FIG. 7, and is executed by the processor 11a.

  First, a user wearing eyewear 100 as illustrated in FIG. 5 walks at a constant speed with a constant rhythm while gazing at a fixed object in front (ST10). Along with this walking, the user's head is shaken (moved / rotated) in a predetermined pattern (see FIGS. 1A and 1B).

  The movement / rotation of the user's head accompanying walking is detected by an acceleration sensor / gyro sensor (11e in FIG. 7) provided in the information processing section 11 in FIG. 5 (ST12). The movement / rotation of the user's head accompanying walking is mixed in the EOG detection signal waveform as noise (see FIG. 8 or FIG. 9). The detection signal of the acceleration sensor / gyro sensor is added to the EOG detection signal waveform at a predetermined phase (ST14).

  As a result of the above signal addition, if the signal amplitude has increased (NO in ST16), the phase of the detection signal to be added is inverted (ST18), and the acceleration sensor / gyro sensor detection signal is added again to the EOG detection signal waveform. (ST14).

  If the signal amplitude decreases as a result of the signal addition (YES in ST16), the level of the signal to be added is adjusted so that the signal amplitude of the addition result is minimized (minimum) (ST20). By minimizing the signal amplitude of the addition result (minimum), noise mixed in the EOG detection signal waveform is canceled (reduced or eliminated).

  Based on the EOG detection result after the noise caused by the shaking (movement / rotation) of the head is canceled, the user's eye movement (gaze movement, blink, eye meditation, wink, etc. in FIGS. 10 to 16) A determination is made (ST22), and processing based on the determination is performed (ST24). For example, when the user winks, the name of the building ahead of the line of sight can be displayed as an AR on the display screen (IM1 / IM2 in FIG. 5) of the eyewear 100.

  When the shaking pattern of the user's head changes, the content of noise cancellation also changes. Therefore, it is detected from the change in the detection result of the acceleration sensor / gyro sensor whether the shaking pattern of the user's head has changed (ST26). For example, it is assumed that a user of eyewear 100 walking at a constant speed while gazing at a fixed object in front stops in front of a descending staircase that descends to a subway, and sweeps a line of sight by stepping back and forth and moving the gaze. . Then, the gaze sweep before and after the step makes the head shaking pattern different from that during walking, and the EOG detection signal waveform also differs from that during walking.

  Therefore, when the shaking pattern of the user's head has changed (YES in ST26), the process returns to step ST12 and the processes of ST14 to ST20 are performed again. By performing the processes of ST14 to ST20 again, the EOG noise caused by the new head shaking pattern is canceled (reduced or eliminated), and EOG corresponding to pure eye movement is detected. The EOG detection result corresponding to this pure line-of-sight movement is compared with the detection result of the acceleration sensor / gyro sensor corresponding to the changed user head shaking pattern. As a result of this comparison, for example, if phase reversal from FIG. 4A to FIG. 4B is detected at the step position of the staircase, it is determined that there is a stair step near the phase reversal point (ST22). ). Then, a mark indicating the presence of a step is AR-displayed with the eyewear 100 (ST24). The AR display can be automatically turned off after a certain time (for example, 10 seconds) after the detection result of the acceleration sensor / gyro sensor corresponding to the shaking pattern of the user's head disappears.

  The AR display can be turned on / off automatically depending on whether the EOG waveform (Ch1 / Ch2 waveform in FIG. 8 or FIG. 9) and the detection signal waveform of the acceleration sensor are synchronized. When the user is looking at the AR display, the eyeball does not rotate to compensate for head sway, so there is no synchronization relationship as described above. However, when the user is looking at the real world instead of the AR display, A synchronization relationship occurs. Based on the same idea, the AR display can be automatically turned on / off depending on whether the EOG waveform (Ch0 waveform in FIG. 8 or 9) and the detection signal waveform of the gyro sensor are synchronized.

  If the user continues to use the eyewear while the user's head shaking pattern does not change (NO in ST28), the processing of ST22 to ST24 continues. If the user instructs the processor 11a of FIG. 7 to end use of the eyewear 100, such as by meditating for a few seconds (YES in ST28), the processing of FIG. 17 ends.

  FIG. 18 is a diagram for explaining an example of mounting the EOG electrode in the eyeglass-type eyewear 100 according to another embodiment (example in which the EOG electrode is arranged around the eyeball). The eyewear 100 of FIG. 18 differs from the eyewear 100 of FIG. 5 in the following points.

  The first difference is that the EOG electrodes 151a and 151b of the right nose pad 150R have moved to the right eye frame 101 side, and the EOG electrodes 152a and 152b of the left nose pad 150L have moved to the left eye frame 102 side. . The EOG electrodes 151a and 151b are arranged at positions that are substantially point-symmetric with respect to the user's right eye center position (not shown), and the EOG electrodes 152a and 152b are with respect to the user's left eye center position (not shown). It is arranged at a position that is substantially point-symmetric. Further, the right oblique line connecting each of the EOG electrodes 151a and 151b and the left oblique line connecting each of the EOG electrodes 152a and 152b are substantially line symmetrical with respect to a vertical line (not shown) along the user's nose. The electrodes 151a, 151b, 152a, and 152b can be formed of a conductive member (metal, conductive polymer, or the like) provided at the tip of an elastic body (sponge, silicone cushion, or the like). Each EOG electrode is lightly pressed against the skin surface of the face of the user wearing the eyewear 100 by the elastic repulsive force of the elastic body.

  When the EOG electrode arrangement structure as shown in FIG. 18 is adopted, the electric field generated around the charged eyeball can be detected more easily than the structure in which the EOG electrode is arranged in the nose pad portion. Therefore, the embodiment of FIG. 18 can detect an EOG signal having a larger amplitude than the embodiment of FIG.

  The second difference is that the sensor unit 11e including an acceleration sensor and a gyro sensor is left in the bridge 103, and other functions of the information processing unit 11 are moved to the temple bar 107 side. For example, when the physical size of the information processing unit 11 increases due to enhancement of information processing capability or addition of a GPS function, when the increased information processing unit 11 is attached to the bridge 103, the design of the eyewear 100 and its mounting There may be a problem with the feeling. If the structure attached to the bridge 103 is only the sensor unit 11e, the structure on the bridge 103 is smaller and lighter than providing the entire information processing unit 11, and thus the design and the mounting feeling may be improved. is there. On the other hand, within the temple bar 107 (and / or 106), even if a relatively large structure is provided, there is little risk of problems in the design of the eyewear 100 and its wearing feeling.

  FIG. 19 is a diagram for explaining an example of mounting EOG electrodes in goggle-type eyewear (a type in which eye frames of left and right eyes are continuous) according to still another embodiment (other EOG electrodes arranged around the eyeball). Example). When the structure as shown in FIG. 19 is adopted, the skin contact stability of the EOG electrode is higher than that in the structure of FIG. For this reason, it is possible to detect the EOG signal with higher accuracy while canceling out noise caused by shaking of the head. In the structure of FIG. 19, the information processing unit 11 and other devices that have been increased in size are easily incorporated. Therefore, in the structure of FIG. 19, the function of the smartphone with GPS can be loaded without worrying about design. Screen display in that case can be performed by AR display on the left and right displays 12L / 12R using film liquid crystal or the like. Although not shown, a small speaker can be attached near the left and right ears of the goggle frame, and a small microphone can be attached near the nose cushion. In addition, command input to the smartphone can be performed by eye movement such as eye movement, blinking, eye meditation, and winking.

  FIG. 20 is a diagram for explaining an example of mounting the EOG electrode in goggle-type eyewear (a type in which the eye cups of the left and right eyes are separated) according to still another embodiment (another example in which the EOG electrode is arranged around the eyeball). Example). In the embodiment of FIG. 20, the same EOG electrode arrangement structure and the arrangement structure of the sensor unit 11e are adopted as in the case of FIG. However, the goggle structure of FIG. 20 can withstand use in water (or use of space swimming training). The head of a diver who swims in the water has a greater sway than walking on the ground, but the large sway (up / down / left / right position movement or rotation) can be detected by a 3-axis acceleration sensor or 3-axis gyro sensor in the sensor unit 11e. The detected shaking component can be used to cancel the noise component mixed in the EOG signal. Various commands can be input based on the eye movements (blinks, eye meditation, etc.) of the user (diver) whose both hands are blocked, using the EOG signal in which noise caused by the movement of the head is suppressed. .

<Summary of Embodiment>
(A) In the conventional gaze detection technology, there is a method of estimating the gaze direction by image processing using an infrared camera. However, the amount of required apparatus (in addition to the high-intensity infrared LED, the entire eyeball visible to the outside is used. Cameras that can be photographed, relatively high-performance computing devices for image processing, and power supply devices that operate these devices become large-scale. For this reason, the conventional line-of-sight detection technology has been applied to stationary devices, but has not been applied to wearable devices.

  On the other hand, gaze detection based on electrooculogram sensing requires a considerably small amount of device (electrodes, ADC, arithmetic device with relatively low performance, and power supply device for operating them) compared to the infrared camera system. It can be applied to wearable devices such as AR glasses. EOG is a technology that can be expected to be applied to wearable devices because of its high affinity with AR glasses and low power consumption compared to infrared systems.

  (B) To cancel the shaking of the eyewear user's head when gazing at a fixed object in the real world. Thereby, the detection accuracy with respect to arbitrary eyeball rotation is improved.

  (C) It is detected whether the real-world fixed object being watched is a distant view or a close view (detection of phase inversion point in FIG. 4). Thereby, the presence / absence of gaze in the real world (fixed object) is detected from the shaking of the head of the eyewear user.

  (D) The head of the eyewear user is used for automatic ON / OFF control of the AR display.

  (E) Only an arbitrary eye movement that cancels the compensated rotation of the eye is extracted.

  (F) Detect whether the object being watched is in the real world (if the detected waveform of the acceleration sensor and / or gyro sensor and the electrooculogram EOG waveform are synchronized during walking, see what is in the real world) If the detection waveform of the acceleration sensor and / or gyro sensor and the electrooculogram EOG waveform are not synchronized, it is determined that the AR display or the like is being viewed during walking).

  (G) Effective as noise cancellation when eye movement is used as UI (user interface).

  (H) When used for collecting work history information using eyewear, it becomes an unprecedented index (for example, in the picking operation of a warehouse, etc., with the effect of noise caused by head shaking, It can be used for pass / fail judgments such as “pass as a worker if the frequency with the line of sight toward is above a certain value”.

<Example of Correspondence Between Contents of Claims Initially Applied and Embodiments>
[1] An electrooculogram detection device according to an embodiment (11 in FIG. 7: including a processor 11a that executes the process in FIG. 17) is based on an electrooculogram (EOG) of a user wearing the eyewear (100). An eye movement detection unit (15; 151a, 151b, 152a, 152b) for detecting eye movement (eye movement, blinking, etc.) including eyeball rotation of the user, and an acceleration sensor (11e) for detecting movement of the eyewear And a gyro sensor (a part of 11e) for detecting the rotation of the eyewear. This eye movement detection device detects a movement and / or rotation of the eyewear by the acceleration sensor and / or the gyro sensor when the eye movement is detected based on the electrooculogram (EOG). (11a that executes ST12) and noise mixed in the electro-oculogram due to movement and / or rotation of the eyewear (an electro-oculogram component added by compensatory eye rotation that works in a direction that cancels the movement of the head) ) Is reduced (combined with signal components detected by the acceleration sensor and / or the gyro sensor (components having a phase opposite to the mixed noise)) (noise reduction means (ST14 to ST20) is executed) 11a).

  [2] The apparatus according to [1] selects a phase of a signal component detected by the acceleration sensor and / or the gyro sensor (ST16 to ST16) so that a signal synthesis result by the reduction unit tends to decrease. 11a) for executing ST18 is further provided.

  [3] The device (11) of the above [1] is incorporated into eyewear (100 in FIGS. 5 and 18 to 20).

  [4] The eyewear (100) of [3] has a display unit (12L, 12R), and displays based on the electrooculogram (EOG) after the noise is reduced by the noise reducing means (AR display) Display means (11a for executing ST22 to ST24) for performing display on the display unit (12L, 12R).

  [5] The device according to [1] detects a rotation corresponding to a change in forward / backward bending of the user's head by the gyro sensor (11e), and changes a user's line-of-sight direction (FIG. 10, FIG. 11: Ch1). / Ch2 +/- level change is detected by the eye movement detection unit (15), and the signal phase of the anteroposterior bending change and the signal phase of the line-of-sight change are in phase (FIG. 4 ( a)) or reverse phase (FIG. 4B) is configured to identify the perspective of the place the user is looking at (whether the real-world fixed object being watched is a distant view or a close view) Can be detected).

  [6] The apparatus of [1] may be configured to detect movement of the eyewear detected by the acceleration sensor and / or the gyro sensor and / or when the eye movement is detected based on the electrooculogram (EOG). Alternatively, when the rotation (FIG. 1) is synchronized with the change of the electrooculogram (EOG) (FIG. 8 or FIG. 9) (when the “predetermined phase relationship” can be maintained in ST14 of FIG. 17), the user moves the real world. It is configured to determine that the user is looking (it can detect whether or not the object being watched is the real world).

  [7] An electrooculogram detection method according to an embodiment includes an eye movement (eye movement, blink, etc.) including eye rotation of the user based on an electrooculogram (EOG) of the user wearing the eyewear (100). , Detecting the movement of the head of the user wearing the eyewear, and detecting the rotation of the head of the user wearing the eyewear. This method includes a step of detecting movement and / or rotation of a user's head wearing the eyewear when the eye movement is detected based on the electrooculogram (EOG) (ST12), and the eye The noise mixed in the electrooculogram due to the movement and / or rotation of the head of the user wearing the wear (the electrooculogram component added by the compensatory ocular rotation that works in the direction of canceling the movement of the head), Steps of reduction (minimization or cancellation) by synthesis of signal components (components having an opposite phase to the mixed noise) detected corresponding to the movement and / or rotation of the head of the user wearing the eyewear (ST14- ST20).

  [8] In the method of [7], the phase of the signal component detected corresponding to the movement and / or rotation of the head of the user wearing the eyewear so that the synthesis result of the signal component decreases. Is further provided (ST16 to ST18).

  Although several embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, replacements, and changes can be made without departing from the scope of the invention.

  For example, in the description of the embodiments, glasses-type and goggles-type devices were introduced as eyewear equipped with EOG electrodes. In addition to this, articles such as eye masks, eye patches, helmets, hats, and hoods are used as EOG electrodes. It can be used for attached eyewear.

  These embodiments and modifications thereof are included in the scope and gist of the invention, and are included in the invention described in the claims and the equivalents thereof. Note that it is possible to combine a part or all of one embodiment of a plurality of disclosed embodiments with a part or all of another embodiment of the plurality of disclosed embodiments. Or included in the abstract.

  DESCRIPTION OF SYMBOLS 100 ... Eyewear (goggles type or glasses type); 110 ... Eye frame; 11 ... Information processing part (processor 11a, non-volatile memory 11b, main memory 11c, communication processing part (GPS processing) which becomes the principal part of an eye movement detection apparatus Part) 11d, integrated circuit including sensor part 11e, etc .; 11e ... sensor part including acceleration sensor, gyro sensor, etc .; BAT ... power source (lithium ion battery etc.); 12 ... display part (right display 12R and left display 12L: IM1 ... Right display image (numeric keypad, alphabet, character string, mark, icon, etc.); IM2 ... Left display image (numeric keypad, alphabet, character string, mark, icon, etc.); 13 ... Camera (right camera 13R) And attached to the left camera 13L or bridge 103 15... Eye movement detector (line-of-sight detection sensor); 1510... Right side (Ch1) AD converter; 1520... Left side (Ch2) AD converter; Left to right (Ch3) AD converter.

Claims (8)

An eye movement detector that detects eye movement including eyeball rotation of the user based on the electrooculogram of the user wearing the eyewear, and outputs a detection signal corresponding to the detected eye movement, and detects movement of the eyewear An eye movement detector comprising an acceleration sensor for detecting the rotation of the eyewear and a gyro sensor for detecting the rotation of the eyewear,
Detecting means for detecting movement and / or rotation of the eyewear by the acceleration sensor and / or the gyro sensor when the eye movement is detected based on the electrooculogram;
Correction for correcting the detection signal by adding a correction signal corresponding to the signal component detected by the acceleration sensor and / or the gyro sensor to the detection signal output from the eye movement detection unit, and outputting the addition result Means,
Adjusting means for adjusting the amplitude and phase of the correction signal added to the detection signal so that the output of the correction means decreases;
An eye movement detection device comprising: 2. The apparatus according to claim 1, wherein the adjustment means adjusts the amplitude and phase of the correction signal so that the output of the correction means is minimized . Rotation corresponding to the longitudinal屈変of the head of the user is detected by the gyro sensor, the change in the viewing direction of the user detected by the Meadow detection unit, the front and rear屈変signal phase as the viewing direction of The apparatus of claim 1, wherein the location of the user's viewing is identified by whether the signal phase of change is in phase or out of phase.   When the eye movement is detected based on the electrooculogram, the movement and / or rotation of the eyewear detected by the acceleration sensor and / or the gyro sensor is synchronized with the change in the electrooculogram. The apparatus of claim 1, wherein the apparatus determines that the user is looking at the real world. Eyewear that can be worn on the user's head,
An eye movement detector that detects eye movement including eyeball rotation of the user based on the user's electrooculogram and outputs a detection signal corresponding to the detected eye movement;
An acceleration sensor for detecting movement;
A gyro sensor that detects rotation;
Detecting means for detecting movement and / or rotation of the eyewear by the acceleration sensor and / or the gyro sensor when the eye movement is detected based on the electrooculogram;
Correction for correcting the detection signal by adding a correction signal corresponding to the signal component detected by the acceleration sensor and / or the gyro sensor to the detection signal output from the eye movement detection unit, and outputting the addition result Means,
Adjusting means for adjusting the amplitude and phase of the correction signal added to the detection signal so that the output of the correction means decreases;
Eyewear equipped with. And de Isupurei part,
A display means for performing display based on the output of the correcting means in the display portion, and further comprising a, eyewear of claim 5. Based on the eye potential of the user who wears the eyewear detects Meadow comprising eyeball rotation of the user, the movement of the head of the user wearing the eyewear detected by the acceleration sensor, mounting the eyewear the rotation of the head of the user who has a method comprising a process of detecting by the gyro sensor,
A step of the in the meadow is detected on the basis of the eye potential, for detecting the movement and / or rotation of the head of the user wearing the eyewear,
Correcting the detection result of the eye movement by adding a correction signal corresponding to the signal component detected by the acceleration sensor and / or the gyro sensor to the detection result of the eye movement;
Adjusting the amplitude and phase of the correction signal so as to reduce the corrected detection result of the eye movement ;
An eye movement detection method comprising: 8. The method of claim 7, wherein the step of adjusting the amplitude and phase of the correction signal adjusts the amplitude and phase of the correction signal so that the corrected detection result of the eye movement is minimized .

Images