JP6535123B2 - Eye movement detection apparatus and eye movement detection method - Google Patents

Description

  Embodiments of the present invention relate to an eye movement detection apparatus and an eye movement detection method to which an eye potential sensing technique is applied.

  Usually, when a person walks while looking forward, the person's head shakes up and down, but the person can clearly see the surrounding scenery. This is because the head movement during walking is suppressed to a range smaller than the other parts of the body (amplitude of about 2 to 3 cm regardless of the distance to the line of sight), within which range the head and eye are stable in line of sight To move in a coordinated manner (see Non-Patent Document 1). This non-patent document 1 teaches that "the head shakes up and down and to the left and right during walking, and rotation of the head and compensated rotation of the eyeball occur as the operation of visual axis stabilization for the shake".

  As for the electro-oculogram sensing technology, there is a known example 1 in which the gaze position of the person is detected based on the electro-oculogram by the human eye movement (see Patent Document 1). In this known example 1, an EOG (Electro-Oculography) electrode is used for electro-oculogram detection (electro-oculography sensing), and the user's gaze position is detected from electro-oculogram data acquired from the EOG electrode (paragraph 0025). Although the EOG electrode is provided on the goggles in the embodiment of the known example 1 (paragraph 0061), there is also a known example 2 in which an electrode for EOG is provided on glasses (see Patent Document 2). Goggles and glasses are a type of eyewear worn near the eyeballs of people. When the user wearing the eyewear walks, the eyewear also shakes with the head.

JP, 2009-288529, A JP, 2013-244370, A

Head Movement and Eye Movement Research to Maintain Gaze Stability while Walking 2000-03 Yazaki, 矢 Journal of Osaka University Graduate School of Human Sciences. 26 P. 177-P. 193 http: //ir.library.osaka-u .ac.jp / dspace / bitstream / 11094/5672/1 / hs26-177.pdf

  In the stationary seating state (a state in which no movement causing disturbance is mixed in the head of the user wearing the eyewear with an EOG electrode), eye gaze / eyeball rotation can be stably detected by eye potential sensing. However, when gazing at a fixed object in the real world in a walking state (a state where up and down and left and right shaking occurs on the head of a user wearing eyewear with an EOG electrode), the target gaze / eyeball is compared to the sitting state. It becomes difficult to detect the rotation. The reason is that eye rotation is performed unconsciously as a complementary motion for visual axis stabilization possessed by the human body, and the signal component of this unconscious eye rotation is also superimposed (as noise or disturbance) on the electrooculogram sensing result. It is.

  That is, when the user wearing the eyewear with the EOG electrode walks, even if the user fixes his gaze to a specific object in front, the electrooculogram data acquired from the EOG electrode also fluctuates due to the movement of the head accompanying the walking . The fluctuation of the electro-oculogram data becomes noise (disturbance) with respect to the electro-oculogram data corresponding to the actual line of sight.

  One of the problems to be solved by the embodiment of the present invention is to suppress the noise component mixed in the electrooculogram by the movement of the head.

According to an embodiment, the eye movement detection apparatus includes a display unit configured to display a real world image or an augmented reality image to which image information is added to the real world image; a first detection unit configured to detect eye movement of a user; The second detection unit for detecting the movement of the head of the user, the user looking at the movement based on the eye movement detected by the first detection unit and the movement of the head detected by the second detection unit Determining means for determining what is being The determination unit determines whether the user is in the real world based on whether the eye movement detected by the first detection unit and the head movement detected by the second detection unit are synchronized. It is determined whether you are looking at an image or looking at the augmented reality image. The display unit is configured to determine whether the eye movement detected by the first detection unit and the head movement detected by the second detection unit are synchronized or not according to the determination result by the determination unit. The display of the augmented reality image is turned on and off.

The figure explaining the head sway (position change by up-and-down left-right motion) of the person in walking while looking ahead. The figure explaining the head sway (the angle change by forward bending / back bending) of the person who is gazing at a front fixed object while walking. The figure which illustrates the relationship between the target distance (the distance from the eye to the gaze target) and the eye movement when the head makes a back and forth change in the longitudinal direction. The figure which illustrates the phase relationship between the angle change / velocity change of head order bending, and the angle change / velocity change of eye movement. A figure explaining an eyeglass type eyewear into which an eye movement detection device concerning one embodiment was built (an example in which an EOG electrode is arranged at a nose pad). The figure which demonstrates the example of mounting of the EOG electrode in the spectacles type eyewear which concerns on one Embodiment. The figure explaining the relationship between the information processing part 11 which can be attached to various embodiment, and its peripheral device. For example, the figure which shows an example of an EOG waveform at the time of starting walking from a right foot and stopping at the right foot of the 13th step while gazing at a fixed object in the real world 25 m away. The figure which shows the other example (confirmation of repetition reproducibility) of the EOG waveform at the same time as FIG. 8 starting from the right foot again and stopping at the right foot of the 13th step under the same conditions. In the case where the noise due to head movement is canceled, the eye movement from the front to the top and the detection signal levels (Ch0, Ch1, Ch2, and so on) obtained from the three analog / digital converters (ADCs) shown in FIG. And an electro-oculogram (EOG) illustrating the relationship between the average levels of Ch1 and Ch2 (Ch1 + 2). In the case where noise due to head sway is canceled, the eye movement from the front to the bottom and the detection signal levels (Ch0, Ch1, Ch2 and Ch1 + 2) obtained from the three ADCs shown in FIG. Electrooculogram illustrating the relationship. When noise due to head movement is canceled out, eye movement from left to right and detection signal levels (Ch0, Ch1, Ch2 and Ch1 + 2) obtained from three ADCs shown in FIG. Electrooculogram illustrating the relationship. In the case where the noise due to head movement is canceled out, when the line of sight is directed to the front, the eye movement that blinks (both eyes) repeated 5 times at intervals of 5 seconds, and from the three ADCs shown in FIG. The electrooculogram which illustrates the relationship with the detection signal level (Ch0, Ch1, Ch2) obtained from. In the case where the noise due to head sway is canceled out, when the sight line is facing the front, the eye movement is a repetition of one second of the eye (both eyes) and four seconds of the eye opening (both eyes) five times FIG. 7 is an electro-oculogram illustrating the relationship between detection signal levels (Ch0, Ch1, Ch2) obtained from the three ADCs shown in FIG. In the case where the noise due to head movement is canceled out, when the line of sight is directed to the front, the eye is obtained by repeating the blink of the left eye 5 times immediately after repeating the blink of both eyes 5 times FIG. 7 is an electro-oculogram illustrating the relationship between the motion and detection signal levels (Ch0, Ch1, Ch2) obtained from three ADCs shown in FIG. When noise from head sway is canceled, when the line of sight is facing the front, the eye is repeated five blinks of the right eye immediately after repeating the blink of both eyes five times FIG. 7 is an electro-oculogram illustrating the relationship between the motion and detection signal levels (Ch0, Ch1, Ch2) obtained from three ADCs shown in FIG. The flowchart explaining an example of the processing (noise cancellation processing) which minimizes the noise resulting from head shaking. The figure which illustrates the example of mounting of the EOG electrode in the spectacles type eyewear which concerns on other embodiment (an example in which the EOG electrode is arrange | positioned around the eyeball). The figure which demonstrates the example of mounting of the EOG electrode in the goggle type eyewear (The type with which the eye frame of the right and left eyes continued) which concerns on other embodiment (The other example in which the EOG electrode is arrange | positioned around the eyeball). The figure which demonstrates the example of mounting of the EOG electrode in the goggle type eyewear (The type which the eye cup of the right and left eyes isolate | separated) which concerns on other embodiment (The further another example in which the EOG electrode is arrange | positioned around the eyeball).

Hereinafter, basic information is provided first, and then various embodiments will be described with reference to the drawings.
<Basic information>
The eyeball diameter of an adult is about 25 mm. It is about 17 mm after birth and becomes larger as it grows.
The interpupillary distance of an adult male is about 65 mm. (Normally commercially available stereo cameras are often made at intervals of 65 mm.)
The interpupillary distance of adult women is several mm shorter than men.
The eye potential is several tens of mV.
The eye has a positive potential on the cornea side and a negative potential on the retina side. When this is measured on the surface of the skin, it appears as a potential difference of several hundred μV.

The types of eye movement related to eye movement detection and the movement range of the eye are, for example, as follows:
<Type of eye movement (eye movement)>
(01) Compensatory eye movement Non-voluntary eye movement developed to stabilize the external image on the retina regardless of head or body movement.
(02) Voluntary eye movement The eye movement developed to center the visual pair image on the retina, and a movement that can be freely controlled.
(03) Impact eye movement (saccade)
Eye movement (easy to detect) that occurs when you change the gaze point to look at things.
(04) Sliding eye movement Smooth eye movement (not easy to detect) generated when tracking a slowly moving object.
<Movement range of eye (in case of general adults)>
(11) Horizontal direction Left direction: 50 ° or less Right direction: 50 ° or less (12) Vertical downward direction: 50 ° or less Upward direction: 30 ° or less (The vertical angle range that can be moved by one's own intention is only upward direction Narrow (There is a “bell phenomenon” in which the eyeball turns up when closed eyes, so when closed eyes the range of eye movement in the vertical direction shifts upward).
(13) Others Convergence angle: 20 ° or less.
<About the movement of the head accompanying walking (the source of Fig. 1 and Fig. 2 is non-patent document 1)>
FIG. 1 is a view for explaining head sway (a change in position due to vertical and horizontal movement) of a person walking while looking forward. The movement of the lower limbs during walking changes the position of the human head up and down (Fig. 1 (a)). A typical adult walking cycle (one walking cycle) is, for example, about 0.6 seconds. At the same time, in this walking, the human head changes position to the left and right (FIG. 1 (b)). The cycle of this lateral position change is about 1.2 seconds, which is twice the walking cycle. The upper and lower position change widths and the left and right position change widths are each about 2 to 3 cm. The vertical and horizontal movements are substantially linear and regular, and have a cycle closely related to the walking cycle.

  FIG. 2 is a view for explaining head sway (a change in angle due to forward bending / back bending) of a person who is gazing at a fixed object in front while walking. When a person gazing at a front visual object (target) starts walking, the head position changes up and down with walking (FIG. 2A). While gazing at the target, when the head comes to the upper position, the head bends forward (pitch down), and when the head comes to the lower position, the head bends backward (pitch up) (FIG. 2 (b)). Such anteroposterior bending (longitudinal rotational movement of the head) occurs in response to (in synchronization with) vertical movement of the head during one walking cycle. Similarly, when the head is displaced to the left and right (FIG. 1 (b)), a rotational movement in the left and right direction (synchronized with the vertical movement of the head) occurs in the head.

  That is, while walking, the head of a person performs a rotational movement that compensates for vertical and horizontal movements, and the head is back and forth at a high and low position so that the line of sight helps capture the visual target (target) ahead. Succumb. The same applies to left and right, and when the body swings to the left and right with walking and the head moves to the left and right, the head turns to the left and right (reciprocal rotation). Such head rotation is called "Compensatory Rotation", and is considered to help stabilize the gaze.

In addition, since the trunk of a person rotates back and forth and left and right during walking, the head needs to compensate for these as well. The angular position of the head is kept horizontal by rotating the head in the opposite direction to the change in the trunk.
<About the movement of the eyeball accompanying walking (the source of FIG. 3, FIG. 4 is nonpatent literature 1)>
FIG. 3 is a view for explaining the relationship between the target distance (the distance from the eye to the gaze target) and the eye movement when the head is bent in the longitudinal direction. Here, it is assumed that the target position when the head is fixed is HFP, and when the actual target is closer (Near Target) and farther (Far Target) than HFP, the fore-and-aft rotational angle (φh) of the head And the compensated rotation angles (φen, φef) of the eyeballs.

As mentioned above, the head performs compensatory rotation, but it alone can not achieve gaze stabilization completely. The head can still suffer from excess and deficiency when it is rotated to compensate for vertical movement. In order to compensate for the displacement due to the excess or deficiency of the compensation rotation, periodic compensation of the eyeball is required during walking. For example, for a near target 30 cm away (Near Target), the eyeball performs compensated rotation (φ en) in the same direction as rotation / rotation (φ h) of the head. As a result, even if the rotation / rotation of the head is insufficient, the eyeball rotates in coordination with the head in the moving direction to compensate for the lack of rotation of the head. On the other hand, for a far target 100 m away (Far Target), for example, the eyeball performs compensated rotation (φef) in the direction opposite to the rotation / rotation (φh) of the head. As a result, even if the head rotation / rotation is excessive, the eyeball rotates in the opposite direction to cancel the excess of head rotation. (Even if the target is near or far, there is no substantial difference in the amount of head movement during walking.)
FIG. 4 is a diagram illustrating a phase relationship between an angle change / velocity change in head longitudinal bending and an angle change / velocity change in eye movement. The anteroposterior bending angle (φh) of the head shown in FIG. 3 can be detected by a three-axis gyro (11e in FIG. 7) attached to the eyewear. The rotation angles (φen, φef) of the eyeball shown in FIG. 3 can be detected by the eye movement detection unit (15 in FIG. 7) using the EOG of the eyeball.

  The vertical movement change (longitudinal bending change) of the head accompanying walking changes in the same phase as the change in eyeball rotation when the target is near (Near Target) (FIG. 4A). On the other hand, when the target is far (Far Target), the vertical movement change (longitudinal bending change in the vertical direction) of the head changes in the opposite phase of the eye rotation change (FIG. 4 (b)). When the target is at the fixed position HFP shown in FIG. 3, the eyeball rotation disappears, and the waveform change of the broken line shown in FIGS. 4 (a) and 4 (b) becomes minimal (substantially horizontal in the figure). And the phase between the longitudinal bending change of the head and the change of the eye rotation is reversed). A similar phase reversal is also seen during changes in the velocity of the head vertical movement and in the velocity of the eye rotation (Fig. 4 (c) (d)).

  Now, the signal change that occurs in the EOG due to the above-mentioned "up and down movement change of the head accompanying walking" will be referred to as "body movement noise". A walking act can be detected by an acceleration sensor or a gyro sensor, and an eye movement can be detected from an EOG. Then, when "moving" and "no eye movement" are detected, if body movement noise does not appear in the EOG, "the eyewear user reverses the body movement noise according to the distance of the visual target. It is indicated that the user is gazing at the point (corresponding to the position HFP in FIG. 3).

  FIG. 5 is a view for explaining an eyeglass-type eyewear 100 in which an eye movement detection device according to an embodiment is incorporated (an example in which an EOG electrode is disposed on a nose pad). In this embodiment, a right eye frame (right rim) 101 and a left eye frame (left rim) 102 are connected to a bridge 103. The left and right eye frames 102 and 101 and the bridge 103 can be made of, for example, an aluminum alloy, titanium or the like. The left outside of the left eye frame 102 is connected to the left temple bar 106 via the left hinge 104, and the left modern (left ear pad) 108 is provided at the tip of the left temple bar 106. Similarly, the right outer side of the right eye frame 101 is connected to the right temple bar 107 via the right hinge 105, and the right modern (right ear pad) 109 is provided at the tip of the right temple bar 107.

  An information processing unit 11 (an integrated circuit of several millimeters square), which is a main part of the eye movement detection apparatus, is attached to the bridge 103. The information processing unit 11 can be configured by an LSI in which a microcomputer, a memory, a communication processing unit, and the like are integrated (the details of the information processing unit 11 will be described later with reference to FIG. 7).

A small battery (BAT) such as a lithium ion battery is embedded, for example, in the left temple bar 106 (or in the right temple bar 107, or in the modern 108 or 109) and becomes a power source necessary for the operation of the eyeglass-type eyewear 100. ing.
The left camera 13L is attached to the end of the left eye frame 102 closer to the left hinge 104, and the right camera 13R is attached to the end of the right eye frame 101 closer to the right hinge 105. These cameras can be configured using a very small CCD image sensor.

  These cameras (13L, 13R) may constitute a stereo camera. Alternatively, an infrared camera (13R) and a laser (13L) may be disposed at the position of these cameras, and a distance sensor by infrared camera + laser may be configured. This distance sensor can also be configured by a small semiconductor microphone (13R) that collects ultrasonic waves, a small piezoelectric speaker (13L) that emits ultrasonic waves, or the like.

  In addition, in place of the left and right cameras 13L / 13R, or in addition to the left and right cameras 13L / 13R, an embodiment may be considered in which a central camera (not shown) is provided in the bridge 103 portion. Conversely, in some embodiments there may be no camera at all (these cameras are shown as camera 13 in FIG. 7).

  The left display 12 L is fitted in the left eye frame 102, and the right display 12 R is fitted in the right eye frame 101. This display is provided on at least one of the left and right eye frames, and can be configured using film liquid crystal or the like. Specifically, one or both of the left and right displays 12L and 12R can be configured using a film liquid crystal display device employing a polymer dispersed liquid crystal (PDLC) that does not use a polarizing plate (this display is a display in FIG. 7) Shown as 12).

Transparent left and right displays 12L / 12R using film liquid crystal can be used as a means for providing augmented reality (AR) that adds image information such as numbers and characters to the real world seen through glasses.
A nose pad portion is provided between the left and right eye frames 102 and 101 below the bridge 103. The nose pad portion is composed of a pair of left nose pad 150L and right nose pad 150R. Right nose pad electrodes 151a and 151b are provided on the right nose pad 150R, and left nose pad electrodes 152a and 152b are provided on the left nose pad 150L.

  The electrodes 151a, 151b, 152a, 152b are electrically separated from each other, and connected to three AD converters (ADCs 1510, 1520, 1512) via insulated wiring members (not shown). The outputs from these ADCs have different signal waveforms according to the eye movement of the user wearing the eyewear 100, and are supplied to the information processing unit 11 as digital data according to the user's eye movement. The electrodes 151a, 151b, 152a, and 152b are used as gaze detection sensors, and are components of the eye movement detection unit 15 of FIG. 7 together with three AD converters.

  The eyewear 100 is fixed to the user's head (not shown) by the left and right nose pads (150L, 150R), the left and right temple bars (106, 107), and the left and right moderns (108, 109). In this embodiment, it is only necessary to touch the head (or face) of the user directly with the left and right nose pads (150L, 150R), the left and right temple bars (106, 107) and the left and right moderns (108, 109). However, there may be embodiments in which portions other than those (nose pad, temple bar, modern) touch the user, such as for voltage alignment between the ADC (1510, 1520, 1512) and the user's body.

  On the film liquid crystal of the right display 12R of FIG. 5, the right display image IM1 including, for example, ten keys (numbers, operators, Enter, etc.), alphabets, marks of a predetermined shape, and other icon groups can be displayed. In addition, the left liquid crystal display 12L can display the left display image IM2 including, for example, an arbitrary character string, an arbitrary shaped mark, various icons and the like (the display contents of the displays 12L and 12R may be anything). The ten keys and alphabets displayed on the right display 12R (or the left display 12L) can be used to input numbers and characters. Character strings, marks, icons, etc. displayed on the right display 12R (or the left display 12L) can be used when searching for a specific information item, selecting / determining a desired item, or alerting the user. .

  The display images IM1 and IM2 can be used as display means of augmented reality (AR) that adds information such as numbers, letters, marks, etc. to the real world seen through glasses, and this AR display can be turned on and off as appropriate. The content of the display image IM1 and the display image IM2 may have the same content (IM1 = IM2) or different content (IM1 ≠ IM2) according to the embodiment. Also, the display of the display image IM1 (or IM2) can be performed by the right display 12R and / or the left display 12. When the content of the AR display is to be a 3D image overlapping (with depth) in the real world viewed through the glasses, IM1 and IM2 can be left and right separate 3D images.

  Further, when the displays (12R, 12L) are present on the left and right, for example, the convergence angle can be adjusted to shift the images of the left and right display images (IM1, IM2) in opposite directions in the left and right. As a result, it is conceivable to reduce the burden on the eyes when alternately viewing an object and an AR display that are visible in the real world. However, usually, the left and right displays (12R, 12L) display images of the same content.

  Display control on the displays 12 </ b> L and 12 </ b> R can be performed by the information processing unit 11 embedded in the right temple bar 107. (Techniques for displaying characters, marks, icons, etc. on the display are well known.) A power source necessary for the operation of the information control unit 11 and the like can be obtained from the battery BAT embedded in the left temple bar 106.

  The designer or the designer may wear the prototype of the eyewear 100 corresponding to the embodiment and feel that the weight balance is bad. If the main cause is the BAT in the left temple bar 106, a "weight (or another equal weight battery)" corresponding to the BAT in the left temple bar 106 can be inserted in the right temple bar 107.

  FIG. 6 is a view for explaining an implementation example of the EOG electrode in the eyeglass-type eyewear 100 according to the embodiment. Right nose pad electrodes 151a and 151b are provided above and below the right nose pad 150R, and left nose pad electrodes 152a and 152b are provided above and below the left nose pad 150L. The outputs of the right nose pad electrodes 151a and 151b are applied to the ADC 1510, and the outputs of the left nose pad electrodes 152a and 152b are applied to the ADC 1520, and the outputs of the lower electrodes 151b and 152b (or upper electrodes 151a and 152a) of the left and right nose pads. Is given to the ADC 1512.

The ADC 1510 provides a Ch1 signal that changes in response to the user's right and left eye movements. The ADC 1520 obtains a Ch2 signal that changes in response to the user's left and right eye movements. From the ADC 1512, a Ch0 signal that changes corresponding to the user's left and right eye movement is obtained. The vertical movement of the left and right eyes can be evaluated by the Ch1 + 2 signal corresponding to the average of the outputs of the ADC 1510 and ADC 1520. (The relationship between the signal waveforms of Ch0, Ch1, Ch2, and Ch1 + 2 and eye movement will be described later.)
FIG. 7 is a view for explaining the relationship between the information processing unit 11 attachable to various embodiments and its peripheral devices. In the example of FIG. 7, the information processing unit 11 includes a processor 11a, a non-volatile memory 11b, a main memory 11c, a communication processing unit 11d, a sensor unit 11e, and the like. The processor 11a can be configured by a microcomputer having processing capability according to product specifications. Various programs executed by the microcomputer and various parameters used at the time of program execution can be stored in the non-volatile memory 11b. The main memory 11 c provides a work area when executing a program.

  The sensor unit 11 e includes a sensor group for detecting the position and / or the orientation of the eyewear 100 (or the head of the user wearing the eyewear). Specific examples of these sensor groups include an acceleration sensor that detects movement in three axial directions (x-y-z direction), a gyro that detects rotation in three axial directions, and a geomagnetic sensor that detects an absolute orientation (a compass function) There are beacon sensors that receive radio waves, infrared rays, and the like to obtain position information and the like. The iBeacon (registered trademark) or Bluetooth (registered trademark) 4.0 can be used to acquire the position information and the like.

  The LSI that can be used for the information processing unit 11 is commercialized. One example is Toshiba Semiconductor & Storage's TZ1000 series for wearable terminals. The product name "TZ1011MBG" in this series includes CPU (11a, 11c), flash memory (11b), Bluetooth Low Energy (registered trademark) (11d), sensor group (acceleration sensor, gyro, geomagnetic sensor) (11e) , 24 bit delta sigma ADC, I / O (USB etc).

The information processing unit 11 is attached to the bridge 103 as an example of a place where the relative positional relationship between the sensor unit 11 e such as an acceleration sensor or a gyro sensor and the EOG electrodes (151a, 151b, 152a, 152b) is not broken.
What to do with the processor 11a can be instructed from an external server (or personal computer) (not shown) via the communication processing unit 11d. The communication processing unit 11 d can use an existing communication method such as ZigBee (registered trademark), Bluetooth (registered trademark), Wi-Fi (registered trademark) or the like. The processing result of the processor 11a can be sent to a server or the like (not shown) via the communication processing unit 11d.

The display 12 (12L and 12R), the cameras 13 (13L and 13R), the eye movement detection unit 15 and the like are connected to the system bus of the information processing unit 11. Each device (11-15) of FIG. 7 is powered by a battery BAT.
The communication processing unit 11 d may incorporate a GPS (Global Positioning System) function well known to mobile phones, smart phones, etc. (Or, a GPS mobile phone function may be installed somewhere in the eyewear 100). If the map service application of the Internet using this GPS function is executed by the processor 11a, the eye compensation rotation at the time of walking can be made to which user the user wearing the eyewear 100 is currently walking while looking at which direction. The noise caused can be suppressed and detected. (Even if the current position of the user can be displayed on the map screen with the conventional mobile phone GPS, it can not be displayed up to where the user's line of sight is facing.)
The eye movement detection unit 15 in FIG. 7 includes four eye movement detection electrodes (151a, 151b, 152a, 152b) that constitute a gaze detection sensor, and three ADCs that extract digital signals corresponding to eye movement from these electrodes ( 1510, 1520, 1512) and a circuit for outputting output data from these ADCs to the processor 11a side. The processor 11a can interpret a command corresponding to the type of eye movement from various eye movements of the user (vertical movement, left and right movement, blinks, eye closing, etc.), and can execute the command.

  As a specific example of a command corresponding to the type of eye movement, if the eye movement is, for example, an eye closing, an information item at the end of the sight line is selected (analogous to one click of a computer mouse), a plurality of consecutive blinks or winks Then there is a command (similar to a computer mouse double click) to start the execution of the process for the selected information item. This command is an example of the information input B using the eye movement detection unit 15.

  FIG. 8 is a diagram showing an example of an EOG waveform in a case where, for example, while gazing at a fixed object in the real world 25 m ahead, the user starts walking from the right foot and stops at the right foot of the 13th step. The compensated rotation of the eye corresponding to the vertical movement of the head accompanying walking is represented by the change in the ADC detection signal level (change in EOG) of Ch1 and Ch2. The vertical movement change of the head synchronized with the EOG change can be detected by the three-axis acceleration sensor possessed by the sensor unit 11 e of the information processing unit 11. In addition, the compensated rotation of the eyeball corresponding to the lateral rotation of the head accompanying walking is represented by the change of the ADC detection signal level (EOG change) of Ch0. The left-right rotation change of the head synchronized with the EOG change can be detected by the three-axis gyro sensor that the sensor unit 11 e of the information processing unit 11 has.

  In the example of FIG. 8, the EOG fluctuation period of Ch1 and Ch2 (about 0.6 seconds) is synchronized with the walking pitch, and the EOG fluctuation period of Ch0 (about 1.2 seconds) is synchronized with twice the walking pitch. ing. Since the EOG fluctuation period corresponding to vertical movement and lateral rotation of the head accompanying walking is clear, it is caused by walking in a period synchronized with the walking pitch (about 0.6 seconds in this example, and twice thereof). EOG fluctuation can be detected.

  FIG. 9 is a diagram showing another example of the EOG waveform (confirmation of repeatability) when starting from the right foot again and stopping at the right foot of the 13th step under the same conditions as FIG. 8. Also in the example of FIG. 9, the EOG fluctuation period of Ch1 and Ch2 (about 0.6 seconds) is synchronized with the walking pitch, and the EOG fluctuation period of Ch0 (about 1.2 seconds) is synchronized with twice the walking pitch. ing. The EOG fluctuation period corresponding to vertical movement and left / right rotation of the head accompanying walking is clearly recognized as well, and caused by walking with a period synchronized with the walking pitch (about 0.6 seconds in this example as well as twice that). Can be detected.

The following features are commonly observed in FIGS. 8 and 9 and it can be confirmed that the event is reproducible:
(1) The eyeball rotational motion in the vertical direction can be confirmed according to the walking cycle (about 0.6 seconds) (refer to the EOG detection signal level of Ch1 and Ch2).
(2) The eyeball rotational motion in the left and right direction can be confirmed in accordance with a cycle (about 1.2 seconds) twice as large as the walking cycle (see the EOG detection signal level of Ch0).

  (3) The amplitude of the EOG detection signal resulting from head sway associated with walking may reach 200 μV to 400 μV. Since the EOG signal amplitude measured when the head is at rest is usually about 1 mV, 200 .mu.V to 400 .mu.V becomes a noise with a non-negligible magnitude when detecting any eyeball rotation. This noise can be canceled (reduced or eliminated) as described below.

  That is, the vertical movement of the head accompanying walking can be detected by the acceleration sensor in a cycle synchronized with the walking pitch (about 0.6 seconds in this example), and another rotation synchronizing the left and right rotation of the head accompanying walking with the walking pitch It can be detected by the gyro sensor in a cycle (about 1.2 seconds in this example). In this case, EOG fluctuation (considered as noise) caused by vertical movement (swing) of the head accompanying walking can be canceled (reduced or eliminated) by the detection result of the acceleration sensor, and the left and right rotation of the head accompanying walking EOG fluctuation (considered as noise) caused by (oscillation) can be canceled (reduced or eliminated) by the detection result of the gyro sensor (a specific example of this cancellation will be described later with reference to FIG. 17).

  FIG. 10 shows the eye movement from the front to the top and the detection signal levels (Ch0, etc.) obtained from the three analog / digital converters (ADCs) shown in FIG. 6 when the noise due to head movement is canceled. It is an electro-oculogram (EOG) which illustrates the relationship between Ch1, Ch2, and the average level of Ch1 and Ch2 (Ch1 + 2). Eye movement detection is performed based on the detection signal waveform in the broken line frame in the figure. The detection criterion is that the user wearing the eyewear 100 is looking straight ahead and there is no eye movement (if the noise due to head sway is canceled, the three ADCs shown in FIG. The output signal waveforms Ch0 to Ch2 are substantially flat in the section looking straight ahead without blinking and hardly change with time). With the line of sight of the left and right eyes of the user facing forward, the line of sight of the two eyes is instantaneously moved up, and the line of sight is kept upward for 1 second, and then the line of sight is immediately returned to the front. Changes in the detection signal level when this is repeated five times are illustrated in FIG.

  FIG. 11 illustrates eye movement detection similar to that of FIG. 10 when the sight line moves downward from the front. If the noise due to head sway is canceled out, even if you are walking, the line of sight is on the basis of the case where the line of sight is facing the front from the waveform changes in FIG. 10 and FIG. You can detect if it is under or under.

  FIG. 12 shows eye movement from left to right and detection signal levels obtained from three ADCs shown in FIG. 6 (Ch0, Ch1, Ch2, and when noise due to head movement is canceled). It is an electro-oculogram which illustrates the relationship with Ch1 + 2). When there is an eye movement from left to right, the change over time of the detection signal waveform of Ch0 rises to the right shoulder (not shown, but when there is an eye movement from right to left, the change over time of the detection signal waveform of Ch0 Is going down the right shoulder). From such a waveform change of Ch0, it is possible to detect whether the line of sight is on the right or to the left based on the case where the line of sight is directed to the front.

If the detection results of FIG. 10 to FIG. 12 are integrated, the following can be understood. That is, if the noise caused by the movement of the head is canceled out, it can be determined whether the line of sight is facing up, down, left, or right based on the case where the line of sight is directed to the front even while walking .
Fig. 13 shows eye movements in which blinks / blinks (both eyes) are repeated five times at 5-second intervals when the line of sight is directed to the front when noise due to head movement is canceled. FIG. 10 is an electrooculogram illustrating the relationship between detection signal levels (Ch0, Ch1, Ch2) obtained from three ADCs shown in FIG. The blink / blink of both eyes can be detected by the pulse appearing on Ch1 and Ch2. Blinks performed by the user involuntarily often have no periodicity. Therefore, the intentional blink / blink of the user can be detected by detecting a plurality of pulses at regular intervals as shown in FIG.

  In addition, according to one theory, the time when the field of vision is interrupted by the time of the "blink / blink" movement of a person or "blink / blink" is about 300 msec (about 100 msec to about 150 msec according to another theory). In either case, a signal component due to “blink / blink” can be removed by a high pass filter (or a band pass filter) below the walking pitch period. The blink / blink interval (for example, about 300 msec) is different from the pitch of normal walking (about 600 msec), and the unintentional blink / blink interval is not synchronized with the normal walking pitch. Therefore, when canceling EOG noise caused by head sway associated with normal walking, EOG waveform change due to blink / blink will not be completely canceled out.

  FIG. 14 shows that when the line of sight is directed to the front when the noise due to head movement is canceled, the one second eye closing (both eyes) and the four second eye opening (both eyes) are repeated five times FIG. 7 is an electro-oculogram illustrating the relationship between the eye movement and the detection signal levels (Ch0, Ch1, Ch2) obtained from the three ADCs shown in FIG. Eye closing in both eyes can be detected by wide pulses appearing in Ch1 and Ch2 (The time to intentionally close the eye is longer than the time to close the eye with blinks / blinks, so the detected pulse width becomes wider) . By detecting the wide pulses of Ch1 and Ch2 as illustrated in FIG. 14, intentional closing of the user can be detected.

Although not shown, a wide pulse with a large amplitude appears on Ch1 when the user only closes the right eye, and a wide pulse with a large amplitude appears on Ch2 when the user closes only the left eye. From this, it is also possible to detect eye closure separately on the left and right.
When only one eye is closed, a small wave with opposite phase on the left and right appears on Ch0. Looking at Ch0, the following small uneven waveform appears.

-Convex waveform because closing the right eye-decreases the positive potential of the electrode;
-Concave waveform because the positive potential of the + electrode decreases when the left eye is closed.
FIG. 15 shows that when the line of sight is directed to the front when the noise due to head movement is canceled, the wink of the left eye (blink of one left eye) is 5 immediately after the blink of both eyes is repeated 5 times. FIG. 7 is an electro-oculogram illustrating the relationship between repeated eye movements and detection signal levels (Ch0, Ch1, Ch2) obtained from three ADCs shown in FIG. 6.

  As illustrated in FIG. 6, the position of the ADC 1512 of Ch0 is offset below the eyeball center line of the left and right eyes. Due to this offset, a potential change appears in the negative direction at both the + input and the − input of the ADC 1512 of FIG. At that time, assuming that the potential changes (amount and direction) of both the + input and the − input are substantially the same, the changes are almost canceled out, and the value of the signal level output from the ADC 1512 of Ch 0 becomes substantially constant ( See the Ch0 level in the left dashed line in FIG. On the other hand, in the blink of one eye (left eye), there is almost no change in potential on the-input side of the ADC 1512, and a relatively large change in negative direction potential appears on the + input side of the ADC 1512. Then, the amount of cancellation of the potential change between the positive input and the negative input of the ADC 1512 is reduced, and a small pulse (small wave of signal level) appears in the negative direction in the signal level output from the ADC 1512 of Ch0 (FIG. See the Ch0 level in the right dashed line). It is possible to detect that the wink of the left eye has been made from the polarity of the small wave (pulse in the negative direction) of this signal level (in the case where the EOG noise due to head sway is canceled, left wink detection using Ch0 An example of

  If the potential changes on the + and-inputs of the ADC 1512 are not equal due to distortion of the user's face or skin condition, the output of the Ch0 ADC is minimum when the user wears the eyewear 100 and blinks simultaneously with both eyes ( Calibration may be performed in advance such that the amount of cancellation between the + input component and the − input component is maximized).

  Further, based on the peak ratio SL1a / SL2a of the detection signal Ch1 / Ch2 when the double-eye blink / light blink is performed, the peak ratio SL1b / SL2b changes when the left-eye wink is performed (SL1b / SL2b Is not equal to SL1a / SL2a). Also from this, the left wink can be detected.

  FIG. 16 shows that the wink (right-handed one-eye blink) is 5 immediately after repeating the blink of both eyes five times when the line of sight is directed to the front when the noise due to head movement is cancelled. FIG. 7 is an electro-oculogram illustrating the relationship between repeated eye movements and detection signal levels (Ch0, Ch1, Ch2) obtained from three ADCs shown in FIG. 6.

  As described above, since the position of the ADC 1512 in FIG. 6 is offset below the eyeball center line of the left and right eyes, a potential change in the negative direction appears in both the + input and the − input of the ADC 1512 in blinks of both eyes simultaneously. However, similar potential changes at the + input and − input are almost canceled out, and the value of the signal level output from the ADC 1512 of Ch 0 becomes substantially constant (see the Ch 0 level in the left dashed line in FIG. 16). On the other hand, in the blink of one eye (right eye), the positive input side of the ADC 1512 hardly changes in potential, and a relatively large negative direction potential change appears on the negative input side of the ADC 1512. Then, the amount of cancellation of the potential change between the − input and the + input of the ADC 1512 is reduced, and a small pulse (small wave of signal level) appears in the positive direction in the signal level output from the ADC 1512 of Ch0 (FIG. 16). See the Ch0 level in the right dashed line). From the polarity of the small wave (positive direction pulse) of this signal level, it is possible to detect that the winking of the right eye has been made (right wink detection using Ch0 in the case where the EOG noise due to head movement is cancelled) An example of

  Further, based on the peak ratio SR1a / SR2a of the detection signal Ch1 / Ch2 when the double-eye blink is performed, the peak ratio SR1b / SR2b changes when the right-eye wink is performed (SR1b / SR2b is SR1a / Not equal to SR2a). Further, the peak ratio SL1b / SL2b at the time of left wink has a value different from the peak ratio SR1b / SR2b at the time of right wink (how much difference can be confirmed by an experiment). From this, the left wink can be detected separately from the right wink (an example of left and right wink detection using Ch1 and Ch2).

The device designer may appropriately determine whether to use Ch0 or Ch1 / Ch2 for the left and right wink detection. The left and right wink detection results using Ch0 to Ch2 can be used as operation commands.
FIG. 17 is a flowchart for explaining an example of processing (noise cancellation processing) for minimizing noise caused by head movement. A computer program corresponding to this flowchart is stored, for example, in the non-volatile memory 11 b of FIG. 7 and executed by the processor 11 a.

First, a user wearing the eyewear 100 as illustrated in FIG. 5 walks at a constant speed with constant rhythm while gazing at a front fixed object (ST10). Along with this walking, the head of the user shakes (moves / rotates) in a predetermined pattern (see FIGS. 1A and 1B).
Movement / rotation of the head of the user accompanying walking is detected by an acceleration sensor / gyro sensor (11e in FIG. 7) provided inside the information processing unit 11 in FIG. 5 (ST12). Movement / rotation of the head of the user accompanying walking is mixed in the EOG detection signal waveform as noise (see FIG. 8 or 9). The detection signal of the acceleration sensor / gyro sensor is added to this EOG detection signal waveform with a predetermined phase (ST14).

As a result of the signal addition, if the signal amplitude increases (NO in ST16), the phase of the detection signal to be added is inverted (ST18), and the detection signal of the acceleration sensor / gyrosensor is added again to the EOG detection signal waveform To do (ST14).
If the signal amplitude decreases as a result of the signal addition (YES in ST16), the level of the signal to be added is adjusted so that the signal amplitude of the addition result becomes minimum (minimum) (ST20). By minimizing (minimizing) the signal amplitude of the addition result, the noise mixed in the EOG detection signal waveform is canceled (reduced or eliminated).

  Based on the EOG detection result after noise due to head movement (movement / rotation) is canceled, the user's eye movement (eye movement in FIG. 10 to FIG. 16, blink, eye closing, wink, etc.) A determination is made (ST22), and a process based on the determination is performed (ST24). For example, when the user winks, it is possible to perform processing such that the name of the building at the end of the line of sight is AR-displayed on the display screen (IM1 / IM2 in FIG. 5) of the eyewear 100.

  When the fluctuation pattern of the user's head changes, the content of the noise cancellation also changes. Therefore, it is detected from the change in the detection result of the acceleration sensor / gyro sensor whether the movement pattern of the user's head has changed (ST26). For example, it is assumed that a user of eyewear 100 who is walking at a constant speed while gazing at a fixed object in front stops in front of the down stairs going down to the subway, and sweeps the steps of adjacent steps of the stairs with forward and backward bending and gaze movement. . Then, the motion pattern of the head is different from that during walking due to the line-of-sight sweep before and after this step, and the EOG detection signal waveform is also different from that during walking.

  Therefore, when the motion pattern of the user's head changes (YES in ST26), the process returns to step ST12, and the processing in ST14 to ST20 is performed again. By repeating the processing of ST14 to ST20, EOG noise caused by the new head movement pattern is canceled (reduced or eliminated), and EOG corresponding to pure eye movement is detected. The EOG detection result corresponding to this pure line-of-sight movement is compared with the detection result of the acceleration sensor / gyro sensor corresponding to the changed movement pattern of the user's head. If, as a result of this comparison, phase inversion from FIG. 4A to FIG. 4B is detected at the step position of the stairs, for example, it is determined that there is a step difference in the vicinity of the phase inversion point (ST22 ). Then, a mark indicating the presence of the step is AR-displayed by the eyewear 100 (ST24). The AR display can be automatically erased after a predetermined time (for example, 10 seconds) after the detection result of the acceleration sensor / gyro sensor corresponding to the movement pattern of the user's head disappears.

  The on / off of the AR display can also be automatically performed depending on whether the EOG waveform (Ch1 / Ch2 waveform in FIG. 8 or 9) and the detection signal waveform of the acceleration sensor are synchronized. When the user is looking at the AR display, the eyeball does not rotate to compensate for head movement, so there is no synchronization relationship as above, but when the user is looking at the real world instead of the AR display, the above A synchronization relationship occurs. With the same idea, the AR display can be automatically turned on / off depending on whether the EOG waveform (Ch0 waveform in FIG. 8 or 9) and the detection signal waveform of the gyro sensor are synchronized.

  If the user continues using the eyewear while the motion pattern of the user's head does not change (NO in ST28), the processing in ST22 to ST24 continues. If the user instructs the processor 11a in FIG. 7 to finish using the eyewear 100 (YES in ST28), the process in FIG. 17 ends.

FIG. 18 is a view for explaining an implementation example of the EOG electrode in the eyeglass-type eyewear 100 according to another embodiment (an example in which the EOG electrode is disposed around the eye). The eyewear 100 of FIG. 18 differs from the eyewear 100 of FIG. 5 in the following respects.
The first difference is that the EOG electrodes 151a and 151b of the right nose pad 150R move toward the right eye frame 101, and the EOG electrodes 152 a and 152 b of the left nose pad 150L move toward the left eye frame 102. . EOG electrodes 151a and 151b are arranged at positions substantially symmetrical with respect to the user's right eye center position (not shown), and EOG electrodes 152a and 152b are with respect to the user's left eye center position (not shown). It is disposed at a position that is substantially point-symmetrical. The right oblique lines connecting the EOG electrodes 151a and 151b and the left oblique lines connecting the EOG electrodes 152a and 152b are substantially line symmetrical with respect to a vertical line (not shown) along the user's nose. The electrodes 151a, 151b, 152a, 152b can be made of a conductive member (metal, conductive polymer, etc.) provided at the tip of an elastic body (sponge, cushion made of silicone, etc.). Each EOG electrode is lightly pressed to the skin surface of the face of the user wearing the eyewear 100 by the elastic repulsive force of the elastic body.

  When the EOG electrode arrangement structure as shown in FIG. 18 is adopted, it becomes easier to detect the electric field generated around the charged eyeball compared with the structure in which the EOG electrode is arranged in the nose pad portion. Therefore, the embodiment of FIG. 18 can detect an EOG signal of larger amplitude than the embodiment of FIG.

  The second difference is that the sensor unit 11 e including an acceleration sensor and a gyro sensor is left on the bridge 103 and the other functions of the information processing unit 11 are shifted to the temple bar 107 side. For example, when the physical size of the information processing unit 11 is increased due to, for example, the enhancement of the information processing capability or the addition of the GPS function, the attached information processing unit 11 is attached to the bridge 103. There may be a problem with the feeling. If the structure attached to the bridge 103 is only the sensor unit 11e, the structure on the bridge 103 can be smaller and lighter than providing the entire information processing unit 11, so the designability and fit may be improved. is there. On the other hand, within the temple bar 107 (and / or 106), even if a relatively large structure is provided, there is little possibility that the design of the eyewear 100 and the feeling of wearing thereof will be problematic.

  FIG. 19 is a view for explaining an implementation example of EOG electrodes in goggle-type eyewear (type in which left and right eye frames are continuous) 100 according to still another embodiment (other EOG electrodes are arranged around the eye) Example). When the structure as shown in FIG. 19 is adopted, the skin contact stability of the EOG electrode becomes higher than that of the structure of FIG. Therefore, it is possible to detect the EOG signal with higher accuracy while canceling out the noise caused by the head movement. Further, in the structure of FIG. 19, the enlarged information processing unit 11 and other devices can be easily incorporated. Therefore, in the structure of FIG. 19, the function of the smartphone with GPS can be loaded without concern for the design. The screen display in that case can be performed by AR display on the left and right displays 12L / 12R using film liquid crystal or the like. Although not shown, small speakers can be attached near the left and right ears of the goggle frame and a small microphone can be attached near the nose cushion. In addition, command input to the smartphone can be performed by eye movement such as eye movement, blink, eye closing, wink and the like.

FIG. 20 is a view for explaining an implementation example of the EOG electrodes in the goggle type eyewear (the type in which the eye cups of the left and right eyes are separated) 100 according to still another embodiment (the EOG electrodes are arranged around the eye) Example). In the embodiment of FIG. 20, the EOG electrode arrangement structure and the arrangement structure of the sensor unit 11e are adopted, which are similar to the case of FIG. However, the goggle structure of FIG. 20 can withstand underwater use (or use of space swimming training). Although the head of a diver who swims in water vibrates more than walking on the ground, the large fluctuation (positional movement or rotation in the vertical and horizontal directions) can be detected by a three-axis acceleration sensor or a three-axis gyro sensor in the sensor unit 11e. The detected motion component can be used to cancel the noise component mixed in the EOG signal. Various commands can be input based on the eye movement (blinks, eye closing, etc.) of the user (diver) whose both hands are blocked by using the EOG signal in which the noise caused by the head movement is suppressed. .
<Late of embodiment>
(A) The conventional gaze detection technology includes a method of estimating the gaze direction by image processing using an infrared camera, but the amount of required equipment (in addition to the high brightness infrared LED, the entire eyeball seen outside is Cameras capable of photographing, relatively high-performance arithmetic devices for image processing, and power supply devices for operating them become large scale. Therefore, although the conventional gaze detection technology is applied to stationary type devices, its application to wearable devices is not advanced.

  On the other hand, eye-gaze detection by electro-oculogram sensing requires much less amount of devices (electrodes, ADCs, relatively low-performance arithmetic devices, power supply devices that operate these) compared to infrared camera methods The scale is also applicable to wearable devices such as AR glasses. EOG is a technology that can be expected to be applied to wearable devices because it has high affinity with AR glasses and lower power consumption compared to infrared methods.

(B) It cancels the shaking of the head of the eyewear user when gazing at a fixed object in the real world. This improves the detection accuracy for arbitrary eyeball rotation.
(C) Detecting whether the fixed object in the real world that is gazing at is a distant view or a near view (detection of phase inversion point in FIG. 4). Thereby, the presence or absence of gaze of the real world (fixed object) is detected from the shaking of the head of the eyewear user.

(D) Use the shaking of the eyewear user's head for automatic on / off control of the AR display.
(E) Extract only arbitrary eye movement that has compensated for the compensated rotation of the eye.
(F) It is detected whether the subject is gazing at the real world or not (If the detection waveform of the acceleration sensor and / or the gyro sensor is synchronized with the electrooculogram EOG waveform at the time of walking, it looks at something in the real world It is determined that the detection waveform of the acceleration sensor and / or the gyro sensor and the electrooculogram EOG waveform are not synchronized, it is determined that the AR display or the like is seen during walking).

(G) Effective as noise cancellation when using eye movement as UI (user interface).
(H) When used for collecting work history information using eyewear, it becomes an index that has never been seen (for example, in the picking operation of a warehouse, etc. with the influence of noise due to head sway eliminated) It can be used for a pass / fail judgment such as “passing as a worker” if the frequency with which the line of sight is directed is a certain value or more
<Example of correspondence relationship between contents of the claim in the initial application and the embodiment>
[1] An electro-oculogram detection apparatus according to an embodiment (11 in FIG. 7: including a processor 11 a for executing the process of FIG. 17) is based on the electro-oculography (EOG) of the user wearing the eyewear (100) Eye movement detection unit (15; 151a, 151b, 152a, 152b) for detecting eye movement (gaze movement, blink etc.) including the eyeball rotation of the user, and an acceleration sensor (11e) for detecting movement of the eyewear And a gyro sensor (part of 11e) for detecting the rotation of the eyewear. The eye movement detecting device is a detecting unit that detects movement and / or rotation of the eyewear by the acceleration sensor and / or the gyro sensor when the eye movement is detected based on the electro-oculogram (EOG). (11a executing ST12) noise added to the electrooculogram due to movement and / or rotation of the eyewear (eoptoelectric component added by compensatory eye rotation acting in a direction to cancel head movement Noise reduction means (ST14 to ST20) for reducing (minimizing or canceling) by combining the signal components (components that are in the reverse phase to the mixed noise) detected by the acceleration sensor and / or the gyro sensor. 11a).

  [2] The apparatus according to the above [1] selects a phase of the signal component detected by the acceleration sensor and / or the gyro sensor such that the result of signal synthesis by the reduction means decreases, (ST16- It further comprises 11a) for performing ST18.

[3] The device (11) of the above [1] is incorporated in eyewear (100 in FIGS. 5 and 18 to 20).
[4] The eyewear (100) of [3] has a display unit (12L, 12R), and displays (AR display) based on the electrooculogram (EOG) after noise is reduced by the noise reduction means And display means (11a for executing ST22 to ST24) for performing the above in the display unit (12L, 12R).

  [5] The apparatus according to the above [1] detects the rotation corresponding to the back-and-forth bending change of the head of the user by the gyro sensor (11e), and changes the sight direction of the user (FIG. 10, FIG. 11: Ch1 The eye movement detection unit (15) can detect the change in the vertical line-of-sight direction from the ± level change of / Ch2, and the signal phase of the longitudinal bending change and the signal phase of the line-of-sight direction change are in phase (FIG. a)) It is configured to identify the perspective of the place where the user is looking by whether it is the opposite phase (FIG. 4 (b)) Can be detected).

  [6] The apparatus according to the above [1], wherein when the eye movement is detected based on the electro-oculogram (EOG), movement of the eyewear detected by the acceleration sensor and / or the gyro sensor and / or Or, if the rotation (FIG. 1) is synchronized with the change in the electro-oculogram (EOG) (FIG. 8 or 9) (if it is possible to maintain the “predetermined phase relationship” in ST14 of FIG. It is configured to determine that it is looking (it can detect whether what you are looking at is the real world or not).

  [7] An eye potential detection method according to an embodiment includes eye movement (eye movement, blink, etc.) including eye rotation of the user based on eye potential (EOG) of the user wearing the eyewear (100) And detecting movement of the head of the user wearing the eyewear, and detecting rotation of the head of the user wearing the eyewear. This method comprises the steps of detecting movement and / or rotation of the head of the user wearing the eyewear when the eye movement is detected based on the electro-oculogram (EOG) (ST12); Noise mixed in the eye potential due to movement and / or rotation of the head of the user wearing the clothing (an eye potential component added by compensatory eye rotation acting in a direction to cancel head movement), Step of reducing (minimizing or canceling) by combining signal components (components that are in anti-phase to mixed noise) detected corresponding to movement and / or rotation of the head of the user wearing the eyewear (ST 14- ST20) is included.

[8] The method according to the above [7], wherein the phase of the signal component detected corresponding to the movement and / or rotation of the head of the user wearing the eyewear causes the synthesis result of the signal component to decrease. Are further included (ST16 to ST18).
While several embodiments of the invention have been described, these embodiments have been presented by way of example only, and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the scope of the invention.

For example, in the description of the embodiment, a glasses-type or goggle-type device is introduced as an eyewear having an EOG electrode, but in addition to that, an EOG electrode is an article having a form such as an eye mask, an eyepiece, a helmet, a hat or a hood It is also conceivable to use it for attached eyewear.
These embodiments and modifications thereof are included in the scope and the gist of the invention, and are included in the invention described in the claims and the equivalent scope thereof. In addition, combining some or all of one embodiment among the plurality of disclosed embodiments and some or all of another embodiment among the plurality of disclosed embodiments is also within the scope of the invention. And included in the abstract.

  100: eyewear (goggle type or glasses type) 110: eye frame 11: information processing unit (processor 11a, non-volatile memory 11b, main memory 11c, communication processing unit (GPS processing) which is a main part of the eye movement detection apparatus 11) integrated circuit including sensor 11e, etc. 11e: sensor including acceleration sensor, gyro sensor etc. BAT: power source (lithium ion battery etc.) 12: display (right display 12R and left display 12L: Film liquid crystal etc.) IM1 ... right display image (ten key, alphabet, character string, mark, icon etc) IM2 ... left display image (ten key, alphabet, character string, mark, icon etc); 13 ... camera (right camera 13R And mounted on the left camera 13L or bridge 103 Center camera (not shown); 15: eye movement detection unit (line of sight detection sensor); 1510: right (Ch1) AD converter; 1520: left (Ch2) AD converter; 1512: left and right (Ch0) AD converter; Left to right (Ch3) AD converter.

Claims (13)

A display unit for displaying a real world image or an augmented reality image for adding image information to the real world image;
A first detection unit that detects eye movement of the user;
A second detection unit that detects the movement of the head of the user;
A determination unit that determines what the user is looking at based on the eye movement detected by the first detection unit and the head movement detected by the second detection unit;
A eye movement detecting apparatus comprising,
The determination unit determines whether the user is in the real world based on whether the eye movement detected by the first detection unit and the head movement detected by the second detection unit are synchronized. Determine if you are looking at the image or looking at the augmented reality image,
The display unit is responsive to the determination result by the determination unit as to whether or not the eye movement detected by the first detection unit and the movement of the head detected by the second detection unit are synchronized. Eye movement detection device which turns on and off the display of the augmented reality image . The determination means
When the eye movement detected by the first detection unit and the movement of the head detected by the second detection unit are synchronized, it is determined that the user is looking at the real world image,
When it is determined that the user is looking at the augmented reality image when the eye movement detected by the first detection unit and the head movement detected by the second detection unit are not synchronized The eye movement detection device according to item 1 .   The determination unit is configured to determine whether the phase of the eye movement detected by the first detection unit is in phase with the phase of the head movement detected by the second detection unit. The eye movement detection device according to claim 1, wherein the perspective of an object being viewed is determined. The determination means
When the phase of the eye movement detected by the first detection unit and the phase of the movement of the head detected by the second detection unit are in phase, the user is at a short distance in the real world image Determine that you are looking at an object,
When the phase of the eye movement detected by the first detection unit and the phase of the movement of the head detected by the second detection unit are opposite in phase, the user is at a far distance in the real world image The eye movement detection apparatus according to claim 3, wherein it is determined that the user is looking at an object of The display unit is configured to determine that the phase of the eye movement detected by the first detection unit and the phase of the head movement detected by the second detection unit change from the same phase to the opposite phase. The eye movement detection device according to claim 3 or 4, which turns on the display of the augmented reality image when it is detected. 3. The image processing apparatus according to claim 1, further comprising: a compensation unit configured to reduce noise included in the eye movement detected by the first detection unit based on the movement detected by the second detection unit. The eye movement detection device according to claim 4 or 5 . The first detection unit detects eye movement in the vertical direction and eye movement in the horizontal direction,
When the determination means determines that the user is looking at the real world image while walking , the eye movement in the vertical direction and the horizontal direction detected by the first detection unit and reduced in noise by the compensation means The eye movement detection device according to claim 6 , wherein it is determined based on eye movement that the fixation point of the user has moved in the vertical direction or in the horizontal direction. Displaying a real world image or an augmented reality image for adding image information to the real world image by a display unit;
Detecting a user's eye movement by the first detection unit;
Detecting the movement of the head of the user by a second detection unit;
And determining by the first based on the motion of the head detected by the second detector and said detected Meadow by the detection unit, determining means what the user is looking,
A eye movement detection method comprising the,
The determining may be performed by the user based on whether the eye movement detected by the first detection unit and the head movement detected by the second detection unit are synchronized. Determining whether the world image is viewed or the augmented reality image is viewed by the determining means;
The displaying may be based on a determination result by the determination unit as to whether or not the eye movement detected by the first detection unit and the head movement detected by the second detection unit are synchronized. Turning on or off the display of the augmented reality image accordingly;
Eye movement detection method comprising. The method further includes reducing noise included in the eye movement detected by the first detection unit by the compensation unit based on the movement detected by the second detection unit.
The detection of the user's eye movement by the first detection unit comprises detecting an eye movement in the vertical direction and an eye movement in the left and right direction,
In the determination, when the determination means determines that the user is looking at the real world image during walking, the first detection unit detects the noise and the compensation means reduces the noise in the vertical direction. based on the Meadow and lateral directions Meadow, claim 8 eye movement detecting method according determining by said determining means that the gazing point of the user moves in the vertical direction or the horizontal direction.   The above determination is
    When the eye movement detected by the first detection unit and the movement of the head detected by the second detection unit are synchronized, it is determined that the user is looking at the real world image,
    When the eye movement detected by the first detection unit and the head movement detected by the second detection unit are not synchronized, it is determined that the user is looking at the augmented reality image The eye movement detection method according to claim 8, comprising   The determining may be performed based on whether or not the phase of the eye movement detected by the first detection unit and the phase of the head movement detected by the second detection unit are in phase. The eye movement detection method according to claim 8, wherein the perspective of the object being viewed is determined.   The above determination is
    When the phase of the eye movement detected by the first detection unit and the phase of the movement of the head detected by the second detection unit are in phase, the user is at a short distance in the real world image Determine that you are looking at an object,
    When the phase of the eye movement detected by the first detection unit and the phase of the movement of the head detected by the second detection unit are opposite in phase, the user is at a far distance in the real world image The eye movement detection method according to claim 11, wherein it is determined that the user is looking at an object.   It is determined that the display changes the phase of the eye movement detected by the first detection unit and the phase of the movement of the head detected by the second detection unit from the same phase to the opposite phase. The eye movement detection method according to claim 11 or 12, wherein the display of the augmented reality image is turned on when is detected.

Images