JP6830981B2 - Wearable device and display method - Google Patents

Description

Embodiments of the present invention relate to wearable devices and display methods.

Normally, when a person walks while looking forward, the person's head sways up, down, left, and right, but the person can clearly see the surrounding scenery. This is because the sway of the head during walking is suppressed to a range smaller than other parts of the body (amplitude of about 2 to 3 cm regardless of the distance of the line of sight), and the head and eyes are stable in the line of sight within that small range. This is because they work cooperatively to maintain the above (see Non-Patent Document 1). This non-patent document 1 teaches that "the head sways up, down, left and right during walking, and rotation of the head and compensation rotation of the eyeball occur as an action of stabilizing the line of sight in response to this sway."

Regarding the electro-oculography sensing technique, there is a known example 1 in which the line-of-sight position of a person is detected based on the electrooculogram caused by the movement of the eyeball of the person (see Patent Document 1). In this known example 1, an EOG (Electro-Oculography) electrode is used for electro-oculography detection (electro-potential sensing), and the user's line-of-sight position is detected from the electro-oculography data acquired from the EOG electrode (paragraph 0025). In the embodiment of Known Example 1, the EOG electrode is provided on the goggles (paragraph 0061), but there is also Known Example 2 in which the electrode for EOG is provided on the glasses (see Patent Document 2). Goggles and glasses are a type of eyewear worn near the human eyeball. When a user wearing eyewear walks, the eyewear also shakes with the head.

Japanese Unexamined Patent Publication No. 2009-288529 Japanese Unexamined Patent Publication No. 2013-244370

Head movements and eye movements that maintain line-of-sight stability during walking 2000-03 Yazaki, Sharpya Osaka University Graduate School of Human Sciences Note. 26 P.177-P.193 http://ir.library.osaka-u .ac.jp/dspace/bitstream/11094/5672/1/hs26-177.pdf

In a stationary sitting state (a state in which disturbing movements are not mixed in the head of a user wearing eyewear with EOG electrodes), eye potential / eye rotation can be stably detected by electro-oculography sensing. However, when gazing at a fixed object in the real world in a walking state (a state in which the head of a user wearing EOG electrode-equipped eyewear sways up, down, left, and right), the target line of sight / eyeball is compared with the sitting state. It becomes difficult to detect the rotation. The reason is that eye rotation is unconsciously performed as a complementary operation for stabilizing the line of sight of the human body, and the signal component of this unconscious eye rotation is superimposed (as noise or disturbance) on the electro-oculography sensing result. Is.

That is, when a user wearing eyewear with EOG electrodes walks, even if the user fixes his / her line of sight to a specific object in front of him / her, the electrocardiographic data acquired from the EOG electrodes also fluctuates due to the shaking of the head accompanying walking. .. The fluctuation of the electrooculogram data becomes noise (disturbance) with respect to the electrooculogram data corresponding to the actual line of sight.

One of the problems to be solved by the embodiment of the present invention is to provide a wearable device and a display method for changing the display content by shaking the head.

The wearable device according to the embodiment includes a display unit that displays a real-world image or an augmented real-world image that adds image information to the real-world image, a first detection unit that detects the movement of the user's head, and the user's wearable device. A second detection unit for detecting the movement of the line of sight is provided, and a component related to the movement of the head included in the detection result of the second detection unit is reduced based on the detection result of the first detection unit, and the component is described. The position on the line of sight of the user in the real-world image is detected based on the detection result of the second detection unit in which the amount is reduced, and the second detection unit in which the component is reduced is detected at the detected position. The augmented reality world image including the image information corresponding to the movement of the line of sight of the user is displayed.

The figure explaining the head sway (position change by up / down / left / right movement) of a person walking while looking forward. The figure explaining the head sway (angle change by forward bending / backward bending) of a person who is gazing at the fixed object in front while walking. The figure explaining the relationship between the target distance (distance from the eye to the gaze target) and the eye movement when the head bends back and forth in the vertical direction. The figure which exemplifies the phase relationship between the angle change / speed change of head anterior-posterior bending and the angle change / speed change of eye movement. The figure explaining the eyeglass type eyewear which incorporated the eye movement detection device which concerns on one Embodiment (an example which EOG electrode was arranged in the nose pad). The figure explaining the mounting example of the EOG electrode in the eyeglass type eyewear which concerns on one Embodiment. The figure explaining the relationship between the information processing unit 11 which can be attached to various embodiments, and the peripheral device thereof. For example, a diagram showing an example of an EOG waveform when a person starts walking from the right foot and stops at the right foot on the 13th step while gazing at a fixed object in the real world 25 m away. The figure which shows another example (confirmation of repetitive reproducibility) of the EOG waveform at the time of starting walking from the right foot again and stopping at the right foot of the 13th step under the same conditions as FIG. When the noise caused by the shaking of the head is canceled, the eye movement from the front to the upper side and the detection signal levels (Ch0, Ch1, Ch2,) obtained from the three analog-to-digital converters (ADCs) shown in FIG. And an electrocardiogram (EOG) illustrating the relationship between Ch1 and the average level of Ch2 (Ch1 + 2). When the noise caused by the shaking of the head is canceled, the eye movement from the front to the lower side and the detection signal levels (Ch0, Ch1, Ch2, and Ch1 + 2) obtained from the three ADCs shown in FIG. 6 An electrocardiogram exemplifying the relationship. When the noise caused by the shaking of the head is canceled, the eye movement from left to right and the detection signal levels (Ch0, Ch1, Ch2, and Ch1 + 2) obtained from the three ADCs shown in FIG. 6 An electrocardiogram exemplifying the relationship. In the case where the noise caused by the shaking of the head is canceled, when the line of sight is facing the front, the blinking (both eyes) is repeated 5 times at 5 second intervals, and from the three ADCs shown in FIG. An electrocardiogram illustrating the relationship with the detection signal level (Ch0, Ch1, Ch2) obtained from the above. When the noise caused by the shaking of the head is canceled, when the line of sight is facing the front, 1 second of eye closure (both eyes) and 4 seconds of eye opening (both eyes) are repeated 5 times. , An electrocardiogram illustrating the relationship with the detection signal levels (Ch0, Ch1, Ch2) obtained from the three ADCs shown in FIG. When the noise caused by the shaking of the head is canceled, the eyes that repeat the blink of the left eye (blink of the left one eye) 5 times immediately after repeating the blink of both eyes 5 times when the line of sight is facing the front. An electrocardiogram illustrating the relationship between motion and detection signal levels (Ch0, Ch1, Ch2) obtained from the three ADCs shown in FIG. When the noise caused by the shaking of the head is canceled, when the line of sight is facing the front, the right eye blink (right one eye blink) is repeated 5 times immediately after repeating the blinks of both eyes 5 times. An electrocardiogram illustrating the relationship between motion and detection signal levels (Ch0, Ch1, Ch2) obtained from the three ADCs shown in FIG. A flowchart illustrating an example of a process (noise canceling process) that minimizes noise caused by head sway. The figure explaining the mounting example of the EOG electrode in the eyeglass type eyewear which concerns on another embodiment (the example which the EOG electrode is arranged around the eyeball). The figure explaining the mounting example of the EOG electrode in the goggle type eyewear (the type in which the eye frames of both left and right eyes are continuous) according to still another embodiment (another example in which the EOG electrode is arranged around the eyeball). The figure explaining the mounting example of the EOG electrode in the goggle type eyewear (the type in which the eye cups of both left and right eyes are separated) according to still another embodiment (the other example in which the EOG electrode is arranged around the eyeball).

Hereinafter, basic information will be provided first, and then various embodiments will be described with reference to the drawings. <Basic information>
The diameter of an adult eyeball is about 25 mm. It is about 17 mm after birth and grows as it grows.
The interpupillary distance of an adult male is about 65 mm. (Many commercially available stereo cameras are made at intervals of 65 mm.)
The interpupillary distance of adult females is several millimeters shorter than that of males.
The electrooculogram is several tens of mV.
The eyeball has a positive potential on the corneal side and a negative potential on the retina side. When this is measured on the surface of the skin, it appears as a potential difference of several hundred μV.

The types of eye movements and the range of eye movements related to eye movement detection include, for example: <Types of eye movements (eye movements)> (01) Compensatory eye movements For head and body movements Regardless, involuntary eye movements developed to stabilize the image of the outside world on the retina. (02) Voluntary eye movement This is an eye movement developed to bring the visual pair image to the center on the retina, and is a movement that can be controlled voluntarily. (03) Impact eye movement (saccade)
Eye movement (easy to detect) that occurs when changing the gazing point to see an object. (04) Sliding eye movement Smooth eye movement that occurs when tracking a slowly moving object (difficult to detect). <Eye movement range (for general adults)> (11) Horizontal left direction: 50 ° or less Right direction: 50 ° or less (12) Vertical direction Down direction: 50 ° or less Up direction: 30 ° or less (self The vertical angle range that can be moved by the intention of is narrow only in the upward direction. (Because there is a "bell phenomenon" in which the eyeballs turn up when the eyes are closed, the vertical eyeball movement range shifts upwards when the eyes are closed.) (13 ) Others Convergence angle: 20 ° or less. <About the movement of the head accompanying walking (Sources in FIGS. 1 and 2 are non-patent documents 1)>
FIG. 1 is a diagram for explaining head sway (position change due to vertical / horizontal movement) of a person walking while looking forward. The position of the human head changes up and down due to the movement of the lower limbs during walking (Fig. 1 (a)). A general adult walking cycle (1 walking cycle) is, for example, about 0.6 seconds. At the same time, in this walking, the position of the human head also changes to the left and right (Fig. 1 (b)). The cycle of this left-right position change is about 1.2 seconds, which is twice the walking cycle. The vertical position change width and the left and right position change width are about 2 to 3 cm, respectively. This vertical and horizontal movement is almost linear and regular, and has a cycle closely related to the walking cycle.

FIG. 2 is a diagram for explaining head sway (angle change due to forward bending / backward bending) of a person who is gazing at a fixed object in front while walking. When a person who is gazing at a visual object (target) in front starts walking, the position of the head changes up and down as the person walks (FIG. 2A). While gazing at the optotype, when the head comes to the upper position, the head bends forward (pitch down), and when the head comes to the lower position, the head bends backward (pitch up). (Fig. 2 (b)). Such anterior-posterior bending (vertical rotational movement of the head) occurs in correspondence (synchronous) with the vertical movement of the head during one walking cycle. Similarly, when the position of the head changes from side to side (FIG. 1 (b)), a rotational movement in the left-right direction (synchronized with the vertical movement of the head) occurs in the head.

That is, while walking, the human head performs a rotational movement that compensates for the vertical and horizontal movements, and the head moves back and forth at high and low positions so that the line of sight helps to catch the visual object (target) in front. Succumb. The same applies to the left and right, and when the body swings left and right and the head moves left and right as the person walks, the head rotates left and right (reciprocating rotation). This rotation of the head is called "Compensatory Rotation" and is thought to help stabilize the line of sight.

In addition, since the human trunk rotates back and forth and left and right while walking, the head needs to compensate for these as well. By rotating the head in the opposite direction in response to this change in the trunk, the angular position of the head is kept horizontal. <About eye movements associated with walking (sources of FIGS. 3 and 4 are non-patent documents 1)>
FIG. 3 is a diagram for explaining the relationship between the target distance (distance from the eye to the gaze target) and the eye movement when the head bends back and forth in the vertical direction. Here, the target position when the head is fixed is HFP, and the anteroposterior flexion rotation angle (φh) of the head when the actual target is closer (Near Target) and farther (Far Target) than the HFP. And the compensatory rotation angle (φen, φef) of the eyeball are illustrated.

As mentioned above, the head makes compensatory rotation, but it cannot completely achieve line-of-sight stabilization by itself. The head often causes excess or deficiency when rotating at the cost of vertical movement. In order to compensate for the deviation due to the excess or deficiency of the compensatory rotation, periodic compensatory rotation of the eyeball is required during walking. For example, for a near target (Near Target) 30 cm away, the eyeball performs compensatory rotation (φen) in the same direction as the rotation / rotation (φh) of the head. As a result, even if the rotation / rotation of the head is insufficient, the eyeball rotates in coordination with the head in the direction of movement to compensate for the insufficient rotation of the head. On the other hand, for example, for a distant target (Far Target) 100 m away, the eyeball performs compensatory rotation (φef) in the direction opposite to the rotation / rotation (φh) of the head. As a result, even if the rotation / rotation of the head becomes excessive, the eyeball rotates in the opposite direction to cancel the excessive rotation of the head. (It should be noted that there is no substantial difference in the amount of movement of the head during walking regardless of whether the target is near or far.)
FIG. 4 is a diagram illustrating the phase relationship between the angle change / speed change of the anterior-posterior bending of the head and the angle change / speed change of the eye movement. The anteroposterior bending angle (φh) of the head shown in FIG. 3 can be detected by a 3-axis gyro (11e in FIG. 7) attached to the eyewear. Further, the rotation angle (φen, φef) of the eyeball shown in FIG. 3 can be detected by the eye movement detection unit (15 in FIG. 7) using the EOG of the eyeball.

The change in vertical movement of the head (change in longitudinal bending in the vertical direction) associated with walking changes in phase with the change in eye rotation when the target is close (Near Target) (Fig. 4 (a)). On the other hand, when the target is far (Far Target), the change in vertical movement of the head (change in longitudinal bending in the vertical direction) changes in the opposite phase to the change in eye rotation (FIG. 4 (b)). When the optotype is at the fixed position HFP in FIG. 3, the eyeball rotation disappears, and the waveform change of the broken line shown in FIGS. Then, the phase between the change in the longitudinal bending of the head and the change in the rotation of the eyeball is reversed). Similar phase inversion is also seen between the speed change of head up and down movement and the speed change of eyeball rotation (FIGS. 4 (c) and 4 (d)).

Now, the signal change that occurs in the EOG due to the above-mentioned "vertical movement change of the head accompanying walking" will be referred to as "body movement noise". Walking behavior can be detected by an acceleration sensor or a gyro sensor, and eye movement behavior can be detected by EOG. Then, when "walking" and "no eye movement" are detected and no body movement noise appears in the EOG, "the eyewear user sees the phase of the body movement noise depending on the distance of the optotype. We are watching the point (corresponding to the position HFP in FIG. 3). "

FIG. 5 is a diagram illustrating a glasses-type eyewear 100 incorporating an eye movement detection device according to an embodiment (an example in which an EOG electrode is arranged on a nose pad). In this embodiment, the right eye frame (right rim) 101 and the left eye frame (left rim) 102 are connected to the bridge 103. The left and right eye frames 102, 101 and the bridge 103 can be made of, for example, an aluminum alloy or titanium. The left outer side of the left eye frame 102 is connected to the left temple bar 106 via a left hinge 104, and a left modern (left ear pad) 108 is provided at the tip of the left temple bar 106. Similarly, the right outer side of the right eye frame 101 is connected to the right temple bar 107 via the right hinge 105, and the right modern (right ear pad) 109 is provided at the tip of the right temple bar 107.

The information processing unit 11 (integrated circuit of several millimeters square), which is the main part of the eye movement detection device, is attached to the bridge 103. The information processing unit 11 can be configured by an LSI in which a microcomputer, a memory, a communication processing unit, and the like are integrated (details of the information processing unit 11 will be described later with reference to FIG. 7).

A small battery (BAT) such as a lithium-ion battery is embedded in, for example, the left temple bar 106 (or the right temple bar 107, or the modern 108 or 109) to serve as a power source necessary for the operation of the eyeglass-type eyewear 100. ing.
A left camera 13L is attached to the end of the left eye frame 102 near the left hinge 104, and a right camera 13R is attached to the end of the right eye frame 101 near the right hinge 105. These cameras can be configured using an ultra-small CCD image sensor.

These cameras (13L, 13R) may constitute a stereo camera. Alternatively, an infrared camera (13R) and a laser (13L) may be arranged at the positions of these cameras to form a distance sensor using an infrared camera and a laser. This distance sensor can also be composed of a small semiconductor microphone (13R) that collects ultrasonic waves and a small piezoelectric speaker (13L) that emits ultrasonic waves.

An embodiment in which a central camera (not shown) is provided on the bridge 103 portion instead of the left and right cameras 13L / 13R or in addition to the left and right cameras 13L / 13R is also conceivable. Conversely, there may be embodiments in which no cameras are provided (these cameras are shown as camera 13 in FIG. 7).

The left display 12L is fitted in the left eye frame 102, and the right display 12R is fitted in the right eye frame 101. This display is provided on at least one of the left and right eye frames, and can be configured by using a film liquid crystal or the like. Specifically, one or both of the left and right displays 12L and 12R can be configured by using a film liquid crystal display device that employs a polymer-dispersed liquid crystal (PDLC) that does not use a polarizing plate (this display is displayed in FIG. 7). (Shown as 12).

The transparent left and right displays 12L / 12R using a film liquid crystal can be used as a means for providing augmented reality (AR) that adds image information such as numbers and characters to the real world seen through glasses.
A nose pad portion is provided between the left and right eye frames 102 and 101 and below the bridge 103. This nose pad portion is composed of a pair of a left nose pad 150L and a right nose pad 150R. The right nose pad 150R is provided with right nose pad electrodes 151a and 151b, and the left nose pad 150L is provided with left nose pad electrodes 152a and 152b.

These electrodes 151a, 151b, 152a, 152b are electrically separated from each other and connected to three AD converters (ADC 1510, 1520, 1512) via an insulated wiring material (not shown). The outputs from these ADCs have different signal waveforms according to the movement of the eyes of the user wearing the eyewear 100, and are supplied to the information processing unit 11 as digital data according to the movements of the eyes of the user. The electrodes 151a, 151b, 152a, and 152b are used as eye-gaze detection sensors, and together with the three AD converters, are components of the eye movement detection unit 15 in FIG.

The eyewear 100 is fixed to the user's head (not shown) by the left and right nose pads (150L, 150R), the left and right temple bars (106, 107), and the left and right modern (108, 109). In this embodiment, only the left and right nose pads (150L, 150R), the left and right temple bars (106, 107), and the left and right modern (108, 109) need to directly touch the user's head (or face). However, there may be an embodiment in which parts other than those (nose pad, temple bar, modern) touch the user in order to adjust the voltage between the ADC (1510, 1520, 1512) and the user's body.

The film liquid crystal of the right display 12R of FIG. 5 can display a right display image IM1 including, for example, a numeric keypad (numbers, operators, Enter, etc.), alphabets, marks of a predetermined shape, and other icon groups. Further, the film liquid crystal of the left display 12L can display the left display image IM2 including, for example, an arbitrary character string, a mark of an arbitrary shape, various icons, and the like (the display contents of the displays 12L and 12R may be anything). The numeric keypad and alphabets displayed on the right display 12R (or left display 12L) can be used when inputting numbers and letters. Character strings, marks, icons, etc. displayed on the right display 12R (or left display 12L) can be used when searching for a specific information item, selecting / determining a target item, or alerting the user. ..

The display images IM1 and IM2 can be used as augmented reality (AR) display means for adding information such as numbers, characters, and marks to the real world seen through glasses, and the AR display can be turned on and off as appropriate. The contents of the display image IM1 and the display image IM2 may have the same contents (IM1 = IM2) or different contents (IM1 ≠ IM2) depending on the embodiment. Further, the display image IM1 (or IM2) can be displayed on the right display 12R and / or the left display 12. If you want the content of the AR display to be a 3D image that overlaps (with depth) in the real world that you can see through the glasses, you can make IM1 and IM2 separate left and right 3D images.

When the displays (12R, 12L) are present on the left and right, for example, the convergence angle can be adjusted to shift the images of the left and right display images (IM1, IM2) in the left and right directions in opposite directions. As a result, it is possible to reduce the burden on the eyes when alternately viewing the object visible in the real world and the AR display. However, normally, the left and right displays (12R, 12L) display the same image.

The display control on the displays 12L and 12R can be performed by the information processing unit 11 embedded in the right temple bar 107. (Technology for displaying characters, marks, icons, etc. on a display is well known.) The power supply required for the information control unit 11 and other operations can be obtained from the battery BAT embedded in the left temple bar 106.

In addition, there is a possibility that a designer or a designer may feel that the weight balance is poor when a designer or a designer wears a prototype of eyewear 100 corresponding to the embodiment. If the main cause is the BAT in the left temple bar 106, a "weight (or another same weight battery)" commensurate with the BAT in the left temple bar 106 can be placed in the right temple bar 107.

FIG. 6 is a diagram illustrating a mounting example of an EOG electrode in the eyeglass-type eyewear 100 according to the embodiment. Right nose pad electrodes 151a and 151b are provided above and below the right nose pad 150R, and left nose pad electrodes 152a and 152b are provided above and below the left nose pad 150L. The output of the right nose pad electrodes 151a, 151b is given to the ADC 1510, the output of the left nose pad electrodes 152a, 152b is given to the ADC 1520, and the output of the lower electrodes 151b, 152b (or upper electrodes 151a, 152a) of the left and right nose pads. Is given to ADC 1512.

From the ADC 1510, a Ch1 signal that changes according to the user's right upper and lower eye movements can be obtained. From the ADC 1520, a Ch2 signal that changes according to the user's left upper and lower eye movements can be obtained. From the ADC 1512, a Ch0 signal that changes according to the left and right eye movements of the user can be obtained. The vertical movement of the left and right eyes can be evaluated by the Ch1 + 2 signal corresponding to the average of the outputs of the ADC 1510 and the ADC 1520. (The relationship between the signal waveforms of Ch0, Ch1, Ch2, Ch1 + 2 and eye movement will be described later.)
FIG. 7 is a diagram illustrating a relationship between the information processing unit 11 that can be attached to various embodiments and peripheral devices thereof. In the example of FIG. 7, the information processing unit 11 is composed of a processor 11a, a non-volatile memory 11b, a main memory 11c, a communication processing unit 11d, a sensor unit 11e, and the like. The processor 11a can be configured by a microcomputer having a processing capacity according to the product specifications. Various programs executed by this microcomputer and various parameters used at the time of program execution can be stored in the non-volatile memory 11b. The main memory 11c provides a work area for executing the program.

The sensor unit 11e includes a group of sensors for detecting the position and / or the orientation of the eyewear 100 (or the head of the user wearing the eyewear). Specific examples of these sensors include an acceleration sensor that detects movement in the three-axis direction (xyz direction), a gyro that detects rotation in the three-axis direction, and a geomagnetic sensor that detects absolute orientation (compass function). , There is a beacon sensor that receives radio waves, infrared rays, etc. to obtain position information and the like. You can use iBeacon® or Bluetooth® 4.0 to obtain this location information and more.

The LSI that can be used in the information processing unit 11 has been commercialized. One example is Toshiba Semiconductor & Storage's "TZ1000 series for wearable terminals". In this series, the product name "TZ1011MBG" includes CPU (11a, 11c), flash memory (11b), Bluetooth Low Energy (registered trademark) (11d), sensor group (accelerometer, gyro, geomagnetic sensor) (11e). , 24-bit Delta Sigma ADC, I / O (USB, etc.).

The information processing unit 11 is attached to the bridge 103 as an example of a place where the relative positional relationship between the sensor unit 11e such as an acceleration sensor and a gyro sensor and the EOG electrodes (151a, 151b, 152a, 152b) is not broken.
What to do with the processor 11a can be instructed from an external server (or personal computer) (not shown) via the communication processing unit 11d. The communication processing unit 11d can use existing communication methods such as ZigBee (registered trademark), Bluetooth, and Wi-Fi (registered trademark). The processing result of the processor 11a can be sent to a server or the like (not shown) via the communication processing unit 11d.

A display 12 (12L and 12R), a camera 13 (13L and 13R), an eye movement detection unit 15, and the like are connected to the system bus of the information processing unit 11. Each device (11 to 15) in FIG. 7 is powered by a battery BAT.
The communication processing unit 11d may incorporate a GPS (Global Positioning System) function that is well known in mobile phones, smartphones, and the like (or a mobile phone function with GPS may be installed somewhere in the eyewear 100). If the Internet map service application using this GPS function is executed on the processor 11a, the user wearing the eyewear 100 can determine where and in which direction he / she is currently walking while compensating for the eyeball during walking. It can be detected by suppressing the noise caused by it. (Although the conventional mobile phone GPS can display the current position of the user on the map screen, it cannot display where the user's line of sight is facing.)
The eye movement detection unit 15 of FIG. 7 has four eye movement detection electrodes (151a, 151b, 152a, 152b) constituting the line-of-sight detection sensor, and three ADCs (151a, 151b, 152a, 152b) that extract digital signals corresponding to eye movements from these electrodes. It includes 1510, 1520, 1512) and a circuit that outputs the output data from these ADCs to the processor 11a side. The processor 11a can interpret a command corresponding to the type of eye movement from various eye movements (vertical movement, left-right movement, blinking, eye closing, etc.) of the user and execute the command.

As a specific example of the command corresponding to the type of eye movement, if the eye movement is, for example, eye meditating, select the information item at the tip of the line of sight (similar to one click of a computer mouse), and multiple consecutive blinks or winks. Then there is a command (similar to double-clicking a computer mouse) to start processing the selected information item. This command is an example of information input B using the eye movement detection unit 15.

FIG. 8 is a diagram showing an example of an EOG waveform when, for example, while gazing at a fixed object in the real world 25 m away, a person starts walking from the right foot and stops at the right foot on the 13th step. The compensatory rotation of the eyeball corresponding to the vertical movement of the head accompanying walking is reflected in the change in ADC detection signal level (change in EOG) of Ch1 and Ch2. The vertical movement change of the head synchronized with the EOG change can be detected by the three-axis acceleration sensor included in the sensor unit 11e of the information processing unit 11. In addition, the compensatory rotation of the eyeball corresponding to the left-right rotation of the head accompanying walking appears in the change in the ADC detection signal level (EOG change) of Ch0. The change in the left-right rotation of the head synchronized with the EOG change can be detected by the 3-axis gyro sensor included in the sensor unit 11e of the information processing unit 11.

In the example of FIG. 8, the EOG fluctuation cycle (about 0.6 seconds) of Ch1 and Ch2 is synchronized with the walking pitch, and the EOG fluctuation cycle (about 1.2 seconds) of Ch0 is synchronized with twice the walking pitch. ing. Since the EOG fluctuation cycle corresponding to the vertical movement and left-right rotation of the head accompanying walking is clear, the cycle synchronized with the walking pitch (about 0.6 seconds in this example and twice that) is caused by walking. EOG fluctuation can be detected.

FIG. 9 is a diagram showing another example (confirmation of repetitive reproducibility) of the EOG waveform when walking is started again from the right foot and stopped at the right foot of the 13th step under the same conditions as in FIG. Also in the example of FIG. 9, the EOG fluctuation cycle (about 0.6 seconds) of Ch1 and Ch2 is synchronized with the walking pitch, and the EOG fluctuation cycle (about 1.2 seconds) of Ch0 is synchronized with twice the walking pitch. ing. The EOG fluctuation cycle corresponding to the vertical movement and left-right rotation of the head accompanying walking is also clearly recognized, and the cycle synchronized with the walking pitch (also about 0.6 seconds in this example and twice that) is caused by walking. It can be seen that the EOG fluctuation can be detected.

The following features are commonly observed in FIGS. 8 and 9, confirming the reproducibility of the event:
(1) Vertical eye rotation can be confirmed according to the walking cycle (about 0.6 seconds) (see EOG detection signal levels of Ch1 and Ch2).
(2) The eyeball rotation movement in the left-right direction can be confirmed according to a cycle (about 1.2 seconds) twice the walking cycle (see the EOG detection signal level of Ch0).

(3) There are places where the amplitude of the EOG detection signal caused by the shaking of the head due to walking reaches 200 μV to 400 μV. Since the EOG signal amplitude measured with the head stationary is usually about 1 mV, 200 μV to 400 μV becomes a noise of a size that cannot be ignored when detecting an arbitrary eye rotation. This noise can be canceled (reduced or eliminated) as described below.

That is, another vertical movement of the head accompanying walking can be detected by the accelerometer at a cycle synchronized with the walking pitch (about 0.6 seconds in this example), and the left-right rotation of the head accompanying walking is synchronized with the walking pitch. It can be detected by the gyro sensor with a cycle (about 1.2 seconds in this example). Then, the EOG fluctuation (considered as noise) caused by the vertical movement (sway) of the head accompanying walking can be canceled (reduced or eliminated) by the detection result of the acceleration sensor, and the left-right rotation of the head accompanying walking. The EOG fluctuation (considered as noise) caused by (sway) can be canceled (reduced or eliminated) by the detection result of the gyro sensor (a specific example of this cancellation will be described later with reference to FIG. 17).

FIG. 10 shows eye movements from the front to the top and detection signal levels (Ch0,) obtained from the three analog-to-digital converters (ADCs) shown in FIG. 6 when the noise caused by the shaking of the head is canceled. It is an electrocardiogram (EOG) exemplifying the relationship between Ch1, Ch2, and the average level of Ch1 and Ch2 (Ch1 + 2). Eye movement detection is performed based on the detection signal waveform in the broken line frame in the figure. The detection standard is when the user wearing the eyewear 100 is looking straight ahead and there is no eye movement (if the noise caused by the shaking of the head is canceled, the three ADCs shown in FIG. 6 are used. The output signal waveforms Ch0 to Ch2 are substantially flat in the section looking straight ahead without blinking, and there is almost no change with the passage of time). With the eyes of both the left and right eyes of the user facing the front, the eyes of both eyes are instantly moved upward, the state of the eyes being directed upward is maintained for 1 second, and then the eyes are instantly returned to the front. The change in the detection signal level when this is repeated 5 times is illustrated in FIG.

FIG. 11 illustrates the same eye movement detection as in FIG. 10 in the case where the line of sight moves downward from the front. If the noise caused by the shaking of the head is canceled, the line of sight is on the upper side based on the case where the line of sight is facing the front from the waveform changes of FIGS. 10 and 11 even while walking. It can detect whether it is below or below.

FIG. 12 shows the eye movement from left to right and the detection signal levels (Ch0, Ch1, Ch2, and Ch2) obtained from the three ADCs shown in FIG. 6 when the noise caused by the shaking of the head is canceled. It is an electrocardiogram which illustrates the relationship with Ch1 + 2). When there is an eye movement from left to right, the change over time of the detection signal waveform of Ch0 increases to the right (not shown, but when there is an eye movement from right to left, the change over time of the detection signal waveform of Ch0 Will fall to the right). From such a waveform change of Ch0, it is possible to detect whether the line of sight is on the right or left, based on the case where the line of sight is facing the front.

The following can be seen by summarizing the detection results of FIGS. 10 to 12. That is, if the noise caused by the shaking of the head is canceled, it is possible to know whether the line of sight is facing up, down, left or right, based on the case where the line of sight is facing the front even while walking. ..
FIG. 13 shows eye movements in which blinks / blinks (both eyes) are repeated 5 times at 5-second intervals when the noise caused by the shaking of the head is canceled and the line of sight is facing the front. 6 is an electrocardiogram illustrating the relationship with the detection signal levels (Ch0, Ch1, Ch2) obtained from the three ADCs shown in 6. Blinks / blinks of both eyes can be detected by the pulses appearing in Ch1 and Ch2. The blinking that the user unconsciously makes is often non-periodic. Therefore, as shown in FIG. 13, the user's intentional blink / blink can be detected by detecting a plurality of pulses at regular intervals.

According to one theory, the time for a person's "blink / blink" movement or the time for the field of vision to be blocked by "blink / blink" is about 300 msec (according to another theory, about 100 msec to 150 msec). In either case, the signal component due to "blink / blink" can be removed by a high-pass filter (or band-pass filter) that is shorter than the walking pitch cycle. The blink / blink interval (eg, about 300 msec) is different from the normal walking pitch (about 600 msec), and the unintentional blink / blink interval is not synchronized with the normal walking pitch. Therefore, when canceling the EOG noise caused by the head shaking caused by normal walking, all the changes in the EOG waveform due to blinking / blinking are not canceled.

FIG. 14 shows that when the noise caused by the shaking of the head is canceled, 1 second of eye closure (both eyes) and 4 seconds of eye opening (both eyes) are repeated 5 times when the line of sight is facing the front. It is an electrocardiogram exemplifying the relationship between the movement of the eye movement and the detection signal level (Ch0, Ch1, Ch2) obtained from the three ADCs shown in FIG. Eye closure of both eyes can be detected by the wide pulses appearing in Ch1 and Ch2 (the pulse width detected is wider because the time of intentional eye closure is longer than the time of blinking / closing the eye with blinking). .. By detecting the wide pulses of Ch1 and Ch2 as illustrated in FIG. 14, the user's intentional eye closure can be detected.

Although not shown, when the user closes only the right eye, a wide pulse having a large amplitude appears in Ch1, and when the user closes only the left eye, a wide pulse having a large amplitude appears in Ch2. From this, it is also possible to detect eye meditation separately on the left and right.
When only one eye is closed, small waves appear in Ch0 that are out of phase on the left and right. When viewed at Ch0, the following small uneven waveform appears.

・ When the right eye is closed, the positive potential of the electrode decreases, so a convex waveform;
・ When the left eye is closed, the positive potential of the + electrode decreases, resulting in a concave waveform.
In FIG. 15, when the noise caused by the shaking of the head is canceled, when the line of sight is facing the front, the blink of the left eye (blink of the left eye) is 5 immediately after repeating the blink of both eyes 5 times. 3 is an electrocardiogram illustrating the relationship between repeated eye movements and detection signal levels (Ch0, Ch1, Ch2) obtained from the three ADCs shown in FIG.

As illustrated in FIG. 6, the position of the ADC 1512 of Ch0 is offset below the centerline of the eyeballs of both the left and right eyes. Due to this offset, when both eyes blink at the same time, a negative potential change appears at both the + input and the-input of the ADC 1512 in FIG. At that time, if the potential changes (amount and direction) of both the + input and the-input are substantially the same, the changes are almost canceled and the value of the signal level output from the ADC 1512 of Ch0 becomes substantially constant (). See Ch0 level in the dashed line on the left side of FIG. 15). On the other hand, in the blink of one eye (left eye), there is almost no potential change on the − input side of the ADC 1512, and a relatively large negative potential change appears on the + input side of the ADC 1512. Then, the amount of cancellation of the potential change between the + input and the-input of the ADC 1512 becomes small, and a small pulse (small wave of the signal level) appears in the negative direction at the signal level output from the ADC 1512 of Ch0 (FIG. 15). See Ch0 level in the dashed line on the right). From the polarity of the small wave (pulse in the negative direction) of this signal level, it is possible to detect that the left eye has been winked (when the EOG noise caused by the shaking of the head is canceled, the left wink detection using Ch0). An example).

If the potential changes of the + input and-input of the ADC 1512 are not equal due to the distortion of the user's face or the condition of the skin, the output of Ch0ADC when the user wears the eyewear 100 and blinks both eyes at the same time is the minimum ( Calibration may be performed in advance so that the amount of cancellation between the + input component and the-input component is maximum).

Further, based on the peak ratio SL1a / SL2a of the detection signals Ch1 / Ch2 when both eyes blink / blink, the peak ratio SL1b / SL2b when the left eye wink is performed changes (SL1b / SL2b). Is not equal to SL1a / SL2a). From this as well, the left wink can be detected.

FIG. 16 shows that when the noise caused by the shaking of the head is canceled and the line of sight is facing the front, the blink of the right eye (blink of the right eye) is 5 immediately after repeating the blink of both eyes 5 times. 3 is an electrocardiogram illustrating the relationship between repeated eye movements and detection signal levels (Ch0, Ch1, Ch2) obtained from the three ADCs shown in FIG.

As described above, since the position of the ADC 1512 in FIG. 6 is offset below the center line of the eyeballs of both the left and right eyes, a negative potential change appears in both the + input and the-input of the ADC 1512 when both eyes blink at the same time. However, similar potential changes at the + and-inputs are almost canceled, and the value of the signal level output from the ADC 1512 of Ch0 becomes substantially constant (see the Ch0 level in the dashed line on the left side of FIG. 16). On the other hand, in the blink of one eye (right eye), there is almost no potential change on the + input side of the ADC 1512, and a relatively large negative potential change appears on the-input side of the ADC 1512. Then, the amount of cancellation of the potential change between the − input and the + input of the ADC 1512 becomes small, and a small pulse (small wave of the signal level) appears in the positive direction at the signal level output from the ADC 1512 of Ch0 (FIG. 16). See Ch0 level in the dashed line on the right). From the polarity of the small wave (pulse in the positive direction) of this signal level, it is possible to detect that the right eye has been winked (when the EOG noise caused by the shaking of the head is canceled, the right wink detection using Ch0). An example).

Further, based on the peak ratio SR1a / SR2a of the detection signal Ch1 / Ch2 when both eyes blink, the peak ratio SR1b / SR2b when the right eye wink is performed changes (SR1b / SR2b is SR1a / Not equal to SR2a). In addition, the peak ratio SL1b / SL2b at the time of left wink has a different value from the peak ratio SR1b / SR2b at the time of right wink (how much the difference can be confirmed by experiment). From this, the left wink can be detected separately from the right wink (an example of left and right wink detection using Ch1 and Ch2).

The device designer may appropriately decide whether to use Ch0 or Ch1 / Ch2 for detecting the left and right winks. The left and right wink detection results using Ch0 to Ch2 can be used as operation commands.
FIG. 17 is a flowchart illustrating an example of a process (noise canceling process) for minimizing noise caused by shaking of the head. The computer program corresponding to this flowchart is stored in, for example, the non-volatile memory 11b of FIG. 7 and executed by the processor 11a.

First, a user wearing the eyewear 100 as illustrated in FIG. 5 walks at a constant speed at a constant rhythm while gazing at a fixed object in front (ST10). Along with this walking, the user's head sways (moves / rotates) in a predetermined pattern (see FIGS. 1 (a) and 1 (b)).
The movement / rotation of the user's head accompanying walking is detected by an acceleration sensor / gyro sensor (11e in FIG. 7) provided inside the information processing unit 11 in FIG. 5 (ST12). The movement / rotation of the user's head accompanying walking is mixed into the EOG detection signal waveform as noise (see FIG. 8 or FIG. 9). The detection signal of the acceleration sensor / gyro sensor is added to the EOG detection signal waveform in a predetermined phase (ST14).

If the signal amplitude increases as a result of the above signal addition (ST16 no), the phase of the detection signal to be added is inverted (ST18), and the detection signal of the acceleration sensor / gyro sensor is added to the EOG detection signal waveform again. (ST14).
If the signal amplitude decreases as a result of the above signal addition (ST16 yes), the level of the signal to be added is adjusted so that the signal amplitude of the addition result becomes the minimum (minimum) (ST20). By minimizing (minimizing) the signal amplitude of the addition result, the noise mixed in the EOG detection signal waveform is canceled (reduced or eliminated).

Based on the EOG detection result after the noise caused by the shaking (movement / rotation) of the head is canceled, the user's eye movements (eye movement, blinking, eye closure, wink, etc. in FIGS. 10 to 16) are performed. A determination is made (ST22), and processing based on the determination is performed (ST24). For example, when the user winks, the name of the building in front of the line of sight can be AR-displayed on the display screen of the eyewear 100 (IM1 / IM2 in FIG. 5).

When the shaking pattern of the user's head changes, the content of noise cancellation also changes. Therefore, whether or not the sway pattern of the user's head has changed is detected from the change in the detection result of the acceleration sensor / gyro sensor (ST26). For example, suppose that a user of eyewear 100 walking at a constant speed while gazing at a fixed object in front stops in front of a down staircase that descends to the subway, and sweeps the step of an adjacent step of the staircase by bending back and forth and moving the line of sight. .. Then, the swaying pattern of the head becomes different from that during walking due to the line-of-sight sweep before and after this step, and the EOG detection signal waveform also becomes different from that during walking.

Therefore, when the shaking pattern of the user's head changes (ST26 yes), the process returns to step ST12 and the processes of ST14 to ST20 are repeated. By redoing the processes of ST14 to ST20, the EOG noise caused by the new head shaking pattern is canceled (reduced or eliminated), and the EOG corresponding to the pure line-of-sight movement is detected. The EOG detection result corresponding to this pure movement of the line of sight is compared with the detection result of the acceleration sensor / gyro sensor corresponding to the changed sway pattern of the user's head. As a result of this comparison, for example, if the phase inversion from FIG. 4 (a) to FIG. 4 (b) is detected at the step position of the stairs, it is determined that there is a step of the stairs near the phase inversion point (ST22). ). Then, the mark indicating the existence of the step is AR-displayed on the eyewear 100 (ST24). This AR display can be automatically turned off after a certain period of time (for example, 10 seconds) after the detection result of the acceleration sensor / gyro sensor corresponding to the shaking pattern of the user's head disappears.

The AR display can be turned on / off automatically depending on whether the EOG waveform (Ch1 / Ch2 waveform in FIG. 8 or FIG. 9) and the detection signal waveform of the acceleration sensor are synchronized. When the user is looking at the AR display, the eyeball is not rotating to compensate for the shaking of the head, so the above synchronization relationship does not occur, but when the user is watching the real world instead of the AR display, the above A synchronization relationship arises. Based on the same idea, the AR display can be automatically turned on / off depending on whether the EOG waveform (Ch0 waveform in FIG. 8 or FIG. 9) and the detection signal waveform of the gyro sensor are synchronized.

If the user continues to use the eyewear while the shaking pattern of the user's head does not change (ST28 no), the processes of ST22 to ST24 continue. When the user instructs the processor 11a of FIG. 7 to end the use of the eyewear 100 by closing for a few seconds (ST28 yes), the process of FIG. 17 ends.

FIG. 18 is a diagram illustrating an example of mounting an EOG electrode in the eyeglass-type eyewear 100 according to another embodiment (an example in which the EOG electrode is arranged around the eyeball). The eyewear 100 of FIG. 18 is different from the eyewear 100 of FIG. 5 in the following points.
The first difference is that the EOG electrodes 151a and 151b of the right nose pad 150R are moved to the right eye frame 101 side, and the EOG electrodes 152a and 152b of the left nose pad 150L are moved to the left eye frame 102 side. .. The EOG electrodes 151a and 151b are arranged at positions substantially point-symmetrical with respect to the user's right eye center position (not shown), and the EOG electrodes 152a and 152b are arranged with respect to the user's left eye center position (not shown). It is placed at a position that is approximately point-symmetrical. Further, the right diagonal line connecting the EOG electrodes 151a and 151b and the left diagonal line connecting the EOG electrodes 152a and 152b are substantially axisymmetric with respect to the vertical line (not shown) along the user's nose. The electrodes 151a, 151b, 152a, 152b can be formed of a conductive member (metal, conductive polymer, etc.) provided at the tip of an elastic body (sponge, silicone cushion, etc.). Each EOG electrode is lightly pressed against the skin surface of the user's face wearing the eyewear 100 by the elastic rebound force of the elastic body.

When the EOG electrode arrangement structure as shown in FIG. 18 is adopted, it becomes easier to detect the electric field generated around the charged eyeball as compared with the structure in which the EOG electrode is arranged on the nose pad portion. Therefore, the EOG signal having a larger amplitude can be detected in the embodiment shown in FIG. 18 than in the embodiment shown in FIG.

The second difference is that the sensor unit 11e including the acceleration sensor and the gyro sensor is left on the bridge 103, and the other functions of the information processing unit 11 are moved to the temple bar 107 side. For example, when the physical size of the information processing unit 11 increases due to the strengthening of the information processing capacity or the addition of the GPS function, if the enlarged information processing unit 11 is attached to the bridge 103, the design of the eyewear 100 and its installation There may be a problem with the feeling. If the structure attached to the bridge 103 is limited to the sensor unit 11e, the structure on the bridge 103 is smaller and lighter than the entire information processing unit 11, so that the design and the fit may be improved. is there. On the other hand, inside the temple bar 107 (and / or 106), even if a relatively large structure is provided, there is little possibility that problems will occur in the design of the eyewear 100 and the wearing feeling thereof.

FIG. 19 is a diagram illustrating an implementation example of EOG electrodes in goggle-type eyewear (a type in which eye frames of both left and right eyes are continuous) 100 according to still another embodiment (another EOG electrode arranged around the eyeball). Example). When the structure shown in FIG. 19 is adopted, the skin contact stability of the EOG electrode is higher than that of the structure shown in FIG. Therefore, the EOG signal can be detected with higher accuracy while canceling the noise caused by the shaking of the head. Further, in the structure of FIG. 19, it is easy to incorporate the enlarged information processing unit 11 and other devices. Therefore, in the structure of FIG. 19, the function of the GPS-equipped smartphone can be incorporated without worrying about the design. In that case, the screen display can be performed by AR display on the left and right displays 12L / 12R using a film liquid crystal or the like. Although not shown, small speakers can be attached near the left and right ears of the goggle frame, and small microphones can be attached near the nose cushion. In addition, command input to the smartphone can be performed by eye movements such as eye movement, blinking, eye closing, and winking.

FIG. 20 is a diagram illustrating an example of mounting the EOG electrode in the goggle type eyewear (type in which the eye cups of the left and right eyes are separated) 100 according to still another embodiment (the EOG electrode is arranged around the eyeball). Example). In the embodiment of FIG. 20, the EOG electrode arrangement structure and the arrangement structure of the sensor unit 11e are adopted as in the case of FIG. However, the goggle structure of FIG. 20 can withstand underwater use (or spacewalk training use). The head of a diver swimming in water sways more than walking on the ground, and the large sway (movement and rotation of position up, down, left, and right) can be detected by a 3-axis acceleration sensor or a 3-axis gyro sensor in the sensor unit 11e. The detected sway component can be used to cancel the noise component mixed in the EOG signal. Using the EOG signal that suppresses noise caused by head sway, it is possible to input various commands based on the eye movements (blinks, eye closures, etc.) of the user (diver) whose hands are closed. .. <Summary of embodiments>
(A) Conventional line-of-sight detection technology includes a method of estimating the line-of-sight direction by image processing using an infrared camera, but the amount of equipment required (in addition to the high-brightness infrared LED, the entire eyeball visible to the outside) A camera capable of taking pictures, a relatively high-performance arithmetic device for image processing, and a power supply device for operating these) will be large-scale. Therefore, although the conventional line-of-sight detection technology has been applied to stationary type devices, its application to wearable devices has not progressed.

On the other hand, the amount of equipment required for eye-gaze detection by electro-oculography sensing (electrodes and ADC, arithmetic unit with relatively low performance, power supply device for operating these) is considerably smaller than that of the infrared camera method. It is a scale and can be applied to wearable devices such as AR glasses. EOG is a technology that can be expected to be applied to wearable devices because it has a high affinity with AR glasses and consumes less power than the infrared method.

(B) Cancels the shaking of the eyewear user's head when gazing at a fixed object in the real world. As a result, the detection accuracy for any rotation of the eyeball is improved.
(C) Detects whether the fixed object in the real world being watched is a distant view or a near view (detection of the phase inversion point in FIG. 4). As a result, the presence or absence of gaze in the real world (fixed object) is detected from the shaking of the head of the eyewear user.

(D) The shaking of the eyewear user's head is used for automatic on / off control of the AR display.
(E) Only arbitrary eye movements that cancel the compensatory rotation of the eye are extracted.
(F) Detect whether or not what you are watching is in the real world (when walking, if the detection waveform of the accelerometer and / or gyro sensor and the electro-oculography EOG waveform are synchronized, look at what is in the real world. If the detection waveform of the accelerometer and / or the gyro sensor and the electro-oculography EOG waveform are not synchronized, it is determined that the AR display or the like is being viewed while walking).

(G) Effective as noise cancellation when the eye movement is used as a UI (user interface).
(H) When used for collecting work history information using eyewear, it becomes an unprecedented index (for example, in a state where the influence of noise due to head shaking is eliminated, in a warehouse picking work, etc., "correct target object" It can be used for pass / fail judgments such as "pass as a worker if the frequency of looking at is above a certain value"). <Example of correspondence between the content of the initial claim and the embodiment>
[1] The electrooculogram detection device (including 11a of FIG. 7: including the processor 11a that executes the process of FIG. 17) according to the embodiment is based on the electrooculogram (EOG) of a user wearing the eyewear (100). The eye movement detection unit (15; 151a, 151b, 152a, 152b) that detects eye movements (eye movement, blinking eyes, etc.) including the movement of the user's eyeball, and an acceleration sensor (11e) that detects the movement of the eyewear. A part of the eyewear) and a gyro sensor (a part of 11e) that detects the rotation of the eyewear. This eye movement detection device is a detecting means for detecting the movement and / or rotation of the eyewear by the acceleration sensor and / or the gyro sensor when the eye movement is detected based on the ocular potential (EOG). (11a for executing ST12) and noise mixed in the ocular potential due to the movement and / or rotation of the eyewear (the ocular potential component added by compensatory eye rotation that works in the direction of canceling the movement of the head). ) Is reduced (minimized or canceled) by synthesizing the signal components (components having opposite phases to the mixed noise) detected by the accelerometer and / or the gyro sensor (ST14 to ST20 are executed). 11a) is provided.

[2] The apparatus of the above [1] is a phase selection means (ST16 to ST16 to) that selects the phase of the signal component detected by the acceleration sensor and / or the gyro sensor so that the signal synthesis result by the reduction means tends to decrease. It further comprises 11a) for executing ST18.

[3] The device (11) of the above [1] is incorporated into eyewear (100 in FIGS. 5, 18 to 20).
[4] The eyewear (100) of the above [3] has a display unit (12L, 12R), and a display (AR display) based on the electro-oculography (EOG) after noise is reduced by the noise reducing means. The display unit (12L, 12R) is provided with display means (11a for executing ST22 to ST24).

[5] The device of the above [1] detects a rotation corresponding to a change in the anteroposterior bending of the user's head by the gyro sensor (11e), and changes in the line-of-sight direction of the user (FIGS. 10 and 11: Ch1). (The change in the vertical line-of-sight direction can be detected from the ± level change of / Ch2) is detected by the eye movement detection unit (15), and the signal phase of the anteroposterior bending change and the signal phase of the line-of-sight direction change are in phase (FIG. 4 (FIG. 4). It is configured to identify the perspective of the place the user is looking at depending on whether it is a)) or the opposite phase (FIG. 4 (b)) (whether the fixed object in the real world being watched is a distant view or a near view). Can be detected).

[6] The device of the above [1] moves and / or moves the eyewear detected by the accelerometer and / or the gyro sensor when the eye movement is detected based on the electrooculogram (EOG). Alternatively, when the rotation (FIG. 1) is synchronized with the change in the electrooculogram (EOG) (FIG. 8 or 9) (when the "predetermined phase relationship" can be maintained in ST14 of FIG. 17), the user sees the real world. It is configured to determine what you are looking at (it can detect whether what you are looking at is the real world).

[7] The electro-oculography detection method according to the first embodiment is based on the electro-oculography (EOG) of the user wearing the eyewear (100), and the eye movement (eye movement, blinking, etc.) including the eye rotation of the user. Is included, the movement of the head of the user wearing the eyewear is detected, and the rotation of the head of the user wearing the eyewear is detected. This method includes a step of detecting the movement and / or rotation of the head of a user wearing the eyewear when the eye movement is detected based on the electrooculogram (EOG) (ST12), and the eye. The noise mixed in the ocular potential due to the movement and / or rotation of the head of the user wearing the garment (the ocular potential component added by the compensatory eyeball rotation acting in the direction of canceling the movement of the head) is said. A step of reducing (minimizing or canceling) by synthesizing signal components (components having opposite phases to mixed noise) detected in response to movement and / or rotation of the head of the user wearing eyewear (ST14 to ST20) is provided.

[8] In the method of [7], the phase of the signal component detected in response to the movement and / or rotation of the head of the user wearing the eyewear so that the synthesis result of the signal component tends to decrease. The steps (ST16 to ST18) for selecting the above are further provided.
Although some embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other embodiments, and various omissions, replacements, and changes can be made without departing from the gist of the invention.

For example, in the description of the embodiment, a glasses-type or goggle-type device was introduced as eyewear provided with an EOG electrode, but other than that, an article in the form of an eye mask, an eye patch, a helmet, a hat, a hood, etc. was introduced as an EOG electrode. It can also be used for attached eyewear.
These embodiments and modifications thereof are included in the scope and gist of the invention, and are also included in the scope of the invention described in the claims and the equivalent scope thereof. It should be noted that it is also within the scope of the invention to combine a part or all of one embodiment of the disclosed embodiments and a part or all of another embodiment of the disclosed embodiments. And included in the abstract.

100 ... Eyewear (goggles type or glasses type); 110 ... Eye frame; 11 ... Information processing unit (processor 11a, non-volatile memory 11b, main memory 11c, communication processing unit (GPS processing)) which is the main part of the eye movement detection device. (Unit) 11d, integrated circuit including sensor unit 11e, etc.); 11e ... Sensor unit including acceleration sensor, gyro sensor, etc .; BAT ... Power supply (lithium ion battery, etc.); 12 ... Display unit (right display 12R and left display 12L: Film liquid crystal, etc.); IM1 ... Right display image (tenkey, alphabet, character string, mark, icon, etc.); IM2 ... Left display image (tenkey, alphabet, character string, mark, icon, etc.); 13 ... Camera (right camera 13R) And left camera 13L, or center camera (not shown) attached to the bridge 103 part); 15 ... Eye movement detection unit (line-of-sight detection sensor); 1510 ... Right side (Ch1) AD converter; 1520 ... Left side (Ch2) AD converter; 1512 ... Left-right (Ch0) AD converter; 1514 ... Left-right (Ch3) AD converter.

Claims (6)

A display unit that displays a real-world image or an augmented real-world image that adds image information to the real-world image,
The first detector that detects the movement of the user's head,
A second detection unit that detects the movement of the user's line of sight is provided.
The component related to the movement of the head included in the detection result of the second detection unit is reduced based on the detection result of the first detection unit, and the component is reduced based on the detection result of the second detection unit. The position on the line of sight of the user in the real-world image is detected, and the detected position includes image information corresponding to the movement of the line of sight of the user detected by the second detection unit in which the component is reduced. A wearable device that displays the augmented reality world image. The first detection unit also detects a change in the movement pattern of the user's head.
The wearable device according to claim 1, wherein when a change in the movement pattern of the head of the user is detected, the process of reducing the component based on the detection result of the first detection unit is executed again. The wearable device according to claim 2, wherein the second detection unit includes a camera. Displaying a real-world image or an augmented real-world image that adds image information to the real-world image,
The movement of the user's head is detected by the first detection unit, and
The movement of the user's line of sight is detected by the second detection unit, and
Reducing the components related to the movement of the head included in the detection result of the second detection unit based on the detection result of the first detection unit, and
Detecting the position on the line of sight of the user in the real-world image based on the detection result of the second detection unit in which the component is reduced,
A display method comprising displaying an augmented reality world image including image information according to the movement of the line of sight of the user detected by the second detection unit in which the component is reduced at the detected position. To detect a change in the movement pattern of the user's head by the first detection unit,
When a change in the movement pattern of the head of the user is detected, a process of reducing components related to the movement of the head included in the detection result of the second detection unit according to the detection result of the first detection unit. 4. The display method according to claim 4, further comprising executing the above again. The display method according to claim 5, wherein the second detection unit includes a camera.

Images