JP7119145B2 - Wearable device and display method - Google Patents

Description

TECHNICAL FIELD Embodiments of the present invention relate to wearable devices and display methods.

Normally, when a person walks while looking forward, the person's head sways up and down, left and right, but the person can see the surrounding scenery clearly. This is because the movement of the head during walking is suppressed to a smaller range than other parts of the body (the amplitude is about 2 to 3 cm regardless of the distance of the line of sight), and within that small range, the head and eyeballs stabilize the line of sight. This is because they are moving cooperatively so as to maintain the (see Non-Patent Document 1). This non-patent document 1 teaches that "when walking, the head sways vertically and horizontally, and rotation of the head and compensation rotation of the eyeballs occur as operations for stabilizing the line of sight against this sway."

Regarding the electro-oculogram sensing technology, there is a known example 1 that detects the line-of-sight position of a person based on the electro-oculogram caused by the eye movement of the person (see Patent Document 1). In this known example 1, EOG (Electro-Oculography) electrodes are used for electrooculogram detection (electrooculography sensing), and the user's gaze position is detected from the electrooculography data acquired from the EOG electrodes (paragraph 0025). In the embodiment of the known example 1, the EOG electrodes are provided on the goggles (paragraph 0061), but there is also a known example 2 in which the EOG electrodes are provided on the glasses (see Patent Document 2). Goggles and spectacles are a type of eyewear worn near the eyeballs of people. When the user wearing the eyewear walks, the eyewear sways together with the head.

JP 2009-288529 A JP 2013-244370 A

Head Movement and Eye Movement to Maintain Gaze Stabilization During Walking Paper 2000-03 Yazaki, Akiya Osaka University Graduate School of Human Sciences, 26 P.177-P.193http://ir.library.osaka-u .ac.jp/dspace/bitstream/11094/5672/1/hs26-177.pdf

In a stationary sitting state (a state in which the head of the user wearing the eyewear with EOG electrodes is not disturbed by movement), electro-oculography sensing can stably detect line-of-sight/eyeball rotation. However, when gazing at a fixed object in the real world in a walking state (a state in which the head of a user wearing eyewear with EOG electrodes sways in the vertical and horizontal directions), the target line of sight/eyeball is different from that in the sitting state. Rotation becomes difficult to detect. The reason for this is that eye rotation is performed unconsciously as a complementary motion of the human body to stabilize the line of sight, and the signal component of this unconscious eye rotation (as noise or disturbance) is superimposed on the electro-oculography sensing results. is.

That is, when a user wearing eyewear with EOG electrodes walks, electro-oculography data obtained from the EOG electrodes fluctuate due to the movement of the head, even if the user's line of sight is fixed on a specific object in front of them. . This variation in the electrooculography data becomes noise (disturbance) with respect to the electrooculography data corresponding to the actual line of sight.

One of the problems to be solved by the embodiments of the present invention is to provide a wearable device and a display method that change the display content according to the movement of the line of sight.

A wearable device according to an embodiment includes a display unit that displays a real world image or an augmented reality image that adds image information to the real world image, a first detection unit that detects eye movement of a user, and a head of the user. a second detector that detects movement; and what the user is looking at is determined based on the eye movement detected by the first detector and the head movement detected by the second detector. and a determination means for determining. When the eye movement detected by the first detection unit and the movement of the head detected by the second detection unit are synchronized, the determination unit determines whether the user sees the real world image. and if the eye movement detected by the first detection unit and the movement of the head detected by the second detection unit are not synchronized, the user looks at the augmented reality image determine that there is The display unit displays the augmented reality image according to whether the eye movement detected by the first detection unit and the movement of the head detected by the second detection unit are synchronized. turn on, turn off

FIG. 10 is a diagram for explaining head sway (positional change due to vertical and horizontal movements) of a person walking while looking forward. FIG. 4 is a diagram for explaining head sway (angular change due to forward bending/backward bending) of a person gazing at a fixed object in front while walking; FIG. 4 is a diagram for explaining the relationship between the target distance (the distance from the eye to the target of gaze) and the eyeball movement when the head undergoes forward and backward bending changes in the vertical direction. FIG. 4 is a diagram illustrating the phase relationship between the angle change/speed change of head forward and backward bending and the angle change/speed change of eye movement; FIG. 4 is a diagram for explaining spectacle-type eyewear incorporating an eye movement detection device according to an embodiment (an example in which EOG electrodes are arranged on a nose pad); FIG. 4 is a diagram for explaining an example of mounting EOG electrodes in spectacle-type eyewear according to an embodiment; FIG. 3 is a diagram for explaining the relationship between an information processing unit 11 that can be attached to various embodiments and its peripheral devices. FIG. 11 is a diagram showing an example of an EOG waveform when, for example, a user gazes at a fixed object in the real world 25 m ahead, starts walking with the right foot, and stops with the right foot on the 13th step. FIG. 9 is a diagram showing another example of EOG waveform (confirmation of repetitive reproducibility) when walking again with the right foot and stopping with the right foot on the 13th step under the same conditions as in FIG. 8 ; When the noise caused by the motion of the head is canceled, the eye movement from the front upward and the detection signal levels (Ch0, Ch1, Ch2, and an electrooculogram (EOG) illustrating the relationship between Ch1 and Ch2 mean levels (Ch1+2). When the noise due to head motion is canceled, the relationship between the frontal downward eye movement and the detected signal levels (Ch0, Ch1, Ch2, and Ch1+2) obtained from the three ADCs shown in FIG. Electrooculogram illustrating the relationship. Left-to-right eye movements and detected signal levels (Ch0, Ch1, Ch2, and Ch1+2) obtained from the three ADCs shown in FIG. Electrooculogram illustrating the relationship. When the noise caused by the motion of the head is canceled, when the line of sight is facing the front, the eye movement obtained by repeating blinking (both eyes) five times at an interval of 5 seconds and the three ADCs shown in FIG. Electrooculogram illustrating the relationship between detection signal levels (Ch0, Ch1, Ch2) obtained from . When the noise caused by the movement of the head is canceled, when the line of sight is facing the front, the eyes are closed for 1 second (both eyes) and the eyes are opened for 4 seconds (both eyes), repeated 5 times. , an electrooculogram illustrating the relationship between detection signal levels (Ch0, Ch1, Ch2) obtained from the three ADCs shown in FIG. When the noise caused by the motion of the head is canceled, the left eye wink (left single eye blink) is repeated 5 times immediately after blinking both eyes 5 times when the line of sight is facing the front. FIG. 7 is an electrooculogram illustrating the relationship between motion and detected signal levels (Ch0, Ch1, Ch2) obtained from the three ADCs shown in FIG. 6; When the noise caused by the motion of the head is canceled, the right eye wink (blink of the right eye) is repeated 5 times immediately after the blink of both eyes is repeated 5 times when the line of sight is facing the front. FIG. 7 is an electrooculogram illustrating the relationship between motion and detected signal levels (Ch0, Ch1, Ch2) obtained from the three ADCs shown in FIG. 6; 4 is a flowchart for explaining an example of processing (noise canceling processing) for minimizing noise caused by shaking of the head. FIG. 11 is a diagram for explaining an example of mounting EOG electrodes in spectacle-type eyewear according to another embodiment (an example in which EOG electrodes are arranged around the eyeball); FIG. 10 is a diagram for explaining an example of mounting EOG electrodes in goggle-type eyewear (a type in which eye frames for both left and right eyes are continuous) according to still another embodiment (another example in which EOG electrodes are arranged around the eyeballs). FIG. 11 is a diagram illustrating an example of mounting EOG electrodes in goggle-type eyewear (a type in which left and right eyecups are separated) according to still another embodiment (still another example in which EOG electrodes are arranged around the eyeballs).

Below, basic information is provided first, followed by a description of various embodiments with reference to the drawings. <Basic information>
The diameter of an adult eyeball is about 25 mm. It is about 17 mm after birth and increases in size as it grows.
The interpupillary distance of an adult male is about 65 mm. (Many commercially available stereo cameras are made at intervals of 65 mm.)
The interpupillary distance of an adult female is several millimeters shorter than that of a male.
Electrooculogram was several tens of mV.
The eyeball has a positive potential on the cornea side and a negative potential on the retina side. When this is measured on the surface of the skin, it appears as a potential difference of several hundred μV.

The types of eye movement and the range of eye movement related to eye movement detection include, for example, the following: <Types of eye movement (eye movement)> (01) Compensatory eye movement Involuntary eye movements developed to stabilize images of the external world on the retina. (02) Voluntary eye movement This eye movement is developed to center the visual image on the retina, and can be voluntarily controlled. (03) Impulsive eye movement (saccade)
The eye movements (easily detectable) that occur when changing the point of gaze in an attempt to see an object. (04) Smooth eye movements Smooth eye movements (difficult to detect) that occur when tracking a slowly moving object. <Range of eyeball movement (for a typical adult)> (11) Horizontal direction Left: 50° or less Right direction: 50° or less (12) Vertical direction Downward: 50° or less Upward: 30° or less The vertical angular range that can be moved at will is narrow only in the upward direction.(Because of the "bell phenomenon" in which the eyeballs rotate upward when the eyes are closed, the vertical eyeball movement range shifts upward when the eyes are closed.) (13 ) Other Convergence angle: 20° or less <Movement of the head accompanying walking (Figures 1 and 2 are sourced from Non-Patent Document 1)>
FIG. 1 is a diagram for explaining head sway (positional change due to vertical and horizontal movements) of a person walking while looking forward. The movement of the lower limbs during walking changes the position of the human head up and down (FIG. 1(a)). A typical adult walking cycle (one walking cycle) is, for example, about 0.6 seconds. At the same time, during this walking, the position of the person's head also changes from side to side (FIG. 1(b)). The cycle of this lateral position change is about 1.2 seconds, which is twice the walking cycle. The vertical positional change width and the horizontal positional change width are each about 2 to 3 cm. This vertical and horizontal movement is substantially linear and regular, and has a period closely related to the walking period.

FIG. 2 is a diagram illustrating head sway (angular change due to forward bending/backward bending) of a person who gazes at a fixed object in front while walking. When a person gazing at a visual target (target) ahead starts walking, the position of the head changes up and down as the person walks (FIG. 2(a)). While gazing at the target, the head bends forward (pitch down) when the head is in the upper position, and bends backward (pitch up) when the head is in the lower position. (Fig. 2(b)). Such anteroposterior bending (longitudinal rotational movement of the head) occurs in correspondence with (synchronization with) the vertical movement of the head during one gait cycle. Similarly, when the position of the head changes to the left and right (FIG. 1(b)), the head rotates in the left and right direction (synchronized with the vertical movement of the head).

That is, during walking, the human head undergoes a rotational motion that compensates for up, down, left, and right movements, and the head moves forward and backward in a high and low position to help the line of sight to grasp the visual target (target) in front of it. succumb The same is true for left and right, and when the body swings left and right as a person walks and the head moves left and right, the head rotates left and right (reciprocating rotation). This head rotation is called "compensatory rotation" and is thought to help stabilize the line of sight.

In addition, since the human trunk rotates back and forth and left and right while walking, the head also needs to compensate for these. By rotating the head in the opposite direction to this change in the trunk, the angular position of the head is kept horizontal. <Movement of Eyeballs Accompanying Walking (Figures 3 and 4 are sourced from Non-Patent Document 1)>
FIG. 3 is a diagram for explaining the relationship between the visual target distance (the distance from the eye to the target of gaze) and the eyeball movement when the head undergoes forward and backward bending changes in the vertical direction. Here, the target position when the head is fixed is HFP, and when the actual target is closer (Near Target) or farther than the HFP (Far Target), the forward and backward bending rotation angle (φh) of the head is and compensatory rotation angles (φen, φef) of the eyeball.

As mentioned above, the head performs compensatory rotation, but this alone cannot achieve full gaze stabilization. When the head rotates to compensate for vertical movement, it often causes excess or deficiency. In order to compensate for the deviation due to the excess or deficiency of this compensatory rotation, periodic compensatory rotation of the eyeball is required during walking. For example, for a near target (Near Target) 30 cm away, the eye performs compensatory rotation (φen) in the same direction as the rotation/rotation (φh) of the head. As a result, even if the rotation/rotation of the head is insufficient, the eyeballs rotate cooperatively with the head in the dynamic direction to compensate for the insufficient rotation of the head. On the other hand, for a far target (Far Target), for example, 100 m away, the eyeball performs compensatory rotation (φef) in the opposite direction to the rotation/rotation (φh) of the head. As a result, even if the head rotates/rotates excessively, the eyeballs rotate in the opposite direction to counteract the excessive amount of head rotation. (There is no substantial difference in the amount of movement of the head during walking whether the target is near or far.)
FIG. 4 is a diagram illustrating the phase relationship between the angle/velocity change of head forward and backward bending and the angle/velocity change of eye movement. The anteroposterior bending angle (φh) of the head shown in FIG. 3 can be detected by a three-axis gyro (11e in FIG. 7) attached to the eyewear. The eyeball rotation angles (φen, φef) shown in FIG. 3 can be detected by an eye movement detector (15 in FIG. 7) using the EOG of the eyeball.

Changes in vertical movement of the head (changes in vertical anteroposterior bending) associated with walking change in phase with changes in eyeball rotation when the target is near (Near Target) (Fig. 4(a)). On the other hand, when the visual target is far (Far Target), the vertical motion change of the head (vertical anteroposterior bending change) changes in opposite phase to the eyeball rotation change (FIG. 4(b)). When the target is at the fixed position HFP in FIG. 3, the eyeball rotation disappears, and the change in the waveform of the broken line shown in FIGS. , the phase between changes in longitudinal anteroposterior bending of the head and changes in eye rotation is reversed). A similar phase reversal is also seen between changes in the velocity of head bobbing and eye rotation (FIGS. 4(c) and 4(d)).

Now, the signal change that occurs in the EOG due to the above-mentioned "change in vertical motion of the head accompanying walking" is called "body motion noise". A walking action can be detected by an acceleration sensor or a gyro sensor, and an eye movement action can be detected by an EOG. Then, when "walking" and "no eye movement action" are detected, if body motion noise does not appear in the EOG, "a user wearing eyewear can detect phase inversion of body motion noise depending on the distance of the target." "I am gazing at the point (corresponding to the position HFP in FIG. 3)" is indicated.

FIG. 5 is a diagram illustrating the spectacle-type eyewear 100 incorporating the eye movement detection device according to one embodiment (an example in which EOG electrodes are arranged on the nose pads). In this embodiment, a right eyeframe (right limb) 101 and a left eyeframe (left limb) 102 are connected by a bridge 103 . The left and right eye frames 102 and 101 and the bridge 103 can be made of aluminum alloy, titanium, or the like. The left outer side of the left eye frame 102 is connected to the left temple bar 106 via the left hinge 104, and the left temple bar 106 is provided with the left temple bar (left ear pad) 108 at its tip. Similarly, the right outer side of the right eye frame 101 is connected to the right temple bar 107 via the right hinge 105, and the right temple bar 107 is provided with the right temple bar (right ear pad) 109 at its tip.

The bridge 103 is attached with the information processing section 11 (integrated circuit of several millimeters square), which is the main part of the eye movement detection device. The information processing section 11 can be configured by an LSI integrated with a microcomputer, a memory, a communication processing section, etc. (Details of the information processing section 11 will be described later with reference to FIG. 7).

A small battery (BAT), such as a lithium-ion battery, is embedded, for example, in the left temple bar 106 (or in the right temple bar 107, or in the moderns 108 or 109) to provide the power necessary for operation of the spectacle eyewear 100. ing.
A left camera 13L is attached to the end of the left eye frame 102 near the left hinge 104, and a right camera 13R is attached to the end of the right eye frame 101 near the right hinge 105. These cameras can be constructed using very small CCD image sensors.

These cameras (13L, 13R) may constitute a stereo camera. Alternatively, an infrared camera (13R) and a laser (13L) may be arranged at the positions of these cameras to form a distance sensor using an infrared camera and a laser. This distance sensor can also be composed of a small semiconductor microphone (13R) that collects ultrasonic waves and a small piezoelectric speaker (13L) that emits ultrasonic waves.

An embodiment in which a central camera (not shown) is provided in the bridge 103 portion instead of or in addition to the left and right cameras 13L/13R is also conceivable. Conversely, embodiments may be equipped with no cameras at all (these cameras are shown as cameras 13 in FIG. 7).

The left eye frame 102 is fitted with the left display 12L, and the right eye frame 101 is fitted with the right display 12R. This display is provided in at least one of the left and right eye frames, and can be constructed using a film liquid crystal or the like. Specifically, one or both of the left and right displays 12L and 12R can be configured using a film liquid crystal display device that employs a polymer-dispersed liquid crystal (PDLC) that does not use a polarizing plate (this display is the display in FIG. 7). 12).

The transparent left and right displays 12L/12R using film liquid crystal can be used as a means of providing augmented reality (AR) that adds image information such as numbers and characters to the real world seen through glasses.
A nose pad portion is provided between the left and right eye frames 102 and 101 and below the bridge 103 . This nose pad portion is composed of a pair of a left nose pad 150L and a right nose pad 150R. The right nose pad 150R is provided with right nose pad electrodes 151a and 151b, and the left nose pad 150L is provided with left nose pad electrodes 152a and 152b.

These electrodes 151a, 151b, 152a, 152b are electrically isolated from each other and connected to three AD converters (ADCs 1510, 1520, 1512) via insulated wiring members (not shown). Outputs from these ADCs have different signal waveforms according to eye movements of the user wearing the eyewear 100, and are supplied to the information processing section 11 as digital data according to the user's eye movements. The electrodes 151a, 151b, 152a, and 152b are used as line-of-sight detection sensors, and together with the three AD converters, constitute the eye movement detector 15 of FIG.

The eyewear 100 is fixed to the user's head (not shown) by left and right nose pads (150L, 150R), left and right temple bars (106, 107), and left and right tip ends (108, 109). In this embodiment, only the left and right nose pads (150L, 150R), the left and right temple bars (106, 107) and the left and right temples (108, 109) are directly in contact with the user's head (or face). However, there may be embodiments where parts other than those (nose pads, temple bars, temples) touch the user, such as for voltage matching between the ADCs (1510, 1520, 1512) and the user's body.

The film liquid crystal of the right display 12R in FIG. 5 can display a right display image IM1 including, for example, numeric keys (numbers, operators, Enter, etc.), alphabets, marks of predetermined shapes, and other icon groups. Also, the film liquid crystal of the left display 12L can display a left display image IM2 including, for example, arbitrary character strings, arbitrary shaped marks, and various icons (the display contents of the displays 12L and 12R can be anything). The numeric keypad and alphabet displayed on the right display 12R (or the left display 12L) can be used when entering numbers and characters. Character strings, marks, icons, and the like displayed on the right display 12R (or the left display 12L) can be used to search for specific information items, select/determine target items, and alert the user. .

The display images IM1 and IM2 can be used as augmented reality (AR) display means for adding information such as numbers, letters, and marks to the real world seen through the glasses, and this AR display can be turned on and off as appropriate. The content of the display image IM1 and the display image IM2 may be the same content (IM1=IM2) or different content (IM1≠IM2) depending on the embodiment. Also, the display image IM1 (or IM2) can be displayed on the right display 12R and/or the left display 12. FIG. When the contents of the AR display are to be a 3D image that overlaps (with depth) the real world seen through the glasses, IM1 and IM2 can be separate left and right 3D images.

Further, when the displays (12R, 12L) are present on the left and right, for example, by adjusting the angle of convergence, it is possible to shift the left and right display images (IM1, IM2) in opposite directions to the left and right. This can reduce the burden on the eyes when viewing objects visible in the real world and AR display alternately. Normally, however, the left and right displays (12R, 12L) display images of the same content.

Display control on the displays 12L and 12R can be performed by the information processing section 11 embedded in the right temple bar 107. FIG. (Techniques for displaying characters, marks, icons, etc. on a display are well known.) The power required for the operation of the information control unit 11 and others can be obtained from a battery BAT embedded in the left temple bar 106 .

Designers may feel that the weight balance is poor when wearing the prototype of the eyewear 100 corresponding to the embodiment. If the primary cause is the BAT in the left temple bar 106, then the right temple bar 107 can have a "weight" (or another battery of the same weight) that matches the BAT in the left temple bar 106.

FIG. 6 is a diagram illustrating an example of mounting EOG electrodes in the spectacle-type eyewear 100 according to one embodiment. Right nose pad electrodes 151a and 151b are provided above and below the right nose pad 150R, and left nose pad electrodes 152a and 152b are provided above and below the left nose pad 150L. The outputs of right nose pad electrodes 151a and 151b are applied to ADC 1510, the outputs of left nose pad electrodes 152a and 152b are applied to ADC 1520, and the outputs of lower electrodes 151b and 152b (or upper electrodes 151a and 152a) of left and right nose pads are applied. is provided to ADC 1512 .

The ADC 1510 obtains a Ch1 signal that changes according to the user's right vertical eye movement. The ADC 1520 provides a Ch2 signal that changes in response to the user's left vertical eye movement. The ADC 1512 provides a Ch0 signal that changes according to the user's left-right eye movement. The vertical motion of both eyes can be evaluated with the Ch1+2 signal corresponding to the average of the ADC 1510 and ADC 1520 outputs. (The relationship between the signal waveforms of Ch0, Ch1, Ch2, and Ch1+2 and eye movements will be described later.)
FIG. 7 is a diagram illustrating the relationship between the information processing unit 11 that can be attached to various embodiments and its peripheral devices. In the example of FIG. 7, the information processing section 11 includes a processor 11a, a nonvolatile memory 11b, a main memory 11c, a communication processing section 11d, a sensor section 11e, and the like. The processor 11a can be composed of a microcomputer having a processing capacity according to product specifications. Various programs to be executed by this microcomputer and various parameters to be used when executing the programs can be stored in the nonvolatile memory 11b. A work area for executing the program is provided by the main memory 11c.

The sensor section 11e includes a group of sensors for detecting the position and/or orientation of the eyewear 100 (or the head of the user wearing the eyewear). Specific examples of these sensors include an acceleration sensor that detects movement in three-axis directions (xyz directions), a gyro that detects rotation in three-axis directions, and a geomagnetic sensor (compass function) that detects absolute orientation. , and beacon sensors that receive radio waves, infrared rays, etc. to obtain location information and other information. Acquisition of this location information and other information can utilize iBeacon (registered trademark) or Bluetooth (registered trademark) 4.0.

An LSI that can be used for the information processing unit 11 has been commercialized. One example is Toshiba Semiconductor & Storage's "TZ1000 series for wearable terminals." Among this series, the product name "TZ1011MBG" is CPU (11a, 11c), flash memory (11b), Bluetooth Low Energy (registered trademark) (11d), sensor group (acceleration sensor, gyro, geomagnetic sensor) (11e) , 24-bit delta-sigma ADC, and I/O (USB, etc.).

The information processing section 11 is attached to the bridge 103 as an example of a place where the relative positional relationship between the sensor section 11e such as an acceleration sensor and a gyro sensor and the EOG electrodes (151a, 151b, 152a, 152b) is not disturbed.
What to do with the processor 11a can be instructed from an external server (or personal computer) (not shown) via the communication processing section 11d. The communication processing unit 11d can use existing communication methods such as ZigBee (registered trademark), Bluetooth, and Wi-Fi (registered trademark). The processing result of the processor 11a can be sent to a server (not shown) or the like via the communication processing unit 11d.

A display 12 (12L and 12R), a camera 13 (13L and 13R), an eye movement detection section 15, and the like are connected to the system bus of the information processing section 11. FIG. Each device (11-15) in FIG. 7 is powered by a battery BAT.
The communication processing unit 11d may incorporate a known GPS (Global Positioning System) function in mobile phones, smart phones, etc. (or a GPS-equipped mobile phone function may be incorporated somewhere in the eyewear 100). If the processor 11a executes the Internet map service application using the GPS function, the location and direction the user wearing the eyewear 100 is currently walking can be determined by eyeball compensation rotation during walking. It can be detected by suppressing the noise caused. (Conventional mobile phone GPS can display the user's current position on the map screen, but cannot display where the user's line of sight is directed.)
The eye movement detection unit 15 in FIG. 7 includes four eye movement detection electrodes (151a, 151b, 152a, 152b) that constitute a line of sight detection sensor, and three ADCs ( 1510, 1520, 1512) and a circuit for outputting output data from these ADCs to the processor 11a side. The processor 11a can interpret a command corresponding to the type of eye movement from various eye movements of the user (vertical movement, left-right movement, eye blinking, eye closing, etc.) and execute the instruction.

As a specific example of a command corresponding to the type of eye movement, if the eye movement is, for example, closing the eyes, select the information item in front of the line of sight (similar to one-click of a computer mouse) and blink or wink several times in succession. If is a command (analogous to a double-click on a computer mouse) that initiates execution of an action on the selected information item. This command is an example of information input B using the eye movement detector 15 .

FIG. 8 is a diagram showing an example of an EOG waveform when, for example, a user gazes at a fixed object in the real world 25 m away, starts walking with the right foot, and stops with the right foot on the 13th step. Compensatory rotation of the eyeball corresponding to the vertical movement of the head accompanying walking appears in the ADC detection signal level changes (EOG changes) of Ch1 and Ch2. The change in vertical motion of the head synchronized with the change in EOG can be detected by the three-axis acceleration sensor of the sensor section 11e of the information processing section 11. FIG. Compensatory rotation of the eyeball corresponding to left-right rotation of the head accompanying walking appears in the ADC detection signal level change (EOG change) of Ch0. The lateral rotation change of the head synchronized with the EOG change can be detected by the 3-axis gyro sensor of the sensor section 11 e of the information processing section 11 .

In the example of FIG. 8, the EOG fluctuation periods of Ch1 and Ch2 (approximately 0.6 seconds) are synchronized with the walking pitch, and the EOG fluctuation period of Ch0 (approximately 1.2 seconds) is synchronized with twice the walking pitch. ing. Since the EOG fluctuation period corresponding to the vertical movement and lateral rotation of the head accompanying walking is clear, the period synchronized with the walking pitch (in this example, about 0.6 seconds and twice that) is attributed to walking. EOG variations can be detected.

FIG. 9 is a diagram showing another example of the EOG waveform (confirmation of repetitive reproducibility) when walking again with the right foot and stopping with the right foot on the 13th step under the same conditions as in FIG. In the example of FIG. 9 as well, the EOG fluctuation periods of Ch1 and Ch2 (approximately 0.6 seconds) are synchronized with the walking pitch, and the EOG fluctuation period of Ch0 (approximately 1.2 seconds) is synchronized with twice the walking pitch. ing. The EOG fluctuation period corresponding to the vertical movement and left-right rotation of the head accompanying walking is clearly recognized, and the period synchronized with the walking pitch (in this example, about 0.6 seconds, which is twice as long) is due to walking. It can be seen that the EOG fluctuations that occur can be detected.

8 and 9 have the following features in common, confirming the reproducibility of the events:
(1) Vertical eyeball rotation can be confirmed in accordance with the walking cycle (approximately 0.6 seconds) (see EOG detection signal levels of Ch1 and Ch2).
(2) Horizontal eyeball rotation can be confirmed in accordance with a cycle (about 1.2 seconds) that is twice the walking cycle (see EOG detection signal level of Ch0).

(3) The amplitude of the EOG detection signal due to the shaking of the head accompanying walking reaches 200 μV to 400 μV in some places. Since the EOG signal amplitude measured when the head is stationary is usually about 1 mV, 200 μV to 400 μV becomes noise that cannot be ignored when detecting any eyeball rotation. This noise can be canceled (reduced or eliminated) as described below.

That is, the vertical movement of the head accompanying walking can be detected by the acceleration sensor at a period (about 0.6 seconds in this example) synchronized with the walking pitch, and the left-right rotation of the head accompanying walking can be detected separately with the walking pitch. It can be detected by the gyro sensor at a period (about 1.2 seconds in this example). Then, the EOG variation (deemed as noise) caused by the vertical movement (sway) of the head during walking can be canceled (reduced or eliminated) by the detection result of the acceleration sensor. EOG fluctuations (deemed as noise) caused by (shaking) can be canceled (reduced or eliminated) by the detection result of the gyro sensor (a specific example of this cancellation will be described later with reference to FIG. 17).

FIG. 10 shows the eye movement from the front to the upper direction and the detected signal levels (Ch0, Electrooculogram (EOG) illustrating the relationship between Ch1, Ch2, and the average level of Ch1 and Ch2 (Ch1+2). Eye movement detection is performed based on the detection signal waveform within the dashed frame in the figure. The detection criterion is that the user wearing the eyewear 100 is looking straight ahead and there is no eye movement (if the noise caused by head shaking is canceled, the three ADCs shown in FIG. The output signal waveforms Ch0 to Ch2 are substantially flat in the section in which the subject is looking straight ahead without blinking, and there is almost no change over time). With the user's right and left eyes facing the front, the eyes are instantly moved upward, the upward gaze is maintained for 1 second, and then the eyes are instantly returned to the front. FIG. 10 exemplifies changes in the detected signal level when this is repeated five times.

FIG. 11 illustrates eye movement detection similar to FIG. 10 when the line of sight moves downward from the front. If the noise caused by the swaying of the head is canceled, even during walking, the waveform changes in FIGS. It can be detected whether the

FIG. 12 shows left-to-right eye movements and detected signal levels (Ch0, Ch1, Ch2, and Ch1+2) is an electrooculogram illustrating the relationship. When there is an eye movement from left to right, the chronological change in the detection signal waveform of Ch0 rises to the right. is downwards to the right). Based on such a change in the waveform of Ch0, it is possible to detect whether the line of sight is to the right or to the left with reference to the case where the line of sight is directed straight ahead.

Integrating the detection results in FIGS. 10 to 12 reveals the following. In other words, if the noise caused by the movement of the head is canceled out, it is possible to know which direction the line of sight is facing, up, down, left, or right, based on the case where the line of sight is looking straight ahead, even while walking. .
FIG. 13 shows an eye movement obtained by repeating blinking/blinking (both eyes) 5 times at 5-second intervals when the line of sight is facing the front when the noise caused by the motion of the head is canceled. 6 is an electrooculogram illustrating the relationship between detected signal levels (Ch0, Ch1, and Ch2) obtained from three ADCs shown in 6. FIG. Blinking/blinking of both eyes can be detected by pulses appearing on Ch1 and Ch2. In many cases, the user's unconscious blinking does not have periodicity. Therefore, by detecting a plurality of pulses at regular intervals as shown in FIG. 13, the user's intentional blink/blink can be detected.

According to one theory, the duration of a person's "blinking/blinking" action or the time during which the "blinking/blinking" action blocks the visual field is about 300 msec (according to another theory, about 100 msec to 150 msec). In either case, a high-pass filter (or band-pass filter) below the walking pitch period can remove the "blink/blink" signal component. The blink/blink interval (eg, about 300 msec) is different from the normal walking pitch (about 600 msec), and the unintentional blink/blink interval is not synchronized with the normal walking pitch. Therefore, when canceling the EOG noise caused by the head shaking that accompanies normal walking, the change in the EOG waveform due to blinking/blinking is not completely canceled.

Fig. 14 shows that when the noise caused by the motion of the head is canceled, the eyes are closed for 1 second (both eyes) and the eyes are opened for 4 seconds (both eyes), which is repeated 5 times when the line of sight is facing the front. 7 is an electrooculogram illustrating the relationship between eye movements and detection signal levels (Ch0, Ch1, Ch2) obtained from the three ADCs shown in FIG. 6. FIG. Binocular eye closure can be detected by the wide pulses appearing on Ch1 and Ch2 (the duration of intentional eye closure is longer than the duration of eye blinking/blink closing, resulting in a wider detected pulse width). . By detecting wide pulses of Ch1 and Ch2 as exemplified in FIG. 14, it is possible to detect the user's intentional eye closing.

Although not shown, when the user closes only the right eye, a wide pulse with a large amplitude appears on Ch1, and when the user closes only the left eye, a wide pulse with a large amplitude appears on Ch2. From this, it is also possible to detect closed eyes separately for the right and left eyes.
When only one eye is closed, a small wave appears in Ch0 with opposite phases on the left and right sides. When viewed at Ch0, the following small uneven waveform appears.

・When the right eye is closed, the positive potential of the - electrode decreases, resulting in a convex waveform;
・When the left eye is closed, the positive potential of the + electrode decreases, resulting in a concave waveform.
FIG. 15 shows that when the noise caused by the shaking of the head is canceled and the line of sight is facing the front, the left eye wink (left eye blink) is repeated five times immediately after repeating the blink of both eyes five times. FIG. 7 is an electrooculogram illustrating the relationship between repeated eye movements and detected signal levels (Ch0, Ch1, Ch2) obtained from the three ADCs shown in FIG. 6;

As illustrated in FIG. 6, the position of the ADC 1512 of Ch0 is offset downward from the eyeball centerlines of both the left and right eyes. Due to this offset, when both eyes are blinked at the same time, a potential change in the negative direction appears at both the +input and -input of the ADC 1512 in FIG. At that time, if the potential changes (amount and direction) of both the + input and the - input are substantially the same, the changes are almost canceled, and the signal level value output from the ADC 1512 of Ch0 becomes substantially constant ( See the Ch0 level inside the dashed line on the left side of FIG. 15). On the other hand, when one eye (left eye) blinks, there is almost no potential change on the − input side of the ADC 1512 and a relatively large negative potential change appears on the + input side of the ADC 1512 . Then, the amount of cancellation of the potential change between the + input and the - input of the ADC 1512 becomes small, and a small negative pulse (signal level small wave) appears in the signal level output from the ADC 1512 of Ch0 (Fig. 15). See Ch0 level in right dashed line). From the polarity of the small wave (negative direction pulse) of this signal level, it is possible to detect that the left eye has been winked. example).

Note that if the potential changes of the +input and -input of the ADC 1512 are not equal due to the distortion of the user's face or the condition of the skin, etc., the output of Ch0ADC will be the minimum ( Calibration may be performed in advance such that the amount of cancellation between the + input component and the - input component is maximized.

Further, when the peak ratio SL1a/SL2a of the detection signals Ch1/Ch2 when blinking/blinking of both eyes is used as a reference, the peak ratio SL1b/SL2b when winking the left eye changes (SL1b/SL2b is not equal to SL1a/SL2a). From this also, a left wink can be detected.

FIG. 16 shows that when the noise caused by the shaking of the head is canceled and the line of sight is facing the front, the blink of the right eye (blink of the right eye) is repeated five times immediately after repeating the blink of both eyes five times. FIG. 7 is an electrooculogram illustrating the relationship between repeated eye movements and detected signal levels (Ch0, Ch1, Ch2) obtained from the three ADCs shown in FIG. 6;

As described above, since the position of the ADC 1512 in FIG. 6 is offset downward from the eyeball center line of both the left and right eyes, a negative potential change appears in both the +input and -input of the ADC 1512 when both eyes are simultaneously blinked. However, similar potential changes in the +input and -input are almost canceled out, and the signal level value output from the Ch0 ADC 1512 becomes substantially constant (see the Ch0 level within the left dashed line in FIG. 16). On the other hand, when one eye (right eye) blinks, there is almost no potential change on the + input side of the ADC 1512 and a relatively large negative potential change appears on the - input side of the ADC 1512 . Then, the amount of cancellation of the potential change between the -input and the +input of the ADC 1512 becomes small, and the signal level output from the ADC 1512 of Ch0 shows a small positive pulse (signal level small wave) (Fig. 16). See Ch0 level in right dashed line). From the polarity of the small wave (positive direction pulse) of this signal level, it is possible to detect that a wink has been made with the right eye. example).

Further, when the peak ratio SR1a/SR2a of the detection signals Ch1/Ch2 when both eyes are blinked is used as a reference, the peak ratio SR1b/SR2b when the right eye wink is performed changes (SR1b/SR2b is SR1a/ not equal to SR2a). Also, the peak ratio SL1b/SL2b for the left wink has a value different from the peak ratio SR1b/SR2b for the right wink (the degree of difference can be confirmed by experiments). Therefore, a left wink can be detected separately from a right wink (an example of left and right wink detection using Ch1 and Ch2).

Whether to use Ch0 or Ch1/Ch2 for right and left wink detection may be determined by the device designer as appropriate. Left and right wink detection results using Ch0 to Ch2 can be used as operation commands.
FIG. 17 is a flowchart illustrating an example of processing (noise canceling processing) for minimizing noise caused by shaking of the head. A computer program corresponding to this flowchart is stored, for example, in the nonvolatile memory 11b in FIG. 7 and executed by the processor 11a.

First, a user wearing eyewear 100 as illustrated in FIG. 5 walks at a constant speed with a constant rhythm while gazing at a fixed object in front (ST10). Along with this walking, the user's head shakes (moves/rotates) in a predetermined pattern (see FIGS. 1(a) and 1(b)).
Movement/rotation of the user's head accompanying walking is detected by an acceleration sensor/gyro sensor (11e in FIG. 7) provided inside the information processing section 11 in FIG. 5 (ST12). Movement/rotation of the user's head accompanying walking is mixed as noise in the EOG detection signal waveform (see FIG. 8 or 9). A detection signal from the acceleration sensor/gyro sensor is added to this EOG detection signal waveform at a predetermined phase (ST14).

As a result of the above signal addition, if the signal amplitude increases (NO in ST16), the phase of the detection signal to be added is inverted (ST18), and the acceleration sensor/gyro sensor detection signal is again added to the EOG detection signal waveform. (ST14).
If the signal amplitude decreases as a result of the above signal addition (YES in ST16), the level of the signal to be added is adjusted so that the signal amplitude of the addition result is minimized (ST20). By minimizing (extremely small) the signal amplitude of the addition result, noise mixed in the EOG detection signal waveform is canceled (reduced or eliminated).

Based on the EOG detection result after the noise caused by the motion (movement/rotation) of the head has been canceled, the user's eye movements (movement of the line of sight, blinking, closing the eyes, winking, etc. in FIGS. 10 to 16) are controlled. A determination is made (ST22), and processing based on the determination is performed (ST24). For example, when the user winks, the name of the building ahead of the line of sight can be AR-displayed on the display screen of the eyewear 100 (IM1/IM2 in FIG. 5).

If the shaking pattern of the user's head changes, the content of noise cancellation also changes. Therefore, whether or not the shaking pattern of the user's head has changed is detected from changes in the detection results of the acceleration sensor/gyro sensor (ST26). For example, assume that the user of the eyewear 100, who is walking at a constant speed while gazing at a fixed object in front of them, stops in front of the stairs leading down to the subway, and sweeps the steps of the adjacent steps of the stairs by bending back and forth and moving the line of sight. . Then, due to the sweep of the line of sight before and after the step, the shaking pattern of the head differs from that during walking, and the waveform of the EOG detection signal also differs from that during walking.

Therefore, when the shaking pattern of the user's head has changed (YES in ST26), the process returns to step ST12 and repeats the processes of ST14 to ST20. By redoing the processing of ST14 to ST20, the EOG noise caused by the new head shaking pattern is canceled (reduced or eliminated), and the EOG corresponding to pure eye movement is detected. The EOG detection result corresponding to this pure line-of-sight movement is compared with the detection result of the accelerometer/gyro sensor corresponding to the changed shaking pattern of the user's head. As a result of this comparison, for example, if a phase inversion from FIG. 4A to FIG. 4B is detected at the step position of the stairs, it is determined that there is a step in the vicinity of the phase inversion point (ST22). ). Then, the eyewear 100 AR-displays a mark indicating the existence of the step (ST24). This AR display can be automatically turned off after a certain period of time (for example, 10 seconds) after the detection result of the acceleration sensor/gyro sensor corresponding to the shaking pattern of the user's head disappears.

AR display can be turned on/off automatically depending on whether the EOG waveform (Ch1/Ch2 waveform in FIG. 8 or 9) and the detection signal waveform of the acceleration sensor are synchronized. When the user is looking at the AR display, the eyeballs do not rotate to compensate for the sway of the head, so the above synchronization relationship does not exist. A synchronous relationship arises. Based on the same idea, the AR display can be automatically turned on/off depending on whether the EOG waveform (Ch0 waveform in FIG. 8 or 9) and the detection signal waveform of the gyro sensor are synchronized.

If the user continues to use the eyewear while the shaking pattern of the user's head does not change (ST28 NO), the processing of ST22 to ST24 continues. If the user closes his/her eyes for a few seconds and instructs the processor 11a in FIG. 7 to stop using the eyewear 100 (YES at ST28), the processing in FIG. 17 ends.

FIG. 18 is a diagram illustrating an example of mounting EOG electrodes in spectacle-type eyewear 100 according to another embodiment (an example in which EOG electrodes are arranged around the eyeball). The eyewear 100 of FIG. 18 differs from the eyewear 100 of FIG. 5 in the following points.
The first difference is that the EOG electrodes 151a and 151b of the right nose pad 150R are moved toward the right eyeframe 101, and the EOG electrodes 152a and 152b of the left nose pad 150L are moved toward the left eyeframe 102. . The EOG electrodes 151a and 151b are arranged at substantially point-symmetrical positions with respect to the central position of the user's right eye (not shown), and the EOG electrodes 152a and 152b are arranged with respect to the central position of the user's left eye (not shown). They are arranged at positions that are substantially point symmetrical. A right oblique line connecting EOG electrodes 151a and 151b and a left oblique line connecting EOG electrodes 152a and 152b are substantially symmetrical with respect to a vertical line (not shown) along the bridge of the user's nose. The electrodes 151a, 151b, 152a, 152b can be composed of a conductive member (metal, conductive polymer, etc.) provided at the tip of an elastic body (sponge, silicone cushion, etc.). Each EOG electrode is lightly pressed against the skin surface of the face of the user wearing the eyewear 100 by the elastic repulsive force of the elastic body.

When the EOG electrode arrangement structure as shown in FIG. 18 is adopted, it becomes easier to detect the electric field generated around the charged eyeball compared to the structure in which the EOG electrodes are arranged in the nose pad portion. Therefore, the embodiment of FIG. 18 can detect a larger amplitude EOG signal than the embodiment of FIG.

The second difference is that the sensor section 11e including the acceleration sensor and the gyro sensor remains in the bridge 103, and the other functions of the information processing section 11 are moved to the temple bar 107 side. For example, when the physical size of the information processing unit 11 increases due to enhancement of information processing capability or addition of a GPS function, attaching the increased information processing unit 11 to the bridge 103 improves the design of the eyewear 100 and its mounting. Sensation problems may occur. If only the sensor section 11e is attached to the bridge 103, the structure on the bridge 103 can be made smaller and lighter than the entire information processing section 11. Therefore, the design and wearing comfort may be improved. be. On the other hand, even if a relatively large structure is provided inside the temple bar 107 (and/or 106), there is little possibility that the design of the eyewear 100 and the comfort of wearing the eyewear 100 will suffer.

FIG. 19 is a diagram illustrating an example of mounting EOG electrodes in goggle-type eyewear (a type in which eye frames for both left and right eyes are continuous) 100 according to still another embodiment (another type in which EOG electrodes are arranged around the eyeballs). example). If the structure shown in FIG. 19 is adopted, the EOG electrode has higher skin contact stability than the structure shown in FIG. Therefore, it is possible to detect the EOG signal with higher accuracy while canceling out the noise caused by the shaking of the head. In addition, in the structure of FIG. 19, it is easy to incorporate the large-sized information processing unit 11 and other devices. Therefore, in the structure of FIG. 19, the function of a smartphone with GPS can be incorporated without worrying about the design. The screen display in that case can be performed by AR display on the left and right displays 12L/12R using film liquid crystal or the like. Although not shown, small speakers can be attached near the left and right ears of the goggle frame, and a small microphone can be attached near the nose cushion. Also, command input to the smartphone can be performed by eye movements such as line of sight movement, eye blinking, eye closing, and winking.

FIG. 20 is a diagram illustrating an example of mounting EOG electrodes in goggle-type eyewear (a type in which eyecups for both left and right eyes are separated) 100 according to still another embodiment (EOG electrodes are arranged around the eyeballs). example). In the embodiment of FIG. 20, the same EOG electrode arrangement structure and sensor section 11e arrangement structure as in the case of FIG. 18 are adopted. However, the goggle structure of FIG. 20 can withstand underwater use (or spacewalk training use). The head of a diver swimming in water shakes more than walking on the ground, and the large shaking (up, down, left, right positional movement and rotation) can be detected by the 3-axis acceleration sensor and 3-axis gyro sensor in the sensor section 11e. The detected fluctuation component can be used to cancel the noise component mixed in the EOG signal. Various commands can be input based on eye movements (blinking, eye closing, etc.) of a user (diver) with both hands occupied by using an EOG signal in which noise caused by head shaking is suppressed. . <Summary of embodiment>
(a) Conventional line-of-sight detection technology uses an infrared camera to estimate the line-of-sight direction through image processing. A camera capable of shooting, a relatively high-performance arithmetic unit for image processing, and a power supply unit for operating these) become large-scale. For this reason, the conventional line-of-sight detection technology has been applied to stationary devices, but has not been applied to wearable devices.

On the other hand, line-of-sight detection by electro-oculography sensing requires a considerably smaller amount of equipment (electrodes, ADC, relatively low-performance arithmetic unit, and power supply unit for operating them) compared to the infrared camera method. It is a large scale and can be applied to wearable devices such as AR glasses. EOG is a technology that can be expected to be applied to wearable devices because it has a high affinity with AR glasses and consumes less power than the infrared method.

(b) Canceling the shaking of the eyewear user's head when gazing at a fixed object in the real world. This improves the detection accuracy for any eyeball rotation.
(c) Detecting whether the gazed fixed object in the real world is a distant view or a near view (detection of the phase reversal point in FIG. 4). This detects whether or not the real world (fixed object) is being watched from the shaking of the eyewear user's head.

(d) Use eyewear user's head shake for automatic on/off control of AR display.
(e) Extract only arbitrary eye motions that cancel the compensation rotation of the eye.
(f) Detecting whether or not the object being gazed at is in the real world (if the detection waveform of the acceleration sensor and/or the gyro sensor and the electro-oculogram EOG waveform are synchronized during walking, the object in the real world is detected) If the detection waveform of the acceleration sensor and/or the gyro sensor and the electro-oculography EOG waveform are not synchronized, it is determined that the AR display or the like is being viewed during walking).

(g) Effective for canceling noise when eye movement is used as a UI (user interface).
(h) When used to collect work history information using eyewear, it becomes an index that has never been seen before (for example, in a warehouse picking work, etc., while eliminating the effects of noise due to head shaking, "correct target object It can be used for pass/fail judgment such as "If the frequency of looking at is above a certain value, you pass as a worker"). <Example of Correspondence between Contents of Claims at the Time of Filing and Embodiments>
[1] An electro-oculogram detection device according to an embodiment (11 in FIG. 7: including a processor 11a that executes the processing in FIG. 17) is based on the electro-oculography (EOG) of a user wearing eyewear (100). an eye movement detection unit (15; 151a, 151b, 152a, 152b) for detecting eye movement (line movement, blink, etc.) including eyeball rotation of the user, and an acceleration sensor (11e ) and a gyro sensor (part of 11e) that detects the rotation of the eyewear. The eye movement detecting device includes detection means for detecting movement and/or rotation of the eyewear by the acceleration sensor and/or the gyro sensor when the eye movement is detected based on the electrooculography (EOG). (11a executing ST12) and noise mixed in the electro-oculogram due to the movement and/or rotation of the eyewear (electro-oculography component added by compensatory eye rotation acting in the direction of canceling the movement of the head) ) is reduced (minimized or canceled) by synthesizing signal components detected by the acceleration sensor and/or the gyro sensor (components opposite in phase to the mixed noise) (ST14 to ST20 are executed). 11a).

[2] The apparatus of [1] includes phase selection means (ST16 to 11a) for executing ST18.

[3] The device (11) of [1] is incorporated in eyewear (100 in FIGS. 5 and 18 to 20).
[4] The eyewear (100) of [3] has display units (12L, 12R), and displays (AR display) based on the electrooculography (EOG) after noise is reduced by the noise reduction means. on the display units (12L, 12R) (11a for executing ST22 to ST24).

[5] The device of [1] detects a rotation corresponding to a change in forward and backward bending of the user's head by the gyro sensor (11e), and detects a change in the direction of the user's line of sight (Figs. 10 and 11: Ch1). /Ch2) is detected by the eye movement detection unit (15), and the signal phase of the change in forward/backward bending and the signal phase of the change in the direction of the line of sight are in phase (Fig. 4 ( a)) or reversed phase (FIG. 4(b)). detectable).

[6] The device of [1] is configured to move and/or move the eyewear detected by the acceleration sensor and/or the gyro sensor when the eye movement is detected based on the electrooculography (EOG). Alternatively, when the rotation (FIG. 1) is synchronized with the change in the electrooculogram (EOG) (FIG. 8 or 9) (when the "predetermined phase relationship" can be maintained in ST14 of FIG. 17), the user can change the real world. It is configured to determine what you are looking at (it can detect whether or not what you are looking at is the real world).

[7] An electro-oculogram detection method according to an embodiment is based on the electro-oculogram (EOG) of the user wearing the eyewear (100), eye movement including eyeball rotation (line-of-sight movement, blinking, etc.) of the user. , detecting movement of the head of the user wearing the eyewear, and detecting rotation of the head of the user wearing the eyewear. This method includes the steps of detecting movement and/or rotation of the head of the user wearing the eyewear when the eye movement is detected based on the electrooculogram (EOG) (ST12); The noise mixed in the electro-oculography due to the movement and/or rotation of the head of the user wearing the wear (the electro-oculography component added by the compensatory eyeball rotation acting in the direction of canceling the movement of the head) is A process of reducing (minimizing or canceling) by synthesizing signal components (components that are inverse to mixed noise) detected corresponding to the movement and/or rotation of the head of the user wearing the eyewear (ST14- ST20).

[8] In the method [7], the phase of the signal component detected corresponding to the movement and/or rotation of the head of the user wearing the eyewear is reduced so that the synthesis result of the signal component decreases. is further provided (ST16 to ST18).
While several embodiments of the invention have been described, these embodiments have been presented by way of example and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, replacements, and modifications can be made without departing from the scope of the invention.

For example, in the description of the embodiments, spectacle-type and goggle-type devices were introduced as eyewear equipped with EOG electrodes. It is also conceivable to use it for attached eyewear.
These embodiments and modifications thereof are included in the scope and gist of the invention, and are included in the scope of the invention described in the claims and equivalents thereof. It is also within the scope of the invention to combine part or all of a certain embodiment out of the multiple disclosed embodiments with part or all of another embodiment out of the multiple disclosed embodiments. and the abstract.

100 Eyewear (goggle type or glasses type); 110 Eye frame; 11 Information processing unit (processor 11a, nonvolatile memory 11b, main memory 11c, communication processing unit (GPS processing part) 11d, integrated circuit including sensor part 11e, etc.); 11e... sensor part including acceleration sensor, gyro sensor, etc.; BAT... power supply (lithium ion battery, etc.); 12... display part (right display 12R and left display 12L: Film liquid crystal, etc.); IM1: Right display image (ten key, alphabet, character string, mark, icon, etc.); IM2: Left display image (ten key, alphabet, character string, mark, icon, etc.); 13: Camera (right camera 13R and left camera 13L, or a center camera (not shown) attached to the bridge 103 portion); ... between left and right (Ch0) AD converter; 1514 ... between left and right (Ch3) AD converter.

Claims (12)

a display unit that displays a real world image or an augmented reality image that adds image information to the real world image;
a first detection unit that detects eye movement of the user;
a second detection unit that detects movement of the user's head;
determining that the user is viewing the real-world image when the eye movement detected by the first detection unit and the movement of the head detected by the second detection unit are synchronized; Determining that the user is viewing the augmented reality image when the eye movement detected by the first detection unit and the movement of the head detected by the second detection unit are not synchronized. means and
A wearable device comprising
The display unit determines whether or not the eye movement detected by the first detection unit and the movement of the head detected by the second detection unit are synchronized according to the determination result of the determination unit. A wearable device that turns on and off the display of the augmented reality image. The determination means determines whether the phase of the eye movement detected by the first detection unit and the phase of the movement of the head detected by the second detection unit are in phase. 3. The wearable device of claim 1, wherein the wearable device determines the perspective of an object being viewed. The determination means is
When the phase of the eye movement detected by the first detection unit and the phase of the movement of the head detected by the second detection unit are in phase, the user is positioned at a short distance in the real-world image. determine that you are looking at an object,
When the phase of the eye movement detected by the first detection unit and the phase of the movement of the head detected by the second detection unit are in opposite phases, the user moves from a long distance in the real world image. 3. The wearable device according to claim 2, wherein the wearable device determines that it is looking at an object of . In the display unit, the determination unit detects that the phase of the eye movement detected by the first detection unit and the phase of the movement of the head detected by the second detection unit change from the same phase to the opposite phase. 4. The wearable device according to claim 2, wherein the display of the augmented reality image is turned on when detected. Claims 1 and 2, further comprising compensating means for reducing noise contained in the eye movement detected by the first detector based on the movement detected by the second detector. The wearable device according to claim 3 or 4. The first detection unit detects eye movement in the vertical direction and eye movement in the horizontal direction,
When the determination means determines that the user is viewing the real world image while walking, the determination means determines that the eye movement in the vertical direction and the eye movement in the horizontal direction detected by the first detection unit is used to compensate for the compensation. 6. The wearable device according to claim 5, wherein it is determined that the user's gaze point has moved in the vertical direction or the horizontal direction based on the vertical eye movement and the horizontal eye movement whose noise has been reduced by means. displaying, on a display unit, a real world image or an augmented reality image in which image information is added to the real world image;
Detecting eye movement of the user by a first detection unit;
Detecting movement of the user's head by a second detection unit;
A display method comprising
determining that the user is viewing the real-world image when the eye movement detected by the first detection unit and the movement of the head detected by the second detection unit are synchronized; Determining that the user is viewing the augmented reality image when the eye movement detected by the first detection unit and the movement of the head detected by the second detection unit are not synchronized. and
The displaying may be performed according to a determination result as to whether or not the eye movement detected by the first detection unit and the head movement detected by the second detection unit are synchronized. turning on and off the display of images;
A display method comprising The determination is based on whether the phase of the eye movement detected by the first detection unit and the phase of the movement of the head detected by the second detection unit are in phase. 8. A display method according to claim 7, wherein determines the perspective of the viewed object. The determining
When the phase of the eye movement detected by the first detection unit and the phase of the movement of the head detected by the second detection unit are in phase, the user is positioned at a short distance in the real-world image. determine that you are looking at an object,
When the phase of the eye movement detected by the first detection unit and the phase of the movement of the head detected by the second detection unit are in opposite phases, the user moves from a long distance in the real world image. 9. The display method according to claim 8, wherein it is determined that the user is looking at an object of . When it is determined that the eye movement phase detected by the first detection unit and the head movement phase detected by the second detection unit change from the same phase to the opposite phase, the augmented reality image is displayed. 10. The display method according to claim 8, wherein the is turned on. Claims 7, 8, and 9, further comprising reducing noise contained in the eye movement detected by the first detection unit based on the movement detected by the second detection unit. Or the display method according to claim 10. Detecting eye movement in the vertical direction and eye movement in the horizontal direction by the first detection unit,
When it is determined that the user is viewing the real world image while walking, the eye movement in the vertical direction and the eye movement in the horizontal direction detected by the first detection unit are reduced in noise. 12. The display method according to claim 11, wherein it is determined that said user's gaze point has moved in the vertical direction or the horizontal direction based on the eye movement in the vertical direction and the eye movement in the horizontal direction.

Images