JP7373032B2 - Wearable devices and display methods - Google Patents

Description

Embodiments of the present invention relate to wearable devices and display methods.

Normally, when a person walks while looking ahead, the person's head is shaking up and down and left and right, but the person can clearly see the surrounding scenery. This means that the movement of the head when walking is suppressed to a smaller range than other parts of the body (amplitude of about 2 to 3 cm, regardless of the distance of the line of sight), and within this small range, the head and eyeballs maintain a stable line of sight. This is because they work in a coordinated manner to maintain this (see Non-Patent Document 1). This non-patent document 1 teaches that ``when walking, the head oscillates vertically and horizontally, and rotation of the head and compensatory rotation of the eyeballs occur as actions to stabilize the line of sight in response to this oscillation.''

Regarding the electro-oculography sensing technology, there is a known example 1 that detects a person's line of sight position based on electro-oculography caused by eye movements of the person (see Patent Document 1). In this known example 1, an EOG (Electro-Oculography) electrode is used for electro-oculography detection (electro-oculography sensing), and the user's gaze position is detected from electro-oculography data acquired from the EOG electrode (paragraph 0025). In the embodiment of the known example 1, the EOG electrode is provided on the goggles (paragraph 0061), but there is also a known example 2 in which the EOG electrode is provided on the glasses (see Patent Document 2). Goggles and glasses are a type of eyewear worn near the human eyeballs. When a user wearing eyewear walks, the eyewear also moves along with the head.

JP2009-288529A Japanese Patent Application Publication No. 2013-244370

Head movement and eye movement that maintain gaze stability while walking Paper 2000-03 Yazaki, Eiya Osaka University Graduate School of Human Sciences Journal Summary.26 P.177-P.193http://ir.library.osaka-u .ac.jp/dspace/bitstream/11094/5672/1/hs26-177.pdf

In a stationary sitting state (a state in which no disturbance is caused to the head of the user wearing eyewear with EOG electrodes), the line of sight/eyeball rotation can be stably detected by electro-oculography sensing. However, when gazing at a fixed object in the real world while walking (a state in which the head of a user wearing eyewear with EOG electrodes shakes vertically and horizontally), the target line of sight/eyeball Rotation becomes difficult to detect. The reason for this is that eyeball rotation is performed unconsciously as a complementary movement for stabilizing the human body's line of sight, and the signal component of this unconscious eyeball rotation is superimposed on the electro-oculography sensing results (as noise or disturbance). It is.

In other words, when a user wearing eyewear with EOG electrodes walks, even if the user's line of sight is fixed on a specific object in front of them, the electrooculographic data obtained from the EOG electrodes will also fluctuate due to the movement of the head as the user walks. . This variation in the electro-oculogram data becomes noise (disturbance) with respect to the electro-oculogram data corresponding to the actual line of sight.

One of the problems to be solved by the embodiments of the present invention is to provide a wearable device and a display method that change display content depending on the movement of the user's line of sight.

The wearable device according to the embodiment includes a display unit that displays an augmented reality image, a first detection unit that detects movement of the user's head, a second detection unit that detects movement of the user's line of sight, and the first detection unit that detects the movement of the user's head. The movement of the head detected by the detection unit and the movement of the line of sight detected by the second detection unit are compared, and the movement of the head and the movement of the line of sight change from a state in which they change in phase to a state in which they change in phase. control for displaying the augmented reality image on the display unit when detecting a change to a changing state, or a change from a state in which the head movement and the line of sight movement change in opposite phases to a state in which they change in the same phase; and a section.

A diagram illustrating head movement (change in position due to vertical and horizontal movements) of a person who is walking while looking ahead. A diagram illustrating head movement (angular change due to forward bending/backward bending) of a person who is gazing at a fixed object in front of him while walking. FIG. 3 is a diagram illustrating the relationship between optotype distance (distance from the eyes to the object of gaze) and eyeball movement when the head bends back and forth in the vertical direction. FIG. 4 is a diagram illustrating the phase relationship between the angle change/speed change of head forward/backward bending and the angle change/speed change of eye movement. FIG. 1 is a diagram illustrating glasses-type eyewear incorporating an eye movement detection device according to an embodiment (an example in which an EOG electrode is disposed on a nose pad). FIG. 3 is a diagram illustrating an example of mounting EOG electrodes in glasses-type eyewear according to an embodiment. FIG. 2 is a diagram illustrating the relationship between an information processing unit 11 that can be attached to various embodiments and its peripheral devices. A diagram showing an example of an EOG waveform when, for example, one starts walking with the right foot and stops at the 13th step with the right foot while gazing at a fixed object in the real world located 25 meters ahead. 9 is a diagram showing another example of the EOG waveform (confirmation of repeatability) when walking starts with the right foot again and stops at the 13th step with the right foot under the same conditions as FIG. 8. FIG. When the noise caused by head movement is canceled, the detection signal level (Ch0, Ch1, Ch2, and an electrooculogram (EOG) illustrating the relationship between the average level of Ch1 and Ch2 (Ch1+2). When the noise caused by head movement is canceled, the detection signal levels (Ch0, Ch1, Ch2, and Ch1+2) obtained from the three ADCs shown in FIG. An electrooculogram illustrating the relationship. When the noise caused by head movement is canceled, the relationship between eye movement from left to right and the detection signal levels (Ch0, Ch1, Ch2, and Ch1+2) obtained from the three ADCs shown in FIG. An electrooculogram illustrating the relationship. When the noise caused by head movement is cancelled, the eye movement of blinking (both eyes) repeated 5 times at 5 second intervals when the line of sight is facing forward, and the three ADCs shown in Figure 6. The electrooculogram which illustrates the relationship with the detection signal level (Ch0, Ch1, Ch2) obtained from. When the noise caused by head movement is canceled, the eye movements are repeated 5 times by closing the eyes for 1 second (both eyes) and opening the eyes for 4 seconds (both eyes) when the line of sight is facing forward. 7 is an electrooculogram illustrating the relationship between detection signal levels (Ch0, Ch1, Ch2) obtained from the three ADCs shown in FIG. When the noise caused by head movement is cancelled, when the line of sight is facing forward, the eye that blinks the left eye 5 times (blinking of the left eye) immediately after repeating the blinking of both eyes 5 times. 7 is an electrooculogram illustrating the relationship between motion and detection signal levels (Ch0, Ch1, Ch2) obtained from the three ADCs shown in FIG. 6. FIG. When the noise caused by head movement is cancelled, when the line of sight is facing forward, the right eye wink (blink of one right eye) is repeated 5 times immediately after the blink of both eyes is repeated 5 times. 7 is an electrooculogram illustrating the relationship between motion and detection signal levels (Ch0, Ch1, Ch2) obtained from the three ADCs shown in FIG. 6. FIG. 5 is a flowchart illustrating an example of a process for minimizing noise caused by head movement (noise cancellation process). FIG. 7 is a diagram illustrating an example of mounting EOG electrodes in glasses-type eyewear according to another embodiment (an example in which EOG electrodes are arranged around the eyeballs). FIG. 7 is a diagram illustrating an example of mounting EOG electrodes in goggle-type eyewear (a type in which the eye frames of both the left and right eyes are continuous) according to yet another embodiment (another example in which EOG electrodes are arranged around the eyeballs). FIG. 7 is a diagram illustrating an example of mounting EOG electrodes in goggle-type eyewear (a type in which the eyecups for the left and right eyes are separated) according to yet another embodiment (yet another example in which the EOG electrodes are arranged around the eyeballs);

In the following, basic information will first be provided, and then various embodiments will be described with reference to the drawings.
<Basic information>
The diameter of an adult eyeball is approximately 25 mm. At birth, it is about 17 mm long and increases in size as it grows.
The interpupillary distance for an adult male is approximately 65 mm. (Many commercially available stereo cameras are made with 65mm spacing.)
The interpupillary distance of adult women is several mm shorter than that of men.
The ocular potential was several tens of mV.
The eyeball has a positive potential on the cornea side and a negative potential on the retina side. When this is measured on the surface of the skin, it appears as a potential difference of several hundred μV.

The types of eye movements and ranges of eye movement related to eye movement detection include, for example, the following:
<Types of eye movements (eye movements)>
(01) Compensatory eye movements
Involuntary eye movements developed to stabilize images of the external world on the retina regardless of head and body movements.
(02) Voluntary eye movements
This is an eye movement that was developed to center the visual image on the retina, and is a movement that can be controlled voluntarily.
(03) Impulsive eye movements (saccades)
Eye movements that occur when changing the point of gaze to look at something (easy to detect).
(04) Gliding eye movements
Smooth eye movements that occur when tracking a slowly moving object (difficult to detect).
<Range of movement of the eyeballs (for a typical adult)>
(11) Horizontal direction
Left direction: 50° or less
Right direction: 50° or less
(12) Vertical direction
Downward: 50° or less
Upward: 30 degrees or less (The range of vertical angles that can be moved by one's will is narrow only in the upward direction. (Due to the "Bell phenomenon," in which the eyeballs roll upward when the eyes are closed, the vertical range of movement of the eyeballs is limited upwards when the eyes are closed.) direction).
(13) Others
Convergence angle: 20° or less.
<Regarding head movements during walking (Source of Figures 1 and 2 is Non-Patent Document 1)>
FIG. 1 is a diagram illustrating head movement (change in position due to vertical and horizontal movements) of a person who is walking while looking ahead. The position of a person's head changes up and down as the lower limbs move while walking (Figure 1(a)). A typical adult's walking cycle (one walking cycle) is, for example, about 0.6 seconds. At the same time, during this walking, the position of the person's head changes from side to side (FIG. 1(b)). The period of this left-right position change is approximately 1.2 seconds, which is twice the walking period. The vertical position change width and the left and right position change width are each about 2 to 3 cm. This vertical and horizontal movement is approximately linear and regular, and has a cycle closely related to the walking cycle.

FIG. 2 is a diagram illustrating head movement (angular change due to forward bending/backward bending) of a person who is gazing at a fixed object in front of him while walking. When a person who is gazing at a visual target (optograph) ahead starts walking, the position of the head changes up and down as the person walks (FIG. 2(a)). While gazing at a visual target, when the head is in the up position, the head bends forward (pitch down), and when the head is in the down position, the head bends back (pitch up). (Figure 2(b)). Such forward and backward bending (vertical rotational movement of the head) occurs in response to (synchronized with) the vertical movement of the head during one gait cycle. Similarly, when the head position changes from side to side (FIG. 1(b)), a rotational movement in the left and right direction (synchronized with the vertical movement of the head) occurs in the head.

In other words, while walking, a person's head performs rotational movements to compensate for vertical and horizontal movements, and the head moves forward and backward in a high and low position to help the line of sight capture the visual target (optotype) in front of it. give in The same goes for left and right; when the body sways left and right and the head moves left and right while walking, the head rotates left and right (reciprocating rotation). This rotation of the head is called ``compensatory rotation,'' and is thought to help stabilize the gaze.

Furthermore, since a person's trunk rotates back and forth and left and right while walking, the head must also compensate for these rotations. By rotating the head in the opposite direction to this change in the trunk, the angular position of the head is maintained horizontally.
<About eyeball movements associated with walking (Source of Figures 3 and 4 is Non-Patent Document 1)>
FIG. 3 is a diagram illustrating the relationship between optotype distance (distance from the eyes to the object of gaze) and eyeball movement when the head bends back and forth in the vertical direction. Here, the optotype position when the head is fixed is defined as HFP, and when the actual optotype is closer than HFP (Near Target) or farther than HFP (Far Target), the forward/backward bending rotation angle (φh) of the head is calculated. and the compensatory rotation angles (φen, φef) of the eyeballs.

As mentioned above, the head performs compensatory rotation, but this alone cannot fully achieve gaze stabilization. When the head rotates to compensate for vertical movement, it often causes too much or too little. In order to compensate for this deviation due to excess or deficiency of compensatory rotation, periodic compensatory rotation of the eyeballs is required while walking. For example, for a near target located 30 cm ahead, the eyeball performs a compensatory rotation (φen) in the same direction as the rotation/rotation (φh) of the head. As a result, even if the rotation/rotation of the head is insufficient, the eyeballs rotate in coordination with the head in the direction of movement to compensate for the insufficient rotation of the head. On the other hand, for a far target that is 100 meters away, for example, the eyeball performs a compensatory rotation (φef) in the opposite direction to the rotation/rotation (φh) of the head. As a result, even if the head rotates excessively, the eyeballs rotate in the opposite direction to cancel out the excessive head rotation. (Note that there is no real difference in the amount of head movement during walking whether the visual target is close or far away.)
FIG. 4 is a diagram illustrating the phase relationship between the angle change/speed change of head forward/backward bending and the angle change/speed change of eye movement. The angle of forward and backward bending of the head (φh) shown in FIG. 3 can be detected by a three-axis gyro (11e in FIG. 7) attached to the eyewear. Further, the rotation angle (φen, φef) of the eyeball shown in FIG. 3 can be detected by an eye movement detection unit (15 in FIG. 7) that uses the EOG of the eyeball.

Changes in vertical movement of the head (vertical forward and backward bending changes) associated with walking change in the same phase as changes in eyeball rotation when the visual target is close (Near Target) (Fig. 4(a)). On the other hand, when the optotype is far away (Far Target), the vertical movement change of the head (vertical forward/backward bending change) changes in the opposite phase to the change in eyeball rotation (FIG. 4(b)). When the optotype is at the fixed position HFP in Fig. 3, there is no eyeball rotation, and the waveform change of the broken line shown in Figs. 4(a) and (b) becomes minimal (approximately horizontal in the illustration) , the phase between the longitudinal head tilt change and the eye rotation change is reversed). A similar phase reversal is also seen between the speed changes of the head vertical movement and the speed change of the eyeball rotation (FIGS. 4(c) and 4(d)).

Now, signal changes occurring in the EOG due to the above-mentioned "changes in vertical movement of the head associated with walking" will be referred to as "body movement noise." Walking behavior can be detected by an acceleration sensor or gyro sensor, and eye movement behavior can be detected by EOG. Then, when ``walking'' and ``no eye movements'' are detected, if body movement noise does not appear in the EOG, ``the eyewear user will notice that the phase of the body movement noise is reversed depending on the distance of the optotype.'' This indicates that the user is gazing at a point (corresponding to position HFP in FIG. 3).

FIG. 5 is a diagram illustrating a glasses-type eyewear 100 incorporating an eye movement detection device according to an embodiment (an example in which an EOG electrode is disposed on a nose pad). In this embodiment, a right eye frame (right rim) 101 and a left eye frame (left rim) 102 are connected by a bridge 103. The left and right eye frames 102, 101 and the bridge 103 can be made of, for example, aluminum alloy, titanium, or the like. The left outer side of the left eye frame 102 is connected to a left temple bar 106 via a left hinge 104, and a left end (left ear pad) 108 is provided at the tip of the left temple bar 106. Similarly, the right outer side of the right eye frame 101 is connected to a right temple bar 107 via a right hinge 105, and a right modern (right ear pad) 109 is provided at the tip of the right temple bar 107.

An information processing unit 11 (an integrated circuit several millimeters square), which is a main part of the eye movement detection device, is attached to the bridge 103. The information processing section 11 can be configured by an LSI that integrates a microcomputer, a memory, a communication processing section, etc. (details of the information processing section 11 will be described later with reference to FIG. 7).

A small battery (BAT) such as a lithium ion battery is embedded, for example, in the left temple bar 106 (or in the right temple bar 107, or in the modern 108 or 109), and serves as a power source necessary for the operation of the glasses-type eyewear 100. ing. A left camera 13L is attached to the end of the left eye frame 102 near the left hinge 104, and a right camera 13R is attached to the end of the right eye frame 101 near the right hinge 105. These cameras can be constructed using ultra-small CCD image sensors.

These cameras (13L, 13R) may constitute a stereo camera. Alternatively, an infrared camera (13R) and a laser (13L) may be arranged at the positions of these cameras to configure a distance sensor using an infrared camera and a laser. This distance sensor can also be configured with a small semiconductor microphone (13R) that collects ultrasonic waves, a small piezoelectric speaker (13L) that emits ultrasonic waves, and the like.

Note that an embodiment may be considered in which a central camera (not shown) is provided in the bridge 103 portion instead of or in addition to the left and right cameras 13L/13R. Conversely, embodiments may be equipped with no cameras at all (these cameras are shown as cameras 13 in FIG. 7).

A left display 12L is fitted into the left eye frame 102, and a right display 12R is fitted into the right eye frame 101. This display is provided in at least one of the left and right eye frames, and can be constructed using film liquid crystal or the like. Specifically, one or both of the left and right displays 12L and 12R can be constructed using a film liquid crystal display device that employs a polymer dispersed liquid crystal (PDLC) that does not use a polarizing plate. 12).

The transparent left and right displays 12L/12R using film liquid crystal can be used as a means to provide augmented reality (AR) that adds image information such as numbers and characters to the real world seen through glasses.
A nose pad portion is provided between the left and right eye frames 102 and 101 and below the bridge 103. This nose pad section is composed of a pair of a left nose pad 150L and a right nose pad 150R. The right nose pad 150R is provided with right nose pad electrodes 151a, 151b, and the left nose pad 150L is provided with left nose pad electrodes 152a, 152b.

These electrodes 151a, 151b, 152a, 152b are electrically isolated from each other and connected to three AD converters (ADCs 1510, 1520, 1512) via insulated wiring materials (not shown). Outputs from these ADCs have different signal waveforms depending on the eye movements of the user wearing the eyewear 100, and are supplied to the information processing unit 11 as digital data corresponding to the user's eye movements. The electrodes 151a, 151b, 152a, and 152b are used as line-of-sight detection sensors, and together with the three AD converters, are constituent elements of the eye movement detection section 15 in FIG. 7.

Eyewear 100 is fixed to the user's head (not shown) by left and right nose pads (150L, 150R), left and right temple bars (106, 107), and left and right temple bars (108, 109). In this embodiment, the only things that directly touch the user's head (or face) are the left and right nose pads (150L, 150R), the left and right temple bars (106, 107), and the left and right temple bars (108, 109). However, there may be embodiments in which parts other than these (nose pad, temple bar, and end) touch the user, such as for voltage adjustment between the ADC (1510, 1520, 1512) and the user's body.

The film liquid crystal of the right display 12R in FIG. 5 can display a right display image IM1 including, for example, a numeric keypad (numbers, operators, Enter, etc.), alphabets, marks of predetermined shapes, and other icon groups. Further, the film liquid crystal of the left display 12L can display a left display image IM2 including, for example, arbitrary character strings, marks of arbitrary shapes, various icons, etc. (the display contents of the displays 12L and 12R may be anything). The numeric keypad and alphabets displayed on the right display 12R (or left display 12L) can be used to input numbers and characters. Character strings, marks, icons, etc. displayed on the right display 12R (or left display 12L) can be used to search for a specific information item, select/determine a desired item, or alert the user. .

The display images IM1 and IM2 can be used as an augmented reality (AR) display means that adds information such as numbers, letters, marks, etc. to the real world seen through the glasses, and this AR display can be turned on and off as appropriate. The content of display image IM1 and display image IM2 may be the same content (IM1=IM2) or different content (IM1≠IM2) depending on the embodiment. Further, the display image IM1 (or IM2) can be displayed on the right display 12R and/or the left display 12. When the content of the AR display is desired to be a 3D image (with depth) that overlaps with the real world seen through glasses, IM1 and IM2 can be made into separate left and right 3D images.

Furthermore, when the displays (12R, 12L) are present on the left and right sides, the images of the left and right display images (IM1, IM2) can be shifted in opposite directions by adjusting the convergence angle, for example. This is thought to reduce the strain on the eyes when alternately viewing an object visible in the real world and the AR display. However, normally, the left and right displays (12R, 12L) display images with the same content.

Display control on the displays 12L and 12R can be performed by the information processing section 11 embedded in the right temple bar 107. (Techniques for displaying characters, marks, icons, etc. on a display are well known.) Power necessary for the information control section 11 and other operations can be obtained from a battery BAT embedded in the left temple bar 106.

Note that when a designer or designer wears a prototype of the eyewear 100 corresponding to the embodiment, there is a possibility that the designer or designer may feel that the weight balance is poor. If the main cause is the BAT in the left temple bar 106, a "weight (or another battery of the same weight)" corresponding to the BAT in the left temple bar 106 can be placed in the right temple bar 107.

FIG. 6 is a diagram illustrating an example of mounting EOG electrodes in the glasses-type eyewear 100 according to one embodiment. Right nose pad electrodes 151a and 151b are provided above and below the right nose pad 150R, and left nose pad electrodes 152a and 152b are provided above and below the left nose pad 150L. The outputs of the right nose pad electrodes 151a, 151b are given to the ADC 1510, the outputs of the left nose pad electrodes 152a, 152b are given to the ADC 1520, and the outputs of the lower electrodes 151b, 152b (or upper electrodes 151a, 152a) of the left and right nose pads. is given to ADC1512.

The ADC 1510 obtains a Ch1 signal that changes in response to the user's right up and down eye movements. The ADC 1520 obtains a Ch2 signal that changes in response to the user's left vertical and vertical eye movements. The ADC 1512 obtains a Ch0 signal that changes in response to the user's left and right eye movements. The vertical movement of both the left and right eyes can be evaluated using the Ch1+2 signal corresponding to the average of the outputs of the ADC 1510 and ADC 1520. (The relationship between the signal waveforms of Ch0, Ch1, Ch2, and Ch1+2 and eye movements will be described later.)
FIG. 7 is a diagram illustrating the relationship between the information processing unit 11 that can be attached to various embodiments and its peripheral devices. In the example of FIG. 7, the information processing section 11 includes a processor 11a, a nonvolatile memory 11b, a main memory 11c, a communication processing section 11d, a sensor section 11e, and the like. The processor 11a can be composed of a microcomputer with processing capacity according to product specifications. Various programs executed by this microcomputer and various parameters used when executing the programs can be stored in the nonvolatile memory 11b. The main memory 11c provides a work area for executing the program.

The sensor unit 11e includes a group of sensors for detecting the position and/or orientation of the eyewear 100 (or the head of the user wearing the eyewear). Specific examples of these sensor groups include an acceleration sensor that detects movement in 3-axis directions (x-y-z directions), a gyro that detects rotation in 3-axis directions, and a geomagnetic sensor that detects absolute direction (compass function). There are also beacon sensors that obtain location information and other information by receiving radio waves and infrared rays. To obtain this location information and other information, iBeacon (registered trademark) or Bluetooth (registered trademark) 4.0 can be used.

LSIs that can be used in the information processing section 11 have been commercialized. One example is Toshiba Semiconductor & Storage Company's "TZ1000 series for wearable terminals." Among this series, the product name "TZ1011MBG" includes CPU (11a, 11c), flash memory (11b), Bluetooth Low Energy (registered trademark) (11d), sensor group (acceleration sensor, gyro, geomagnetic sensor) (11e) , 24-bit delta-sigma ADC, and I/O (USB, etc.).

The information processing unit 11 is attached to the bridge 103 as an example of a location where the relative positional relationship between the sensor unit 11e such as an acceleration sensor or a gyro sensor and the EOG electrodes (151a, 151b, 152a, 152b) is maintained.
What to do with the processor 11a can be instructed from an external server (or personal computer, not shown) via the communication processing section 11d. The communication processing unit 11d can use existing communication methods such as ZigBee (registered trademark), Bluetooth, and Wi-Fi (registered trademark). The processing results in the processor 11a can be sent to a server (not shown) or the like via the communication processing section 11d.

A display 12 (12L and 12R), a camera 13 (13L and 13R), an eye movement detection unit 15, and the like are connected to the system bus of the information processing unit 11. Each device (11-15) in FIG. 7 is powered by a battery BAT.
Note that the communication processing unit 11d may incorporate a GPS (Global Positioning System) function, which is well known in mobile phones, smartphones, etc. (or a mobile phone function with GPS may be installed somewhere in the eyewear 100). If the processor 11a executes an Internet map service application that uses this GPS function, it will be possible to determine where the user wearing the eyewear 100 is currently walking and in what direction he or she is looking in the eyeball compensation rotation while walking. It is possible to suppress the noise caused by the detection. (Even though conventional mobile phone GPS can display the user's current location on the map screen, it is not possible to display where the user's line of sight is pointing.)
The eye movement detection unit 15 in FIG. 7 includes four eye movement detection electrodes (151a, 151b, 152a, 152b) constituting a line of sight detection sensor, and three ADCs ( 1510, 1520, 1512) and a circuit that outputs output data from these ADCs to the processor 11a side. The processor 11a can interpret a command corresponding to the type of eye movement (up and down movement, left and right movement, blinking, eye closing, etc.) of the user, and can execute the command.

Specific examples of commands that correspond to the type of eye movement include, for example, if the eye movement is closing the eyes, selecting an information item in front of the line of sight (similar to a single click on a computer mouse), blinking multiple times in a row, or winking. then there is a command (analogous to a double-click on a computer mouse) that starts the execution of an operation on the selected information item. This command is an example of information input B using the eye movement detection unit 15.

FIG. 8 is a diagram showing an example of an EOG waveform when, for example, a person starts walking with his right foot and stops at the 13th step with his right foot while gazing at a fixed object in the real world located 25 meters ahead. The compensatory rotation of the eyeballs corresponding to the vertical movement of the head accompanying walking is reflected in changes in the ADC detection signal levels (EOG changes) of Ch1 and Ch2. The vertical movement change of the head in synchronization with this EOG change can be detected by the three-axis acceleration sensor included in the sensor section 11e of the information processing section 11. Further, the compensatory rotation of the eyeball corresponding to the left-right rotation of the head accompanying walking is reflected in the change in the ADC detection signal level (EOG change) of Ch0. The left-right rotational change of the head in synchronization with this EOG change can be detected by the 3-axis gyro sensor included in the sensor section 11e of the information processing section 11.

In the example in Figure 8, the EOG fluctuation period of Ch1 and Ch2 (approximately 0.6 seconds) is synchronized with the walking pitch, and the EOG fluctuation period of Ch0 (approximately 1.2 seconds) is synchronized with twice the walking pitch. ing. The period of EOG fluctuation corresponding to the vertical movement and left/right rotation of the head associated with walking is clear, so the period is synchronized with the walking pitch (in this example, approximately 0.6 seconds and twice that), and is caused by walking. EOG fluctuations can be detected.

FIG. 9 is a diagram showing another example of the EOG waveform (confirmation of repeatability) when the user starts walking with the right foot again and stops at the 13th step under the same conditions as FIG. 8. In the example of Fig. 9, the EOG fluctuation period of Ch1 and Ch2 (approximately 0.6 seconds) is synchronized with the walking pitch, and the EOG fluctuation period of Ch0 (approximately 1.2 seconds) is synchronized with twice the walking pitch. ing. The EOG fluctuation period corresponding to the vertical movement and left/right rotation of the head accompanying walking is clearly recognized, and the period synchronized with the walking pitch (approximately 0.6 seconds in this example, twice that), indicating that it is caused by walking. It can be seen that EOG fluctuations can be detected.

The following features are commonly observed in Figures 8 and 9, confirming that the phenomenon is reproducible:
(1) Eyeball rotation movement in the vertical direction can be confirmed in accordance with the walking cycle (approximately 0.6 seconds) (see EOG detection signal levels of Ch1 and Ch2).
(2) Eyeball rotation movement in the left-right direction can be confirmed at a cycle twice the walking cycle (approximately 1.2 seconds) (see EOG detection signal level of Ch0).

(3) The amplitude of the EOG detection signal caused by head movement associated with walking reaches 200 μV to 400 μV in some places. Since the EOG signal amplitude measured when the head is stationary is usually about 1 mV, 200 μV to 400 μV becomes noise that cannot be ignored when detecting arbitrary eyeball rotation. This noise can be canceled out (reduced or eliminated) as described below.

In other words, the vertical movement of the head associated with walking can be detected by an acceleration sensor at a cycle synchronized with the walking pitch (approximately 0.6 seconds in this example), and the left-right rotation of the head associated with walking can be detected with another cycle synchronized with the walking pitch. It can be detected by a gyro sensor at a cycle (approximately 1.2 seconds in this example). This makes it possible to cancel (reduce or eliminate) EOG fluctuations (considered as noise) caused by the vertical movement (sway) of the head associated with walking, using the detection results of the acceleration sensor, and the left-right rotation of the head associated with walking. EOG fluctuations (regarded as noise) caused by (shaking) can be canceled out (reduced or eliminated) by the detection results of the gyro sensor (a specific example of this cancellation will be described later with reference to FIG. 17).

FIG. 10 shows upward eye movement from the front and the detection signal levels (Ch0, Ch0, 2 is an electrooculogram (EOG) illustrating the relationship between Ch1, Ch2, and the average level of Ch1 and Ch2 (Ch1+2). Eye movement detection is performed based on the detection signal waveform within the broken line frame in the figure. The detection standard is when the user wearing the eyewear 100 is looking straight ahead and there is no eye movement (if the noise caused by head movement is canceled, the detection from the three ADCs shown in FIG. The output signal waveforms Ch0 to Ch2 are substantially flat in the section where the user is looking straight ahead without blinking, and there is almost no change over time). With the user's left and right eyes facing forward, the user's eyes instantly move upward, maintain the upward gaze for 1 second, and then instantly return the gaze to the front. FIG. 10 shows an example of changes in the detection signal level when this process is repeated five times.

FIG. 11 illustrates eye movement detection similar to that in FIG. 10 in a case where the line of sight moves downward from the front. As long as the noise caused by head movement is canceled out, even while walking, the line of sight is upward, based on the waveform changes in Figures 10 and 11, compared to when the line of sight is facing forward. It can detect whether it is above or below.

FIG. 12 shows the detected signal levels (Ch0, Ch1, Ch2, and It is an electrooculogram illustrating the relationship with Ch1+2). When there is an eye movement from left to right, the Ch0 detection signal waveform changes over time to the right (although not shown, when there is an eye movement from right to left, the Ch0 detection signal waveform changes over time) , the right shoulder is downward). From such a waveform change of Ch0, it is possible to detect whether the line of sight is to the right or to the left, based on the case where the line of sight is facing forward.

By integrating the detection results shown in FIGS. 10 to 12, the following can be found. In other words, if the noise caused by head movement is canceled out, even when walking, you can tell whether your line of sight is facing up, down, left, or right, based on when your line of sight is facing forward. .
Figure 13 shows the eye movements of blinking/blinking (both eyes) repeated 5 times at 5-second intervals when the line of sight is facing forward, and the figure when the noise caused by head movement is canceled. 6 is an electrooculogram illustrating the relationship with detection signal levels (Ch0, Ch1, Ch2) obtained from the three ADCs shown in FIG. Blinking/blinking of both eyes can be detected by the pulses appearing on Ch1 and Ch2. Blinks that users unconsciously perform often have no periodicity. Therefore, by detecting a plurality of pulses at regular intervals as shown in FIG. 13, intentional blinking/blinking of the user can be detected.

According to one theory, the time required for a person's "blink/blink" action or the time that vision is obstructed by "blink/blink" to be about 300 msec (according to other theories, it is about 100 msec to 150 msec). In either case, signal components due to "blinks/blinks" can be removed by a high-pass filter (or band-pass filter) below the walking pitch period. The blink/blink interval (for example, about 300 msec) is different from the normal walking pitch (about 600 msec), and the unintentional blink/blink interval is not synchronized with the normal walking pitch. Therefore, when canceling EOG noise caused by head movement associated with normal walking, EOG waveform changes due to blinking/blinking are not completely canceled.

Figure 14 shows the repetition of 1 second of eyes closed (both eyes) and 4 seconds of eyes open (both eyes) when the line of sight is facing forward, when the noise caused by head movement is cancelled. 7 is an electrooculogram illustrating the relationship between the eye movements and detection signal levels (Ch0, Ch1, Ch2) obtained from the three ADCs shown in FIG. 6. FIG. Eye closure of both eyes can be detected by wide pulses appearing on Ch1 and Ch2 (the time of intentional eye closure is longer than the time of blinking/blinking, so the detected pulse width becomes wider) . By detecting wide pulses of Ch1 and Ch2 as illustrated in FIG. 14, intentional eye closure of the user can be detected.

Although not shown, when the user closes only his right eye, a wide pulse with large amplitude appears on Ch1, and when the user closes only his left eye, a wide pulse with large amplitude appears on Ch2. From this, it is also possible to detect eye closure separately for the left and right eyes.
When only one eye is closed, small waves appear on Ch0 with opposite phases on the left and right sides. When viewed on Ch0, a small uneven waveform as shown below appears.

・When you close your right eye, the positive potential of the - electrode decreases, resulting in a convex waveform;
・When you close your left eye, the positive potential of the + electrode decreases, resulting in a concave waveform.
Figure 15 shows that when the noise caused by head movement is canceled out, when the line of sight is facing forward, the left eye wink (blink of one left eye) is repeated 5 times immediately after repeating the blink of both eyes 5 times. 7 is an electrooculogram illustrating the relationship between repeated eye movements and detection signal levels (Ch0, Ch1, Ch2) obtained from the three ADCs shown in FIG. 6. FIG.

As illustrated in FIG. 6, the position of the Ch0 ADC 1512 is offset downward from the eyeball center line of both the left and right eyes. Due to this offset, when both eyes blink simultaneously, negative potential changes appear at both the + and - inputs of the ADC 1512 in FIG. At this time, if the potential changes (amount and direction) of both the + input and - input are approximately the same, the changes are almost canceled out, and the value of the signal level output from the ADC 1512 of Ch0 becomes approximately constant ( (See Ch0 level within the dashed line on the left side of FIG. 15). On the other hand, when one eye (left eye) blinks, there is almost no potential change on the - input side of the ADC 1512, and a relatively large negative potential change appears on the + input side of the ADC 1512. Then, the amount of cancellation of the potential change between the + and - inputs of the ADC 1512 becomes smaller, and a small pulse (small wave of signal level) appears in the negative direction in the signal level output from the ADC 1512 of Ch0 (see Fig. 15). (See Ch0 level within the dashed line on the right). From the polarity of this signal level small wave (pulse in the negative direction), it is possible to detect that the left eye has winked (left wink detection using Ch0 when EOG noise caused by head movement is canceled) example).

Note that if the potential changes at the + and - inputs of the ADC 1512 are not equal due to the distortion of the user's face or the condition of the skin, the output of the Ch0ADC will be the minimum ( Calibration may be performed in advance so that the amount of cancellation between the + input component and the - input component is maximized.

Furthermore, if the peak ratio SL1a/SL2a of detection signals Ch1/Ch2 when both eyes blink/blink is performed as a reference, the peak ratio SL1b/SL2b when a left eye wink is performed changes (SL1b/SL2b is not equal to SL1a/SL2a). From this reason as well, a left wink can be detected.

Figure 16 shows that when the noise caused by head movement is cancelled, when the line of sight is facing forward, the right eye wink (blink of one right eye) is repeated 5 times immediately after repeating the blink of both eyes 5 times. 7 is an electrooculogram illustrating the relationship between repeated eye movements and detection signal levels (Ch0, Ch1, Ch2) obtained from the three ADCs shown in FIG. 6. FIG.

As described above, since the position of the ADC 1512 in FIG. 6 is offset downward from the eyeball center line of both the left and right eyes, a negative potential change appears in both the + input and the - input of the ADC 1512 when both eyes blink at the same time. However, similar potential changes at the + input and - input are almost canceled out, and the value of the signal level output from the ADC 1512 of Ch0 becomes approximately constant (see the Ch0 level within the broken line on the left side of FIG. 16). On the other hand, when one eye (right eye) blinks, there is almost no potential change on the + input side of the ADC 1512, and a relatively large negative potential change appears on the - input side of the ADC 1512. Then, the amount of cancellation of the potential change between the - input and + input of the ADC 1512 becomes small, and a small pulse (small wave of signal level) appears in the positive direction in the signal level output from the ADC 1512 of Ch0 (see Fig. 16). (See Ch0 level within the dashed line on the right). From the polarity of this small signal level wave (pulse in the positive direction), it is possible to detect a wink of the right eye (right wink detection using Ch0 when EOG noise caused by head movement is canceled). example).

Furthermore, if the peak ratio SR1a/SR2a of detection signals Ch1/Ch2 when both eyes are blinked is used as a reference, the peak ratio SR1b/SR2b when a right eye wink is performed changes (SR1b/SR2b is SR1a/SR2b). (not equal to SR2a). Furthermore, the peak ratio SL1b/SL2b during the left wink has a different value from the peak ratio SR1b/SR2b during the right wink (the degree of difference can be confirmed by experiment). From this, the left wink can be detected separately from the right wink (an example of left and right wink detection using Ch1 and Ch2).

A device designer may appropriately decide whether to use Ch0 or Ch1/Ch2 for left and right wink detection. The left and right wink detection results using Ch0 to Ch2 can be used as operation commands.
FIG. 17 is a flowchart illustrating an example of processing for minimizing noise caused by head movement (noise cancellation processing). A computer program corresponding to this flowchart is stored, for example, in the nonvolatile memory 11b of FIG. 7, and executed by the processor 11a.

First, a user wearing eyewear 100 as illustrated in FIG. 5 walks at a constant speed at a constant rhythm while gazing at a fixed object in front of him (ST10). As the user walks, the user's head oscillates (moves/rotates) in a predetermined pattern (see FIGS. 1(a) and 1(b)).
The movement/rotation of the user's head while walking is detected by an acceleration sensor/gyro sensor (11e in FIG. 7) provided inside the information processing unit 11 in FIG. 5 (ST12). The movement/rotation of the user's head accompanying walking is mixed into the EOG detection signal waveform as noise (see FIG. 8 or 9). The detection signal of the acceleration sensor/gyro sensor is added to this EOG detection signal waveform at a predetermined phase (ST14).

As a result of the above signal addition, if the signal amplitude has increased (ST16 NO), the phase of the detection signal to be added is inverted (ST18), and the acceleration sensor/gyro sensor detection signal is added to the EOG detection signal waveform again. (ST14).
If the signal amplitude decreases as a result of the signal addition described above (ST16 YES), the level of the signals to be added is adjusted so that the signal amplitude of the addition result becomes the minimum (minimum) (ST20). By minimizing the signal amplitude of the addition result (minimum), noise mixed into the EOG detection signal waveform is canceled out (reduced or eliminated).

Based on the EOG detection results after noise caused by head movement (movement/rotation) has been canceled, the user's eye movements (gaze movement, blinking, closing, winking, etc. in Figures 10 to 16) are detected. A determination is made (ST22), and processing based on the determination is performed (ST24). For example, when the user winks, a process can be performed in which the name of the building in front of the user's line of sight is displayed in AR on the display screen of the eyewear 100 (IM1/IM2 in FIG. 5).

When the movement pattern of the user's head changes, the content of noise cancellation also changes. Therefore, whether the movement pattern of the user's head has changed is detected from the change in the detection results of the acceleration sensor/gyro sensor (ST26). For example, assume that a user wearing eyewear 100 who is walking at a constant speed while keeping his eyes fixed on a fixed object in front of him stops in front of the stairs leading down to the subway, and sweeps his gaze by bending forward and backward and moving his gaze over the difference between adjacent steps of the stairs. . Then, the movement pattern of the head becomes different from that when walking due to the line of sight sweep before and after this step, and the EOG detection signal waveform also becomes different from that when walking.

Therefore, when the motion pattern of the user's head changes (ST26 YES), the process returns to step ST12 and the processes from ST14 to ST20 are redone. By redoing the processing from ST14 to ST20, EOG noise caused by a new head movement pattern is canceled out (reduced or eliminated), and EOG corresponding to pure line of sight movement is detected. The EOG detection result corresponding to this pure line of sight movement is compared with the detection result of the acceleration sensor/gyro sensor corresponding to the changed movement pattern of the user's head. As a result of this comparison, for example, if a phase reversal from FIG. 4(a) to FIG. 4(b) is detected at the step position of the stairs, it is determined that there is a step of the stairs near the phase reversal point (ST22 ). Then, a mark indicating the presence of a step is displayed in AR on the eyewear 100 (ST24). This AR display can be automatically turned off after a certain period of time (for example, 10 seconds) after the detection result of the acceleration sensor/gyro sensor corresponding to the movement pattern of the user's head disappears.

The AR display can also be turned on/off automatically depending on whether the EOG waveform (Ch1/Ch2 waveform in FIG. 8 or 9) and the detection signal waveform of the acceleration sensor are synchronized. When the user is looking at the AR display, the eyeballs do not rotate to compensate for the movement of the head, so the above synchronization relationship does not exist, but when the user is gazing at the real world rather than the AR display, the above synchronization relationship does not exist. A synchronous relationship occurs. In a similar way, the AR display can be automatically turned on/off depending on whether the EOG waveform (Ch0 waveform in FIG. 8 or 9) and the detection signal waveform of the gyro sensor are synchronized.

If the user continues to use the eyewear while the motion pattern of the user's head does not change (ST28 NO), the processing in ST22 to ST24 continues. If the user instructs the processor 11a in FIG. 7 to end the use of the eyewear 100 by, for example, closing his eyes for several seconds (ST28 YES), the process in FIG. 17 ends.

FIG. 18 is a diagram illustrating an example of mounting EOG electrodes in glasses-type eyewear 100 according to another embodiment (an example in which EOG electrodes are arranged around the eyeballs). Eyewear 100 in FIG. 18 differs from eyewear 100 in FIG. 5 in the following points.
The first difference is that the EOG electrodes 151a and 151b of the right nose pad 150R have moved to the right eye frame 101 side, and the EOG electrodes 152a and 152b of the left nose pad 150L have moved to the left eye frame 102 side. . EOG electrodes 151a and 151b are arranged at positions that are approximately symmetrical with respect to the center position of the user's right eye (not shown), and EOG electrodes 152a and 152b are arranged at positions that are substantially symmetrical with respect to the center position of the user's left eye (not shown). They are placed in positions that are approximately point symmetrical. Further, the right diagonal line connecting each of the EOG electrodes 151a and 151b and the left diagonal line connecting each of the EOG electrodes 152a and 152b are approximately symmetrical with respect to a vertical line (not shown) along the bridge of the user's nose. The electrodes 151a, 151b, 152a, and 152b can be composed of a conductive member (metal, conductive polymer, etc.) provided at the tip of an elastic body (sponge, silicone cushion, etc.). Each EOG electrode is lightly pressed against the skin surface of the face of the user wearing the eyewear 100 by the elastic repulsive force of the elastic body.

If the EOG electrode arrangement structure as shown in FIG. 18 is adopted, it becomes easier to detect the electric field generated around the charged eyeball compared to a structure in which the EOG electrode is arranged in the nose pad portion. Therefore, an EOG signal with a larger amplitude can be detected in the embodiment of FIG. 18 than in the embodiment of FIG.

The second difference is that the sensor section 11e including an acceleration sensor and a gyro sensor is left in the bridge 103, and the other functions of the information processing section 11 are moved to the temple bar 107 side. For example, if the physical size of the information processing unit 11 increases due to enhanced information processing capacity or the addition of a GPS function, attaching the enlarged information processing unit 11 to the bridge 103 will improve the design of the eyewear 100 and its attachment. There is a risk that problems may arise. If only the sensor section 11e is attached to the bridge 103, the structure on the bridge 103 will be smaller and lighter than when the entire information processing section 11 is provided, so the design and wearing comfort may be improved. be. On the other hand, if a relatively large structure is provided inside the temple bar 107 (and/or 106), there is little risk of problems with the design of the eyewear 100 or its wearing comfort.

FIG. 19 is a diagram illustrating an implementation example of EOG electrodes in goggle-type eyewear (a type in which the eye frames of both the left and right eyes are continuous) according to yet another embodiment (an example of implementation of EOG electrodes in goggle-type eyewear 100 in which the EOG electrodes are arranged around the eyeballs). Example). When the structure shown in FIG. 19 is adopted, the EOG electrode has higher skin contact stability than the structure shown in FIG. 18. Therefore, the EOG signal can be detected with higher accuracy while canceling out noise caused by head movement. Furthermore, with the structure of FIG. 19, it is easy to incorporate the enlarged information processing section 11 and other devices. Therefore, with the structure shown in FIG. 19, the functions of a GPS smartphone can be incorporated without worrying too much about the design. In that case, the screen display can be performed by AR display on the left and right displays 12L/12R using film liquid crystal or the like. Although not shown, small speakers may be attached near the left and right ears of the goggle frame, and a small microphone may be attached near the nose cushion. In addition, commands to the smartphone can be input by eye movements such as line of sight movement, blinking, closing the eyes, and winking.

FIG. 20 is a diagram illustrating an example of mounting EOG electrodes in goggle-type eyewear (a type in which the left and right eyecups are separated) 100 according to yet another embodiment (a further embodiment in which EOG electrodes are arranged around the eyeballs). example). The embodiment of FIG. 20 employs the same EOG electrode arrangement structure and sensor section 11e arrangement structure as in the case of FIG. 18. However, the goggle structure shown in FIG. 20 can withstand underwater use (or spacewalk training use). The head of a diver swimming underwater moves more than when walking on land, and this large movement (up, down, left and right position movement and rotation) can be detected by the 3-axis acceleration sensor and 3-axis gyro sensor in the sensor section 11e. The detected oscillation component can be used to cancel noise components mixed into the EOG signal. Using EOG signals with suppressed noise caused by head movements, it is possible to input various commands based on the eye movements (blinks, closing, etc.) of the user (diver) with both hands occupied. .
<Summary of embodiments>
(a) Conventional line-of-sight detection technology uses an infrared camera to estimate the line-of-sight direction through image processing, but the amount of equipment required (in addition to high-brightness infrared LEDs, Cameras that can take pictures, relatively high-performance computing devices for image processing, and power supply devices that operate these devices will be large-scale. For this reason, conventional line of sight detection technology has been applied to stationary devices, but its application to wearable devices has not progressed.

On the other hand, line of sight detection using electro-oculography sensing requires a much smaller amount of equipment (electrodes, ADC, relatively low-performance arithmetic unit, and power supply to operate these) compared to the infrared camera method. Due to its large scale, it can also be applied to wearable devices such as AR glasses. EOG is a technology that can be expected to be applied to wearable devices because it is highly compatible with AR glasses and consumes less power than infrared systems.

(b) To cancel the shaking of the eyewear user's head when gazing at a fixed object in the real world. This improves detection accuracy for arbitrary eyeball rotations.
(c) Detect whether the fixed object in the real world that is being watched is a distant view or a near view (detection of phase inversion points in FIG. 4). Thereby, the presence or absence of gaze on the real world (fixed object) is detected from the shaking of the eyewear user's head.

(d) The eyewear user's head shaking is used for automatic on/off control of the AR display.
(e) Extract only arbitrary eyeball movements that cancel out the compensatory rotation of the eyeballs.
(f) Detect whether what you are looking at is in the real world (if the detected waveform of the acceleration sensor and/or gyro sensor and the electrooculography EOG waveform are synchronized when walking, you can see what is in the real world) If the detection waveform of the acceleration sensor and/or gyro sensor and the electro-oculogram EOG waveform are not synchronized, it is determined that the user is looking at an AR display or the like while walking).

(g) Effective as noise cancellation when eye movement is used as a UI (user interface).
(h) If used to collect work history information using eyewear, it will become an unprecedented indicator (for example, when removing the influence of noise caused by head movement, it is possible to determine the correct target in warehouse picking work, etc.). It can be used for pass/fail judgments such as "If the number of times a person looks at the person is above a certain value, he/she passes the job as a worker."
<Example of correspondence between content of original claims and embodiments>
[1] The electro-oculography detection device according to one embodiment (11 in FIG. 7: includes a processor 11a that executes the process shown in FIG. an eye movement detection unit (15; 151a, 151b, 152a, 152b) that detects eye movement (gaze movement, blinking, etc.) including eyeball rotation of the user; and an acceleration sensor (11e) that detects movement of the eyewear. (part of 11e) and a gyro sensor (part of 11e) that detects rotation of the eyewear. This eye movement detection device includes a detection means for detecting movement and/or rotation of the eyewear using the acceleration sensor and/or the gyro sensor when the eye movement is detected based on the electrooculogram (EOG). (Executing ST12 11a) and noise mixed into the electro-oculography due to movement and/or rotation of the eyewear (electro-oculography component added by compensatory eyeball rotation that acts in a direction to cancel head movement) ) by combining signal components (components having a phase opposite to mixed noise) detected by the acceleration sensor and/or the gyro sensor (by performing ST14 to ST20). 11a).

[2] The device of [1] above includes phase selection means (ST16 to The apparatus further includes 11a) that executes ST18.

[3] The device (11) of [1] above is incorporated into eyewear (100 in FIGS. 5 and 18 to 20).
[4] The eyewear (100) of [3] has a display section (12L, 12R), and displays a display (AR display) based on the electro-oculography (EOG) after noise has been reduced by the noise reduction means. The display section (12L, 12R) is provided with display means (11a for executing ST22 to ST24).

[5] The device of [1] detects the rotation of the user's head corresponding to a change in forward/backward bending using the gyro sensor (11e), and detects the change in the user's line of sight (Figs. 10 and 11: Ch1 The eye movement detecting section (15) detects the change in the vertical direction of the line of sight from the ± level change of /Ch2, and the signal phase of the anteroposterior change and the signal phase of the change in the direction of the line of sight are in phase (see FIG. 4). a)) or reverse phase (Fig. 4(b)). (can be detected).

[6] The device of [1] above is configured to detect movement of the eyewear and/or detected by the acceleration sensor and/or the gyro sensor when the eye movement is detected based on the electro-oculogram (EOG). Alternatively, if the rotation (Fig. 1) is synchronized with the change in the electro-oculogram (EOG) (Fig. 8 or 9) (if the "predetermined phase relationship" can be maintained in ST14 of Fig. 17), the user can It is configured to determine that it is what it is looking at (it can detect whether what it is gazing at is in the real world or not).

[7] The electro-oculography detection method according to one embodiment detects eye movements (gaze movement, blinking, etc.) including eyeball rotation of the user based on the electro-oculogram (EOG) of the user wearing the eyewear (100). , detecting movement of the head of the user wearing the eyewear, and detecting rotation of the head of the user wearing the eyewear. This method includes a step of detecting movement and/or rotation of the head of the user wearing the eyewear when the eye movement is detected based on the electrooculography (EOG) (ST12); The noise mixed into the electro-oculogram due to the movement and/or rotation of the head of the user wearing the wear (the electro-oculogram component added by compensatory eyeball rotation that acts in a direction to cancel the head movement) is Step of reducing (minimizing or canceling) by synthesizing signal components (components with opposite phase to mixed noise) detected in response to movement and/or rotation of the head of the user wearing the eyewear (ST14 ~ ST20).

[8] The method of [7] above includes adjusting the phase of the signal components detected in response to the movement and/or rotation of the head of the user wearing the eyewear so that the result of combining the signal components tends to decrease. The method further includes a step of selecting (ST16 to ST18).
Although several embodiments of the invention have been described, these embodiments are presented by way of example and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, substitutions, and changes can be made without departing from the gist of the invention.

For example, in the description of the embodiments, glasses-type and goggle-type devices were introduced as eyewear equipped with EOG electrodes, but other items such as eye masks, eye patches, helmets, hats, and hoods can also be used with EOG electrodes. It is also possible to use it for eyewear with accessories.
These embodiments and their modifications are included within the scope and gist of the invention, as well as within the scope of the invention described in the claims and its equivalents. Note that it is within the scope of the invention to combine part or all of one embodiment of the plurality of disclosed embodiments with part or all of another embodiment of the plurality of disclosed embodiments. and abstracts.

100... Eyewear (goggle type or glasses type); 110... Eye frame; 11... Information processing unit (processor 11a, non-volatile memory 11b, main memory 11c, communication processing unit (GPS processing part) 11d, an integrated circuit including sensor part 11e, etc.; 11e... sensor part including an acceleration sensor, gyro sensor, etc.; BAT... power source (lithium ion battery, etc.); 12... display part (right display 12R and left display 12L: IM1...Right display image (numeric keypad, alphabet, character string, mark, icon, etc.); IM2...Left display image (numeric keypad, alphabet, character string, mark, icon, etc.);13...Camera (right camera 13R) and the left camera 13L or a center camera (not shown) attached to the bridge 103); 15...Eye movement detection unit (line of sight detection sensor); 1510...Right side (Ch1) AD converter; 1520...Left side (Ch2) AD converter; 1512 ... Left-right AD converter (Ch0); 1514... Left-right AD converter (Ch3).

Claims (4)

a display unit that displays an augmented reality image;
a first detection unit that detects movement of the user's head;
a second detection unit that detects movement of the user's line of sight;
The movement of the head detected by the first detection unit and the movement of the line of sight detected by the second detection unit are compared, and from a state in which the movement of the head and the movement of the line of sight change in phase. When detecting a change to a state in which the movement of the head and the movement of the line of sight change in opposite phases to a state in which they change in opposite phases to a state in which they change in phase, the augmented reality image is displayed on the display unit. A control unit to display,
A wearable device equipped with The wearable device according to claim 1, wherein the control section stops the display section of the augmented reality image after a certain period of time after the first detection section stops detecting the movement of the head. a display unit that displays an augmented reality image;
a first detection unit that detects movement of the user's head;
a second detection unit that detects movement of the user's line of sight;
A display method for a wearable device comprising:
Comparing the movement of the head detected by the first detection unit and the movement of the line of sight detected by the second detection unit,
The head movement and the line of sight movement change from a state in which they change in phase to a state in which they change in antiphase, or the head movement and the line of sight movement change from a state in which they change in antiphase to a state in which they change in phase. A display method comprising: displaying the augmented reality image on the display unit when a change to a state is detected. 4. The display method according to claim 3, wherein the display unit for the augmented reality image is stopped a certain period of time after the first detection unit stops detecting the movement of the head.

Images