JP2021009719A - Electronic apparatus and display method - Google Patents

Abstract

To provide an eyeball rotation detection device that can individually detect the left and right rotation of the right eyeball and the left and right rotation of the right eyeball.SOLUTION: According to an embodiment, an eyeball rotation detection device is provided with first to fifth electrodes and a detection circuit. The first electrode is provided on the right of the right eyeball, the second electrode on the left of the left eyeball, the third electrode on the left of the right eyeball, and the fourth electrode on the left of the left eyeball; a line connecting the first and third electrode passes through the right eyeball; a line connecting the second and fourth electrodes passes through the left eyeball; the distance between the fifth and first electrodes and the distance between the fifth and second electrodes are equal to each other; the distance between the fifth and third electrodes and the distance between the fifth and fourth electrodes are equal to each other. The detection circuit detects the levels of signals from the first to fourth electrodes with the level of a signal from the fifth electrode as a reference level, detects a first eye potential based on the difference between the signals from the first and second electrodes, detects a second eye potential based on the difference between the signals from the first and third electrodes, detects a third eye potential based on the difference between the signals from the second and fourth electrodes, and detects the left and right rotation of the right eyeball and the left and right rotation of the right eyeball based on the waveforms of the first to third eye potentials.SELECTED DRAWING: Figure 4

Description

Embodiments of the present invention relate to electronic devices and display methods.

As one method for detecting the rotation of the eyeball, there is an eyeball potential (hereinafter, also referred to as an ocular potential) method (Electro-Oculografy: also referred to as an EOG method). By attaching electrodes to the skin near the left and right eyeballs, the electrooculograms of the left and right eyeballs can be detected. The rotation of the eyeball can be detected based on the change pattern of the electrooculogram of the left and right eyeballs.

Japanese Unexamined Patent Publication No. 2011-125693 U.S. Pat. No. 8,449,116 Japanese Unexamined Patent Publication No. 2011-125692 U.S. Pat. No. 8,434,868 Japanese Unexamined Patent Publication No. 2013-240469 Japanese Unexamined Patent Publication No. 2000-259336 U.S. Patent Application Publication No. 2011 / 0,178,784 Japanese Unexamined Patent Publication No. 2013-244370 Japanese Unexamined Patent Publication No. 2013-215356

The conventional eyeball rotation detection device cannot detect the left-right rotation of the right eyeball and the left-right rotation of the left eyeball individually. Therefore, it is not possible to distinguish whether the right eyeball and the left eyeball rotate left and right in the same direction or in opposite directions.

An object of the present invention is to provide an electronic device and a display method for controlling display based on the detection results of left-right rotation of the right eyeball and left-right rotation of the left eyeball.

According to the embodiment, the electronic device includes a display unit, a detector, and a display controller.

The display unit superimposes an augmented reality image of both eyes consisting of a right image and a left image on the image in the real world. The detector detects the angle at which the line-of-sight direction of the user's right eyeball and the line-of-sight direction of the left eyeball intersect. The display controller adjusts the distance between the right image and the left image of the augmented reality image displayed on the display unit according to the angle detected by the detector.

It is a figure which looked at an example of the glasses-type eyeball rotation detection device which concerns on embodiment from the front. It is a figure which looked at an example of the eyeglass type eye rotation detection device from the back upper side. It is the figure which looked at the user who put on an example of the eyeglass type eye rotation detection device from the right front. It is a block diagram which shows an example of the electric structure of the eyeball rotation detection device. It is a figure which shows the 1st modification of the arrangement of a neutral electrode 46. It is a figure which shows the 2nd modification of the arrangement of a neutral electrode 46. It is a figure which shows the 3rd modification of the arrangement of a neutral electrode 46. It is a figure which shows the EOG signal in the state that the line of sight is facing the front. It is a figure which shows an example of the change of the waveform of the EOG signal when both eyeballs are rotated to the left from the state where the line of sight is facing the front. It is a figure which shows an example of the change of the waveform of the EOG signal when both eyeballs are rotated to the right from the state where the line of sight is facing the front. It is a figure which shows an example of the change of the waveform of the EOG signal when both eyeballs are rotated in the direction of increasing the convergence angle from the state where the line of sight is facing the front. It is a figure which shows an example of the change of the waveform of the EOG signal when the binocular eyeball is rotated in the direction which the convergence angle becomes small from the state where the line of sight is facing the front. It is a figure which shows an example of the waveform of the EOG signal for various eye movements. It is a figure which shows an example of the experimental result which detects the change of the convergence angle. It is a figure which looked at an example of the glasses-type eyeball rotation detection apparatus which concerns on 2nd Embodiment from the front view. It is a block diagram which shows an example of the electrical structure of the operation support system including the eyeglass type eye rotation detection device. It is a figure which shows an example of the operation of a glasses-type eyeball rotation detection device. It is a figure which shows another example of the operation of the eyeglass type eye rotation detection device. It is a figure which shows an example of the operation of the eyeglass type eyeball rotation detection apparatus which concerns on the modification of 2nd Embodiment. It is a figure which shows an example of the electric structure of the system including the eyeglass type eyeball rotation detection device which concerns on 3rd Embodiment.

Hereinafter, embodiments will be described with reference to the drawings. The disclosure is merely an example, and the invention is not limited by the contents described in the following embodiments. Modifications that can be easily conceived by those skilled in the art are naturally included in the scope of disclosure. In order to clarify the explanation, in the drawings, the size, shape, etc. of each part may be changed with respect to the actual embodiment and represented schematically. In a plurality of drawings, the corresponding elements may be given the same reference numbers and detailed description may be omitted.

The basic information of the eyeball will be explained. The diameter of an adult eyeball is about 25 mm. It is about 17 mm after birth and grows as it grows. The interpupillary distance of an adult male is about 65 mm. Therefore, many commercially available stereo cameras are made at intervals of 65 mm. The interpupillary distance of adult females is several millimeters shorter than that of males. The electrooculogram is several tens of mV. The eyeball has a positive potential on the corneal side and a negative potential on the retina side. When this is measured on the surface of the skin, it appears as a potential difference of several hundred μV (called an ocular potential).

The range of rotation of the eyeball (in the case of a general adult) is left direction: 50 ° or less, right direction: 50 ° or less in the left-right direction (also called horizontal direction), and up-down direction (also called vertical direction). Downward direction: 50 ° or less, upward direction: 30 ° or less. The vertical angle range that you can move by yourself is narrow in the upward direction. This is because there is a "Bell's phenomenon" in which the eyeballs rotate upward when the eyes are closed, and the range of movement of the eyeballs in the vertical direction shifts upward when the eyes are closed. The convergence angle (the angle at which the line-of-sight directions of the left and right eyeballs intersect) is 20 ° or less.

[First Embodiment]
An example of the configuration of the eyeball rotation detection device according to the embodiment will be described with reference to FIGS. 1, 2, and 3. There are various forms of the eye rotation detection device, but here, an example of the eye rotation detection device in the form of eyewear is shown. Examples of eyewear include goggles and glasses (sunglasses are equivalent to glasses), but here, a glasses-type eyeball rotation detection device will be described. FIG. 1 is a front view of an example of a glasses-type eye rotation detection device, and FIG. 2 is a rear view of an example of a glasses-type eye rotation detection device. FIG. 3 is a view of a user wearing an example of a glasses-type eye rotation detection device as viewed from the front right.

There are two types of eyeball rotation: vertical rotation and horizontal rotation. Vertical rotation includes blinking, eye closure, winking, etc. Left-right rotation includes slow motion (Slow Eye Movement) in which the left and right eyes rotate unconsciously in the same direction, eye movement in which the left and right eyes consciously rotate in the same direction, and movement in which the left and right eyes rotate in opposite directions. It is roughly divided into congestion and divergence. Convergence is the intersection of the line-of-sight directions of the left and right eyes, and divergence is the divergence of the line-of-sight directions of the left and right eyes. Eye rotation is detected based on changes in the electrooculogram. The electrooculogram can be detected by the difference in voltage from a pair of electrodes sandwiching the eyeball. The direction of sandwiching the eyeball may be left-right, up-down, front-back, or diagonally. Blinks, eye closures, winks, etc. can be detected from the electro-oculography detected by a pair of electrodes arranged so as to sandwich the eyeball from above and below. From the electro-oculography detected by the electrode pairs arranged so as to sandwich the eyeball from above and below and from the left and right, blinking, eye closure, wink, slow movement, and eye movement can be detected. Slow movement, eye movement, and congestion / divergence can be detected from the electro-oculography detected by the electrode pairs arranged so as to sandwich the eyeball from the front and back and from the left and right.

[Electrode arrangement]
The glasses include a right frame 12, a left frame 14, and a bridge 26 connecting both frames 12, 14. In this specification, the right and left are the right and left as seen by the user wearing the glasses. It is reversed in FIG. 1 when viewed from the front, and the frame on the right side of FIG. 1 is the left frame 14. If the device is only for electro-oculography detection, it is not necessary to fit a lens or glass in the right frame 12 and the left frame 14, but when the user regularly uses glasses, the power suitable for the user is replaced with the regularly used glasses. The lens may be fitted into the right frame 12 and the left frame 14. If the user does not use glasses regularly, simple glass may be fitted into the right frame 12 and the left frame 14. When configuring a product that detects eye movement or change in convergence angle from the electro-oculography and applies the detection result, for example, a glasses-type wearable device capable of AR display, the right frame 12; A liquid crystal panel for AR display or an organic EL panel may be fitted in at least a part of the left frame 14.

In the embodiment, in order to detect congestion / divergence, in the same plane, the front-back positions are in phase (same vector) with respect to the left and right eyeballs, and the opposite phase (reverse) with respect to the left and right eyeballs. Electrodes are arranged so as to sandwich the eyeball from the left and right positions that form a vector).

In order to detect the electrooculogram of the right eyeball ER, as shown in FIG. 2, a right temple electrode 32 is provided on the right side of the right eyeball ER, for example, a portion of the right temple 18 that hangs on the ear, and is provided on the left side of the right eyeball ER, for example. The right nose pad electrode 42 is provided on the surface of the right nose pad 22 attached to the vicinity of the connection point between the right frame 12 and the bridge 16 in contact with the nose. In the plan view (FIG. 2 is regarded as a plan view), the right temple electrode 32 and the right nose pad electrode 42 are arranged so that the line connecting the right temple electrode 32 and the right nose pad electrode 42 passes through the right eyeball ER. ing.

In the front view (FIG. 1 is regarded as the front view), the right temple electrode 32 is provided on the left side of the right eyeball ER, and the right nose pad electrode 42 is provided on the right side of the right eyeball ER. The right temple electrode 32 and the right nose pad electrode 42 are arranged so that the line connecting the right temple electrode 32 and the right nose pad electrode 42 passes through the right eyeball ER. Further, in the front view, the right nose pad electrode 42 is provided slightly above the right temple electrode 32.

In the side view, the right temple electrode 32 is provided behind the right eyeball ER, that is, on the left side of the right eyeball ER in the right side view, on the right side of the right eyeball ER in the left side view, and the right nose pad electrode 42 is on the right eyeball ER. It is provided on the front side, that is, on the right side of the right eyeball ER in the right side view, and on the left side of the right eyeball ER in the left side view. The right temple electrode 32 and the right nose pad electrode 42 are arranged so that the line connecting the right temple electrode 32 and the right nose pad electrode 42 passes through the right eyeball ER.

FIG. 3 shows how the line connecting the right temple electrode 32 and the right nose pad electrode 42 passes through the right eyeball ER in the front view, the plan view, and the side view of the head. The line connecting the two electrodes is not limited to the center of the right eyeball ER, but may pass through any part of the eyeball. Although it is hidden in the face in FIG. 3, the same applies to the left eyeball.

The right temple electrode 32 and the right nose pad electrode 42 that detect the electrooculogram of the right eyeball ER have a line connecting the right temple electrode 32 and the right nose pad electrode 42 in any of the plan view, the front view, and the side view. It is arranged so as to pass through the right eyeball ER, but in at least one of the plan view, the front view, and the side view, the line connecting the right temple electrode 32 and the right nose pad electrode 42 passes through the right eyeball ER. It suffices if it is arranged in.

Similarly, in order to detect the electrooculogram of the left eyeball EL, the left nose is on the right side of the left eyeball EL in the plan view, for example, on the surface of the left nose pad 24 attached near the connection point between the left frame 14 and the bridge 16 in contact with the nose. The pad electrode 44 is provided, and the left temple electrode 36 is provided on the left side of the left eyeball EL, for example, a portion of the left temple 20 that hangs on the ear. The left nose pad electrode 44 and the left temple electrode 36 are arranged so that the line connecting the left nose pad electrode 44 and the left temple electrode 36 passes through the left eyeball EL.

The right temple electrode 32 and the left temple electrode 36 are axisymmetric with respect to a straight line orthogonal to the midpoint of the straight line connecting the right frame 12 and the left frame 14 (for example, a straight line extending from the center of the nose to the back of the head).

In the front view, the left nose pad electrode 44 is provided on the left side of the left eyeball EL, and the left temple electrode 36 is provided on the right side of the left eyeball EL. The left nose pad electrode 44 and the left temple electrode 36 are arranged so that the line connecting the left nose pad electrode 44 and the left temple electrode 36 passes through the left eyeball EL. Further, in the front view, the left nose pad electrode 44 is provided slightly above the second left electrode 42.

In the side view, the left nose pad electrode 44 is provided on the front side of the left eyeball EL, that is, on the right side of the left eyeball EL in the right side view, on the left side of the left eyeball EL in the left side view, and the left temple electrode 36 is behind the left eyeball EL. It is provided on the side, that is, on the left side of the left eyeball EL in the right side view, and on the right side of the left eyeball EL in the left side view. The left nose pad electrode 44 and the left temple electrode 36 are arranged so that the line connecting the left nose pad electrode 44 and the left temple electrode 36 passes through the left eyeball EL.

The right temple electrode 32 is provided over the side surface (contacting the temporal region) and the lower surface (contacting the base of the ear) of the right temple 18, and when the glasses are worn on the face, the weight of the temple 18 causes the right temple electrode 32. 32 comes into contact with the non-hairy area at the base of the ear. The left temple electrode 36 is provided over the side surface (contacting the temporal region) and the lower surface (contacting the base of the ear) of the left temple 20, and when the glasses are worn on the face, the weight of the temple 20 causes the left temple electrode 36. 36 comes into contact with the non-hairy area at the base of the ear. As a result, the right temple electrode 32 and the left temple electrode 36 are in close contact with the user's skin, and the electrooculogram can be accurately sensed.

The left nose pad electrode 44 and the left temple electrode 36 that detect the electrooculogram of the left eyeball EL also connect the left nose pad electrode 44 and the left temple electrode 36 in at least one of the plan view, the front view, and the side view. The line may be arranged so as to pass through the left eyeball EL.

A forehead pad 26 in contact with the forehead is provided inside the bridge 16, and a neutral electrode 46 is provided on the surface of the forehead pad 26 in contact with the forehead. The neutral electrode 46 is an electrode for securing a neutral potential for detecting the electrooculogram, and is in contact with the skin, for example, the forehead. In the neutral electrode 46, the distance between the neutral electrode 46 and the right temple electrode 32 is equal to the distance between the neutral electrode 46 and the left temple electrode 36, and the distance between the neutral electrode 46 and the right nose pad electrode 42 is neutral. The electrodes 46 and the left nose pad electrode 44 are arranged so that the distances are equal to each other. The reason why the neutral electrode 46 is arranged at such a position is for detecting the convergence angle described later. This is because the convergence angle is detected based on the result of detecting the symmetrical eye rotations when viewed from the front with respect to the left and right eyeballs. For example, in an electrocardiograph, a neutral potential is taken at a part of the body where the influence of eye rotation can be ignored, for example, at the end of the right foot. Although it is slightly affected by eye rotation, by taking a neutral potential at the center of the forehead, where it is evenly affected by the left and right eyes, the neutral electrodes are evenly affected by the left and right eye potentials. can do.

The right temple electrode 32, the right nose pad electrode 42, the left nose pad electrode 44, the left temple electrode 36, and the neutral electrode 46 are metal foils such as copper, metal pieces, metal balls such as stainless steel, or conductive silicon rubber sheets. Consists of. Since these electrodes 32, 42, 44, and 36 are electrodes for detecting the ocular potential as described later, they are also referred to as EOG electrodes.

[EOG signal]
As shown in FIG. 2, a processing unit 30 for detecting an electro-oculography is built-in or externally attached to one of the temples, for example, a portion of the right temple 18 near the frame 12. A battery 34 for the processing unit 30 is built-in or externally attached to the other temple, for example, a portion of the left temple 20 near the frame 12. The processing unit 30 may perform not only the electro-oculography detection but also the display control in the glasses-type wearable device capable of AR display. The processing unit 30 may be provided outside the glasses instead of being built in the glasses, and the glasses and the processing unit 30 may be connected wirelessly or by wire. In that case, the battery 34 can be built in the processing unit 30 and provided outside the glasses. Further, the function of the processing unit 30 is divided into two, and only the first processing unit that senses the signal from the electrode is provided on the glasses, and the second processing that detects the electrooculogram from the sense signal and controls according to the detection result. The portion may be provided outside the glasses. A mobile terminal such as a smartphone can be used as the second processing unit. The second processing unit is not limited to a mobile terminal or the like directly connected to the glasses, but also includes a server or the like connected via a network.

The signal from the right temple electrode 32 is input to the-terminal of the first analog / digital (A / D) converter 62, and the signal from the second left electrode 36 is input to the + terminal of the first A / D converter 62. Then, the first EOG signal ADC Ch0, which is a difference signal, is output. Since the right temple electrode 32 and the left temple electrode 36 sandwich the eyeball from the left and right, the first EOG signal ADC Ch0 indicates the left and right rotation of the left and right eyeballs.

The signal from the right temple electrode 32 is input to the-terminal of the second A / D converter 64, the signal from the right nose pad electrode 42 is input to the + terminal of the second A / D converter 64, and the difference signal is used. A second EOG signal ADC Ch1 is output. Since the right temple electrode 32 and the right nose pad electrode 42 sandwich the right eyeball from the top and bottom and left and right, the second EOG signal ADC Ch1 indicates the left and right rotation and the vertical rotation of the right eyeball.

The signal from the left nose pad electrode 44 is input to the + terminal of the third A / D converter 66, the signal from the left temple electrode 36 is input to the-terminal of the second A / D converter 66, and the difference signal is used. A third EOG signal ADC Ch2 is output. Since the left nose pad electrode 44 and the left temple electrode 36 sandwich the left eyeball from the top and bottom and left and right, the third EOG signal ADC Ch2 indicates the left and right rotation and the vertical rotation of the left eyeball.

Since the left and right positions of the two electrodes with respect to the second EOG signal ADC Ch1 and the left and right positions of the two electrodes with respect to the third EOG signal ADC Ch2 are opposite (the +/- of the input of the A / D converter is inverted), the second From the waveforms of the EOG signal ADC Ch1 and the third EOG signal ADC Ch2, it can be detected whether the left and right eyeballs are rotating left and right in the same direction or in opposite directions.

Since the voltage signals from the right temple electrode 32, the right nose pad electrode 42, the left nose pad electrode 44, and the left temple electrode 36 are weak, the influence of noise is large. In order to cancel this noise, a series circuit of resistors R1 and R2 is connected between the reference analog voltage Vcc (= 3.3V or 5.5V) of the A / D converters 62, 64, 66 and the ground (GND). Then, the neutral electrode 46 is connected to the connection point between the resistors R1 and R2. The resistors R1 and R2 have equal values, for example 1 MΩ. The A / D converters 62, 64, and 66 can detect an analog voltage from 0 V (ground) to a reference analog voltage Vcc, and are at the midpoint of the detectable range, for example, a voltage halved from 3.3 V (midpoint voltage). The input analog voltage is converted into a digital value in the range of 0V to 3.3V centering on). Since the connection point of the resistors R1 and R2 is connected to the midpoint voltage end and the neutral electrode 46 is connected to the connection point of the resistors R1 and R2, the midpoint voltage of the A / D converters 62, 64, 66 is the human body. It becomes the same as the voltage. As a result, the midpoint voltage of the A / D converters 62, 64, 66 fluctuates in conjunction with the voltage of the human body, and the noise mixed in the voltage signals from the EOG electrodes 32, 42, 44, 36 is mixed in the A / D converter. It does not mix with the digital values that are the outputs of 62, 64, and 66. Thereby, the S / N of the detection of the electro-oculography can be improved.

FIG. 4 is a block diagram showing an example of the electrical configuration of the eyeball rotation detection device. The processing unit 30 may include A / D converters 62, 64, 66, and the A / D converters 62, 64, 66 may be externally attached to the processing unit 30.

The signal from the right temple electrode 32 is input to the-terminal of the first A / D converter 62, the signal from the left temple electrode 36 is input to the + terminal of the first A / D converter 62, and the first channel The EOG signal ADC Ch0 is obtained. The signal from the right temple electrode 32 is input to the-terminal of the second A / D converter 64, the signal from the right nose pad electrode 42 is input to the + terminal of the second A / D converter 64, and the second channel EOG signal ADC Ch1 is obtained. The signal from the left nose pad electrode 44 is input to the + terminal of the third A / D converter 66, the signal from the left temple electrode 36 is input to the-terminal of the second A / D converter 66, and the third channel EOG signal ADC Ch2 is obtained.

The signal from the neutral electrode 46 is supplied to the midpoint voltage end of the A / D converters 62, 64, 66, and the midpoint voltage of the A / D converters 62, 64, 66 is detected by the neutral electrode 46 of the human body. It is considered to be a voltage.

The EOG signal output from the A / D converters 62, 64, 66 is input to the eye movement detection unit 75 that detects eyeball rotation (hereinafter, may be referred to as eye movement). The eye movement detection unit 75 may be composed of hardware or software. In the latter case, the CPU 74, ROM 76, and RAM 78 are connected to the bus line, and the eye movement detection unit 75 is also connected to the bus line. The eye movement detection unit 75 is realized by the CPU 74 executing a program stored in the ROM 76. A wireless LAN device 80 is also connected to the bus line, and the processing unit 30 is connected to a mobile terminal 84 such as a smartphone via the wireless LAN device 80. The mobile terminal 84 may be connected to the server 88 via a network 86 such as the Internet. The eye movement detection unit 75 detects the ocular potential based on the EOG signal output from the A / D converters 62 and 64, and from the detected ocular potential, the left and right eyeballs rotate left and right (convergence / divergence) and the eyeball left and right. (Gaze movement) and vertical rotation (blinking, eye closure) can be detected. Further, the eye movement detection unit 75 is subjected to various states of the user (for example, a state of lack of concentration and restlessness or concentration, a state of being tense and mentally stressed, and a state of being tired) from the detected eye movement. It is possible to estimate (a state in which it is difficult to concentrate on work or work). Which type of eye rotation is detected and which type of state is estimated can be changed by changing the program executed by the CPU 74. This change instruction may be given from the mobile terminal 84.

Instead of the wireless LAN device 80, a communication device using a communication method such as ZigBee (registered trademark), Bluetooth Low Energy (registered trademark), Wi-Fi (registered trademark) may be used. The detection result (eye movement detection result, state estimation result) of the eye movement detection unit 75 may be temporarily stored in the RAM 78 and then sent to the mobile terminal 84 via a communication device such as a wireless LAN device 80. .. Alternatively, the detection result of the eye movement detection unit 75 may be sent to the mobile terminal 84 in real time. The mobile terminal 84 may store the detection result of the eye movement detection unit 75 in a built-in memory (not shown), or may transfer the detection result to the server 88 via the network 86. The mobile terminal 84 may start some processing according to the detection result of the eye movement detection unit 75, may store the processing result in the built-in memory, or store the processing result in the server 88 via the network 86. You may transfer to. The server 88 may perform so-called big data analysis by aggregating the detection results from many eye movement detection units 75 and the processing results of many mobile terminals 84.

[Modification of electrode arrangement]
5, FIG. 6 and FIG. 7 show modified examples of the arrangement of the neutral electrode 46. In the above description, the left and right nose pads 22 and 24 are provided separately, but in the modified example of FIG. 5, the left and right integrated inverted V-shaped or inverted U-shaped nose pads 52 are provided. A right nose pad electrode 42 is provided on the right inner side of both sides of the nose pad 52, a left nose pad electrode 44 is provided on the left inner side, and a neutral electrode 46 is provided inside a V-shaped or U-shaped apex portion. Be done. As a result, the neutral electrode 46 in contact with the forehead can be provided without providing the forehead pad 26.

Also in the modified example of FIG. 6, the left and right integrated V-shaped or U-shaped nose pad 54 is provided. The nose pad 52 extends downward, except that the nose pad 54 extends toward the front side. A right nose pad electrode 42 is provided on the right side of the nose pad 54, a left nose pad electrode 44 is provided on the left side, and a neutral electrode 46 is provided in the center.

In the modified examples of FIGS. 5 and 6, since a nose pad having a wider area than the normal nose pad is used, a glasses-type electro-oculography detector heavier than normal glasses or a glasses-type wearable terminal glass capable of AR display can be used for a long time. Even if you use it, the weight does not hurt your nose.

In the modified example of FIG. 7, the left and right nose pads 22 and 24 are used separately, but the forehead pad 26 is unnecessary. Here, the right nose pad electrode 42 and the right neutral electrode 46a are provided on the surface of the right nose pad 22 in contact with the nose, and the left nose pad electrode 44 and the left neutral electrode 46b are provided on the surface of the left nose pad 24 in contact with the nose. It is provided. The right neutral electrode 46a and the left neutral electrode 46b are electrically short-circuited to be equivalent to one neutral electrode 46.

[Relationship between line-of-sight movement and EOG signal]
With reference to FIGS. 8 to 12, the EOG signal ADC Ch0, A / D converter output from the A / D converter 62 when the right eyeball and the left eyeball are rotated left and right from the state where the line-of-sight direction is facing the front direction. An example of the change in the waveform of the EOG signal ADC Ch1 output from 64 and the EOG signal ADC Ch2 output from the A / D converter 66 is shown.

FIG. 8 shows a state in which the user's line-of-sight direction is the front direction. When looking at infinity, the line-of-sight direction of the right eyeball and the line-of-sight direction of the left eyeball are parallel, but when looking at a finite point, the line-of-sight direction of the right eyeball and the line-of-sight direction of the left eyeball are far Cross at a point. From this state, as shown in FIG. 9, when both the right eyeball ER and the left eyeball EL rotate counterclockwise (the line-of-sight direction of the right eyeball and the line-of-sight direction of the left eyeball move to the left), the positively charged cornea of the right eyeball ER Approaches the right nose pad electrode 42, and the negatively charged retina approaches the right temple electrode 32. Similarly, the positively charged cornea of the left eyeball EL approaches the left temple electrode 36, and the negatively charged retina approaches the left nose pad electrode 44. When both the left and right eyeballs are rotated clockwise in this state, the state returns to the state shown in FIG. Therefore, the first EOG signal ADC Ch0 output from the first A / D converter 62 to which the right and left temple electrodes 32 and 36 are connected becomes a signal having a convex waveform (upwardly convex waveform). The second EOG signal ADC Ch1 output from the second A / D converter 64 to which the right nose pad electrode 42 and the right temple electrode 32 are connected becomes a signal having a convex waveform (upwardly convex waveform). The third EOG signal ADC Ch2 output from the third A / D converter 66 to which the left nose pad electrode 44 and the left temple electrode 36 are connected becomes a concave waveform (downwardly convex waveform) signal.

Since the left and right eyeballs rotate in the same direction (counterclockwise rotation) in this way, the second EOG signal ADC
Opposite phase EOG signals appear in Ch1 and the third EOG signal ADC Ch2. An EOG signal having the same phase as that of the second EOG signal ADC Ch1 having the same +/- relationship appears in the first EOG signal ADC Ch0.

When both the right eyeball ER and the left eyeball EL rotate to the right (the line of sight direction of the right eyeball and the line of sight direction of the left eyeball move to the right) as shown in FIG. 10 from the state where the line-of-sight direction shown in FIG. 8 is the front direction. The positively charged cornea of the right eyeball ER approaches the right temple electrode 32, and the negatively charged retina approaches the right nose pad electrode 42. Similarly, the positively charged cornea of the left eyeball EL approaches the left nose pad electrode 44, and the negatively charged retina approaches the left temple electrode 36. When both the left and right eyeballs are rotated counterclockwise in this state, the state returns to the state shown in FIG. Therefore, the first EOG signal ADC Ch0 output from the first A / D converter 62 to which the right and left temple electrodes 32 and 36 are connected becomes a concave waveform (downwardly convex waveform) signal. The second EOG signal ADC Ch1 output from the second A / D converter 64 to which the right nose pad electrode 42 and the right temple electrode 32 are connected becomes a concave waveform (downwardly convex waveform) signal. The third EOG signal ADC Ch2 output from the third A / D converter 66 to which the left nose pad electrode 44 and the left temple electrode 36 are connected becomes a signal having a convex waveform (upwardly convex waveform).

Since the left and right eyeballs rotate in the same direction (rotate to the right) in this way, the second EOG signal ADC
Opposite phase EOG signals appear in Ch1 and the third EOG signal ADC Ch2. However, both the right eyeball ER and the left eyeball EL are out of phase with respect to the case where they rotate to the left. An EOG signal having the same phase as that of the second EOG signal ADC Ch1 having the same +/- relationship appears in the first EOG signal ADC Ch0. However, the first EOG signal ADC Ch0 when both the right eyeball ER and the left eyeball EL rotate clockwise is opposite to the first EOG signal ADC Ch0 when both the right eyeball ER and the left eyeball EL rotate counterclockwise. It is a phase.

From the state where the line-of-sight direction shown in FIG. 8 is the front direction, the right eyeball ER rotates counterclockwise (the line-of-sight direction of the right eyeball moves to the left) and the left eyeball EL rotates to the right (the line-of-sight direction of the left eyeball) as shown in FIG. Moves to the right), causing congestion at which the line-of-sight directions of the left and right eyes intersect, so-called cross-eyed eyes, where the positively charged cornea of the right eyeball ER approaches the right nose pad electrode 42 and becomes negatively charged. The retina is approaching the right temple electrode 32. Similarly, the positively charged cornea of the left eyeball EL approaches the left nose pad electrode 44, and the negatively charged retina approaches the left temple electrode 36. When the right eyeball rotates to the right and the left eyeball rotates to the right in this state, the state returns to the state shown in FIG. Therefore, the first EOG signal ADC Ch0 output from the first A / D converter 62 to which the right and left temple electrodes 32 and 36 are connected does not change, and neither a convex waveform nor a concave waveform appears. The second EOG signal ADC Ch1 output from the second A / D converter 64 to which the right nose pad electrode 42 and the right temple electrode 32 are connected becomes a signal having a convex waveform (upwardly convex waveform). The third EOG signal ADC Ch2 output from the third A / D converter 66 to which the left nose pad electrode 44 and the left temple electrode 36 are connected becomes a signal having a convex waveform (upwardly convex waveform).

Since the left and right eyeballs rotate in opposite directions in this way, in-phase waveforms appear in the second EOG signal ADC Ch1 and the third EOG signal ADC Ch2.

When the ocular potentials of the left and right eyeballs are the same and the absolute value of the rotation angle is also the same, both the + terminal and the-terminal of the A / D converter 62 change by the same amount in the same direction (minus direction). No change is seen in the relative value, and no change in the ocular potential appears in the first EOG signal ADC Ch0.
However, in reality, the plane connecting the nose pad electrode and the temple electrode is slightly displaced from the center of the left and right eyeballs, so that a slight change appears according to the amount of these displacements.

From the state where the line-of-sight direction shown in FIG. 8 is the front direction, the right eyeball ER rotates to the right (the line-of-sight direction of the right eyeball moves to the right) and the left eyeball EL rotates to the left (the line-of-sight direction of the left eyeball) as shown in FIG. Moves to the left), that is, when the left and right eyeballs diverge in the line-of-sight direction, that is, at the so-called distant eye, the positively charged corneum of the right eyeball ER approaches the right temple electrode 32 and is negatively charged. The retinal is approaching the right nose pad electrode 42. Similarly, the positively charged cornea of the left eyeball EL approaches the left temple electrode 36, and the negatively charged retina approaches the left nose pad electrode 44. When the right eyeball rotates counterclockwise and the left eyeball rotates counterclockwise in this state, the state returns to the state shown in FIG. Therefore, the first EOG signal ADC Ch0 output from the first A / D converter 62 to which the right and left temple electrodes 32 and 36 are connected does not change, and neither a convex waveform nor a concave waveform appears. The second EOG signal ADC Ch1 output from the second A / D converter 64 to which the right nose pad electrode 42 and the right temple electrode 32 are connected becomes a concave waveform (downwardly convex waveform) signal. The third EOG signal ADC Ch2 output from the third A / D converter 66 to which the left nose pad electrode 44 and the left temple electrode 36 are connected becomes a concave waveform (downwardly convex waveform) signal. Since the left and right eyeballs rotate in opposite directions in this way, the second EOG signal ADC
In-phase EOG signals appear in Ch1 and the third EOG signal ADC Ch2. However, the second EOG signal ADC Ch1 and the third EOG signal ADC Ch2 in the case of cross-eyed eyes are in opposite phases with respect to the second EOG signal ADC Ch1 and the third EOG signal ADC Ch2 in the case of cross-eyed eyes, respectively. is there.

When the ocular potentials of the left and right eyeballs are the same and the absolute value of the angle of rotation is also the same, the + terminal and the-terminal of the A / D converter 62 change by the same amount in the same direction (plus direction). No change is seen in the relative value, and no change in the ocular potential appears in the first EOG signal ADC Ch0. However, in reality, the plane connecting the nose pad electrode and the temple electrode is slightly displaced from the center of the left and right eyeballs, so that a slight change appears according to the amount of these displacements.

FIG. 13 is an electrocardiogram (EOG) illustrating an example of the relationship between various eye movements of the user and the EOG signals ADC Ch0, ADC Ch1, and ADC Ch2 obtained from the A / D converters 62, 64, and 66. The vertical axis shows sample values of A / D converters 62, 64, 66 (for example, 3.3 V, 24-bit A / D converter), and the horizontal axis shows time.

As shown in FIG. 11, neither a convex waveform nor a concave waveform appears in the EOG signal ADC Ch0, and a convex waveform appears in the EOG signals ADC Ch1 and ADC Ch2, so that the eye movement detection unit 75 is congested in the line-of-sight direction of the left and right eyeballs. Detects a "cross-eyed" condition that causes Although not shown in FIG. 13, as shown in FIG. 12, neither a convex waveform nor a concave waveform appears in the EOG signal ADC Ch0, and a concave waveform appears in the EOG signals ADC Ch1 and ADC Ch2, so that the eye movement detection unit 75 Detects a "distant eye" condition in which the left and right eyeballs diverge in the line-of-sight direction. As described above, congestion and divergence are the same except that the waveform irregularities of the EOG signals ADC Ch1 and ADC Ch2 are different. Therefore, in the following description, congestion and divergence may be collectively referred to as congestion. The amplitude of the waveforms of the EOG signals ADC Ch1 and ADC Ch2 depends on the degree of convergence (convergence angle) and the degree of divergence. As will be described later with reference to FIG. 14, the degree of change in amplitude is larger as the distance is shorter and becomes smaller as the distance is longer. Therefore, the sensitivity of detecting the amplitude change increases as the distance increases, and decreases as the distance increases.

As shown in FIG. 9, a convex waveform appears in the EOG signal ADC Ch0, a convex waveform appears in the EOG signal ADC Ch1, and a concave waveform appears in the EOG signal ADC Ch2, so that the eye movement detection unit 75 moves to the left in the line-of-sight direction. Is detected.

As shown in FIG. 10, a concave waveform appears in the EOG signal ADC Ch0, a concave waveform appears in the EOG signal ADC Ch1, and a convex waveform appears in the EOG signal ADC Ch2, so that the eye movement detection unit 75 moves to the right in the line-of-sight direction. Is detected.

In the EOG signal ADC Ch1 and the EOG signal ADC Ch2, the level rises momentarily and returns to the original state. Due to the convex pulse waveforms of 1 to 3 waves of the same phase, the first blink 1, the second blink 2, 3 The second blink 3 is detected. In the EOG signal ADC Ch1 and the EOG signal ADC Ch2, a combination waveform of a convex waveform (upward convex waveform) when the line of sight is directed upward and a concave waveform (downward convex waveform) when the line of sight is directed downward is used. , The eye movement detection unit 75 detects vertical eye rotation, that is, eye closure. As described above, the rotation of the eyeball generated by the vertical blink and the eye closure can be detected based on either one of the EOG signals ADC Ch1 or ADC Ch2, and in the application of detecting only the blink and the eye closure. It is not necessary to provide electrode pairs for each of the left and right eyeballs, and electrode pairs may be provided for only one of the eyeballs.

An example of the experimental result for detecting the change in the congestion state according to the embodiment is shown. Using the prototype of the electro-oculography detector shown in FIGS. 1 to 4, in the line-of-sight direction of the left and right eyeballs so that the subject looks at the marker farther from the state where the subject is looking at the fingertip 10 cm ahead of the nose. FIG. 14 shows an example of the change of the second EOG signal ADC Ch1 when the intersection position is changed. In this case, since the intersection in the line-of-sight direction changes from near to far, the convergence angle decreases. The horizontal axis is the moving distance of the intersection in the line-of-sight direction from 10 cm ahead to the depth direction, and the first plot shows the EOG amplitude when the intersection in the line-of-sight direction is moved from 10 cm ahead to 20 cm ahead. Note that 10 cm is the shortest distance at which a person can stare stably. The minimum detection voltage of the EOG signal is 50 μV. That is, when the EOG signal changes by 50 μV or more, the eye motion detector 75 can detect the change in the EOG signal, and based on this, can detect the change in the intersection position in the line-of-sight direction, that is, the change in the convergence angle, and the EOG signal changes by 50 μV or more. Otherwise, the change in the EOG signal cannot be detected. If the average value obtained by integrating the measured values is used, the minimum detection voltage of the EOG signal becomes a smaller voltage, but here it is set to 50 μV. The amplitude of the EOG signal depends on the contact resistance of the electrode, and if an electrode material having a small contact resistance is used, the amplitude of the EOG signal becomes large and the minimum detection voltage of the EOG signal becomes large.

For example, when the convergence angle is reduced so that the subject looks at the marker 30 cm in front of the nose and looks at the marker farther away, when the EOG amplitude increases by 50 μV, the eye movement detection unit 75 moves the EOG amplitude. Detect changes in. The EOG amplitude as a result of adding 50 μV to the EOG amplitude when looking at the marker 30 cm ahead corresponds to 40 cm ahead. That is, when the convergence angle is reduced from the state where the subject is looking 30 cm ahead to the state where the subject is looking 40 cm ahead, the eye movement detection unit 75 can detect the change in the EOG amplitude. It can be said that the resolution of detection is quite good that the change in EOG amplitude corresponding to the decrease in the convergence angle corresponding to the change of 10 cm can be detected. From FIG. 14, when the convergence angle decreases from the state where the subject looks 50 cm ahead to the state where the subject looks 65 cm or farther away, the change in EOG amplitude can be detected, and the subject looks 70 cm ahead. It can be seen that a change in EOG amplitude can be detected when the convergence angle is reduced so as to look at 1.4 m ahead or farther from the state of being.

As described above, according to the first embodiment, the right and left temple electrodes, the right and left nose pad electrodes, and the neutral electrodes arranged so as to be evenly affected by the rotation of the right and left eyeballs. Provided is a spectacle-type eyeball rotation position detection device that detects congestion by independently detecting the left and right rotations of the right and left eyeballs. Since the neutral potential from the neutral electrode is the midpoint potential of the A / D converter that samples the EOG signal from the electrode, the EOG signal from the electrode is not affected by noise and the ocular potential is detected accurately. Therefore, accurate eye rotation is detected. According to the first embodiment, further, left-right rotation of both eyeballs (left-right movement in the line-of-sight direction) and up-down rotation of the eyeball can be detected.

According to the electro-oculography detection device of the first embodiment, it is possible to detect congestion, which is the conscious eye rotation of the subject, and therefore, by performing control according to the detection result, the subject's intention is achieved. It is possible to realize an application example in which control is performed according to the situation. For example, in eyewear capable of displaying Augmented Reality (hereinafter referred to as AR), it is possible to control the ON / OFF of the AR display and the adjustment of the display position of the AR image hands-free according to the detection of congestion. Can be done.

In addition, some tasks may require work-specific congestion changes, and by comparing the user's congestion change pattern with the reference pattern, the user's proficiency level in the work and whether or not the work is being performed correctly. It is possible to judge.

An application example of the congestion detection result will be described below. One of the application examples is to incorporate an electro-oculography detection device into a glasses-type wearable device capable of AR display, and control the AR display based on the detection of congestion. For example, congestion detection can be applied as a function changeover switch. For example, if the AR display is turned off when the user is looking at a distance in front of the user as shown in FIG. 8, and the line-of-sight direction of both eyes is changed so as to look at a short distance (congested state) as shown in FIG. It is possible to control the on / off of the display so that the AR display is performed. Since the display position of the AR image is set to a short distance, when the user is looking at the AR image, the AR image is in a congested state, so the AR display is continued and the line-of-sight direction of both eyes is changed so that the user looks at a distance. Then, the AR display is turned off. As a result, AR display control as intended by the user becomes possible. Hereinafter, as a second embodiment, a glasses-type wearable device for surgical support, which is an application example, will be described.

[Second Embodiment]
FIG. 15 is a front view of an example of the glasses-type electro-oculography detection device 100 according to the second embodiment. The difference from the detection device of the first embodiment is that AR display displays (for example, an organic EL panel or a liquid crystal panel) 102 and 104 are fitted in at least a part of the right frame 12 and the left frame 14. .. When the AR image is not displayed in 3D, the display 102 or 104 may not be fitted in one of the right frame 12 and the left frame 14. Here, it is assumed that the AR image is displayed in 3D. The arrangement of the EOG electrodes is the same as that of the first embodiment. There are various application examples of the glasses-type electro-oculography detection device 100 capable of displaying AR, but as a second embodiment, a surgical support system will be described as an example. The doctor who operates the operation and the staff related to the operation wear the glasses-type electro-oculography detector 100.

FIG. 16 is a block diagram showing an example of an electrical configuration of a surgical support system including a glasses-type electro-oculography detection device 100. A display controller 112 is added to the processing unit 30 in addition to the configuration of the first embodiment shown in FIG. The display controller 112 controls the AR display of the displays 102 and 104. The AR display control includes ON / OFF control of the AR display, display position control of the AR image (convergence angle control of the AR image), and the like.

In the surgery support system, since a plurality of doctors and staff are involved in the same surgery and a plurality of glasses are used, a personal computer or the like having higher performance than a mobile terminal 82 such as a smartphone is connected to the processing unit 30 of the glasses. The control device 120 of the above is preferable. Although not shown, a plurality of eyeglass processing units 30 are connected to the control device 120. The control device 120 includes a vital data memory 124, a support image memory 122, and a support information memory 126. The eye movement detection unit 75 may also be provided in the control device 120.

A vital data measuring unit 127 is connected to the control device 120, and the electrocardiogram, blood pressure, pulse, cumulative blood transfusion amount, etc. of the patient are measured and stored in the vital data memory 124 in the control device 120 for each patient. The support image memory 122 stores support images for supporting surgery. If necessary, the support image in the surgery support database 130 in the server 88 is downloaded to the control device 120 and stored in the support image memory 122. The support information memory 126 stores support texts for supporting surgery. If necessary, the support text in the surgery support database 130 in the server 88 is downloaded to the control device 120 and stored in the support information memory 126.

An example of the operation of the glasses-type electro-oculography detection device 100 will be described with reference to FIGS. 17 and 18. Here, it is assumed that the distance from the eyeball to the affected area (real world) during surgery is about 40 cm. When the doctor is looking at a point in front of the affected area (real world) during surgery, for example, at a distance of about 30 cm from the eyeball, the eye movement detection unit 75 detects congestion. As shown in FIG. 11, the eye movement detection unit 75 is congested based on the fact that the waveform of the EOG signal ADC Ch0 does not change and that the waveforms of the EOG signal ADC Ch1 and the EOG signal ADC Ch2 are convex upward. Is detected. When the eye movement detection unit 75 detects congestion, it requests the control device 120 for an AR image related to surgical support, and causes the display controller 112 to perform AR display. An example of AR display is to superimpose a vital data window (AR image) translucently on the affected area (real world) during surgery as shown in FIG. 17 (a). The window contains multiple pages, and each page displays an electrocardiogram, blood pressure, pulse rate, cumulative blood transfusion volume, etc. The window is confined to a portion of the screen so that the affected area is not hidden. In response to the request from the processing unit 30, the control device 120 reads the patient's vital data from the vital data memory 124 and transfers it to the processing unit 30.

When the AR image is displayed in 3D, the display position can be arbitrarily set by adjusting the convergence angles of the left and right images. Here, the display position of the vital data window is set to a position about 30 cm forward, which is equal to the distance at which the eye movement detection unit 75 detects congestion. Therefore, when the vital data window is displayed, the doctor is looking at a point at the same distance as the display position, so that the contents of the window can be checked instantly and the eyeball is rotated to gaze at the window. There is no need to adjust the convergence angle, and eye strain does not occur.

The page switching of the vital data window may be automatically performed for a certain period of time, for example, every second, or may be performed by the user's intention based on the rotation of the eyeball in the other direction by the eye movement detector 75. For example, if 0.5 seconds or more of eye closure is detected, the pages of the window may be switched. Further, the pages of the window may be switched based on the moving direction of the line of sight. For example, when the line of sight moves to the right, the page may be switched to the next page, and when the line of sight moves to the left, the page may be switched to the previous page. FIG. 17B shows an example in which the pages of the window are switched.

In the state of FIG. 17A or FIG. 17B, when a doctor or the like changes the line-of-sight direction so as to look at the affected area (real world) during surgery (farther than about 40 cm), the eye movement detection unit 75 is congested. It detects that the angle becomes smaller (the distance to the intersection in the line-of-sight direction of both eyes becomes longer), and stops the AR display on the display controller 112. Therefore, doctors and the like observe only the affected area during surgery as shown in FIG. 17C through the right frame 12 and the left frame 14 of the glasses.

When the doctor or the like wants to refer to the vital data in the state of FIG. 17 (c), the doctor or the like changes the line-of-sight direction so as to look at the hand (about 30 cm). The eye movement detection unit 75 detects that the convergence angle becomes large (the distance to the intersection in the line-of-sight direction of both eyes becomes short), causes the display controller 112 to perform AR display, and shows FIG. 17 (a) or The vital data window as shown in (b) is displayed.

The judgment of the increase / decrease of the convergence angle is based on the relationship between the EOG amplitude and the moving distance of the line-of-sight intersection as shown in FIG. 14, and when the EOG voltage decreases / increases by a certain voltage or more, the convergence angle increases / decreases. Can be judged.

In this way, the doctor can display or turn off the AR image that supports the surgery hands-free during the surgery. Since the hands are closed during the operation, it is very effective to switch the AR display on / off hands-free.

Another example of the AR display may be a support image as shown in FIG. 18 (a). The support image may be an image of a past operation example for another patient with similar symptoms, or an image during another operation for the patient. The image may be a still image or a moving image. Therefore, an image during surgery is taken by a camera (not shown), the captured image is uploaded to the server 88, and is stored in the surgery support database 130. Further, another example of AR display may be support information as shown in FIG. 18 (b). Support information is various textual data regarding the surgery being performed.

The display position of the support information is set close to the vital data window. However, the display position of the support image may be set at a distance of about 40 cm, which is the same as the affected area (real world) during surgery. Vital data windows and support information are not seen at the same time as the affected area during surgery, but support images are expected to be compared side-by-side with the affected area. If the display position is different when comparing the affected area and the support image, it is necessary to rotate the eyeball left and right when comparing, which may cause eye strain. Therefore, unlike the above description, the support image is displayed when the eyeball is in the state shown in FIG. 8, and is turned off when congestion as shown in FIG. 11 is detected. When the position of the support image and the affected part (real world) during the operation are slightly deviated, the convergence angle detected by the eye movement detection unit 75 slightly changes when comparing. If the display controller 112 adjusts the display position of the support image (convergence angle of the left and right images) so that this change becomes small, the possibility of eye strain can be further reduced.

The switching from the vital data window to the support image and the support information may be performed by the user's intention based on the rotation of the eyeball in the other direction by the eye movement detector 75, similarly to the page switching of the vital data window.

In the above example, the change of the convergence angle was used to switch the AR display on / off, but the detected convergence angle is used for another control, and the AR display on / off switching is based on another eye movement. You may. For example, when a plurality of eye closures are detected, the AR display may be switched on / off. Then, the detected convergence angle may be used to control the display position of the AR image. That is, the distance between the left and right images of the AR image (sometimes referred to as the convergence angle of the image) is adjusted based on the distance to the intersection of the left and right eyeballs in the line-of-sight direction at the start of displaying the AR image. The convergence angle of the image is adjusted by the display controller 112. As a result, it is not necessary to rotate the eyeball left and right in order to gaze at the AR display, so that eye strain does not occur when viewing the AR display.

Another application example of the change of the convergence angle and the control of the AR display will be described with reference to FIG. This example is an example of picking in a warehouse, and an operator wears a glasses-type electro-oculography detection device 100 during picking work.

Since the configuration of the processing unit 30 may be the same as that in FIG. 16, the illustration is omitted. The configuration of the control device 120 is the same except that the vital data, the support image, and the support information are changed to the pickup list, and thus the illustration is omitted. Instead of the control device 120, a mobile terminal 84 such as a smartphone may be used as in the first embodiment shown in FIG. The mobile terminal 84 may acquire the position information. The configuration of the server 88 is the same except that the map information regarding the position of the shelves in the warehouse, the list of products on each shelf in the warehouse, and the list of products to be picked up are changed instead of the surgical support data. Therefore, the illustration is omitted. The pick-up list for each worker is downloaded from the server 88 to the control device 120.

Also in this application example, the AR display is initially turned off, and the displays of the right frame 12 and the left frame 14 of the glasses do not display anything and are transparent. Therefore, the operator watches the inside of the warehouse as shown in FIG. 19A through the right frame 12 and the left frame 14 of the glasses. When the mobile terminal 84 as the control device 120 can acquire the position information, the air tag may be AR-displayed on the products on each shelf based on the map information in the server 88 and the acquired position information. At this time, the worker is looking ahead by several meters or several tens of meters or more.

The left and right eyeballs are moved from the state where the operator is looking at a distant target to the state where the worker is looking at a near target as shown in FIG. 11, in opposite directions so that congestion occurs, that is, so-called cross-eyed eyes. Rotates left and right, and the eye movement detection unit 75 detects congestion.

In response to the detection of congestion, the processing unit 30 requests the control device 120 for a pickup list indicating the products to be picked up by the worker, and causes the display controller 112 to start the AR display. An example of the pickup list is shown in FIG. 19 (b). The display position of the pickup list is also set nearby. When the left and right eyeballs are rotated from the state of looking at the pickup list to the state of looking at the distant warehouse shown in FIG. 8, the eye movement detection unit 75 does not detect congestion, so the pickup is picked up with respect to the display controller 112. Stop displaying the list.

As with surgery, the pickup work is a situation where you cannot take your hands off, so the effect of switching the AR display on / off hands-free is great.

Although the air tag is always displayed in FIG. 19, the air tag may be displayed only when the user looks at a distant target, and may be turned off when the user looks at a near target. For example, when a worker goes out for maintenance work on the elevator of a building, and is looking for the target building during the visit, the worker is looking at a distant target, so the name of the building, distance, work purpose, etc. The air tag containing is displayed overlaid on the target building. Here, when the worker looks at his / her hand to check the map or the like, the air tag display may be turned off. When you arrive at your destination and come in front of the elevator, an air tag will be displayed at the inspection point when the operator is looking at the inspection point. When the worker looks at his / her hand, the work manual or the like is displayed in AR instead of the air tag. This example can also be applied to tourist information. For example, when looking at a landscape at a tourist spot, an air tag related to the spot may be displayed, and when looking at a hand, the air tag may be turned off and a detailed tourist guide may be displayed instead.

As another example of AR display by detecting congestion, it is conceivable to incorporate an electro-oculography detection device into AR glasses used for watching sports. The state of the game is shot from various angles and stored on the server as a replay video. The AR glass is connected to the server via the network. Spectators are always watching the game and looking at distant targets. At this time, the AR display is off. When a spectator in the outfield seat of a baseball stadium wants to see a magnified image of the close play on the home base, he or she can rotate the eyeball so as to be cross-eyed and start the AR display. In the AR display, various replay images are downloaded and displayed from the server. As a result, the spectator can instantly enjoy the replay image by rotating the eyeball.

In the above description, the AR display is started when the congestion is detected, and the AR display is stopped when the congestion is no longer detected, but the reverse is also possible. That is, AR display may be performed in a state where congestion is not detected, and AR display may be stopped when congestion is detected.

[Third Embodiment]
Another application of congestion detection is confirmation of work procedures. FIG. 20 is a block diagram showing an example of an electrical configuration of a work confirmation system including a glasses-type electro-oculography detection device. The AR display is not essential for work confirmation, but the AR display may be used to immediately convey the confirmation result to the worker. The processing unit 30 is the same as the second embodiment shown in FIG. Instead of the control device 120, a mobile terminal 84 such as a smartphone may be used as in the first embodiment shown in FIG. The control device 120 includes a line-of-sight movement determination unit 148 and a line-of-sight movement pattern memory 142. The eye movement determination unit 148 discriminates eye movement (eye movement right movement, eye direction left movement, eye direction crossing (congestion)) from the change pattern of eyeball rotation detected by the eye movement detection unit 75, and moves the eye movement pattern. Stored in the pattern memory 142. The data in the line-of-sight movement pattern memory 142 is uploaded to the server 88 via the network 86.

The server 88 includes a line-of-sight movement standard pattern memory 144 and a work procedure determination unit 146. Some of the work requires movement of the line of sight peculiar to the work. For example, in the railway industry, in pointing confirmation after departure, it is required to confirm a distant target and a near target in a predetermined order. Further, even in the visual inspection of structures such as bridges and tunnels, since the points to be visually inspected are determined, it is required to move the line-of-sight direction in a predetermined pattern. In this way, the standard pattern memory 144 stores a standard pattern of movement in the line-of-sight direction peculiar to work. The work procedure determination unit 146 can compare the line-of-sight movement pattern of the worker uploaded from the control device 120 with the line-of-sight movement standard pattern in the standard pattern memory 144, and determine whether or not the worker is working correctly. Each worker may be stored in a database (not shown). The proficiency level of the worker can be estimated from the change in the judgment result over time. Alternatively, when the work procedure determination unit 146 determines that the operator is not working correctly, the display controller 112 of the processing unit 30 is notified via the control device 120, and a caution message is sent to the displays 102 and 104. It may be displayed.

Another example of a standard pattern of eye movement is a pattern related to driving a car. In the training when renewing the license, a video of the safety confirmation by a good driver may be introduced. At this time, the proficiency level of the safety confirmation may be determined by comparing the line-of-sight movement pattern when the same safety confirmation is actually performed with the line-of-sight movement pattern of a good driver, instead of simply looking at the driver. In the transportation industry as well, accidents can be reduced by having a new driver learn the same line-of-sight movement pattern as that of a veteran driver.

The above description is applicable not only to glasses but also to goggles. For example, instead of the nose pad, an electrode may be provided on a part of the foam on the front surface of the goggles, and instead of the temple, an electrode may be provided on the belt.

[Calibration]
In the electrocardiogram, the specific angle of the convergence angle cannot be obtained, but only the rotation direction of the eyeball in the congestion state can be obtained. Therefore, in the above embodiment, congestion is detected, and in the application example, control is performed based on a binary state of whether or not congestion is detected, regardless of the congestion angle itself. However, when there are multiple targets with known distances, by looking at the multiple targets and periodically measuring the electrocardiogram for the multiple distances, the electrooculogram for the multiple known distances can be measured. Values can be determined and three or more congestion states can be identified. For example, the AR display is off when looking at a distant target, the first AR image is displayed when looking at a medium-distance target, and the first when looking at a short-distance target. The AR image of 2 may be displayed.

Since the contact resistance of the electrodes changes (decreases with time), the absolute value of the ocular potential with respect to the distance changes with the passage of time, and the absolute value of the distance cannot be guaranteed in the long term. For example, immediately after the start of use of the electrode, since the contact resistance is large, the amplitude of the electrocardiogram is small, and as time passes, the resistance becomes small and the amplitude of the electrocardiogram becomes large. However, if the electrooculogram with respect to the distance is calibrated regularly, the change with time of the electrooculogram can be estimated, and the absolute value of the electrooculogram with respect to the distance can be guaranteed. For example, if the work has a constant focal length such as surgery or desk work, or if it is a work process that guarantees that the target at a known distance (level) is being viewed at a certain point in time, the distance is known. It is possible to calibrate regularly in the state.

Furthermore, the potential of the human body itself may fluctuate, which can also be compensated for by regular calibration.

[Summary of effects of embodiments]
According to the above-described embodiment, since the left-right rotation of the right eyeball and the left-right rotation of the left eyeball can be detected independently, it is possible to detect the convergence which is the intersection of the left and right eyeballs in the line-of-sight direction. Congestion does not occur unconsciously, but is caused by the subject's conscious eye rotation. Therefore, hands-free control can be performed according to the intention of the subject by performing control according to the detection of congestion. For example, in a glasses-type wearable terminal that performs AR display, it is possible to start displaying an AR image by making a cross-eyed eye and end displaying the AR image by looking at a distant target.

The AR display includes a monocular display and a binocular display. This embodiment can also be applied to monocular display, but in the case of binocular display, detection of congestion in the left and right eyes is effective. In the 3D display of AR images by both eyes, the distance between the left and right images (also called the convergence angle) is arbitrarily set, but the convergence angles of the left and right eyes when looking at the real world and the convergence of the AR image. If the angles are different, the convergence angles of the left and right eyeballs must be adjusted each time when comparing the AR image with the real world, which causes eye fatigue. However, if the convergence angles of the left and right images of the AR image are set to match the convergence angles of the left and right eyeballs, the rotation of the left and right eyeballs for adjusting the convergence angle becomes unnecessary, and eye strain does not occur. ..

The work content can be objectively confirmed by detecting the change in the rotation of the eyeball during the work, obtaining the time-dependent pattern, and comparing this with the standard pattern. For example, when it is necessary to confirm a distant target or a near target in work such as maintenance and inspection, it is possible to confirm whether the work is performed according to the procedure by detecting congestion during the actual work. The confirmation result may be notified to the operator as an alarm. Although it is not a maintenance inspection, the embodiment can also be applied to pointing confirmation after the train departs.

The comparison with the standard pattern can be used not only for confirming the work contents but also for mastering the work procedure with congestion. For example, in the training at the time of license renewal, a video of a good driver confirming safety may be introduced. At this time, it is possible to understand the ideal congestion change in more detail by comparing the line-of-sight movement pattern when the same safety confirmation is actually performed with the line-of-sight movement pattern of a good driver, rather than simply looking at it.

The left-right rotation of the eyeball is not limited to the rotation of the right eyeball and the left eyeball in opposite directions (convergence), but also includes the right eye movement and the left eye movement in which the right eyeball and the left eyeball rotate in the same direction. May be combined with congestion detection to perform hands-free control. In addition, it is possible to detect nystagmus based on the left-right movement of the line of sight and the left-hand movement of the line of sight in the left-right direction that occur with the occurrence of a congestion state in a state of increased drowsiness. This can also alert a drowsy subject. Further, not only the horizontal rotation of the eyeball but also the vertical rotation may be detected, and the hands-free control may be executed by combining the horizontal rotation of the eyeball and the vertical rotation of the eyeball.

The present invention is not limited to the above embodiment as it is, and at the implementation stage, the components can be modified and embodied within a range that does not deviate from the gist thereof. In addition, various inventions can be formed by an appropriate combination of a plurality of components disclosed in the above-described embodiment. For example, some components may be removed from all the components shown in the embodiments. In addition, components from different embodiments may be combined as appropriate.

32 ... right temple electrode, 36 ... left temple electrode, 42 ... right nose pad electrode, 44 ... left nose pad electrode, 46 ... neutral electrode, 62, 64, 66 ... A / D converter, 75 ... eye movement detection unit.

Claims (6)

A display unit that superimposes an augmented reality image of both eyes consisting of a right image and a left image on an image in the real world,
A detector that detects the angle at which the line-of-sight direction of the user's right eyeball and the line-of-sight direction of the left eyeball intersect,
A display controller that adjusts the distance between the right image and the left image of the augmented reality image displayed on the display unit according to the angle detected by the detector.
Electronic device equipped with. The electronic device according to claim 1, wherein the display controller changes the augmented reality image displayed on the display unit according to the angle detected by the detector. The electronic device according to claim 1, wherein the display controller controls the display or non-display of the augmented reality image displayed on the display unit according to the angle detected by the detector. The display unit superimposes the augmented reality image of both eyes consisting of the right image and the left image on the image in the real world.
The detector detects the angle at which the line-of-sight direction of the user's right eyeball and the line-of-sight direction of the left eyeball intersect.
The display controller adjusts the distance between the right image and the left image of the augmented reality image displayed on the display unit according to the angle detected by the detector.
Display method including. The display method according to claim 4, wherein the display controller changes the augmented reality image displayed on the display unit according to the angle detected by the detector. The display method according to claim 4, wherein the display controller controls the display or non-display of the augmented reality image displayed on the display unit according to the angle detected by the detector.

Images