JP2017227941A - Program, information processing device, and eyewear - Google Patents

Abstract

PROBLEM TO BE SOLVED: To prevent blinking and visual line movement from being erroneously detected.SOLUTION: A program causes a computer to execute an acquisition step for acquiring a sensor signal detected by an acceleration sensor and/or an angular velocity sensor, and an electrooculogram signal based on an ocular potential detected by each electrode coming into contact with an eye periphery, a detection step for performing detection processing of blinking or visual line movement on the basis of the electrooculogram signal, a determination step for determining whether the sensor signal is a prescribed pattern, and a stop step for stopping the detection processing when it is determined that the sensor signal is the prescribed pattern.SELECTED DRAWING: Figure 8

Description

  The present invention relates to a program, an information processing apparatus, and eyewear.

  There is known eyewear provided with an electrode for inputting an electrooculogram attached to a frame of spectacles, an acceleration sensor, and an angular velocity sensor (for example, see Patent Document 1).

US Patent Application Publication No. 2004/0070729

  However, when each electrode is brought into contact with the periphery of the eye and eye blink or eye movement is detected using the electrooculogram signal detected by each electrode, noise may be generated depending on the type of action of the user wearing the glasses. There is a problem that erroneous detection of blinking or eye movement occurs.

  Therefore, an object of the present invention is to prevent erroneous detection of blinks and eye movements.

  The program according to one aspect of the present invention is based on a sensor signal indicating a motion of a subject detected by an acceleration sensor and / or an angular velocity sensor, and an electrooculogram detected by each electrode that contacts the periphery of the subject's eyes. An acquisition step of acquiring an electrooculogram signal, a detection step of performing a blink or eye movement detection process based on the electrooculogram signal, and a determination step of determining whether or not the sensor signal is a predetermined pattern When the sensor signal is determined to be the predetermined pattern, the computer is caused to execute a stop step of stopping the detection process.

  According to the present invention, it is possible to prevent erroneous detection of blinks and eye movements.

It is a perspective view which shows an example from the front of the glasses in an Example. It is a perspective view which shows an example from the back of the spectacles in an Example. It is a block diagram which shows an example of the processing apparatus in an Example. It is a figure which shows roughly the contact position of the electrode with respect to a user. It is a figure which shows an example of a structure of the amplification part in an Example. It is a figure for demonstrating the reason for providing a buffer amplifier. It is a figure which shows the other example of a structure of the amplification part in an Example. It is a block diagram which shows an example of a structure of the external device in an Example. It is a figure which shows an example of the electrooculogram signal and sensor signal at the time of a walk. It is a figure which shows an example of the electrooculogram signal which shows the motion of the perpendicular | vertical direction of eyes. It is a flowchart which shows an example of the process regarding the detection process of the blink and eye movement in an Example. It is a flowchart which shows an example of the blink detection process in an Example.

  Embodiments of the present invention will be described below with reference to the drawings. However, the embodiment described below is merely an example, and there is no intention to exclude various modifications and technical applications that are not explicitly described below. That is, the present invention can be implemented with various modifications without departing from the spirit of the present invention. In the following description of the drawings, the same or similar parts are denoted by the same or similar reference numerals. The drawings are schematic and do not necessarily match actual dimensions and ratios. In some cases, the dimensional relationships and ratios may be different between the drawings.

[Example]
FIG. 1 is a perspective view illustrating an example from the front of the glasses 100 according to the embodiment. FIG. 2 is a perspective view illustrating an example from the rear of the glasses 100 in the embodiment. The glasses 100 include a lens 110 and a frame 120. Glasses 100 and frame 120 are examples of eyewear.

  The frame 120 supports the pair of lenses 110. The frame 120 includes a rim 122, an eyebrow portion (for example, a bridge) 124, an armor 126, a hinge 128, a temple 130, a modern 132, a pair of nose pads 140, a first electrode 152, and a second electrode. 154, a third electrode 156, an electric wire (not shown), a processing device 200, and an amplification unit 250. Depending on the type of glasses 100, there may be no bridge portion of the frame by using a single lens. In this case, the portion between the eyebrows of the single lens is defined as the portion between the eyebrows.

  The pair of nose pads 140 includes a right nose pad 142 and a left nose pad 144. The rim 122, the armor 126, the hinge 128, the temple 130, and the modern 132 are provided in a pair on the left and right.

  The rim 122 holds the lens 110. The armor 126 is provided outside the rim 122 and holds the temple 130 rotatably with a hinge 128. The temple 130 presses the upper part of the user's ear to pinch this part. The modern 132 is provided at the tip of the temple 130. The modern 132 contacts the upper part of the user's ear. The modern 132 is not necessarily provided in the glasses 100.

  The first electrode 152 and the second electrode 154 are provided on the respective surfaces of the pair of nose pads 140 and detect the electrooculogram. For example, the first electrode 152 is provided on the right nose pad 142, and the second electrode 154 is provided on the left nose pad 144.

  The first electrode 152 detects the electrooculogram of the user's right eye. The second electrode 154 detects the electrooculogram of the user's left eye. As described above, the electrode for detecting the electrooculogram is provided on the surface of the nose pad that inevitably contacts the skin of the user. Thereby, the burden given to a user's skin can be reduced compared with making a pair of electrodes contact the circumference | surroundings of a user's eye.

  The third electrode 156 is provided on the surface of the eyebrow portion 124 and detects an electrooculogram. A ground electrode (not shown) is provided on the surface of the modern 132. When the glasses 100 do not have the modern 132, the ground electrode is provided at the tip of the temple 130. In the embodiment, the potential detected by the first electrode 152, the second electrode 154, and the third electrode 156 may be based on the potential detected by the ground electrode.

  The processing apparatus 200 may be provided in the temple 130, for example. Thus, the design when the glasses 100 are viewed from the front is not impaired. The installation position of the processing apparatus 200 is not necessarily the temple 130, but may be positioned in consideration of the balance when the glasses 100 are worn. The processing device 200 is connected to the amplifying unit 250 via an electric wire. Note that the processing device 200 and the amplifying unit 250 may be connected via wireless.

  The amplifying unit 250 is provided in the vicinity of the first electrode 152, the second electrode 154, and the third electrode 156, and is connected to each electrode to be amplified via an electric wire. The amplifying unit 250 acquires an electrooculogram signal indicating the electrooculogram detected by each electrode. For example, the amplification unit 250 amplifies an electrooculogram signal indicating an electrooculogram detected by the first electrode 152, the second electrode 154, and the third electrode 156.

  In addition, if the amplification unit 250 includes a processing unit that processes an electrooculogram signal, the amplification unit 250 may perform addition / subtraction processing on each electrooculogram signal before amplification or after amplification. For example, the amplifying unit 250 may obtain a reference electrooculogram signal indicating the potential of the first electrode 152 with respect to the third electrode 156. The amplifying unit 250 may obtain a reference electrooculogram signal indicating the potential of the second electrode 154 with respect to the third electrode 156. The signal amplified or processed by the amplification unit 250 is output to the processing device 200.

  The external device 300 is an information processing device having a communication function. For example, the external device 300 is a mobile communication terminal such as a mobile phone and a smartphone possessed by the user. The external device 300 executes processing based on the electrooculogram signal received from the transmission unit 220. For example, the external device 300 detects blinks and eye movements from the received electrooculogram signal. As a response when detecting blinks, the external device 300 issues a warning for preventing a doze when detecting that the number of blinks of the user has increased. Details of the external device 300 will be described later.

<Configuration of processing device>
FIG. 3 is a block diagram illustrating an example of the processing apparatus 200 in the embodiment. As illustrated in FIG. 3, the processing device 200 includes a processing unit 210, a transmission unit 220, a power supply unit 230, and a motion sensor 240. The first electrode 152, the second electrode 154, and the third electrode 156 are connected to the processing unit 210 via, for example, the amplification unit 250.

  The processing unit 210 acquires and processes the amplified electrooculogram signal from the amplification unit 250. For example, the processing unit 210 may process a reference electrooculogram signal indicating the potential of the first electrode 152 with respect to the third electrode 156. In addition, although the reference | standard electrocardiogram signal attached | subjected "reference | standard" for convenience of explanation, it is contained in the electrooculogram signal. Further, the processing unit 210 may process a reference electrooculogram signal indicating the potential of the second electrode 154 with respect to the third electrode 156.

  At this time, the processing unit 210 performs processing so as to obtain an electrooculogram signal indicating the vertical and / or horizontal movement of the eye based on the electrooculogram detected from each electrode in the right eye and the left eye. You may go.

  In addition, if the acquired electrooculogram signal is not digitized, the processing unit 210 performs the digitization process or acquires an electrooculogram signal amplified from each electrode. Add or subtract signals. The processing unit 210 may transmit the electrooculogram signal acquired from the amplification unit 250 to the transmission unit 220 as it is.

  The transmission unit 220 transmits the electrooculogram signal processed by the processing unit 210 to the external device 300. For example, the transmission unit 220 transmits an electrooculogram signal to the external device 300 by wireless communication such as Bluetooth (registered trademark) and wireless LAN, or wired communication. The power supply unit 230 supplies power to the processing unit 210, the transmission unit 220, the motion sensor 240, and the amplification unit 250.

  The motion sensor 240 is a sensor that detects the movement of the user wearing the glasses 100. The motion sensor 240 is, for example, a 3-axis acceleration sensor and / or a 3-axis angular velocity sensor, and is preferably a 6-axis sensor capable of detecting acceleration and angular velocity. A signal (hereinafter also referred to as a sensor signal) detected by the motion sensor 240 is output to the transmission unit 220 and transmitted to the external device 300 by the transmission unit 220. Note that the motion sensor 240 may be provided separately from the glasses 100 as long as it can communicate with the external device 300.

  FIG. 4 is a diagram schematically showing the contact position of the electrode with respect to the user. The first contact position 452 represents the contact position of the first electrode 152. The second contact position 454 represents the contact position of the second electrode 154. The third contact position 456 represents the contact position of the third electrode 156. A horizontal center line 460 represents a horizontal center line connecting the center of the right eye 402 and the center of the left eye 404. The vertical center line 462 represents a center line orthogonal to the horizontal center line 460 at the center of the right eye 402 and the left eye 404.

  It is desirable that the first contact position 452 and the second contact position 454 are located below the horizontal center line 460. Further, it is desirable that the first contact position 452 and the second contact position 454 are arranged so that the line connecting the centers of the first contact position 452 and the second contact position 454 is parallel to the horizontal center line 460.

  Further, the first contact position 452 and the second contact position 454 are desirably arranged so that the distance from the first contact position 452 to the right eye 402 and the distance between the second contact position 454 and the left eye 404 are equal. . Further, it is desirable that the first contact position 452 and the second contact position 454 are separated from each other by a certain distance or more.

  The third contact position 456 is preferably located on the vertical center line 462. The third contact position 456 is preferably located above the horizontal center line 460 and away from the first contact position 452 and the second contact position 454. In addition, for example, the distance between the third contact position 456 and the right eye 402 is separated from the distance between the right eye 402 and the first contact position 452, and the distance from the left eye 404 is the second contact with the left eye 404. The distance from the position 454 may be greater than the distance.

  The eyeball is positively charged on the corneal side and negatively charged on the retinal side. Therefore, when the line of sight moves upward, the potential of the first electrode 152 with respect to the third electrode 156 and the potential of the second electrode 154 with respect to the third electrode 156 become negative. When the line of sight moves downward, the potential of the first electrode 152 with respect to the third electrode 156 and the potential of the second electrode 154 with respect to the third electrode 156 become positive.

  When the line of sight moves to the right, the potential of the first electrode 152 with respect to the third electrode 156 becomes negative, and the potential of the second electrode 154 with respect to the third electrode 156 becomes positive. When the line of sight moves to the left, the potential of the first electrode 152 with respect to the third electrode 156 becomes positive, and the potential of the second electrode 154 with respect to the third electrode 156 becomes negative.

  By detecting the potential of the first electrode 152 with respect to the third electrode 156 and the potential of the second electrode 154 with respect to the third electrode 156, the influence of noise can be suitably reduced. In order to make the third contact position 456 as far as possible from the first contact position 452 and the second contact position 454, the inter-brow portion 124 may be disposed at or near the upper end of the rim 122. Further, the third electrode 156 may be provided above the center of the eyebrow portion 124. In this case, it is desirable to adopt the eyebrow portion 124 having a wide vertical width as the arrangement position of the third electrode 156.

  Instead of detecting the potential of the first electrode 152 based on the third electrode 156, the processing unit 210 detects the third electrode based on the reference electrode from the potential of the first electrode 152 based on the reference electrode. The potential of 156 may be reduced. Similarly, instead of detecting the potential of the second electrode 154 with respect to the third electrode 156, the processing unit 210 detects the potential of the second electrode 154 with respect to the reference electrode as a reference. The potential of the three electrodes 156 may be reduced.

  A ground electrode may be used as the reference electrode. Further, a reference electrode may be separately provided in the glasses 100 at a position away from the first electrode 152, the second electrode 154, and the third electrode 156. For example, the reference electrode may be provided on the modern 132 on the right side. Further, the reference electrode may be provided at a portion of the right temple 130 that is in contact with the user's skin.

  The process of subtracting the potential of the third electrode 156 from the potential of the first electrode 152 relative to the reference electrode and the process of subtracting the potential of the third electrode 156 from the potential of the second electrode 154 relative to the reference electrode are as follows: The processing unit 210 may execute, or the amplification unit 250 or the external device 300 may execute. In this case, the signal indicating the potential to be processed is amplified by the amplification unit 250.

<Configuration of amplification unit>
Next, the configuration of the amplification unit 250 will be described. FIG. 5 is a diagram illustrating an example of a configuration of the amplification unit 250 in the embodiment. As illustrated in FIG. 5, the amplification unit 250 includes a first amplifier 260 and a second amplifier 270. The first amplifier 260 is an amplifier that is positioned in front of the second amplifier 270 and functions as a buffer amplifier. Hereinafter, the first amplifier 260 is also referred to as a buffer amplifier 260. The second amplifier 270 is an amplifier that functions as a main amplifier. Hereinafter, the second amplifier 270 is also referred to as a main amplifier 270. The signal amplified by the main amplifier 270 is output to the processing device 200 by wire or wireless.

  As for the installation position of the amplification part 250, it is desirable that it is the part 124 between eyebrows. Note that the amplification unit 250 may be provided so as to be embedded in the eyebrow portion 124. As described above, it is desirable that the electrodes be separated as much as possible. However, since the installation positions of the electrodes depend on the shape of the frame 120, there is a limit even if they are separated.

  For this reason, the potential difference between the electrodes may not be sufficiently large, and if noise is mixed in an electrooculogram signal indicating a small potential detected at each electrode, a sufficiently accurate potential can be detected. Will become difficult.

  Therefore, in the embodiment, the amplifying unit 250 is provided in the vicinity of the first electrode 152, the second electrode 154, and the third electrode 156 for the purpose of amplifying the detected electrooculogram signal before noise is mixed therein. Provided. For example, the amplifying unit 250 is preferably provided in a portion between the eyebrows 124 that is close to each electrode and has a relatively large space in the frame 120. Thereby, while the electrooculogram signal detected by each electrode passes an electric wire, the risk that noise mixes and reduces the accuracy of the electrooculogram signal can be reduced.

  Next, the reason why the buffer amplifier 260 is provided at a position preceding the main amplifier 270 will be described with reference to FIG. FIG. 6 is a diagram for explaining the reason why the buffer amplifier 260 is provided. The example shown in FIG. 6 uses the third electrode 156, but the same applies to the first electrode 152 and the second electrode 154.

Since the third electrode 156 touches human skin when wearing the glasses 100, it may be considered that a resistance R ₀ exists between the third electrode 156 and the ground. At this time, the resistance R ₀ is, for example, several hundred kΩ. The main amplifier 270 has an internal resistance R ₁ . At this time, when a normal amplifier is used as the main amplifier 270, the internal resistance R ₁ is several tens kΩ to several hundreds kΩ.

Here, although ideally is that does not flow a current to the main amplifier 270, the internal resistance R ₁ is smaller than the resistance R _0, the current flows to the main amplifier 270 side. Then, the voltage Vi of the electrode and the voltage Vx of the main amplifier 270 are divided and observed. Therefore, a buffer amplifier 260 is provided at a position before the main amplifier 270 so that no current flows into the main amplifier 270 side.

  FIG. 7 is a diagram illustrating another example of the configuration of the amplifying unit in the embodiment. The amplifying unit shown in FIG. 7 is denoted by reference numeral 250A. The amplification unit 250A includes a buffer amplifier 260, a main amplifier 270, an A / D conversion unit 280, and a wireless communication unit 290. Since the buffer amplifier 260 and the main amplifier 270 have the same functions as those shown in FIG. 5, the A / D conversion unit 280 and the wireless communication unit 290 will be mainly described below.

  The A / D converter 280 converts the signal amplified by the main amplifier 270 from analog to digital. The A / D conversion unit 280 outputs the digitally converted signal to the wireless communication unit 290.

  The wireless communication unit 290 transmits the digital signal converted by the A / D conversion unit 280 to the processing device 200 using wireless communication. Therefore, the wireless communication unit 290 functions as a transmission unit. The wireless communication unit 290 uses wireless communication such as Bluetooth (registered trademark) and wireless LAN. The wireless communication unit 290 may directly transmit a digital signal to the external device 300.

  In the embodiment, an example is shown in which one buffer amplifier 260 and one main amplifier 270 are provided, but in this case, the order of the electrooculogram signals from the respective electrodes may be determined and amplified. Further, a buffer amplifier 260 and a main amplifier 270 may be provided for each electrode.

<Configuration of external device>
Next, the configuration of the external device 300 will be described. FIG. 8 is a block diagram illustrating an example of the configuration of the external device 300 in the embodiment. As illustrated in FIG. 8, the external device 300 includes a communication unit 310, a storage unit 320, and a control unit 330.

  The communication unit 310 receives an electrooculogram signal and a sensor signal by wireless communication such as Bluetooth (registered trademark) and wireless LAN, or wired communication. The communication unit 310 outputs an electrooculogram signal or a sensor signal received from the communication unit 220 of the processing device 200 to the control unit 330.

  The control unit 330 is a CPU (Central Processing Unit), for example, and controls each unit and performs various arithmetic processes. In the example illustrated in FIG. 8, the control unit 330 includes an acquisition unit 340, a determination unit 342, and a detection unit 346.

  The acquisition unit 340 receives the sensor signal indicating the movement of the subject detected by the acceleration sensor and / or the angular velocity sensor, and the electrooculogram signal based on the electrooculogram detected by each electrode in contact with the periphery of the subject's eyes. get. For example, when acquiring the electrooculogram signal or sensor signal received by the communication unit 310, the acquisition unit 340 outputs the electrooculogram signal to the detection unit 346 and outputs the sensor signal to the determination unit 342. The movement of the subject may be a direct movement of the subject or an indirect movement by riding on a vehicle.

  The control unit 330 stores the maximum value and the minimum value of the electrooculogram signal indicating the movement of the eyes in the vertical direction and the horizontal direction in the storage unit 320 with respect to the acquired electrooculogram signal. Further, the control unit 330 may store the maximum value and / or the minimum value of the electrooculogram signal indicating the eye movement in the vertical direction and the horizontal direction in the storage unit 320 for each predetermined period. The predetermined period is, for example, 200 msec, but is not limited thereto. Further, the predetermined period may be temporally varied by allowing overlap by using a time window. Further, the control unit 330 may store the acquired sensor signal in the storage unit 320. Here, the vertical direction refers to the direction from the top to the bottom of the head with respect to the face, and the horizontal direction refers to a direction orthogonal to the vertical direction.

  The storage unit 320 is, for example, a RAM (Random Access Memory), and has a local maximum value (or a maximum value for each predetermined period) and / or a minimum value (or a minimum value for each predetermined period) of the electrooculogram signals in the vertical direction and the horizontal direction. Value). For example, the storage unit 320 includes a maximum value FIFO buffer and a minimum value FIFO buffer. When the storage capacity of the FIFO buffer is full due to the maximum or minimum value data, the oldest data is erased and the latest data is stored, whereby the data stored in the storage area is updated. . The storage unit 320 may store sensor signals.

  In addition, the storage unit 320 stores a program that causes the computer to execute blink detection processing and eye movement detection processing described later. This program may be installed in the external device 300 via the Internet or a recording medium such as an SD card, or may be preinstalled. In addition, the storage unit that stores the program may be different from the storage unit 320.

  Based on the sensor signal acquired from the acquisition unit 340, the determination unit 342 determines whether or not the operation of the user (user) wearing the glasses 100 is a predetermined operation. The predetermined operation is an operation that makes it difficult to detect blinks and eye movements such as walking, running, coughing, sneezing, sneezing, and shaking the head. The determination unit 342 includes a recognition unit 344 in order to perform operation determination based on the sensor signal.

  The recognition unit 344 recognizes a predetermined operation by performing pattern recognition. For example, the recognizing unit 344 learns each pattern of sensor signals such as walking, running, coughing, sneezing, sneezing, and shaking the head, and stores them in the storage unit 320. The recognizing unit 344 determines whether the acquired sensor signal matches one or more learned patterns. The recognition unit 344 recognizes the operation from the matched pattern. A known technique can be applied to the pattern matching technique in the recognition unit 344.

  When the recognition unit 344 recognizes that the sensor signal is a predetermined pattern, the determination unit 342 determines that the user's action based on the sensor signal is a predetermined operation, and the recognition unit 344 does not identify the predetermined pattern. In this case, it is determined that the user's action based on the sensor signal is not a predetermined action. The determination unit 342 outputs a control signal based on the determination result to the detection unit 346. The control signal is, for example, an execution signal when the sensor signal is not a predetermined operation, and a stop signal when the sensor signal is a predetermined operation.

  In addition to the pattern recognition, the determination unit 342 may determine whether the sensor signal simply exceeds the threshold value. In this case, the determination unit 342 determines that the predetermined operation is performed when the threshold value is exceeded. This threshold is a value used for detecting an abnormal sensor signal, and is set to a value larger than the amplitude of a normal sensor signal.

  The detection unit 346 performs or stops blink detection processing based on the electrooculogram signal acquired from the acquisition unit 340 according to the determination result of the determination unit 342. The detection unit 346 stops the detection process when it is determined that the sensor signal has a predetermined pattern, that is, the user's operation is a predetermined operation. The detection unit 346 performs detection processing when it is determined that the sensor signal is not a predetermined pattern, that is, when it is determined that the user's motion is not the predetermined motion. Specifically, the detection unit 346 performs a detection process when an execution signal that is a control signal is acquired from the determination unit 342, and stops the detection process when a stop signal that is a control signal is acquired from the determination unit 342. .

  Thereby, when the user is performing a predetermined operation with a high possibility that a lot of noise is mixed, erroneous detection can be prevented by stopping the blink or eye movement detection processing. Moreover, the stop of the detection process also contributes to power saving.

  After stopping the detection process, the detection unit 346 may continue the stop of the detection process until a predetermined time elapses after it is determined that the pattern is not a predetermined pattern. That is, once the detection process is stopped, the detection unit 346 stops the detection process for a predetermined time even if the detection process is enabled.

  Thereby, when the sensor signal is not a predetermined pattern (negative) from the sensor signal having a predetermined pattern (positive), an electrooculogram signal that is unstable due to a predetermined operation is used. It is not necessary to perform detection processing immediately. Moreover, the detection process can be stopped until the electrooculogram signal is stabilized. As a result, it is possible to prevent erroneous detection of blinks and eye movements. The predetermined time may be changed according to a predetermined pattern or a predetermined operation. The predetermined time may be set to an appropriate time until the electrooculogram signal is stabilized according to each pattern and operation by a prior experiment or the like.

  Next, stop of the detection process will be described by taking an electrooculogram signal and a sensor signal when the user is walking as a specific example. FIG. 9 is a diagram illustrating an example of an electrooculogram signal and a sensor signal during walking.

  FIG. 9A shows an electrooculogram signal S1 indicating the eye movement in the vertical direction during walking. A period W1 illustrated in FIG. 9 indicates a period during which the user is walking. As shown in FIG. 9A, the electrooculogram signal S1 during walking has a large amplitude and is not in a state in which blinking or eye movement can be detected. If the process of detecting blinks and eye movements is performed during this period W1, many blinks and eye movements are detected because the electrooculogram signal S1 exceeds the determination threshold for blinks and eye movements. Will occur. The electroocular intensity on the vertical axis is a unit indicating the intensity of the electrooculogram, and is represented by, for example, measured value of electrooculogram signal × 1.5 (V) ÷ 2048.

  FIG. 9B shows a sensor signal S2 indicating the vertical acceleration during walking. In the period W1, when the user walks, the sensor signal S2 indicating the vertical acceleration greatly fluctuates. Therefore, the learning unit 342 can recognize the user's walking by learning the movement pattern of the sensor signal S2. Note that the value on the vertical axis of the graph shown in FIG. 9B is a value representing acceleration.

  In the example shown in FIG. 9, the case of walking has been described as an example, but the operation of running, coughing, sneezing, squeezing, shaking the head, etc. is performed by the determination unit 342 in the same manner as in FIG. It is only necessary to recognize the signal pattern. Note that the determination unit 342 may improve the recognition accuracy by using not only the vertical acceleration but also the angular velocity signal for the sensor signal.

≪Gaze movement detection≫
Next, detection of eye movement will be described. The detection unit 346 may detect line-of-sight movement using a known method. For example, the detection unit 346 divides the acquired electrooculogram signal into a right electrocardiogram and a left electrocardiogram, and when a negative potential is indicated in the right electrocardiogram and the left electrocardiogram, the line of sight is displayed. Detect that is facing up. Further, the detection unit 346 has a line of sight when a positive potential is shown in the right electrogram and the left electrogram, a negative potential is shown in the right electrogram, and a positive potential is shown in the left electrogram. Is detected, it is detected that the line of sight is directed to the left when the line of sight is right, the right electrogram shows a positive potential and the left electrogram shows a negative potential.

  Furthermore, the detection unit 346 can increase the detection accuracy of the line of sight by adding and subtracting the potential V1 indicated by the right electrogram and the potential V2 indicated by the left electrogram. For example, when V1 + V2 is negative and V1-V2 is substantially zero, it can be detected that the line of sight is directed upward. When V1 + V2 is positive and V1-V2 is substantially zero, it can be determined that the line of sight is directed downward. When V1 + V2 is substantially zero and V1-V2 is negative, it can be determined that the line of sight is directed to the right. When V1 + V2 is substantially zero and V1-V2 is positive, it can be determined that the line of sight is directed to the left. By adding and subtracting V1 and V2, the calculated positive value and negative value are increased. Therefore, since the threshold value can be set larger accordingly, it is possible to reduce erroneous detection that erroneously detects noise as eye movement.

≪Blink detection≫
Next, blink detection will be described. The detection unit 346 can use a known blink detection algorithm using an electrooculogram signal. In this embodiment, an algorithm for changing a threshold value used for blink determination will be described. The detection unit 346 includes a threshold value calculation unit 350.

  The threshold calculation unit 350 calculates a threshold using the maximum value and / or the minimum value stored in the storage unit 320. For example, the threshold value calculation unit 350 may calculate the absolute value of the threshold value from the average value of the absolute value of the local maximum value or the local minimum value in order to simplify the processing.

  Further, the threshold value calculation unit 350 may calculate the first threshold value using the local maximum value stored in the storage unit 320, and may calculate the second threshold value using the local minimum value stored in the storage unit 320. . Here, the first threshold value is used to determine that the eye has moved in the vertical direction, and the second threshold value is used to determine that the eye has moved in the vertical direction. Accordingly, threshold values can be set for each of the upward eye movement and the downward eye movement, so that appropriate threshold determination can be performed.

  The threshold calculation unit 350 includes a first calculation unit 352 and a second calculation unit 354. The first calculator 352 calculates the average value and standard deviation of the maximum values stored in the maximum value FIFO buffer. The first calculation unit 352 calculates the average value and standard deviation of the minimum values stored in the minimum value FIFO buffer.

  The second calculation unit 354 calculates a first threshold based on the average value and standard deviation of the maximum values, and calculates a second threshold value based on the average value and standard deviation of the minimum values. Thereby, for example, the threshold value can be set using an electrooculogram signal indicating the latest state of the subject. Further, the first threshold value and the second threshold value can be changed in accordance with the intensity of a signal indicating the most recent state of the user.

  For example, the second calculation unit 354 sets a value obtained by adding a value obtained by multiplying the standard value of the maximum value by a coefficient to the average value of the maximum value as the first threshold value. The second calculation unit 354 sets a value obtained by subtracting a value obtained by multiplying the standard deviation of the minimum value by a coefficient from the average value of the minimum value as the second threshold value. Thereby, an appropriate threshold value can be set.

  Further, the threshold value calculation unit 350 updates the first threshold value and the second threshold value each time the maximum value and / or the minimum value are stored in the storage unit 320. As a result, the threshold calculation unit 350 can set the threshold based on the past electrooculogram signal, so that even if the electrooculogram signal becomes weak due to sleepiness and slowing of eye movement. Since the threshold can be set according to the weakened signal, the blink can be detected appropriately.

  Further, the threshold calculation unit 350 sets the average of the predetermined maximum values stored in the storage unit 320 as the first threshold, or sets the average of the predetermined minimum values stored in the storage unit 320 as the second threshold. May be. Further, the threshold value calculation unit 350 may calculate the first threshold value and the second threshold value using the standard deviations of the local maximum value and the local minimum value stored in the storage unit 320.

  The detection unit 346 detects blinks from an electrooculogram signal indicating the vertical movement of the eye using the first threshold value and the second threshold value calculated by the second calculation unit 354. For example, the detection unit 346 determines the difference between the first time of the maximum value stored in the storage unit 320 that is equal to or greater than the first threshold and the second time of the minimum value stored in the storage unit 320 that is equal to or less than the second threshold. If is within a predetermined time, blink is detected. Here, the second time is the latest time after the first time. The predetermined time is, for example, 500 msec, but is not limited thereto.

  Note that the detection unit 346 detects blinks using the electrocardiogram signals of the right eye and the left eye, and detects the final blink when the blinks are detected in both eyes at a timing within a predetermined range. May be. The detection unit 346 may detect blinks using the average of both electrooculogram signals on the assumption that the right eye and the left eye move in the same manner. Next, a blink detection algorithm will be described using specific electrooculogram signals.

  FIG. 10 is a diagram illustrating an example of an electrooculogram signal indicating the vertical movement of the eye. The electrooculogram signal S3 shown in FIG. 10 is an electrooculogram signal indicating the vertical movement of one eye. Here, the blink detection algorithm will be described with reference to FIG. The blink detection algorithm described below is an algorithm for calculating a threshold value using the maximum value and the minimum value of a predetermined period.

(1) Obtaining maximum and minimum values for each predetermined period The control unit 330 obtains maximum and minimum values for each predetermined period T1 (for example, 500 msec) of the electrooculogram signal S3.

(2) When the maximum value is a maximum value, it is stored in the first FIFO buffer. When the maximum value obtained in (1) is a maximum value, the control unit 330 stores it in the first FIFO buffer (storage unit 320).

(3) When the minimum value is a minimum value, the controller 330 stores the value in the second FIFO buffer. When the minimum value obtained in (1) is the minimum value, the control unit 330 stores the minimum value in a second FIFO buffer in a different area from the first FIFO buffer. Save (storage unit 320). The order of (2) and (3) is not limited. Hereinafter, when the first FIFO buffer and the second FIFO buffer are combined, they are simply referred to as buffers. In FIG. 10, a black dot on the electrooculogram signal S3 represents a value detected as a maximum value or a minimum value in each period. Note that the maximum value and the minimum value can be obtained using differentiation, for example, using a difference signal of the electrooculogram signal S3.

(4) When the buffer capacity data is stored, the average value and the standard deviation are calculated. The first calculation unit 352 calculates the average value (a1) and the standard deviation (b1) of the maximum values stored in the first FIFO buffer. Is calculated. The first calculation unit 352 calculates the average value (a2) and standard deviation (b2) of the maximum values stored in the second FIFO buffer.

(5) Calculating the threshold The second calculation unit 354 calculates the first threshold and the second threshold using the average value and the standard deviation calculated in (4). The first threshold value (Th1) and the second threshold value (Th2) are calculated by the following equations.
Th1 = a1 + E × b1 Formula (1)
Th2 = a2-E × b2 Formula (2)
Here, the coefficient E is set to 2, for example.
In addition, you may set a lower limit to the absolute value of a 1st threshold value (Th1) and a 2nd threshold value (Th2). As a result, it is possible to prevent erroneous detection of a slight vertical movement of the eye as blinking. Further, the coefficient E may be variably set according to the signal strength.

  In FIG. 10, the example of the first threshold is represented by the symbol TH1, and the example of the second threshold is represented by the symbol TH2. As shown in FIG. 10, each threshold value varies with the passage of time of the electrooculogram signal, and further, the threshold value is appropriately changed following the intensity (magnitude) of the electrooculogram signal. . The first threshold value TH1 and the second threshold value TH2 shown in FIG. 8 are merely conceptually showing threshold value fluctuations.

(6) Identify the maximum value greater than or equal to the first threshold and the minimum value less than or equal to the second threshold The detection unit 342 identifies the maximum value greater than or equal to the first threshold calculated in (5). Here, the maximum value to be subjected to the threshold determination is a maximum value that is stored in the first FIFO buffer and has not yet been subjected to the threshold determination.

  In addition, the detection unit 342 specifies a minimum value equal to or less than the second threshold calculated in (5). Here, the minimum value to be subjected to the threshold determination is a minimum value that is stored in the second FIFO buffer and has not yet been subjected to the threshold determination. In FIG. 10, the specified maximum and minimum values are represented by black dots surrounded by a square.

(7) Detecting blinks The detection unit 342 includes, for each specified maximum value, the first time of the maximum value and the second time of the specified minimum value, and the latest time after the first time If the difference from the second time is within a predetermined time, the vertical movement of the eye is detected as a blink.

  In FIG. 10, for example, if the difference between the specified maximum value time t11 and the minimum value time t21 specified most recently after t11 is within a predetermined time, the eye movement is detected as a blink. . The predetermined time is, for example, 500 msec.

  The blink detection algorithm has been described above, but this algorithm is merely an example, and the present invention is not limited to this example. In the blink and eye movement detection processing, the same processing as described above may be performed using the difference signal in the time direction of the electrooculogram signal. The difference signal is, for example, a signal obtained by subtracting an electrooculogram signal a predetermined time before time t from an electrooculogram signal at time t. Thereby, detection accuracy can be improved by using a difference signal with strong noise tolerance.

<Operation>
Next, the operation of the external device 300 in the embodiment will be described. FIG. 11 is a flowchart illustrating an example of processing related to detection processing for blinking and eye movement in the embodiment. The flowchart shown in FIG. 11 shows a state in which the user wears the glasses 100 and the first electrode 152, the second electrode 154, the third electrode 156, and the ground electrode are in contact with the user's skin. Is started when the operation mode is set to detect blink or eye movement.

  In step S <b> 102 illustrated in FIG. 11, the acquisition unit 340 acquires an electrooculogram signal and a sensor signal from the glasses 100.

  In step S104, the determination unit 342 determines whether the sensor signal is a predetermined pattern. For example, the determination unit 342 performs determination processing using pattern recognition by the recognition unit 344. If it is determined that the sensor signal is a predetermined pattern (YES in step S104), the determination unit 342 outputs a stop signal to the detection unit 346 in order to stop the blink or eye movement detection process. Return to S102. If it is determined that the sensor signal is not a predetermined pattern (step S104—NO), the determination unit 342 outputs an execution signal to the detection unit 346, and the process proceeds to step S106.

  In step S <b> 106, when the detection unit 346 acquires an execution signal from the determination unit 342, the detection unit 346 performs blink processing or eye movement detection processing. That is, the detection unit 346 performs detection processing unless a stop signal is acquired from the determination unit 342.

  Note that the timing when the detection process is once stopped and restarted is when the detection unit 346 acquires the execution signal. Note that the detection unit 346 may continue to stop the detection process for a predetermined time even if the execution signal is acquired or the stop signal is not acquired after the detection process is stopped.

  Through the above processing, it is possible to determine the user's motion that is likely to cause erroneous detection of blinking or eye movement. When this operation is performed, the detection processing is stopped and erroneous detection is performed. Can not be.

  Next, FIG. 12 is a flowchart illustrating an example of blink detection processing in the embodiment. In the blink detection process described below, a determination threshold is calculated using a moving average of the maximum value and the minimum value.

  In step S <b> 202, the control unit 330 stores the calculated maximum value and / or minimum value in the storage unit 320.

  In step S <b> 204, the threshold value calculation unit 350 calculates a threshold value (first threshold value and second threshold value) using the maximum value and / or the minimum value stored in the storage unit 320. At this time, the threshold value calculation unit 350 sets, for example, the average value of the ten latest local maximum values as the first threshold value, or the average value of the ten latest local minimum values as the second threshold value. In addition, you may exclude from the average calculation process after the next time the maximum value and minimum value which are determined to be a blink afterward.

  In step S206, the detection unit 346 specifies the first time of the maximum value that is equal to or greater than the first threshold value, or the second time of the minimum value that is equal to or less than the second threshold value. At this time, the detection part 360 specifies the time of the nearest minimum value after the specified 1st time as 2nd time.

  In step S208, the detection unit 346 determines whether the second time-first time is less than a predetermined time. If this condition is satisfied (step S208—YES), the process proceeds to step S210. If this condition is not satisfied (step S208—NO), the process returns to step S202.

  In step S210, the detection unit 346 detects an eye movement between the first time and the second time as a blink.

  Through the above processing, the external apparatus 300 can detect blinks using an appropriate threshold value.

  As described above, according to the embodiment, the sensor signal from the motion sensor 240 is used to determine whether the user is performing an operation in which the electrooculogram signal is greatly shaken, and when the user is performing this operation, By stopping the eye and eye movement detection processing, erroneous detection can be prevented.

  In the present embodiment, the case where the eyewear is glasses has been described. However, eyewear is not limited to this. The eyewear may be any device related to the eye, and may be a face wearing device or a head wearing device such as glasses, sunglasses, goggles and a head mounted display and their frames.

  In this embodiment, the example in which the glasses 100 include the third electrode 156 has been described. However, the glasses 100 are not limited to this. The glasses 100 may not include the third electrode 156. In this case, an electrooculogram indicated by the potential of the first electrode 152 relative to the reference electrode and an electrooculogram indicated by the potential of the second electrode 154 relative to the reference electrode may be transmitted to the external device 300. Here, a ground electrode may be provided at the position of the third electrode 156 to serve as a reference electrode. In addition, a ground electrode provided in the left modern may be used as a reference electrode, or an electrode provided separately from the first electrode 152 and the second electrode 154 may be used as a reference electrode.

  In the present embodiment, the example in which the glasses 100 include the nose pad 140 integrated with the rim 122 has been described. However, the glasses 100 are not limited to this. The glasses 100 may include a klings provided on the rim 122 and a nose pad 140 attached to the krings. In this case, the electrode provided on the surface of the nose pad 140 is electrically connected to the electric wire embedded in the frame via the krings.

  In the present embodiment, the first electrode 152 and the second electrode 154 have been described as examples provided below the center of the nose pad 140. However, it is not limited to this. The nose pad 140 may include an extending portion that extends downward, and the first electrode 152 and the second electrode 154 may be provided in the extending portion. This allows the first electrode 152 and the second electrode 154 to be in contact below the eye position even for a user whose nose pad is located directly beside the eye due to individual differences in eye and nose positions. Can do.

  In the present embodiment, the example in which the third electrode 156 is provided on the surface of the eyebrow portion 124 has been described. However, it is not limited to this. The eyebrow portion 124 may include an extending portion that extends upward, and the third electrode 156 may be provided in the extending portion. Furthermore, a movable part that moves the extending part up and down between the extending part and the eyebrow part 124 may be provided so that the position of the third electrode 156 can be adjusted up and down. As a result, even if the user makes the contact position of the third electrode 156 near the eye due to individual differences in the eye position, the contact position of the third electrode 156 can be separated from the eye by adjustment. . Further, in the present embodiment, the position of each electrode is not limited to the above-described position, and may be disposed at a position where an electrooculogram signal indicating the vertical and horizontal movements of the eye can be acquired.

  In the present embodiment, as an example of the external device 300, a mobile communication terminal such as a mobile phone and a smartphone that is separate from the processing device 200 has been described. However, it is not limited to this. The external device 300 may be a unit integrated with the processing device 200. In this case, the external device 300 is provided integrally with the eyewear.

  In the present embodiment, a shield cable may be used as the electric wire to prevent noise from being mixed.

  Further, in this embodiment, the configuration using three electrodes is illustrated in FIG. 1, but a configuration using four or more electrodes may be used. In this case, the glasses have an upper electrode, a lower electrode, a left electrode, and a right electrode. For example, the upper electrode and the lower electrode are provided on the rim 122 shown in FIG. 1, the left electrode is provided on the left temple 130, and the right electrode is provided on the right temple 130. There is no. Note that these electrodes are in contact with a part of the face.

  In the example of four electrodes, the vertical direction of the eye can be detected by the voltage difference between the upper electrode and the lower electrode, and the horizontal direction of the eye can be detected by the voltage difference between the left electrode and the right electrode. .

  As mentioned above, although this invention was demonstrated using the Example, the technical scope of this invention is not limited to the range as described in the said Example. It will be apparent to those skilled in the art that various modifications or improvements can be made to the above-described embodiments. It is apparent from the description of the scope of claims that embodiments with such changes or improvements can be included in the technical scope of the present invention.

100 glasses 120 frame 124 eyebrow portion 140 nose pad 152 first electrode 154 second electrode 156 third electrode 200 processing device 300 external device 320 storage unit 330 control unit 340 acquisition unit 342 determination unit 344 recognition unit 346 detection unit 350 threshold calculation unit

Claims (6)

An acquisition step of acquiring a sensor signal indicating the movement of the subject detected by the acceleration sensor and / or the angular velocity sensor, and an electrooculogram signal based on an electrooculogram detected by each electrode in contact with the periphery of the eye of the subject. When,
Based on the electrooculogram signal, a detection step of performing detection processing of blink or eye movement;
A determination step of determining whether the sensor signal is a predetermined pattern;
A stop step of stopping the detection process when the sensor signal is determined to be the predetermined pattern;
A program that causes a computer to execute. The stopping step includes
The program according to claim 1, wherein after the detection process is stopped, the detection process is stopped until a predetermined time elapses after it is determined that the pattern is not the predetermined pattern.   The program according to claim 2, wherein the predetermined time is changed according to the predetermined pattern. The obtaining step includes
The program according to any one of claims 1 to 3, wherein the sensor signal and the electrooculogram signal are acquired from eyewear provided with the acceleration sensor, the angular velocity sensor, and the electrodes. An acquisition unit that acquires a sensor signal indicating the movement of a subject detected by an acceleration sensor and / or an angular velocity sensor, and an electrooculogram signal based on an electrooculogram detected by each electrode that contacts the periphery of the subject's eyes. When,
A determination unit for determining whether the sensor signal is one or a plurality of predetermined patterns;
When it is determined that the pattern is not the predetermined pattern, a detection process for performing blink or eye movement based on the electrooculogram signal, and when the pattern is determined to be the predetermined pattern, a detection unit that stops the detection process;
An information processing apparatus comprising: Between the eyebrows,
A frame having a pair of nose pads;
A first electrode and a second electrode provided on the surface of each of the pair of nose pads;
A third electrode provided on the surface of the portion between the eyebrows;
An acceleration sensor and / or an angular velocity sensor;
An acquisition unit for acquiring an electrooculogram signal based on the sensor signal detected by the acceleration sensor and / or the angular velocity sensor and the electrooculogram detected by each electrode;
A determination unit for determining whether the sensor signal is one or a plurality of predetermined patterns;
When it is determined that the pattern is not the predetermined pattern, a detection process for performing blink or eye movement based on the electrooculogram signal, and when the pattern is determined to be the predetermined pattern, a detection unit that stops the detection process;
Eyewear to prepare.

Images