KR102306111B1 - Method and apparatus for eog-based eye tracking protocol using baseline drift removal algorithm for long-term eye movement detection - Google Patents

Abstract

To a method and apparatus for an EOG (electrooculography)-based eye tracking protocol using a baseline drift removal algorithm for long-time eye movement detection according to various embodiments, an initial calibration is performed on an EOG signal input from a user, Measuring an initial baseline, separating the EOG signal input from the user into a plurality of EOG signals according to a predetermined time unit, and generating a plurality of differential signals calculated from the separated EOG signals by averaging, calculating a differential EOG signal from which baseline drift is removed, and comparing the differential EOG signal to the initial baseline, to identify eye movement of the user.

Description

METHOD AND APPARATUS FOR EOG-BASED EYE TRACKING PROTOCOL USING BASELINE DRIFT REMOVAL ALGORITHM FOR LONG-TERM EYE MOVEMENT DETECTION

Various embodiments relate to a method and apparatus for an electrooculography (EOG)-based eye tracking protocol using a baseline drift cancellation algorithm for long-term eye movement detection.

In recent years, numerous studies have been conducted on human-computer interfaces using eye movement. A user's attention range, characteristics, visual memory, and perceptual learning can be determined by eye movement. Eye movements can also be used to generate equipment control commands. The most popular methods of tracking eye movements are using infrared cameras, red-green-blue (RGB) cameras, and image-based video oculography (VOG). Another method used for eye tracking is electrooculography (EOG) (hereinafter referred to as EOG). EOG measures eye movement through the voltage difference between the cornea and retina. Therefore, EOG has a simple acquisition approach that is relatively economical, robust, and user-responsive. In addition, since the EOG signal is a biosignal, the user's intention can be directly recognized. In addition, the EOG signal is convenient for the user because the position of the electrode attached for EOG is similar to that of a pair of eye goggle frames. Therefore, many studies on equipment control using biosignals adopt EOG signals. For example, an assistive robot application using radio frequency identification and EOG has been proposed. This study uses a portable wireless EOG to control the arm of the robot.

Studies using EOG signals must address the baseline drift problem. The baseline of the EOG signal varies continuously with time due to baseline drift. Baseline can be altered by different nerve signals and small movements of different body parts, and most baseline drifts are independent of eye movements. Therefore, in previous studies, EOG signals were measured over a short period of time, calibration was repeated for long-term activity analysis using EOG signals, and signal transduction was used to eliminate baseline drift. For example, wavelet transforms and bandpass filters (BPFs) have been used to solve the baseline drift problem.

In addition, many design studies have been conducted to improve the portability of equipment control using EOG signals. As an example, there is a study that proposes wearable EOG goggles and eye movement analysis. This study forms a new form of context recognition, which is currently impossible, and the analysis criteria for eye movements in everyday environments. As another example, there is a study that proposes computer control of EOG glasses. However, the design of the user interface is limited because the electrodes for recognizing vertical eye movement are located above and below the side of the eyeball. Numerous wearable eye trackers using EOG signals have been developed to be worn over the face or the entire eye. Existing eye trackers using EOG signals have been mainly used for research due to poor design and inconvenient equipment. Since the lower electrode is fixed to the adipose tissue of the cheekbones, many users felt uncomfortable when attaching the vertical EOG signal electrode for a long time. In addition, the position of the electrodes was arranged around the eyeball, so the size of the EOG signal acquisition device could not be reduced and commercialized. Therefore, it is necessary to solve the problem of baseline drift of the eye tracker by using the EOG signal and to reposition the electrode for the convenience and comfort of the user.

Various embodiments propose a new electrode positioning method based on glasses and a differential EOG signal based on fixation curve (DOSbFC) (hereinafter referred to as a differential EOG signal) based on a fixed curve for eliminating baseline drift.

A method of operating an electronic device according to various embodiments of the present disclosure includes measuring an initial baseline by performing initial calibration on an electrooculography (EOG) signal input from a user, and an EOG signal input from the user. is divided into a plurality of EOG signals according to a predetermined time unit, and a differential EOG signal from which baseline drift is removed by averaging a plurality of differential signals calculated from the separated EOG signals calculating, and comparing the differential EOG signal with the initial baseline to identify the user's eye movement.

An electronic device according to various embodiments may include a detection device that is wearable on the user's face and configured to detect an electrooculography (EOG) signal using a plurality of electrodes disposed around the user's eyeball, and Based on the input EOG signal, it may include a control device configured to track the eye movement of the user.

According to various embodiments, the control device performs initial calibration on the EOG signal input from the detection device, measures an initial baseline, and pre-prelims the EOG signal input from the detection device. Separating a plurality of EOG signals according to a predetermined time unit and averaging a plurality of differential signals calculated from the separated EOG signals to calculate a differential EOG signal from which baseline drift is removed, and compare the differential EOG signal to the initial baseline to identify eye movement of the user.

According to various embodiments, the electronic device may detect a differential EOG signal from which a baseline drift is removed from an EOG signal input from the user, and track the user's eye movement based on the differential EOG signal. Through this, the baseline drift problem in the electronic device may be solved. In addition, the electronic device may detect the EOG signal from the user by using a detection device in which a plurality of electrodes are fixedly disposed at a desired position on a spectacle frame wearable on the user's face. Accordingly, the portability and mobility of the electronic device may be improved, and user convenience and comfort may be improved.

1 is a diagram for explaining an EOG signal according to eye movement.
2 is a diagram for explaining a baseline drift.
3 is a diagram illustrating an electronic device according to various embodiments of the present disclosure;
4 is a diagram illustrating a method of operating an electronic device according to various embodiments of the present disclosure;
5A and 5B are diagrams illustrating an initial baseline measurement operation of FIG. 4 .
6A and 6B are diagrams illustrating an activity recognition operation of eye movement using the EOG signal of FIG. 4 .
FIG. 7 is a view for explaining an operation of removing the baseline drift of FIGS. 6A and 6B .
FIG. 8 is a diagram for explaining an operation of recognizing an eye movement of FIGS. 6A and 6B .
9 is a diagram illustrating an electronic device according to an exemplary embodiment.
10, 11, and 12 are diagrams for explaining an operation of an electronic device according to an exemplary embodiment.
13 is a diagram illustrating an electronic device according to another exemplary embodiment.
14 and 15 are diagrams for explaining an operation of an electronic device according to another exemplary embodiment.

Hereinafter, various embodiments of the present document will be described with reference to the accompanying drawings.

1 is a diagram for explaining an EOG signal according to eye movement. 2 is a diagram for explaining a baseline drift.

Referring to FIG. 1 , the EOG signal may be divided into a horizontal signal and a vertical signal according to the position of the electrode. When the user's gaze is forward (0°), the reference line of the EOG signal may be measured. In general, the baseline may be zero. The EOG signal may have a linear relationship between the visual angle and the amplitude of the signal. Accordingly, eye movements can be quantitatively analyzed. Eye movements may include eye blinking, fixation, and saccades.

The EOG signal according to the blink of an eye may appear as shown in (a) of FIG. 1 . When the user blinks, the eyelids close, the eyeballs contract at the same time, and the eyelids can then open. The blinking of the eye lasts about 100 to 150 ms, and the average range of motion of the eyeball may be about 30°. Thus, blinking can be measured using the vertical signal component of the EOG signal. In general, a person can blink 20 to 30 times per minute.

The EOG signal according to the fixation may appear as shown in (b) of FIG. 1 . Fixation refers to fixing the eyeball in order to focus, and the fixation time refers to the time to focus after stopping the eyeball.

The EOG signal according to the rapid motion may appear as any one of (c), (d) or (e) of FIG. 1 . The rapid movement may refer to an action of moving the gaze, which is currently the focus of attention, to another object. Rapid movements can occur over short periods of 10-100 ms, as opposed to smooth tracking (slow movements). The maximum detection ranges for horizontal and vertical rapid motion are approximately 45° and 40°, respectively. Immobilization can be described as a situation in which eye movement stops for about 150 to 200 ms between two rapid movements.

However, the reference line of the EOG signal may continuously change with time due to the drift of the reference line as shown in FIG. 2 . Baseline drift of the EOG signal can lead to very great difficulties in accurate visualization of waveforms and computerized detection of waveform complexes based on threshold determination. Also, baseline drift is a superposition of slow signal changes in the EOG signal, and may not be mostly related to eye movement. At this time, FIG. 2 may show four types of baseline drift when a 0.1-10 Hz hardware BPF and a 60 Hz notch filter are applied to the EOG signal. Fig. 2(a) shows a linear type of baseline drift, Fig. 2(b) shows a non-linear type of baseline drift, and Fig. 2(c) shows a wavering type may represent the baseline drift of , and FIG. 2( d ) may represent the baseline drift of an oscillating type. However, since the EOG signal does not have a repeating characteristic, it is impossible to remove the baseline drift by a general method. Since the EOG signal has no repeatability, it may not be easy to remove the baseline drift from the EOG signal. On the other hand, when the baseline drift is removed using the existing wavelet transform and BPF, the original EOG signal may be distorted, and thus the recognition rate may decrease.

Noise in the EOG signal can generally be generated by residential power lines such as mains power, measurement circuitry, electrodes, wires, and other sources of physiological interference. Noise in the EOG signal cannot be predicted and corrected since it has no repeating reference signal and pattern. The denosing algorithm of the EOG signal must preserve its properties to enable rapid movement recognition and eye movement analysis. The denoising algorithm must not introduce artificial signals that could be accidentally interpreted as rapid motion during signal processing. Therefore, the denoising algorithm of the EOG signal can use a method of reducing noise by using adjacent characteristics without changing the signal, such as a median filter, a bandpass filter, a lowpass filter, a wavelet, and a moving average filter.

3 is a diagram illustrating an electronic device 300 according to various embodiments.

Referring to FIG. 3 , an electronic device 300 according to various embodiments may include a detection device 310 and a control device 330 .

The detection device 310 may be implemented in the form of glasses. The detection device 310 may include a spectacle frame 311 and a plurality of electrodes 331 , 333 , 335 , 337 , and 339 . The spectacle frame 311 is detachable from the user's face, and the two spectacle frames 313, the nose bridge 315 connecting the spectacle frames 313, and the spectacles legs extending from the spectacle frames 313, respectively. 317 ) and supports 319 respectively disposed at the distal ends of the temples 317 . For example, each spectacle frame 313 may include a nose support. The electrodes 331 , 333 , 335 , 337 , and 339 are dispersedly mounted on the glasses frame 311 , and when the user wears the glasses frame 311 , they may come into contact with the user's eyeballs. For example, the electrodes 331 , 333 , 335 , 337 , 339 may include two vertical electrodes 331 , 333 , two horizontal electrodes 335 , 337 , and one ground electrode 339 . have. The vertical electrodes 331 and 333 may be disposed on at least one of the eyeglass frames 313 and the eyeglass bridge 315 . As an example, the vertical electrodes 331 and 333 may be disposed on the nose support of any one of the bridge of the glasses 315 and the antenna rim 313 . The horizontal electrodes 335 and 337 may be respectively disposed on the temples 317 to be in contact with the user's temple. The ground electrode 339 may be disposed on any one of the supports 319 . It can be in contact with the user's mastoid region.

The control device 330 may track the user's eye movement based on the EOG signal input from the detection device 310 . The control device 330 may perform initial calibration on the EOG signal input from the detection device 310 to measure an initial baseline. Then, the control device 330 calculates a differential EOG signal from which the baseline drift is removed from the EOG signal input from the detection device 310, compares the differential EOG signal with an initial baseline, and controls the user's eye movement. can be identified. In this case, the control device 330 may calculate the differential EOG signal by dividing the EOG signal into a plurality of EOG signals according to a predetermined time unit and averaging the plurality of differential signals calculated from the separated EOG signals.

4 is a diagram illustrating a method of operating an electronic device 300 according to various embodiments of the present disclosure.

Referring to FIG. 4 , the electronic device 300 may measure the user's initial baseline in operation 410 . The electronic device 300 may measure the user's initial reference line based on the input vertical EOG signal and the horizontal EOG signal. In this case, the electronic device 300 may measure the user's initial reference line by performing initial calibration on the input vertical EOG signal and the horizontal EOG signal. Here, operation 410 will be described later in more detail with reference to FIGS. 5A and 5B .

The electronic device 300 may recognize an eye movement activity using the EOG signal in operation 420 . The electronic device 300 may detect a differential EOG signal based on fixation curve (DOSbFC) (hereinafter, referred to as a differential EOG signal) based on the input EOG signal. Through this, the electronic device 300 may remove the baseline drift and noise from the input EOG signal and detect the differential EOG signal. And, based on the differential EOG signal, the electronic device 300 may recognize at least one of eye blinking, vertical rapid movement, horizontal rapid movement, or fixed eye movement. When blinking or rapid movement is detected, the electronic device 300 may sequentially record voltage values of the continuously input EOG signals, and calculate the voltage amplitude after detecting a fixed state. The electronic device 300 may analyze the detection ranges (10°, 20°, and 30°) of the rapid motion by comparing the voltage amplitude of the input EOG signal with an initial reference line for the EOG signal. Here, operation 420 will be described later in more detail with reference to FIGS. 6A and 6B .

5A and 5B are diagrams illustrating an initial baseline measurement operation of FIG. 4 .

Referring to FIG. 5A , the electronic device 300 may detect a vertical EOG signal in operation 511 . The electronic device 300 may perform initial calibration on the vertical EOG signal in operations 513 and 515 . In operation 513 , the electronic device 300 may perform calibration on the vertical EOG signal in each of the upward and downward directions, based on a voltage value for each user corresponding to an eye movement at an angle of 1°. This calibration may be performed separately from chain processing of the input vertical EOG signal. In operation 515 , for each of the upward and downward directions, the electronic device 300 performs the vertical EOG signal and the vertical EOG signal based on the voltage value between the positive and negative signals due to the difference in the user's eye shape, eye voltage, and biological characteristics. Calibration can be performed on the signal. Through this, an initial reference line for the vertical EOG signal may be measured.

Referring to FIG. 5B , the electronic device 300 may detect a horizontal EOG signal in operation 531 . The electronic device 300 may perform initial calibration on the vertical EOG signal in operations 533 and 535 . In operation 533 , the electronic device 300 may perform calibration on the horizontal EOG signal in each of the right direction and the left direction based on a voltage value for each user corresponding to an eye movement at an angle of 1°. This calibration may be performed separately from chain processing of the input horizontal EOG signal. In operation 535 , the electronic device 300 performs the horizontal EOG signal in the right direction and the left direction, respectively, based on the voltage value between the horizontal EOG signal and the positive and negative signals due to differences in the user's eye shape, eye voltage, and biological characteristics. Calibration can be performed on the signal. Through this, an initial reference line for the horizontal EOG signal may be measured.

6A and 6B are diagrams illustrating an activity recognition operation of eye movement using the EOG signal of FIG. 4 .

Referring to FIG. 6A , the electronic device 300 may detect a vertical EOG signal in operation 611 . The electronic device 300 may detect a differential EOG signal in operation 613 . Through this, the baseline drift of the vertical EOG signal is removed, and the differential EOG signal can be detected. The electronic device 300 may determine whether the differential EOG signal corresponds to vertical eye movement in operation 615 . If it is determined in operation 615 that the differential EOG signal does not correspond to vertical eye movement and includes noise, the electronic device 300 may remove noise from the differential EOG signal in operation 617 and return to operation 611 .

If it is determined that the differential EOG signal corresponds to the vertical eye movement in operation 615 , the electronic device 300 may determine whether the vertical eye movement corresponds to blinking in operation 619 . If it is determined that the vertical eye movement corresponds to blinking in operation 619 , the electronic device 300 may recognize the vertical eye movement as blinking in operation 621 . If it is determined in operation 619 that the vertical eye movement does not correspond to blinking, the electronic device 300 may compare an initial reference line for the differential EOG signal and the vertical EOG signal in operation 623 . Here, operation 623 will be described later in more detail with reference to FIG. 8 . Based on the comparison result of operation 623 , the electronic device 300 may recognize the vertical eye movement as a rapid downward movement in operation 625 or recognize the vertical eye movement as a rapid upward movement in operation 627 .

When the electronic device 300 detects eye blinking or rapid movement, the electronic device 300 may sequentially record voltage values of continuously input vertical EOG signals, and calculate the voltage amplitude after detecting a fixed state. . The electronic device 300 may analyze the detection ranges (10°, 20°, and 30°) of the rapid motion by comparing the voltage amplitude of the input vertical EOG signal with an initial reference line for the vertical EOG signal. The vertical EOG signal can be guaranteed linearity of up to 30°. To minimize the linearity error of the EOG signal, the fast motion detection range can be limited to 30° (±5°).

Referring to FIG. 6B , the electronic device 300 may detect a horizontal EOG signal in operation 631 . The electronic device 300 may detect a differential EOG signal in operation 633 . Through this, the baseline drift of the horizontal EOG signal may be removed, and the differential EOG signal may be detected. The electronic device 300 may determine whether the differential EOG signal corresponds to a horizontal eye movement in operation 635 . If it is determined in operation 635 that the differential EOG signal does not correspond to the horizontal eye movement and includes noise, the electronic device 300 may remove the noise from the differential EOG signal in operation 637 and return to operation 631 .

If it is determined in operation 635 that the differential EOG signal corresponds to the horizontal eye movement, the electronic device 300 may compare the differential EOG signal and an initial reference line for the horizontal EOG signal in operation 643 . Here, operation 643 will be described later in more detail with reference to FIG. 8 . Based on the comparison result in operation 643 , the electronic device 300 may recognize the horizontal eye movement as the fast left movement in operation 645 or the horizontal eye movement as the rapid right movement in operation 647 .

When a rapid motion is detected, the electronic device 300 may sequentially record voltage values of continuously input horizontal EOG signals, and calculate the voltage amplitude after detecting a fixed state. The electronic device 300 may analyze the detection ranges (10°, 20°, and 30°) of the rapid motion by comparing the voltage amplitude of the input horizontal EOG signal with an initial reference line for the horizontal EOG signal. The horizontal EOG signal can be guaranteed linearity of up to 30°. To minimize the linearity error of the EOG signal, the fast motion detection range can be limited to 30° (±5°).

FIG. 7 is a view for explaining an operation of removing the baseline drift of FIGS. 6A and 6B .

Referring to FIG. 7 , a new method for detecting eye movement and removing baseline drift and noise using EOG signals to control a mobile human-computer interface application program may be proposed. In general, when looking at an object in front, the EOG signal should have a reference voltage of 0 V. The proposed method allows long-term recording of eye movements and requires only a single calibration. For initial calibration, voltage values for each object corresponding to movement of 10°, 20°, and 30° angles in four directions, left, right, up, and down, may be measured. This process is repeated 5 times, through which the average voltage value can be calculated. For the initial calibration, the user should see a point located at a specified angle on the screen. Based on the linearity of the EOG signal, voltage values increased by 1° in the four directions of the eye line of sight can be automatically calculated. The time required to fix eye movements following eye blinking or rapid movement may be referred to as a fixation time. Depending on the user, the fixed time may be 150 to 300 ms. Here, the fixed time may be defined as at least 150 ms.

The electronic device 300 may convert the horizontal and vertical input EOG signals into differential EOG signals to minimize baseline drift. Compared with the original EOG signal, the proposed differential EOG signal can change the amplitude of the rapid motion. 7A may represent the original EOG signal generated by the rapid motion. All EOG signals measured between the initial eye movement and the fixation time can be recorded. At this time, for the original EOG signal, a window may be determined. Since the fixed time for motion recognition is at least 150 ms, the size of the window may be set to include all EOG signals generated during 150 ms to obtain a differential EOG signal c _{i with respect to the i-th input signal f i .} A window can be used to distinguish the start and end points of the input signal for real-time processing. 7 (b) is calculated as the difference in the voltage amplitude of the EOG signal separated by 10 ms from the signals _{of one window f i+1} and f _i , ..., f _i+14 and f _i+13 shows the differential signals. Since the neuron activation time to generate a measurable synaptic potential is about 10 ms, and the minimum duration of a rapid movement is 10 ms, an interval of 10 ms can be used. _{7C may represent a differential EOG signal c i} obtained by averaging the differential signals of the window. The interval of the window may be set to 10 ms in order to recognize the change in amplitude for rapid motion. That is, the electronic device 300 may gradually calculate the _{differential EOG signal c i+1} by using the EOG signal of the next window including the EOG signal f _{i+1 after 10 ms.} In this case, the differential EOG signal may be calculated as in Equation 1 below.

Here, w is 0, 1, ..., 14, and represents an index value of an EOG signal selected at intervals of 10 ms in the window, and w may be used to select an input signal within the window. f _i may represent the voltage amplitude of the i-th EOG signal.

For each EOG signal f _i , the voltage of the corresponding differential EOG signal c _i may be calculated by averaging the difference values of the EOG signals between the initial eye movement and the fixation time. The proposed method can select and average the most similar dislocations within the window range. It can slide the window once along the sample to find the differences between the selected signals and average them to get _{the c i signal.} The average can be obtained by setting the next interval and selecting a new set of dislocations. Instead of simply averaging the signal differences, an optional differential averaging of windows can be used. The averaging process has a similar shape to the various potentials of a sample set and can be based on the selection of signal sections using uniform weights of the selected potentials. The baseline of the differential EOG signal is close to the baseline (zero voltage) when sudden changes such as rapid motion do not occur, so that both baseline drift and noise due to the moving average effect can be minimized. Thus, the baseline of the differential EOG signal can be used to track the eye gaze point coordinates.

By comparing the differential EOG signal with a user-specific voltage threshold of 1° eye movement in all directions, eye movement or noise recognition can be calculated as shown in Equation 2 below.

Here, b _N is the horizontal (vertical) EOG signal and represents the user-specific voltage (negative) of 1° eye movement in the left (down) direction, and b _P is the horizontal (vertical) EOG signal, which is 1° in the right (vertical) direction. It can represent the user-specific voltage (positive number) of eye movement.

The EOG signal may have a potential difference depending on the position of the eyeball and the electrode. Since the linearity of the EOG signal is maintained in one direction, positive and negative user-specific voltage values can be measured for high-precision recognition of rapid motion. Thus, b _P and b _N can be calculated. Voltage values for 1° eye movement in each direction can be calculated using a one-time calibration. A difference value within a given range may indicate no eye movement (E _mov = 0). Otherwise, eye movement can be considered to have occurred (E _mov = 1). This process can be repeated for successive vertical and horizontal input EOG signals. Eye movement with a viewing angle of less than 5° may be recognized as noise.

FIG. 8 is a diagram for explaining an operation of recognizing an eye movement of FIGS. 6A and 6B .

Referring to FIG. 8 , eye movement may be classified into eye blinking, fixation, and rapid movement. For rapid motion, 12 examples (4 directions: left, right, up and down, 3 sensing ranges: 10°, 20°, 30°) can be considered. If E _{mov becomes one, the proposed method can calculate c i} , which is the sum of the difference values between the initial eye movement time and the fixation time. This may be due to the linear relationship between the EOG signal and the eye gaze angle. 8 may show examples of eye movement indicated by a difference value of an EOG signal. The x-axis and y-axis may represent time and voltage, respectively. Horizontal and vertical eye movements can be recognized separately. v may be defined as a vertical signal, and h may be defined as a horizontal signal. For example, if the voltage value for each user is 1°, it may be _{b vN.} Vertical eye movement recognition can consist of two stages: eye blinking and up/down rapid movement recognition. In the first step of vertical eye movement recognition, eye blink recognition may be calculated as shown in Equation 3 below.

, c_vi < 0 이면

이고, c_vi는 상기 [수학식 1]을 이용하여 계산된 EOG 신호의 차이 값을 나타낼 수 있다. vi = 1,2, … , n이고, n은 안구 움직임 시작 시간(E_mov = 1)부터 고정 시간까지 EOG 신호의 입력 수일 수 있다. c_vi의 합계는, c_vi가 양수일 때 P(c_v)와 같고, 음수일 때 N(c_v)와 같다고 정의될 수 있다. 사용자의 눈이 깜박일 때(B_mov = 1) 안구는 위아래로 움직인 다음 원래 위치로 돌아올 수 있다. 또 안구 움직임의 각도가 10°보다 크면 깜박임(B_mov = 1)으로 감지될 수 있다. 그렇지 않으면 위/아래 신속 운동으로 감지될 수 있다(B_mov = 0). 만약 B_mov가 1이 된다면 수직 안구 움직임 인식은 종료될 수 있다. 수직 안구 움직임 인식의 두 번째 단계(B_mov = 0)에서는, 하기 [수학식 4]와 같이 위/아래 신속 운동 인식이 계산될 수 있다.Here, b _vP represents the user-specific voltage of the positive vertical EOG signal, b _vN represents the user-specific voltage of the negative vertical EOG signal, and _{if c vi} > 0

, if c _vi < 0

, and c _vi may represent a difference value between EOG signals calculated using Equation 1 above. vi = 1,2, … , n, and n may be the number of inputs of the EOG signal from the eye movement start time (E _{mov = 1) to the fixed time.} the sum of c is _{vi, vi} c when a positive number equal to P _(v c), may be defined the same as N (c _v) when a negative number. When the user's eye blinks (B _mov = 1), the eye can move up and down and then return to its original position. Also, if the angle of eye movement is greater than 10°, it can be detected as _{blinking (B mov = 1).} Otherwise, it can be detected as an up/down rapid movement (B _mov = 0). If B _mov becomes 1, vertical eye movement recognition may be terminated. In the second step of vertical eye movement recognition (B _mov = 0), up/down rapid movement recognition may be calculated as shown in Equation 4 below.

As shown in the up/down rapid movement of FIG. 8 , when the user makes an up/down rapid movement of the eyeball, the eyeball may move up/down and then return to the original position. Unless the user intentionally moves the eye, the pattern of rapid movement can only be created in one direction. Accordingly, the electronic device 300 may recognize the direction of the rapid motion by comparing the value of _{P(c v} ) with the absolute value of N(c _{v ).} If P(cv) is _{greater than N(c v} ), it may be determined as an upward rapid motion, otherwise it may be determined as a downward rapid motion. The up/down rapid motion calculated as in [Equation 4] may be expressed in degrees (degrees). P(c _v ) and N(c _v ) may represent the total eye movement distance, and b _vP and b _vN may represent 1° user-specific voltage. Accordingly, the angle of the rapid motion may be calculated by dividing the voltage for each user and the larger of the two comparison values.

In addition, in the same way as applied to recognize vertical up/down rapid movement, horizontal eye movement may be recognized as shown in Equation 5 below.

, c_hi < 0 이면

이며, c_hi는 수평 EOG 신호의 차이값을 나타낼 수 있다. 사용자가 오른쪽 신속 운동의 안구 움직임을 할 때, 도 8과 같이 서로 다른 EOG 신호 값이 위쪽으로 이동한 다음 원래 위치로 되돌아가고, 이 신속 운동에 따라 약 150 ms의 고정 상태가 발생할 수 있다. 안구 움직임의 각도가 +5°(E_mov = 1)보다 클 경우 수평 EOG 신호가 있는 우측 신속 운동이 검출될 수 있다. 사용자가 왼쪽 신속 운동의 안구 움직임을 할 때, 도 8과 같이 다른 EOG 신호 값이 아래로 이동한 후 원래 위치로 되돌아가고, 이 신속 운동에 따라 대략 150 ms의 고정 상태가 발생할 수 있다. 눈 움직임의 각도가 -5°(E_mov = 1) 미만이면 수평 EOG 신호가 있는 좌측 신속 운동이 검출될 수 있다. 눈 깜박임이나 신속 운동이 발생한 후 전압이 기준선으로 복귀하면 고정 상태가 감지될 수 있다.Here, b _hP represents the user-specific voltage of the positive horizontal EOG signal, b _hN represents the user-specific voltage of the negative horizontal EOG signal, and _{if c hi} > 0

, if c _hi < 0

, and c _hi may represent a difference value of the horizontal EOG signal. When the user makes an eye movement of the right rapid movement, different EOG signal values move upward and then return to the original position as shown in FIG. 8, and a stationary state of about 150 ms may occur according to this rapid movement. When the angle of eye movement is greater than +5° (E _mov = 1), a right sacral motion with a horizontal EOG signal can be detected. When the user makes an eye movement of the left rapid movement, as shown in FIG. 8 , another EOG signal value moves down and then returns to the original position, and a fixed state of approximately 150 ms may occur according to this rapid movement. If the angle of the eye movement _{is less than -5° (E mov} = 1), a left rapid movement with a horizontal EOG signal can be detected. A stationary state can be detected when the voltage returns to baseline after a blink or rapid movement has occurred.

9 is a diagram illustrating an electronic device (eg, the electronic device 300 of FIG. 3 ) 900 according to an exemplary embodiment. 10, 11, and 12 are diagrams for explaining the operation of the electronic device 900 according to an embodiment.

Referring to FIG. 9 , an electronic device 900 according to an embodiment may be implemented as a desktop computer. To this end, a desktop application controlling a human-computer interface can be implemented for use on any desktop platform. Desktop applications can also run as background processes. In order to increase performance and memory efficiency, an application may include an input/output (I/O) thread 910 , a main thread 920 , a control thread 930 , and a render thread 940 . This multithreaded configuration can increase resource sharing efficiency and responsiveness, and enable code modularity. The input/output thread 910 may support the EOG signal acquisition equipment (eg, the detection device 310 of FIG. 3 ) when managing the input EOG signal and communicating with the desktop computer. The input/output thread 910 may identify vertical, horizontal, and forward signals while waiting in the signal processing function block. The main thread 920 uses the difference value of the EOG signal to control the remaining three threads 910 , 930 , and 940 and the recognized eye movement. The control thread 930 may execute instructions to control hardware such as speakers, mice, keyboards, and external applications. The render thread 940 may manage the graphical interface to display the status and display the input EOG.

A master and settings application can be implemented to control the human-computer interface using the EOG signal to set the initial command and control level. The desktop master application and the desktop setting application may be represented as shown in FIG. 10 . The master application may analyze the input EOG signal as shown in (a) of FIG. 10 and execute the command according to a specified criterion. Also, when the eye movement recognition is completed, a command may be executed by the input/output device according to a function set through the setting application. This allows the master application to be used as a secondary controller in any Windows-based application. Eye protection to prevent visual display terminal (VDT) syndrome may be implemented. Eye protection can generate a warning message and sound a 'beep' when it detects less than 10 blinks in a minute (solution to reduce VDT syndrome). The setting application may set the start motion, command type, and command level as shown in FIG. 10( b ). The start motion can consist of right click, middle mouse button click, double blink and triple blink.

The command may be divided into three levels as shown in FIG. 11 according to the type of action. Commands can be configured using combo boxes. According to level 1 as shown in (a) of FIG. 11 , there may be four commands according to four movements in which the eyeball moves vertically or horizontally by 30° or more. For example, as shown in (a) of FIG. 12 , a command corresponding to the right eye movement of level 1 may be generated. According to level 2 as shown in (b) of FIG. 11 , there may be 8 commands according to 8 movements of the eyeball sequentially moving 30° or more in vertical and horizontal directions, or in horizontal and vertical directions. For example, as shown in FIG. 12B , commands corresponding to the upper and left eye movements of level 2 may be generated. According to level 3 as shown in (c) of FIG. 11 , there may be 12 commands according to 12 movements of the eyeball moving horizontally or vertically by 10°, 20°, and 30°. For example, as shown in FIG. 12C , a command corresponding to an upward eye movement of 20° of level 3 may be generated. The command can be executed when the eye returns to its original position within 300 ms after movement. Since it is not easy to move the eyeball in this way, the diagonal movement of the eyeball may not be considered. In addition, when moving both eyes simultaneously, noise may occur due to interference from both eyes, so multi-level stepwise control can be used to improve usability while maintaining linearity in the EOG signal. For example, the setting command of the desktop application may be defined as follows [Table 1], [Table 2] and [Table 3].

13 is a diagram illustrating an electronic device (eg, the electronic device 300 of FIG. 3 ) 1300 according to another exemplary embodiment. 14 and 15 are diagrams for explaining an operation of the electronic device 1300 according to another exemplary embodiment.

Referring to FIG. 13 , an electronic device 1300 according to another embodiment may be implemented as a mobile device. To this end, the mobile application architecture design can be implemented similar to the desktop application design. Mobile devices may have lower performance and less memory than desktop computers. Therefore, optimization can be performed by isolating threads. In order to improve performance and memory efficiency, it may be composed of an input/output thread 1310 , a main thread 1320 , a control thread 1330 , a render thread 1340 , and a multimedia application 1350 . The input/output thread 1310 may manage the input EOG signal and connect the EOG signal acquisition apparatus (eg, the detection apparatus 310 of FIG. 3 ) and the mobile device through Bluetooth. The main thread 1320 may control other threads 1310 , 1330 , 1340 , and 1350 and manage memory using a garbage collector, EOG signal processing, eye movement detection, and the like. The control thread 1330 may execute instructions to control the application. The render thread 1340 may manage view components and other components.

The mobile application can implement the multimedia control function of the mobile device in 7 ways by using the EOG signal. An eye protection function can also be applied that alerts you when the blinking rate per minute is insufficient. Screenshots of the application may be presented as shown in FIG. 14 . The default start motion can be set to double blink. The setting function can be implemented in the same way as the desktop application. The setting values can be saved in the SD card. The MP3 player function may be activated as a background service. For example, the comic viewer function may be controlled as shown in FIG. 15 , based on the command according to the level 1 eye movement. When the start motion (twice blink) is executed, the application will access the previous page, when the eye returns to its original position after a 30° left rapid movement, and the eye returns to its original position after a 30° right rapid movement. When you return, you can access the next page. For example, the setting command of the mobile application may be defined as [Table 1], [Table 2], and [Table 3].

The electronic devices 300 , 900 , and 1300 according to various embodiments of the present disclosure use a plurality of electrodes 331 , 333 , 335 , 337 , 339 that can be worn on the user's face and are disposed around the user's eyeball. a detection device 310 configured to detect an electrooculography (EOG) signal; and a control device 330 configured to track the eye movement of the user based on the EOG signal input from the detection device 310 . can

According to various embodiments, the control device 330 performs initial calibration on the EOG signal input from the detection device 310 to measure an initial baseline, and the detection device ( The EOG signal input from 310 is divided into a plurality of EOG signals according to a predetermined time unit, and a plurality of differential signals calculated from the separated EOG signals are averaged to prevent baseline drift. and calculate a differential EOG signal to be removed and compare the differential EOG signal to the initial baseline to identify eye movement of the user.

According to various embodiments, the detection device 310 is wearable on the electrodes 331 , 333 , 335 , 337 , 339 and the user's face, and the electrodes 331 , 333 , 335 , 337 , 339) may include a spectacle frame 311 in which the dispersed arrangement.

According to various embodiments, the EOG signal input from the detection device 310 may include a horizontal EOG signal and a vertical EOG signal.

According to various embodiments, the control device 330 may be configured to identify the eye movement as any one of eye blinking, horizontal rapid movement, vertical rapid movement, or fixed movement.

According to various embodiments, the electrodes 331 , 333 , 335 , 337 , 339 are at least one horizontal electrode 335 , 337 configured to detect the horizontal EOG signal, configured to detect the vertical EOG signal It may include at least one vertical electrode 331 and 333, and a ground electrode 339 for grounding the horizontal electrodes 335 and 337 and the vertical electrodes 331 and 333.

According to various embodiments, the spectacle frame 311 includes two spectacle frames 313 , a nose bridge 315 connecting the spectacle frames 313 , and extending from the spectacle frames 313 , respectively. It may include temples 317 and supports 319 respectively disposed at distal ends of the temples 317 .

According to various embodiments, the horizontal electrodes 335 and 337 are disposed inside at least one of the temples 317, and when the glasses frame 311 is worn by the user, the user's temple portion is in contact with, the vertical electrodes 331 and 333 are disposed on at least one of the nose bridge 315 and the eyeglass frames 313 , and the ground electrode 319 is at least one of the supports 339 . It is disposed on any one, and when the glasses frame 311 is worn by the user, it may be in contact with the mastoid region of the user.

According to various embodiments, the control device determines a window corresponding to a predetermined fixed time from the EOG signal input from the user, and applies the EOG signal corresponding to the determined window to a plurality of EOGs according to a predetermined time unit. may be configured to separate into signals.

A method of operating an electronic device 300 , 900 , 1300 according to various embodiments includes performing an initial calibration on an electrooculography (EOG) signal input from a user to measure an initial baseline; Separating the EOG signal input from the user into a plurality of EOG signals according to a predetermined time unit, averaging a plurality of differential signals calculated from the separated EOG signals, thereby performing baseline drift It may include calculating a differential EOG signal from which is removed, and comparing the differential EOG signal with the initial reference line to identify the user's eye movement.

According to various embodiments, the EOG signal input from the user includes a horizontal EOG signal and a vertical EOG signal, and the identification operation performs the eye movement as any one of blinking, horizontal rapid movement, vertical rapid movement, or fixed movement. It may include an action to identify.

According to various embodiments, the separation operation may include determining a window corresponding to a predetermined fixed time from the EOG signal input from the user, and a plurality of EOG signals corresponding to the determined window according to a predetermined time unit. It may include an operation of dividing into EOG signals.

According to various embodiments, the measuring operation may include detecting the vertical EOG signal, in each of the upward and downward directions, based on the user's voltage value corresponding to 1° angular eye movement, the vertical EOG calibrating the signal, and for each of the upward and downward directions, based on the voltage value between the vertical EOG signal and a positive and negative signal due to a difference in the user's eye shape, eye voltage, and biological characteristics , an operation of performing calibration on the vertical EOG signal, and through this, an initial reference line for each of the upward and downward directions may be measured.

According to various embodiments, the measuring operation may include detecting the horizontal EOG signal, based on the user's voltage value corresponding to 1° angular eye movement, in each of the left and right directions, the horizontal EOG calibrating the signal, and for each of the left and right directions, based on the voltage value between the horizontal EOG signal and a positive and negative signal due to a difference in the user's eye shape, eye voltage, and biological characteristics , an operation of performing calibration on the horizontal EOG signal, and through this, an initial reference line for each of the left direction and the right direction may be measured.

According to various embodiments, the differential EOG signal includes a vertical differential EOG signal calculated from the vertical EOG signal, and the identification operation includes the eye movement based on the vertical differential EOG signal and the user's voltage value. the operation of identifying as blinking, the operation of identifying the eye movement as a vertical rapid movement if the eye movement is not identified as the blinking of the eye, the distance of the eye movement based on the vertical differential EOG signal and the initial baseline determining, and recognizing the vertical rapid motion in either the upward direction or the downward direction based on the distance of the eye movement and the user's voltage.

According to various embodiments, the differential EOG signal includes a horizontal differential EOG signal calculated from the horizontal EOG signal, and the identification operation includes the eye movement based on the horizontal differential EOG signal and the user's voltage value. the operation of identifying as horizontal rapid movement, the operation of determining the distance of the eye movement based on the horizontal differential EOG signal and the initial baseline, and the operation of the horizontal rapid movement based on the distance of the eye movement and the user's voltage It may include an operation of recognizing one of the left direction and the right direction.

According to various embodiments, the identification operation is

The method may further include identifying a detection range of the horizontal rapid motion or a detection range of the vertical rapid motion.

According to various embodiments, the electronic device 300 , 900 , 1300 may detect a differential EOG signal from which a baseline drift is removed from an EOG signal input from the user, and track the user's eye movement based on the differential EOG signal. have. Through this, the baseline drift problem in the electronic device may be solved. In addition, the electronic devices 300, 900, and 1300 are a detection device 310 in which a plurality of electrodes 331, 333, 335, 337, and 339 are fixedly disposed at a desired position on a spectacle frame 311 that can be worn on the user's face. can be used to detect the EOG signal from the user. Accordingly, the portability and mobility of the electronic devices 300 , 900 , and 1300 may be improved, and user convenience and comfort may be improved.

The electronic device according to various embodiments disclosed in this document may have various types of devices. The electronic device may include, for example, a portable communication device (eg, a smart phone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a home appliance device. Electronic devices according to various embodiments of the present document are not limited to the aforementioned devices.

It should be understood that the various embodiments of this document and the terms used therein are not intended to limit the technology described in this document to a specific embodiment, and include various modifications, equivalents, and/or substitutions of the embodiments. In connection with the description of the drawings, like reference numerals may be used for like components. The singular expression may include the plural expression unless the context clearly dictates otherwise. In this document, expressions such as “A or B”, “at least one of A and/or B”, “A, B or C” or “at least one of A, B and/or C” refer to all of the items listed together. Possible combinations may be included. Expressions such as “first”, “second”, “first” or “second” can modify the corresponding components regardless of order or importance, and are only used to distinguish one component from another. The components are not limited. When an (eg, first) component is referred to as being “connected (functionally or communicatively)” or “connected” to another (eg, second) component, that component is It may be directly connected to the component or may be connected through another component (eg, a third component).

As used herein, the term “module” includes a unit composed of hardware, software, or firmware, and may be used interchangeably with terms such as, for example, logic, logic block, component, or circuit. A module may be an integrally formed part or a minimum unit or a part of one or more functions. For example, the module may be configured as an application-specific integrated circuit (ASIC).

Various embodiments of the present document are stored in a storage medium (eg, memory 125, memory 139) readable by a machine (eg, server 120, electronic device 130). It may be implemented as software comprising one or more stored instructions. For example, the processor (eg, the processor 127 or the processor 140 ) of the device may call at least one of one or more instructions stored from a storage medium and execute it. This enables the device to be operated to perform at least one function according to at least one command called. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The device-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' only means that the storage medium is a tangible device and does not contain a signal (eg, electromagnetic wave), and this term is used in cases where data is semi-permanently stored in the storage medium and It does not distinguish between temporary storage cases.

According to various embodiments, each component (eg, a module or a program) of the described components may include a singular or a plurality of entities. According to various embodiments, one or more components or operations among the above-described corresponding components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (eg, a module or a program) may be integrated into one component. In this case, the integrated component may perform one or more functions of each component of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to integration. According to various embodiments, operations performed by a module, program, or other component are executed sequentially, in parallel, repeatedly, or heuristically, or one or more of the operations are executed in a different order, omitted, or , or one or more other operations may be added.

Claims (17)

A method of operating an electronic device, comprising:
measuring an initial baseline by performing initial calibration on an electrooculography (EOG) signal input from a user;
dividing the EOG signal input from the user into a plurality of EOG signals according to a predetermined time unit;
calculating a differential EOG signal from which baseline drift is removed by averaging a plurality of differential signals calculated from the separated EOG signals; and
Comprising the operation of comparing the differential EOG signal with the initial baseline to identify the user's eye movement,
The separation operation is
determining a window corresponding to a predetermined fixed time in the EOG signal input from the user; and
and dividing the EOG signal corresponding to the determined window into a plurality of EOG signals according to a predetermined time unit.
The method of claim 1,
The EOG signal input from the user includes a horizontal EOG signal and a vertical EOG signal,
The identification operation is
and identifying the eye movement as any one of blinking, horizontal rapid movement, vertical rapid movement, or fixation.
delete According to claim 1, wherein the calculation operation,
A method of calculating the differential EOG signal based on the following equation.

Here, w denotes an index value for the separated EOG signals, and f _i denotes the voltage amplitude of the EOG signal of the i-th window.
The method of claim 2, wherein the measuring operation comprises:
detecting the vertical EOG signal;
performing calibration on the vertical EOG signal based on the user's voltage value corresponding to 1° angular eye movement in each of the upward and downward directions; and
For each of the upward and downward directions, calibration is performed on the vertical EOG signal based on a voltage value between the vertical EOG signal and a positive and negative signal due to a difference in the user's eye shape, eye voltage, and biological characteristics. including the action to be performed;
Through this, an initial reference line for each of the upward and downward directions is measured.
The method of claim 2, wherein the measuring operation comprises:
detecting the horizontal EOG signal;
performing calibration on the horizontal EOG signal in each of the left and right directions, based on the user's voltage value corresponding to 1° angular eye movement; and
For each of the left and right directions, calibration is performed on the horizontal EOG signal based on a voltage value between the horizontal EOG signal and a positive and negative signal due to a difference in the user's eye shape, eye voltage, and biological characteristics including the action to be performed;
Through this, an initial reference line for each of the left direction and the right direction is measured.
6. The method of claim 5,
The differential EOG signal includes a vertical differential EOG signal calculated from the vertical EOG signal,
The identification operation is
identifying the eye movement as blinking based on the vertical differential EOG signal and the user's voltage value;
if the eye movement is not identified as the eye blink, identifying the eye movement as a vertical rapid movement;
determining the distance of the eye movement based on the vertical differential EOG signal and the initial reference line; and
and recognizing the vertical rapid movement in either the upward direction or the downward direction based on the distance of the eye movement and the user's voltage.
7. The method of claim 6,
The differential EOG signal includes a horizontal differential EOG signal calculated from the horizontal EOG signal,
The identification operation is
identifying the eye movement as a horizontal rapid movement based on the horizontal differential EOG signal and the user's voltage value;
determining the distance of the eye movement based on the horizontal differential EOG signal and the initial reference line; and
and recognizing the horizontal rapid movement as either the left direction or the right direction based on the distance of the eye movement and the user's voltage.
The method of claim 2, wherein the identification operation comprises:
and identifying the detection range of the horizontal rapid movement or the detection range of the vertical rapid movement.
In an electronic device,
a detection device that is wearable on the user's face and is configured to detect an electrooculography (EOG) signal using a plurality of electrodes disposed around the user's eyeball; and
Based on the EOG signal input from the detection device, comprising a control device configured to track the eye movement of the user,
The control device is
By performing an initial calibration (calibration) on the EOG signal input from the detection device, measuring an initial baseline (initial baseline),
Separating the EOG signal input from the detection device into a plurality of EOG signals according to a predetermined time unit,
calculating a differential EOG signal from which baseline drift is removed by averaging a plurality of differential signals calculated from the separated EOG signals;
and compare the differential EOG signal to the initial baseline to identify eye movements of the user;
The control device is
Determining a window corresponding to a predetermined fixed time from the EOG signal input from the user,
and splitting the EOG signal corresponding to the determined window into a plurality of EOG signals according to a predetermined time unit.
delete 11. The method of claim 10,
The EOG signal input from the detection device includes a horizontal EOG signal and a vertical EOG signal,
The control device is
an apparatus configured to identify the eye movement as any one of eye blink, horizontal saccade, vertical saccade, or fixation.
13. The method of claim 12, wherein the electrodes
at least one horizontal electrode configured to detect the horizontal EOG signal;
at least one vertical electrode configured to detect the vertical EOG signal; and
and a ground electrode for grounding the horizontal electrode and the vertical electrode.
14. The method of claim 13,
The detection device is
the electrodes; and
It is wearable on the user's face and includes a glasses frame in which the electrodes are dispersed,
The glasses frame,
two eyeglass frames;
a nose bridge connecting the glasses frames;
temples each extending from the spectacle frames; and
an apparatus comprising supports respectively disposed at the distal ends of the temples.
15. The method of claim 14,
The horizontal electrode is disposed inside at least one of the temples, so that when the glasses frame is worn by the user, it is in contact with the user's temple region,
The vertical electrode is disposed on at least one of the spectacles nose bridge or the spectacle frames,
The ground electrode is disposed on at least one of the supports, and when the glasses frame is worn by the user, the device is in contact with the mastoid region of the user.
delete 11. The method of claim 10, wherein the control device,
An apparatus configured to calculate the differential EOG signal based on the following equation.

Here, w denotes an index value for the separated EOG signals, and f _i denotes the voltage amplitude of the EOG signal of the i-th window.

Images