JP6338445B2 - Blink detection system and method - Google Patents

Description

  The present invention relates to a blink detection system and method using a Doppler sensor.

  In recent years, in order to prevent traffic accidents caused by falling asleep during driving or lowering of consciousness level, a technique for detecting the degree of fatigue and drowsiness of a driver while driving has attracted attention. Although a method for detecting a decrease in the consciousness level by looking at the driving situation of the driver, for example, a change in the movement of the vehicle, is conceivable, it is desirable that the decrease in the consciousness level can be detected before it appears in the driving situation. For example, the degree of consciousness reduction can be estimated by acquiring biological information representing a physical state such as fatigue level or sleepiness level. Examples of biological information include heartbeat, brain waves, blinks, and the like. In particular, blink measurement is known to more directly reflect a decrease in consciousness than other methods. Acquisition of blink information is also useful for preventing dry eyes associated with a decrease in the number of blinks during work concentration, such as screen operations on information devices.

  Many researches aimed at detecting blinks are technologies that use contact-type devices and cameras, and are accompanied by discomfort such as the feeling of pressure that comes with wearing and the feeling of surveillance by the camera. As a technique for detecting blinks in a non-contact manner without using a camera, driver blink detection using a Doppler sensor has been proposed (for example, see Non-Patent Document 1). In this method, the circuit design is performed so that the output waveform of the Doppler sensor when blinking is detected is composed of three half sine waves, and three continuous peaks are extracted from the sensor output waveform to blink. Detected. This document targets a driver and performs blink detection during a period in which only blinks occur in the absence of other body movements.

Staszek, K., Wincza, K., Gruszczynski, S., "Driver's Drowsiness Monitoring System Utilizing Microwave Doppler Sensor", 19th International Confeene on Microwave Radar and Wireless Communications, p623-626, Ma, 2012

  In the known blink detection method using the Doppler sensor, the premise that the waveform relating to blinking becomes three half sine waves depends on the circuit design of the Doppler sensor, and is detected when the number of half sine waves included in the output waveform increases. Accuracy will be reduced. In addition, since blink detection is performed during a period in which only blinking occurs, no distinction is made between the operation of shaking the head, changing the orientation of the face, and the like. Furthermore, the effects of individual differences are not taken into account.

  Therefore, it is an object to provide a blink sensor that can detect a blink by analyzing a signal waveform without depending on circuit design. Moreover, the structure which can distinguish large body movements, such as shaking a head, and blink, and the structure which can reduce the influence of an individual difference are provided.

  In order to solve the above-described problem, in the present invention, blinking is detected by discriminating between blink and non-blink (body movement such as the head) from the output waveform of the Doppler sensor without special circuit design. In a preferred embodiment, a calibration process tailored to the individual is performed to improve detection accuracy.

In one aspect of the invention, a blink detection system is provided. The blink detection system
A Doppler sensor,
A frequency analysis unit that generates a power spectrum of the Doppler signal by performing frequency analysis on the output of the Doppler sensor;
Smoothing the power spectrum and separating it into a plurality of discontinuous peak waveforms to generate a feature amount waveform; and
Based on the feature amount of the feature amount waveform, a blink identifying unit that identifies a peak waveform representing blink among the plurality of peak waveforms;
Have

In another aspect of the present invention, a blink detection method is provided. The blink detection method is
The Doppler sensor receives the reflected wave from the subject and generates a Doppler signal.
Frequency analysis is performed on the Doppler signal to generate a power spectrum of the Doppler signal,
Smoothing the power spectrum and separating it into a plurality of discontinuous peak waveforms to generate a feature waveform;
Based on the feature quantity of the feature quantity waveform, a peak waveform representing blinking is identified from the plurality of peak waveforms.

  The blink can be detected by analyzing the signal waveform without depending on the circuit design. Further, it is possible to distinguish between blinks and non-blinks, and to reduce the influence of individual differences.

It is a flowchart of the blink detection method of an embodiment. It is a figure which shows the acquisition example of the output data of a Doppler sensor. It is a figure which shows the addition process of the energy component based on a FFT spectrogram. It is a figure which shows the smoothing of the waveform in feature-value extraction. It is a figure which shows the noise removal and waveform separation in feature-value extraction. It is a figure which shows the peak detection of blink. It is a figure for demonstrating the influence of an individual difference. It is a figure for demonstrating identification of blink and non-blink. It is a figure which shows the example of a detection result. It is a schematic structure figure of the blink detection system of an embodiment. It is a figure which shows the positional relationship of a test subject and a sensor. It is a figure which shows the detection result of the blink sensor of embodiment.

  FIG. 1 is a flowchart of blink detection processing according to the embodiment. First, a Doppler signal is acquired using a Doppler sensor as a blink detection sensor (S11). The Doppler sensor detects a movement of an observation target by observing a frequency shift between a transmission signal and a reception signal due to the Doppler effect. In the embodiment, an unmodulated continuous wave (CW) is used as a transmission wave. A signal reflected at or near the eyelid of a subject working on an image display terminal (VDT: Visual Display Terminal) is received, and a Doppler signal (raw data) is obtained by multiplying the received signal by the frequency of the transmission signal.

  FIG. 2 shows an example of the acquired raw data. A Doppler signal representing the frequency shift between the transmitted signal and the received signal is acquired as a function of time. For convenience of explanation, the position on the time axis where the blink actually occurred is indicated by a triangular mark. As can be seen from FIG. 2, not all peaks represent blinks. This is because it includes operations other than blinking (non-blinking) such as shaking the head and changing the direction of the face. In the embodiment, the method described below is used to distinguish between blinks and non-blinks and detect blinks easily and accurately.

  Next, frequency analysis is performed on the data acquired in S11. Specifically, the acquired Doppler data is subjected to wavelet transform to remove noise (S12), and fast Fourier transform (FFT) is applied (S13). The energy component of the power spectrum obtained by FFT is added in the vertical axis direction for each time bin (S14). The wavelet transform of S12 is not essential, but it is desirable to apply it in order to improve blink detection accuracy. The FFT spectrogram acquired in S13 can also be used for analysis of large body movements, but in the embodiment, blinking and non-blink can be identified in a later process by performing energy addition for each time bin in S14. Generate frequency analysis data for feature analysis.

  3A shows an example of the FFT spectrogram acquired in S13, and FIG. 3B shows an example of energy addition for each time bin in S14. The sum of the FFT spectrograms in the vertical axis direction for each time bin is defined as “Psum”. The FFT spectrogram represents plus or minus fluctuations centered on zero, whereas Psum represents the magnitude of the energy component as a function of time.

  Next, in S15, a feature amount is extracted from the information of Psum. As the feature quantity extraction step, the waveform is smoothed and the peak waveform is separated for Psum. FIG. 4 shows an example of waveform smoothing. FIG. 4A is a partial extract of FIG. 3B and represents Psum as a function of time. FIG. 4B shows a waveform after the smoothing process is performed on the waveform of FIG. Although the waveform in FIG. 4A has a large number of peaks, a smooth waveform can be obtained by smoothing.

  Smoothing can be performed by any method. In the embodiment, the peak-to-peak is detected by setting the minimum distance between peaks to three time bins, and the waveform of FIG. 4B is obtained by linear interpolation. By setting the minimum distance between peaks to three time bins, peaks that occur within three time bins from a certain peak are eliminated. This prevents a difference in the number of peaks in the time interval from causing an error. If the minimum distance between peaks is set too large, blinking with a short time width cannot be detected. If the minimum distance between peaks is set too small, the number of peaks cannot be reduced, and a smoothing effect cannot be obtained. The three time bins are values empirically selected for blink detection, and can be appropriately adjusted according to measurement conditions, individual differences, and the environment.

  By removing noise from the peak waveform smoothed in FIG. 4B, the peak waveform can be separated as shown in FIG. This separated peak waveform is a waveform representing a feature amount. FIG. 6 shows detection of peaks for each waveform separated in FIG. As shown in FIG. 5, in the smoothed and separated feature amount waveform, it is expected that there is one peak for each blink. When a plurality of peaks appear in one waveform, there is a high probability of non-blinks such as shaking the head or changing the direction of the face, not blinking.

  In parallel with S15 or in random order, calibration data for analyzing the feature quantity with higher accuracy is acquired (S16). The calibration data is reference data relating to blinking for each subject. By using the reference data, it is possible to reduce the influence of individual differences in blink detection.

  FIG. 7 is a diagram for explaining the influence of individual differences. As shown in FIG. 7A and FIG. 7B, the feature amount of blinking varies depending on the positional relationship with the Doppler sensor and individual characteristics. The subject A in FIG. 7 (A) has a smaller amplitude of Psum and the width of the waveform than the subject in FIG. 7 (B). Therefore, for each subject, before the start of work in the VDT, blink data serving as a reference is acquired, and information regarding the state of shaking of the body is acquired. The reference data desirably includes a measurement result of a plurality of blinks, and an average of a plurality of blink measurements may be used as the reference data. For example, a Doppler waveform for 5 to 10 blinks may be acquired in advance before work, and the threshold for blink detection of the subject may be determined from Psum in FIG. Alternatively, the threshold for blink detection may be set after performing the smoothing process of FIG. 4B and the waveform separation process of FIG. Further, the separation waveform itself acquired in FIG. 5 may be used as calibration data for blink identification.

  Finally, blinking is detected in S17. The blink detection is performed using the reference data acquired in S16 based on the feature amount analysis result in S15. The blink detection includes a process of discriminating between blinks and non-blinks.

  FIG. 8A shows an example of blink identification based on a threshold value. Since the height and width of the blink energy addition waveform Psum vary depending on the individual, when blinking is detected, at least one threshold of peak height and peak width is used for each subject to distinguish between blinking and non-blinking. For example, a peak waveform having a height or width equal to or smaller than a threshold value is detected as a blink, and a peak waveform exceeding the threshold value is detected as a non-blink. The effect of individual differences can be reduced by blink identification (calibration process) using this threshold value.

  FIG. 8B shows an example of blink identification by waveform classification. As described above, when waveform separation is performed as shown in FIG. 5 in the feature amount analysis step (S15), it is expected that one peak is detected for each blink. Therefore, when two or more peaks are included in the separated waveform, it is reasonable to determine that the blink is not blinking. By blink identification (calibration process) based on the waveform classification, it is possible to distinguish between blink and body movement other than blink (non-blink).

  FIG. 9 is a diagram illustrating a detection result obtained by the blink detection method according to the embodiment. The white triangle in the figure indicates the time when blinking was actually performed. The black inverted triangle in the figure indicates the blink position detected by the blink detection method of the embodiment. Circles in the figure indicate peaks detected as non-blinks by the blink detection method of the embodiment. As can be seen from FIG. 9, according to the blink detection method of the embodiment, it is possible to detect blinks with high accuracy in distinction from body movements other than blinks. Further, when the threshold value calibration of FIG. 8A is further combined with the waveform classification, the influence of individual differences can be eliminated and the accuracy of blink detection can be further increased.

  FIG. 10 is a schematic configuration diagram of the blink detection system 10 of the embodiment. The blink detection system 10 includes a Doppler sensor 11 and a detection processing unit 15. The detection processing unit 15 includes a frequency analysis unit 16, a feature amount analysis unit 17, a blink identification unit 18, and a storage unit 19. The frequency analysis unit 16 has a wavelet calculation function, an FFT calculation function, an FFT spectrogram analysis function, and the like. The feature amount analysis unit 17 performs smoothing processing and waveform separation processing on the output data of the frequency analysis unit 16. The feature amount analysis unit 17 also determines a feature amount (a peak height threshold value, a peak width threshold value, a waveform type, etc.) for blink identification of the subject from the reference data of the subject acquired in advance, and stores it in the storage unit 19. Store. The blink identification unit 18 identifies blinks from the output of the feature amount analysis unit 17 based on the waveform threshold value and / or waveform type for each subject stored in the storage unit 19 and outputs the identification result.

  The output of the blink identification unit 18 can be used as the output of the blink detection system 10 to determine the fatigue level and the consciousness level of the operator and the driver. The Doppler sensor 11 is installed in the vicinity of the work screen or on the handle of the vehicle, and the reference data can be acquired under the initiative of the worker or driver before the work starts or before the operation starts. By acquiring a Doppler signal during work or driving and performing calibration processing using feature amount analysis and reference data, it is possible to measure the degree of fatigue and the like in a non-contact manner without imposing a burden on the worker or the driver. The identification result of the blink identification unit 18 may be output as the number of blinks per unit time, and an alarm may be generated when the number of blinks falls below a certain value.

  FIG. 11 is a diagram illustrating a positional relationship of a characteristic evaluation experiment of the blink detection method of the embodiment. The subject sits on a chair and performs VDT work toward the desktop 5 on the desk. The distance between the subject's eyes and the screen of the desktop 5 is 50 cm, and the height of the subject's eyes from the desk is 50 cm. The Doppler sensor 11 is arranged on the desk at a position closer to the subject by 15 cm than the desktop 5, and the sensor surface is inclined toward the subject's eye at an angle of 55 ° from the surface of the desk. The transmission frequency of the Doppler sensor 11 is 24 GHz, and the sampling frequency is 8192 Hz. The output of the Doppler sensor 11 is connected to the input of a personal computer (not shown). The detection processing unit 15 in FIG. 10 is realized by a personal computer, and performs frequency conversion, feature amount analysis, and blink identification.

  In Experiment 1, five subjects (subjects A, B, C, D, and E) play a typing game for one minute toward the desktop 5. In the case of Experiment 1, the movement of each subject's head is small. As Experiment 2, three subjects (subjects A, B, and C) perform VDT work while changing their face freely while blinking naturally. In the case of Experiment 2, each subject has a large amount of head movement.

  In both Experiment 1 and Experiment 2, blink data of 5 times is acquired as reference data for each subject before the start of the experiment, and threshold values of the peak waveform height and width corresponding to each subject are set from the reference data. Thereafter, the experiment is started, blink detection is performed by the method of the embodiment, and the actual blink of the subject is confirmed by the web camera in each of Experiments 1 and 2.

  The blink detection rate of the embodiment is calculated as a recall rate (Recall). The recall is expressed by the following formula.

Reproducibility = (number of blinks correctly detected / number of actual blinks) × 100
FIG. 12 is a diagram showing experimental results. 12A shows the blink detection result of Experiment 1, and FIG. 12B shows the blink detection result of Experiment 2.

  In Experiment 1, blinks are detected with an average of 95% accuracy among different subjects by detecting peaks from the separated waveform (see FIG. 5) using the reference data (calibration data) of each subject. Even in a situation where there is a lot of head movement as in Experiment 2, 80% blink detection accuracy is achieved. Therefore, blinking is detected with high accuracy during VDT work or driving with little head movement, and the fatigue level of the worker or driver can be estimated based on the detection result.

DESCRIPTION OF SYMBOLS 10 Blink detection system 11 Doppler sensor 15 Detection process part 16 Frequency analysis part 17 Feature-value analysis part 18 Blink identification part

Claims (10)

A Doppler sensor,
A frequency analysis unit that generates a power spectrum of the Doppler signal by performing frequency analysis on the output of the Doppler sensor;
Smoothing the power spectrum and separating it into a plurality of discontinuous peak waveforms to generate a feature amount waveform; and
Based on the feature amount of the feature amount waveform, a blink identifying unit that identifies a peak waveform representing blink among the plurality of peak waveforms;
A blink detection system characterized by comprising:   The blink detection system according to claim 1, wherein the blink identification unit identifies a peak waveform having a single peak from the discontinuous peak waveforms as a blink.   The blink detection system according to claim 2, wherein the blink identification unit identifies a peak waveform having two or more peaks from the discontinuous peak waveforms as a non-blink. The feature amount analysis unit determines a threshold value for identifying a peak waveform representing the blink from a reference feature amount waveform generated based on reference data of a subject acquired in advance,
The blink identification unit identifies a peak waveform below the threshold from the discontinuous peak waveform as a peak waveform representing the blink,
The blink detection system according to claim 1. The frequency analysis unit generates a summed power spectrum by adding a fast Fourier transform spectrum obtained by applying a fast Fourier transform to the Doppler signal in a vertical axis direction for each time bin,
5. The blink detection system according to claim 1, wherein the feature amount analysis unit generates the feature amount waveform by smoothing the added power spectrum. The Doppler sensor receives the reflected wave from the subject and generates a Doppler signal.
Frequency analysis is performed on the Doppler signal to generate a power spectrum of the Doppler signal,
Smoothing the power spectrum and separating it into a plurality of discontinuous peak waveforms to generate a feature waveform;
Identifying a peak waveform representing blinking from the plurality of peak waveforms based on the feature amount of the feature amount waveform;
A blink detection method characterized by that.   The blink detection method according to claim 1, wherein a peak waveform having a single peak is identified as a blink from among the discontinuous peak waveforms.   The blink detection method according to claim 7, wherein a peak waveform having two or more peaks is identified as a non-blink from the discontinuous peak waveforms. Generate a reference feature waveform from the reference data of the subject acquired in advance,
Determining a threshold for identifying a peak waveform representing the blink from the reference feature amount waveform;
Further comprising a step,
Identifying a peak waveform below the threshold from the discontinuous peak waveform as a peak waveform representing the blink,
The blink detection method according to claim 6. Applying a fast Fourier transform to the Doppler signal to obtain a fast Fourier transform spectrum,
The fast Fourier transform spectrum is added in the vertical axis direction for each time bin to generate an added power spectrum,
Smoothing the added power spectrum to generate the feature amount waveform;
The blink detection method according to any one of claims 6 to 9.