JP2019030582A - Blink detection system, blink detection method - Google Patents

Abstract

To improve blink detection accuracy.SOLUTION: The blink detection system includes: a Doppler sensor for receiving a reflection wave from a subject to acquire an I signal and a Q signal; a signal calculation unit for calculating a Doppler signal, an amplitude signal, and a phase signal on the basis of the I signal and the Q signal; a spectrogram calculation unit for calculating a spectrogram from each of the Doppler signal, the amplitude signal, and the phase signal; an integral value calculation unit for integrating, for each time, amplitudes of respective frequencies included in a predetermined frequency band in the spectrogram in each of the Doppler signal, the amplitude signal, and the phase signal to calculate temporal variations in integral values; and a blink detection unit for detecting the blink of the subject on the basis of the temporal variations in the respective integral values.SELECTED DRAWING: Figure 4

Description

  The present invention relates to a blink detection system and a blink detection method.

  In recent years, a blink detection system that detects blinks in a non-contact manner using, for example, a Doppler sensor has been studied in order to grasp a subject's fatigue level, stress, health condition, and the like. In a blink detection system using a Doppler sensor, improving the blink detection accuracy is an important issue, and various proposals have been made for that purpose.

  For example, a method for detecting blinks based on machine learning has been proposed (see, for example, Non-Patent Document 1). In this method, five feature values (maximum voltage value, time width, variance, raw signal variance, raw signal voltage maximum value) are defined for blink signals, and SVM (Support Vector Machine) is one of machine learning. A certain detection accuracy is ensured by discriminating between blinks and non-blinks.

  However, in the method proposed in the past, for example, in an environment where the noise level is high, the face direction is easy to change, and fast body movements including frequency components similar to blinking occur, such as in a car environment. However, sufficient detection accuracy was not obtained.

C, Tamba and T. Ohtsuki, "Learning-based Blink Detection Using a Doppler Sensor," IEICE technical report, vol. 114, no. 418, ASN 2014-123, pp. 97-102. Jan. 2015.

  The present invention has been made in view of the above points, and an object thereof is to improve blink detection accuracy.

  The blink detection system receives a reflected wave from a subject and obtains an I signal and a Q signal, and calculates a Doppler signal, an amplitude signal, and a phase signal based on the I signal and the Q signal. A signal calculation unit; a spectrogram calculation unit for calculating a spectrogram from each of the Doppler signal, the amplitude signal, and the phase signal; and a predetermined frequency of the spectrogram of each of the Doppler signal, the amplitude signal, and the phase signal An integral value calculation unit that integrates the amplitude of each frequency included in the band for each time and calculates a time change of the integral value, and a blink that detects the subject's blink based on the time change of each of the integral values And a detection unit.

  According to the disclosed technique, it is possible to improve blink detection accuracy.

It is a figure which illustrates schematic structure of the blink detection system which concerns on this Embodiment. It is an example of the signal obtained with the Doppler sensor. It is a figure which illustrates the hardware block of the signal processing part which concerns on this Embodiment. It is a figure which illustrates the functional block of the signal processing part which concerns on this Embodiment. It is an example of the flowchart which shows operation | movement of the blink detection system which concerns on this Embodiment. It is an example of the signal obtained from I signal and Q signal. It is an example of the spectrogram calculated from the amplitude signal. It is an example of the integral value of the energy integral on a spectrogram. It is a figure explaining CA-CFAR detection. It is an example of the window comprised from a some bin. It is an example of the matrix Me which makes the energy contained in each bin in a window an element. It is a figure explaining formation of a feature-value vector. It is an example of the data classified as blink and the data classified as non-blink.

  Hereinafter, embodiments will be described with reference to the drawings. In addition, in each drawing, the same code | symbol is attached | subjected to the same component and the overlapping description may be abbreviate | omitted.

  FIG. 1 is a diagram illustrating a schematic configuration of a blink detection system according to the present embodiment. As shown in FIG. 1, the blink detection system 1 includes a Doppler sensor 10 and a signal processing unit 20 as main components.

  The Doppler sensor 10 is a sensor that detects the movement of the observation target (subject) by observing the frequency shift of the transmission signal and the reception signal due to the Doppler effect. In the present embodiment, as an example, a non-modulated continuous wave (CW) is used as a transmission wave.

  The Doppler sensor 10 is disposed in the vicinity of the subject, can receive a signal (reflected wave) reflected in the subject's eyelid or in the vicinity thereof, and can obtain a signal generated by the movement of the observation target. Examples of the test subject include a driver of a vehicle and a worker working on an image display terminal (VDT).

  FIG. 2 is an example of a signal obtained by the Doppler sensor. The signal shown in FIG. 2 is obtained by obtaining a signal representing a frequency shift between the transmission signal and the reception signal as a function of time. The I signal which is an in-phase component of the transmission signal and the quadrature phase ( Quadrature) component Q signal.

  Movements of the body surface such as blinking, heartbeat and breathing, and body movement (movement by the body) can be observed by the Doppler sensor 10. When performing blink detection, it is preferable to appropriately remove noise using a high-pass filter, a band-pass filter, or the like.

  Returning to FIG. 1, the signal processing unit 20 detects the blink of the subject based on the output signal of the Doppler sensor 10. The signal processing unit 20 appropriately uses the I signal and Q signal of the signal received by the Doppler sensor 10 as they are, or generates various signals (amplitude, phase, their integrated values, etc.) based on the I signal and the Q signal. You can do it.

  FIG. 3 is a diagram illustrating a hardware block of the signal processing unit according to the present embodiment. Referring to FIG. 3, the signal processing unit 20 includes a CPU 21, a ROM 22, a RAM 23, an I / F 24, and a bus line 25. The CPU 21, ROM 22, RAM 23, and I / F 24 are connected to each other via a bus line 25.

  The CPU 21 controls each function of the signal processing unit 20. The ROM 22 serving as a storage unit stores a program executed by the CPU 21 to control each function of the signal processing unit 20 and various types of information. The RAM 23 serving as storage means is used as a work area for the CPU 21. The RAM 23 can temporarily store predetermined information. The I / F 24 is an interface for connecting the blink detection system 1 to other devices. The blink detection system 1 may be connected to an external network or the like via the I / F 24.

  However, part or all of the signal processing unit 20 may be realized only by hardware. Examples of hardware include an application specific integrated circuit (ASIC), a digital signal processor (DSP), and a field programmable gate array (FPGA). The signal processing unit 20 may be physically configured by a plurality of devices.

  FIG. 4 is a diagram illustrating functional blocks of the signal processing unit according to the present embodiment. Referring to FIG. 4, the signal processing unit 20 includes, as functional blocks, a signal calculation unit 201, a spectrogram calculation unit 202, an integral value calculation unit 203, a product calculation unit 204, a signal detection unit 205, and a feature amount. A vector forming unit 206 and a blink determination unit 207 are provided. Specific functions of each functional block will be described later in the description of FIG.

  FIG. 5 is an example of a flowchart showing the operation of the blink detection system according to the present embodiment. The blink detection method according to the present embodiment will be described with reference to FIG. 5 and other drawings as appropriate.

  First, in step S <b> 11, the signal calculation unit 201 receives a reflected wave from the subject by the Doppler sensor 10 and acquires a signal. The signal acquired here is composed of, for example, an I signal and a Q signal as shown in FIG.

  Next, in step S12, noise components of the I signal and Q signal received in step S11 are removed. The removal of the noise component can be performed by hardware by inserting a band-pass filter between the Doppler sensor 10 and the signal processing unit 20, for example. Alternatively, the output signal of the Doppler sensor 10 may be directly input to the signal processing unit 20 and digital signal processing (digital filter or the like) may be performed in the signal processing unit 20. The passband of the bandpass filter can be set to about 4 Hz to 30 Hz, for example.

  Next, in step S13, the signal calculation unit 201 calculates Doppler signal = I (t) + jQ (t) from the I signal and Q signal from which noise components have been removed by the band pass filter. For example, the Doppler signal shown in FIG. 6A can be obtained from the signal shown in FIG.

Next, in step S < ^b > 14, the signal calculation unit 201 calculates an amplitude signal = √ (I (t) ² + Q (t) ² ) from the I signal and Q signal from which noise components have been removed by the band pass filter. For example, the amplitude signal shown in FIG. 6B can be obtained from the signal shown in FIG.

Next, in step S15, the signal calculation unit 201 calculates phase signal = tan ⁻¹ (Q (t) / I (t)) from the I signal and the Q signal from which the noise component has been removed by the band pass filter. For example, the phase signal shown in FIG. 6C can be obtained from the signal shown in FIG. Note that the processing order of S13 to S15 may be arbitrary.

  Next, in step S16, the spectrogram calculation unit 202 calculates a spectrogram from each of the Doppler signal calculated in step S13, the amplitude signal calculated in step S14, and the phase signal calculated in step S15. That is, three types of spectrograms are calculated.

  A spectrogram is a representation of the frequency distribution of energy contained in a signal on a plane with time on the horizontal axis and frequency on the vertical axis. For example, a short-time Fourier transform (STFT) ). The short-time Fourier transform can be performed, for example, under the conditions of a window size of 512 ms and an overlap of 5 ms.

  FIG. 7 is an example of a spectrogram calculated from the amplitude signal in step S16, and displays time, Doppler frequency, and signal component strength in gray scale. Although the spectrograms of the Doppler signal and the phase signal are not shown, the time, Doppler frequency, and signal component strength are displayed as in FIG. However, the spectrogram of the Doppler signal includes a positive Doppler frequency corresponding to the proximity motion and a negative Doppler frequency corresponding to the separation motion.

  Next, in step S17, the integral value calculation unit 203 performs energy integration on the spectrograms of the Doppler signal, the amplitude signal, and the phase signal. The integration ranges of the Doppler signal, the amplitude signal, and the phase signal can be set to, for example, −30 Hz to −4 Hz, 4 Hz to 30 Hz, 4 Hz to 30 Hz, and 4 Hz to 30 Hz.

  That is, the integral value calculation unit 203 integrates the amplitude of each frequency included in −30 Hz to −4 Hz and 4 Hz to 30 Hz for each time in the spectrogram of the Doppler signal, and calculates the time change of the integral value of the amplitude. For example, the integral value Ed shown in FIG.

  Similarly, the integral value calculation unit 203 integrates the amplitude of each frequency included in 4 Hz to 30 Hz for each time in the spectrogram of the amplitude signal, and calculates the time change of the integral value of the amplitude. For example, an integral value Ea shown in FIG. 8B is obtained.

  Similarly, the integral value calculation unit 203 integrates the amplitude of each frequency included in 4 Hz to 30 Hz for each time in the spectrogram of the phase signal, and calculates the time change of the integral value of the amplitude. For example, an integral value Ep shown in FIG. 8C is obtained.

  Next, in step S18, the product calculation unit 204 calculates the time change of the product by multiplying the time changes of the three integral values Ed, Ea, and Ep for each time. For example, the product Ec (= Ed × Ea × Ep) shown in FIG. 8D is obtained. In FIGS. 8A to 8D, for the sake of convenience, the time at which the blink actually occurred is indicated by a broken line.

Next, in step S <b> 19, the signal detection unit 205 detects a time when blinking may occur based on CA-CFAR (Cell Averaging Constant False Alarm Rate) processing. Specifically, the signal detection unit 205 determines the threshold value TH ₁ in the CA-CFAR process, compares the product Ec calculated in step S18 with the threshold value TH _1, and detects a signal at a time that is equal to or greater than the threshold value TH _1. .

Here, in order to determine the threshold TH ₁ in the CA-CFAR process, as shown in FIG. 9, the product Ec is divided into a plurality of cells that are continuous in the time axis direction, and the cell of interest (Cut: Cell under test) is divided. decide.

Next, to determine the threshold value TH ₁ based on the amplitude values before and after the cell not adjacent to the Cut. Specifically, a predetermined number of cells immediately preceding the Cut and guard cell G _F, a predetermined number of cells immediately after the Cut and guard cell G _R, further, guard cell G _F front cell F of a predetermined number of cells immediately before, guard cell a predetermined number of cells immediately after the G _R and Riaseru R. The duration of the blink signal observed by the Doppler sensor 10 is empirically found to be about 0.2 seconds. Therefore, the cell length of each cell in FIG. 9 is preferably set to about 0.2 seconds.

In FIG. 9, the guard cell G _F , the guard cell G _R , the front cell F, and the rear cell R are each composed of three cells, but this is an example, and the guard cell G _F , the guard cell G _R , the front cell F, The number of cells constituting the rear cell R can be determined as appropriate.

Next, an average value A _AVE of the amplitude value of the front cell F and the amplitude value of the rear cell R is calculated, and a value obtained by multiplying the average value A _AVE by a coefficient α is set as a threshold value TH ₁ . That is, the threshold TH ₁ = A _AVE × α. Incidentally, guard cell _{G F} and _{G R} are not used for calculating the threshold value TH _1. This is to prevent the Cut itself is a cell of interest can affect the threshold value TH _1.

When the threshold value TH ₁ is too low, detected FP (False Positive) erroneously blink much noise is increased. On the other hand, if the threshold TH ₁ is too high, the more the detection omission blinking FN (False Negative) increases. Therefore, it is necessary to set the coefficient α to an appropriate value, but it has been found by experiments conducted by the inventors that FP and FN can be reduced by setting α to about 3.4.

When the threshold value TH ₁ is determined as described above, the signal detection unit 205 detects a threshold value TH ₁ or more signal components in Cut. By comparing the product Ec with a threshold value TH ₁ by repeating the same detection while continuing moving the position of the Cut, detecting the time of the signal is the threshold value TH ₁ or more.

  In addition, the above steps S17-S19 are prior detection steps, and detect the time and signal at which blinking may occur. On the other hand, the following steps S20 and S21 are classification steps, and the time and signal at which blinking may occur are changed to the time and signal at which blinking occurs and the time and signal at which blinking does not occur (non-blink). Classify.

Specifically, in step S20, the feature quantity vector forming unit 206, spectrogram calculated from the amplitude signal in step S16 (e.g., 7) in, the signal including a time at which the threshold value TH ₁ or more signal components is detected Based on this, a feature vector is formed. The reason for using the spectrogram calculated from the amplitude signal is that the spectrogram calculated from the amplitude signal is more blinking and non-blinking than the spectrogram calculated from the Doppler signal or the phase signal. This is because the difference appears remarkably.

In order to form a feature vector, first, the feature vector forming unit 206 is a window composed of a plurality of bins (unit regions) arranged vertically and horizontally within a predetermined frequency band of a spectrogram calculated from an amplitude signal. Is set around the time when a signal component equal to or greater than the threshold TH ₁ is detected. Then, the feature vector forming unit 206 generates a matrix Me whose elements are the energy included in each bin in the set window.

  For example, if a window having a width of 500 ms is set in a frequency range of 14 Hz to 30 Hz and one bin is set to 2 Hz in length and 5 ms in width, as shown in FIG. 10, an 8 × 100 bin is formed in the set window. A matrix Me (8 × 100) whose elements are energy contained in each bin in the window can be generated. FIG. 11 shows a specific example of the matrix Me.

  The blinking frequency is about 4 Hz to 30 Hz, but the body shakes other than blinking easily occur in the car, and it easily appears in a frequency band of about 4 Hz to 14 Hz. That is, since the frequency band from 4 Hz to 14 Hz has a lot of noise, the window is set in a frequency range of 14 Hz to 30 Hz excluding this frequency band.

  The blinking time is about 200 ms to 1200 ms, but considering the spread of energy, the blinking can be sufficiently captured if the window width is 500 ms. The bin width of 2 Hz is determined from the short-time Fourier transform conditions (for example, window size 512 ms, overlap 5 ms).

  Energy generation differs between blinking and non-blinking. On the other hand, how the energy is generated in the time and frequency domains is similar in every blink. Therefore, here, the feature amount spectrum is extracted by the covariance matrix while paying attention to the energy generation on the spectrogram.

Specifically, the feature vector forming unit 206 calculates a covariance matrix X of the matrix Me. FIG. 12A shows a specific example of the covariance matrix X. Then, the covariance elements of the covariance matrix X, arranged in a row along the line L ₁ in FIG. 12 (a). For example, as shown in FIG. 12B, [1.2948, 0.6550,..., 0.0244].

  The diagonal component of the covariance matrix X is not used because it is self-dispersing and has no meaning. Since the covariance element is symmetric with respect to the diagonal component of the covariance matrix X, only the covariance element on the upper right side of the diagonal component is used here.

Next, the feature vector forming unit 206 also calculates a similarly covariance matrix with respect transposed matrix Me ^T matrix Me, arranging the covariance elements of the covariance matrix of the transposed matrix Me ^T in a row. For example, as shown in FIG. 12C, [3.2315, 1.0102,.

Next, the feature vector forming section 206, based on the covariance elements of the covariance matrix of the covariance elements of the covariance matrix of the matrix Me and transposed matrix Me ^T, to form a feature vector. Specifically, the covariance elements arranged in a line in FIGS. 12B and 12C are combined to form a feature vector. For example, as shown in FIG. 12D, [1.2948, 0.6550,..., 0.0244, 3.2315, 1.0102,.

  It should be noted that the feature vector is formed at all times at which the blink detected in step S19 may occur on the spectrogram of the amplitude signal.

  Next, in step S21, the blink determination unit 207 detects a subject's blink by applying a machine learning algorithm to the feature amount vector formed in step S20. Specifically, for example, a random forest is used as a machine learning algorithm, and blinks and non-blinks are classified by the random forest.

  The random forest is a classifier that uses a plurality of decision trees and outputs the mode value determined by each decision tree as a final output. Here, as an example, the number of decision trees is 50, and the maximum number of divisions (depth of the decision tree) is 20. FIG. 13 shows an example of data classified as blink and data classified as non-blink. As shown in FIG. 13, feature vectors differ greatly between blinks and non-blinks, and therefore, accurate classification using a random forest is possible.

  Thus, in the blink detection system 1, the Doppler signal, the amplitude signal, and the phase signal are calculated based on the I signal and the Q signal acquired by the Doppler sensor 10, and the spectrogram is obtained from each of the Doppler signal, the amplitude signal, and the phase signal. Is calculated. Then, the amplitude of each frequency included in a predetermined frequency band of each spectrogram is integrated at each time to calculate the temporal changes of the integrated values Ed, Ea, and Ep, and each integrated value Ed, Ea, And the time change of Ep is calculated for every time, and the time change of the product Ec is calculated.

  The time change of the product Ec is calculated by multiplying the time changes of the integral values Ed, Ea, and Ep every time, so that the noise level of the product Ec is lower than the noise levels of the integral values Ed, Ea, and Ep. . For example, FP decreases near 27 to 28 seconds and 37 seconds in FIG. 8D, and FN decreases near 29 seconds in FIG. 8D. Therefore, the blink detection accuracy can be improved by performing blink detection based on the product Ec.

However, the blink determination unit 207 may detect the blink of the subject based on temporal changes of the three integrated values Ed, Ea, and Ep according to the required detection accuracy. Further, blink determination unit 207, depending on the required detection accuracy, may detect blinking of the subject based on the time change of the product Ec, it detects the blink of the subject based on a threshold TH ₁ or more signal components May be.

[Example]
In the example, blink detection was performed using the blink detection system 1 according to the present embodiment. Specifically, the blink detection system 1 is used, and a driver who is a subject when driving on a general road and stopping at a vehicle on which a total of four people including a driver and three occupants rides 81 times (when driving: 64 times). At the time of stopping: 17 times) I tried to detect the blink that I went. However, it was excluded within 1 second before and after the stop. The Doppler sensor 10 was placed on the dashboard so that the transmitted / received signal was not shielded by the steering wheel while the subject was seated in the driver's seat, and was separated by about 40-50 cm in front of the subject's face.

  Table 1 shows the specifications of the experiment, and Table 2 shows the results of the experiment. As a comparative example, a method of obtaining an amplitude signal from an I signal and a Q signal, selecting a blink candidate by performing CA-CFAR processing on the obtained amplitude signal, and detecting a blink by removing body movement from the selected blink candidate Table 2 also shows the results when using.

As shown in Table 2, in the embodiment, the Recall (detection rate) during traveling is 72%, the Precision (adaptation rate) during traveling is 70%, the Recall (detection rate) when stopped is 82%, and when the vehicle is stopped Precision (conformity) was 78%.

  On the other hand, in the comparative example, the Recall (detection rate) when traveling is 20%, the Precision (adaptation rate) when traveling is 33%, the Recall (detection rate) when stopped is 35%, and the Precision (adaptation rate) when stopping is Was 43%.

  As described above, in the example, it was confirmed that both Recall (detection rate) and Precision (matching rate) were significantly improved as compared with the comparative example, and the accuracy of blink detection could be improved.

  In the embodiment, the detection accuracy is improved even when the vehicle is stopped compared to the comparative example. However, this is because the engine is started even when the vehicle is stopped, and noise generated due to fine shaking of the body accompanying the vibration of the engine. This is considered to be because it could be removed in the example but not in the comparative example. Thus, the example is an algorithm that is more resistant to noise than the comparative example.

  The preferred embodiments and the like have been described in detail above, but the present invention is not limited to the above-described embodiments and the like, and various modifications can be made to the above-described embodiments and the like without departing from the scope described in the claims. Variations and substitutions can be added.

DESCRIPTION OF SYMBOLS 1 Blink detection system 10 Doppler sensor 20 Signal processing part 21 CPU
22 ROM
23 RAM
24 I / F
25 bus line 201 signal calculation unit 202 spectrogram calculation unit 203 integral value calculation unit 204 product calculation unit 205 signal detection unit 206 feature quantity vector formation unit 207 blink determination unit

Claims (9)

A Doppler sensor that receives a reflected wave from a subject and obtains an I signal and a Q signal;
A signal calculation unit that calculates a Doppler signal, an amplitude signal, and a phase signal based on the I signal and the Q signal;
A spectrogram calculating unit that calculates a spectrogram from each of the Doppler signal, the amplitude signal, and the phase signal;
An integration value calculation unit that integrates the amplitude of each frequency included in a predetermined frequency band of the spectrogram of each of the Doppler signal, the amplitude signal, and the phase signal for each time, and calculates a time change of the integration value; ,
A blink detection system comprising: a blink detection unit that detects blinks of the subject based on a time change of each of the integral values. A product calculation unit that calculates the time change of the product by multiplying the time change of each of the integral values for each time,
The blink detection system according to claim 1, wherein the blink determination unit detects the blink of the subject based on a time change of the product. The time change of the product is divided into a plurality of cells that are continuous in the time axis direction, a threshold value is determined based on amplitude values of preceding and following cells that are not adjacent to the cell of interest, and a signal component equal to or greater than the threshold value in the cell of interest A signal detection unit for detecting
The blink detection system according to claim 2, wherein the blink determination unit detects the blink of the subject based on a signal component equal to or greater than the threshold. In the spectrogram calculated from the amplitude signal, a feature amount vector forming unit that forms a feature amount vector based on a signal including a time when a signal component equal to or greater than the threshold is detected;
The blink detection system according to claim 3, wherein the blink determination unit detects the blink of the subject based on the feature amount vector. The feature vector forming unit includes:
Within a predetermined frequency band of the spectrogram calculated from the amplitude signal, a window composed of a plurality of unit regions arranged vertically and horizontally is set around the time when a signal component equal to or greater than the threshold is detected,
Generating a matrix having energy included in each of the unit regions as an element and a transpose matrix of the matrix;
Calculating a covariance matrix of each of the matrix and the transpose matrix;
The blink detection system according to claim 4, wherein the feature amount vector is formed based on a covariance element of each of the covariance matrices.   The blink detection system according to claim 5, wherein the blink determination unit detects a blink of the subject by applying a machine learning algorithm to the feature vector.   The blink detection system according to claim 6, wherein the machine learning algorithm is a random forest.   The blink detection system according to any one of claims 5 to 7, wherein the predetermined frequency band is 14 Hz or more and 30 Hz or less. A signal acquisition step of receiving a reflected wave from a subject with a Doppler sensor and acquiring an I signal and a Q signal;
A signal calculating step of calculating a Doppler signal, an amplitude signal, and a phase signal based on the I signal and the Q signal;
A spectrogram calculating step of calculating a spectrogram from each of the Doppler signal, the amplitude signal, and the phase signal;
An integration value calculation step of integrating the amplitude of each frequency included in a predetermined frequency band of the spectrogram of each of the Doppler signal, the amplitude signal, and the phase signal at each time, and calculating a time change of the integration value; ,
A blink detection method comprising: a blink detection step of detecting blinks of the subject based on a time change of each of the integral values.