CN109101881A - A kind of real-time blink detection method based on multiple dimensioned timing image - Google Patents

Abstract

The invention discloses a real-time blink detection method based on multi-scale time-series images, comprising: positioning the first frame of images in the time-series images to obtain the positions of both eyes; using it to extract human eye images from the first frame of images to obtain human eye Template; use the human eye template to initialize the human eye tracker, use the updated human eye tracker to track the time-series images, obtain the time-series human eye images, and update the human eye tracker to extract the artificial descriptors of the processed time-series human eye images, Then extract the differ feature of the artificial descriptor, and concatenate it with the artificial descriptor to obtain the human eye feature. Encode the human eye features into a blink behavior feature heat map according to the time sequence; then input the blink behavior feature heat map into the LSTM network line by line to obtain multiple hidden states; finally, multiple hidden states are concatenated to obtain multi-scale time series features. Blink detection, to determine whether the time-series images contain blinking behavior. The invention improves the accuracy and stability of blink detection under unconstrained conditions.

Description

A real-time blink detection method based on multi-scale time-series images

technical field

The invention belongs to the technical field of digital image recognition, and more specifically relates to a real-time blink detection method based on multi-scale time-series images.

Background technique

With the popularization of various intelligent application devices, it should become a good way of human interaction to reflect the current state of the target by observing the blinking behavior of people, so whether it is living body verification, fatigue driving detection, polygraph detection, etc. There is a great demand.

The current main blink detection algorithms are mainly divided into the following two types: one is to extract the traditional features of a single frame (LBP, HOG, etc.), and then use the classifier (SVM, Adboost, etc.) state; the other is to perform blink detection based on some heuristic rules, such as hough transform to detect pupils, etc.

The above method has the following shortcomings. For the first method, since blinking is a sequential behavior, the characteristics of a single picture can not explain the behavior of the eye state, and the detection algorithm of a single picture has a higher success rate under natural conditions. Low, poor robustness. Another algorithm is eye blink detection based on some heuristic rules. This algorithm has insufficient model ability under unconstrained conditions, and is prone to misjudgment, and rules such as detecting changes in the center of connected domains will appear regular under natural conditions. Crashes, undetectable phenomena.

Contents of the invention

In view of the above defects or improvement needs of the prior art, the present invention provides a real-time blink detection method based on multi-scale time-series images. The blink behavior feature map is obtained by encoding, and then different timing scales are considered to improve the accuracy and stability of blink detection under natural conditions. Therefore, the technical problems of low correct rate and poor stability in the prior art are solved.

In order to achieve the above object, the present invention provides a real-time blink detection method based on multi-scale time-series images, including:

(1) Establishing a database containing time-series images of human eyes, and positioning the first frame image of the time-series images in the database to obtain the positions of the eyes;

(2) Extract the human eye image from the first frame image by using the position of both eyes to obtain the human eye template;

(3) Utilize the human eye template to initialize the human eye tracker, utilize the initialized human eye tracker to track the follow-up frame images of the first frame image in the sequential images and update the human eye tracker;

(4) Utilize the updated human eye tracker to track the time-series images to obtain the time-series human eye images, and sequentially perform grayscale processing and image equalization processing on the time-series human eye images to obtain the processed time-series human eye images;

(5) Extract the artificial descriptor of the processed time-series human eye image, extract the difference feature of the artificial descriptor, and then concatenate the difference feature and the artificial descriptor to obtain the human eye feature, and then encode the human eye feature according to the time-series information into a A heat map of blinking behavior characteristics;

(6) Input the blink behavior feature heat map into the LSTM network line by line to obtain multiple hidden states;

(7) After concatenating multiple hidden states to obtain multi-scale time series features, perform eye blink detection to determine whether the time series images contain eye blink behavior.

Further, step (1) includes:

The video database containing human eyes is used to establish a database containing time-series images of human eyes, and the face alignment algorithm is used to locate the first frame of the time-series images in the database to obtain the positions of the eyes.

Further, step (2) includes:

(2-1) Utilize the position of both eyes to obtain the distance between the two eyes, determine the width and height of the human eye area based on the distance between the two eyes, and obtain the human eye area;

(2-2) According to the human eye area, extract the human eye image from the first frame image, and then judge whether to reduce the human eye image by K times according to the size of the human eye image after grayscale processing of the human eye image, K≥ 0, and finally use the obtained human eye image as the human eye template.

Further, step (3) includes:

(3-1) Initialize the human eye tracker using the human eye template;

(3-2) Use the initialized human eye tracker to track the subsequent frame images of the first frame image in the time series images, obtain the human eye position Pos of the subsequent frame images, and feed back the confidence C of each tracking;

(3-3) Set the tracking score threshold T, if C>T, use the human eye position Pos to extract a new human eye image, and obtain a new human eye template, if C≤T, then perform step (1);

(3-4) The tracker after initialization of the new eye template and the eye tracker after initialization are weighted and fused, and the eye tracker is updated.

Further, the differ feature is: a difference feature obtained by subtracting the artificial descriptor of each frame extracted from the processed time-series human eye image from the corresponding bit of the artificial descriptor of the previous frame.

Further, step (6) includes:

(6-1) Establish an LSTM network with n cells, each cell contains h hidden layers, the input of the LSTM network does not use the dropout technology that eliminates overfitting, and the cell state of the LSTM network is transmitted between cells using a range of d dropout technology, d<1;

(6-2) Train the LSTM network by using the sample heat map containing the label to obtain multiple sample hidden states, and then train the sphereface model by concatenating the multiple sample hidden states to obtain the multi-scale time series features of the sample, and obtain the parameters and training of the sphereface model A good LSTM network;

(6-3) Input the heat map of eye blinking behavior features into the trained LSTM network line by line to obtain multiple hidden states.

Further, step (7) includes:

Multiple hidden states are concatenated to obtain multi-scale time-series features and then input to the sphereface model for eye blink detection to determine whether the time-series images contain eye blink behavior.

Generally speaking, compared with the prior art, the above technical solutions conceived by the present invention can achieve the following beneficial effects:

(1) Firstly, the present invention realizes real-time computing, and then concatenates the differ features and artificial descriptors to obtain human eye features, which has stronger semantic information; secondly, the blink behavior is encoded into a heat map of blink behavior characteristics, which is more It is beneficial to the intuitive observation and the processing of the algorithm in the image field. Finally, the LSTM network is used to obtain multiple hidden states to adapt to the problem of different blinking speeds of different people. Compared with the traditional eye blink detection algorithm, the present invention has better performance in speed and accuracy in the time-series eye blink database under unconstrained conditions.

(2) Blinking is a timing-based behavior. The introduction of timing information is of great significance for identifying the essential characteristics of blinking. Based on the prior art, there is no blink database based on timing information. The present invention utilizes a video database containing human eyes to establish a database containing A time-series image database of the human eye. The positioning algorithm can accurately determine the position of the human eye. Since it is in the initialization step, its time-consuming problem will not affect the real-time performance of the program. The invention is based on a relatively accurate human eye image obtained by a positioning algorithm, and the initialized human eye tracker can track the human eye area more accurately and quickly, ensuring that the program can obtain satisfactory results in terms of real-time and positioning accuracy.

(3) The present invention utilizes image equalization processing to reduce the influence caused by different light intensities to a certain extent, and enhance the robustness of the program. Because the program is affected by factors such as the environment and positioning accuracy, noise will inevitably be introduced. The present invention uses a differential mechanism to weaken low-frequency noise to a certain extent. LSTM has great advantages in extracting timing information of timing samples, and can extract timing information in timing samples very well.

Description of drawings

1 is a flowchart of a real-time blink detection method based on multi-scale time-series images provided by an embodiment of the present invention;

Fig. 2 (a) is a schematic diagram of time-series images with blinking behavior provided by Embodiment 1 of the present invention;

Fig. 2(b) is a schematic diagram of time-series images without blinking behavior provided by Embodiment 1 of the present invention;

FIG. 3 is a schematic flow chart of a blink detection method provided in Embodiment 1 of the present invention;

FIG. 4 is a schematic flow chart of acquiring human eye features provided by Embodiment 1 of the present invention.

Detailed ways

In order to make the object, technical solution and advantages of the present invention clearer, the present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described here are only used to explain the present invention, not to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not constitute a conflict with each other.

As shown in Figure 1, a real-time blink detection method based on multi-scale time series images, including:

(1) Use the video database containing human eyes to establish a database containing time-series images of human eyes, and use the face alignment algorithm to locate the first frame of the time-series images in the database to obtain the positions of the eyes.

(2) Utilize the position of both eyes to obtain the distance between the two eyes, determine the width and height of the human eye area based on the distance between the two eyes, and obtain the human eye area; according to the human eye area, extract the human eye image from the first frame image, and After grayscale processing of the human eye image, it is judged whether to reduce the human eye image by K times according to the size of the human eye image, K≥0, and finally the obtained human eye image is used as a human eye template.

(3) Use the human eye template to initialize the human eye tracker; use the initialized human eye tracker to track the subsequent frame images of the first frame image in the sequence image, obtain the human eye position Pos of the subsequent frame images, and feed back each tracking Confidence C; set the tracking score threshold T, if C>T, use the human eye position Pos to extract a new human eye image, get a new human eye template, if C≤T, then perform step (1); set the new The tracker after the initialization of the human eye template and the weighted fusion of the initialized human eye tracker are used to update the human eye tracker.

(4) Utilize the updated human eye tracker to track the time-series images to obtain the time-series human eye images, and sequentially perform grayscale processing and image equalization processing on the time-series human eye images to obtain the processed time-series human eye images;

(5) Extract the artificial descriptor of the processed time-series human eye image, extract the difference feature of the artificial descriptor, and then concatenate the difference feature and the artificial descriptor to obtain the human eye feature, and then encode the human eye feature according to the time-series information into a A heat map of blinking behavior features; the differ feature is: the difference feature obtained by subtracting the artificial descriptor of each frame extracted from the processed time-series human eye image from the corresponding bit of the previous frame of artificial descriptor.

(6) Establish an LSTM network with n cells, each cell contains h hidden layers, the input of the LSTM network does not use the dropout technology that eliminates overfitting, and the cell state transmitted between the nodes of the LSTM network uses a dropout with an amplitude of d technology, d<1; train the LSTM network by using the sample heat map with labels to obtain the hidden states of multiple samples, and then train the sphereface model by concatenating the hidden states of multiple samples to obtain the multi-scale time series features of the samples, and obtain the parameters and training of the sphereface model A good LSTM network; input the blink behavior feature heatmap line by line into the trained LSTM network to obtain multiple hidden states.

(7) Multiple hidden states are concatenated to obtain multi-scale time-series features, and then input to the sphereface model for eye blink detection to determine whether the time-series images contain eye blink behavior.

Example 1

Step 1, use the video database containing human eyes to establish a database containing time-series images of human eyes, and use the face alignment algorithm to locate the first frame of images in the database in time-series images to obtain the positions of the eyes, including the following steps:

(1-1) Establish a video database containing human eyes under natural conditions. The video database contains more than 200 time-series images containing human eyes with different frame lengths. Use the video database containing human eyes to establish the image shown in Figure 2(a) Timing images with blinking behavior and timing images without blinking behavior shown in Figure 2(b);

In the embodiment of the present invention, the video database containing human eyes is used to establish more than 370 time-series images containing human eyes under natural conditions and about 400 indoors;

(1-2) As shown in Figure 3, the database image is a time-series image containing human eyes, and the face alignment algorithm is used to locate the first frame of images in the time-series image to obtain the positions of both eyes.

Step 2, make the initial template of the tracking algorithm, including the following sub-steps:

(2-1) Use the position of the eyes to obtain the distance between the eyes, and determine the width and height of the human eye area based on the distance between the eyes, as follows:

in, Indicates the x-coordinate of the right eye, Indicates the x-coordinate of the left eye, Indicates the y coordinate of the right eye, Indicates the y coordinate of the left eye, width indicates the width of the eye area, height indicates the height of the eye area, centering on the position of the eyes, the human eye area is determined according to the width and height of the eye area;

(2-2) According to the human eye area, extract the human eye image and perform grayscale processing,

K＝0.2989×R+0.5870×G+0.1140×B

Among them, K is the grayscale image, R, G, and B are the three channels of the human eye image;

(2-3) According to the size of the human eye image, judge whether to adjust the human eye image to 0.7 times the original size, and finally use the adjusted human eye image as the human eye template.

Step 3, using the human eye template, combined with the KCF (Kernel Correlation Filter) algorithm, to perform real-time tracking and update the tracker, including the following sub-steps:

(3-1) Input the eye template into the eye tracker, and use the eye template to initialize the eye tracker:

tracker=init(image)

Among them, tracker is the human eye tracker obtained after initialization, and image is the human eye template used for initialization;

(3-2) Use the initialized human eye tracker tracker to track subsequent frame images, obtain the human eye position Pos, and feed back the confidence C of each tracking:

[Pos, C] = tracker(image _{0_c} )

Among them, image ₀ is the subsequent frame image, Pos is the position of the human eye corresponding to the image, and C is its confidence level;

(3-3) Set the tracking score threshold to 0.35. If C>0.35, use the human eye position Pos to extract a new human eye image to obtain a new human eye template. If C≤0.35, perform step (1);

(3-4) The tracker after initialization of the new eye template and the eye tracker after initialization are weighted and fused, and the eye tracker is updated:

Among them, image is the current frame image, tracker_new is the tracker after initialization of the new human eye template, image_next is the next frame of the image to be tested, and Pos is the positioning result corresponding to the next frame of the image to be tested.

Step 4, extract the human eye image for preprocessing, including the following steps:

(4-1) For time series images, use the updated human eye tracker to track the time series images, obtain the human eye images in each frame of images, and arrange them in time series to form time series human eye images E ₁ , E ₂ , .. .E _n .

E=image(Pos)

Among them, image is the current frame image, and E is the extracted human eye image;

(4-2) For each frame of image E _i (i=1, 2,...n) in the time-series human eye image, first perform grayscale processing, and then use the histogram equalization algorithm to process the grayscale, and obtain Processed time-series human eye images

E ^pro =histgram(rgb2gray(E))

Among them, E is the current human eye image, and E ^pro is the processed time-series human eye image.

Step 5, extract the artificial descriptor of the processed time-series human eye image, and encode it into a blink behavior feature heat map:

(5-1) For time series human eye images Take the LBP (Local Binary Pattern) feature, extract the LBP feature of each frame image, and get the LBP feature map of the frame image

Among them, (x _c , y _c ) is the central coordinate point, p is the pth pixel of the neighborhood, i _p is the gray value of the neighborhood pixel, _ic is the gray value of the center pixel, s(x) is Symbolic functions, as follows:

(5-2) For the obtained LBP feature map Binarize each pixel first to obtain a long string of binarized data:

S={S _xy [b ₁ , b ₂ , . . . b ₈ ]{bi=0 or 1(i=1, 2, ..8)}}.

where S _xy represents The binary value at the middle coordinate (x, y);

(5-3) For each S _xy in S, first count the sequences without jumps in the long string, that is, all 0 or all 1 sequences, which are divided into two categories;

(5-3) For each S _xy in S, count the hopping sequences that have only one (0→1 or 1→0) jump sequence in the long string, and divide these sequences according to the pth position that occurs in the sequence Divided into 14 categories;

(5-4) After counting the sequence S _xy of two jumps in S, find the difference between the first and second jump positions, and then divide these sequences into 42 categories;

(5-5) Classify the remaining features into the last category, and mark the same value for all sequences of the same category, then replace the LBP feature with a uniform-LBP feature with a dimension of 59;

(5-6) Subtract the uniform-LBP feature (that is, the artificial descriptor) of the nth frame from the corresponding bit of the uniform-LBP feature of the previous frame to obtain the difference feature (difference feature), and then concatenate the differ feature After the uniform-LBP feature of the nth frame, the LBP-differ feature (human eye feature) of the nth frame is obtained;

LBP_differ _i ＝[uniform_LBP _i ，(uniform_LBP _i ⊙uniform_LBP _i-1 )]

in:

Specifically, as shown in Figure 4, the uniform-LBP features of Frame1 and Frame2 are extracted, and the corresponding bits of the uniform-LBP feature of Frame2 are subtracted from the uniform-LBP features of Framel to obtain the differ feature of Frame2, and then the differ feature string of Frame2 After the uniform-LBP feature of Frame2, the LBP-differ feature of Frame2 is obtained.

(5-7) Stitch n frames of human eye features in time series into a 9*118 blink behavior feature heat map:

map = [LBP_differ ₁ ; LBP_differ ₂ ; ...; LBP_differ _n ]

Step 6, LSTM processes the blink behavior feature heat map, and obtains the result, including the following steps:

(6-1) Establish an LSTM network with 9 cells, each cell contains 2 hidden layers, and the number of hidden layers is 128. The input of the network does not use dropout, and the cell state transmitted between nodes uses dropout with an amplitude of 0.5;

(6-2) Use the 9*118 eye blinking behavior feature heatmap according to the time series it represents, and use it as the input of the trained LSTM neural network row by row to obtain multiple hidden states.

Step 7, concatenating multiple hidden states to obtain multi-scale time-series features and then input the sphereface model for eye blink detection to determine whether the time-series images contain eye blinking behavior, including the following steps:

(7-1) Take out the 128-dimensional output hidden state O _i (i=1, 2,...n) of the last two hidden layers of the LSTM network, and concatenate them in sequence:

O ^comb ＝[O _n-1 ，O _n ]

Among them, O ^comb is the 256-dimensional multi-time scale output of the LSTM network, and O _i is the output of the i-th cell;

(7-2) Input O ^comb as the sphereface model with trained parameters, and calculate the result res[res1, res2]; use:

state=argmax(res)

Among them, state is the final result, 0 means that the time-series image has not blinked, 1 means that the time-series image blinked, and the expression of the argmax function is:

The sphereface model uses the loss function of the A-softmax class, as follows:

Among them, the i function means that the 1st means the i-th category, and the representation of the number is: 1 means that the blinking behavior occurs in the image sequence, Indicates the angle between y _i and w _i ,

Firstly, the present invention realizes real-time computing, and secondly connects the differ feature and the artificial descriptor to obtain the human eye feature, which has stronger semantic information; secondly, the blinking behavior is encoded into a blinking behavior feature heat map, which is more conducive to intuitive observation The processing with the algorithm in the image field, and finally, using the LSTM network to obtain multiple hidden states, and concatenating them into multi-scale temporal features for training can adapt to the problem of different blinking speeds of different people. Compared with the traditional eye blink detection algorithm, the present invention has better performance in speed and accuracy in the time-series eye blink database under unconstrained conditions.

It is easy for those skilled in the art to understand that the above descriptions are only preferred embodiments of the present invention, and are not intended to limit the present invention. Any modifications, equivalent replacements and improvements made within the spirit and principles of the present invention, All should be included within the protection scope of the present invention.

Claims (7)

1. A real-time blink detection method based on multi-scale time-series images, characterized in that, comprising: (1) Establishing a database containing time-series images of human eyes, and positioning the first frame image of the time-series images in the database to obtain the positions of the eyes; (2) Extract the human eye image from the first frame image by using the position of both eyes to obtain the human eye template; (3) Utilize the human eye template to initialize the human eye tracker, utilize the initialized human eye tracker to track the follow-up frame images of the first frame image in the sequential images and update the human eye tracker; (4) Utilize the updated human eye tracker to track the time-series images to obtain the time-series human eye images, and sequentially perform grayscale processing and image equalization processing on the time-series human eye images to obtain the processed time-series human eye images; (5) Extract the artificial descriptor of the processed time-series human eye image, extract the difference feature of the artificial descriptor, concatenate the difference feature and the artificial descriptor to obtain the human eye feature, and encode the human eye feature into a blink according to the time-series information Behavioral characteristic heat map; (6) Input the blink behavior feature heat map into the LSTM network line by line to obtain multiple hidden states; (7) Multiple hidden states are concatenated to obtain multi-scale time-series features, and then blink detection is performed to determine whether the time-series images contain blinking behavior. 2. a kind of real-time blink detection method based on multi-scale time series image as claimed in claim 1, is characterized in that, described step (1) comprises: The video database containing human eyes is used to establish a database containing time-series images of human eyes, and the face alignment algorithm is used to locate the first frame of the time-series images in the database to obtain the positions of the eyes. 3. a kind of real-time blink detection method based on multi-scale time series images as claimed in claim 1 or 2, is characterized in that, described step (2) comprises: (2-1) Utilize the position of both eyes to obtain the distance between the two eyes, determine the width and height of the human eye area based on the distance between the two eyes, and obtain the human eye area; (2-2) According to the human eye area, extract the human eye image from the first frame of image, after grayscale processing of the human eye image, judge whether to reduce the human eye image by K times according to the size of the human eye image, K ≥0, and finally the obtained human eye image is used as the human eye template. 4. a kind of real-time blink detection method based on multi-scale time series images as claimed in claim 1 or 2, is characterized in that, described step (3) comprises: (3-1) Initialize the human eye tracker using the human eye template; (3-2) Use the initialized human eye tracker to track the subsequent frame images of the first frame image in the time series images, obtain the human eye position Pos of the subsequent frame images, and feed back the confidence C of each tracking; (3-3) Set the tracking score threshold T, if C>T, use the human eye position Pos to extract a new human eye image, and obtain a new human eye template, if C≤T, then perform step (1); (3-4) The tracker after initialization of the new eye template and the eye tracker after initialization are weighted and fused, and the eye tracker is updated. 5. A kind of real-time blink detection method based on multi-scale time-series images as claimed in claim 1 or 2, wherein said differ feature is: each frame of artificial descriptor extracted from the processed time-series human eye images The difference feature obtained by subtracting the corresponding bit from the artificial descriptor of the previous frame. 6. a kind of real-time blink detection method based on multi-scale time series images as claimed in claim 1 or 2, is characterized in that, described step (6) comprises: (6-1) Establish an LSTM network with n cells, each cell contains h hidden layers, the input of the LSTM network does not use the dropout technology that eliminates overfitting, and the cell state of the LSTM network is transmitted between cells using a range of d dropout technology, d<1; (6-2) Use the sample heat map containing labels to train the LSTM network to obtain the hidden states of multiple samples, concatenate the hidden states of multiple samples according to the time sequence to obtain the multi-scale time series features of the samples, train the sphereface model, and obtain the sphereface model. Parameters and trained LSTM network; (6-3) Input the heat map of eye blinking behavior features into the trained LSTM network line by line to obtain multiple hidden states. 7. a kind of real-time blink detection method based on multi-scale time series images as claimed in claim 6, is characterized in that, described step (7) comprises: The multi-scale time-series features obtained by concatenating multiple hidden states according to the time series are input into the sphereface model for blink detection, and it is judged whether the time-series images contain eye blinking behavior.