CN111062292B - Fatigue driving detection device and method - Google Patents

Abstract

The invention discloses a fatigue driving detection device and method. A main control module is electrically connected with an image acquisition device, a storage module and an alarm module respectively; the alarm module includes LED lights and an alarm; the image acquisition module includes a camera and four infrared LEDs Light source; the main control module outputs control signals to the camera, and the camera collects the driver's image and inputs it to the main control module; the storage module is used to store the hash fingerprint information, the corresponding face positioning frame and the fatigue detection standard parameters; Receive the data for processing, obtain the fatigue characteristic parameters in real time, and read the fatigue detection standard parameters in the storage module, obtain the fatigue degree according to the result of the comparison between the two, and output the control signal to the alarm module to control the flashing of the LED light and the buzzer of the alarm. Inventions can be detected around the clock. Distraction detection can give early warning of fatigue. Using multiple features to comprehensively judge the degree of fatigue, the results are good in both real-time and accuracy.

Description

Fatigue driving detection device and method

technical field

The invention belongs to the technical field of safe driving, in particular to a fatigue driving detection device and method.

Background technique

With the improvement of people's life happiness index, the popularity of automobiles is also getting higher and higher. However, the public's attention to traffic safety issues has not increased accordingly, especially the lack of awareness of the hidden causes of accidents such as fatigue driving. The consequences of fatigue driving are sometimes extremely serious. Therefore, the detection and timely warning of fatigue driving through technical means can effectively reduce the harm caused by traffic accidents.

At present, most of the researches are based on visual feature methods. After collecting the driver's face image, fatigue is detected by analyzing the facial features such as the eyes and mouth. However, this process is extremely complicated. On the one hand, facial features are easily affected by individual factors and the lighting environment, and there is a problem with the posture and angle of the face during driving, which greatly interferes with the detection accuracy. On the other hand, some methods cannot achieve a good balance in terms of accuracy and real-time performance. Some methods have improved accuracy, but do not consider the real-time problem; some methods can achieve rapid detection, but face the problem of poor accuracy. Therefore, this technology needs further improvement and research in practical application.

SUMMARY OF THE INVENTION

In view of the above-mentioned prior art, the technical problem to be solved by the present invention is to provide a fatigue driving detection device and method that takes both real-time performance and accuracy into consideration, and is not easily affected by illumination changes and attitude angles.

In order to solve the above technical problems, a fatigue driving detection device of the present invention includes an image acquisition module, a main control module, a storage module, and an alarm module, and the main control module is respectively electrically connected with the image acquisition device, the storage module, and the alarm module; The alarm module includes LED lights and an alarm; the image acquisition module includes a camera and 4 infrared LED light sources; the main control module outputs control signals to the camera, and the camera collects driver images and inputs them to the main control module; the storage module It is used to store the hash fingerprint information, the face positioning frame corresponding to the hash fingerprint and the fatigue detection standard parameters; the main control module processes the received data, obtains the fatigue characteristic parameters in real time, and reads the fatigue detection in the storage module. According to the standard parameters, the fatigue level is obtained according to the result of the comparison between the two, and the corresponding control signal is output to the alarm module, which controls the alarm module to issue a set alarm signal.

A detection method using the above-mentioned fatigue driving detection device, comprising the following steps:

S1: Use the image acquisition device to collect the driver's driving image, the image includes the image of the driver when he is awake, obtain the initial value of the eye aspect ratio, and use it as the fatigue detection standard parameter;

S2: Use the improved fast face detection algorithm to locate the face position;

S3: When a face is detected, use the face feature point positioning algorithm to obtain 68 face feature point positions;

S4: Extract the eye information according to the position of the facial feature points, calculate the eye aspect ratio, calculate the ratio of the number of closed eye frames per unit time to the total number of frames, the PERCLOS value and the blinking frequency, and determine the PERCLOS value T and the given threshold. relationship: when T ≥ T _Z , it is judged as severe fatigue, T _Z is the threshold of severe fatigue, and the alarm module sends out the set alarm signal; when T _Q < T < T _Z , T _Q is the threshold of mild fatigue, and the execution S5; when T≤T _Q , execute step S6;

S5: further determine whether the blinking frequency is too fast: when the blinking frequency is greater than the given blinking frequency threshold, it is determined as normal driving, and step S6 is performed; otherwise, it is determined as mildly fatigued driving, and the alarm module sends a set alarm signal;

S6: Calculate the head posture angle. When the angle in the vertical direction of the head posture is greater than the given angle threshold, it is determined as bowing, and when the time when the angle in the vertical direction of the head posture is greater than the given angle threshold is greater than the given time threshold, If it is determined to be severe fatigue, the alarm module sends a set alarm signal; otherwise, step S7 is performed;

S7: Determine whether the eyes are closed according to the aspect ratio of the eyes: when the eyes are closed, the distraction detection cannot be performed, and it is determined as normal driving; when the eyes are not closed, step S8 is performed;

S8: Perform distraction detection, calculate the horizontal declination angle θ _l and vertical declination angle θ _v , divide the area in front of the driver into a normal gaze area and a deviated gaze area, and obtain the driver's gaze according to θ _l and θ _v In the area where the landing point is located, when the time of looking at the landing point in the line-of-sight deviation area exceeds the given time threshold, it is determined as mild fatigue, and the alarm module sends out the set alarm signal; otherwise, it is determined as normal driving.

The present invention also includes:

1. Preprocess the sampled images of subsequent video streams in S1, including adaptive median filtering and Laplacian-based image enhancement.

2. Using the improved fast face detection algorithm in S2 to locate the face position is as follows:

First, calculate the average hash fingerprint of the driver image to be detected. If there is fingerprint information that differs from the average hash fingerprint by no more than 2 bits in the cache module, the face location frame corresponding to the fingerprint in the cache is directly output, and the cache Add 1 to the number of times the fingerprint is called in ;

If there is fingerprint information that differs from the average hash fingerprint by more than 2 bits but less than 5 bits in the cache module, call the AdaBoost face detection algorithm to get the positioning frame, and judge whether the cache capacity is full. The hash fingerprint is stored in the cache database; when the cache capacity is full, the fingerprint with the smallest number of calls in the cache is deleted, and then the current average hash fingerprint is stored in the cache database;

If the difference between the fingerprint information in the cache module and the average hash fingerprint is more than 5 bits, the AdaBoost face detection algorithm is called to obtain the positioning frame.

3. Using the facial feature point positioning algorithm in S3, the coordinates of 68 feature points are obtained as follows:

First, use the HOG-SVM algorithm to judge the face orientation: select different face orientations, extract the HOG feature of the image to be detected to obtain the HOG feature vector, and use it as the SVM input parameter to obtain the face orientation estimation classifier;

According to the different face orientation results, different face feature point initialization models are selected;

For each face feature point position, the trained random forest is used to extract the LBF feature, and the trained linear regressor is used to obtain the regression result to update the face feature point position until the maximum number of iterations is reached, and the face feature point position is output.

4. Using the trained random forest to extract the LBF feature is to normalize the gray levels of the two pixels near the feature point as a classification feature.

5. The calculation of the eye aspect ratio in S4 is as follows:

The eye aspect ratio WHR satisfies:

Among them, W is the eye width, H is the height, and WHR _init is the initial value of the eye aspect ratio when the driver is awake;

The ratio of the number of closed-eye frames to the total number of frames in the calculation unit time in S4 PERCLOS value T is specifically:

When the WHR is greater than the given threshold, it is determined that the human eye is in the closed state, the total number of frames per unit time is Z, and the number of frames in the closed state of the human eye is z, then T satisfies:

The calculation of the blinking frequency described in S4 is specifically:

If the WHR of the current frame image is less than or equal to 3, and the WHR of the current frame image is greater than 3, it is determined that one blink occurs, and the number of blinks in a given unit time is the blink frequency.

6. The calculation of the head attitude angle in step S6 is as follows:

After obtaining the position of the facial feature points, by solving the rotation and translation matrices, the two-dimensional N feature points on the image are mapped to the corresponding N feature points on the standard three-dimensional face model, and finally the rotation matrix R is used to calculate the head. Attitude angle α, βγ:

Among them, β is the angle of the vertical direction of the head posture, α is the angle of the horizontal direction of the head posture, and γ is the angle of the front and rear directions of the head posture.

7. The attention distraction detection is performed in S8, and the horizontal declination angle θ _l and the vertical declination angle θ _v of the line of sight are calculated as follows:

垂直偏角η，然后用头部姿态α、β修正视线方向，求出最终视线水平偏角为

垂直偏角为θ_v＝η+β。For the displacement vector of a point in the pupil and the boundary point of the pupil and the gradient vector of the boundary point, the point on the pupil corresponding to the maximum value of the inner product of the two is the center of the pupil. The Purkin light spot is formed by 4 infrared LED light sources distributed around the area in front of the driver. Calculate the relative position of the pupil center and the center of the rectangle surrounded by the four Purkin light spots, and obtain the horizontal declination angle of the two.

The vertical declination angle η, and then use the head posture α, β to correct the line of sight direction, and the final horizontal declination angle of the line of sight is obtained as

The vertical deflection angle is θ _v =η+β.

Beneficial effects of the present invention:

1. The use of infrared devices solves the problem of low accuracy in low light conditions such as night, so as to achieve the purpose of all-weather fatigue detection. After improving the face detection algorithm, the position of the face can be quickly and accurately determined;

2. By improving the initialization strategy of the LBF algorithm, the influence of the face orientation problem on the positioning of face feature points is reduced, and the robustness to problems such as wearing glasses and lighting changes is enhanced;

3. The driver distraction detection is applied to the fatigue driving detection algorithm, so that the system can warn the driver before the driver falls into deep fatigue, so as to ensure the safety of the driver;

4. Using multiple features to comprehensively judge the degree of fatigue, the algorithm has good effects in both real-time and accuracy, and successfully solves the balance problem between the two.

Description of drawings

FIG. 1 is a schematic flowchart of a driver fatigue driving detection method according to an embodiment of the present invention.

FIG. 2 is a flowchart of a driver face detection algorithm according to an embodiment of the present invention.

FIG. 3 is a schematic diagram of initialization models of different facial feature points according to an embodiment of the present invention.

FIG. 4 is a flowchart of a facial feature point location algorithm according to an embodiment of the present invention.

FIG. 5 is a schematic diagram of dividing a driver's gaze area according to an embodiment of the present invention.

FIG. 6 is a structural diagram of a hardware device for fatigue driving detection according to an embodiment of the present invention.

Detailed ways

The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

1 is a flowchart of a method for detecting driver fatigue driving according to an embodiment of the present invention, which includes the following steps:

S1. Collect driver images through a camera and perform preprocessing.

Specifically, the camera is an infrared camera, and four infrared LED light sources are arranged around the front windshield of the car. Used to capture driver images. When the system runs initially, the driver is awake, and the input image is collected at this time to obtain the initial value of the eye aspect ratio. The video stream is then sampled to obtain the driver input image.

In a preferred embodiment, the video stream is preprocessed after the image is captured. The preprocessing techniques include adaptive median filtering and Laplacian-based image enhancement. Adaptive median filtering replaces the intermediate pixels by the median of all pixel grayscales in the template, and the template size can be adjusted to reduce the interference of smooth non-impact noise. Laplacian image enhancement traverses the entire image, and then uses the Laplacian operator to obtain the value after the pixel according to the gray value of the pixel neighborhood.

S2. Use the improved face detection algorithm to mark the face position.

Specifically, as shown in Figure 2, for the driver image to be detected, its mean hash fingerprint is first calculated. Mean hash fingerprint is a series of binary numbers obtained by corresponding calculation of the image. Use it to characterize how similar two images are. Since the driver's movements are generally not very large during fatigue driving detection, for the images of several frames before and after, the similarity can be judged by comparing their mean hash fingerprints, and the cache mechanism can be used to directly retrieve them from the storage module. Face positioning frame, thereby speeding up the detection rate.

The process of the improved fast face detection algorithm is as follows: reduce the size of the input image to 8*8, and then calculate the average gray level of the image. The relationship between the gray value of each pixel of the image and the mean value is counted. If it is greater than the mean value, it is set to 1, and if it is less than the mean value, it is set to 0, so as to obtain a string of mean hash fingerprint information hash_finger consisting of 0 and 1 characters. If there is fingerprint data with a difference of less than or equal to 2 from the hash_finger in the cache module, the face positioning frame corresponding to the fingerprint data in the cache is directly output; otherwise, the AdaBoost algorithm is used to output the face positioning frame. Determines whether to save the face positioning frame to the cache module. If the difference is greater than or equal to 5, it will not be saved; if the difference is greater than 2 and less than 5, the hash_finger and the face positioning frame will be stored in the cache module. In order to solve the problem of cache capacity, the number of calls is counted for each hash fingerprint, so that when the cache is full, the hash fingerprint with the minimum number of calls and the corresponding face location frame are deleted.

S3. Using the improved facial feature point positioning algorithm, the positions of 68 facial feature points are obtained.

The traditional algorithm based on local binary feature (LBF) needs to input a face feature point to initialize the model at the beginning, and then make the model more and more close to the real position through continuous regression. In order to improve the detection accuracy and speed, the present invention optimizes the LBF algorithm and adopts different initialization strategies. Instead of simply using the standard average face for initialization, the HOG-SVM algorithm is first used to determine the face orientation, so as to use different face feature points to initialize the model.

Specifically, the HOG feature divides the target image into several cells, and the gradient histograms of the pixels in each cell are combined to form a vector representing the image. HOG has no rotation and scale invariance, so the computation is much faster. Moreover, it is sensitive to shape changes and can well represent the face contour, especially suitable for face orientation classification tasks. After the HOG feature is extracted, the HOG feature vector can be obtained for subsequent SVM classification, and finally the face orientation classifier is obtained.

In the embodiment of the present invention, five kinds of face orientations are selected during training, namely, the face without deflection, the face with a 22-degree deflection to the left, a face with a 45-degree deflection to the left, a face with a 22-degree deflection to the right, and a face with a deflection to the right. 45 degree face. So create 5 new folders in the training dataset folder, name them 0, -1, -2, 1, 2 and store the corresponding images in them. During SVM training, the name of the folder where the image is located is used as the classification label, and the HOG feature vector is extracted as the SVM input parameter to obtain the face orientation estimation classifier.

As shown in Figure 3, different face feature point initialization models are selected according to the different face orientation results. After that, as shown in Figure 4, the trained random forest is used to extract the LBF feature, and the trained linear regressor is used to obtain the regression result to update the position of the face feature point until the maximum number of iterations is reached. Specifically, the random forest training process: the training sample is divided into multiple small samples, and the pictures in each small sample have 68 feature points. In the circle with each feature point as the center, 500 pixel points are generated, and the difference between the two is calculated. Using these differences as features, a threshold is selected to divide all the pictures in the current small sample into two categories: left and right subtrees, and the variance attenuation before and after classification is calculated. The threshold value that maximizes the value of statistics is the final threshold value of this classification, and the feature at this time is the final feature of this time. Keep splitting up to the maximum depth of the tree. A decision tree for a feature point is established, and then a small sample set is taken and the above steps are repeated. Through this method, multiple subtrees are constructed for each feature point to form a random forest. There are 68 points on the face in this article, which is 68 random forests.

In order to improve the classification effect, the present invention no longer simply uses the pixel difference value as the feature, but uses the normalized value:

x and y are two pixel values, using NOL(x, y) as the feature to classify, this normalization makes the feature more sensitive to lighting. At the same time, the amount of calculation does not even change after this processing, but the classification effect is significantly improved.

Then use the trained random forest to extract local binary features. For each feature point of each picture, the picture must be classified into a certain leaf node, and the leaf node is encoded as 1 at this time. Otherwise encode it as 0. The binary code of a tree can be obtained, and the combination of all binary codes in the random forest of the feature point is the LBF feature. Finally, all LBFs are combined to form the feature map Φ ^t .

的目标函数来学习。随着层级的不断深入，回归产生的特征点越来越逼近真实位置。The position increment is used as the learning target during regression, and the linear regressor W ^t is trained. The LBF algorithm multiplies the linear regression matrix W ^t and the feature mapping function Φ ^t in the regression of each level, and obtains a position variable ΔS according to the face feature point initialization model and the current feature point position information, thereby correcting the position information of this level S ^t , that is, S ^t = S ^t-1 +W ^t Φ ^t (I, S ^t-1 ). Then continue regression and iteration. By minimizing the formula

the objective function to learn. With the deepening of the level, the feature points generated by regression are getting closer and closer to the real position.

S4. Extract eye information and calculate head posture according to the positions of the facial feature points, and use them as parameter features for fatigue driving detection;

Specifically, the present invention first calculates the eye aspect ratio WHR by using the eye width W and the height H according to the shape of the eye feature points and edge detection.

WHR _init is the initial value of the driver's eye aspect ratio when awake. Preferably, in the embodiment of the present invention, when the eye closure exceeds 80%, it is considered that the human eye is currently in a closed state, that is, when the WHR is greater than 3, the human eye is closed. The PERCLOS value is obtained by counting the ratio of the number of closed eye frames to the total number of frames per unit time. Take the unit time as 30 seconds, set the threshold of mild fatigue as 0.3, and set the threshold of severe fatigue as 0.5. If the detected PERCLOS value is greater than 0.5, it indicates that the driver is severely fatigued; if the PERCLOS value is less than 0.3, the driver is normal; if it is between the two, the blink frequency is calculated. If the previous frames have WHR≤3 and the current frame is greater than 3, then a blink occurs. Set the blink frequency threshold to 5 times/5 seconds. If it is greater than the threshold, the blink frequency is too fast, indicating that it is not caused by fatigue, but the PERCLOS value caused by rapid blinking under special circumstances (such as strong wind, strong light). If it rises, it is judged to be normal driving, and it is mild fatigue driving anyway.

If the above characteristic parameters are used to determine that the driver is not in a fatigued state, the head attitude angle is calculated. First, after obtaining the position of the facial feature points, by solving the rotation and translation matrix, the two-dimensional N feature point positions on the image are mapped to the corresponding N feature point positions on the standard three-dimensional face model. This step involves regularizing the feature points. Finally, the rotation matrix R is used to calculate the head attitude angles α, β, γ.

Among them, β is the angle of the vertical direction of the head posture. In view of the fact that the driver will bow his head for a long time when fatigued, the threshold of the head posture angle is set to 20°. If the β for 3 consecutive seconds is greater than the threshold, it is determined as severe Fatigue, α is the angle in the horizontal direction of the head posture, and γ is the angle in the front and rear direction of the head posture.

S5: Determine the gaze direction of the human eye according to the relative positions of the pupil in the human eye and the Purkins spot formed by the infrared light source, and perform the driver's distraction detection.

垂直偏角η。然后用头部姿态α、β修正视线方向，求出最终视线水平偏角为

垂直偏角为θ_v＝η+β。Fatigue is progressive, and distraction is often the beginning of the fatigue process. It is possible that the driver's eye features and head posture are normal, but his attention is distracted, which also affects driving safety. In the present invention, distraction is taken as a mild fatigue indicator, and if fatigue is not detected through the step S4, distraction detection is performed. Specifically, for the displacement vector of a point in the pupil and the pupil boundary point and the gradient vector of the boundary point, the point on the pupil corresponding to the maximum value of the inner product of the two is the pupil center. After locating the center of the pupil, search for the location of the Purkin spot near it. The Purkin light spot is formed by four infrared LED light sources distributed around the area in front of the driver, so it has the property of projection space. Calculate the relative position of the pupil center and the center of the rectangle surrounded by the four Purkin light spots, and obtain the level of the two. declination

Vertical declination angle η. Then use the head posture α, β to correct the line of sight direction, and obtain the final horizontal declination angle of the line of sight as

The vertical deflection angle is θ _v =η+β.

As shown in FIG. 5 , in the embodiment of the present invention, the front area is divided into 9 parts of 3*3, and the angle boundary of each area is marked in the figure, and the driver's gaze area is obtained according to the angles of θ _l and θ _v . When the driver is driving normally, the line of sight is almost always straight ahead. Except for areas 1, 4, and 5, the gaze points in other areas belong to the line of sight deviation. Long-term sight deviation is a phenomenon of distraction, so the threshold for detection of distraction is determined as: if the duration of the gaze is not in areas 1, 4, and 5 for more than 3 seconds, the driver has been distracted, and it is judged as mild fatigue.

The embodiment of the present invention also provides a fatigue driving detection device, as shown in FIG. 6 . Including image acquisition module, main control module, storage module, alarm module. Preferably, the image acquisition module uses an infrared CCD camera and is equipped with 4 infrared LED light sources. The main control module adopts Raspberry Pi 3B motherboard. The memory module uses 1GB of LPDDR2 SDRAM memory. The alarm module includes LED lights, sirens and horns.

The invention includes the entire detection process of image acquisition, image preprocessing, face detection, facial feature point positioning, eye feature extraction, head posture solution, attention distraction detection and fatigue state judgment. In different situations, the fatigue state is divided into three levels, namely normal driving, mild fatigue, and severe fatigue. According to the detected PERCLOS value, eye blink frequency, head posture angle, driver's sight deviation and other multi-feature parameters, the current fatigue state level of the driver is judged, and different warning levels are adopted to reduce traffic caused by fatigued driving. ACCIDENT. The above descriptions are only preferred embodiments of the present invention, and do not specifically limit the patent scope of the present invention. Although the above preferred embodiments are described in detail, researchers in this field should understand that, under the inventive concept of the present invention, various changes can be made to the present invention in detail or structure, without departing from the limitations of the claims of the present invention range.

The specific embodiment of the present invention also includes:

In view of the fact that the accuracy and real-time performance of the fatigue driving detection technology cannot be well balanced in practical applications, and the accuracy of facial features is affected by changes in illumination and existing posture angles, the present invention provides a Fatigue driving detection method and device. The technical solution of the present invention can realize all-weather detection and enhance the robustness to problems such as wearing glasses, face orientation and illumination changes. Distraction detection can provide early warning of fatigue. Using multiple features to comprehensively judge the degree of fatigue, both real-time and accuracy are effective, and the balance problem between the two is successfully solved.

The technical solution of the present invention to solve the above problems: a multi-feature-based fatigue driving detection method, comprising the following steps:

S1. Use an infrared device to collect a driver's driving image;

S2. Use the improved fast face detection algorithm to locate the face position;

S3. Use the facial feature point positioning algorithm to obtain 68 facial feature point coordinates;

S4. Extract eye information and calculate head posture according to the positions of the facial feature points, and use them as parameter features for fatigue driving detection;

S5. Determine the gaze direction of the human eye according to the relative position of the pupil in the human eye and the Purkin light spot caused by the infrared LED, and perform the driver's distraction detection;

S6. Perform fatigue state detection by comparing the fatigue multi-features detected in real time in steps S4 and S5 with a threshold, wherein the fatigue features include the driver's PERCLOS value, blinking frequency, head posture angle, and gaze area placement.

In the step S1, the infrared device uses four infrared LED light sources to cooperate with the camera, and on the one hand, its function is to collect the driver's image and reduce the interference caused by the change of illumination. On the other hand, in the step S5, four infrared light sources are used to determine the direction of the driver's line of sight. Four infrared LED light sources are installed around the front windshield of the car.

When performing face detection in the step S2, Haar features are extracted from positive and negative face samples, and input into the AdaBoost algorithm to obtain a face detection classifier. Aiming at the low real-time performance of traditional AdaBoost algorithm, an improved algorithm is proposed. The mean hash algorithm is introduced to convert face feature information into hash fingerprint information, and a caching mechanism is added to store hash fingerprint information similar to images, thereby reducing detection time. First, calculate the mean hash fingerprint information of the current input image, and if there is a hash fingerprint similar to the information in the cache device, directly output the face location frame in the cache. Otherwise, the face detection algorithm based on AdaBoost is used, and the current fingerprint information and the face positioning frame are stored in the cache, so that the similar pictures can be judged in the next frame.

In the step S3, before using the LBF algorithm to locate the facial feature points, a step is added to improve the detection rate, that is, the HOG-SVM algorithm is used to judge the facial orientation, and different facial feature points are selected according to the detection results of the facial orientation. Initialize the model, then use the improved LBF algorithm for localization, resulting in the location of 68 points. First extract the HOG feature of the face image, then use the SVM face orientation classifier, and use different face feature points to initialize the model according to the different results of the face orientation. Then, the LBF optimization algorithm is used to make the initialization model get closer and closer to the accurate position through continuous regression and iteration. The LBF optimization algorithm means that when training the LBF random forest, the difference between the two pixels near the feature point is no longer used as the random forest classification feature, but the grayscale of the two pixels is normalized as the classification feature.

The parameter features of the fatigue driving detection in the step S4 include human eye closure, blink frequency, PERCLOS parameter and head posture angle. The eye image is obtained according to the position of the facial feature points, and the parameter used to evaluate the degree of eye closure and blinking frequency is the eye aspect ratio. By setting the aspect ratio threshold, it is judged whether the human eye is closed, so as to count the blink frequency threshold. The PERCLOS parameter is obtained according to the proportion of human eye closure time to unit time. The center of the pupil is located using a gradient-based algorithm. The head pose angle is obtained through the mapping relationship between the 2D 68 face feature points and the 3D standard face model.

The step S5 firstly uses the improved gradient-based pupil center positioning algorithm to obtain the pupil position, and uses the properties of the mapping space to divide the area in front of the driver into 9 parts. Calculate the relative position of the Purchin spot formed by the four infrared light sources and the pupil, and combine the head posture correction angle to obtain the gaze area landing point, count the deviation of the sight line, and judge whether the driver's attention is scattered.

In the step S6, fatigue is detected according to the multi-features obtained above. First, judge the closing situation of the human eye according to the eye aspect ratio, update the PERCLOS value and blink frequency, judge whether the two exceed the threshold, and make corresponding warnings according to different situations. If fatigue is not detected, the head posture angle is calculated, and it is judged whether the head is bowed for a long time, and if so, it is judged as severe fatigue. When the above characteristics belong to normal driving and the human eyes are open, it is judged whether the attention is distracted according to the deviation of the sight. The fatigue standard parameters include the initial value of the driver's eye aspect ratio, the first PERCLOS threshold, the second PERCLOS threshold, the blink frequency threshold, the head posture threshold and the distribution statistics of the human eye gaze area. The fatigue level is divided into three levels, and the driver's fatigue level is judged by the different manifestations of the above-mentioned multiple features, and corresponding warnings are given.

The invention also provides a fatigue driving detection device.

A fatigue driving detection device includes an image acquisition module, a main control module, a storage module, and an alarm module. The main control module is respectively electrically connected with the image acquisition module, the storage module and the alarm module. The image acquisition module includes a CCD camera installed above the instrument panel shell, four infrared LED light sources installed above the instrument panel case and located around the windshield; the main control module is a Raspberry Pi main board and peripheral circuits , the peripheral circuit includes a power supply module and a communication module; the alarm module includes an LED light and an alarm device; the storage module is an LPDDR2 SDRAM memory for storing hash fingerprint information, a face positioning frame corresponding to the hash fingerprint, and fatigue Check standard parameters. The main control module outputs the control signal to the camera, the camera collects the driver's image, inputs it to the main control module through the USB data interface, executes the fatigue driving detection program, and obtains the aforementioned fatigue characteristic parameters in real time. And read the fatigue detection standard parameters in the storage module, get the fatigue degree according to the results of the comparison between the two, and output a control signal to the alarm module to control the flashing of the LED light and the buzzer of the alarm.

Claims (8)

1. A fatigue driving detection device is characterized in that: comprising an image acquisition module, a main control module, a storage module, an alarm module, and the main control module is electrically connected with the image acquisition device, the storage module, and the alarm module respectively; the alarm module Including LED lights and alarms; the image acquisition module includes a camera and 4 infrared LED light sources; the main control module outputs control signals to the camera, and the camera collects the driver's image and inputs it to the main control module; the storage module is used to store Hash fingerprint information, a face positioning frame corresponding to the hash fingerprint, and fatigue detection standard parameters; the main control module processes the received data, obtains fatigue characteristic parameters in real time, and reads the fatigue detection standard parameters in the storage module, Obtain the fatigue level according to the result of the comparison between the two, output the corresponding control signal to the alarm module, and control the alarm module to send out the set alarm signal; the detection steps of using the device include: S1: Use the image acquisition device to collect the driver's driving image, the image includes the image of the driver when he is awake, obtain the initial value of the eye aspect ratio, and use it as the fatigue detection standard parameter; S2: Use the improved fast face detection algorithm to locate the face position, specifically: first calculate the average hash fingerprint of the driver image to be detected, if there is a difference between the average hash fingerprint and the average hash fingerprint in the cache module by no more than 2 bits If the fingerprint information is stored in the cache, the face location frame corresponding to the fingerprint in the cache is directly output, and the number of calls of the fingerprint in the cache is increased by 1; If there is fingerprint information that differs from the average hash fingerprint by more than 2 bits but less than 5 bits in the cache module, call the AdaBoost face detection algorithm to get the positioning frame, and judge whether the cache capacity is full. The hash fingerprint is stored in the cache database; when the cache capacity is full, the fingerprint with the smallest number of calls in the cache is deleted, and then the current average hash fingerprint is stored in the cache database; If the difference between the fingerprint information in the cache module and the mean hash fingerprint is more than 5 bits, call the AdaBoost face detection algorithm to obtain the positioning frame; S3: When a face is detected, use the face feature point positioning algorithm to obtain 68 face feature point positions; S4: Extract the eye information according to the position of the facial feature points, calculate the eye aspect ratio, calculate the ratio of the number of closed eye frames per unit time to the total number of frames, the PERCLOS value and the blinking frequency, and determine the PERCLOS value T and the given threshold. relationship: when T ≥ T _Z , it is judged as severe fatigue, T _Z is the threshold of severe fatigue, and the alarm module sends out the set alarm signal; when T _Q < T < T _Z , T _Q is the threshold of mild fatigue, and the execution S5; when T≤T _Q , execute step S6; S5: further determine whether the blinking frequency is too fast: when the blinking frequency is greater than the given blinking frequency threshold, it is determined as normal driving, and step S6 is performed; otherwise, it is determined as mildly fatigued driving, and the alarm module sends a set alarm signal; S6: Calculate the head posture angle. When the angle in the vertical direction of the head posture is greater than the given angle threshold, it is determined as bowing, and when the time when the angle in the vertical direction of the head posture is greater than the given angle threshold is greater than the given time threshold, If it is determined to be severe fatigue, the alarm module sends a set alarm signal; otherwise, step S7 is performed; S7: Determine whether the eyes are closed according to the aspect ratio of the eyes: when the eyes are closed, the distraction detection cannot be performed, and it is determined as normal driving; when the eyes are not closed, step S8 is performed; S8: Perform distraction detection, calculate the horizontal declination angle θ _l and vertical declination angle θ _v , divide the area in front of the driver into a normal gaze area and a deviated gaze area, and obtain the driver's gaze according to θ _l and θ _v In the area where the landing point is located, when the time of looking at the landing point in the line-of-sight deviation area exceeds the given time threshold, it is determined as mild fatigue, and the alarm module sends out the set alarm signal; otherwise, it is determined as normal driving. 2. a detection method that adopts the fatigue driving detection device of claim 1, is characterized in that, comprises the following steps: S1: Use the image acquisition device to collect the driver's driving image, the image includes the image of the driver when he is awake, obtain the initial value of the eye aspect ratio, and use it as the fatigue detection standard parameter; S2: Use the improved fast face detection algorithm to locate the face position, specifically: first calculate the average hash fingerprint of the driver image to be detected, if there is a difference between the average hash fingerprint and the average hash fingerprint in the cache module by no more than 2 bits If the fingerprint information is stored in the cache, the face location frame corresponding to the fingerprint in the cache is directly output, and the number of calls of the fingerprint in the cache is increased by 1; If there is fingerprint information that differs from the average hash fingerprint by more than 2 bits but less than 5 bits in the cache module, call the AdaBoost face detection algorithm to get the positioning frame, and judge whether the cache capacity is full. The hash fingerprint is stored in the cache database; when the cache capacity is full, the fingerprint with the smallest number of calls in the cache is deleted, and then the current average hash fingerprint is stored in the cache database; If the difference between the fingerprint information in the cache module and the mean hash fingerprint is more than 5 bits, call the AdaBoost face detection algorithm to obtain the positioning frame; S3: When a face is detected, use the face feature point positioning algorithm to obtain 68 face feature point positions; S4: Extract the eye information according to the position of the facial feature points, calculate the eye aspect ratio, calculate the ratio of the number of closed eye frames per unit time to the total number of frames, the PERCLOS value and the blinking frequency, and determine the PERCLOS value T and the given threshold. relationship: when T ≥ T _Z , it is judged as severe fatigue, T _Z is the threshold of severe fatigue, and the alarm module sends out the set alarm signal; when T _Q < T < T _Z , T _Q is the threshold of mild fatigue, and the execution S5; when T≤T _Q , execute step S6; S5: further determine whether the blinking frequency is too fast: when the blinking frequency is greater than the given blinking frequency threshold, it is determined as normal driving, and step S6 is performed; otherwise, it is determined as mildly fatigued driving, and the alarm module sends a set alarm signal; S6: Calculate the head posture angle. When the angle in the vertical direction of the head posture is greater than the given angle threshold, it is determined as bowing, and when the time when the angle in the vertical direction of the head posture is greater than the given angle threshold is greater than the given time threshold, If it is determined to be severe fatigue, the alarm module sends a set alarm signal; otherwise, step S7 is performed; S7: Determine whether the eyes are closed according to the aspect ratio of the eyes: when the eyes are closed, the distraction detection cannot be performed, and it is determined as normal driving; when the eyes are not closed, step S8 is performed; S8: Perform distraction detection, calculate the horizontal declination angle θ _l and vertical declination angle θ _v , divide the area in front of the driver into a normal gaze area and a deviated gaze area, and obtain the driver's gaze according to θ _l and θ _v In the area where the landing point is located, when the time of looking at the landing point in the line-of-sight deviation area exceeds the given time threshold, it is determined as mild fatigue, and the alarm module sends out the set alarm signal; otherwise, it is determined as normal driving. 3. A detection method using the fatigue driving detection device according to claim 1 according to claim 2, characterized in that: preprocessing the driving image of the driver in S1, including adaptive median filtering and based on Laplace's image enhancement. 4. a kind of detection method that adopts the fatigue driving detection device of claim 1 according to claim 2, is characterized in that: the described use face feature point positioning algorithm of S3, obtains 68 feature point coordinates specific for: First, use the HOG-SVM algorithm to judge the face orientation: select different face orientations, extract the HOG feature of the image to be detected to obtain the HOG feature vector, and use it as the SVM input parameter to obtain the face orientation estimation classifier; According to the different face orientation results, different face feature point initialization models are selected; For each face feature point position, the trained random forest is used to extract the LBF feature, and the trained linear regressor is used to obtain the regression result to update the face feature point position until the given maximum number of iterations is reached, and the face feature point position is output. 5 . A detection method using the fatigue driving detection device according to claim 1 according to claim 4 , wherein the extraction of LBF features by using the trained random forest is to extract two pixels near the feature point. 6 . The grayscale is normalized as a classification feature. 6. A detection method using the fatigue driving detection device according to claim 1 according to claim 2, characterized in that: the calculation of the eye aspect ratio in S4 is specifically: The eye aspect ratio WHR satisfies:

Among them, W is the eye width, H is the height, and WHR _init is the initial value of the eye aspect ratio when the driver is awake; The ratio of the number of closed-eye frames to the total number of frames in the calculation unit time in S4 PERCLOS value T is specifically: When the WHR is greater than the given threshold, it is determined that the human eye is in the closed state, the total number of frames per unit time is Z, and the number of frames in the closed state of the human eye is z, then T satisfies:

The calculation of the blinking frequency described in S4 is specifically: If the WHR of the current frame image is less than or equal to 3, and the WHR of the current frame image is greater than 3, it is determined that one blink occurs, and the number of blinks in a given unit time is the blink frequency. 7. A detection method using the fatigue driving detection device according to claim 1 according to claim 2, characterized in that: the calculation of the head attitude angle described in step S6 is specifically: After obtaining the position of the facial feature points, by solving the rotation and translation matrices, the two-dimensional N feature points on the image are mapped to the corresponding N feature points on the standard three-dimensional face model, and finally the rotation matrix R is used to calculate the head. Attitude angles α, β, γ:

Among them, β is the angle of the vertical direction of the head posture, α is the angle of the horizontal direction of the head posture, and γ is the angle of the front and rear directions of the head posture. 8. a kind of detection method that adopts the fatigue driving detection device according to claim 1 according to claim 2, is characterized in that: carry out distraction detection described in S8, calculate and obtain line of sight horizontal declination angle θ ₁ and line of sight vertical The declination angle θ _v is specifically: For the displacement vector of a point in the pupil and the boundary point of the pupil and the gradient vector of the boundary point, the point on the pupil corresponding to the maximum value of the inner product of the two is the center of the pupil. The Purkin light spot is formed by 4 infrared LED light sources distributed around the area in front of the driver. Calculate the relative position of the pupil center and the center of the rectangle surrounded by the four Purkin light spots, and obtain the horizontal declination angle of the two.

The vertical declination angle η, and then use the head posture α, β to correct the line of sight direction, and the final horizontal declination angle of the line of sight is obtained as

The vertical deflection angle is θ _v =η+β.

Images