CN114332829A - Driver fatigue detection method based on multiple strategies - Google Patents

Abstract

The invention belongs to the field of artificial intelligence, and discloses a multi-strategy-based driver fatigue detection method, comprising the steps of: calibrating a target area and inputting a relevant threshold; using an improved DeepLabV3+ algorithm to segment images of the target area to obtain eyes and mouths According to the contour, the correlation values of the eyes and mouth are calculated, and the state of the eyes and mouth is judged; according to the adaptive fatigue judgment rule and the state of the eyes and the mouth, it is judged whether the driver is driving fatigued. The present invention better obtains the information of eyes and mouth, and obtains a more accurate outline. Rectangle fitting is performed on the segmented contours to eliminate interfering target information; effectively avoid the misjudgment of closed eyes or yawning caused by factors such as small eyes and large mouths of drivers; adaptive fatigue judgment rules avoid incomplete information and insufficient use Misjudgment caused by calculation information and other reasons.

Description

A multi-strategy-based driver fatigue detection method

technical field

The invention belongs to the technical field of artificial intelligence, and in particular relates to a multi-strategy-based driver fatigue detection method, aiming at the driver fatigue detection method and technology, which can effectively detect the driver's state and promote the driver to drive safely.

Background technique

With the development of economy and the improvement of quality of life, automobiles have gradually become one of the main means of transportation and a serious problem on global roads. The driving skills and driving status of car drivers will directly affect the safety of themselves and others. For example, long-distance travel, lack of sleep and monotonous environment will lead to a dangerous decrease in vigilance and even microsleep attacks. Fatigue driving subconsciously will become a traffic accident. one of the main incentives. Therefore, the detection and early warning of the driver's driving state can help reduce the incidence of traffic accidents and promote safer roads, which is of great significance in reality.

With the rapid development of computer and artificial intelligence technologies, deep learning algorithms have become more abundant, and the application fields have been continuously expanded, which has led to the rapid development of intelligent driving and driver fatigue detection technologies. At present, the methods of driver face detection can be divided into traditional algorithms and deep learning algorithms. Among them, deep learning algorithms have gradually become the mainstream and have achieved relatively ideal results. In the deep learning algorithm, segmentation, detection and recognition algorithms are gradually applied to the problem of driver fatigue detection, such as judging whether the driver is in a fatigued state by the closed degree of the driver's eyes and the degree of opening and closing of the mouth, so as to make corresponding warnings to avoid The occurrence of dangerous driving behavior.

Although the existing algorithm has achieved certain results, due to the complex environment of the cab and the special detection target, it is easy to misjudge and cause early warning, which affects the driving state of the driver. The existing problems can be summarized into three aspects. First, when using the deep learning algorithm to segment the contours of the eyes and mouth, it is easy to be disturbed by the cab environment, and the segmentation results will incorporate redundant targets; second, for the cab scene, the There are few segmentation algorithms, and it is more difficult to extract the features of the driver's eyes and mouth; third, the fatigue state judgment rules are less adaptive, and when applied to drivers with small eyes and drivers wearing masks, It is easy to cause misjudgment. Therefore, there is a need for a fatigue detection method that simultaneously solves the above three problems, improves the detection accuracy, and promotes road safety.

SUMMARY OF THE INVENTION

In view of the problems mentioned in the background art, the present invention proposes a multi-strategy-based driver fatigue detection method. In order to solve the first problem, the first strategy is proposed. When installing the equipment, the moving range of the driver's head is calibrated. The upper part of the seat in the cab is used as the reference object, and the variable range of the seat is used as the calibration area and as the input. The data is analyzed to reduce the interference of the complex environment to the target. For the second problem, based on the DeepLabV3+ algorithm, combined with the characteristics of eye and mouth segmentation, the algorithm is improved so that it can obtain more target information to obtain more accurate eye and mouth contours. For the third question, first, combined with the advantages of human-computer interaction, input the judgment thresholds for closing eyes and yawning, and then applying the proposed adaptive judgment rules to calculate the degree of eye closure or mouth opening according to whether a mask is worn or not. And according to the input threshold to judge whether it is eyes closed and yawning, at the same time, according to the fatigue judgment rules to judge whether it is fatigue driving, and output the corresponding results.

Specifically, the multi-strategy-based driver fatigue detection method proposed by the present invention includes the following steps:

Calibrate the target area and input related thresholds;

Use the improved DeepLabV3+ algorithm to segment the target area image to obtain the contours of the eyes and mouth;

According to the outline, calculate the correlation value of the eyes and the mouth, and judge the state of the eyes and the mouth;

According to the adaptive fatigue determination rule and the state of eyes and mouth, it is determined whether the driver is driving fatigued.

Further, the calibration target area and input related thresholds include:

When installing the equipment, through the human-computer interaction mode, the cab seat and the active area of the driver's head are calibrated, and this area is used as the subsequent input data for analysis;

According to the driver's facial condition, the eye closing threshold and the yawn threshold are calculated, which are used to judge the driver's eyes and mouth status.

Further, the improved DeepLabV3+ algorithm performs segmentation on the target area image, including the following steps:

Analyze the input image and extract features using the pre-trained resnet101 algorithm;

By analyzing the characteristics of the eyes and mouth, the original coding module is improved, a new convolutional structure is constructed by adding a convolution layer and modifying parameter values, and a new coding module is constructed to obtain richer target information;

The specific contour of the target is obtained through the decoding module, that is, the contours of the eyes and mouth are obtained.

Further, calculating the correlation values of the eyes and the mouth, and judging the states of the eyes and the mouth include:

First perform rectangle fitting on the obtained contour to obtain the corresponding height and width;

Then calculate the eye closure and mouth opening according to the actual application and driver feedback. The calculation formula is:

Among them, b is the eye closure degree, o is the mouth opening degree, h is the contour height, w is the contour width, and i is the number of contours of the same type;

Finally, according to the input threshold to determine the state of the eyes and mouth, the judgment rules are:

Among them, α is the driver's eye closing threshold, if the closing degree is less than the closing threshold, the eyes are closed, β is the driver's mouth opening threshold, and if the opening degree is greater than the opening threshold, it is yawning.

Further, the adaptive fatigue determination rule is:

When wearing a mask, only the eye fatigue is used to judge the fatigue state. If the mask is not worn, the fatigue state is comprehensively judged by the fatigue calculated by the eye fatigue and the number of yawns.

Further, the eye fatigue calculation formula is:

Among them, pe is the fatigue degree calculated according to the state of the eyes _{, T e is} the number of frames in which the eyes are closed in the detection period, and T _all is the total number of frames in the detection period.

Further, the formula for calculating the fatigue degree of the number of yawns is:

Among them, p _m is the fatigue degree calculated according to yawning, T _m is the number of yawning frames in the detection period, and T _all is the total number of frames in the detection period.

Further, if the described mask is not worn, the fatigue state comprehensively judged by the fatigue degree calculated by eye fatigue degree and the number of yawns is:

Among them, p _e is the degree of eye fatigue, and p _m is the degree of fatigue calculated by the number of yawns.

Compared with the prior art, the present invention has the following advantages:

First: the present invention calibrates the driver's head activity area and input judgment threshold by means of human-computer interaction, which can effectively reduce the impact of the complex environment of the cab on the performance of the algorithm. for different drivers.

Second: the present invention improves the DeepLabV3+ algorithm by analyzing the features of the eyes and mouth, so that it can better obtain the information of the eyes and the mouth, effectively improve the segmentation accuracy, and obtain a more accurate contour.

Third: the present invention performs rectangle fitting on the segmented contour, removes the interference target information, calculates the height and width of the contour, and proposes a new calculation method for the degree of eye closure and mouth opening, combined with the input judgment threshold, which can Effectively avoid the misjudgment of closed eyes or yawning caused by factors such as the driver's small eyes and large mouth.

Fourth: the present invention proposes an adaptive fatigue judgment rule according to whether or not to wear a mask, that is, according to whether or not mouth information is detected, different fatigue judgment rules can be selected, which can not only make single-factor judgments, but also combine multiple factors to make comprehensive judgments. , which can avoid misjudgments caused by incomplete information and insufficient use of computational information.

Description of drawings

Fig. 1 is the flow chart of the present invention;

Fig. 2(a) is the real-time imaging area of the DSM camera, and Fig. 2(b) is the calibration area with the upper part of the cab seat as the reference object, which is used as the input image for the next step;

Figure 3(a) is the coding module of DeepLabV3+, and Figure 3(b) is the improved DeepLabV3+ algorithm coding module after analyzing the features of eyes and mouth;

Figure 4. Performance comparison of segmentation algorithms. Figure 4(a) is the original image, and Figure 4(b) is the unet segmentation result.

Figure 4(c) is the segmentation result of DeepLabV3+, and Figure 4(d) is the segmentation result of the improved algorithm;

Figure 5 is a schematic diagram of the contour rectangle fitting of the segmented, and the height and width selection determination, wherein, the height of the rectangle after fitting is the height of the contour, and the width of the rectangle is the width of the contour;

Figure 6 is a schematic diagram of the selection of the threshold for closing eyes, Figure 6(a) is the normal state of the eye, Figure 6(b) is the segmentation result of the normal state of the eye, Figure 6(c) is the selection state of the closing threshold, and Figure 6(d) is the closed state Threshold state segmentation result;

Figure 7 is a schematic diagram of yawning mouth threshold selection, Figure 7(a) is the mouth in the yawning state, Figure 7(b) is the segmentation result of the mouth yawning state, Figure 7(c) is the yawning threshold selection state, Figure 7(d) Split the results for the yawn threshold state.

Detailed ways

The present invention is further described below in conjunction with the accompanying drawings, but the present invention is not limited in any way, and any transformation or replacement based on the teachings of the present invention belongs to the protection scope of the present invention.

1, the specific steps of the embodiment of the present invention are as follows:

(1) Install a DSM camera in the vehicle cab, and calibrate the target area and calculate the judgment threshold through human-computer interaction, such as the eye closure threshold α and the mouth opening threshold β, refer to Figure 2, Figure 6 and Figure 7;

(2) Apply the improved DeepLabV3+ algorithm to segment the input image and extract the contours of the eyes and mouth. It is divided into three stages:

Firstly, the ResNet101 algorithm is used to extract image features, and then the rich target information is obtained through the encoding module, and finally the corresponding target contour image is obtained through the decoding module, and the segmented contour image is output. Among them, combined with the segmented eye and mouth feature information, the encoder module is improved on the basis of DeepLabV3+. As shown in Figure 3, two feature fusion layers are added, that is, two 1x1 convolution layers. Use target information more effectively to improve the final segmentation accuracy. ResNet is the abbreviation of Residual Network, which is widely used in the field of target classification and a part of the backbone classical neural network of computer vision tasks. DeepLabV3+ is a branch with a relatively large influence in the field of semantic segmentation, which is not repeated in the present invention.

(3) To judge whether the eyes are closed and yawning, the specific process is as follows:

(3a) Rectangle fitting is performed on the output contour, and the experiment is shown in Figure 5.

(3b) Calculate the degree of eye closure and mouth opening, and judge whether the eyes are closed or yawning according to the input threshold. The formula is as follows:

Among them, b is the eye closure degree, o is the mouth opening degree, h is the contour height, w is the contour width, and i is the number of contours of the same type, such as 2 for eyes and 1 for mouth.

(3c) Judging whether the eyes are closed or yawning according to the input threshold α and yawning threshold β, the judgment rule is:

Among them, α is the driver's eye closing threshold, if the closing degree is less than the closing threshold, the eyes are closed, β is the driver's mouth opening threshold, and if the opening degree is greater than the opening threshold, it is yawning. Eye closure thresholds and mouth opening thresholds are determined experimentally.

(4) Select the corresponding judgment rule to judge whether it is fatigue driving, as follows:

(4a) When wearing a mask, there is only eye closure information, and fatigue is calculated only by eyes. The formula is:

Among them, pe is the fatigue degree calculated according to the state of the eyes _{, T e is} the number of frames in which the eyes are closed in the detection period, and T _all is the total number of frames in the detection period.

(4b) When not wearing a mask, combined with eyes closed and yawning, the calculation formula is:

Among them, p _m is the fatigue degree calculated according to yawning, T _m is the number of yawning frames in the detection period, and T _all is the total number of frames in the detection period.

(4c) Preferably, when there are only eyes, the detection period is set to 60 frames, and the eyes closed state reaches 18 frames as fatigue, that is, p _e ≥ 0.3.

When not wearing a mask, set the detection period to 180 frames, and yawn for 48 frames or close your eyes for 60 frames to be fatigued. Therefore, the rules for judging fatigue status when not wearing a mask are:

The threshold in the rule for judging the fatigue state in this embodiment may also be other values, which are not limited in the present invention.

(5) If it is in the fatigue state, give an early warning, otherwise return to step (2) to continue the detection.

Example

The effect of the present invention is further illustrated by the simulation of the following examples.

1. Simulation conditions and instance sources:

In order to avoid problems such as the need to affect road traffic in the demonstration, the required materials are from the laboratory or internal closed roads, and all steps are completed in the same computing equipment and environment, where the training parameter settings and training data of all algorithms are also the same.

2. Simulation content

First, refer to the calibration and input correlation judgment thresholds of FIG. 2 . Then, use Unet, DeepLabV3+ and the improved algorithm to segment the instance image to obtain the contours of the eyes and mouth. The results are shown in Figure 4, where Figure 4(a) is the input image, and Figure 4(b) is the unet segmentation result , 4(c) are the segmentation results of DeepLabV3+ and 4(d) are the segmentation results of the improved algorithm. The segmentation results include the contour of the eye area and the contour area of the mouth, and the segmentation results of Unet also include eyebrows, glasses, etc. Finally, fit the contour, calculate the closing degree and the closing degree, and make a fatigue judgment according to the selected rule.

3. Analysis of results

In order to measure the performance of the improved algorithm more intuitively, Mean Intersection over Union (MIoU) is used to quantitatively analyze the segmentation results. The formula is:

Among them, p _ij represents the number of the true value i is predicted to be j, k+1 is the number of categories, p _ii is the number of the true value i is predicted to be i, and the larger the value, the better the segmentation effect.

From FIG. 4 , the effect of the improved algorithm in the present invention is obviously better than that of unet, especially the segmentation of the eye area, which avoids the interference of the non-target area on the performance of the algorithm. The segmentation effect of DeepLabV3+ and the improved algorithm of the present invention are similar, but the effect of the present invention is closer to the input image in terms of details. In order to better measure the performance of DeepLabV3+ and the improved algorithm of the present invention, quantitative analysis is carried out. Under the same conditions, the MIoU value of the DeepLabV3+ algorithm is 0.864, and the MIoU value of the improved algorithm in the present invention is 0.8706. The algorithm achieves the best results. Referring to Figure 5 and the judgment rule, the segmentation result of the algorithm directly affects the calculation of the degree of closure and the degree of tension and closure, which further affects the judgment result. Therefore, the excellent algorithm segmentation result helps to obtain accurate fatigue judgment results.

Compared with the prior art, the present invention has the following advantages:

First: the present invention calibrates the driver's head activity area and input judgment threshold by means of human-computer interaction, which can effectively reduce the impact of the complex environment of the cab on the performance of the algorithm. for different drivers.

Second: the present invention improves the DeepLabV3+ algorithm by analyzing the features of the eyes and mouth, so that it can better obtain the information of the eyes and the mouth, effectively improve the segmentation accuracy, and obtain a more accurate contour.

Third: the present invention performs rectangle fitting on the segmented contour, removes the interference target information, calculates the height and width of the contour, and proposes a new calculation method for the degree of eye closure and mouth opening, combined with the input judgment threshold, which can Effectively avoid the misjudgment of closed eyes or yawning caused by factors such as the driver's small eyes and large mouth.

Fourth: the present invention proposes an adaptive fatigue judgment rule according to whether or not to wear a mask, that is, according to whether or not mouth information is detected, different fatigue judgment rules can be selected, which can not only make single-factor judgments, but also combine multiple factors to make comprehensive judgments. , which can avoid misjudgments caused by incomplete information and insufficient use of computational information.

As used herein, the word "preferred" means serving as an example, instance, or illustration. Any aspect or design described herein as "preferred" is not necessarily to be construed as advantageous over other aspects or designs. Rather, use of the word "preferred" is intended to present concepts in a specific manner. The term "or" as used in this application is intended to mean an inclusive "or" rather than an exclusive "or." That is, unless specified otherwise or clear from context, "X employs A or B" is meant to naturally include either of the permutations. That is, "X uses A or B" is satisfied in any of the preceding examples if X uses A; X uses B; or X uses both A and B.

Furthermore, although the present disclosure has been shown and described with respect to one implementation or implementation, equivalent variations and modifications will occur to those skilled in the art based on a reading and understanding of this specification and drawings. The present disclosure includes all such modifications and variations and is limited only by the scope of the appended claims. In particular with respect to the various functions performed by the above-described components (eg, elements, etc.), the terms used to describe such components are intended to correspond to any component that performs the specified function of the component (eg, which is functionally equivalent) (unless otherwise indicated), even if not structurally equivalent to the disclosed structures that perform the functions of the exemplary implementations of the present disclosure shown herein. Furthermore, although a particular feature of the present disclosure has been disclosed with respect to only one of several implementations, such feature may be combined with one or other features of other implementations as may be desired and advantageous for a given or particular application combination. Also, to the extent that the terms "including," "having," "containing," or variations thereof, are used in the detailed description or the claims, such terms are intended to include in a manner similar to the term "comprising."

Each functional unit in this embodiment of the present invention may be integrated into one processing module, or each unit may exist physically alone, or multiple or more units may be integrated into one module. The above-mentioned integrated modules can be implemented in the form of hardware, and can also be implemented in the form of software function modules. If the integrated modules are implemented in the form of software functional modules and sold or used as independent products, they may also be stored in a computer-readable storage medium. The above-mentioned storage medium may be a read-only memory, a magnetic disk or an optical disk, and the like. The above-mentioned apparatuses or systems may execute the storage methods in the corresponding method embodiments.

To sum up, the above-mentioned embodiment is an embodiment of the present invention, but the embodiment of the present invention is not limited by the embodiment, and any other changes that deviate from the spirit and principle of the present invention, Modifications, substitutions, combinations, and simplifications should all be equivalent substitutions, which are all included within the protection scope of the present invention.

Claims (8)

1. a driver fatigue detection method based on multiple strategies, is characterized in that, comprises the following steps: Calibrate the target area and input related thresholds; Use the improved DeepLabV3+ algorithm to segment the target area image to obtain the contours of the eyes and mouth; According to the outline, calculate the correlation value of the eyes and the mouth, and judge the state of the eyes and the mouth; According to the adaptive fatigue determination rule and the state of eyes and mouth, it is determined whether the driver is driving fatigued. 2. driver fatigue detection method technology based on multi-strategy according to claim 1, is characterized in that, described calibration target area and input relevant threshold value comprise: When installing the equipment, through the human-computer interaction mode, the cab seat and the active area of the driver's head are calibrated, and this area is used as the subsequent input data for analysis; According to the driver's facial condition, the eye closing threshold and the yawn threshold are calculated, which are used to judge the driver's eyes and mouth status. 3. driver fatigue detection method technology based on multi-strategy according to claim 1, is characterized in that, described improved DeepLabV3+ algorithm is segmented to target area image, may further comprise the steps: Analyze the input image and extract features using the pre-trained resnet101 algorithm; By analyzing the characteristics of the eyes and mouth, the original encoding module is improved, adding a convolution layer and modifying the parameter values to construct a new atrous convolution structure, and constructing a new encoding module to obtain richer target information; The specific contour of the target is obtained through the decoding module, that is, the contours of the eyes and mouth are obtained. 4. the driver fatigue detection method technology based on multi-strategy according to claim 1, is characterized in that, described calculating the correlation value of eye and mouth, judging eye and mouth state comprises: First perform rectangle fitting on the obtained contour to obtain the corresponding height and width; Then calculate the eye closure and mouth opening according to the actual application and driver feedback. The calculation formula is:

Among them, b is the eye closure degree, o is the mouth opening degree, h is the contour height, w is the contour width, and i is the number of contours of the same type; Finally, according to the input threshold to determine the state of the eyes and mouth, the judgment rules are:

Among them, α is the driver's eye closing threshold, if the closing degree is less than the closing threshold, the eyes are closed, β is the driver's mouth opening threshold, and if the opening degree is greater than the opening threshold, it is yawning. 5. a kind of driver fatigue detection method technology based on multi-strategy according to claim 1, is characterized in that, described self-adaptive fatigue determination rule is: When wearing a mask, only the eye fatigue is used to judge the fatigue state. If the mask is not worn, the fatigue state is comprehensively judged by the fatigue calculated by the eye fatigue and the number of yawns. 6. a kind of driver fatigue detection method technology based on multi-strategy according to claim 5, is characterized in that, described eyestrain calculation formula is:

Among them, pe is the fatigue degree calculated according to the state of the eyes _{, T e is} the number of frames in which the eyes are closed in the detection period, and T _all is the total number of frames in the detection period. 7. a kind of driver fatigue detection method technology based on multi-strategy according to claim 5, is characterized in that, the formula of the degree of fatigue that described yawn times calculates is:

Among them, p _m is the fatigue degree calculated according to yawning, T _m is the number of yawning frames in the detection period, and T _all is the total number of frames in the detection period. 8. a kind of driver fatigue detection method technology based on multi-strategy according to claim 5, is characterized in that, described if not wearing a mask, the fatigue state comprehensively judged by the fatigue degree calculated by eye fatigue degree and the number of yawns is :

Among them, p _e is the degree of eye fatigue, and p _m is the degree of fatigue calculated by the number of yawns.

Images