CN114332829A - Driver fatigue detection method based on multiple strategies - Google Patents

Driver fatigue detection method based on multiple strategies Download PDF

Info

Publication number
CN114332829A
CN114332829A CN202111479261.2A CN202111479261A CN114332829A CN 114332829 A CN114332829 A CN 114332829A CN 202111479261 A CN202111479261 A CN 202111479261A CN 114332829 A CN114332829 A CN 114332829A
Authority
CN
China
Prior art keywords
fatigue
mouth
eye
driver
eyes
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202111479261.2A
Other languages
Chinese (zh)
Inventor
仲从建
李征
付本刚
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Jiangsu Aerospace Dawei Technology Co Ltd
Original Assignee
Jiangsu Aerospace Dawei Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Jiangsu Aerospace Dawei Technology Co Ltd filed Critical Jiangsu Aerospace Dawei Technology Co Ltd
Priority to CN202111479261.2A priority Critical patent/CN114332829A/en
Priority to PCT/CN2022/081162 priority patent/WO2023103206A1/en
Publication of CN114332829A publication Critical patent/CN114332829A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G06T7/12—Edge-based segmentation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/20—Image preprocessing
    • G06V10/22—Image preprocessing by selection of a specific region containing or referencing a pattern; Locating or processing of specific regions to guide the detection or recognition
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/20—Image preprocessing
    • G06V10/26—Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features
    • G06V10/44—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/764—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using classification, e.g. of video objects
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
    • G06V10/774—Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/50—Context or environment of the image
    • G06V20/59—Context or environment of the image inside of a vehicle, e.g. relating to seat occupancy, driver state or inner lighting conditions
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/16—Human faces, e.g. facial parts, sketches or expressions
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/18—Eye characteristics, e.g. of the iris
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/70—Multimodal biometrics, e.g. combining information from different biometric modalities

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Multimedia (AREA)
  • General Health & Medical Sciences (AREA)
  • Health & Medical Sciences (AREA)
  • Evolutionary Computation (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Computing Systems (AREA)
  • Software Systems (AREA)
  • Artificial Intelligence (AREA)
  • Human Computer Interaction (AREA)
  • Databases & Information Systems (AREA)
  • Medical Informatics (AREA)
  • Biophysics (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Biomedical Technology (AREA)
  • Oral & Maxillofacial Surgery (AREA)
  • Computational Linguistics (AREA)
  • Data Mining & Analysis (AREA)
  • Molecular Biology (AREA)
  • General Engineering & Computer Science (AREA)
  • Mathematical Physics (AREA)
  • Ophthalmology & Optometry (AREA)
  • Image Analysis (AREA)

Abstract

The invention belongs to the field of artificial intelligence, and discloses a driver fatigue detection method based on multiple strategies, which comprises the following steps: calibrating a target area and inputting a related threshold; segmenting the target region image by using a modified DeepLabV3+ algorithm to obtain the outlines of the eyes and the mouth; calculating the correlation values of the eyes and the mouth according to the contour, and judging the states of the eyes and the mouth; and judging whether the driver is in fatigue driving or not according to the self-adaptive fatigue judgment rule and the eye and mouth states. The invention can better acquire the information of eyes and mouth and obtain more accurate contour. Performing rectangular fitting on the divided outline, and eliminating interference target information; the condition of eye closure or yawning misjudgment caused by the factors of small eyes, large mouth and the like of a driver is effectively avoided; the self-adaptive fatigue judgment rule avoids misjudgment caused by insufficient information, insufficient use of calculation information and the like.

Description

Driver fatigue detection method based on multiple strategies
Technical Field
The invention belongs to the technical field of artificial intelligence, and particularly relates to a driver fatigue detection method based on multiple strategies.
Background
With the development of economy and the improvement of quality of life, automobiles are gradually becoming one of the main vehicles for traveling and also becoming a serious problem on roads all over the world. The driving technique and the driving state of the automobile driver directly influence the safety of the automobile driver and others, such as long-distance travel, insufficient sleep and monotonous environment can cause dangerous vigilance reduction and even micro sleep attack, and fatigue driving is performed subconsciously and becomes one of the main causes of traffic accidents. Therefore, the driving state of the driver is detected and early-warned, the traffic accident rate is reduced, the road safety is promoted, and the method has important significance in reality.
With the rapid development of computers and artificial intelligence technologies, deep learning algorithms are richer, and the application field is continuously expanded, so that the intelligent driving and driver fatigue detection technology is rapidly developed. At present, methods for detecting the face of a driver can be divided into a traditional algorithm and a deep learning algorithm, wherein the deep learning algorithm gradually becomes the mainstream and obtains a relatively ideal effect. In the deep learning algorithm, the segmentation, detection and recognition algorithm is gradually applied to the fatigue detection problem of the driver, for example, whether the driver is in a fatigue state is judged according to the closing degree of eyes and the opening and closing degree of mouth of the driver, so that corresponding early warning is made, and dangerous driving behaviors are avoided.
Although the existing algorithm achieves a certain effect, due to the reasons of complex cab environment, special detection target and the like, misjudgment is easy to cause early warning, and the driving state of a driver is influenced. For existing problems, the existing problems can be summarized into three aspects, firstly, when the eye and mouth contours are segmented by using a deep learning algorithm, the eye and mouth contours are easily interfered by the environment of a cab, and redundant targets can be mixed in the segmentation result; secondly, the segmentation algorithm for the cab scene is less, and the difficulty is higher when the eye and mouth features of the driver are extracted; thirdly, the fatigue state judgment rule has poor adaptivity, and is easy to cause misjudgment when being applied to drivers with small eyes and drivers wearing masks. Therefore, a fatigue detection method for simultaneously solving the three problems is needed, so that the detection accuracy is improved, and the road safety is promoted.
Disclosure of Invention
Aiming at the problems mentioned in the background technology, the invention provides a driver fatigue detection method based on multiple strategies. In order to solve the first problem, a first strategy is proposed, in which the range of movement of the driver's head is calibrated when the device is installed, the upper part of the seat in the cab is used as a reference object, the range of seat variability is used as a calibration area, and the calibration area is analyzed as input data to reduce the interference of a complex environment on a target. Aiming at the second problem, on the basis of the DeepLabV3+ algorithm, the eye and mouth segmentation characteristics are combined, and the algorithm is improved, so that more target information can be obtained, and more accurate eye and mouth contours can be obtained. Aiming at the third problem, firstly, combining the advantages of human-computer interaction, inputting judgment threshold values of eye closure and yawning, then, applying the proposed self-adaptive judgment rule, calculating eye closure degree or mouth opening degree according to whether the mask is worn, judging whether the eyes are closed or the yawning is performed according to the input threshold values, meanwhile, judging whether the driving is fatigue according to the fatigue judgment rule, and outputting a corresponding result.
Specifically, the method for detecting the fatigue of the driver based on the multiple strategies comprises the following steps:
calibrating a target area and inputting a related threshold;
segmenting the target region image by using a modified DeepLabV3+ algorithm to obtain the outlines of the eyes and the mouth;
calculating the correlation values of the eyes and the mouth according to the contour, and judging the states of the eyes and the mouth;
and judging whether the driver is in fatigue driving or not according to the self-adaptive fatigue judgment rule and the eye and mouth states.
Further, the calibrating the target area and the input correlation threshold value comprise:
when the equipment is installed, the seat of the cab and the head movement area of a driver are calibrated through a man-machine interaction mode, and the area is used as subsequent input data for analysis;
and calculating an eye closing threshold value and a yawning threshold value according to the face condition of the driver, and judging the states of the eyes and the mouth of the driver.
Further, the improved DeepLabV3+ algorithm segments the target area image, and comprises the following steps:
analyzing the input image by using a pre-trained resnet101 algorithm and extracting features;
the original coding module is improved by analyzing the characteristics of eyes and mouths, convolution layers are added, parameter values are modified to construct a new cavity convolution structure, and a new coding module is constructed to obtain richer target information;
and obtaining the specific contour of the target, namely obtaining the contour of the eyes and the mouth through a decoding module.
Further, the calculating the correlation value between the eyes and the mouth and the judging the states of the eyes and the mouth include:
performing rectangle fitting on the obtained contour to obtain corresponding height and width;
and then calculating the eye closure degree and the mouth opening degree according to the practical application and the feedback of the driver, wherein the calculation formula is as follows:
Figure BDA0003394381910000031
Figure BDA0003394381910000032
wherein b is eye closure degree, o is mouth opening degree, h is profile height, w is profile width, and i is the number of profiles of the same type;
finally, judging the states of eyes and mouth according to the input threshold value, wherein the judgment rule is as follows:
Figure BDA0003394381910000033
wherein alpha is an eye closing threshold value of the driver, the eye is closed when the closure degree is smaller than the closing threshold value, beta is an mouth opening degree threshold value of the driver, and yawning is realized when the opening degree is larger than the opening threshold value.
Further, the adaptive fatigue determination rule is:
when the mask is worn, the fatigue state is judged only by using the eye fatigue, and if the mask is not worn, the fatigue state is comprehensively judged by the eye fatigue and the fatigue calculated by the yawning times.
Further, the eye fatigue calculation formula is as follows:
Figure BDA0003394381910000041
wherein p iseFor fatigue calculated from eye state, TeNumber of frames in the detection period for which the eyes are closed, TallIs the total number of frames of the detection period.
Further, the formula for calculating the fatigue degree by the yawning times is as follows:
Figure BDA0003394381910000042
wherein p ismFor fatigue calculated from yawning, TmFor detecting the number of frames yawned in a period, TallIs the total number of frames of the detection period.
Further, if the mask is not worn, the fatigue state is comprehensively judged according to the fatigue degree calculated by the eye fatigue degree and the yawning times as follows:
Figure BDA0003394381910000043
wherein p iseFor eye fatigue, pmFatigue calculated for the number of yawns.
Compared with the prior art, the invention has the following advantages:
firstly, the method comprises the following steps: the invention calibrates the head activity area of the driver and the input judgment threshold value in a man-machine interaction mode, can effectively reduce the influence of the complex environment of the cab on the algorithm performance, and simultaneously inputs the relevant threshold value according to the condition of the driver to ensure that the judgment rule is suitable for different drivers.
Secondly, the method comprises the following steps: according to the method, the characteristics of the eyes and the mouth are analyzed, and the DeepLabV3+ algorithm is improved, so that the information of the eyes and the mouth can be better acquired, the segmentation precision is effectively improved, and a more accurate outline is acquired.
Thirdly, the method comprises the following steps: the invention carries out rectangle fitting on the segmented outline, eliminates the interference target information, calculates the height and the width of the outline, provides a novel calculation method of the eye closure degree and the mouth opening degree, combines the input judgment threshold value, and can effectively avoid the condition of eye closure or yawning misjudgment caused by the factors of small eyes, large mouth and the like of a driver.
Fourthly: the invention provides a self-adaptive fatigue judgment rule according to whether a mask is worn or not, namely different fatigue judgment rules are selected according to whether mouth information is detected or not, so that single-factor judgment can be carried out, comprehensive judgment can also be carried out by combining multiple factors, and misjudgment caused by insufficient information, insufficient calculation information and other reasons can be avoided.
Drawings
FIG. 1 is a flow chart of the present invention;
fig. 2(a) is a real-time camera shooting area of the DSM camera, and fig. 2(b) is a calibration area with the upper part of the driver's seat as a reference, which is used as an input image for the next processing;
FIG. 3(a) is the DeepLabV3+ encoding module, and FIG. 3(b) is the DeepLabV3+ algorithm encoding module modified after analysis of the ocular and mouth features;
FIG. 4 shows a comparison of the performance of the segmentation algorithm, where FIG. 4(a) shows the original image, FIG. 4(b) shows the unet segmentation result,
FIG. 4(c) shows the DeepLabV3+ segmentation results, and FIG. 4(d) shows the segmentation results of the improved algorithm;
FIG. 5 is a schematic diagram of the rectangle fitting and height and width determination of the divided outline, wherein the height of the rectangle after fitting is the height of the outline, and the width of the rectangle is the width of the outline;
fig. 6 is a schematic diagram of eye-closing threshold selection, fig. 6(a) shows the eye in a normal state, fig. 6(b) shows the segmentation result of the eye in a normal state, fig. 6(c) shows the eye-closing threshold selection state, and fig. 6(d) shows the segmentation result of the eye-closing threshold state;
fig. 7 is a schematic diagram of threshold value selection for yawning, where fig. 7(a) shows a mouth in a yawning state, fig. 7(b) shows a segmentation result of the yawning state of the mouth, fig. 7(c) shows a yawning threshold value selection state, and fig. 7(d) shows a segmentation result of the yawning threshold value state.
Detailed Description
The invention is further described with reference to the accompanying drawings, but the invention is not limited in any way, and any alterations or substitutions based on the teaching of the invention are within the scope of the invention.
Referring to fig. 1, the embodiment of the present invention includes the following specific steps:
(1) a DSM camera is installed in a vehicle cab, a target area is calibrated and a judgment threshold value, such as an eye closing threshold value alpha and a mouth opening threshold value beta, is calculated through a man-machine interaction mode, and the method refers to fig. 2, fig. 6 and fig. 7;
(2) the input image is segmented using the modified DeepLabV3+ algorithm to extract the eye and mouth contours. The method is specifically divided into three stages:
firstly, using ResNet101 algorithm to extract image characteristics, then obtaining rich target information through a coding module, finally obtaining a corresponding target contour image through a decoding module, and outputting the segmented contour image. The encoder module is improved on the basis of the DeepLabV3+ by combining the segmented eye and mouth feature information, and as shown in FIG. 3, two feature fusion layers, namely two 1x1 convolution layers, are added, so that the target information can be used more effectively, and the final segmentation precision is improved. ResNet is an abbreviation of Residual Network, which is widely used in the field of object classification and a part of a computer vision task backbone classical neural Network, ResNet101 is a classical Network therein, ResNet is common knowledge in the field, and the invention is not described in detail. DeepLabV3+ is a branch with a large influence in the semantic segmentation field, and is not described in detail herein.
(3) Judging whether the eyes are closed or yawned, wherein the specific process comprises the following steps:
(3a) a rectangular fit was made to the output profile and the experiment was with reference to figure 5.
(3b) Calculating the eye closure degree and the mouth opening degree, and judging whether the eyes are closed or yawned according to an input threshold value, wherein the formula is as follows:
Figure BDA0003394381910000071
Figure BDA0003394381910000072
wherein b is eye closure degree, o is mouth opening degree, h is contour height, w is contour width, i is the number of same type of contour, for example, 2 for eyes, 1 for mouth.
(3c) Judging whether the eye is closed or yawned according to the input eye closing threshold value alpha and yawning threshold value beta, wherein the judgment rule is as follows:
Figure BDA0003394381910000073
wherein alpha is an eye closing threshold value of the driver, the eye is closed when the closure degree is smaller than the closing threshold value, beta is an mouth opening degree threshold value of the driver, and yawning is realized when the opening degree is larger than the opening threshold value. The eye closure threshold and mouth opening threshold are determined experimentally.
(4) Selecting a corresponding judgment rule to judge whether the driver is fatigue driving, and specifically:
(4a) when the mask is worn, only eye-closing information exists, the fatigue degree is calculated only through eyes, and the formula is as follows:
Figure BDA0003394381910000074
wherein p iseFor fatigue calculated from eye state, TeNumber of frames in the detection period for which the eyes are closed, TallIs the total number of frames of the detection period.
(4b) When the mask is not worn, the eye closing and yawning conditions are combined for comprehensive judgment, and the calculation formula is as follows:
Figure BDA0003394381910000075
wherein p ismFor fatigue calculated from yawning, TmFor detecting the number of frames yawned in a period, TallIs the total number of frames of the detection period.
(4c) Preferably, when only eyes are present, the detection period is set to 60 frames, and fatigue, i.e., p, occurs when the eye-closed state reaches 18 framese≥0.3。
When the mask is not worn, the detection period is set to be 180 frames, and the fatigue state is set to be 48 frames or 60 frames of the eye closure. Therefore, the fatigue state determination rule when the mask is not worn is:
Figure BDA0003394381910000081
the threshold in the rule for determining the fatigue state in this embodiment may also be other values, which is not limited in the present invention.
(5) And (4) if the fatigue state is the fatigue state, early warning is carried out, and if not, the step (2) is returned to continue detection.
Examples
The effect of the invention is further illustrated by the simulation of the following example.
1. Simulation conditions and example sources:
in order to avoid the problems that the road traffic is influenced by demonstration, and the like, required materials come from a laboratory or an internal closed road, all steps are completed in the same computing equipment and environment, and training parameter settings and training data of all algorithms are the same.
2. Emulated content
First, referring to the calibration of fig. 2 and the input of the correlation determination threshold value. Then, the example image is segmented using the Unet, the deplab v3+ and the improved algorithm to obtain the eye and mouth contours, and the result is shown in fig. 4, where fig. 4(a) is the input image, 4(b) is the Unet segmentation result, 4(c) is the deplab v3+ segmentation result, and 4(d) is the segmentation result of the improved algorithm, and the segmentation result includes the eye region contour and the mouth contour region, and the eyebrow, glasses, etc. are also included in the segmentation result of the Unet. And finally, fitting the profile, calculating the closure degree and the opening degree, and judging fatigue according to the selected rule.
3. Analysis of results
In order to more intuitively measure and improve the performance of the algorithm, quantitative analysis is carried out on the segmentation result by adopting a Mean interaction over Union (MIoU), wherein the formula is as follows:
Figure BDA0003394381910000091
wherein p isijThe true value is the number of predicted j for i, k +1 is the number of categories, piiThe number of i is predicted for the fact that i is also the number of i, and the larger the numerical value is, the better the segmentation effect is.
From fig. 4, the improved algorithm effect in the present invention is obviously better than that of unet, and especially for the segmentation of the eye region, the interference of the non-target region to the algorithm performance is avoided. DeepLabV3+ is close to the segmentation effect of the improved algorithm of the present invention, but the effect of the present invention is closer to the input image in terms of details. In order to better measure the performance of the DeepLabV3+ and the improved algorithm of the invention, quantitative analysis is carried out, under the same condition, the MIoU value of the DeepLabV3+ algorithm is 0.864, and the MIoU value of the improved algorithm of the invention is 0.8706, so that the improved algorithm of the invention has the best effect. Referring to fig. 5 and the judgment rule, the segmentation result of the algorithm directly affects the closure degree and opening degree calculation, further affecting the judgment result, and thus, the excellent segmentation result of the algorithm helps to obtain an accurate fatigue judgment result.
Compared with the prior art, the invention has the following advantages:
firstly, the method comprises the following steps: the invention calibrates the head activity area of the driver and the input judgment threshold value in a man-machine interaction mode, can effectively reduce the influence of the complex environment of the cab on the algorithm performance, and simultaneously inputs the relevant threshold value according to the condition of the driver to ensure that the judgment rule is suitable for different drivers.
Secondly, the method comprises the following steps: according to the method, the characteristics of the eyes and the mouth are analyzed, and the DeepLabV3+ algorithm is improved, so that the information of the eyes and the mouth can be better acquired, the segmentation precision is effectively improved, and a more accurate outline is acquired.
Thirdly, the method comprises the following steps: the invention carries out rectangle fitting on the segmented outline, eliminates the interference target information, calculates the height and the width of the outline, provides a novel calculation method of the eye closure degree and the mouth opening degree, combines the input judgment threshold value, and can effectively avoid the condition of eye closure or yawning misjudgment caused by the factors of small eyes, large mouth and the like of a driver.
Fourthly: the invention provides a self-adaptive fatigue judgment rule according to whether a mask is worn or not, namely different fatigue judgment rules are selected according to whether mouth information is detected or not, so that single-factor judgment can be carried out, comprehensive judgment can also be carried out by combining multiple factors, and misjudgment caused by insufficient information, insufficient calculation information and other reasons can be avoided.
The word "preferred" is used herein to mean serving as an example, instance, or illustration. Any aspect or design described herein as "preferred" is not necessarily to be construed as advantageous over other aspects or designs. Rather, use of the word "preferred" is intended to present concepts in a concrete fashion. The term "or" as used in this application is intended to mean an inclusive "or" rather than an exclusive "or". That is, unless specified otherwise or clear from context, "X employs A or B" is intended to include either of the permutations as a matter of course. That is, if X employs A; b is used as X; or X employs both A and B, then "X employs A or B" is satisfied in any of the foregoing examples.
Also, although the disclosure has been shown and described with respect to one or an implementation, equivalent alterations and modifications will occur to others skilled in the art based upon a reading and understanding of this specification and the annexed drawings. The present disclosure includes all such modifications and alterations, and is limited only by the scope of the appended claims. In particular regard to the various functions performed by the above described components (e.g., elements, etc.), the terms used to describe such components are intended to correspond, unless otherwise indicated, to any component which performs the specified function of the described component (e.g., that is functionally equivalent), even though not structurally equivalent to the disclosed structure which performs the function in the herein illustrated exemplary implementations of the disclosure. In addition, while a particular feature of the disclosure may have been disclosed with respect to only one of several implementations, such feature may be combined with one or other features of the other implementations as may be desired and advantageous for a given or particular application. Furthermore, to the extent that the terms "includes," has, "" contains, "or variants thereof are used in either the detailed description or the claims, such terms are intended to be inclusive in a manner similar to the term" comprising.
Each functional unit in the embodiments of the present invention may be integrated into one processing module, or each unit may exist alone physically, or a plurality of or more than one unit are integrated into one module. The integrated module can be realized in a hardware mode, and can also be realized in a software functional module mode. The integrated module, if implemented in the form of a software functional module and sold or used as a stand-alone product, may also be stored in a computer readable storage medium. The storage medium mentioned above may be a read-only memory, a magnetic or optical disk, etc. Each apparatus or system described above may execute the storage method in the corresponding method embodiment.
In summary, the above-mentioned embodiment is an implementation manner of the present invention, but the implementation manner of the present invention is not limited by the above-mentioned embodiment, and any other changes, modifications, substitutions, combinations, and simplifications which do not depart from the spirit and principle of the present invention should be regarded as equivalent replacements within the protection scope of the present invention.

Claims (8)

1. A driver fatigue detection method based on multiple strategies is characterized by comprising the following steps:
calibrating a target area and inputting a related threshold;
segmenting the target region image by using a modified DeepLabV3+ algorithm to obtain the outlines of the eyes and the mouth;
calculating the correlation values of the eyes and the mouth according to the contour, and judging the states of the eyes and the mouth;
and judging whether the driver is in fatigue driving or not according to the self-adaptive fatigue judgment rule and the eye and mouth states.
2. The multi-strategy based driver fatigue detection technique of claim 1, wherein the calibrating the target zone and inputting the relevant threshold comprises:
when the equipment is installed, the seat of the cab and the head movement area of a driver are calibrated through a man-machine interaction mode, and the area is used as subsequent input data for analysis;
and calculating an eye closing threshold value and a yawning threshold value according to the face condition of the driver, and judging the states of the eyes and the mouth of the driver.
3. The multi-strategy based driver fatigue detection technique according to claim 1, wherein the modified deep lab v3+ algorithm segments the target area image, comprising the steps of:
analyzing the input image by using a pre-trained resnet101 algorithm and extracting features;
the original coding module is improved by analyzing the characteristics of eyes and mouths, convolution layers are added, parameter values are modified to construct a new cavity convolution structure, and a new coding module is constructed to obtain richer target information;
and obtaining the specific contour of the target, namely obtaining the contour of the eyes and the mouth through a decoding module.
4. The multi-strategy based driver fatigue detection method technology as claimed in claim 1, wherein the calculating eye and mouth correlation values and the determining eye and mouth states comprise:
performing rectangle fitting on the obtained contour to obtain corresponding height and width;
and then calculating the eye closure degree and the mouth opening degree according to the practical application and the feedback of the driver, wherein the calculation formula is as follows:
Figure FDA0003394381900000021
Figure FDA0003394381900000022
wherein b is eye closure degree, o is mouth opening degree, h is profile height, w is profile width, and i is the number of profiles of the same type;
finally, judging the states of eyes and mouth according to the input threshold value, wherein the judgment rule is as follows:
Figure FDA0003394381900000023
wherein alpha is an eye closing threshold value of the driver, the eye is closed when the closure degree is smaller than the closing threshold value, beta is an mouth opening degree threshold value of the driver, and yawning is realized when the opening degree is larger than the opening threshold value.
5. The multi-strategy based driver fatigue detection method technology as claimed in claim 1, wherein the adaptive fatigue determination rule is:
when the mask is worn, the fatigue state is judged only by using the eye fatigue, and if the mask is not worn, the fatigue state is comprehensively judged by the eye fatigue and the fatigue calculated by the yawning times.
6. The multi-strategy based driver fatigue detection method technology as claimed in claim 5, wherein the eye fatigue calculation formula is:
Figure FDA0003394381900000024
wherein p iseFor fatigue calculated from eye state, TeNumber of frames in the detection period for which the eyes are closed, TallIs the total number of frames of the detection period.
7. The technology of claim 5, wherein the fatigue degree calculated by the yawning times is expressed by the following formula:
Figure FDA0003394381900000025
wherein p ismFor fatigue calculated from yawning, TmFor detecting the number of frames yawned in a period, TallIs the total number of frames of the detection period.
8. The technology of claim 5, wherein if the mask is not worn, the fatigue state is comprehensively determined by the fatigue degree calculated by the eye fatigue degree and the yawning frequency as follows:
Figure FDA0003394381900000031
wherein p iseFor eye fatigue, pmFatigue calculated for the number of yawns.
CN202111479261.2A 2021-12-06 2021-12-06 Driver fatigue detection method based on multiple strategies Pending CN114332829A (en)

Priority Applications (2)

Application Number Priority Date Filing Date Title
CN202111479261.2A CN114332829A (en) 2021-12-06 2021-12-06 Driver fatigue detection method based on multiple strategies
PCT/CN2022/081162 WO2023103206A1 (en) 2021-12-06 2022-03-16 Driver fatigue detection method based on multiple strategies

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202111479261.2A CN114332829A (en) 2021-12-06 2021-12-06 Driver fatigue detection method based on multiple strategies

Publications (1)

Publication Number Publication Date
CN114332829A true CN114332829A (en) 2022-04-12

Family

ID=81049306

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202111479261.2A Pending CN114332829A (en) 2021-12-06 2021-12-06 Driver fatigue detection method based on multiple strategies

Country Status (2)

Country Link
CN (1) CN114332829A (en)
WO (1) WO2023103206A1 (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN118279878A (en) * 2024-03-01 2024-07-02 杭州圆点科技有限公司 Multi-mode physiological information fusion vehicle-mounted driver fatigue state intelligent recognition method and system

Families Citing this family (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117115894A (en) * 2023-10-24 2023-11-24 吉林省田车科技有限公司 Non-contact driver fatigue state analysis method, device and equipment
CN117341715B (en) * 2023-12-05 2024-02-09 山东航天九通车联网有限公司 Vehicle driving safety early warning method based on joint self-checking
CN118494499B (en) * 2024-07-17 2024-10-22 武汉车凌智联科技有限公司 Fatigue driving detection reminding system based on camera

Family Cites Families (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106781282A (en) * 2016-12-29 2017-05-31 天津中科智能识别产业技术研究院有限公司 A kind of intelligent travelling crane driver fatigue early warning system
CN108294759A (en) * 2017-01-13 2018-07-20 天津工业大学 A kind of Driver Fatigue Detection based on CNN Eye state recognitions
CN111753674A (en) * 2020-06-05 2020-10-09 广东海洋大学 Fatigue driving detection and identification method based on deep learning
CN113343926A (en) * 2021-07-01 2021-09-03 南京信息工程大学 Driver fatigue detection method based on convolutional neural network

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN118279878A (en) * 2024-03-01 2024-07-02 杭州圆点科技有限公司 Multi-mode physiological information fusion vehicle-mounted driver fatigue state intelligent recognition method and system

Also Published As

Publication number Publication date
WO2023103206A1 (en) 2023-06-15

Similar Documents

Publication Publication Date Title
CN114332829A (en) Driver fatigue detection method based on multiple strategies
CN105286802B (en) Driver Fatigue Detection based on video information
CN106652465A (en) Method and system for identifying abnormal driving behavior on road
CN112793576B (en) Lane change decision method and system based on rule and machine learning fusion
CN111738337B (en) Driver distraction state detection and identification method in mixed traffic environment
CN101692980A (en) Method for detecting fatigue driving
CN111753674A (en) Fatigue driving detection and identification method based on deep learning
CN110532925B (en) Driver fatigue detection method based on space-time graph convolutional network
CN111460950B (en) Cognitive distraction method based on head-eye evidence fusion in natural driving conversation behavior
CN106570516A (en) Obstacle recognition method using convolution neural network
CN113609482B (en) Back door detection and restoration method and system for image classification model
CN111626272A (en) Driver fatigue monitoring system based on deep learning
CN118238847B (en) Autonomous lane change decision planning method and system adaptive to different driving styles and road surface environments
CN112308005A (en) Traffic video significance prediction method based on GAN
EP3885973A1 (en) Measuring the sensitivity of neural network image classifiers against adversarial attacks
CN113837154B (en) Open set filtering system and method based on multitask assistance
CN114492043A (en) Personalized driver following modeling method considering perception limited characteristics
CN114220158A (en) Fatigue driving detection method based on deep learning
CN112052829B (en) Pilot behavior monitoring method based on deep learning
CN113642114A (en) Modeling method for humanoid random car following driving behavior capable of making mistakes
CN105046285A (en) Abnormal behavior identification method based on motion constraints
CN117333852A (en) Deep learning-based driver safety belt detection method
CN116946183A (en) Commercial vehicle driving behavior prediction method considering driving capability and vehicle equipment
CN116011503A (en) Traffic vehicle multi-mode track prediction system and method based on track primitives
CN115352454A (en) Interactive auxiliary safe driving system

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination