CN116386277A - Fatigue driving detection method and device, electronic equipment and medium - Google Patents
Fatigue driving detection method and device, electronic equipment and medium Download PDFInfo
- Publication number
- CN116386277A CN116386277A CN202211514826.0A CN202211514826A CN116386277A CN 116386277 A CN116386277 A CN 116386277A CN 202211514826 A CN202211514826 A CN 202211514826A CN 116386277 A CN116386277 A CN 116386277A
- Authority
- CN
- China
- Prior art keywords
- fatigue
- driver
- determining
- face
- voice
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000001514 detection method Methods 0.000 title claims abstract description 43
- 230000001815 facial effect Effects 0.000 claims abstract description 53
- 238000000034 method Methods 0.000 claims abstract description 48
- 208000011293 voice disease Diseases 0.000 claims abstract description 47
- 230000004399 eye closure Effects 0.000 claims description 52
- 238000004590 computer program Methods 0.000 claims description 6
- 206010016256 fatigue Diseases 0.000 description 221
- 230000008569 process Effects 0.000 description 10
- 238000004891 communication Methods 0.000 description 9
- 241001282135 Poromitra oscitans Species 0.000 description 5
- 206010048232 Yawning Diseases 0.000 description 5
- 230000009471 action Effects 0.000 description 4
- 238000010586 diagram Methods 0.000 description 3
- 230000000694 effects Effects 0.000 description 3
- 230000006870 function Effects 0.000 description 3
- 238000012545 processing Methods 0.000 description 3
- 230000009286 beneficial effect Effects 0.000 description 2
- 230000008901 benefit Effects 0.000 description 2
- 230000008859 change Effects 0.000 description 2
- 238000000802 evaporation-induced self-assembly Methods 0.000 description 2
- 210000003128 head Anatomy 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000002093 peripheral effect Effects 0.000 description 2
- 238000003491 array Methods 0.000 description 1
- 230000004397 blinking Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 238000013145 classification model Methods 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 238000013527 convolutional neural network Methods 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 230000007613 environmental effect Effects 0.000 description 1
- 210000004709 eyebrow Anatomy 0.000 description 1
- 210000000887 face Anatomy 0.000 description 1
- 238000005286 illumination Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000010354 integration Effects 0.000 description 1
- 238000003062 neural network model Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
- 230000003867 tiredness Effects 0.000 description 1
- 208000016255 tiredness Diseases 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G08—SIGNALLING
- G08B—SIGNALLING OR CALLING SYSTEMS; ORDER TELEGRAPHS; ALARM SYSTEMS
- G08B21/00—Alarms responsive to a single specified undesired or abnormal condition and not otherwise provided for
- G08B21/02—Alarms for ensuring the safety of persons
- G08B21/06—Alarms for ensuring the safety of persons indicating a condition of sleep, e.g. anti-dozing alarms
Landscapes
- Business, Economics & Management (AREA)
- Emergency Management (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Measurement Of The Respiration, Hearing Ability, Form, And Blood Characteristics Of Living Organisms (AREA)
Abstract
The invention discloses a fatigue driving detection method, a fatigue driving detection device, electronic equipment and a fatigue driving medium, and relates to the technical field of intelligent driving. The method comprises the following steps: acquiring face images and voices of a driver according to a preset acquisition strategy to obtain a face image set and a voice data set; acquiring a face fatigue index of a driver based on the face image set; acquiring a voice fatigue index of a driver based on the voice data set; based on the facial fatigue index and the speech fatigue index, it is determined whether the driver is in fatigue driving. According to the method, the face image set and the voice data set of the driver are respectively analyzed, whether the driver is in fatigue driving is judged from two dimensions of the face fatigue index and the voice fatigue index, and compared with a single-dimension detection method, the accuracy and the reliability of detection can be effectively improved, and missing detection and false detection are reduced.
Description
Technical Field
The invention relates to the technical field of intelligent driving, in particular to a fatigue driving detection method, a fatigue driving detection device, electronic equipment and a medium.
Background
Research shows that most automobile safety accidents are caused by fatigue driving of a driver, and the fatigue driving detection method of the driver is also called a hot spot problem. At present, common fatigue driving detection methods are mainly divided into two types. The first type is mainly to judge whether the driver is tired or not by detecting the body related physiological information such as electrocardiogram, electroencephalogram, electrooculogram and the like of the driver. The second type is to judge whether or not the driving state is tired by the running characteristics of the vehicle such as the running speed of the vehicle, the running track of the tire, the turning angle of the steering wheel, and the like. However, the two detection methods are both based on the characteristic of one dimension, so that the accuracy is low, and the conditions of missing detection, false detection and the like are easy to occur.
Disclosure of Invention
In order to solve the technical problems or at least partially solve the technical problems, embodiments of the present invention provide a method, an apparatus, an electronic device, and a medium for detecting fatigue driving.
In a first aspect, an embodiment of the present invention provides a method for detecting fatigue driving, including: acquiring face images and voices of a driver according to a preset acquisition strategy to obtain a face image set and a voice data set; acquiring a face fatigue index of the driver based on the face image set; acquiring a voice fatigue index of the driver based on the voice data set; based on the facial fatigue index and the speech fatigue index, it is determined whether the driver is in a fatigue driving state.
Optionally, the determining whether the driver is in a fatigue driving state based on the facial fatigue index and the voice fatigue index includes: determining a first weight corresponding to the facial fatigue index; determining a second weight corresponding to the voice fatigue index; calculating a weighted sum of the facial fatigue index and the speech fatigue index based on the first weight and the second weight; based on the weighted sum, it is determined whether the driver is in a state of fatigue driving.
Optionally, the facial fatigue index includes one or more of: eye closure, blink frequency, mouth opening and closing.
Optionally, the determining the first weight corresponding to the facial fatigue index includes: determining a first face weight corresponding to the eye closure degree, a second face weight corresponding to the blink frequency and a third face weight corresponding to the mouth opening and closing degree;
the calculating a weighted sum of the facial fatigue index and the speech fatigue index based on the first weight and the second weight, comprising: calculating a weighted sum of the eye closure, the blink frequency, the mouth opening and closing degree, and the speech fatigue index based on the first face weight, the second face weight, the third face weight, and the second weight;
The determining whether the driver is in a state of fatigue driving based on the weighted sum includes: and determining whether the driver is in a fatigue driving state based on the weighted sum and one of the eye closure degree, the blink frequency and the mouth opening and closing degree.
Optionally, the determining whether the driver is in a fatigue driving state based on the weighted sum and one of the eye closure degree, the blink frequency, and the mouth opening and closing degree includes: determining whether the weighted sum is greater than or equal to a first target threshold; determining that the driver is in fatigue driving if the weighted sum is greater than or equal to the first target threshold; determining if one of the eye closure, the blink frequency, and the mouth opening and closing degree is greater than or equal to a corresponding second target threshold value if the weighted sum is less than the first target threshold value; and determining that the driver is in fatigue driving under the condition that one of the eye closure degree, the blink frequency and the mouth opening and closing degree is greater than or equal to a corresponding second target threshold value.
Optionally, the method further comprises: determining the fatigue level of the driver under the condition that the driver is determined to be in fatigue driving; and outputting fatigue early warning information corresponding to the fatigue grade.
Optionally, the method further comprises: detecting the driving characteristics of a vehicle under the condition of outputting the fatigue early warning information or the fatigue prompt information corresponding to the fatigue grade, and determining whether the vehicle stops driving; and outputting preset warning information under the condition that the vehicle does not stop running.
Optionally, the step of acquiring the face image and the voice of the driver according to a preset acquisition strategy to obtain a face image set and a voice data set includes: determining the running time of a vehicle, and determining an acquisition period and an acquisition duration according to the running time; and acquiring face images and voices of the driver based on the acquisition period and the acquisition time length.
Optionally, the acquiring the face fatigue index of the driver based on the face image set includes:
determining a first target image of which the eyes of a driver are in a closed state in the face image set, and determining the eye closure degree according to the frame number of the first target image and the total frame number of the face image set;
and/or
Determining the blink times of the driver based on the face image set; determining the blink frequency according to the blink times;
And/or
And determining a second target image of which the mouth outline of the driver is in a preset state in the face image set, and determining the mouth opening and closing degree according to the frame number of the second target image and the total frame number of the face image set.
Optionally, the acquiring, based on the voice data, a voice fatigue index of the driver includes: identifying the voice data set, and determining the times of the voice data set belonging to fatigue voice; and determining the voice fatigue index of the driver according to the times of the fatigue voice in the voice data set.
In a second aspect, an embodiment of the present invention further provides a fatigue driving detection apparatus, including:
the acquisition module is used for acquiring face images and voices of a driver according to a preset acquisition strategy to obtain a face image set and a voice data set;
the image recognition module is used for acquiring the face fatigue index of the driver based on the face image set;
the voice recognition module is used for acquiring the voice fatigue index of the driver based on the voice data set;
and the state determining module is used for determining whether the driver is in a fatigue driving state or not based on the facial fatigue index and the voice fatigue index.
Optionally, the state determining module is further configured to: determining a first weight corresponding to the facial fatigue index; determining a second weight corresponding to the voice fatigue index; calculating a weighted sum of the facial fatigue index and the speech fatigue index based on the first weight and the second weight; based on the weighted sum, it is determined whether the driver is in a state of fatigue driving.
Optionally, the state determining module is further configured to: determining a first face weight corresponding to the eye closure degree, a second face weight corresponding to the blink frequency and a third face weight corresponding to the mouth opening and closing degree; calculating a weighted sum of the eye closure, the blink frequency, the mouth opening and closing degree, and the speech fatigue index based on the first face weight, the second face weight, the third face weight, and the second weight; and determining whether the driver is in a fatigue driving state based on the weighted sum and one of the eye closure degree, the blink frequency and the mouth opening and closing degree.
Optionally, the state determining module is further configured to: determining whether the weighted sum is greater than or equal to a first target threshold; determining that the driver is in fatigue driving if the weighted sum is greater than or equal to the first target threshold; determining if one of the eye closure, the blink frequency, and the mouth opening and closing degree is greater than or equal to a corresponding second target threshold value if the weighted sum is less than the first target threshold value; and determining that the driver is in fatigue driving under the condition that one of the eye closure degree, the blink frequency and the mouth opening and closing degree is greater than or equal to a corresponding second target threshold value.
Optionally, the state determining module is further configured to determine a fatigue level of the driver if it is determined that the driver is in fatigue driving; and outputting fatigue early warning information corresponding to the fatigue grade.
Optionally, the device further comprises a driving detection module for: detecting the driving characteristics of a vehicle under the condition of outputting the fatigue early warning information or the fatigue prompt information corresponding to the fatigue grade, and determining whether the vehicle stops driving; and outputting preset warning information under the condition that the vehicle does not stop running.
Optionally, the acquisition module is configured to: determining the running time of a vehicle, and determining an acquisition period and an acquisition duration according to the running time; and acquiring face images and voices of the driver based on the acquisition period and the acquisition time length.
Optionally, the image recognition module is further configured to: determining a first target image of which the eyes of a driver are in a closed state in the face image set, and determining the eye closure degree according to the frame number of the first target image and the total frame number of the face image set; and/or determining the blink times of the driver based on the face image set; determining the blink frequency according to the blink times; and/or determining a second target image of which the outline of the mouth of the driver is in a preset state in the face image set, and determining the mouth opening and closing degree according to the frame number of the second target image and the total frame number of the face image set.
Optionally, the voice recognition module is further configured to: identifying the voice data set, and determining the times of the voice data set belonging to fatigue voice; and determining the voice fatigue index of the driver according to the times of the fatigue voice in the voice data set.
In a third aspect, an embodiment of the present invention further provides an electronic device, including: one or more processors; and the storage device is used for storing one or more programs, and when the one or more programs are executed by the one or more processors, the one or more processors are enabled to realize the fatigue driving detection device according to any embodiment of the invention.
In a fourth aspect, an embodiment of the present invention further provides a computer readable medium having stored thereon a computer program, which when executed by a processor, implements the fatigue driving detection device according to any of the embodiments of the present invention.
One embodiment of the above invention has the following advantages or benefits:
according to the fatigue driving detection method, the face fatigue index and the voice fatigue index are obtained by respectively analyzing the face image set and the voice data set of the driver, the face fatigue index and the voice fatigue index are comprehensively analyzed to judge whether the driver is in fatigue driving or not, and the states of the driver are judged from two dimensions.
Further effects of the above-described non-conventional alternatives are described below in connection with the embodiments.
Drawings
The drawings are included to provide a better understanding of the invention and are not to be construed as unduly limiting the invention. Wherein:
FIG. 1 shows a flow chart of a fatigue driving detection method of an embodiment of the present invention;
FIG. 2 is a flow chart of a fatigue driving detection method according to another embodiment of the present invention;
FIG. 3 is a flowchart of a fatigue driving detection method according to another embodiment of the present invention;
fig. 4 shows a schematic structural diagram of a fatigue driving detection device according to an embodiment of the present invention;
fig. 5 shows a schematic structural diagram of an electronic device according to an embodiment of the present invention.
Detailed Description
Exemplary embodiments of the present invention will now be described with reference to the accompanying drawings, in which various details of the embodiments of the present invention are included to facilitate understanding, and are to be considered merely exemplary. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the invention. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.
According to the embodiment of the invention, if the driver is tired in the process of driving the vehicle, obvious change of the facial features of the driver is considered, and the voice information representing the fatigue is also sent out, if the facial features of the driver are captured by utilizing the image acquisition and recognition device and the voice features of the driver are captured by utilizing the voice acquisition and recognition device, the facial features and the voice features are used as the basis for comprehensively judging whether the driver is tired or not, so that the detection accuracy and reliability can be effectively improved.
Fig. 1 shows a flowchart of a fatigue driving detection method according to an embodiment of the present invention, as shown in fig. 1, the method includes:
step S101: and acquiring face images and voices of the driver according to a preset acquisition strategy to obtain a face image set and a voice data set.
In the present embodiment, the face image of the driver may be acquired by the vehicle-mounted camera, and the voice of the driver may be acquired by the vehicle-mounted voice receiving device such as the vehicle-mounted microphone.
In an alternative embodiment, the face image and voice of the driver during driving can be acquired in real time.
In other alternative embodiments, the face image and voice of the driver may be acquired at intervals. For example, the interval of collection may be longer when the vehicle has just started to travel, and shorter when the vehicle has traveled for a longer period of time, that is, the collection period may be determined according to the travel time of the vehicle. As a specific example, the present embodiment may start timing at the time of starting the vehicle to determine the running time of the vehicle, and may collect every 20 minutes during a period of time in which the running time of the vehicle is less than 1 hour, collect every 10 minutes during a period of time in which the running time of the vehicle is 1 hour to 2 hours, collect every 5 minutes during a period of time in which the running time of the vehicle is 2 hours to 3 hours, and collect in real time when the running time of the vehicle is 4 hours or more.
In the process of intermittently acquiring the face image and the voice of the driver, the duration of each acquisition may be fixed, for example, two minutes of face image and voice are acquired each time as the face image set and the voice data set. The duration of the collection may also be dynamically variable, for example, when the vehicle is just started to travel, the duration of the collection may be longer, and when the vehicle is traveling for a longer period of time, the duration of the collection may be shorter, i.e., the duration of the collection may be determined according to the travel time of the vehicle. As a specific example, the present embodiment may start timing at the time of starting the vehicle to determine the travel time of the vehicle. The method comprises the steps of collecting face images and voices every 20 minutes in a time period that the running time of a vehicle is less than 1 hour, collecting face images and voices every 4 minutes, collecting face images and voices every 10 minutes in a time period that the running time of the vehicle is 1 hour to 2 hours, collecting face images and voices every 3 minutes, collecting face images and voices every 5 minutes in a time period that the running time of the vehicle is 2 hours to 3 hours, collecting face images and voices every 1 minute, and collecting face images and voices in real time when the running time of the vehicle is more than 4 hours.
After the face image set and the voice data set of the driver are collected, the face image set and the voice data set can be preprocessed, and the preprocessed face image set and voice data set are used as data to be analyzed.
Step S102: and acquiring the face fatigue index of the driver based on the face image set.
Wherein the facial fatigue index is used for representing the fatigue degree of the driver from the facial feature dimension, and the higher the facial fatigue index is, the higher the fatigue degree of the driver is. In this step, the pre-trained neural network model may be used to identify the face key points in the face image, such as eyebrows, eyes, mouth, nose, etc., and obtain the face fatigue index (may also be referred to as the face fatigue index) of the driver according to the features of the face key points. Specifically, the step can utilize a pre-trained face detection model to locate faces in a face image set, and utilize a face key point locating model to identify face key points. As a specific example, the face image may be subjected to face positioning and face key point positioning by using a pre-trained MTCNN network (Multi-task Convolutional Neural Network ), a positioned face feature image is output, and a face fatigue index is calculated based on the face feature image. The MTCNN network is a multi-task cascade convolution network, and is composed of P-Net, R-Net and O-Net, so that the MTCNN network can simultaneously, rapidly and accurately perform face detection and face key point positioning, and compared with other face detection models, the model has better accuracy and instantaneity, has better robustness on illumination, background and other influences, and is beneficial to improving the speed and accuracy of face image recognition.
The embodiment of the invention considers that under the condition of fatigue driving of a driver, the facial features of the driver can be obviously changed, and most obvious is that eyes are obviously closed, blinks frequently, the mouth opening degree caused by yawning becomes large and the head sags. Accordingly, the facial fatigue index of embodiments of the present invention may include one or more of eye closure, blink frequency, mouth opening and closing.
As an example, the eye closure, blink frequency, and mouth opening and closing degree of the driver may be determined according to the following procedure:
determining a first target image of which the eyes of a driver are in a closed state in the face image set, and determining the eye closure degree according to the frame number of the first target image and the total frame number of the face image set;
determining the blink times of a driver based on the face image set; determining blink frequency according to blink times;
and determining a second target image of which the mouth outline of the driver is in a preset state in the face image set, and determining the mouth opening degree according to the frame number of the second target image and the total frame number of the face image set.
Eye closure represents the proportion of the image that the eye is closed to within a given time (i.e., the target period of time). Let N be close Representing the total number of frames in which the eye closure is in state, N, for a given time total Representing the total number of frames of face images in a given time interval, the eye closure degree P is calculated according to the following equation:
the blink frequency identifies the proportion of blink times in a given time. Let N be b Indicating the number of blinks in a given time interval, N l Representing the total number of frames of face images in a given time interval, the blink frequency f BF Calculated according to the following formula:
mouth opening and closing degree means the proportion of an image that is significantly larger in the mouth in a given time. Let N be m Representing the number of frames, N, of significantly larger mouth portions in a given time interval h Representing the total number of frames of face images within a given time interval, the mouth opening degree OC rate Calculated according to the following formula:
the eye closing, blinking and mouth opening images can be determined according to the positions of the positioned eye key points and the positioned mouth key points.
Step S103: and acquiring the voice fatigue index of the driver based on the voice data set.
According to the embodiment of the invention, if a driver has fatigue driving in the process of driving the vehicle, the driver can send out voice information representing fatigue, such as: ' I am good o ', ' good o! The driver directly sounds a yawning voice when he wants to sleep. Such voice information indicative of fatigue may be used to determine whether the driver is driving fatigue. Therefore, the embodiment of the invention collects the voice data of the driver, analyzes the voice data to obtain the voice fatigue index, and takes the voice fatigue index as a basis for judging whether the driver is in fatigue driving or not. The voice fatigue index characterizes the fatigue degree of the driver from the voice characteristic dimension, and the higher the voice fatigue index is, the higher the fatigue degree of the driver is.
In an alternative embodiment, the driver's voice fatigue index may be determined according to the following procedure:
identifying the voice data set, and determining the times of the voice data set belonging to fatigue voice;
and determining the voice fatigue index of the driver according to the times of the fatigue voice in the voice data set.
Specifically, the voice in the voice data set can be subjected to text recognition by using the voice recognition model, the voice text corresponding to the voice data is determined, and whether the recognized voice text belongs to a preset fatigue voice text or not is judged. The preset fatigue voice text is a voice text which can be preset and represents fatigue, such as "good tiredness". Meanwhile, the voice in the voice data set can be classified by utilizing the voice classification model, and the yawning sound in the voice data can be determined. Then, the number of times of fatigue-indicating voices and the number of times of yawning occurring in the voice data set are counted, and the sum of the number of times of fatigue-indicating voices and the number of times of yawning is taken as the number of times C of fatigue-indicating voices in the voice data set tired Substituting the following formula to calculate to obtain the voice fatigue index S:
where C represents the length of a voice frame in the voice data set, and may also represent the duration of the target time period.
Step S104: based on the facial fatigue index and the speech fatigue index, it is determined whether the driver is in a fatigue driving state.
In an alternative embodiment, it may be determined whether the driver is in fatigue driving according to the following procedure:
respectively determining a first threshold corresponding to the facial fatigue index and a second threshold corresponding to the voice fatigue index;
comparing the facial fatigue index to a first threshold and comparing the voice fatigue index to a second threshold;
and determining that the driver is in fatigue driving under the condition that the facial fatigue index is greater than or equal to a first threshold value and the voice fatigue index is greater than or equal to a second threshold value.
Wherein, the facial fatigue index can be any one or more of eye closure, blink frequency and mouth opening and closing degree. When the facial fatigue index includes a plurality of eye closure, blink frequency, mouth opening and closing, different facial fatigue indexes correspond to different thresholds. For example, the facial fatigue index includes an eye closure degree and a mouth opening degree, and the eye closure degree and the mouth opening degree correspond to different thresholds, respectively.
In an alternative embodiment, it may also be determined whether the driver is in fatigue driving according to the following procedure:
Determining a first weight corresponding to the facial fatigue index;
determining a second weight corresponding to the voice fatigue index;
calculating a weighted sum of the facial fatigue index and the speech fatigue index based on the first weight and the second weight;
based on the weighted sum, it is determined whether the driver is in fatigue driving.
The facial fatigue index may include any one or more of eye closure, blink frequency, and mouth opening and closing. When the facial fatigue index includes a plurality of eye closure, blink frequency, mouth opening and closing, different facial fatigue indexes correspond to different weights. For example, in the case where the facial fatigue index includes eye closure, blink frequency, mouth opening and closing, determining the first weight corresponding to the facial fatigue index includes: a first face weight corresponding to eye closure, a second face weight corresponding to blink frequency, and a third face weight corresponding to mouth opening and closing are determined. The first face weight, the second face weight, and the third face weight may be preset, and the present invention is not limited herein. Then, a weighted sum of eye closure, blink frequency, mouth opening and closing degree and voice fatigue index is calculated, whether the weighted sum is larger than or equal to a first target threshold value is judged, and the driver is determined to be in fatigue driving under the condition that the weighted sum is larger than or equal to the first target threshold value. The weighted sum may be used as a fatigue index for the driver, with a greater fatigue index indicating a greater degree of fatigue for the driver. The calculation formula of the weighted sum can be as follows:
f＝α 1 P+α 2 f BF +α 3 OC rate +α 4 S
Wherein alpha is 1 、α 2 、α 3 、α 4 Respectively represent a first face weight, a second face weight, a third face weight, a second weight, alpha 1 +α 2 +α 3 +α 4 =1. In alternative embodiments, alpha may be set 1 >α 2 >α 3 >α 4 。
According to the embodiment of the invention, on the basis of fatigue detection based on facial features, voice information representing fatigue driving is sent out in the fatigue driving process of a driver, and the acoustic feature information representing fatigue driving is fused into the fatigue driving detection result based on facial features, so that the problem of incomplete capture of face pictures caused by head inclination or body position change of the driver in the driving process is solved, and the accuracy and reliability of the fatigue driving detection are improved.
In an alternative embodiment, the method further comprises: and under the condition of determining the fatigue driving of the driver, outputting fatigue early warning information.
The fatigue reminding information can be voice information or other action information which does not influence safe driving, for example, the fatigue reminding information can control the seat of the driver to vibrate under the condition of determining the fatigue driving of the driver.
In an alternative embodiment, in the case of determining fatigue driving of the driver, the process of outputting the fatigue reminding information may include:
Under the condition of determining fatigue driving of a driver, determining the fatigue level of the driver;
and outputting fatigue early warning information corresponding to the fatigue grade.
Under the condition that the driver is judged to belong to fatigue driving according to the combination of the facial fatigue index and the voice fatigue index, the fatigue grade of the driver can be determined according to one or more of eye closure degree, blink frequency, mouth opening and closing degree and the voice fatigue index. For example, when determining the fatigue level according to the eye closure degree, a plurality of value intervals may be preset, different value intervals correspond to different fatigue levels, then the value interval corresponding to the eye closure degree of the driver is determined, and the fatigue level corresponding to the data interval is that after determining the fatigue level of the driver for the fatigue level of the driver, the fatigue early warning information corresponding to the fatigue level is output, and the higher the fatigue level is, the stronger the fatigue early warning information is output. For example, the "please rest in time" voice may be played when the fatigue level is low, the "please stop to rest with care" voice may be played when the fatigue level is medium, and the voice may be played and the driver's seat may be vibrated when the fatigue level is high.
According to the fatigue driving detection method provided by the embodiment of the invention, the fatigue early warning information is output under the condition that the fatigue driving of the driver is determined, so that the driver can be reminded to stop the fatigue driving in time, the driving safety is improved, the fatigue grade of the driver can be further determined, the fatigue early warning information corresponding to the fatigue grade is output, the driver can be reminded dynamically, and the reminding effect can be enhanced.
Fig. 2 shows a flowchart of a fatigue driving detection method according to another embodiment of the present invention, as shown in fig. 2, the method includes:
step S201: acquiring face images and voices of a driver according to a preset acquisition strategy to obtain a face image set and a voice data set;
step S202: acquiring a face fatigue index of the driver based on the face image set; the facial fatigue index includes one or more of the following: eye closure, blink frequency, mouth opening and closing;
step S203: acquiring a voice fatigue index of the driver based on the voice data set;
step S204: determining a first face weight corresponding to the eye closure degree, a second face weight corresponding to the blink frequency and a third face weight corresponding to the mouth opening and closing degree; determining a second weight corresponding to the voice fatigue index;
Step S205: calculating a weighted sum of the eye closure, the blink frequency, the mouth opening and closing degree, and the speech fatigue index based on the first face weight, the second face weight, the third face weight, and the second weight;
step S206: and determining whether the driver is in a fatigue driving state based on the weighted sum and one of the eye closure degree, the blink frequency and the mouth opening and closing degree.
Step S207: under the condition of determining fatigue driving of a driver, determining the fatigue level of the driver;
step S208: outputting fatigue early warning information corresponding to the fatigue grade;
the steps S201 to S205 and the steps S207 to S208 may refer to the embodiment shown in fig. 1, and the present invention is not described herein.
According to the embodiment of the invention, the situation that the collected audio data set has a large amount of environmental sounds caused by the fact that the driver possibly plays audio and video in the driving process, and further the voice fatigue index is too small or too large and cannot accurately represent the fatigue degree of the driver and finally cause false detection or omission is considered, so that when judging whether the driver is in fatigue driving, judgment can be carried out not only based on the weighted sum of the face fatigue index and the voice fatigue index, but also based on the face fatigue index. Specifically, it can be determined according to the following procedure:
Judging whether the weighted sum of the facial fatigue index and the voice fatigue index is larger than or equal to a first target threshold value;
determining that the driver is in fatigue driving if the weighted sum is greater than or equal to a first target threshold;
judging whether one of eye closure degree, blink frequency and mouth opening and closing degree is larger than or equal to a corresponding second target threshold value under the condition that the weighted sum is smaller than the first target threshold value;
and determining that the driver is in fatigue driving under the condition that one of eye closure degree, blink frequency and mouth opening and closing degree is greater than or equal to a corresponding second target threshold value.
Fig. 3 shows a flowchart of a fatigue driving detection method according to another embodiment of the present invention, and as shown in fig. 3, the method includes:
step S301: acquiring face images and voices of a driver according to a preset acquisition strategy to obtain a face image set and a voice data set;
step S302: acquiring a face fatigue index of the driver based on the face image set;
step S303: acquiring a voice fatigue index of the driver based on the voice data set;
step S304: determining whether the driver is in a fatigue driving state based on the facial fatigue index and the voice fatigue index;
Step S305: outputting fatigue early warning information under the condition that the driver is determined to be in fatigue driving;
step S306: under the condition of outputting fatigue early warning information, detecting the driving characteristics of the vehicle, and determining whether the vehicle stops driving;
step S307: and outputting preset warning information under the condition that the vehicle does not stop running.
The steps S301 to S305 may refer to the embodiments shown in fig. 1 and fig. 2, and the disclosure is not repeated here.
With respect to steps S306-S307, after the fatigue warning information is output, the running characteristics of the vehicle such as the running speed, the state of the engine, etc. may be detected to determine whether the vehicle is stopped, and whether the driver is stopped for rest. If the vehicle does not stop running, warning information can be further output so as to ensure that the driver stops fatigue driving. The warning information may be the same as the fatigue warning information, or may be information having stronger warning effect than the fatigue warning information, which is not limited herein.
According to the fatigue driving detection method provided by the embodiment of the invention, whether the driver stops driving for rest is determined by detecting the state of the vehicle after the fatigue early warning information is output, and the means for reminding the driver is further provided under the condition that the driver does not stop driving for rest is determined, so that the driving safety can be further ensured.
Fig. 4 shows a schematic structural diagram of a fatigue driving detection device 400 according to an embodiment of the present invention, as shown in fig. 4, the device 400 includes:
the acquisition module 401 is configured to acquire a face image and voice of a driver according to a preset acquisition strategy, so as to obtain a face image set and a voice data set;
an image recognition module 402, configured to obtain a face fatigue index of the driver based on the face image set;
a voice recognition module 403, configured to obtain a voice fatigue index of the driver based on the voice data set;
a state determination module 404 for determining whether the driver is in a state of tired driving based on the facial fatigue index and the speech fatigue index.
According to the fatigue driving detection device, the face fatigue index and the voice fatigue index are obtained by respectively analyzing the face image set and the voice data set of the driver, the face fatigue index and the voice fatigue index are comprehensively analyzed to judge whether the driver is in fatigue driving or not, and the states of the driver are judged from two dimensions.
Optionally, the state determining module is further configured to: determining a first weight corresponding to the facial fatigue index; determining a second weight corresponding to the voice fatigue index; calculating a weighted sum of the facial fatigue index and the speech fatigue index based on the first weight and the second weight; based on the weighted sum, it is determined whether the driver is in a state of fatigue driving.
Optionally, the state determining module is further configured to: : determining a first face weight corresponding to the eye closure degree, a second face weight corresponding to the blink frequency and a third face weight corresponding to the mouth opening and closing degree; calculating a weighted sum of the eye closure, the blink frequency, the mouth opening and closing degree, and the speech fatigue index based on the first face weight, the second face weight, the third face weight, and the second weight; and determining whether the driver is in a fatigue driving state based on the weighted sum and one of the eye closure degree, the blink frequency and the mouth opening and closing degree.
Optionally, the state determining module is further configured to: determining whether the weighted sum is greater than or equal to a first target threshold; determining that the driver is in fatigue driving if the weighted sum is greater than or equal to the first target threshold; determining if one of the eye closure, the blink frequency, and the mouth opening and closing degree is greater than or equal to a corresponding second target threshold value if the weighted sum is less than the first target threshold value; and determining that the driver is in fatigue driving under the condition that one of the eye closure degree, the blink frequency and the mouth opening and closing degree is greater than or equal to a corresponding second target threshold value.
Optionally, the state determining module is further configured to: determining the fatigue level of the driver under the condition that the driver is determined to be in fatigue driving; and outputting fatigue early warning information corresponding to the fatigue grade.
Optionally, the device further comprises a driving detection module for: detecting the driving characteristics of a vehicle under the condition of outputting the fatigue early warning information or the fatigue prompt information corresponding to the fatigue grade, and determining whether the vehicle stops driving; and outputting preset warning information under the condition that the vehicle does not stop running.
Optionally, the acquisition module is configured to: determining the running time of a vehicle, and determining a target time period according to the running time; and acquiring a face image set and a voice data set of the driver in the target time period.
Optionally, the image recognition module is further configured to: determining a first target image of which the eyes of a driver are in a closed state in the face image set, and determining the eye closure degree according to the frame number of the first target image and the total frame number of the face image set; and/or, based on the face image set, determining
The number of blinks of the driver; determining the blink frequency according to the blink times; and/or 5 determining a second target image of the face image set, wherein the outline of the mouth of the driver is in a preset state, according to
And determining the opening and closing degree of the mouth by the frame number of the second target image and the total frame number of the face image set.
Optionally, the voice recognition module is further configured to: identifying the voice data set, and determining the times of the voice data set belonging to fatigue voice; and determining the voice fatigue index of the driver according to the times of the fatigue voice in the voice data set.
The device can execute the method provided by the embodiment of the invention and has the corresponding functions of the execution method
Module and beneficial effect. Technical details not described in detail in this embodiment may be found in the methods provided in the embodiments of the present invention.
The embodiment of the invention also provides an electronic device, as shown in fig. 5, which comprises a processor 501, a communication interface 502, a memory 503 and a communication bus 504, wherein the processor 501, the communication interface 5 and the memory 503 complete the communication between each other through the communication bus 504,
a memory 503 for storing a computer program;
the processor 501 is configured to execute the program stored in the memory 503, and implement the following steps:
according to a preset acquisition strategy, acquiring a face image and voice of a driver to obtain the face image
A set and a speech data set;
0, acquiring a face fatigue index of the driver based on the face image set;
acquiring a voice fatigue index of the driver based on the voice data set;
based on the facial fatigue index and the speech fatigue index, it is determined whether the driver is in a fatigue driving state.
The communication bus 504 mentioned by the above terminal may be a Peripheral component interconnect standard (PCI) bus of Peripheral component interconnect standard 5Component Interconnect, abbreviated as EISA bus, or an extended industry standard architecture (Extended Industry Standard Architecture, abbreviated as EISA) bus, or the like. The communication bus 504 may be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, the figures are shown with only one bold line, but not with only one bus or one type of bus.
The communication interface 502 is used for communication between the above-described terminal and other devices.
The memory 503 may include a random access memory (Random Access Memory, simply referred to as RAM) or may include a non-volatile memory (non-volatile memory), such as at least one magnetic disk memory. Optionally, the memory may also be at least one memory device located remotely from the aforementioned processor 501.
The processor 501 may be a general-purpose processor, including a central processing unit (Central Processing Unit, CPU for short), a network processor (Network Processor, NP for short), etc.; but also digital signal processors (Digital Signal Processing, DSP for short), application specific integrated circuits (Application Specific Integrated Circuit, ASIC for short), field-programmable gate arrays (Field-Programmable Gate Array, FPGA for short) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.
In yet another embodiment of the present invention, a computer readable medium is provided, in which instructions are stored, which when run on a computer, cause the computer to perform the fatigue driving detection method according to any of the above embodiments.
In yet another embodiment of the present invention, a computer program product comprising instructions, which when run on a computer, causes the computer to perform the method for detecting fatigue driving according to any of the above embodiments is also provided.
In the above embodiments, it may be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented in software, may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When loaded and executed on a computer, produces a flow or function in accordance with embodiments of the present invention, in whole or in part. The computer may be a general purpose computer, a special purpose computer, a computer network, or other programmable apparatus. The computer instructions may be stored in or transmitted from one computer-readable storage medium to another, for example, by wired (e.g., coaxial cable, optical fiber, digital Subscriber Line (DSL)), or wireless (e.g., infrared, wireless, microwave, etc.). The computer readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. that contains an integration of one or more available media. The usable medium may be a magnetic medium (e.g., floppy Disk, hard Disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid State Disk (SSD)), etc.
It is noted that relational terms such as first and second, and the like are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.
In this specification, each embodiment is described in a related manner, and identical and similar parts of each embodiment are all referred to each other, and each embodiment mainly describes differences from other embodiments. In particular, for system embodiments, since they are substantially similar to method embodiments, the description is relatively simple, as relevant to see a section of the description of method embodiments.
The foregoing description is only of the preferred embodiments of the present invention and is not intended to limit the scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.
Claims (13)
1. The fatigue driving detection method is characterized by comprising the following steps of:
acquiring face images and voices of a driver according to a preset acquisition strategy to obtain a face image set and a voice data set;
acquiring a face fatigue index of the driver based on the face image set;
acquiring a voice fatigue index of the driver based on the voice data set;
based on the facial fatigue index and the speech fatigue index, it is determined whether the driver is in a fatigue driving state.
2. The method of claim 1, wherein the determining whether the driver is in a state of tired driving based on the facial fatigue index and the speech fatigue index comprises:
determining a first weight corresponding to the facial fatigue index;
determining a second weight corresponding to the voice fatigue index;
calculating a weighted sum of the facial fatigue index and the speech fatigue index based on the first weight and the second weight;
Based on the weighted sum, it is determined whether the driver is in a state of fatigue driving.
3. The method of claim 2, wherein the facial fatigue index comprises one or more of: eye closure, blink frequency, mouth opening and closing.
4. A method according to claim 3, wherein said determining a first weight corresponding to the facial fatigue index comprises: determining a first face weight corresponding to the eye closure degree, a second face weight corresponding to the blink frequency and a third face weight corresponding to the mouth opening and closing degree;
the calculating a weighted sum of the facial fatigue index and the speech fatigue index based on the first weight and the second weight, comprising: calculating a weighted sum of the eye closure, the blink frequency, the mouth opening and closing degree, and the speech fatigue index based on the first face weight, the second face weight, the third face weight, and the second weight;
the determining whether the driver is in a state of fatigue driving based on the weighted sum includes:
and determining whether the driver is in a fatigue driving state based on the weighted sum and one of the eye closure degree, the blink frequency and the mouth opening and closing degree.
5. The method of claim 4, wherein the determining whether the driver is in a state of tired driving based on the weighted sum and one of the eye closure, the blink frequency, and the mouth opening and closing comprises:
determining whether the weighted sum is greater than or equal to a first target threshold;
determining that the driver is in fatigue driving if the weighted sum is greater than or equal to the first target threshold;
determining if one of the eye closure, the blink frequency, and the mouth opening and closing degree is greater than or equal to a corresponding second target threshold value if the weighted sum is less than the first target threshold value;
and determining that the driver is in fatigue driving under the condition that one of the eye closure degree, the blink frequency and the mouth opening and closing degree is greater than or equal to a corresponding second target threshold value.
6. The method according to claim 1, wherein the method further comprises:
determining the fatigue level of the driver under the condition that the driver is determined to be in fatigue driving;
and outputting fatigue early warning information corresponding to the fatigue grade.
7. The method of claim 6, wherein the method further comprises:
detecting the driving characteristics of a vehicle under the condition of outputting the fatigue early warning information, and determining whether the vehicle stops driving;
and outputting preset warning information under the condition that the vehicle does not stop running.
8. The method according to claim 1, wherein the capturing face images and voices of the driver according to a preset capturing strategy includes:
determining the running time of a vehicle, and determining an acquisition period and an acquisition duration according to the running time;
and acquiring face images and voices of the driver based on the acquisition period and the acquisition time length.
9. A method according to claim 3, wherein said obtaining a facial fatigue index of the driver based on the set of face images comprises:
determining a first target image of which the eyes of a driver are in a closed state in the face image set, and determining the eye closure degree according to the frame number of the first target image and the total frame number of the face image set;
and/or
Determining the blink times of the driver based on the face image set; determining the blink frequency according to the blink times;
And/or
And determining a second target image of which the mouth outline of the driver is in a preset state in the face image set, and determining the mouth opening and closing degree according to the frame number of the second target image and the total frame number of the face image set.
10. The method of claim 1, wherein the obtaining the driver's voice fatigue index based on the voice data comprises:
identifying the voice data set, and determining the times of the voice data set belonging to fatigue voice;
and determining the voice fatigue index of the driver according to the times of the fatigue voice in the voice data set.
11. A fatigue driving detection device, characterized by comprising:
the acquisition module is used for acquiring face images and voices of a driver according to a preset acquisition strategy to obtain a face image set and a voice data set;
the image recognition module is used for acquiring the face fatigue index of the driver based on the face image set;
the voice recognition module is used for acquiring the voice fatigue index of the driver based on the voice data set;
and the state determining module is used for determining whether the driver is in a fatigue driving state or not based on the facial fatigue index and the voice fatigue index.
12. An electronic device, comprising:
one or more processors;
storage means for storing one or more programs,
when executed by the one or more processors, causes the one or more processors to implement the method of any of claims 1-10.
13. A computer readable medium, on which a computer program is stored, characterized in that the program, when being executed by a processor, implements the method according to any of claims 1-10.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211514826.0A CN116386277A (en) | 2022-11-28 | 2022-11-28 | Fatigue driving detection method and device, electronic equipment and medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211514826.0A CN116386277A (en) | 2022-11-28 | 2022-11-28 | Fatigue driving detection method and device, electronic equipment and medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116386277A true CN116386277A (en) | 2023-07-04 |
Family
ID=86964342
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211514826.0A Pending CN116386277A (en) | 2022-11-28 | 2022-11-28 | Fatigue driving detection method and device, electronic equipment and medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116386277A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117079255A (en) * | 2023-10-17 | 2023-11-17 | 江西开放大学 | Fatigue driving detection method based on face recognition and voice interaction |
| CN117237926A (en) * | 2023-11-14 | 2023-12-15 | 广州斯沃德科技有限公司 | Vehicle driving state determining method, system, equipment and readable storage medium |
| CN117576668A (en) * | 2024-01-17 | 2024-02-20 | 江西科技学院 | Multi-feature perception driving fatigue state detection method and system based on video frame |
-
2022
- 2022-11-28 CN CN202211514826.0A patent/CN116386277A/en active Pending
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117079255A (en) * | 2023-10-17 | 2023-11-17 | 江西开放大学 | Fatigue driving detection method based on face recognition and voice interaction |
| CN117079255B (en) * | 2023-10-17 | 2024-01-05 | 江西开放大学 | Fatigue driving detection method based on face recognition and voice interaction |
| CN117237926A (en) * | 2023-11-14 | 2023-12-15 | 广州斯沃德科技有限公司 | Vehicle driving state determining method, system, equipment and readable storage medium |
| CN117237926B (en) * | 2023-11-14 | 2024-02-06 | 广州斯沃德科技有限公司 | Vehicle driving state determining method, system, equipment and readable storage medium |
| CN117576668A (en) * | 2024-01-17 | 2024-02-20 | 江西科技学院 | Multi-feature perception driving fatigue state detection method and system based on video frame |
| CN117576668B (en) * | 2024-01-17 | 2024-04-05 | 江西科技学院 | Multi-feature perception driving fatigue state detection method and system based on video frame |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN116386277A (en) | Fatigue driving detection method and device, electronic equipment and medium | |
| CN109473123B (en) | Voice activity detection method and device | |
| CN108122556B (en) | Method and device for reducing false triggering of voice wake-up instruction words of driver | |
| WO2015192618A1 (en) | Driving monitoring method and device and computer storage medium | |
| CN108280332B (en) | Biological characteristic authentication, identification and detection method, device and equipment of mobile terminal | |
| CN108701229B (en) | Driving behavior analysis method and driving behavior analysis device | |
| WO2021017329A1 (en) | Method and device for detecting when driver is distracted | |
| CN110765807A (en) | Driving behavior analysis method, driving behavior processing method, driving behavior analysis device, driving behavior processing device and storage medium | |
| Sathyanarayana et al. | Information fusion for robust ‘context and driver aware’active vehicle safety systems | |
| CN104239847B (en) | Driving warning method and electronic device for vehicle | |
| CN113723292A (en) | Driver-ride abnormal behavior recognition method and device, electronic equipment and medium | |
| CN110555346A (en) | Driver emotion detection method and device, electronic equipment and storage medium | |
| KR102314824B1 (en) | Acoustic event detection method based on deep learning | |
| CN112818939A (en) | Behavior detection method and device and electronic equipment | |
| CN116563829A (en) | Driver emotion recognition method and device, electronic equipment and storage medium | |
| CN108108651B (en) | Method and system for detecting driver non-attentive driving based on video face analysis | |
| CN115394318A (en) | Audio detection method and device | |
| CN112927688B (en) | Voice interaction method and system for vehicle | |
| WO2022068304A1 (en) | Sound quality detection method and device | |
| CN112380977A (en) | Smoking behavior detection method and device | |
| CN107539038B (en) | Vehicle tire pressure state monitoring method and device | |
| CN110782622A (en) | Safety monitoring system, safety detection method, safety detection device and electronic equipment | |
| CN112633387A (en) | Safety reminding method, device, equipment, system and storage medium | |
| JP7013722B2 (en) | Driving support device | |
| CN112052774B (en) | Behavior detection method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |