WO2023241358A1 - 一种疲劳驾驶的确定方法、装置及电子设备 - Google Patents

一种疲劳驾驶的确定方法、装置及电子设备 Download PDF

Info

Publication number
WO2023241358A1
WO2023241358A1 PCT/CN2023/097394 CN2023097394W WO2023241358A1 WO 2023241358 A1 WO2023241358 A1 WO 2023241358A1 CN 2023097394 W CN2023097394 W CN 2023097394W WO 2023241358 A1 WO2023241358 A1 WO 2023241358A1
Authority
WO
WIPO (PCT)
Prior art keywords
driver
image
face
eye
head
Prior art date
Application number
PCT/CN2023/097394
Other languages
English (en)
French (fr)
Inventor
孔繁昊
陈明轩
Original Assignee
京东方科技集团股份有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 京东方科技集团股份有限公司 filed Critical 京东方科技集团股份有限公司
Publication of WO2023241358A1 publication Critical patent/WO2023241358A1/zh

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/50—Context or environment of the image
    • G06V20/59—Context or environment of the image inside of a vehicle, e.g. relating to seat occupancy, driver state or inner lighting conditions
    • G06V20/597—Recognising the driver's state or behaviour, e.g. attention or drowsiness
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/20—Image preprocessing
    • G06V10/26—Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/764—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using classification, e.g. of video objects
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
    • G06V10/774—Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/16—Human faces, e.g. facial parts, sketches or expressions
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/18—Eye characteristics, e.g. of the iris

Definitions

  • the present application relates to the field of image processing technology, and in particular to a method, device and electronic equipment for determining fatigue driving.
  • the purpose of the embodiments of the present application is to provide a fatigue driving determination method, device and electronic equipment to improve the accuracy of fatigue driving determination.
  • the specific technical solutions are as follows:
  • embodiments of the present application provide a method for determining fatigue driving.
  • the method includes: obtaining multiple frames of image data collected for the driver within a specified time period; for each of the image data, from the In the image data, obtain the driver's face image; obtain the face key points of the driver's target face area in each of the face images, and obtain the face key points of the target face area according to the face key points of the target face area.
  • the target face image Click to obtain the target face image; input the target face image into the pre-trained face state model to obtain the face state of the driver in each target face image; wherein, the face state model It is obtained by using the true value training of the sample target face image and the face state corresponding to the sample target face image; based on the face state of the driver in each target face image, it is determined whether the driver is tired. drive.
  • the target face image includes at least one of an eye image of the driver's eyes, a head image of the head, and a mouth image of the mouth; the input of the target face image Using a pre-trained face state model to obtain the driver's face state in each of the target face images includes: using the pre-trained face state model to obtain the driver's face state in each of the eye images.
  • the open and closed eyes state, and/or the driver's head posture information in each of the head images, and/or the driver's mouth opening and closing state in each of the mouth images; the based on each target Determining whether the driver is driving fatigued based on the driver's face state in the face image includes: based on the driver's eye opening and closing state in each of the eye images and/or each of the head
  • the driver's head posture information in the image and/or the driver's mouth opening and closing state in each mouth image are used to determine whether the driver is driving fatigued.
  • the method is based on the driver's eye opening and closing states in each of the eye images and/or the driver's head posture information in each of the head images and/or each of the mouths.
  • the driver's mouth opening and closing state in each eye image is used to determine whether the driver is driving fatigued, including: estimating whether the driver is fatigued based on the driver's eye opening and closing state in each eye image.
  • Driving obtaining a first prediction result; and/or, based on the driver's head posture information in each head image, predicting whether the driver is fatigued while driving, and obtaining a second prediction result; and/or Or, based on the driver's mouth opening and closing state in each mouth image, estimate whether the driver is driving fatigued, and obtain a third estimate result; based on the first estimate result, and/or the The second prediction result and/or the third prediction result are used to determine whether the driver is driving fatigued.
  • estimating whether the driver is driving fatigued based on the driver's eye opening and closing status in each of the eye images includes: based on the driver's opening and closing eyes in each of the eye images.
  • the eyes-closed state determines the driver's eyes-closing information within the specified time period; wherein the eye-closing information indicates at least one of the driver's eye-closing times, eye-closing duration, and eye-closing frequency. ; Determine whether the eye-closing information satisfies the preset eye-closing fatigue detection conditions; If satisfied, it is determined that the driver is driving fatigued as the first estimated result; otherwise, it is determined that the driver is not driving fatigued as the first estimated result.
  • estimating whether the driver is driving fatigued based on the driver's head posture information in each of the head images includes: based on the driver's head posture information in each of the head images. head posture information to determine the head shaking information of the driver within the specified time period; wherein the head shaking information indicates at least one of the number of nodding times, nodding duration, and nodding frequency of the driver; Determine whether the head shaking information satisfies the preset head fatigue detection conditions; if satisfied, determine that the driver is driving fatigued as the second estimated result; otherwise, determine that the driver is not driving fatigued as the second estimated result. Evaluate the results.
  • estimating whether the driver is driving tiredly based on the driver's mouth opening and closing state in each mouth image includes: based on the driver's mouth in each mouth image The opening and closing state determines the driver's mouth opening information within the specified time period; wherein the mouth opening information indicates at least one of the driver's mouth opening times, mouth opening duration, and mouth opening frequency; determining the mouth opening information Whether the information satisfies the preset mouth fatigue detection conditions; if satisfied, it is determined that the driver is driving fatigued as the third estimated result; otherwise, it is determined that the driver is not driving fatigued as the third estimated result.
  • determining whether the driver is driving fatigued based on the first prediction result, and/or the second prediction result, and/or the third prediction result includes: if the In the first prediction result, and/or the second prediction result, and/or the third prediction result, the proportion indicating that the driver is driving fatigued is greater than the proportion indicating that the driver is not driving fatigued. , then it is determined that the driver is driving fatigued, otherwise, it is determined that the driver is driving without fatigue.
  • the method further includes: determining whether the driver's mouth and the driver's eyes are blocked within the specified time period; and using a pre-trained face state model to obtain each of the eyes.
  • the status includes: if the driver's mouth is blocked and the driver's eyes are not blocked within the specified time period, obtaining the driver's head posture information and The driver's eyes are open and closed in each eye image; if the driver's eyes are blocked and the driver's mouth is not blocked within the specified time period, obtain each head The driver's head posture information in the mouth image and the driver's mouth opening and closing state in each mouth image; if within the specified time period, the driver's eyes and the driver's If the mouths are all blocked, the head posture information of the driver in each head image is obtained.
  • obtaining each of the eye images of the driver includes: performing head detection on the image data, determining the head area corresponding to each head included in the image data, and performing head detection on the head.
  • Perform face detection on the image data to determine the face area corresponding to each face contained in the image data; associate each head area and each face area based on the positions of each head area and each face area ; From the face areas associated with the head area, determine the face area that is within the specified area in the image data, or occupies the largest area, as the face area of the driver; from the driver In the human face area, the image of the area where the eyes are located is extracted as the eye image of the driver's eyes.
  • embodiments of the present application also provide a device for determining fatigue driving, which device includes: a first acquisition module, used to acquire multiple frames of image data collected for the driver within a specified time period; a second acquisition module A module for obtaining the driver's face image from the image data for each of the image data; a third acquisition module for obtaining the driver's face image from each of the face images. face key points of the target face area, and obtain the target face image according to the face key points of the target face area; the fourth acquisition module is used to input the target face image into the pre-trained face state model to obtain the face state of the driver in each target face image; wherein the face state model is based on a sample target face image and a face state corresponding to the sample target face image. Obtained from true value training; the first determination module is used to determine whether the driver is driving fatigue based on the face state of the driver in each target face image.
  • embodiments of the present application provide an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory complete communication with each other through the communication bus; the memory is used to store Computer program; processor, used to implement the steps of the above method for determining fatigue driving when executing the program stored in the memory.
  • embodiments of the present application provide a computer-readable storage medium.
  • a computer program is stored in the computer-readable storage medium.
  • the steps of the above method for determining fatigue driving are implemented.
  • Figure 1 is a flow chart of a method for determining fatigue driving provided by an embodiment of the present application
  • Figure 2 is another flow chart of a method for determining fatigue driving provided by an embodiment of the present application
  • FIG. 3 is a schematic diagram of key points of the mouth in the embodiment of the present application.
  • Figure 4 is another flow chart of the fatigue driving determination method provided by the embodiment of the present application.
  • Figure 5 is another flow chart of the fatigue driving determination method provided by the embodiment of the present application.
  • Figure 6 is a schematic structural diagram of a device for fatigue driving provided by an embodiment of the present application.
  • FIG. 7 is a schematic structural diagram of an electronic device provided by an embodiment of the present application.
  • embodiments of the present application provide a method, device and electronic device for determining fatigue driving.
  • the fatigue driving determination method provided by the embodiments of the present application can be applied to the image collection device in the vehicle, such as a vehicle-mounted camera, or the fatigue driving method provided by the embodiments of the present application.
  • the determination method can also be applied to other types of electronic devices, such as smartphones, personal computers, servers and other devices with data processing capabilities.
  • the electronic devices can also be vehicle-mounted central controls. It should be noted that when applied to other types of electronic devices, the electronic device can communicate with the image acquisition device in the vehicle, thereby acquiring image data collected by the image acquisition device in the vehicle.
  • the fatigue driving determination method provided by the embodiments of the present application can be implemented by software, hardware, or a combination of software and hardware.
  • a method for determining fatigue driving may include the following steps: obtain multiple frames of image data collected for the driver within a specified time period; for each image data, obtain the image data containing Eye images of the driver's eyes; use the pre-trained eye opening and closing model to identify the driver's eye opening and closing state in each eye image; among them, the eye opening and closing model uses sample eye images and sample eye images The corresponding true value of the eye opening and closing state is obtained by training; based on the driver's eye opening and closing state in each eye image, it is determined whether the driver is driving fatigued.
  • the open and closed eye state is identified by using a pre-trained open and closed eye model. Since the open and closed eye model uses the entire eye image when determining the eye state, compared with the key points of the human eye in related technologies, The method of determination is more accurate, which can improve the accuracy of determining the state of open and closed eyes, thereby improving the accuracy of determining fatigue driving.
  • a method for determining fatigue driving may include steps S101 to S104.
  • embodiments of the present application can be applied to vehicles equipped with image acquisition equipment to determine driver fatigue driving.
  • the image acquisition device can be installed directly opposite the driver, or at the central control position of the vehicle, etc. Since it is necessary to obtain an eye image including the driver's eyes from each frame of image data, it is necessary to keep the field of view of the image acquisition device including the driver's face.
  • the multi-frame image data may be multi-frame image data in the video stream captured by the image acquisition device.
  • the above specified time period can be any time period after the vehicle is started. For example, multiple frames of image data in the video stream captured by the image acquisition device within 10 seconds can be periodically collected every five minutes.
  • the acquired multiple frames of image data may be multiple frames of image data acquired by the image acquisition device.
  • the acquired multi-frame image data may be the multi-frame image data acquired from the video stream captured by the image capture device.
  • embodiments of the present application may obtain, for each frame of image data, an eye image including the driver's eyes from the image data.
  • the eye image may be the entire image data, or, in order to improve the efficiency of subsequent recognition, the eye image may be an image of the region where the eye part of the image data is located.
  • the acquired eye image may be an image of the area where a single eye is located, such as a left eye image of the area where the left eye is located, or a right eye image of the area where the right eye is located.
  • the acquired eye image can include images of both eyes.
  • One way is to image the common area where the left eye and the right eye are located. In this case, both the eye image and the Including the left eye and the right eye.
  • the acquired eye image may include two images: a left eye image and a right eye image. Using two eyes to determine fatigue driving is more accurate than using one eye.
  • the left eye image and the right eye image can be detected to determine whether the left eye image and the right eye image belong to eye images, Avoid incorrect eye image recognition due to the driver's profile. Obtain the driver's left eye image and right eye image, and detect whether eyes exist in the two eye images. If there are eyes in only one eye image, only use the eye image with eyes in the driver's eye for fatigue driving. It is determined that if eyes exist in both eye images, the two eye images can be synthesized To determine fatigue driving.
  • Optional methods include at least one of the following two ways:
  • the first eye image acquisition method can use a pre-trained eye image extraction model to obtain an eye image containing the driver's eyes from the image data, where the eye image extraction model uses a sample face image And the ground-truth training of eye images.
  • the second eye image acquisition method may include the following steps A1-A4:
  • Step A1 Perform head detection on the image data to determine the head area corresponding to each head included in the image data, and perform face detection on the image data to determine the face area corresponding to each face included in the image data. ;
  • a target detection algorithm or a semantic segmentation algorithm can be used to simultaneously determine the head area and face area in the image data.
  • Step A2 associate each head area and each face area based on the positions of each head area and each face area;
  • the position of the face area belonging to the same person should be included in the head area. Therefore, the face area included in the head area can be determined as the face area associated with the head area.
  • the head area The head region and the associated face region belong to the same person.
  • Step A3 From the face areas associated with the head area, determine the face area within the specified area in the image data, or the face area occupying the largest area, as the driver's face area.
  • the driver's face area in the image data when determining the driver's face area in the image data, different methods can be used to determine the face area in the image data according to different situations. For example, in the image data When the image only contains the driver, you can directly use methods such as target detection algorithm or semantic segmentation algorithm to determine the driver's head area and face area from the image data. If the image data contains not only the driver but also other persons, the head areas and associated face areas of all persons in the image data can be determined first, and then the head areas and associated face areas of each person associated with the head area can be determined. In the face area, the face area within the specified area in the image data, or the face area occupying the largest area, will be regarded as the driver's face area.
  • target detection algorithm semantic segmentation algorithm
  • the designated area may be the position of the driving position in the image data.
  • the area where the driving position is located is also fixed.
  • the area where the driving position is located in the image data is used as the designated area.
  • the above designated area may be the middle area of the image data. Since the image acquisition device is facing the driving position, the face area with the largest area among the face areas associated with the head area can also be used as the driver's face area.
  • each head area and each face area By associating each head area and each face area based on the positions of each head area and each face area, and determining, from each face area associated with the head area, a person located within a specified area in the image data.
  • the face area, or the face area that occupies the largest area, serves as the driver's face area to avoid interference from people other than the driver.
  • the associated head area can be used as the driver's head area.
  • Step A4 From the determined face area, extract the image of the area where the eyes are located as the eye image of the driver's eyes.
  • an image of the area where the eyes are located can be further extracted from the determined face area as an eye image of the driver's eyes.
  • a key point recognition algorithm can be used to identify key points in the determined face area. After identifying the key points in the face area, determine the key points from the identified key points. The key points of the human eye are obtained, and then the area where the driver's eyes are located is determined based on the key points of the human eye, and the determined area is further segmented to obtain an eye image of the driver's eyes.
  • S103 use the pre-trained eye opening and closing model to identify the driver's eye opening and closing state in each eye image; where the eye opening and closing model is: using the sample eye image and the eye opening and closing state corresponding to the sample eye image.
  • the true value of is obtained by training;
  • the above-mentioned eye-opening and closing states include an eye-opening state and an eye-closing state.
  • the pre-trained eye opening and closing model can be used to identify the driver's eye opening and closing state in each eye image.
  • each eye image can be input to the eye opening and closing model, and then the driver's eye opening and closing state in the eye image output by the eye opening and closing model can be obtained.
  • the multi-frame image data includes image data 1, image data 2 and image data 3. Eye image 1 is extracted from image data 1, eye image 2 is extracted from image data 2, and eye image 2 is extracted from image data 3.
  • the above open and closed eye model can be trained by using the sample eye image and the true value of the open and closed eye state corresponding to the sample eye image.
  • the sample eye image can be multiple, and the eyes in a part of the sample eye image are The eyes are open, and the other part is the eyes closed;
  • the true value of the eye opening and closing state corresponding to the sample eye image is: the actual eye opening and closing state of the eye in the sample eye image.
  • the true value of the eye opening and closing state corresponding to the sample eye image can be manually marked. get.
  • the training methods of the above open and closed eyes model can include:
  • the corresponding recognition result and the true value of the open and closed eye state corresponding to the sample eye image are used to calculate the model loss of the open and closed eye model; then the parameters of the open and closed eye model are adjusted based on the model loss until the model loss converges and training is obtained Completed open and closed eye model.
  • the above-mentioned eye-opening and closing models are pre-trained.
  • the pre-trained eye-opening and closing models can be directly used to identify the driver's eye-opening and closing state in each eye image.
  • S104 Determine whether the driver is driving fatigued based on the driver's eye opening and closing status in each eye image.
  • the eyes blink about 10-15 times per minute, and each blink takes about 0.2-0.4 seconds.
  • the blinking speed will slow down and the blinking frequency will increase.
  • the method of determining whether the driver is driving fatigued may at least include the following steps: Based on the driver's eye opening and closing state in each eye image, determine whether the driver is driving tiredly. Eye-closing information within a specified time period; wherein the eye-closing information indicates at least one of the driver's eye-closing times, eye-closing duration, and eye-closing frequency; that is, determining whether the eye-closing information satisfies the preset eye-closing fatigue detection conditions ; If satisfied, it is determined that the driver is driving fatigued, otherwise, it is determined that the driver is not driving fatigued.
  • Optional methods include at least one of the following three ways:
  • the first determination method is that when the preset eyes-closed fatigue detection condition is that the driver's eyes-closed state reaches the preset duration threshold within the specified time period, the number of times the driver's eyes are closed within the specified time period can be counted. Whether the duration reaches the preset duration threshold. It can be understood that since each frame of image data is used to identify the driver's eye opening and closing status in the embodiment of the present application, the image acquisition device can collect one frame of image data every predetermined time, for example, 24 frames of image data can be collected in one second. , at this time, the duration occupied by each frame of image data can be regarded as 1/24 second.
  • the duration of the driver's eyes-closed state in the specified time period can be obtained. If the driver's eyes-closed state reaches the preset duration threshold within the specified time period, it is determined that the preset eyes-closed fatigue detection conditions are met.
  • the second determination method is that when the preset eye-closing fatigue detection condition is that the number of times the driver closes his eyes in the specified time period reaches the preset number threshold, it can be counted whether the number of times the driver closes his eyes in the specified time period reaches the preset threshold.
  • Set a threshold for times It can be understood that when the image acquisition device collects one frame of image data every predetermined time, when the eye-opening state corresponding to the previous frame of image data appears, the eye-opening state corresponding to the next frame of image data appears. state change For the case of eyes-closed state, it is considered that the driver closes his eyes once. If the number of times the driver closes his eyes within a specified period of time reaches the preset threshold, it is determined that the preset eye-closing fatigue detection conditions are met.
  • the third determination method is when the preset eye-closed fatigue detection condition is that there is a predetermined number of consecutive frames of first target image data within a specified time period, where the first target image data is when the driver is in a closed-eyes state.
  • the image data can detect whether there is a continuous predetermined number of frames of first target image data within a specified time period. If so, it is determined that the preset eye-closed fatigue detection conditions are met.
  • the open and closed eye state is identified by using a pre-trained open and closed eye model. Since the open and closed eye model uses the entire eye image when determining the eye state, compared with the key points of the human eye in related technologies, The method of determination is more accurate, which can improve the accuracy of determining the state of open and closed eyes, thereby improving the accuracy of determining fatigue driving.
  • a method provided by the embodiment of the present application is The method for determining fatigue driving also includes S201: S201, obtaining the driver's head posture information and/or the driver's mouth opening and closing state in each image data; wherein the head posture information may include the angle of the head posture angle,
  • the head attitude angle refers to the pitch angle, yaw angle and flip angle of the head.
  • the head posture information can be used as a basis for determining whether the driver is driving fatigued.
  • the mouth opening and closing state may include an open mouth state and a closed mouth state.
  • the frequency of yawning will also increase. Therefore, the mouth opening and closing state can also be used as a basis for determining whether the driver is driving fatigued.
  • Determining whether the driver is driving fatigued based on the driver's eye opening and closing states in each eye image may include steps S202: S202, based on the driver's eye opening and closing states in each eye image and the driver's eyes in each image data. Head posture information and/or the driver's mouth opening and closing status determine whether the driver is driving fatigued.
  • the driver's eyes opening and closing state can be used, combined with the driver's head posture information and the driver's mouth opening and closing state, or any of the latter two methods, as the determination method.
  • the above method is based on the driver's eye opening and closing status in each eye image, as well as the driver's head posture information and/or the driver's mouth opening and closing status in each image data, to determine whether the driver is driving fatigued, which can at least include the following methods.
  • the driver's eye opening and closing state, the driver's head posture information, the driver's One of the three bases of mouth opening and closing status is only used to predict whether the driver is driving fatigued and obtain the prediction result, and is not directly used to determine whether the driver is driving fatigued.
  • each prediction result can be characterized as driver fatigue driving or driver non-fatigue driving.
  • each prediction result can be characterized as driver fatigue driving or driver non-fatigue driving.
  • the second estimated result and/or the third estimated result there can be multiple ways to determine whether the driver is driving fatigued, which can at least include one of the following two ways:
  • the score is recorded as 0; when it is represented that the driver is driving fatigued, the first estimated result corresponds to The first score, the second estimated result corresponds to the second score, and the third estimated result corresponds to the third score. Finally, the scores of each estimated result are added to obtain the total score. If the total score reaches the predetermined score threshold, it is determined that the driver is driving fatigued. If it does not, it is determined that the driver is not driving fatigued.
  • the second method if in the first estimation result, the second estimation result and/or the third estimation result, the proportion of the driver being instructed to drive while fatigued is greater than the proportion of the driver being instructed to drive without fatigue, then it is determined that the driver is driving without fatigue.
  • the driver is driving fatigued, otherwise, the driver is determined to be driving without fatigue.
  • two of the estimated results are used to determine whether the driver is driving fatigued, it is necessary to determine whether the driver is driving fatigued when both of the two estimated results are used. If three of the three estimated results are used, The estimated results are used to determine whether the driver is driving fatigued. When two of the first estimated results, the second estimated result and the third estimated result are the driver driving fatigued, it is determined that the driver is driving fatigued. .
  • the driver's head posture information and/or the driver's mouth opening and closing state in each image data are acquired. ; and determine whether the driver is driving fatigued based on the driver's eye opening and closing status in each eye image, the driver's head posture information in each image data, and/or the driver's mouth opening and closing status. It can be seen that in this solution, the driver's eye opening and closing state can be used according to the actual application situation, combining the driver's head posture information and the driver's mouth opening and closing state, or one of the latter two methods. As a basis for determining whether the driver is driving fatigued, the method of determining whether the driver is driving fatigued can be flexibly adjusted according to actual application conditions to improve the accuracy of determining whether the driver is driving fatigued in different scenarios.
  • the above method of estimating whether the driver is driving fatigue based on the driver's eye opening and closing state in each eye image is similar to the above method of determining whether the driver is driving fatigue based on the driver's eye opening and closing state in each eye image.
  • the driver's eye-closing information within a specified time period is determined; wherein the eye-closing information indicates at least one of the driver's eye-closing times, eye-closing duration, and eye-closing frequency. one;
  • Step B1 based on the driver's head posture information in each image data, determine the driver's head shaking information within the specified time period; wherein, the head shaking information indicates the number of nodding times, nodding duration, and nodding frequency of the driver. at least one;
  • the driver's head area image can be extracted from the image data, and then the driver's head area image can be further used to obtain the driver's head posture information.
  • the head posture information can be the pitch angle and/or roll angle of the driver's head.
  • the head posture information can also be obtained by using a pre-trained head posture estimation model; similar to the open and closed eyes model, the head posture estimation The model may be trained using the sample head image and the true values of the pitch angle and/or roll angle corresponding to the sample head image.
  • Count at least one of the number of noddings, nodding duration, and nodding frequency of the driver within the specified time period to obtain the driver's head shaking information within the specified time period.
  • Step B2 Determine whether the head shaking information satisfies the preset head fatigue detection conditions; if so, it is determined that the driver is driving fatigued as the second estimated result; otherwise, it is determined that the driver is not driving fatigued as the second estimated result.
  • the head fatigue detection condition may be that the driver's number of nods within a specified time period reaches a predetermined number threshold and/or the duration of the nod reaches a predetermined duration threshold.
  • the head fatigue detection condition may also be that the proportion of the driver's nodding duration in the specified time period reaches a predetermined proportion threshold.
  • the head fatigue detection conditions may not be limited to this.
  • the head shaking information meets the preset head fatigue detection conditions, it is determined that the driver is driving fatigued as the second estimated result. If not, it is determined that the driver is not driving fatigued as the second estimated result.
  • the driver's head shaking information within a specified time period is determined; it is determined whether the head shaking information satisfies the preset head fatigue detection conditions; if it satisfies , then it is determined that the driver is driving fatigued as the second prediction result, otherwise, it is determined that the driver is not driving fatigued as the second prediction result. It can be seen that this solution provides a way to estimate whether the driver is driving fatigued by using the driver's head posture information in each image data, and adds the driver's head posture information in each image data as a method to determine whether the driver is driving fatigued. basis, thereby improving the accuracy of determining whether a driver is driving fatigued in different scenarios.
  • estimating whether the driver is driving fatigue may include the following steps C1-C2:
  • Step C1 Based on the driver's mouth opening and closing status in each image data, determine the driver's mouth opening information within the specified time period; wherein the mouth opening information indicates at least one of the driver's mouth opening times, mouth opening duration, and mouth opening frequency;
  • the above-mentioned ways of determining the driver's mouth opening and closing status based on the positions of key points can be multiple, optional, including at least the following ways:
  • Figure 3 shows a schematic diagram of using the key point identification algorithm to mark key points on the driver's mouth.
  • points 50 to 68 are the key points of the mouth identified by the key point identification algorithm. , respectively represented by p50-p68 in the examples of this application.
  • Key points can be used to calculate the aspect ratio of the mouth.
  • Aspect ratio vertical distance/horizontal distance. Taking Figure 3 as an example, the vertical distance can be the distance between the midpoint of the line connecting the p51 point and the p53 point, and the midpoint of the line connecting the p59 point and the p57 point; the horizontal distance can be the distance between the p49 point and the p55 point. distance.
  • the determination method of the vertical distance may not be limited to this.
  • the vertical distance may also be the distance from point p52 to point p58, etc. If the calculated aspect ratio reaches a predetermined aspect ratio threshold, which is usually set to 0.5, it is determined that the driver's mouth is in an open state; otherwise, it is determined that the driver's mouth is in a closed state.
  • a predetermined aspect ratio threshold which is usually set to 0.5
  • the driver's mouth area image can be extracted from the image data, and then the driver's mouth area image can be further used to obtain the driver's mouth opening and closing state.
  • the mouth opening and closing status can be obtained using the pre-trained mouth opening and closing model. Similar to the eye opening and closing model, the mouth opening and closing model is trained using the sample mouth image and the true value of the mouth opening and closing state corresponding to the sample mouth image.
  • the mouth opening information is used to indicate at least one of the number of mouth openings, mouth opening duration, and mouth opening frequency of the driver within a specified time period.
  • Step C2 Determine whether the mouth opening information satisfies the preset mouth fatigue detection conditions; if so, it is determined that the driver is driving fatigued as the third estimated result; otherwise, it is determined that the driver is not driving fatigued as the third estimated result.
  • the method of determining whether the mouth opening information satisfies the preset mouth fatigue detection condition is similar to the above-mentioned method of determining whether the eye closing information satisfies the preset eye closing fatigue detection condition.
  • the preset mouth fatigue detection condition may be that there is a predetermined number of continuous frames of second target image data within a specified time period, where the second target image data is image data in which the driver's mouth is in an open mouth state. In this case, it can be detected whether there is a continuous predetermined number of frames of second target image data within the specified time period, and if so, it is determined that the preset mouth fatigue detection conditions are met. Other methods of determining whether the mouth opening information meets the preset mouth fatigue detection conditions will not be described again here.
  • the driver's mouth opening information within the specified time period is determined; it is determined whether the mouth opening information satisfies the preset mouth fatigue detection conditions; if so, it is determined that the driver is driving If the driver is driving fatigued, it is the third estimated result; otherwise, it is determined that the driver is not driving fatigued, which is the third estimated result. It can be seen that this solution provides a way to estimate whether the driver is driving fatigued by using the driver's mouth opening and closing state in each image data, and adds the driver's mouth opening and closing state in each image data as a method to determine whether the driver is driving fatigued. basis, thereby improving the accuracy of determining whether a driver is driving fatigued in different scenarios.
  • the fatigue driving method provided by the embodiment of the present application is Determination methods can also include:
  • Determining whether the driver's mouth is blocked during a specified period of time can include at least the following methods:
  • a face alignment algorithm can be used to align the faces in the face image to a unified shape, and then Input the aligned face image to the first face quality model.
  • the above-mentioned first face quality model is used to detect whether the mouth of the face image is blocked, and can be trained using the sample face image and the true value of whether the mouth is blocked corresponding to the sample face image.
  • Obtaining the driver's head posture information and/or the driver's mouth opening and closing status in each image data may include: if the driver's mouth is blocked within a specified time period, obtaining the driver's head posture information in each image data Partial posture information; when it is determined that the driver's mouth is blocked within a specified period of time, it means that the driver's mouth opening and closing state cannot be used to predict whether the driver is driving fatigued. In this case, there is no need to obtain the driver's mouth opening and closing state. However, the driver's head posture information can be used to predict whether the driver is driving fatigued.
  • the driver's head posture information and the driver's mouth opening and closing status in each image data are obtained.
  • the driver's mouth opening and closing state can be used to estimate whether the driver is driving fatigued.
  • the driver's head posture information in each image data can be obtained. and the driver's mouth opening and closing status, so that based on the driver's head posture information in each image data, it can be estimated whether the driver is driving fatigued, and a second prediction result can be obtained.
  • a second prediction result can be obtained.
  • the driver's mouth opening in each image data In the combined state it is estimated whether the driver is driving fatigued, and the third estimated result is obtained. Both the second estimated result and the third estimated result are used to determine whether the driver is driving fatigued.
  • the driver's eyes may also be blocked, for example, if the driver wears sunglasses, etc. At this time, it is impossible to determine the driver's eye opening and closing status, and thus the driver's eye opening and closing status cannot be used to predict driving. Whether the driver is tired while driving.
  • the method for determining fatigue driving provided by the embodiment of the present application may further include: determining whether the driver's eyes are in a specified period of time. obscured.
  • the driver's face area can be determined after obtaining each frame of image data, and the driver's face image can be intercepted from the image data, and then The face image can be input to a pre-trained second face quality model to detect whether the driver's eyes are occluded in the face image.
  • the face alignment algorithm can be used to align the faces in the face image to a unified shape, and then The aligned face image is input to the second face quality model.
  • the above-mentioned second face quality model is used to detect whether the eyes of the face image are blocked, and can be trained using the sample face image and the true value of whether the eyes are blocked corresponding to the sample face image.
  • the step of obtaining an eye image including the driver's eyes from the image data is performed.
  • the driver's eye opening and closing status can be used to evaluate whether the driver is driving fatigued.
  • the driver's eyes When the driver's eyes are occluded within a specified period of time, it means that the driver's eye opening and closing state cannot be used to evaluate whether the driver is driving fatigued. At this time, it is not necessary to execute the image data for each image data, and obtain the driver's information from the image data.
  • the face image in order to improve detection efficiency and simultaneously detect whether the driver's eyes and mouth are blocked, at this time, after obtaining each frame of face image, the face image can be input to the third person Face quality model to simultaneously obtain detection results of whether the driver's eyes and mouth are occluded.
  • the above-mentioned third face quality model can be trained using the sample face image and the true value of whether the eyes and mouth corresponding to the sample face image are occluded.
  • the above detection results will have the following four situations: neither the eyes nor the mouth are blocked; the eyes are blocked, but the mouth is not blocked; the eyes are not blocked, but the mouth is blocked; the eyes and the mouth are both blocked.
  • the driver's eyes and mouth are not blocked within the specified period of time.
  • the driver's eye opening and closing status, the driver's head posture information and the driver's facial expressions can be obtained from each image data. Mouth opening and closing state; based on the driver's eye opening and closing state in each eye image, it is estimated whether the driver is driving fatigued, and the first estimation result is obtained; based on the driver's head posture information in each image data, driving is estimated Based on the driver's mouth opening and closing status in each image data, it is estimated whether the driver is driving fatigued and the third estimation result is obtained; finally, based on the first estimation result, the third estimation result is obtained.
  • the second prediction result and the third prediction result determine whether the driver is driving fatigued.
  • the driver's eyes are blocked and the mouth is not blocked within the specified time period.
  • the driver's head posture information and the driver's mouth opening and closing status in each image data can be obtained; based on each Based on the driver's head posture information in the image data, it is estimated whether the driver is driving fatigued and the second estimation result is obtained; and based on the driver's mouth opening and closing state in each image data, it is estimated whether the driver is driving fatigued and the third estimation result is obtained. three prediction results; finally, based on the second prediction result and the third prediction result, it is determined whether the driver is driving fatigued.
  • the driver's eyes are not blocked and the mouth is blocked within the specified time period.
  • the driver's eye opening and closing status and the driver's head posture information in each image data can be obtained; based on each image data
  • the driver's eye opening and closing state in the eye image is used to estimate whether the driver is driving fatigued, and the first estimation result is obtained; based on the driver's head posture information in each image data, the driver is estimated to be driving fatigued, and the second estimation result is obtained. 2.
  • the driver's eyes and mouth are blocked within the specified time period.
  • only the driver's head posture information in each image data can be obtained; finally, based on the driver's head in each image data Attitude information directly determines whether the driver is driving fatigued.
  • the embodiment of the present application may include the following steps:
  • Step 1 First, perform head detection and face detection on the collected image data, and generate identification frames for the head areas and face areas of all persons in the image data, thereby determining each head area and face area.
  • Step 2 According to the position of the identification frame of each head area and the identification frame of the face area, associate each head area with each face area, and select the largest head area as the driver's head area. Thus, the driver's head image and face image are intercepted.
  • Step 3 Determine whether there is a head area in the image data. If so, obtain the head posture information.
  • Step 4 Use the key point recognition algorithm to generate key points of the face in the face image, determine the eye area, and then obtain the eye image.
  • Step 5 Use the face alignment algorithm (similarity transformation or radial transformation) to align the faces in the face image to a unified shape.
  • Step 6 Use the image quality evaluation algorithm to evaluate the quality of the face image to obtain the image quality score of the face image.
  • detect whether the driver's mouth is blocked to obtain the mouth quality score For example, the driver's mouth quality score can be obtained.
  • the corresponding mouth quality score is set when the mouth is blocked, and another corresponding mouth quality score is set for the situation when the driver's mouth is not blocked; similarly, it is detected whether the driver's eyes are blocked, and the eyes are obtained.
  • face quality score weighted addition of image quality score, mouth quality score and eye quality score to obtain the face quality score.
  • Step 7 Determine whether the face quality score reaches the predetermined score threshold. If not, estimate whether the driver is driving fatigued based only on the driver's head posture information in each image data, and obtain the second estimate result; if so, then Based on the driver's head posture information in each image data, it is estimated whether the driver is driving fatigued, a second estimation result is obtained, and step 8 is performed.
  • Step 8 Determine whether the driver's eyes are blocked; if not, use the eye opening and closing model to identify the driver's eye opening and closing state in each eye image. Based on the driver's eye opening and closing state in each eye image, It is estimated whether the driver is driving fatigued, the first estimation result is obtained, and step 9 is executed; if so, step 9 is executed directly.
  • Step 9 Determine whether the driver's mouth is blocked.
  • Step 10 If the driver's mouth is not blocked, obtain the driver's mouth opening and closing status in each image data. Based on the driver's mouth opening and closing status in each image data, estimate whether the driver is driving fatigued, and obtain the third Three estimated results.
  • Step 11 Determine whether the driver is driving fatigued based on each estimated result obtained in the above process.
  • a method for determining fatigue driving may include steps S301 to S305.
  • S301 Obtain multiple frames of image data collected for the driver within a specified time period.
  • the collection of multi-frame image data in S301 may refer to the above-mentioned S101, which will not be described again in this application.
  • S303 Obtain the face key points of the driver's target face area in each of the face images, and obtain the target face image based on the face key points of the target face area.
  • S304 Input the target face image into a pre-trained face state model, and obtain the face state of the driver in each target face image; wherein the face state model uses sample target faces.
  • the face image and the true value of the face state corresponding to the sample target face image are trained.
  • inputting the target face image into a pre-trained face state model and obtaining the face state of the driver in each target face image includes: using a pre-trained face state model Facial state model, obtains the driver's eye opening and closing state in each of the eye images, and/or the driver's head posture information in each of the head images, and/or each of the mouths The driver's mouth shown in the image.
  • S305 Determine whether the driver is driving fatigued based on the driver's face state in each target face image.
  • determining whether the driver is driving fatigued based on the face state of the driver in each target face image includes: based on the driver's opening in each of the eye images.
  • the eyes-closed state, and/or the driver's head posture information in each head image, and/or the driver's mouth opening and closing state in each mouth image determine whether the driver is tired. drive.
  • the method is based on the driver's eye opening and closing state in each of the eye images and/or the driver's head posture information and/or in each of the head images. Or the driver's mouth opening and closing state in each of the mouth images, determining whether the driver is driving fatigued, including: estimating the driver's eye opening and closing state in each of the eye images. Predict whether the driver is driving fatigued and obtain a first prediction result; and/or estimate whether the driver is driving fatigued based on the head posture information of the driver in each head image and obtain a second prediction result.
  • estimating whether the driver is driving tiredly based on the driver's eye opening and closing state in each of the eye images includes: based on the driver's eye opening and closing states in each of the eye images. Describe the driver's eye-opening and closing status, and determine the driver's eye-closing information within the specified time period; wherein the eye-closing information indicates the driver's eye-closing times, eye-closing duration, and eye-closing frequency. At least one of: Determine whether the eyes-closed information satisfies the preset eyes-closed fatigue detection condition; if satisfied, determine that the driver's fatigue driving is the first estimated result, otherwise, determine that the driver is not fatigued Driving is the first estimate.
  • estimating whether the driver is driving fatigue based on the driver's head posture information in each head image includes: based on the driver's head posture information in each head image.
  • the driver's head posture information is used to determine the driver's head shaking information within the specified time period; wherein the head shaking information indicates the number of nodding, nodding duration, and nodding frequency of the driver. At least one of; determine whether the head shaking information satisfies the preset head fatigue detection condition; if satisfied, determine that the driver's fatigue driving is the second estimated result, otherwise, determine that the driver is not fatigued Driving is the second estimated outcome.
  • estimating whether the driver is driving fatigue based on the driver's mouth opening and closing state in each mouth image includes: based on the driver's mouth opening and closing state in each mouth image. Describe the driver's mouth opening and closing status, and determine the driver's mouth opening information within the specified time period; wherein the mouth opening information indicates at least one of the driver's mouth opening times, mouth opening duration, and mouth opening frequency. ; Determine whether the mouth opening information satisfies the preset mouth fatigue detection conditions; if satisfied, determine that the driver is driving fatigued as the third estimated result, otherwise, determine that the driver is not driving fatigued as the third estimated result result.
  • determining whether the driver is driving fatigued including: if the first prediction result, and/or the second prediction result, and/or the third prediction In the results, if the proportion indicating that the driver is fatigued driving is greater than the proportion indicating that the driver is not fatigued driving, it is determined that the driver is driving fatigued; otherwise, it is determined that the driver is driving not fatigued.
  • the method further includes: determining whether the driver's mouth and the driver's eyes are blocked within the specified time period; using a pre-trained face state model, Obtain the driver's eye opening and closing state in each of the eye images, and/or the driver's head posture information in each of the head images, and/or the driving state in each of the mouth images.
  • the driver's mouth opening and closing state includes: if the driver's mouth is blocked and the driver's eyes are not blocked within the specified time period, obtaining the driver's face in each head image.
  • Head posture information and the driver's eye opening and closing status in each eye image if the driver's eyes are blocked and the driver's mouth is not blocked within the specified time period, then Obtain the driver's head posture information in each head image and the driver's mouth opening and closing state in each mouth image; if the driver's eyes and If the driver's mouths are all blocked, the head posture information of the driver in each head image is obtained.
  • obtaining each eye image of the driver includes: performing head detection on the image data, and determining the head corresponding to each head contained in the image data. area, perform face detection on the image data, and determine the face area corresponding to each face included in the image data; based on the positions of each head area and each face area, perform face detection on each head area and each face area. Correlate the face areas; determine from the face areas associated with the head area the face area that is within the specified area in the image data or occupies the largest area as the driver's face area; From the driver's face area, an image of the area where the eyes are located is extracted as an eye image of the driver's eyes.
  • the embodiment of the present application also provides a device for determining fatigue driving, as shown in Figure 6.
  • the device includes: a first acquisition module 510, used to acquire multiple frames of image data collected for the driver within a specified time period;
  • the second acquisition module 520 is used to acquire the face image of the driver from the image data for each of the image data;
  • the third acquisition module 530 is used to acquire the face image of each of the face images.
  • the face key points of the driver's target face area and obtain the target face image according to the face key points of the target face area;
  • the fourth acquisition module 540 is used to input the target face image Pre-trained face state model to obtain the face state of the driver in each target face image; wherein the face state model uses sample target face images and the sample target face image The corresponding true value training of the face state is obtained;
  • the first determination module 550 is used to determine whether the driver is driving fatigue based on the face state of the driver in each target face image.
  • the driver's mouth opening and closing state in the mouth image; the first determination module is specifically configured to: based on the driver's eye opening and closing state in each of the eye images, and/or each of the heads
  • the driver's head posture information in the image and/or the driver's mouth opening and closing state in each mouth image are used to determine whether the driver is driving fatigued.
  • the first determination module includes: a first prediction sub-module, used to estimate whether the driver is driving fatigue based on the driver's eye opening and closing state in each eye image, and obtain a third An estimation result; and/or, based on the driver's head posture information in each head image, it is estimated whether the driver is driving fatigued, and a second estimation result is obtained; and/or, based on each mouth
  • the driver's mouth opening and closing state in the image is used to estimate whether the driver is driving fatigued, and a third estimate result is obtained;
  • a first determination sub-module is used to determine based on the first estimate result and/or the The second prediction result and/or the third prediction result are used to determine whether the driver is driving fatigued.
  • the first prediction sub-module includes: a first determination unit configured to determine, based on the driver's eye opening and closing status in each eye image, whether the driver will open or close his eyes within the specified time period.
  • the eye-closing information wherein the eye-closing information indicates at least one of the driver's eye-closing times, eye-closing duration, and eye-closing frequency; the second determination unit, Used to determine whether the eyes-closed information satisfies the preset eyes-closed fatigue detection conditions; if satisfied, determine that the first estimated result is driver fatigue driving; otherwise, determine that the first estimated result is driver fatigue driving. Driving without fatigue.
  • the first prediction sub-module estimates whether the driver is driving fatigued based on the driver's head posture information in each head image, and obtains a second prediction result, including: based on each head image The driver's head posture information in the head image is used to determine the driver's head shaking information within the specified time period; wherein the head shaking information indicates the driver's number of nods and the duration of the nod. , at least one of nodding frequencies; determine whether the head shaking information satisfies the preset head fatigue detection conditions; if satisfied, determine that the second estimated result is driver fatigue driving, otherwise, determine the The second estimated result is that the driver is not driving fatigued.
  • the first prediction sub-module estimates whether the driver is driving fatigued based on the mouth opening and closing status of the driver in each mouth image, including: based on the driver's mouth opening and closing status in each mouth image.
  • the mouth opening and closing status of the driver is determined to determine the driver's mouth opening information within the specified time period; wherein the mouth opening information indicates at least one of the driver's mouth opening times, mouth opening duration, and mouth opening frequency; determining the driver's mouth opening and closing status; Whether the mouth opening information satisfies the preset mouth fatigue detection conditions; if satisfied, it is determined that the third estimated result is that the driver is driving fatigued; otherwise, it is determined that the third estimated result is that the driver is not driving fatigued.
  • the device further includes: a second determination module for determining whether the driver's mouth and the driver's eyes are blocked within the specified time period; the first determination module is specifically configured to : If the driver's mouth is blocked and the driver's eyes are not blocked within the specified time period, obtain the driver's head posture information and each head posture information in each head image. The driver's eyes are opened and closed in the eye image; if the driver's eyes are blocked and the driver's mouth is not blocked within the specified time period, obtain the state of the driver's eyes in each head image.
  • the driver's head posture information and the driver's mouth opening and closing status in each of the mouth images if within the specified time period, the driver's eyes and the driver's mouth are both If the driver is blocked, the head posture information of the driver in each head image is obtained.
  • the second acquisition module includes: a second determination sub-module for performing head detection on the image data, determining the head area corresponding to each head contained in the image data, and The image data performs face detection to determine the face area corresponding to each face contained in the image data; the correlation submodule is used to detect each head area based on the position of each head area and each face area. associated with each face area; the third determination sub-module is used to determine, from each face area associated with the head area, the face area within the specified area in the image data, or the face area that occupies the largest area The face area is used as the driver's face area; the extraction submodule is used to extract the image of the area where the eye parts are located from the determined face area as the eye image of the driver's eyes.
  • a second determination sub-module for performing head detection on the image data, determining the head area corresponding to each head contained in the image data, and The image data performs face detection to determine the face area corresponding to each face contained in the image data
  • the correlation submodule is
  • the embodiment of the present application also provides an electronic device, as shown in Figure 7, including a processor 601, a communication interface 602, a memory 603, and a communication bus 604.
  • the processor 601, the communication interface 602, and the memory 603 communicate through the communication bus 604.
  • the memory 603 is used to store the computer program; the processor 601 is used to implement the above-mentioned method for determining fatigue driving when executing the program stored in the memory 603.
  • the communication bus mentioned in the above-mentioned electronic equipment can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc.
  • PCI Peripheral Component Interconnect
  • EISA Extended Industry Standard Architecture
  • the communication bus can be divided into address bus, data bus, control bus, etc. For ease of presentation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.
  • the communication interface is used for communication between the above-mentioned electronic devices and other devices.
  • the memory may include random access memory (Random Access Memory, RAM) or non-volatile memory (Non-Volatile Memory, NVM), such as at least one disk memory.
  • RAM Random Access Memory
  • NVM Non-Volatile Memory
  • the memory may also be at least one storage device located far away from the aforementioned processor.
  • the above-mentioned processor may be a general-purpose processor, including a central processing unit (Central Processing Unit, CPU), Network Processor (NP), etc.; it can also be Digital Signal Processor (DSP), Application Specific Integrated Circuit (ASIC), Field-Programmable Gate Array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.
  • CPU Central Processing Unit
  • NP Network Processor
  • DSP Digital Signal Processor
  • ASIC Application Specific Integrated Circuit
  • FPGA Field-Programmable Gate Array
  • FPGA Field-Programmable Gate Array
  • a computer-readable storage medium stores a computer program.
  • the computer program is executed by a processor, the above-described method for determining fatigue driving is implemented. A step of.
  • a computer program product containing instructions is also provided, which when run on a computer causes the computer to perform the above method for determining fatigue driving.
  • the computer program product includes one or more computer instructions.
  • the computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device.
  • the computer instructions may be stored in or transmitted from one computer-readable storage medium to another, e.g., the computer instructions may be transferred from a website, computer, server, or data center Transmission to another website, computer, server or data center by wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) means.
  • the computer-readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. that contains one or more available media integrated.
  • the available media may be magnetic media (eg, floppy disk, hard disk, magnetic tape), optical media (eg, DVD), or semiconductor media (eg, Solid State Disk (SSD)), etc.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Multimedia (AREA)
  • General Physics & Mathematics (AREA)
  • Physics & Mathematics (AREA)
  • Health & Medical Sciences (AREA)
  • General Health & Medical Sciences (AREA)
  • Human Computer Interaction (AREA)
  • Computing Systems (AREA)
  • Software Systems (AREA)
  • Medical Informatics (AREA)
  • Evolutionary Computation (AREA)
  • Databases & Information Systems (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Artificial Intelligence (AREA)
  • Oral & Maxillofacial Surgery (AREA)
  • Ophthalmology & Optometry (AREA)
  • Measurement Of The Respiration, Hearing Ability, Form, And Blood Characteristics Of Living Organisms (AREA)

Abstract

本申请提供了一种疲劳驾驶的确定方法、装置及电子设备，应用于图像处理技术领域，该方法包括：获取指定时间段内，针对驾驶员所采集的多帧图像数据；针对每一所述图像数据，从所述图像数据中，获取所述驾驶员的人脸图像；获取每一所述人脸图像中所述驾驶员的目标人脸区域的人脸关键点，并根据所述目标人脸区域的人脸关键点获取目标人脸图像；将所述目标人脸图像输入预先训练的人脸状态模型，获取每一所述目标人脸图像中所述驾驶员的人脸状态；其中，所述人脸状态模型是利用样本目标人脸图像以及所述样本目标人脸图像对应的人脸状态的真值训练得到的；基于各目标人脸图像中所述驾驶员的人脸状态，确定所述驾驶员是否疲劳驾驶。

Description

一种疲劳驾驶的确定方法、装置及电子设备 技术领域
本申请涉及图像处理技术领域，特别是涉及一种疲劳驾驶的确定方法、装置及电子设备。
背景技术
随着公路交通事业逐渐繁荣，车辆数量不断上升，道路交通安全问题变得日益严重起来。由于疲劳驾驶导致的交通事故比一般交通事故要严重很多，因此，对驾驶员的疲劳程度进行实时确定与预警，对避免交通事故的发生具有重要的意义。
发明内容
本申请实施例的目的在于提供一种疲劳驾驶的确定方法、装置及电子设备，用以提高疲劳驾驶确定的准确度。具体技术方案如下：
第一方面，本申请实施例提供了一种疲劳驾驶的确定方法，该方法包括：获取指定时间段内，针对驾驶员所采集的多帧图像数据；针对每一所述图像数据，从所述图像数据中，获取所述驾驶员的人脸图像；获取每一所述人脸图像中所述驾驶员的目标人脸区域的人脸关键点，并根据所述目标人脸区域的人脸关键点获取目标人脸图像；将所述目标人脸图像输入预先训练的人脸状态模型，获取每一所述目标人脸图像中所述驾驶员的人脸状态；其中，所述人脸状态模型是利用样本目标人脸图像以及所述样本目标人脸图像对应的人脸状态的真值训练得到的；基于各目标人脸图像中所述驾驶员的人脸状态，确定所述驾驶员是否疲劳驾驶。
可选地，所述目标人脸图像包括所述驾驶员的眼睛的眼部图像、头部的头部图像和嘴巴的嘴部图像中的至少一种；所述将所述目标人脸图像输入预先训练的人脸状态模型，获取每一所述目标人脸图像中所述驾驶员的人脸状态，包括：利用预先训练的人脸状态模型，获取各所述眼部图像中所述驾驶员的睁闭眼状态、和/或各所述头部图像中所述驾驶员的头部姿态信息和/或各所述嘴部图像中所述驾驶员的嘴巴张合状态；所述基于各目标人脸图像中所述驾驶员的人脸状态，确定所述驾驶员是否疲劳驾驶，包括：基于各所述眼部图像中所述驾驶员的睁闭眼状态、和/或各所述头部图像中所述驾驶员的头部姿态信息和/或各所述嘴部图像中所述驾驶员的嘴巴张合状态，确定所述驾驶员是否疲劳驾驶。
可选地，所述基于各所述眼部图像中所述驾驶员的睁闭眼状态、和/或各所述头部图像中所述驾驶员的头部姿态信息和/或各所述嘴部图像中所述驾驶员的嘴巴张合状态，确定所述驾驶员是否疲劳驾驶，包括：基于各所述眼部图像中所述驾驶员的睁闭眼状态，预估所述驾驶员是否疲劳驾驶，得到第一预估结果；和/或，基于各所述头部图像中所述驾驶员的头部姿态信息，预估所述驾驶员是否疲劳驾驶，得到第二预估结果；和/或，基于各所述嘴部图像中所述驾驶员的嘴巴张合状态，预估所述驾驶员是否疲劳驾驶，得到第三预估结果；基于所述第一预估结果、和/或所述第二预估结果和/或所述第三预估结果，确定所述驾驶员是否疲劳驾驶。
可选地，所述基于各所述眼部图像中所述驾驶员的睁闭眼状态，预估所述驾驶员是否疲劳驾驶，包括：基于各所述眼部图像中所述驾驶员的睁闭眼状态，确定所述驾驶员在所述指定时间段内的闭眼信息；其中，所述闭眼信息指示所述驾驶员的闭眼次数、闭眼时长、闭眼频率中的至少一种；确定所述闭眼信息是否满足预设的闭眼疲劳检测条件； 若满足，则确定所述驾驶员疲劳驾驶为第一预估结果，否则，确定所述驾驶员未疲劳驾驶为第一预估结果。
可选地，所述基于各所述头部图像中所述驾驶员的头部姿态信息，预估所述驾驶员是否疲劳驾驶，包括：基于各所述头部图像中所述驾驶员的头部姿态信息，确定所述驾驶员在所述指定时间段内的头部晃动信息；其中，所述头部晃动信息指示所述驾驶员的点头次数、点头时长、点头频率中的至少一种；确定所述头部晃动信息是否满足预设的头部疲劳检测条件；若满足，则确定所述驾驶员疲劳驾驶为第二预估结果，否则，确定所述驾驶员未疲劳驾驶为第二预估结果。
可选地，所述基于各所述嘴部图像中所述驾驶员的嘴巴张合状态，预估所述驾驶员是否疲劳驾驶，包括：基于各所述嘴部图像中所述驾驶员的嘴巴张合状态，确定所述驾驶员在所述指定时间段内的张嘴信息；其中，所述张嘴信息指示所述驾驶员的张嘴次数、张嘴时长、张嘴频率中的至少一种；确定所述张嘴信息是否满足预设的嘴部疲劳检测条件；若满足，则确定所述驾驶员疲劳驾驶为第三预估结果，否则，确定所述驾驶员未疲劳驾驶为第三预估结果。
可选地，所述基于所述第一预估结果、和/或所述第二预估结果和/或所述第三预估结果，确定所述驾驶员是否疲劳驾驶，包括：若所述第一预估结果、和/或所述第二预估结果和/或所述第三预估结果中，指示所述驾驶员疲劳驾驶的占比大于指示所述驾驶员未疲劳驾驶的占比，则确定所述驾驶员疲劳驾驶，否则，确定所述驾驶员为未疲劳驾驶。
可选地，所述方法还包括：确定所述指定时间段内所述驾驶员的嘴巴和所述驾驶员的眼睛是否被遮挡；所述利用预先训练的人脸状态模型，获取各所述眼部图像中所述驾驶员的睁闭眼状态、和/或各所述头部图像中所述驾驶员的头部姿态信息和/或各所述嘴部图像中所述驾驶员的嘴巴张合状态，包括：若在所述指定时间段内所述驾驶员的嘴巴被遮挡且所述驾驶员的眼睛未被遮挡，则获取各所述头部图像中所述驾驶员的头部姿态信息和各所述眼部图像中所述驾驶员的睁闭眼状态；若在所述指定时间段内所述驾驶员的眼睛被遮挡且所述驾驶员的嘴巴未被遮挡，则获取各所述头部图像中所述驾驶员的头部姿态信息和各所述嘴部图像中所述驾驶员的嘴巴张合状态；若在所述指定时间段内所述驾驶员的眼睛和所述驾驶员的嘴巴均被遮挡，则获取各所述头部图像中所述驾驶员的头部姿态信息。
可选地，获取所述驾驶员的各所述眼部图像，包括：对所述图像数据进行头部检测，确定所述图像数据中包含的每一头部对应的头部区域，并对所述图像数据进行人脸检测，确定所述图像数据中包含的每一人脸对应的人脸区域；基于各头部区域和各人脸区域的位置，对各头部区域和各人脸区域进行关联；从各与头部区域关联的人脸区域中，确定处于所述图像数据中指定区域内，或所占面积最大的人脸区域，作为所述驾驶员的人脸区域；从所述驾驶员的人脸区域中，提取眼睛部位所在区域的图像，作为所述驾驶员的眼睛的眼部图像。
第二方面，本申请实施例还提供了一种疲劳驾驶的确定装置，该装置包括：第一获取模块，用于获取指定时间段内，针对驾驶员所采集的多帧图像数据；第二获取模块，用于针对每一所述图像数据，从所述图像数据中，获取所述驾驶员的人脸图像；第三获取模块，用于获取每一所述人脸图像中所述驾驶员的目标人脸区域的人脸关键点，并根据所述目标人脸区域的人脸关键点获取目标人脸图像；第四获取模块，用于将所述目标人脸图像输入预先训练的人脸状态模型，获取每一所述目标人脸图像中所述驾驶员的人脸状态；其中，所述人脸状态模型是利用样本目标人脸图像以及所述样本目标人脸图像对应的人脸状态的真值训练得到的；第一确定模块，用于基于各目标人脸图像中所述驾驶员的人脸状态，确定所述驾驶员是否疲劳驾驶。
第三方面，本申请实施例提供了一种电子设备，包括处理器、通信接口、存储器和通信总线，其中，处理器，通信接口，存储器通过通信总线完成相互间的通信；存储器，用于存放计算机程序；处理器，用于执行存储器上所存放的程序时，实现上述疲劳驾驶的确定方法的步骤。
第四方面，本申请实施例提供了一种计算机可读存储介质，所述计算机可读存储介质内存储有计算机程序，所述计算机程序被处理器执行时实现上述疲劳驾驶的确定方法的步骤。
附图说明
为了更清楚地说明本申请实施例中的技术方案，下面将对实施例描述中所需要使用的附图作简单地介绍，显而易见地，下面描述中的附图仅仅是本申请的一些实施例，对于本领域普通技术人员来讲，还可以根据这些附图获得其他的实施例。
图1为本申请实施例所提供的疲劳驾驶的确定方法的流程图；
图2为本申请实施例所提供的疲劳驾驶的确定方法的另一流程图；
图3为本申请实施例中嘴巴的关键点的示意图；
图4为本申请实施例所提供的疲劳驾驶的确定方法的又一流程图；
图5为本申请实施例所提供的疲劳驾驶的确定方法的又一流程图；
图6为本申请实施例所提供的疲劳驾驶的装置的结构示意图；
图7为本申请实施例所提供的电子设备的结构示意图。
具体实施方式
下面将结合本申请实施例中的附图，对本申请实施例中的技术方案进行清楚、完整地描述，显然，所描述的实施例仅仅是本申请一部分实施例，而不是全部的实施例。基于本申请中的实施例，本领域普通技术人员基于本申请所获得的所有其他实施例，都属于本申请保护的范围。
随着公路交通事业逐渐繁荣，车辆数量不断上升。道路交通安全问题变得日益严重起来。据统计，由于疲劳驾驶导致的交通事故占总交通事故数的比例超过20％。且由于驾驶疲劳导致的交通事故通常为重大交通事故，驾驶员在交通事故中死亡概率较高。据有关研究表明，如果在发生交通事故前对疲劳驾驶的驾驶员进行提醒，可以成功避免90％左右的因疲劳驾驶造成的交通事故。因此，若能对驾驶员准确进行疲劳驾驶的确定，从而及时对疲劳驾驶的驾驶员进行提醒和预警，对避免交通事故的发生具有重要的意义。
相关技术中，多通过确定驾驶员是否频繁闭眼来确定驾驶员是否疲劳驾驶，具体的，先通过车载摄像头在预定时间内连续拍摄多帧人脸图像；针对每一人脸图像，利用关键点识别算法，识别眼部位置的关键点，进而基于各人眼关键点之间的位置关系随时间的变化情况，来确定驾驶员是否频繁闭眼。然而，由于关键点是标记在一个个像素点上的，而眼部区域所占整张人脸图像的比例较小，所获取的眼部区域的像素点较少，导致关键点所标记的位置不准确，因此，利用人眼关键点的位置关系来确定驾驶员是否频繁闭眼的准确度不高，进一步地，会导致确定驾驶员是否处于疲劳驾驶状态的准确度不高。
为了解决上述问题，提高确定驾驶员是否疲劳驾驶的准确度，本申请实施例提供了一种疲劳驾驶的确定方法、装置及电子设备。
需要说明的是，在具体应用中，本申请实施例所提供的疲劳驾驶的确定方法可以应用于车辆内的图像采集设备中，例如车载摄像头，或者本申请实施例所提供的疲劳驾驶 的确定方法还可以应用于其他各类电子设备，例如，智能手机、个人电脑、服务器以及其他具有数据处理能力的设备，该电子设备还可以为车载中控。需要说明的是，当应用于其他各类电子设备时，该电子设备可以与车辆内的图像采集设备相互通信，从而获取车辆内的图像采集设备所采集的图像数据。并且，本申请实施例提供的疲劳驾驶的确定方法可以通过软件、硬件或软硬件结合的方式实现。
本申请实施例所提供的一种疲劳驾驶的确定方法，可以包括以下步骤：获取指定时间段内，针对驾驶员所采集的多帧图像数据；针对每一图像数据，从图像数据中，获取包含驾驶员的眼睛的眼部图像；利用预先训练的睁闭眼模型，识别每一眼部图像中驾驶员的睁闭眼状态；其中，睁闭眼模型是利用样本眼部图像以及样本眼部图像对应的睁闭眼状态的真值训练得到的；基于各眼部图像中驾驶员的睁闭眼状态，确定驾驶员是否疲劳驾驶。
本实施例中，通过利用预先训练的睁闭眼模型识别睁闭眼状态，由于睁闭眼模型在确定眼睛状态时所利用的是整个眼部图像，相比于相关技术中通过人眼关键点的确定的方式，准确度更高，从而可以提高睁闭眼状态确定的准确度，进而提高疲劳驾驶确定的准确度。
下面结合附图，对本申请实施例所提供的一种疲劳驾驶的确定方法进行介绍。
如图1所示，本申请实施例所提供的一种疲劳驾驶的确定方法，可以包括步骤S101-步骤S104。
S101，获取指定时间段内，针对驾驶员所采集的多帧图像数据；
其中，本申请实施例可以应用于配置有图像采集设备的车辆，对驾驶员进行疲劳驾驶的确定。图像采集设备可以安装于正对驾驶员的位置，或者车载中控的位置处等等。由于需要从每一帧图像数据中获取包含驾驶员的眼睛的眼部图像，需要保持图像采集设备的视野范围包含驾驶员的人脸。
多帧图像数据可以为图像采集设备所拍摄的视频流中的多帧图像数据。上述指定时间段可以为车辆启动后的任一时间段，示例性的，可以周期性地每隔五分钟，就采集10秒内图像采集设备所拍摄的视频流中的多帧图像数据。
在本申请实施例应用于图像采集设备的情况下，获取的多帧图像数据可以为图像采集设备所采集的多帧图像数据。
在本申请实施例应用于独立于图像采集设备的电子设备的情况下，获取的多帧图像数据可以为从图像采集设备拍摄的视频流中获取的多帧图像数据。
S102，针对每一图像数据，从图像数据中，获取包含驾驶员的眼睛的眼部图像；
本步骤中，为了提高疲劳驾驶确定的准确度，本申请实施例可以针对每一帧图像数据，从该图像数据中获取包含驾驶员的眼睛的眼部图像。其中，该眼部图像可以为整张图像数据，或者，为了提高后续识别的效率，该眼部图像可以为图像数据眼睛部位所在区域的图像。
可选的，所获取的眼部图像可以为单个眼睛所在区域的图像，例如左眼所在区域的左眼图像，或者右眼所在区域的右眼图像。或者，为了提高睁闭眼状态识别的准确度，所获取的眼部图像可以包括双眼的图像，一种方式，可以为左眼和右眼所在的共同区域的图像，此时眼部图像中既包含左眼，也包含右眼，另一种方式，所获取的眼部图像可以包括左眼图像和右眼图像两种图像。综合两只眼睛来进行疲劳驾驶的确定，相比一只眼睛更加准确。
示例性的，在一种实现方式中，还可以在获取到左眼图像和右眼图像之后，对左眼图像和右眼图像进行检测，确定左眼图像和右眼图像是否属于眼部图像，避免由于驾驶员侧脸导致识别出错误的眼部图像。获取驾驶员的左眼图像和右眼图像，并检测这两帧眼部图像是否存在眼睛，若只有一帧眼部图像中存在眼睛，则仅利用该张存在眼睛的眼部图像进行疲劳驾驶的确定，若两帧眼部图像中均存在眼睛，则可以综合两帧眼部图像 来进行疲劳驾驶的确定。
本申请实施例中，从图像数据中，获取包含驾驶员的眼睛的眼部图像的方式可以有多种，可选的，至少包括以下两种方式中的一种：
第一种眼部图像获取方式，可以利用预先训练完成的眼部图像提取模型从图像数据中，获取包含驾驶员的眼睛的眼部图像，其中，该眼部图像提取模型为利用样本人脸图像以及眼部图像的真值训练得到的。
第二种眼部图像获取方式，可以包括以下步骤A1-A4：
步骤A1，对图像数据进行头部检测，确定图像数据中包含的每一头部对应的头部区域，并对图像数据进行人脸检测，确定图像数据中包含的每一人脸对应的人脸区域；
在该步骤中，针对每一帧图像数据，可以利用目标检测算法或语义分割算法等方式，同时确定出该图像数据中的头部区域和人脸区域。
步骤A2，基于各头部区域和各人脸区域的位置，对各头部区域和各人脸区域进行关联；
一般地，属于同一个人员的人脸区域的位置应当包含在头部区域内，因此，可以将头部区域所包含的人脸区域，确定为该头部区域相关联的人脸区域，该头部区域和相关联的人脸区域属于同一人员。
步骤A3，从各与头部区域关联的人脸区域中，确定处于图像数据中指定区域内的人脸区域，或所占面积最大的人脸区域，作为驾驶员的人脸区域。
本方式中，针对每一帧图像数据，在确定该图像数据中驾驶员的人脸区域时，可以根据不同的情况，采用不同的方式确定该图像数据中的人脸区域，例如在该图像数据中只仅包含驾驶员的情况下，可以直接利用目标检测算法或者语义分割算法等方法，从该图像数据中确定驾驶员的头部区域和人脸区域。而在该图像数据中，不仅包含驾驶员，还包含其他人员的情况下，可以先确定图像数据中所有人员的头部区域和相关联的人脸区域，再从各与头部区域关联的人脸区域中，将处于该图像数据中指定区域内的人脸区域，或所占面积最大的人脸区域，作为驾驶员的人脸区域。其中，指定区域可以为驾驶位置在该图像数据中得位置，本领域技术人员所知的，在图像采集设备固定后，图像采集设备与驾驶位置之间的相对位置即固定，使得图像采集设备所采集的图像数据中，驾驶位置所在的区域也固定，本申请实施例中，将图像数据中所在的驾驶位置所在的区域作为指定区域，以图像采集设备安装在正对驾驶位置为例，上述指定区域可以为图像数据的中间区域。由于图像采集设备正对驾驶位置，也可以将各与头部区域关联的人脸区域中所占面积最大的人脸区域，作为驾驶员的人脸区域。
通过基于各头部区域和各人脸区域的位置，对各头部区域和各人脸区域进行关联，并从各与头部区域关联的人脸区域中，确定处于图像数据中指定区域内的人脸区域，或所占面积最大的人脸区域，作为驾驶员的人脸区域，可以避免除驾驶员以外的人员的干扰。
得到驾驶员的人脸区域后，可以将相关联的头部区域作为驾驶员的头部区域。或者，也可以先确定处于图像数据中指定区域内的头部区域，或所占面积最大的头部区域，作为驾驶员的头部区域，再将相关联的人脸区域作为驾驶员的人脸区域，也是可以的。
步骤A4，从所确定的人脸区域中，提取眼睛部位所在区域的图像，作为驾驶员的眼睛的眼部图像。
在确定出驾驶员的人脸区域之后，可以进一步的从所确定的人脸区域中，提取眼睛部位所在区域的图像，作为驾驶员的眼睛的眼部图像。可选的，在确定出人脸区域后，可以利用关键点识别算法，对所确定的人脸区域进行关键点识别，在识别出人脸区域的关键点之后，从所识别的关键点中确定出人眼关键点，进而根据人眼关键点确定出驾驶员的人眼所在的区域，并进一步的对所确定的区域进行分割，得到驾驶员的眼睛的眼部图像。
S103，利用预先训练的睁闭眼模型，识别每一眼部图像中驾驶员的睁闭眼状态；其中，睁闭眼模型为：利用样本眼部图像以及样本眼部图像对应的睁闭眼状态的真值训练得到的；
其中，上述睁闭眼状态包括睁眼状态和闭眼状态。在得到各眼部图像之后，可以利用预先训练的睁闭眼模型，识别每一眼部图像中驾驶员的睁闭眼状态。可选的，可以将每一眼部图像输入至睁闭眼模型，进而得到睁闭眼模型输出的该眼部图像中驾驶员的睁闭眼状态。示例性的，多帧图像数据包括图像数据1、图像数据2以及图像数据3，从图像数据1中提取出眼部图像1、从图像数据2中提取出眼部图像2、从图像数据3中提取出眼部图像3，则可以将眼部图像1输入至睁闭眼模型，得到眼部图像1中驾驶员的睁闭眼状态，将眼部图像2输入至睁闭眼模型，得到眼部图像2中驾驶员的睁闭眼状态，将眼部图像3输入至睁闭眼模型，得到眼部图像3中驾驶员的睁闭眼状态。
上述睁闭眼模型可以为利用样本眼部图像以及样本眼部图像对应的睁闭眼状态的真值训练得到的，其中，样本眼部图像可以为多张，一部分样本眼部图像中的眼睛为睁眼状态，另一部分为闭眼状态；
样本眼部图像对应的睁闭眼状态的真值为：该样本眼部图像中的眼睛的实际的睁闭眼状态，样本眼部图像对应的睁闭眼状态的真值可以通过人工标记的方式得到。
上述睁闭眼模型的训练方式可以包括：
将多张样本眼部图像输入该待训练的睁闭眼模型中，使得待训练的睁闭眼模型输出关于各张样本眼部图像是睁闭眼状态的识别结果；再利用各个样本眼部图像对应的识别结果与该样本眼部图像对应的睁闭眼状态的真值计算该睁闭眼模型的模型损失；进而基于模型损失调整该睁闭眼模型的参数，直到该模型损失收敛，得到训练完成的睁闭眼模型。
上述睁闭眼模型是预先训练完成的，在实际应用过程中可直接利用已经预先训练完成的睁闭眼模型，识别每一眼部图像中驾驶员的睁闭眼状态。
S104，基于各眼部图像中驾驶员的睁闭眼状态，确定驾驶员是否疲劳驾驶。
正常情况下，眼睛每分钟大约要眨动10-15次，每次眨眼大概0.2-0.4秒，当人疲劳时，眨眼的速度会变慢，眨眼的频率也会增多。
可选的，基于各眼部图像中驾驶员的睁闭眼状态，确定驾驶员是否疲劳驾驶的方式至少可以包括以下步骤：基于各眼部图像中驾驶员的睁闭眼状态，确定驾驶员在指定时间段内的闭眼信息；其中，闭眼信息指示驾驶员的闭眼次数、闭眼时长、闭眼频率中的至少一者；即确定闭眼信息是否满足预设的闭眼疲劳检测条件；若满足，则确定驾驶员疲劳驾驶，否则，确定驾驶员未疲劳驾驶。
上述确定闭眼信息是否满足预设的闭眼疲劳检测条件的方式可以有很多种，可选的，至少可以包括以下三种方式中的一种：
第一种确定方式，在预设的闭眼疲劳检测条件为指定时间段内驾驶员处于闭眼状态的时长达到预设时长阈值的情况下，可以统计指定时间段内驾驶员处于闭眼状态的时长是否达到预设时长阈值。可以理解的，由于本申请实施例中是利用每一帧图像数据来识别驾驶员的睁闭眼状态，图像采集设备可以每隔预定时间就采集一帧图像数据，例如一秒采集24帧图像数据，此时，可将每一帧图像数据所占的时长视为1/24秒。在这种情况下，统计指定时间段内驾驶员处于闭眼状态的图像数据的帧数，就能得到指定时间段内驾驶员处于闭眼状态的时长。若指定时间段内驾驶员处于闭眼状态的时长达到预设时长阈值，则确定满足预设的闭眼疲劳检测条件。
第二种确定方式，在预设的闭眼疲劳检测条件为指定时间段内驾驶员闭眼的次数达到预设次数阈值的情况下，可以统计指定时间段内驾驶员闭眼的次数是否达到预设次数阈值。可以理解的，在图像采集设备每隔预定时间就采集一帧图像数据的情况下，当出现前一帧图像数据对应的睁闭眼状态为睁眼状态，下一帧图像数据对应的睁闭眼状态变 为了闭眼状态的情况，则认为驾驶员闭眼一次。若指定时间段内驾驶员闭眼的次数达到预设次数阈值，则确定满足预设的闭眼疲劳检测条件。
第三种确定方式，在预设的闭眼疲劳检测条件为指定时间段内存在连续的预定数量帧第一目标图像数据的情况下，其中，第一目标图像数据是驾驶员处于闭眼状态的图像数据，可以检测指定时间段内是否存在连续的预定数量帧第一目标图像数据，若是，则确定满足预设的闭眼疲劳检测条件。
本实施例中，通过利用预先训练的睁闭眼模型识别睁闭眼状态，由于睁闭眼模型在确定眼睛状态时所利用的是整个眼部图像，相比于相关技术中通过人眼关键点的确定的方式，准确度更高，从而可以提高睁闭眼状态确定的准确度，进而提高疲劳驾驶确定的准确度。
为了进一步提高确定驾驶员是否疲劳驾驶的准确度，本申请实施例还引入了其他确定驾驶员是否疲劳驾驶的方式。可选的，如图2所示，在本申请的另一实施例中，在获取指定时间段内，针对驾驶中的驾驶员采集的多帧图像数据之后，本申请实施例所提供的一种疲劳驾驶的确定方法还包括S201：S201，获取各图像数据中，驾驶员的头部姿态信息和/或驾驶员的嘴巴张合状态；其中，头部姿态信息可以包括头部姿态角的角度，头部姿态角指头部的俯仰角、偏航角和翻转角。当驾驶员疲劳驾驶时，往往会出现打瞌睡的情况，人在打瞌睡时，头部会下垂或者向两边倾斜，其中，头部下垂对应头部的俯仰角，头部向两边倾斜对应头部的翻转角。因此，可以将头部姿态信息作为确定驾驶员是否疲劳驾驶的依据。
嘴巴张合状态可以包括张嘴状态和闭嘴状态。当驾驶员疲劳驾驶时，打哈欠的频率也会增多。因此，也可以将嘴巴张合状态作为确定驾驶员是否疲劳驾驶的依据。
基于各眼部图像中驾驶员的睁闭眼状态，确定驾驶员是否疲劳驾驶，可以包括步骤S202：S202，基于各眼部图像中驾驶员的睁闭眼状态、以及各图像数据中驾驶员的头部姿态信息和/或驾驶员的嘴巴张合状态，确定驾驶员是否疲劳驾驶。
本申请实施例中，可以利用驾驶员的睁闭眼状态，结合驾驶员的头部姿态信息和驾驶员的嘴巴张合状态两种方式，或者后两种方式中的任一种，来作为确定驾驶员是否疲劳驾驶的依据。
上述基于各眼部图像中驾驶员的睁闭眼状态、以及各图像数据中驾驶员的头部姿态信息和/或驾驶员的嘴巴张合状态，确定驾驶员是否疲劳驾驶，至少可以包括以下方式：基于各眼部图像中驾驶员的睁闭眼状态，预估驾驶员是否疲劳驾驶，得到第一预估结果；基于各图像数据中驾驶员的头部姿态信息，预估驾驶员是否疲劳驾驶，得到第二预估结果；和/或，基于各图像数据中驾驶员的嘴巴张合状态，预估驾驶员是否疲劳驾驶，得到第三预估结果；由于要结合各图像数据中驾驶员的头部姿态信息，和/或各图像数据中驾驶员的嘴巴张合状态共同作为确定驾驶员是否疲劳驾驶的依据，因此，驾驶员的睁闭眼状态、驾驶员的头部姿态信息、驾驶员的嘴巴张合状态三种依据中的单独一个仅用于预估驾驶员是否疲劳驾驶，得到预估结果，不直接用于确定驾驶员是否疲劳驾驶。
最后，基于第一预估结果，以及第二预估结果和/或第三预估结果，确定驾驶员是否疲劳驾驶。
其中，各个预估结果可以表征为，驾驶员疲劳驾驶，或驾驶员未疲劳驾驶。上述基于第一预估结果，以及第二预估结果和/或第三预估结果，确定驾驶员是否疲劳驾驶的方式可以有多种，至少可以包括以下两种方式中的一种：
第一种方式，第一预估结果、第二预估结果、第三预估结果表征为驾驶员未疲劳驾驶时，得分记为0；表征为驾驶员疲劳驾驶时，第一预估结果对应第一得分，第二预估结果对应第二得分，第三预估结果对应第三得分。最后将各个预估结果的得分相加得到总分，若总分达到预定的得分阈值，则确定驾驶员疲劳驾驶，若未达到，则确定驾驶员未疲劳驾驶。
第二种方式，若第一预估结果，以及第二预估结果和/或第三预估结果中，指示驾驶员疲劳驾驶的占比大于指示驾驶员未疲劳驾驶的占比，则确定驾驶员疲劳驾驶，否则，确定驾驶员为未疲劳驾驶。在该方式下，若利用了其中的两种预估结果来确定驾驶员是否疲劳驾驶，则需要这两个预估结果均表征为驾驶员疲劳驾驶时，确定驾驶员疲劳驾驶；若利用了三预估结果来确定驾驶员是否疲劳驾驶，则在第一预估结果、第二预估结果和第三预估结果中的两个预估结果为驾驶员疲劳驾驶时，则确定驾驶员疲劳驾驶。
本申请实施例中，在获取指定时间段内，针对驾驶中的驾驶员采集的多帧图像数据之后，获取各图像数据中，驾驶员的头部姿态信息和/或驾驶员的嘴巴张合状态；并基于各眼部图像中驾驶员的睁闭眼状态、以及各图像数据中驾驶员的头部姿态信息和/或驾驶员的嘴巴张合状态，确定驾驶员是否疲劳驾驶。可见，本方案中，可以根据实际应用情况利用驾驶员的睁闭眼状态，结合驾驶员的头部姿态信息和驾驶员的嘴巴张合状态两种方式，或者后两种方式中的一种方式来作为确定驾驶员是否疲劳驾驶的依据，从而可以灵活地根据实际应用情况调整确定驾驶员是否疲劳驾驶的方式，以提高不同场景下确定驾驶员是否疲劳驾驶的准确度。
上述基于各眼部图像中驾驶员的睁闭眼状态，预估驾驶员是否疲劳驾驶，与上述基于各眼部图像中驾驶员的睁闭眼状态，确定驾驶员是否疲劳驾驶的方式类似，可以包括：
基于各眼部图像中驾驶员的睁闭眼状态，确定驾驶员在指定时间段内的闭眼信息；其中，闭眼信息指示驾驶员的闭眼次数、闭眼时长、闭眼频率中的至少一者；
确定闭眼信息是否满足预设的闭眼疲劳检测条件；若满足，则确定第一预估结果为驾驶员疲劳驾驶，否则，确定第一预估结果为驾驶员未疲劳驾驶。
由于上文中已介绍基于各眼部图像中驾驶员的睁闭眼状态，确定驾驶员是否疲劳驾驶，在此不再赘述。下面分别具体地介绍基于各图像数据中驾驶员的头部姿态信息，预估驾驶员是否疲劳驾驶，和基于各图像数据中驾驶员的嘴巴张合状态，预估驾驶员是否疲劳驾驶。
首先介绍基于各图像数据中驾驶员的头部姿态信息，预估驾驶员是否疲劳驾驶。
需要说明的是，基于各图像数据中驾驶员的头部姿态信息，预估驾驶员是否疲劳驾驶的方式可以有多种，至少可以包括以下步骤B1-B2：
步骤B1，基于各图像数据中驾驶员的头部姿态信息，确定驾驶员在指定时间段内的头部晃动信息；其中，头部晃动信息指示驾驶员的点头次数、点头时长、点头频率中的至少一者；
在确定各个图像数据中驾驶员的头部区域后，可以从图像数据中提取出驾驶员的头部区域图像，再进一步地利用驾驶员的头部区域图像得到驾驶员的头部姿态信息。其中，头部姿态信息可以为驾驶员的头部的俯仰角和/或翻滚角，头部姿态信息也可以利用预先训练完成的人头姿态估计模型得到；与睁闭眼模型类似，该人头姿态估计模型可以是利用样本头部图像以及样本头部图像对应的俯仰角和/或翻滚角的真值训练得到的。
当驾驶员的头部的俯仰角和/或翻滚角达到预定的角度阈值时，则可以认为驾驶员发生了点头的情况。统计指定时间段内驾驶员的点头次数、点头时长、点头频率中的至少一者，得到驾驶员在指定时间段内的头部晃动信息。
步骤B2，确定头部晃动信息是否满足预设的头部疲劳检测条件；若满足，则确定驾驶员疲劳驾驶为第二预估结果，否则，确定驾驶员未疲劳驾驶为第二预估结果。
其中，头部疲劳检测条件可以为指定时间段内驾驶员的点头次数达到预定次数阈值和/或点头时长达到预定时长阈值。头部疲劳检测条件还可以为指定时间段内驾驶员的点头时长所占指定时间段的时间比例达到预定比例阈值。当然，头部疲劳检测条件也可以不限于此。
当头部晃动信息满足预设的头部疲劳检测条件时，则确定驾驶员疲劳驾驶为第二预估结果，若不满足，则确定驾驶员未疲劳驾驶为第二预估结果。
示例性的，在10s内，当驾驶员的头部的俯仰角或翻滚角的角度达到了25度，且时长占比达到了30％，即3s，则可以确定驾驶员疲劳驾驶为第二预估结果。
本实施例中，基于各图像数据中驾驶员的头部姿态信息，确定驾驶员在指定时间段内的头部晃动信息；确定头部晃动信息是否满足预设的头部疲劳检测条件；若满足，则确定驾驶员疲劳驾驶为第二预估结果，否则，确定驾驶员未疲劳驾驶为第二预估结果。可见，本方案中，提供了利用各图像数据中驾驶员的头部姿态信息预估驾驶员是否疲劳驾驶的方式，增加了各图像数据中驾驶员的头部姿态信息作为确定驾驶员是否疲劳驾驶的依据，从而提高不同场景下确定驾驶员是否疲劳驾驶的准确度。
下面介绍基于各图像数据中驾驶员的嘴巴张合状态，预估驾驶员是否疲劳驾驶。可选的，基于各图像数据中驾驶员的嘴巴张合状态，预估驾驶员是否疲劳驾驶可以包括以下步骤C1-C2：
步骤C1，基于各图像数据中驾驶员的嘴巴张合状态，确定驾驶员在指定时间段内的张嘴信息；其中，张嘴信息指示驾驶员的张嘴次数、张嘴时长、张嘴频率中的至少一者；
类似的，为了确定驾驶员的嘴巴张合状态，也可以先利用关键点识别算法，在上述人脸区域中驾驶员的嘴巴上标记出关键点，再根据关键点的位置，确定驾驶员的嘴巴张合状态。
示例性的，上述根据关键点的位置，确定驾驶员的嘴巴张合状态的方式可以有多种，可选的，至少包括以下方式：
如图3所示，图3展示了利用关键点识别算法在驾驶员的嘴巴上标记出关键点的示意图，图3中，点50-点68为关键点识别算法所识别的嘴部的关键点，本申请实施例中用p50-p68分别表示。可以利用关键点计算嘴巴的纵横比，纵横比＝垂直距离/水平距离。以图3为例，垂直距离可以为p51点和p53点的连线的中点，与p59点和p57点连线的中点之间的距离；水平距离可以为p49点和p55点之间的距离。其中，垂直距离的确定方式也可以不仅仅局限于此，例如，垂直距离还可以为p52点到p58点的距离等。若计算出的纵横比达到了预定纵横比阈值，预定纵横比阈值通常设置为0.5，则确定驾驶员的嘴巴为张开状态，否则，确定驾驶员的嘴巴为闭合状态。
在确定各个图像数据中驾驶员的嘴巴区域后，可以从图像数据中提取出驾驶员的嘴巴区域图像，再进一步地利用驾驶员的嘴巴区域图像得到驾驶员的嘴巴张合状态。嘴巴张合状态可以利用预先训练完成的嘴巴张合模型得到。与睁闭眼模型类似，该嘴巴张合模型是利用样本嘴部图像以及样本嘴部图像对应的嘴巴张合状态的真值训练得到的。
张嘴信息则用于指示驾驶员的在指定时间段内的张嘴次数、张嘴时长、张嘴频率中的至少一者。
步骤C2，确定张嘴信息是否满足预设的嘴部疲劳检测条件；若满足，则确定驾驶员疲劳驾驶为第三预估结果，否则，确定驾驶员未疲劳驾驶为第三预估结果。
其中，确定张嘴信息是否满足预设的嘴部疲劳检测条件与上述确定闭眼信息是否满足预设的闭眼疲劳检测条件的方式类似。例如，预设的嘴部疲劳检测条件可以为，指定时间段内存在连续的预定数量帧第二目标图像数据，其中，第二目标图像数据为驾驶员嘴巴处于张开嘴状态的图像数据。该情况下，可以检测指定时间段内是否存在连续的预定数量帧第二目标图像数据，若是，则确定满足预设的嘴部疲劳检测条件。其他确定张嘴信息是否满足预设的嘴部疲劳检测条件的方式在此不再赘述。
本实施例中，基于各图像数据中驾驶员的嘴巴张合状态，确定驾驶员在指定时间段内的张嘴信息；确定张嘴信息是否满足预设的嘴部疲劳检测条件；若满足，则确定驾驶员疲劳驾驶为第三预估结果，否则，确定驾驶员未疲劳驾驶为第三预估结果。可见，本方案中，提供了利用各图像数据中驾驶员的嘴巴张合状态预估驾驶员是否疲劳驾驶的方式，增加了各图像数据中驾驶员的嘴巴张合状态作为确定驾驶员是否疲劳驾驶的依据，从而提高不同场景下确定驾驶员是否疲劳驾驶的准确度。
可选地，在本申请的另一实施例中，在上述获取各图像数据中，驾驶员的头部姿态信息和/或驾驶员的嘴巴张合状态之前，本申请实施例所提供的疲劳驾驶的确定方法还可以包括：
确定指定时间段内驾驶员的嘴巴是否被遮挡；
考虑到在驾驶员嘴部存在障碍物的情况，例如口罩等，此时无法判断驾驶员的嘴巴张合状态，从而无法利用驾驶员的嘴巴张合状态来预估驾驶员是否疲劳驾驶。因此，可以在获取各图像数据中，驾驶员的头部姿态信息和/或驾驶员的嘴巴张合状态之前，确定指定时间段内驾驶员的嘴巴是否被遮挡。确定指定时间段内驾驶员的嘴巴是否被遮挡，至少可以包括以下方式：
针对每一图像数据，从图像数据中，确定驾驶员的人脸区域，并从图像数据中截取出驾驶员的人脸图像，进而可以将人脸图像输入至预先训练的第一人脸质量模型，以检测该人脸图像中驾驶员的嘴巴是否被遮挡。可选的，为了提高识别的准确度，可以在将人脸图像输入至预先训练的第一人脸质量模型之前，利用人脸对齐算法将人脸图像中的人脸对齐到统一的形状，进而将对齐后的人脸图像输入至第一人脸质量模型。上述第一人脸质量模型用于检测人脸图像的嘴巴是否被遮挡，可以利用样本人脸图像、以及样本人脸图像对应的是否嘴巴被遮挡的真值训练得到。
获取各图像数据中，驾驶员的头部姿态信息和/或驾驶员的嘴巴张合状态，可以包括：若在指定时间段内驾驶员的嘴巴被遮挡，获取各图像数据中，驾驶员的头部姿态信息；当确定指定时间段内驾驶员的嘴巴被遮挡，则说明无法利用驾驶员的嘴巴张合状态来预估驾驶员是否疲劳驾驶，此时可以不用获取驾驶员的嘴巴张合状态，但可以利用驾驶员的头部姿态信息预估驾驶员是否疲劳驾驶。
若在指定时间段内驾驶员的嘴巴未被遮挡，获取各图像数据中，驾驶员的头部姿态信息以及驾驶员的嘴巴张合状态。
当确定指定时间段内驾驶员的嘴巴未被遮挡，则说明可以利用驾驶员的嘴巴张合状态来预估驾驶员是否疲劳驾驶，此时可以获取各图像数据中，驾驶员的头部姿态信息以及驾驶员的嘴巴张合状态，从而可以基于各图像数据中驾驶员的头部姿态信息，预估驾驶员是否疲劳驾驶，得到第二预估结果，同时基于各图像数据中驾驶员的嘴巴张合状态，预估驾驶员是否疲劳驾驶，得到第三预估结果。第二预估结果和第三预估结果均用于确定驾驶员是否疲劳驾驶。
同样的，驾驶员的眼睛也可能存在被遮挡的情况，例如驾驶员戴了墨镜等情况，此时无法判断驾驶员的睁闭眼状态，从而无法利用驾驶员的睁闭眼状态来预估驾驶员是否疲劳驾驶。在针对每一图像数据，从图像数据中，获取包含驾驶员的眼睛的眼部图像之前，本申请实施例所提供的疲劳驾驶的确定方法还可以包括：确定指定时间段内驾驶员的眼睛是否被遮挡。
与确定指定时间段内驾驶员的嘴巴是否被遮挡的方式类似，可以在获取每一帧图像数据之后，确定驾驶员的人脸区域，并从图像数据中截取出驾驶员的人脸图像，进而可以将人脸图像输入至预先训练的第二人脸质量模型，以检测该人脸图像中驾驶员的眼睛是否被遮挡。同样的，为了提高识别的准确度，可以在将人脸图像输入至预先训练的第二人脸质量模型之前，利用人脸对齐算法将人脸图像中的人脸对齐到统一的形状，进而将对齐后的人脸图像输入至第二人脸质量模型。上述第二人脸质量模型用于检测人脸图像的眼睛是否被遮挡，可以利用样本人脸图像、以及样本人脸图像对应的是否眼睛被遮挡的真值训练得到。
若未被遮挡，则执行针对每一图像数据，从图像数据中，获取包含驾驶员的眼睛的眼部图像的步骤。
当指定时间段内驾驶员眼睛未被遮挡，则可以利用驾驶员的睁闭眼状态来评估驾驶员是否疲劳驾驶。
若被遮挡，则基于所获取的驾驶员的头部姿态信息，和/或驾驶员的嘴巴张合状态，确定驾驶员是否疲劳驾驶。
当指定时间段内驾驶员眼睛被遮挡，说明无法利用驾驶员的睁闭眼状态来评估驾驶员是否疲劳驾驶，此时，可以不执行针对每一图像数据，从图像数据中，获取包含驾驶员的眼睛的眼部图像的步骤，而执行获取各图像数据中，驾驶员的头部姿态信息，和/或驾驶员的嘴巴张合状态的步骤，从而利用驾驶员的头部姿态信息，和/或驾驶员的嘴巴张合状态，确定驾驶员是否疲劳驾驶。
可选的，在一种实现方式中，为了提高检测效率，同时检测驾驶员眼睛和嘴巴是否被遮挡，此时，在获取每一帧人脸图像之后，可以将人脸图像输入至第三人脸质量模型，以同时得到驾驶员眼睛和嘴巴是否被遮挡的检测结果。此时，上述第三人脸质量模型可以利用样本人脸图像、以及样本人脸图像对应的眼睛和嘴巴是否被遮挡的真值训练得到。上述检测结果会存在以下四种情况：眼睛和嘴巴均未被遮挡；眼睛被遮挡，嘴巴未被遮挡；眼睛未被遮挡，嘴巴被遮挡；眼睛和嘴巴均被遮挡。
此时，考虑到驾驶员的嘴巴和眼睛被遮挡的情况，本申请实施例可以分为以下四种情况：
第一种情况，指定时间段内驾驶员的眼睛和嘴巴均未被遮挡，该情况下，可以获取各图像数据中，驾驶员的睁闭眼状态，驾驶员的头部姿态信息和驾驶员的嘴巴张合状态；基于各眼部图像中驾驶员的睁闭眼状态，预估驾驶员是否疲劳驾驶，得到第一预估结果；基于各图像数据中驾驶员的头部姿态信息，预估驾驶员是否疲劳驾驶，得到第二预估结果；并基于各图像数据中驾驶员的嘴巴张合状态，预估驾驶员是否疲劳驾驶，得到第三预估结果；最后基于第一预估结果、第二预估结果和第三预估结果，确定驾驶员是否疲劳驾驶。
第二种情况，指定时间段内驾驶员的眼睛被遮挡，嘴巴未被遮挡，该情况下，可以获取各图像数据中，驾驶员的头部姿态信息和驾驶员的嘴巴张合状态；基于各图像数据中驾驶员的头部姿态信息，预估驾驶员是否疲劳驾驶，得到第二预估结果；并基于各图像数据中驾驶员的嘴巴张合状态，预估驾驶员是否疲劳驾驶，得到第三预估结果；最后基于第二预估结果和第三预估结果，确定驾驶员是否疲劳驾驶。
第三种情况，指定时间段内驾驶员的眼睛未被遮挡，嘴巴被遮挡，该情况下，可以获取各图像数据中，驾驶员的睁闭眼状态和驾驶员的头部姿态信息；基于各眼部图像中驾驶员的睁闭眼状态，预估驾驶员是否疲劳驾驶，得到第一预估结果；基于各图像数据中驾驶员的头部姿态信息，预估驾驶员是否疲劳驾驶，得到第二预估结果；最后基于第一预估结果和第二预估结果，确定驾驶员是否疲劳驾驶。
第四种情况，指定时间段内驾驶员的眼睛和嘴巴均被遮挡，该情况下，可以仅获取各图像数据中，驾驶员的头部姿态信息；最后基于各图像数据中驾驶员的头部姿态信息直接确定驾驶员是否疲劳驾驶。
本实施例中，考虑了在指定时间段内，驾驶员眼睛和/或嘴巴是否被遮挡的不同情况，从而解决了驾驶员眼睛或嘴巴被遮挡时，无法利用驾驶员眼睛或嘴巴评估驾驶员是否疲劳驾驶的问题。
为了方便理解，下面结合附图对本申请所提供的一种疲劳驾驶的确定方法进行示例性介绍。
如图4所示，在一实施例中，本申请实施例可以包括如下步骤：
步骤1，首先对采集的图像数据进行头部检测和人脸检测，生成图像数据中所有人员的头部区域的识别框，和人脸区域的识别框，从而确定各个头部区域和人脸区域。
步骤2，根据各个头部区域的识别框，和人脸区域的识别框的位置，对各个头部区域和各人脸区域进行关联，选取面积最大的头部区域作为驾驶员的头部区域，从而截取出驾驶员头部图像和人脸图像。
步骤3，判断图像数据中是否存在头部区域，若是，获取头部姿态信息。
步骤4，利用关键点识别算法，在人脸图像中生成人脸的关键点，确定眼部区域，进而得到眼部图像。
步骤5，利用人脸对齐算法(相似性变换或放射变换)，把人脸图像中的人脸对齐到统一的形状。
步骤6，利用图像质量评价算法评估人脸图像的质量，得到人脸图像的图像质量得分，同时，检测驾驶员的嘴巴是否被遮挡，得到嘴部质量得分，示例性的，可以针对驾驶员的嘴巴被遮挡的情况设定对应的嘴部质量得分，以及针对驾驶员的嘴巴未被遮挡的情况设定另一对应的嘴部质量得分；同样的，检测驾驶员的眼睛否被遮挡，得到眼部质量得分；加权相加图像质量得分、嘴部质量得分和眼部质量得分，得到人脸质量得分。
步骤7，判断人脸质量得分是否达到预定得分阈值，若否，则仅基于各图像数据中驾驶员的头部姿态信息，预估驾驶员是否疲劳驾驶，得到第二预估结果；若是，则基于各图像数据中驾驶员的头部姿态信息，预估驾驶员是否疲劳驾驶，得到第二预估结果，并执行步骤8。
步骤8，判断驾驶员的眼睛是否被遮挡；若否，则利用睁闭眼模型识别每一眼部图像中驾驶员的睁闭眼状态，基于各眼部图像中驾驶员的睁闭眼状态，预估驾驶员是否疲劳驾驶，得到第一预估结果，并执行步骤9；若是，则直接执行步骤9。
步骤9，判断驾驶员的嘴巴是否被遮挡。
步骤10，若驾驶员的嘴巴未被遮挡，则获取各图像数据中，驾驶员的嘴巴张合状态，基于各图像数据中驾驶员的嘴巴张合状态，预估驾驶员是否疲劳驾驶，得到第三预估结果。
步骤11，根据上述过程中获取的各个预估结果，确定驾驶员是否疲劳驾驶。
可见，本实施例中，通过利用预先训练的睁闭眼模型识别睁闭眼状态，由于睁闭眼模型在确定眼睛状态时所利用的是整个眼部图像，相比于相关技术中通过人眼关键点的确定的方式，准确度更高，从而可以提高睁闭眼状态确定的准确度，进而提高疲劳驾驶确定的准确度。进一步地，还将驾驶员的头部姿态信息和驾驶员的嘴巴张合状态也作为确定驾驶员是否疲劳驾驶的依据，从而可以灵活地根据实际应用情况调整确定驾驶员是否疲劳驾驶的方式，以提高不同场景下确定驾驶员是否疲劳驾驶的准确度。
如图5所示，本申请实施例所提供的一种疲劳驾驶的确定方法，可以包括步骤S301-步骤S305。
S301，获取指定时间段内，针对驾驶员所采集的多帧图像数据。
在本申请的实施例中，S301中多帧图像数据的采集可参考上述S101，本申请对此不再赘述。
S302，针对每一所述图像数据，从所述图像数据中，获取所述驾驶员的人脸图像。
在本申请的实施例中，针对每一图像数据，从图像数据中，确定驾驶员的人脸区域，并从图像数据中截取出驾驶员的人脸图像。在本申请的实施例中，驾驶员的人脸区域可以包括驾驶员的头部区域，驾驶员的人脸图像可以包含驾驶员的头部图像。
S303，获取每一所述人脸图像中所述驾驶员的目标人脸区域的人脸关键点，并根据所述目标人脸区域的人脸关键点获取目标人脸图像。
在本申请的一些实施例中，所述目标人脸区域包括眼部区域、头部区域和嘴部区域中的至少一种。在本申请的一些实施例中，所述目标人脸图像包括所述驾驶员的眼睛的眼部图像、所述驾驶员的头部的头部图像和所述驾驶员的嘴巴的嘴部图像中的至少一种。
S304，将所述目标人脸图像输入预先训练的人脸状态模型，获取每一所述目标人脸图像中所述驾驶员的人脸状态；其中，所述人脸状态模型是利用样本目标人脸图像以及所述样本目标人脸图像对应的人脸状态的真值训练得到的。
在本申请的一些实施例中，人脸状态模型包括睁闭眼模型、嘴巴张合模型和人头姿 态估计模型。其中，睁闭眼模型可以为利用样本眼部图像以及样本眼部图像对应的睁闭眼状态的真值训练得到的，人头姿态估计模型可以为利用样本头部图像以及样本头部图像对应的俯仰角和/或翻滚角的真值训练得到的，嘴巴张合模型可以为利用样本嘴部图像以及样本嘴部图像对应的嘴巴张合状态的真值训练得到的。
在本申请的一些实施例中，将所述目标人脸图像输入预先训练的人脸状态模型，获取每一所述目标人脸图像中所述驾驶员的人脸状态，包括：利用预先训练的人脸状态模型，获取各所述眼部图像中所述驾驶员的睁闭眼状态、和/或各所述头部图像中所述驾驶员的头部姿态信息和/或各所述嘴部图像中所述驾驶员的嘴巴张合状态。
在本申请的一些实施例中，将所述目标人脸图像输入预先训练的人脸状态模型，获取每一所述目标人脸图像中所述驾驶员的人脸状态，包括：利用预先训练的睁闭眼模型，获取每一眼部图像中驾驶员的睁闭眼状态；和/或，利用预先训练的嘴巴张合模型，获取每一嘴部图像中驾驶员的嘴巴张合状态；和/或，利用预先训练的人头姿态估计模型，获取每一头部图像中驾驶员的头部姿态信息。
S305，基于各目标人脸图像中所述驾驶员的人脸状态，确定所述驾驶员是否疲劳驾驶。
在本申请的一些实施例中，基于各目标人脸图像中所述驾驶员的人脸状态，确定所述驾驶员是否疲劳驾驶，包括：基于各所述眼部图像中所述驾驶员的睁闭眼状态、和/或各所述头部图像中所述驾驶员的头部姿态信息和/或各所述嘴部图像中所述驾驶员的嘴巴张合状态，确定所述驾驶员是否疲劳驾驶。
在本申请的一些实施例中，所述基于各所述眼部图像中所述驾驶员的睁闭眼状态、和/或各所述头部图像中所述驾驶员的头部姿态信息和/或各所述嘴部图像中所述驾驶员的嘴巴张合状态，确定所述驾驶员是否疲劳驾驶，包括：基于各所述眼部图像中所述驾驶员的睁闭眼状态，预估所述驾驶员是否疲劳驾驶，得到第一预估结果；和/或，基于各所述头部图像中所述驾驶员的头部姿态信息，预估所述驾驶员是否疲劳驾驶，得到第二预估结果；和/或，基于各所述嘴部图像中所述驾驶员的嘴巴张合状态，预估所述驾驶员是否疲劳驾驶，得到第三预估结果；基于所述第一预估结果、和/或所述第二预估结果和/或所述第三预估结果，确定所述驾驶员是否疲劳驾驶。
在本申请的一些实施例中，所述基于各所述眼部图像中所述驾驶员的睁闭眼状态，预估所述驾驶员是否疲劳驾驶，包括：基于各所述眼部图像中所述驾驶员的睁闭眼状态，确定所述驾驶员在所述指定时间段内的闭眼信息；其中，所述闭眼信息指示所述驾驶员的闭眼次数、闭眼时长、闭眼频率中的至少一种；确定所述闭眼信息是否满足预设的闭眼疲劳检测条件；若满足，则确定所述驾驶员疲劳驾驶为第一预估结果，否则，确定所述驾驶员未疲劳驾驶为第一预估结果。
在本申请的一些实施例中，所述基于各所述头部图像中所述驾驶员的头部姿态信息，预估所述驾驶员是否疲劳驾驶，包括：基于各所述头部图像中所述驾驶员的头部姿态信息，确定所述驾驶员在所述指定时间段内的头部晃动信息；其中，所述头部晃动信息指示所述驾驶员的点头次数、点头时长、点头频率中的至少一种；确定所述头部晃动信息是否满足预设的头部疲劳检测条件；若满足，则确定所述驾驶员疲劳驾驶为第二预估结果，否则，确定所述驾驶员未疲劳驾驶为第二预估结果。
在本申请的一些实施例中，所述基于各所述嘴部图像中所述驾驶员的嘴巴张合状态，预估所述驾驶员是否疲劳驾驶，包括：基于各所述嘴部图像中所述驾驶员的嘴巴张合状态，确定所述驾驶员在所述指定时间段内的张嘴信息；其中，所述张嘴信息指示所述驾驶员的张嘴次数、张嘴时长、张嘴频率中的至少一种；确定所述张嘴信息是否满足预设的嘴部疲劳检测条件；若满足，则确定所述驾驶员疲劳驾驶为第三预估结果，否则，确定所述驾驶员未疲劳驾驶为第三预估结果。
在本申请的一些实施例中，所述基于所述第一预估结果、和/或所述第二预估结果 和/或所述第三预估结果，确定所述驾驶员是否疲劳驾驶，包括：若所述第一预估结果、和/或所述第二预估结果和/或所述第三预估结果中，指示所述驾驶员疲劳驾驶的占比大于指示所述驾驶员未疲劳驾驶的占比，则确定所述驾驶员疲劳驾驶，否则，确定所述驾驶员为未疲劳驾驶。
在本申请的一些实施例中，所述方法还包括：确定所述指定时间段内所述驾驶员的嘴巴和所述驾驶员的眼睛是否被遮挡；所述利用预先训练的人脸状态模型，获取各所述眼部图像中所述驾驶员的睁闭眼状态、和/或各所述头部图像中所述驾驶员的头部姿态信息和/或各所述嘴部图像中所述驾驶员的嘴巴张合状态，包括：若在所述指定时间段内所述驾驶员的嘴巴被遮挡且所述驾驶员的眼睛未被遮挡，则获取各所述头部图像中所述驾驶员的头部姿态信息和各所述眼部图像中所述驾驶员的睁闭眼状态；若在所述指定时间段内所述驾驶员的眼睛被遮挡且所述驾驶员的嘴巴未被遮挡，则获取各所述头部图像中所述驾驶员的头部姿态信息和各所述嘴部图像中所述驾驶员的嘴巴张合状态；若在所述指定时间段内所述驾驶员的眼睛和所述驾驶员的嘴巴均被遮挡，则获取各所述头部图像中所述驾驶员的头部姿态信息。
在本申请的一些实施例中，获取所述驾驶员的各所述眼部图像，包括：对所述图像数据进行头部检测，确定所述图像数据中包含的每一头部对应的头部区域，并对所述图像数据进行人脸检测，确定所述图像数据中包含的每一人脸对应的人脸区域；基于各头部区域和各人脸区域的位置，对各头部区域和各人脸区域进行关联；从各与头部区域关联的人脸区域中，确定处于所述图像数据中指定区域内，或所占面积最大的人脸区域，作为所述驾驶员的人脸区域；从所述驾驶员的人脸区域中，提取眼睛部位所在区域的图像，作为所述驾驶员的眼睛的眼部图像。
本申请实施例还提供了一种疲劳驾驶的确定装置，如图6所示，该装置包括：第一获取模块510，用于获取指定时间段内，针对驾驶员所采集的多帧图像数据；第二获取模块520，用于针对每一所述图像数据，从所述图像数据中，获取所述驾驶员的人脸图像；第三获取模块530，用于获取每一所述人脸图像中所述驾驶员的目标人脸区域的人脸关键点，并根据所述目标人脸区域的人脸关键点获取目标人脸图像；第四获取模块540，用于将所述目标人脸图像输入预先训练的人脸状态模型，获取每一所述目标人脸图像中所述驾驶员的人脸状态；其中，所述人脸状态模型是利用样本目标人脸图像以及所述样本目标人脸图像对应的人脸状态的真值训练得到的；第一确定模块550，用于基于各目标人脸图像中所述驾驶员的人脸状态，确定所述驾驶员是否疲劳驾驶。
可选地，所述目标人脸图像包括所述驾驶员的眼睛的眼部图像、头部的头部图像和嘴巴的嘴部图像中的至少一种；第四获取模块具体用于：利用预先训练的人脸状态模型，获取各所述眼部图像中所述驾驶员的睁闭眼状态、和/或各所述头部图像中所述驾驶员的头部姿态信息和/或各所述嘴部图像中所述驾驶员的嘴巴张合状态；所述第一确定模块具体用于：基于各所述眼部图像中所述驾驶员的睁闭眼状态、和/或各所述头部图像中所述驾驶员的头部姿态信息和/或各所述嘴部图像中所述驾驶员的嘴巴张合状态，确定所述驾驶员是否疲劳驾驶。
可选地，所述第一确定模块，包括：第一预估子模块，用于基于各眼部图像中所述驾驶员的睁闭眼状态，预估所述驾驶员是否疲劳驾驶，得到第一预估结果；和/或，基于各头部图像中所述驾驶员的头部姿态信息，预估所述驾驶员是否疲劳驾驶，得到第二预估结果；和/或，基于各嘴部图像中所述驾驶员的嘴巴张合状态，预估所述驾驶员是否疲劳驾驶，得到第三预估结果；第一确定子模块，用于基于所述第一预估结果、和/或所述第二预估结果和/或所述第三预估结果，确定所述驾驶员是否疲劳驾驶。
可选地，所述第一预估子模块，包括：第一确定单元，用于基于各眼部图像中所述驾驶员的睁闭眼状态，确定所述驾驶员在所述指定时间段内的闭眼信息；其中，所述闭眼信息指示所述驾驶员的闭眼次数、闭眼时长、闭眼频率中的至少一种；第二确定单元， 用于确定所述闭眼信息是否满足预设的闭眼疲劳检测条件；若满足，则确定所述第一预估结果为驾驶员疲劳驾驶，否则，确定所述第一预估结果为驾驶员未疲劳驾驶。
可选地，所述第一预估子模块基于各头部图像中所述驾驶员的头部姿态信息，预估所述驾驶员是否疲劳驾驶，得到第二预估结果，包括：基于各头部图像中所述驾驶员的头部姿态信息，确定所述驾驶员在所述指定时间段内的头部晃动信息；其中，所述头部晃动信息指示所述驾驶员的点头次数、点头时长、点头频率中的至少一种；确定所述头部晃动信息是否满足预设的头部疲劳检测条件；若满足，则确定所述第二预估结果为驾驶员疲劳驾驶，否则，确定所述第二预估结果为驾驶员未疲劳驾驶。
可选地，所述第一预估子模块基于各嘴部图像中所述驾驶员的嘴巴张合状态，预估所述驾驶员是否疲劳驾驶，包括：基于各嘴部图像中所述驾驶员的嘴巴张合状态，确定所述驾驶员在所述指定时间段内的张嘴信息；其中，所述张嘴信息指示所述驾驶员的张嘴次数、张嘴时长、张嘴频率中的至少一种；确定所述张嘴信息是否满足预设的嘴部疲劳检测条件；若满足，则确定所述第三预估结果为驾驶员疲劳驾驶，否则，确定所述第三预估结果为驾驶员未疲劳驾驶。
可选地，所述第一确定子模块具体用于：若所述第一预估结果、和/或所述第二预估结果和/或所述第三预估结果中，指示所述驾驶员疲劳驾驶的占比大于指示所述驾驶员未疲劳驾驶的占比，则确定所述驾驶员疲劳驾驶，否则，确定所述驾驶员为未疲劳驾驶。
可选地，所述装置还包括：第二确定模块，用于确定所述指定时间段内所述驾驶员的嘴巴和所述驾驶员的眼睛是否被遮挡；所述第一确定模块具体用于：若在所述指定时间段内所述驾驶员的嘴巴被遮挡且所述驾驶员的眼睛未被遮挡，则获取各所述头部图像中所述驾驶员的头部姿态信息和各所述眼部图像中所述驾驶员的睁闭眼状态；若在所述指定时间段内所述驾驶员的眼睛被遮挡且所述驾驶员的嘴巴未被遮挡，则获取各所述头部图像中所述驾驶员的头部姿态信息和各所述嘴部图像中所述驾驶员的嘴巴张合状态；若在所述指定时间段内所述驾驶员的眼睛和所述驾驶员的嘴巴均被遮挡，则获取各所述头部图像中所述驾驶员的头部姿态信息。
可选地，所述第二获取模块包括：第二确定子模块，用于对所述图像数据进行头部检测，确定所述图像数据中包含的每一头部对应的头部区域，并对所述图像数据进行人脸检测，确定所述图像数据中包含的每一人脸对应的人脸区域；关联子模块，用于基于各头部区域和各人脸区域的位置，对各头部区域和各人脸区域进行关联；第三确定子模块，用于从各与头部区域关联的人脸区域中，确定处于所述图像数据中指定区域内的人脸区域，或所占面积最大的人脸区域，作为所述驾驶员的人脸区域；提取子模块，用于从所确定的人脸区域中，提取眼睛部位所在区域的图像，作为所述驾驶员的眼睛的眼部图像。
本申请实施例还提供了一种电子设备，如图7所示，包括处理器601、通信接口602、存储器603和通信总线604，其中，处理器601，通信接口602，存储器603通过通信总线604完成相互间的通信，存储器603，用于存放计算机程序；处理器601，用于执行存储器603上所存放的程序时，实现上述疲劳驾驶的确定方法。
上述电子设备提到的通信总线可以是外设部件互连标准(Peripheral Component Interconnect，PCI)总线或扩展工业标准结构(Extended Industry Standard Architecture，EISA)总线等。该通信总线可以分为地址总线、数据总线、控制总线等。为便于表示，图中仅用一条粗线表示，但并不表示仅有一根总线或一种类型的总线。
通信接口用于上述电子设备与其他设备之间的通信。
存储器可以包括随机存取存储器(Random Access Memory，RAM)，也可以包括非易失性存储器(Non-Volatile Memory，NVM)，例如至少一个磁盘存储器。可选的，存储器还可以是至少一个位于远离前述处理器的存储装置。
上述的处理器可以是通用处理器，包括中央处理器(Central Processing Unit，CPU)、 网络处理器(Network Processor，NP)等；还可以是数字信号处理器(Digital Signal Processor，DSP)、专用集成电路(Application Specific Integrated Circuit，ASIC)、现场可编程门阵列(Field-Programmable Gate Array，FPGA)或者其他可编程逻辑器件、分立门或者晶体管逻辑器件、分立硬件组件。
在本申请提供的又一实施例中，还提供了一种计算机可读存储介质，该计算机可读存储介质内存储有计算机程序，所述计算机程序被处理器执行时实现上述疲劳驾驶的确定方法的步骤。
在本申请提供的又一实施例中，还提供了一种包含指令的计算机程序产品，当其在计算机上运行时，使得计算机执行上述疲劳驾驶的确定方法。
在上述实施例中，可以全部或部分地通过软件、硬件、固件或者其任意组合来实现。当使用软件实现时，可以全部或部分地以计算机程序产品的形式实现。所述计算机程序产品包括一个或多个计算机指令。在计算机上加载和执行所述计算机程序指令时，全部或部分地产生按照本申请实施例所述的流程或功能。所述计算机可以是通用计算机、专用计算机、计算机网络、或者其他可编程装置。所述计算机指令可以存储在计算机可读存储介质中，或者从一个计算机可读存储介质向另一个计算机可读存储介质传输，例如，所述计算机指令可以从一个网站站点、计算机、服务器或数据中心通过有线(例如同轴电缆、光纤、数字用户线(DSL))或无线(例如红外、无线、微波等)方式向另一个网站站点、计算机、服务器或数据中心进行传输。所述计算机可读存储介质可以是计算机能够存取的任何可用介质或者是包含一个或多个可用介质集成的服务器、数据中心等数据存储设备。所述可用介质可以是磁性介质，(例如，软盘、硬盘、磁带)、光介质(例如，DVD)、或者半导体介质(例如固态硬盘Solid State Disk(SSD))等。
需要说明的是，在本文中，诸如第一和第二等之类的关系术语仅仅用来将一个实体或者操作与另一个实体或操作区分开来，而不一定要求或者暗示这些实体或操作之间存在任何这种实际的关系或者顺序。而且，术语“包括”、“包含”或者其任何其他变体意在涵盖非排他性的包含，从而使得包括一系列要素的过程、方法、物品或者设备不仅包括那些要素，而且还包括没有明确列出的其他要素，或者是还包括为这种过程、方法、物品或者设备所固有的要素。在没有更多限制的情况下，由语句“包括一个……”限定的要素，并不排除在包括所述要素的过程、方法、物品或者设备中还存在另外的相同要素。
本说明书中的各个实施例均采用相关的方式描述，各个实施例之间相同相似的部分互相参见即可，每个实施例重点说明的都是与其他实施例的不同之处。
以上所述仅为本申请的较佳实施例，并非用于限定本申请的保护范围。凡在本申请的精神和原则之内所作的任何修改、等同替换、改进等，均包含在本申请的保护范围内。

Claims (12)

  1. 一种疲劳驾驶的确定方法，其特征在于，所述方法包括：
    获取指定时间段内，针对驾驶员所采集的多帧图像数据；
    针对每一所述图像数据，从所述图像数据中，获取所述驾驶员的人脸图像；
    获取每一所述人脸图像中所述驾驶员的目标人脸区域的人脸关键点，并根据所述目标人脸区域的人脸关键点获取目标人脸图像；
    将所述目标人脸图像输入预先训练的人脸状态模型，获取每一所述目标人脸图像中所述驾驶员的人脸状态；其中，所述人脸状态模型是利用样本目标人脸图像以及所述样本目标人脸图像对应的人脸状态的真值训练得到的；
    基于各目标人脸图像中所述驾驶员的人脸状态，确定所述驾驶员是否疲劳驾驶。
  2. 根据权利要求1所述的方法，其特征在于，所述目标人脸图像包括所述驾驶员的眼睛的眼部图像、头部的头部图像和嘴巴的嘴部图像中的至少一种；
    所述将所述目标人脸图像输入预先训练的人脸状态模型，获取每一所述目标人脸图像中所述驾驶员的人脸状态，包括：
    利用预先训练的人脸状态模型，获取各所述眼部图像中所述驾驶员的睁闭眼状态、和/或各所述头部图像中所述驾驶员的头部姿态信息和/或各所述嘴部图像中所述驾驶员的嘴巴张合状态；
    所述基于各目标人脸图像中所述驾驶员的人脸状态，确定所述驾驶员是否疲劳驾驶，包括：
    基于各所述眼部图像中所述驾驶员的睁闭眼状态、和/或各所述头部图像中所述驾驶员的头部姿态信息和/或各所述嘴部图像中所述驾驶员的嘴巴张合状态，确定所述驾驶员是否疲劳驾驶。
  3. 根据权利要求2所述的方法，其特征在于，所述基于各所述眼部图像中所述驾驶员的睁闭眼状态、和/或各所述头部图像中所述驾驶员的头部姿态信息和/或各所述嘴部图像中所述驾驶员的嘴巴张合状态，确定所述驾驶员是否疲劳驾驶，包括：
    基于各所述眼部图像中所述驾驶员的睁闭眼状态，预估所述驾驶员是否疲劳驾驶，得到第一预估结果；和/或，
    基于各所述头部图像中所述驾驶员的头部姿态信息，预估所述驾驶员是否疲劳驾驶，得到第二预估结果；和/或，
    基于各所述嘴部图像中所述驾驶员的嘴巴张合状态，预估所述驾驶员是否疲劳驾驶，得到第三预估结果；
    基于所述第一预估结果、和/或所述第二预估结果和/或所述第三预估结果，确定所述驾驶员是否疲劳驾驶。
  4. 根据权利要求3所述的方法，其特征在于，所述基于各所述眼部图像中所述驾驶员的睁闭眼状态，预估所述驾驶员是否疲劳驾驶，包括：
    基于各所述眼部图像中所述驾驶员的睁闭眼状态，确定所述驾驶员在所述指定时间段内的闭眼信息；其中，所述闭眼信息指示所述驾驶员的闭眼次数、闭眼时长、闭眼频率中的至少一种；
    确定所述闭眼信息是否满足预设的闭眼疲劳检测条件；若满足，则确定所述驾驶员疲劳驾驶为第一预估结果，否则，确定所述驾驶员未疲劳驾驶为第一预估结果。
  5. 根据权利要求3所述的方法，其特征在于，所述基于各所述头部图像中所述驾驶员的头部姿态信息，预估所述驾驶员是否疲劳驾驶，包括：
    基于各所述头部图像中所述驾驶员的头部姿态信息，确定所述驾驶员在所述指定时间段内的头部晃动信息；其中，所述头部晃动信息指示所述驾驶员的点头次数、点头时长、点头频率中的至少一种；
    确定所述头部晃动信息是否满足预设的头部疲劳检测条件；若满足，则确定所述驾 驶员疲劳驾驶为第二预估结果，否则，确定所述驾驶员未疲劳驾驶为第二预估结果。
  6. 根据权利要求3所述的方法，其特征在于，所述基于各所述嘴部图像中所述驾驶员的嘴巴张合状态，预估所述驾驶员是否疲劳驾驶，包括：
    基于各所述嘴部图像中所述驾驶员的嘴巴张合状态，确定所述驾驶员在所述指定时间段内的张嘴信息；其中，所述张嘴信息指示所述驾驶员的张嘴次数、张嘴时长、张嘴频率中的至少一种；
    确定所述张嘴信息是否满足预设的嘴部疲劳检测条件；若满足，则确定所述驾驶员疲劳驾驶为第三预估结果，否则，确定所述驾驶员未疲劳驾驶为第三预估结果。
  7. 根据权利要求3-6中任一项所述的方法，其特征在于，所述基于所述第一预估结果、和/或所述第二预估结果和/或所述第三预估结果，确定所述驾驶员是否疲劳驾驶，包括：
    若所述第一预估结果、和/或所述第二预估结果和/或所述第三预估结果中，指示所述驾驶员疲劳驾驶的占比大于指示所述驾驶员未疲劳驾驶的占比，则确定所述驾驶员疲劳驾驶，否则，确定所述驾驶员为未疲劳驾驶。
  8. 根据权利要求2所述的方法，还包括：
    确定所述指定时间段内所述驾驶员的嘴巴和所述驾驶员的眼睛是否被遮挡；
    所述利用预先训练的人脸状态模型，获取各所述眼部图像中所述驾驶员的睁闭眼状态、和/或各所述头部图像中所述驾驶员的头部姿态信息和/或各所述嘴部图像中所述驾驶员的嘴巴张合状态，包括：
    若在所述指定时间段内所述驾驶员的嘴巴被遮挡且所述驾驶员的眼睛未被遮挡，则获取各所述头部图像中所述驾驶员的头部姿态信息和各所述眼部图像中所述驾驶员的睁闭眼状态；
    若在所述指定时间段内所述驾驶员的眼睛被遮挡且所述驾驶员的嘴巴未被遮挡，则获取各所述头部图像中所述驾驶员的头部姿态信息和各所述嘴部图像中所述驾驶员的嘴巴张合状态；
    若在所述指定时间段内所述驾驶员的眼睛和所述驾驶员的嘴巴均被遮挡，则获取各所述头部图像中所述驾驶员的头部姿态信息。
  9. 根据权利要求2所述的方法，其特征在于，获取所述驾驶员的各所述眼部图像，包括：
    对所述图像数据进行头部检测，确定所述图像数据中包含的每一头部对应的头部区域，并对所述图像数据进行人脸检测，确定所述图像数据中包含的每一人脸对应的人脸区域；
    基于各头部区域和各人脸区域的位置，对各头部区域和各人脸区域进行关联；
    从各与头部区域关联的人脸区域中，确定处于所述图像数据中指定区域内，或所占面积最大的人脸区域，作为所述驾驶员的人脸区域；
    从所述驾驶员的人脸区域中，提取眼睛部位所在区域的图像，作为所述驾驶员的眼睛的眼部图像。
  10. 一种疲劳驾驶的确定装置，其特征在于，所述装置包括：
    第一获取模块，用于获取指定时间段内，针对驾驶员所采集的多帧图像数据；
    第二获取模块，用于针对每一所述图像数据，从所述图像数据中，获取所述驾驶员的人脸图像；
    第三获取模块，用于获取每一所述人脸图像中所述驾驶员的目标人脸区域的人脸关键点，并根据所述目标人脸区域的人脸关键点获取目标人脸图像；
    第四获取模块，用于将所述目标人脸图像输入预先训练的人脸状态模型，获取每一所述目标人脸图像中所述驾驶员的人脸状态；其中，所述人脸状态模型是利用样本目标人脸图像以及所述样本目标人脸图像对应的人脸状态的真值训练得到的；
    第一确定模块，用于基于各目标人脸图像中所述驾驶员的人脸状态，确定所述驾驶员是否疲劳驾驶。
  11. 一种电子设备，其特征在于，包括处理器、通信接口、存储器和通信总线，其中，处理器，通信接口，存储器通过通信总线完成相互间的通信；
    存储器，用于存放计算机程序；
    处理器，用于执行存储器上所存放的程序时，实现权利要求1-9任一所述的方法步骤。
  12. 一种计算机可读存储介质，其特征在于，所述计算机可读存储介质内存储有计算机程序，所述计算机程序被处理器执行时实现权利要求1-9任一所述的方法步骤。
PCT/CN2023/097394 2022-06-17 2023-05-31 一种疲劳驾驶的确定方法、装置及电子设备 WO2023241358A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202210686090.9 2022-06-17
CN202210686090.9A CN114973215A (zh) 2022-06-17 2022-06-17 一种疲劳驾驶的确定方法、装置及电子设备

Publications (1)

Publication Number Publication Date
WO2023241358A1 true WO2023241358A1 (zh) 2023-12-21

Family

ID=82964573

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2023/097394 WO2023241358A1 (zh) 2022-06-17 2023-05-31 一种疲劳驾驶的确定方法、装置及电子设备

Country Status (2)

Country Link
CN (1) CN114973215A (zh)
WO (1) WO2023241358A1 (zh)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117935231B (zh) * 2024-03-20 2024-06-07 杭州臻稀生物科技有限公司 一种无感式疲劳驾驶监测干预方法

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114973215A (zh) * 2022-06-17 2022-08-30 京东方科技集团股份有限公司 一种疲劳驾驶的确定方法、装置及电子设备

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR20190083155A (ko) * 2018-01-03 2019-07-11 한국전자통신연구원 운전자 상태 검출 장치 및 그 방법
CN110532976A (zh) * 2019-09-03 2019-12-03 湘潭大学 基于机器学习和多特征融合的疲劳驾驶检测方法及系统
CN110826521A (zh) * 2019-11-15 2020-02-21 爱驰汽车有限公司 驾驶员疲劳状态识别方法、系统、电子设备和存储介质
CN113420656A (zh) * 2021-06-23 2021-09-21 展讯通信(天津)有限公司 一种疲劳驾驶检测方法、装置、电子设备及存储介质
CN114973215A (zh) * 2022-06-17 2022-08-30 京东方科技集团股份有限公司 一种疲劳驾驶的确定方法、装置及电子设备

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR20190083155A (ko) * 2018-01-03 2019-07-11 한국전자통신연구원 운전자 상태 검출 장치 및 그 방법
CN110532976A (zh) * 2019-09-03 2019-12-03 湘潭大学 基于机器学习和多特征融合的疲劳驾驶检测方法及系统
CN110826521A (zh) * 2019-11-15 2020-02-21 爱驰汽车有限公司 驾驶员疲劳状态识别方法、系统、电子设备和存储介质
CN113420656A (zh) * 2021-06-23 2021-09-21 展讯通信(天津)有限公司 一种疲劳驾驶检测方法、装置、电子设备及存储介质
CN114973215A (zh) * 2022-06-17 2022-08-30 京东方科技集团股份有限公司 一种疲劳驾驶的确定方法、装置及电子设备

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117935231B (zh) * 2024-03-20 2024-06-07 杭州臻稀生物科技有限公司 一种无感式疲劳驾驶监测干预方法

Also Published As

Publication number Publication date
CN114973215A (zh) 2022-08-30

Similar Documents

Publication Publication Date Title
US11783601B2 (en) Driver fatigue detection method and system based on combining a pseudo-3D convolutional neural network and an attention mechanism
US11182592B2 (en) Target object recognition method and apparatus, storage medium, and electronic device
US11398084B2 (en) Method, apparatus and application system for extracting a target feature
Craye et al. Driver distraction detection and recognition using RGB-D sensor
CN110765807B (zh) 驾驶行为分析、处理方法、装置、设备和存储介质
AU2009243442B2 (en) Detection of abnormal behaviour in video objects
CN113366487A (zh) 基于表情组别的操作确定方法、装置及电子设备
Ibrahim et al. Embedded system for eye blink detection using machine learning technique
CN105612533A (zh) 活体检测方法、活体检测系统以及计算机程序产品
Dua et al. AutoRate: How attentive is the driver?
WO2023241358A1 (zh) 一种疲劳驾驶的确定方法、装置及电子设备
EP3798900A1 (en) Child state analysis method and apparatus, vehicle, electronic device and storage medium
Celona et al. A multi-task CNN framework for driver face monitoring
CN111460950A (zh) 自然驾驶通话行为中基于头-眼证据融合的认知分心方法
CN110543848B (zh) 一种基于三维卷积神经网络的驾驶员动作识别方法及装置
CN112115803B (zh) 口罩状态提醒方法、装置及移动终端
JP2014532947A (ja) 人物検出モダリティ結果を融合して人物カウンティングを向上させる方法及びシステム
CN112949345A (zh) 疲劳监测方法及系统、行车记录仪和智能座舱
US20200218878A1 (en) Personalized eye openness estimation
CN105718896A (zh) 一种具有目标识别功能的智能机器人
CN112926364A (zh) 头部姿态的识别方法及系统、行车记录仪和智能座舱
US20170309040A1 (en) Method and device for positioning human eyes
CN111860437A (zh) 一种基于面部表情判断疲劳程度的方法及装置
CN113420667A (zh) 人脸活体检测方法、装置、设备及介质
Lollett et al. Driver’s drowsiness classifier using a single-camera robust to mask-wearing situations using an eyelid, lower-face contour, and chest movement feature vector gru-based model

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 23822928

Country of ref document: EP

Kind code of ref document: A1