WO2023108364A1 - 驾驶员状态检测方法、装置及存储介质 - Google Patents

驾驶员状态检测方法、装置及存储介质 Download PDF

Info

Publication number
WO2023108364A1
WO2023108364A1 PCT/CN2021/137537 CN2021137537W WO2023108364A1 WO 2023108364 A1 WO2023108364 A1 WO 2023108364A1 CN 2021137537 W CN2021137537 W CN 2021137537W WO 2023108364 A1 WO2023108364 A1 WO 2023108364A1
Authority
WO
WIPO (PCT)
Prior art keywords
image frame
eye
detection information
state
detection
Prior art date
Application number
PCT/CN2021/137537
Other languages
English (en)
French (fr)
Inventor
田勇
徐文康
Original Assignee
华为技术有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 华为技术有限公司 filed Critical 华为技术有限公司
Priority to CN202180022842.5A priority Critical patent/CN116615765A/zh
Priority to PCT/CN2021/137537 priority patent/WO2023108364A1/zh
Publication of WO2023108364A1 publication Critical patent/WO2023108364A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/70—Determining position or orientation of objects or cameras
    • G—PHYSICS
    • G08—SIGNALLING
    • G08B—SIGNALLING OR CALLING SYSTEMS; ORDER TELEGRAPHS; ALARM SYSTEMS
    • G08B21/00—Alarms responsive to a single specified undesired or abnormal condition and not otherwise provided for
    • G08B21/02—Alarms for ensuring the safety of persons
    • G08B21/06—Alarms for ensuring the safety of persons indicating a condition of sleep, e.g. anti-dozing alarms
    • G—PHYSICS
    • G08—SIGNALLING
    • G08B—SIGNALLING OR CALLING SYSTEMS; ORDER TELEGRAPHS; ALARM SYSTEMS
    • G08B23/00—Alarms responsive to unspecified undesired or abnormal conditions

Definitions

  • the present application relates to the technical field of computer vision, and in particular to a driver state detection method, device and storage medium.
  • the human eye detection technology includes performing image detection on multiple image frames including human eyes, thereby estimating whether the eyes are currently closed or open. With the popularization and development of video processing and video surveillance technology, human eye detection technology has become an indispensable part of the eye image analysis process.
  • the camera is located relative to the driver’s face, and the system acquires multiple image frames including human eyes through the camera, and recognizes the driver’s eye opening and closing status based on the multiple image frames, and then combines the head
  • the facial posture can realize accurate monitoring of user fatigue status.
  • the system cannot accurately judge the driver's eye opening and closing state through the images acquired by the camera.
  • the three image frames acquired by the camera are shown in FIG. 1 , wherein the image frame 12 corresponds to the normal eye-closed situation, the image frame 14 corresponds to the head-down and eye-opening situation, and the image frame 16 corresponds to the head-down dozing situation.
  • the eyes of the driver seem to be closed in these three situations, and there are obvious deficiencies in accuracy and robustness in identifying the opening and closing of the eyes directly through the image, which greatly reduces the follow-up
  • the accuracy of the user's fatigue status determined based on the eye opening and closing status.
  • driver state detection method, device, and storage medium are proposed, which ensures high-precision and robust eye opening and closing state recognition effects, and helps to obtain stable and accurate user fatigue states in the future.
  • an embodiment of the present application provides a method for detecting a driver's state, the method comprising:
  • both of the first detection information and the second detection information indicate the state of the driver's eyes and head Department posture
  • the second detection information indicates that the first eye state corresponding to the second image frame is a closed eye state
  • the first detection information of the first image frame and the second detection information of the second image frame are obtained, for different scenarios, such as bowing the head
  • the second eye state corresponding to the second image frame can be determined according to the first detection information and the second detection information.
  • the determining the second eye state corresponding to the second image frame according to the first detection information and the second detection information includes:
  • the first preset condition is that the second detection information indicates that the head posture corresponding to the second image frame jumps in the pitch angle direction, and the first detection information indicates that the first image frame The corresponding eye state is the eye open state.
  • the second detection information indicates that the head posture corresponding to the second image frame jumps in the pitch angle direction
  • the first detection information indicates that the eye state corresponding to the first image frame is an eye-open state
  • the eye-closed state in the nap scene further improves the recognition effect of the eye state.
  • the determining the second eye state corresponding to the second image frame according to the first detection information and the second detection information includes:
  • the second preset condition is that the second detection information indicates that the head posture corresponding to the second image frame jumps in the pitch angle direction, and the first detection information indicates that the first image frame The corresponding eye state is the closed eye state.
  • the second detection information indicates that the head posture corresponding to the second image frame jumps in the pitch angle direction
  • the first detection information indicates that the eye state corresponding to the first image frame is the closed eye state
  • the second eye state corresponding to the second image frame is determined to be the eye-closed state, that is, the head-down dozing scene is accurately determined by combining the head posture sequence and the eye state sequence, which further improves the recognition effect of the eye state.
  • the method further includes:
  • the second detection information indicates that the head posture corresponding to the second image frame does not change in the pitch angle direction
  • the second detection information indicates that the head posture corresponding to the second image frame does not jump in the direction of the pitch angle
  • the method further includes:
  • the method further includes:
  • the alarm information is output, so that when dangerous driving behavior is monitored, the driver can be reminded and stopped in time of possible traffic accidents.
  • an embodiment of the present application provides a device for detecting a driver's state, the device comprising:
  • the first acquisition unit is configured to acquire a first image frame and a second image frame, the first image frame and the second image frame are image frames including a driver's face, and the first image frame is obtained at an image frame collected before the second image frame;
  • a second acquiring unit configured to acquire first detection information of the first image frame and second detection information of the second image frame, both of the first detection information and the second detection information indicate the driving The eye condition and head posture of the personnel;
  • a determining unit configured to determine the first eye state according to the first detection information and the second detection information when the second detection information indicates that the first eye state corresponding to the second image frame is a closed eye state.
  • the second eye state corresponding to the two image frames.
  • the determining unit is further configured to:
  • the first preset condition is that the second detection information indicates that the head posture corresponding to the second image frame jumps in the pitch angle direction, and the first detection information indicates that the first image frame The corresponding eye state is the eye open state.
  • the determining unit is further configured to:
  • the second preset condition is that the second detection information indicates that the head posture corresponding to the second image frame jumps in the pitch angle direction, and the first detection information indicates that the first image frame The corresponding eye state is the closed eye state.
  • the determining unit is further configured to:
  • the second detection information indicates that the head posture corresponding to the second image frame does not change in the pitch angle direction
  • the device further includes: a detection module
  • the detection module is configured to determine a fatigue state detection result according to the head pose and the second eye state corresponding to the second image frame.
  • the device further includes: an alarm module;
  • the alarm module is configured to output alarm information when the fatigue state detection result meets a preset alarm condition.
  • an embodiment of the present application provides a device for detecting a driver's state, the device comprising:
  • memory for storing processor-executable instructions
  • the processor is configured to implement the first aspect or the method provided in any possible implementation manner of the first aspect when executing the instruction.
  • the embodiments of the present application provide a non-volatile computer-readable storage medium, on which computer program instructions are stored, and when the computer program instructions are executed by a processor, the above-mentioned first aspect or the first aspect is realized A method provided by any of the possible implementations in .
  • embodiments of the present application provide a computer program product, where the computer program product includes computer-readable codes, or a non-volatile computer-readable storage medium bearing the computer-readable codes, when the When the computer readable code is run in the electronic device, the processor in the electronic device executes the method provided in the first aspect or any possible implementation manner of the first aspect.
  • embodiments of the present application provide a vehicle, where the vehicle includes the second aspect or the device provided in any possible implementation manner of the second aspect.
  • FIG. 1 shows a schematic diagram of 3 frames of image frames acquired by a camera.
  • Fig. 2 shows a schematic diagram of the architecture of a DMS provided according to an exemplary embodiment of the present application.
  • Fig. 3 shows a schematic structural diagram of a device for detecting a driver's state according to an exemplary embodiment of the present application.
  • Fig. 4 shows a schematic flowchart of a method for detecting a driver's state according to an exemplary embodiment of the present application.
  • Fig. 5 shows a schematic flowchart of a method for detecting a driver's state according to another exemplary embodiment of the present application.
  • Fig. 6 shows a schematic diagram of the principle of an eye state detection method provided according to an exemplary embodiment of the present application.
  • Fig. 7 shows a schematic diagram of the principle of a head posture detection method provided according to an exemplary embodiment of the present application.
  • Fig. 8 shows a schematic diagram of a state sequence correction process provided according to an exemplary embodiment of the present application.
  • Fig. 9 shows a schematic diagram of a state sequence correction process provided according to another exemplary embodiment of the present application.
  • Fig. 10 shows a schematic diagram of a state sequence correction process provided according to another exemplary embodiment of the present application.
  • Fig. 11 shows a schematic diagram of three scenarios and a correction process involved in a method for detecting a driver's state according to an exemplary embodiment of the present application.
  • Fig. 12 shows a block diagram of a device for detecting a driver's state provided by an exemplary embodiment of the present application.
  • the fatigue driving warning system accurately identifies the driver's eye state, yawning movements, head posture and abnormal behavior through the driver monitoring system (DMS) in the cockpit, and uses this information to accurately warn the driver Judging the fatigue status and giving warning information to protect the driving safety of users.
  • Eye state recognition technology is an important module of the fatigue driving warning system.
  • the driver's fatigue level can be analyzed through continuous eye closure and blinking frequency, and whether the driver is distracted can be analyzed through continuous eye closure time, so as to accurately identify the driver Eye status, so as to achieve accurate fatigue warning.
  • High-precision and robust eye state recognition helps to obtain stable and accurate user fatigue status, especially in vehicle scenarios, where not only high precision is required, but the algorithm must also be able to adapt to various environmental lighting and different camera conditions.
  • the embodiment of the present application provides a driver state detection method, device, and storage medium.
  • the system determines that the first eye state corresponding to the second image frame is the eye-closed state, according to the first image frame (that is, in the second The first detection information of the image frame collected before the image frame) and the second detection information of the second image frame determine the second eye state corresponding to the second image frame, thereby correcting the eye state corresponding to the second image frame to obtain
  • the correct eye state solves the problem in the related art that the system cannot accurately judge the eye opening and closing state directly through the image when the driver lowers his head, and ensures high-precision and robust eye opening and closing state recognition effect, which helps Follow-up to obtain stable and accurate user fatigue status. It should be noted that, for definitions of the first image frame and the second image frame, reference may be made to relevant descriptions in the following embodiments.
  • the product involved in the embodiment of this application is DMS, which is divided into front-installation and rear-installation.
  • the front-installation DMS is generally designed and developed by the OEM, and only adapts to the current model and installation location, while the rear-installation DMS is mainly supplied by related software and hardware. Designed and developed by the manufacturer, it can be easily installed on various models and locations.
  • This product is mainly used in the cockpit environment to monitor the status of the driver. On the one hand, it is for the safety of the driver and reminds in time in a dangerous state. On the other hand, the state stipulates that DMS must be installed in certain scenarios for timely monitoring Dangerous driving behavior, timely stop the driver's possible traffic accidents.
  • Fig. 2 shows a schematic diagram of the architecture of a DMS provided according to an exemplary embodiment of the present application.
  • DMS can comprise vehicle 21, and vehicle 21 can be the vehicle that has wireless communication function, and wherein, wireless communication function can be arranged on the vehicle-mounted terminal of this vehicle 21, vehicle-mounted module, vehicle-mounted unit, chip (system) or other parts or components.
  • the vehicle 21 in the embodiment of the present application is used to monitor the state of the driver, and send out reminder information in the case of fatigue and distraction, and the reminder information instructs the driver to pay attention to safe driving.
  • Vehicle 21 may be provided with at least one sensor 22, such as vehicle radar (such as millimeter wave radar, laser radar, ultrasonic radar, etc.), rain sensor, camera, vehicle attitude sensor (such as gyroscope), inertial measurement unit (inertial measurement unit, IMU), global navigation satellite system (global navigation satellite system, GNSS), etc., other sensors may also be arranged on the vehicle 21.
  • vehicle radar such as millimeter wave radar, laser radar, ultrasonic radar, etc.
  • rain sensor such as gyroscope
  • IMU inertial measurement unit
  • GNSS global navigation satellite system
  • other sensors may also be arranged on the vehicle 21.
  • a camera may include a photosensitive element such as a lens group and an image sensor, wherein the lens group includes a plurality of lenses (convex lens or concave lens) for collecting light signals reflected by an object to be photographed and transmitting the collected light signals to the image sensor.
  • the image sensor generates an original image of the object to be photographed according to the light signal.
  • the camera is a DMS infrared camera.
  • the driver's image frame can be collected by at least one camera set on the vehicle 21 , and the driver's image frame is an image frame including the face of the driver.
  • data such as the point cloud data of the road surface and the vertical acceleration of the vehicle 21 (that is, the acceleration data of the vehicle 21 in the direction perpendicular to the road surface) can also be collected.
  • data such as point cloud data of the road surface and vertical acceleration of the vehicle 21 (ie, acceleration data of the vehicle 21 in a direction perpendicular to the road surface) can also be collected.
  • the vehicle 21 may also be provided with a driver monitoring system 23, which is used to monitor the state of the driver.
  • the vehicle 21 can also be provided with an automatic driving system 24, and the driver monitoring system 23 assists the automatic driving system 24.
  • the driver monitoring system 23 determines that the driver is fatigued or distracted, the automatic driving system 24 can take over and the automatic driving
  • the system 24 can be used to generate an automatic driving strategy for dealing with road conditions according to the data collected by the sensors, and realize the automatic driving of the vehicle 21 according to the generated strategy.
  • a man-machine interface (human machine interface, HMI) 25 can also be set on the vehicle 21, and the man-machine interface 25 can be used to broadcast the current road surface conditions and the strategies adopted by the automatic driving system 24 to the vehicle 21 through visual icons and voice broadcasting modes. To remind the relevant drivers and passengers.
  • HMI human machine interface
  • the vehicle 21 may also be provided with a processor 26, for example, the processor 26 is a high-performance computing processor.
  • the processor 26 is used to obtain the first image frame and the second image frame through the camera, the first image frame and the second image frame are image frames including the driver's face, and the first image frame is collected before the second image frame the image frame;
  • the processor 26 is used to obtain the first detection information of the first image frame and the second detection information of the second image frame through the driver monitoring system 23, the first detection information and the second detection information both indicate the driver's Eye state and head posture; when the second detection information indicates that the first eye state corresponding to the second image frame is an eye-closed state, determine the second eye state corresponding to the second image frame according to the first detection information and the second detection information. Eye condition.
  • the DMS in the embodiment of the present application may also include a server, which may be located on the vehicle 21 as a vehicle-mounted computing unit, or located in the cloud, and may be a physical device or a virtual device such as a virtual machine , containers, etc., have a wireless communication function, wherein the wireless communication function can be set on the chip (system) or other parts or components of the server.
  • the server and the vehicle 21 can communicate through a wireless connection, such as mobile communication technologies such as 2G/3G/4G/5G, Wi-Fi, Bluetooth, frequency modulation (frequency modulation, FM), digital radio, satellite communication, etc.
  • the server can be carried on the vehicle 21 and communicate with the vehicle 21 through a wireless connection.
  • the server can collect information on one or more vehicles 21. , or data collected by sensors installed on the road or other places for calculation, and the calculation result is sent back to the corresponding vehicle 21 .
  • Fig. 3 shows a schematic structural diagram of a device for detecting a driver's state according to an exemplary embodiment of the present application.
  • This driver's state detection device can be through special-purpose hardware circuit, perhaps, the combination of software and hardware realizes and becomes all or a part of DMS in Fig. 2, and this driver's state detection device comprises: image acquisition module 310, eye state detection module 320 and Head posture detection module 330 .
  • the image acquisition module 310 is used for acquiring driver image frames.
  • the eye state detection module 320 is used to detect the driver's eye state in the driver image frame, and the head posture detection module 330 is used to detect the driver's head posture in the driver image frame.
  • the driver state detection device may also include another fatigue dependence detection module 340 and a fatigue state detection module 350 .
  • the other fatigue-dependent detection module 340 is used to detect other designated states of the driver in the driver image frame, and the other designated states are biometric states related to the driver's fatigue state, for example, the other designated states are the yawn state.
  • the eye state detection module 320 is also used to input the detected eye state of the driver to the fatigue state detection module 350
  • the head posture detection module 330 is also used to input the detected driver's head posture to the fatigue state
  • the detection module 350 and other fatigue-dependent detection modules 340 are also used to input other detected states of the driver to the fatigue state detection module 350 .
  • the fatigue state detection module 350 is used to comprehensively judge the driver's fatigue state according to the input state (such as eye state, head posture and other specified states).
  • Fig. 4 shows a schematic flowchart of a method for detecting a driver's state according to an exemplary embodiment of the present application.
  • the flow process of the driver state detection method includes:
  • Step 401 acquire a first image frame and a second image frame, both of which include the driver's face, and the first image frame is an image frame collected before the second image frame.
  • the system collects a sequence of image frames through a camera, and the sequence of image frames includes at least two image frames, that is, at least one first image frame and one second image frame.
  • both the first image frame and the second image frame are image frames including the driver's face, that is, both the first image frame and the second image frame include features of the entire face of the driver.
  • the first image frame is an image frame acquired before the second image frame.
  • the first image frame is at least two image frames acquired before the second image frame.
  • At least one first image frame and one second image frame are a plurality of consecutive image frames.
  • Step 402 acquiring first detection information of the first image frame and second detection information of the second image frame, both of which indicate the driver's eye state and head posture.
  • the system performs eye state detection and head posture detection on the first image frame to obtain first detection information, and the first detection information indicates the driver's eye state and head posture corresponding to the first image frame.
  • the first detection information includes first eye detection information and first head detection information
  • the first eye detection information indicates the driver's eye state corresponding to the first image frame
  • the first head detection information Indicate the driver's head posture corresponding to the first image frame.
  • the eye state includes a closed eye state or an eye open state.
  • the system performs eye state detection and head posture detection on the second image frame to obtain second detection information, and the second detection information indicates the driver's eye state and head posture corresponding to the second image frame.
  • the second detection information includes second eye detection information and second head detection information, the second eye detection information indicates the driver's eye state corresponding to the second image frame, and the second head detection information Indicate the driver's head posture corresponding to the second image frame.
  • eye state detection and the head posture detection may be executed in parallel or sequentially, which is not limited in this embodiment of the present application.
  • eye state detection and head posture detection refer to the relevant description in the following embodiments, which will not be introduced here.
  • Step 403 When the second detection information indicates that the first eye state corresponding to the second image frame is the eye-closed state, determine the second eye state corresponding to the second image frame according to the first detection information and the second detection information.
  • the system judges whether the second detection information indicates that the first eye state corresponding to the second image frame is the closed eye state, and if the first eye state corresponding to the second image frame is the closed eye state, then according to the first detection
  • the information and the second detection information determine the second eye state corresponding to the second image frame; if the first eye state corresponding to the second image frame is the eye open state, the process ends.
  • the system determines that the second eye state corresponding to the second image frame is the eye-open state.
  • the first preset condition is that the second detection information indicates that the head posture corresponding to the second image frame jumps in the pitch angle direction, and the first detection information indicates that the eye state corresponding to the first image frame is an eye-open state.
  • the first image frame includes a plurality of first image frames
  • the meaning of "the first detection information indicates that the eye state corresponding to the first image frame is an open eye state” includes: the first detection information indicates that the eye state is an open eye state.
  • the ratio of the number of the first image frames of the eye state to the total number of the first image frames is greater than a first preset threshold.
  • the first preset threshold is a default setting or a custom setting.
  • the first preset threshold is 0.9. This embodiment of the present application does not limit it.
  • the system determines that the second eye state corresponding to the second image frame is the eye-closed state.
  • the second preset condition is that the second detection information indicates that the head posture corresponding to the second image frame jumps in the pitch angle direction, and the first detection information indicates that the eye state corresponding to the first image frame is a closed eye state.
  • the first image frame includes a plurality of first image frames
  • the meaning of "the first detection information indicates that the eye state corresponding to the first image frame is a closed eye state” includes: the first detection information indicates that the eye state is a closed eye state
  • the ratio of the number of the first image frames in the eye state to the total number of the first image frames is greater than the second preset threshold.
  • the second preset threshold is a default setting or a custom setting.
  • the second preset threshold is 0.95. This embodiment of the present application does not limit it.
  • the system determines that the second eye state corresponding to the second image frame is the eye-closed state.
  • the system determines the fatigue state detection result according to the head posture and the second eye state corresponding to the second image frame.
  • an alarm message is output.
  • the system acquires a preset fatigue detection model, invokes the fatigue detection model according to the head pose and the second eye state corresponding to the second image frame, and outputs the fatigue state detection result.
  • the fatigue detection model indicates the correlation between head posture, eye state and fatigue state
  • the fatigue detection model is a pre-trained model based on sample image frames.
  • the fatigue detection model is a model based on the fusion of eye state and head posture.
  • the fatigue detection model is a model based on the fusion of eye state, head posture and yawn state.
  • the fatigue detection model is a model based on the fusion of eye state, head posture, yawn state and other information. This embodiment of the present application does not limit it.
  • the fatigue state detection result includes one of a first detection result and a second detection result
  • the first detection result indicates that the driver is in a fatigue state
  • the second detection result indicates that the driver is in a non-fatigue state .
  • outputting alarm information includes: outputting alarm information when the fatigue state detection result is the first detection result.
  • the fatigue state detection result includes a fatigue state level
  • the fatigue state level is related to the predicted fatigue intensity of the driver.
  • the fatigue state level is positively correlated with the predicted driver's fatigue intensity, that is, the higher the fatigue state level, the greater the predicted driver's fatigue intensity.
  • outputting the alarm information includes: outputting the alarm information when the fatigue state level is greater than the preset level threshold.
  • the preset grade threshold is a default setting or a custom setting. This embodiment of the present application does not limit it.
  • the system outputs the alarm information in a preset prompt format
  • the preset prompt format includes at least one of voice, text, image, and animation.
  • the embodiment of the present application does not limit the output mode and output content of the alarm information.
  • the embodiment of the present application aims at different Scenes, such as looking down, looking down, and taking a nap, etc., when determining that the first eye state corresponding to the second image frame is a closed eye state, determine the eye state corresponding to the second image frame according to the first detection information and the second detection information.
  • the second eye state is to correct the eye state corresponding to the second image frame to obtain the correct eye state, which solves the problem in the related art that the system cannot accurately judge the eye opening and closing state directly through the image when the driver bows his head. It ensures high-precision and robust eye opening and closing state recognition effect, and helps to obtain stable and accurate user fatigue state in the future.
  • Fig. 5 shows a schematic flowchart of a method for detecting a driver's state according to another exemplary embodiment of the present application.
  • the flow process of the driver state detection method includes:
  • step 501 image acquisition is performed through a camera to obtain an image frame sequence.
  • the system collects images through a camera deployed in at least one position in the vehicle to obtain a sequence of image frames.
  • the camera may be an infrared camera.
  • the at least one position includes any one or more of the following positions: above or near the steering column, the instrument panel, above or near the center console, at or near the A-pillar, and at or near the rearview mirror.
  • the recognition of the eye state is the optimal position, and it is not easy to correctly identify the eye state under the event of lowering the head, but this position is prone to the steering wheel blocking the face and causing the eye state to fail Recognition, when the camera is placed on the A-pillar or the rearview mirror, the image acquisition will not cause the steering wheel to be blocked, but there must be a scene where the eye state cannot be correctly recognized under the head-down event.
  • the image acquisition through the camera to obtain the image frame sequence includes: acquiring the image through the camera when the vehicle is in a driving state to obtain the image frame sequence; Image acquisition, to obtain an image frame sequence; and/or, to acquire an image frame sequence through a camera after detecting that the vehicle is ignited; and/or, to acquire an image frame through a camera when a vehicle start instruction is detected sequence; and/or, when a control command to the vehicle or a component or system in the vehicle is detected, image acquisition is performed through the camera to obtain a sequence of image frames.
  • the implementation of this application does not limit the trigger conditions and acquisition methods of image acquisition.
  • Step 502 performing face detection on the acquired image frame sequence.
  • Face detection is the foundation of other face applications. Eye state detection and head posture algorithms based on faces are affected by the front-end face detection algorithm. During the inspection, the misidentification of the eye state and the relatively large jitter of the head posture will cause the system to misjudge the normal state as a head down event. It is possible to obtain more stable and accurate eye condition results.
  • the system uses a preset face detection algorithm to perform face detection on the collected image frame sequence.
  • the preset face detection algorithm is a dedicated face detection algorithm, such as a multi-task face detection algorithm, which is used to detect faces and output face key point information , so that the follow-up system can correct the face through the face key point information, provide better input for the back-end eye state detection algorithm, and improve the accuracy of the single-frame eye state recognition algorithm.
  • the dedicated face detection algorithm has a very high accuracy rate, but the end-to-end deployment of this algorithm is very time-consuming and long, and is not suitable for end-to-end deployment.
  • the preset face detection algorithm is a lightweight face detection algorithm, which is suitable for device-side deployment.
  • the preset face detection algorithm is a general-purpose target detection algorithm, such as a single-stage target detection algorithm. Algorithms for face detection. It should be noted that the embodiment of the present application does not limit the preset face detection algorithm.
  • Step 503 judging whether a human face is detected in the sequence of image frames.
  • step 501 If the system does not detect a human face in the sequence of image frames, proceed to step 501; if the system detects a human face in the sequence of image frames, proceed to step 504.
  • Step 504 if a human face is detected in the image frame sequence, perform eye state detection to obtain an eye state sequence, and perform head posture detection to obtain a head posture sequence.
  • the system performs eye state detection and head posture detection to obtain eye state detection data and head posture detection data corresponding to the image frame, thereby obtaining the image frame sequence
  • Eye state sequence and head pose sequence that is, the eye state sequence includes eye state detection data corresponding to each of the multiple image frames in the image frame sequence
  • the head pose sequence includes multiple images in the image frame sequence Head pose detection data corresponding to each frame.
  • the system can use the target detection algorithm for eye state detection.
  • the eye state recognition based on the target detection algorithm has strong robustness, and the eye state The recognition is less likely to be disturbed, and it is easy to support scenes with masks, hands covering the face and makeup. Only when the human eye is blocked or disturbed, the recognition of the eye state will be affected. However, based on the target detection algorithm, it is impossible to distinguish between the squinting scene and the squinting scene due to fatigue, resulting in mutual misidentification between the two states.
  • the eye state recognition algorithm is suitable for scenarios where the amount of face data is small and the face samples of the driving environment cannot be fully covered.
  • the system can use the general target detection algorithm after pruning to detect the eye state, and only use part of the network branches to predict the eye state, which can While ensuring the accuracy rate, the detection speed has been greatly improved, and the end-side advantages are greater.
  • the system may use a preset classification algorithm to detect the state of the eye.
  • the eye state detection based on the preset classification algorithm can be divided into two cases.
  • One case is to judge the eye state based on the image frame including the face. Since the proportion of the human eye is very small, it is easy to be interfered by other parts of the face. Misidentification, such as wearing a mask, makeup, etc., may lead to misidentification. At the same time, this method cannot distinguish the invalid state caused by whether the eyes are blocked.
  • Another situation is to add the human eye detection model, and then input the image frame including the human eye to the human eye detection model to output the eye state, where the human eye detection model is pre-trained to identify the eye state in the image frame
  • This method is free from interference from other parts of the human eye, and has higher robustness and higher accuracy than the previous case.
  • This method also has the inability to distinguish between squinting scenes and squinting due to fatigue.
  • the method adds a human eye detection model, which increases resource consumption and reduces device-side performance.
  • the system may use a key point algorithm for eye state detection.
  • the system detects the positions of six key points "p 43 , p 44 , p 45 , p 46 , p 47 , p 48 " on the upper and lower eyelids and corners of the eyes, and then calculates by the following formula Normalized distance EAR:
  • the normalized distance is used as the eye opening index.
  • the preset opening threshold is the default It is set or customized, for example, the preset opening threshold is 20% of the normal eye opening value, which is not limited in this embodiment of the present application.
  • the eye opening and closing state based on the eye opening index is very suitable as the basis for subsequent judgment of the driver's fatigue state, but this method has high requirements for the stability of the key point algorithm, and requires that the position of the key point will not experience large jitter, that is, the corresponding The eye opening is required to be stable.
  • a filtering method can be added, that is, the system filters the output eye opening index to obtain a more stable eye opening index.
  • the system maps the two-dimensional image frame to the three-dimensional image frame to obtain the head pose detection data, which indicates the orientation of the driver's face pose.
  • the head posture detection data is represented by three degrees of freedom (dof), that is, a pitch angle (pitch), a roll angle (roll) and a yaw angle (yaw).
  • the head posture detection data includes the pitch angle between the driver's head and the horizontal axis of the device coordinate system, the yaw angle between the head and the vertical axis of the device coordinate system, and the distance between the driver's head and the device coordinate system.
  • the roll angle between the vertical axes of the coordinate system.
  • the accuracy and stability of head pose detection is crucial.
  • the head pose detection algorithm needs to support large-angle head poses.
  • the system performs head pose detection based on key points of the face.
  • This method can be trained based on a large number of existing key point data; the key point information of each part of the face can be obtained; because the disturbance of the point has less influence on the angle, the head posture is more stable.
  • the system performs head pose detection based on a regression algorithm.
  • This method can directly return Euler angles, which is simple and convenient; the model design is simple, and it does not need to be solved by post-processing or secondary algorithm optimization.
  • the head pose tags collected in the real camera coordinate system can cover 360° head pose data, but the equipment is expensive, the deployment is complicated, the collection cycle is long, and the manpower input is large.
  • the data generation method includes: fitting the key point data in the two-dimensional image frame with a three-dimensional deformable model (3D Morphable Model, 3DMM), and then enhancing the head pose to obtain the head pose label under a large angle.
  • 3D Morphable Model, 3DMM three-dimensional deformable model
  • the method can support head pose data at ⁇ 90°. This method only relies on 2D key point annotation, and the feasibility and cost are relatively reasonable.
  • the system performs head pose detection based on face key points and a regression algorithm.
  • the system inputs the image frame into the head pose detection model, and outputs head pose data.
  • the head pose detection model is a multi-task neural network model based on face key points and regression methods, that is, tasks such as Euler angle regression and face key points are done in one network model, and by sharing the skeleton network, multiple A head network does multiple different tasks, and this method can reduce the prediction error of Euler angles to a certain extent.
  • the embodiment of the present application does not limit the methods of the eye state detection algorithm and the head posture detection algorithm.
  • Step 505 judging whether the first eye state corresponding to the second image frame is a closed eye state.
  • the system After performing eye state detection and head pose detection on each image frame in the image frame sequence, the system obtains the eye state queue and head pose queue corresponding to the image frame sequence, and the system judges the latest frame in the image frame sequence That is, whether the first eye state in the second image frame is the eye-closed state.
  • the eye state queue and head pose queue corresponding to the frame sequence are divided into the following three scenarios: normal eye-closed scene, head-down-open-eye-looking-down scene, and head-down-nap scene.
  • Step 506 if the first eye state corresponding to the second image frame is the eye-closed state, then determine whether the head posture corresponding to the second image frame jumps in the pitch angle direction.
  • the system judges whether the head posture corresponding to the second image frame jumps in the direction of the pitch angle. If there is no jump in the angular direction, execute step 507 ; if the head posture corresponding to the second image frame jumps in the pitch angle direction, execute step 508 .
  • Step 507 if the head posture corresponding to the second image frame does not jump in the pitch angle direction, output the second eye state corresponding to the second image frame as the eye-closed state.
  • the head posture corresponding to the second image frame does not jump in the pitch angle direction, it is determined as a normal eye-closed scene, and the second eye state corresponding to the second image frame is output as the eye-closed state.
  • Step 508 if the head posture corresponding to the second image frame jumps in the pitch angle direction, then determine whether the eye state corresponding to the first image frame is the closed eye state.
  • step 509 is executed, and if the eye state corresponding to the first image frame is the closed eye state, then step 510 is executed.
  • Step 509 if the eye state corresponding to the first image frame is the eye open state, then output the second eye state corresponding to the second image frame as the eye open state.
  • the eye state corresponding to the first image frame is the eye open state, it is determined to look down at the scene with the head down and the eyes open, and output the second eye state corresponding to the second image frame as the eye open state.
  • Step 510 if the eye state corresponding to the first image frame is the closed eye state, then output the second eye state corresponding to the second image frame as the closed eye state.
  • the eye state corresponding to the first image frame is the eye-closed state, it is determined to be a head-down dozing scene, and the second eye state corresponding to the second image frame is output as the eye-closed state.
  • the system can distinguish between normal eye-closing, looking down with eyes open, and dozing with head down according to the eye state sequence and the head posture sequence. Scenes. When only the eye state sequence is used as the input parameter of the fatigue state, false alarms will occur in these three states, which will affect the effectiveness of the alarm system. By adding the head posture detection module, we can easily distinguish between normal eye-closed and eyes-down eyes looking down.
  • the eyes-closed state is detected at the same time, check whether the head posture jumps: if not If there is a jump, it will be judged as a normal closed-eyes scene, and the original eye state will be maintained if the result is not corrected; if a jump occurs, it will be judged to be looking down at the scene with eyes open; and the original closed-eyes state will be corrected to open eyes state.
  • the event of lowering the head and dozing will be misjudged as the situation of lowering the head and opening the eyes, resulting in an error in the correction of the eye state.
  • the eye stable state queue is added.
  • the eye state corresponding to the first image frame By checking the eye state of the historical continuous frames, that is, the eye state corresponding to the first image frame, to distinguish looking down with head down and open eyes In the two scenarios of head down and snoozing, if the eye state corresponding to the first image frame is the closed eye state, it will be judged as the head down snooze scene, and the original closed eye state will remain if the result is not corrected. If the eye state corresponding to the first image frame is In the eyes-open state, it is judged to be looking down at the scene with the head down and eyes open, and the original closed-eyes state is corrected as the eyes-open state.
  • the three scenarios are distinguished and the correction process is shown in FIG. 11 .
  • the first row represents the head pose sequence of the head pose in the pitch angle direction
  • the second row represents the eye state sequence corresponding to the head pose without any post-processing.
  • the third row represents the corrected eye state sequence after adding the head posture detection module.
  • the eye state of the scene of bowing the head and snoozing is mistakenly corrected to the eye-open state
  • the fourth row represents the addition of the eye state Steady-state queue, further correcting the eye state to obtain an accurate eye state.
  • the driver state detection method provided by the embodiment of the present application can accurately distinguish three kinds of normal eye-closed, glance-down and doze scenes which are almost indistinguishable from the image analysis, and at the same time use the steady-state queue to accurately
  • the eye-closed false detection caused by glancing down is restored to the open-eye state, and the closed-eye state in the dozing scene will not be incorrectly modified.
  • the fault tolerance is higher. Significantly improves the accuracy of eye state recognition in driving scenes.
  • FIG. 12 shows a block diagram of a driver state detection device provided by an exemplary embodiment of the present application.
  • the device can be implemented as all or part of the DMS provided in FIG. 2 or the driver state detection device provided in FIG. 3 through software, hardware or a combination of the two.
  • the apparatus may include: a first acquiring unit 1210, a second acquiring unit 1220, and a determining unit 1230;
  • the first acquisition unit 1210 is configured to acquire a first image frame and a second image frame, the first image frame and the second image frame are image frames including the face of the driver, and the first image frame is before the second image frame The captured image frame;
  • the second acquisition unit 1220 is configured to acquire the first detection information of the first image frame and the second detection information of the second image frame, both the first detection information and the second detection information indicate the driver's eye state and head posture ;
  • the determination unit 1230 is configured to determine the second eye corresponding to the second image frame according to the first detection information and the second detection information when the second detection information indicates that the first eye state corresponding to the second image frame is a closed eye state. state.
  • the determining unit 1230 is further configured to:
  • the first preset condition is that the second detection information indicates that the head posture corresponding to the second image frame jumps in the pitch angle direction, and the first detection information indicates that the eye state corresponding to the first image frame is an eye-open state.
  • the determining unit 1230 is further configured to:
  • the second preset condition is that the second detection information indicates that the head posture corresponding to the second image frame jumps in the pitch angle direction, and the first detection information indicates that the eye state corresponding to the first image frame is a closed eye state.
  • the determining unit 1230 is further configured to:
  • the second detection information indicates that the head posture corresponding to the second image frame does not change in the pitch angle direction
  • the device further includes: a detection unit;
  • the detection unit is configured to determine the fatigue state detection result according to the head posture and the second eye state corresponding to the second image frame.
  • the device further includes: an alarm unit;
  • the alarm unit is configured to output alarm information when the fatigue state detection result meets a preset alarm condition.
  • the division of the above-mentioned functional modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional modules according to the needs.
  • the internal structure of the device is divided into different functional modules to complete all or part of the functions described above.
  • the device and the method embodiment provided by the above embodiment belong to the same idea, and the specific implementation process thereof is detailed in the method embodiment, and will not be repeated here.
  • An embodiment of the present application provides a driver state detection device, the driver state detection device includes: a processor; a memory for storing processor-executable instructions; wherein, the processor is configured to implement the above-mentioned embodiments when executing the instructions in the method executed by DMS.
  • An embodiment of the present application provides a computer program product, including computer-readable codes, or a non-volatile computer-readable storage medium carrying computer-readable codes.
  • the processor executes the method executed by the DMS in the foregoing embodiments.
  • An embodiment of the present application provides a non-volatile computer-readable storage medium, on which computer program instructions are stored, and when the computer program instructions are executed by a processor, the methods performed by the DMS in the foregoing embodiments are implemented.
  • a computer readable storage medium may be a tangible device that can retain and store instructions for use by an instruction execution device.
  • a computer readable storage medium may be, for example, but is not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing.
  • Non-exhaustive list of computer-readable storage media include: portable computer disk, hard disk, random access memory (Random Access Memory, RAM), read only memory (Read Only Memory, ROM), erasable Electrically Programmable Read-Only-Memory (EPROM or flash memory), Static Random-Access Memory (Static Random-Access Memory, SRAM), Portable Compression Disk Read-Only Memory (Compact Disc Read-Only Memory, CD -ROM), Digital Video Disc (DVD), memory sticks, floppy disks, mechanically encoded devices such as punched cards or raised structures in grooves with instructions stored thereon, and any suitable combination of the foregoing .
  • RAM Random Access Memory
  • ROM read only memory
  • EPROM or flash memory erasable Electrically Programmable Read-Only-Memory
  • Static Random-Access Memory SRAM
  • Portable Compression Disk Read-Only Memory Compact Disc Read-Only Memory
  • CD -ROM Compact Disc Read-Only Memory
  • DVD Digital Video Disc
  • Computer readable program instructions or codes described herein may be downloaded from a computer readable storage medium to a respective computing/processing device, or downloaded to an external computer or external storage device over a network, such as the Internet, local area network, wide area network, and/or wireless network.
  • the network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and/or edge servers.
  • a network adapter card or a network interface in each computing/processing device receives computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in a computer-readable storage medium in each computing/processing device .
  • Computer program instructions for performing the operations of the present application may be assembly instructions, instruction set architecture (Instruction Set Architecture, ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or in one or more source or object code written in any combination of programming languages, including object-oriented programming languages—such as Smalltalk, C++, etc., and conventional procedural programming languages—such as the “C” language or similar programming languages.
  • Computer readable program instructions may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer, or entirely on the remote computer or server implement.
  • the remote computer can be connected to the user's computer through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or it can be connected to an external computer such as use an Internet service provider to connect via the Internet).
  • electronic circuits such as programmable logic circuits, field-programmable gate arrays (Field-Programmable Gate Array, FPGA) or programmable logic arrays (Programmable Logic Array, PLA), the electronic circuit can execute computer-readable program instructions, thereby realizing various aspects of the present application.
  • These computer-readable program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine such that when executed by the processor of the computer or other programmable data processing apparatus , producing an apparatus for realizing the functions/actions specified in one or more blocks in the flowchart and/or block diagram.
  • These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause computers, programmable data processing devices and/or other devices to work in a specific way, so that the computer-readable medium storing instructions includes An article of manufacture comprising instructions for implementing various aspects of the functions/acts specified in one or more blocks in flowcharts and/or block diagrams.
  • each block in a flowchart or block diagram may represent a module, a portion of a program segment, or an instruction that includes one or more Executable instructions.
  • the functions noted in the block may occur out of the order noted in the figures. For example, two blocks in succession may, in fact, be executed substantially concurrently, or they may sometimes be executed in the reverse order, depending upon the functionality involved.
  • each block in the block diagrams and/or flowcharts, and combinations of blocks in the block diagrams and/or flowcharts can be implemented with hardware (such as circuits or ASIC (Application Specific Integrated Circuit, application-specific integrated circuit)), or it can be realized by a combination of hardware and software, such as firmware.
  • hardware such as circuits or ASIC (Application Specific Integrated Circuit, application-specific integrated circuit)
  • firmware such as firmware

Abstract

本申请实施例涉及计算机视觉技术领域，尤其涉及一种驾驶员状态检测方法、装置及存储介质。该方法包括：获取第一图像帧和第二图像帧，第一图像帧和第二图像帧均为包括驾驶员人脸的图像帧，第一图像帧是在第二图像帧之前采集的图像帧；获取第一图像帧的第一检测信息和第二图像帧的第二检测信息，第一检测信息和第二检测信息均指示驾驶员的眼部状态和头部姿态；在第二检测信息指示第二图像帧对应的第一眼部状态为闭眼状态时，根据第一检测信息和第二检测信息确定第二图像帧对应的第二眼部状态。本申请实施例可以准确的区分三种从图像上分析几乎没有区别的正常闭眼、往下一瞥和打盹场景，保证了高精度和强鲁棒性的眼睛开闭状态识别效果。

Description

驾驶员状态检测方法、装置及存储介质 技术领域
本申请涉及计算机视觉技术领域，尤其涉及一种驾驶员状态检测方法、装置及存储介质。
背景技术
人眼检测技术包括对包括人眼的多帧图像帧进行图像检测，从而估计目前眼睛处于闭眼状态还是睁眼状态的技术。随着视频处理与视频监控技术的普及发展，人眼检测技术成为眼睛图像分析过程中不可缺少的重要部分。
相关技术中，以车载场景为例，摄像头位于相对于驾驶员面部的位置，系统通过摄像头获取包括人眼的多帧图像帧，根据多帧图像帧识别驾驶员的眼睛开闭状态，再结合头部姿态可实现精准的用户疲劳状态的监控。
但是在上述方法中，在驾驶员头部往下看的情况下，系统通过摄像头获取的图像无法准确判断出驾驶员的眼睛开闭状态。在一个示意性的例子，通过摄像头获取的3帧图像帧如图1所示，其中图像帧12对应正常闭眼情况，图像帧14对应低头睁眼情况，图像帧16对应低头打盹情况。而从摄像头的角度来看，驾驶员的眼睛在这三种情况下似乎都是闭着的，直接通过图像来识别眼睛开闭状态在精度和鲁棒性上存在明显的不足，大大降低了后续基于眼睛开闭状态所确定的用户疲劳状态的准确度。
发明内容
有鉴于此，提出了一种驾驶员状态检测方法、装置及存储介质，保证了高精度和强鲁棒性的眼睛开闭状态识别效果，有助于后续获取稳定、精准的用户疲劳状态。
第一方面，本申请的实施例提供了一种驾驶员状态检测方法，所述方法包括：
获取第一图像帧和第二图像帧，所述第一图像帧和所述第二图像帧均为包括驾驶员人脸的图像帧，所述第一图像帧是在所述第二图像帧之前采集的图像帧；
获取所述第一图像帧的第一检测信息和所述第二图像帧的第二检测信息，所述第一检测信息和所述第二检测信息均指示所述驾驶员的眼部状态和头部姿态；
在所述第二检测信息指示所述第二图像帧对应的第一眼部状态为闭眼状态时，根据所述第一检测信息和所述第二检测信息确定所述第二图像帧对应的第二眼部状态。
在该实现方式中，获取第一图像帧的第一检测信息和第二图像帧的第二检测信息(比如驾驶员的连续眼部状态和对应的连续头部姿态)，针对不同场景，比如低头往下看、低头打盹等场景，可以在确定第二图像帧对应的第一眼部状态为闭眼状态时，根据第一检测信息和第二检测信息确定第二图像帧对应的第二眼部状态，即对第二图像帧对应的眼部状态进行修正得到正确的眼部状态，解决了相关技术中系统在驾驶员低头时直接通过图像无法准确判断眼睛开闭状态的问题，保证了高精度和强鲁棒性的 眼睛开闭状态识别效果，有助于后续获取稳定、精准的用户疲劳状态。
在一种可能的实现方式中，所述根据所述第一检测信息和所述第二检测信息确定所述第二图像帧对应的第二眼部状态，包括：
当所述第一检测信息和所述第二检测信息满足第一预设条件时，确定所述第二图像帧对应的第二眼部状态为睁眼状态；
其中，所述第一预设条件为所述第二检测信息指示所述第二图像帧对应的头部姿态在俯仰角方向发生跳变，且所述第一检测信息指示所述第一图像帧对应的眼部状态为睁眼状态。
在该实现方式中，当第二检测信息指示第二图像帧对应的头部姿态在俯仰角方向发生跳变，且第一检测信息指示第一图像帧对应的眼部状态为睁眼状态时，确定第二图像帧对应的第二眼部状态为睁眼状态，即结合头部姿态序列和眼部状态序列准确地将往下一瞥导致的闭眼误检测恢复为睁眼状态，不会错误修改打盹场景下的闭眼状态，进一步提高了眼睛状态的识别效果。
在另一种可能的实现方式中，所述根据所述第一检测信息和所述第二检测信息确定所述第二图像帧对应的第二眼部状态，包括：
当所述第一检测信息和所述第二检测信息满足第二预设条件时，确定所述第二图像帧对应的第二眼部状态为闭眼状态；
其中，所述第二预设条件为所述第二检测信息指示所述第二图像帧对应的头部姿态在俯仰角方向发生跳变，且所述第一检测信息指示所述第一图像帧对应的眼部状态为闭眼状态。
在该实现方式中，当第二检测信息指示第二图像帧对应的头部姿态在俯仰角方向发生跳变，且第一检测信息指示第一图像帧对应的眼部状态为闭眼状态时，确定第二图像帧对应的第二眼部状态为闭眼状态，即结合头部姿态序列和眼部状态序列准确地确定出低头打盹场景，进一步提高了眼睛状态的识别效果。
在另一种可能的实现方式中，所述方法还包括：
当所述第二检测信息指示所述第二图像帧对应的头部姿态在俯仰角方向未发生跳变时，确定所述第二图像帧对应的第二眼部状态为闭眼状态。
在该实现方式中，当第二检测信息指示第二图像帧对应的头部姿态在俯仰角方向未发生跳变时，确定第二图像帧对应的第二眼部状态为闭眼状态，即通过头部姿态序列准确地确定出正常闭眼场景，进一步提高了眼睛状态的识别效果。
在另一种可能的实现方式中，所述方法还包括：
根据所述第二图像帧对应的所述头部姿态和所述第二眼部状态，确定疲劳状态检测结果。
在另一种可能的实现方式中，所述方法还包括：
当所述疲劳状态检测结果满足预设报警条件时，输出报警信息。
在该实现方式中，当疲劳状态检测结果满足预设报警条件时，输出报警信息，以便在监控到危险驾驶行为时，及时提醒并制止驾驶员可能产生的交通事故。
第二方面，本申请的实施例提供了一种驾驶员状态检测装置，所述装置包括：
第一获取单元，用于获取第一图像帧和第二图像帧，所述第一图像帧和所述第二 图像帧均为包括驾驶员人脸的图像帧，所述第一图像帧是在所述第二图像帧之前采集的图像帧；
第二获取单元，用于获取所述第一图像帧的第一检测信息和所述第二图像帧的第二检测信息，所述第一检测信息和所述第二检测信息均指示所述驾驶员的眼部状态和头部姿态；
确定单元，用于在所述第二检测信息指示所述第二图像帧对应的第一眼部状态为闭眼状态时，根据所述第一检测信息和所述第二检测信息确定所述第二图像帧对应的第二眼部状态。
在另一种可能的实现方式中，所述确定单元，还用于：
当所述第一检测信息和所述第二检测信息满足第一预设条件时，确定所述第二图像帧对应的第二眼部状态为睁眼状态；
其中，所述第一预设条件为所述第二检测信息指示所述第二图像帧对应的头部姿态在俯仰角方向发生跳变，且所述第一检测信息指示所述第一图像帧对应的眼部状态为睁眼状态。
在另一种可能的实现方式中，所述确定单元，还用于：
当所述第一检测信息和所述第二检测信息满足第二预设条件时，确定所述第二图像帧对应的第二眼部状态为闭眼状态；
其中，所述第二预设条件为所述第二检测信息指示所述第二图像帧对应的头部姿态在俯仰角方向发生跳变，且所述第一检测信息指示所述第一图像帧对应的眼部状态为闭眼状态。
在另一种可能的实现方式中，所述确定单元，还用于：
当所述第二检测信息指示所述第二图像帧对应的头部姿态在俯仰角方向未发生跳变时，确定所述第二图像帧对应的第二眼部状态为闭眼状态。
在另一种可能的实现方式中，所述装置还包括：检测模块；
所述检测模块，用于根据所述第二图像帧对应的所述头部姿态和所述第二眼部状态，确定疲劳状态检测结果。
在另一种可能的实现方式中，所述装置还包括：报警模块；
所述报警模块，用于当所述疲劳状态检测结果满足预设报警条件时，输出报警信息。
第三方面，本申请的实施例提供了一种驾驶员状态检测装置，所述装置包括：
处理器；
用于存储处理器可执行指令的存储器；
其中，所述处理器被配置为执行所述指令时实现上述第一方面或第一方面中的任意一种可能的实现方式所提供的方法。
第四方面，本申请的实施例提供了一种非易失性计算机可读存储介质，其上存储有计算机程序指令，所述计算机程序指令被处理器执行时实现上述第一方面或第一方面中的任意一种可能的实现方式所提供的方法。
第五方面，本申请的实施例提供了一种计算机程序产品，所述计算机程序产品包括计算机可读代码，或者承载有所述计算机可读代码的非易失性计算机可读存储介质， 当所述计算机可读代码在电子设备中运行时，所述电子设备中的处理器执行上述第一方面或第一方面中的任意一种可能的实现方式所提供的方法。
第六方面，本申请的实施例提供了一种车辆，所述车辆包含上述第二方面或第二方面中的任意一种可能的实现方式所提供的装置。
附图说明
包含在说明书中并且构成说明书的一部分的附图与说明书一起示出了本申请的示例性实施例、特征和方面，并且用于解释本申请的原理。
图1示出通过摄像头获取的3帧图像帧的示意图。
图2示出根据本申请一个示例性实施例提供的DMS的架构示意图。
图3示出根据本申请一个示例性实施例提供的驾驶员状态检测装置的架构示意图。
图4示出根据本申请一个示例性实施例提供的驾驶员状态检测方法的流程示意图。
图5示出根据本申请另一个示例性实施例提供的驾驶员状态检测方法的流程示意图。
图6示出根据本申请一个示例性实施例提供的眼部状态检测方式的原理示意图。
图7示出根据本申请一个示例性实施例提供的头部姿态检测方式的原理示意图。
图8示出根据本申请一个示例性实施例提供的状态序列修正过程的示意图。
图9示出根据本申请另一个示例性实施例提供的状态序列修正过程的示意图。
图10示出根据本申请另一个示例性实施例提供的状态序列修正过程的示意图。
图11示出根据本申请一个示例性实施例提供的驾驶员状态检测方法涉及的三种场景和修正过程的原理示意图。
图12示出了本申请一个示例性实施例提供的驾驶员状态检测装置的框图。
具体实施方式
以下将参考附图详细说明本申请的各种示例性实施例、特征和方面。附图中相同的附图标记表示功能相同或相似的元件。尽管在附图中示出了实施例的各种方面，但是除非特别指出，不必按比例绘制附图。
在这里专用的词“示例性”意为“用作例子、实施例或说明性”。这里作为“示例性”所说明的任何实施例不必解释为优于或好于其它实施例。
另外，为了更好的说明本申请，在下文的具体实施方式中给出了众多的具体细节。本领域技术人员应当理解，没有某些具体细节，本申请同样可以实施。在一些实例中，对于本领域技术人员熟知的方法、手段、元件和电路未作详细描述，以便于凸显本申请的主旨。
疲劳驾驶预警系统通过座舱内的驾驶员监控系统(driver monitoring system，DMS)精确地识别驾驶员的眼部状态、哈欠动作、头部姿态和异常行为等，并利用这些信息来准确地对驾驶员疲劳状态进行判断，并给出告警信息，为用户的驾驶安全保驾护航。眼部状态识别技术是疲劳驾驶预警系统的重要模块，通过连续闭眼和眨眼频率可分析出驾驶员的疲劳程度，通过连续闭眼时长可分析出驾驶员是否视线分神，精准地识别 驾驶员的眼部状态，从而实现精准的疲劳预警。高精度、强鲁棒性的眼部状态识别有助于获取稳定、精准的用户疲劳状态，尤其是在车载场景下，不仅要求精度高，算法还要能够适配各种环境光照和摄像头不同的安装位置场景，其中摄像头为DMS中用于监控驾驶员的摄像头。当摄像头安装在汽车的A柱或者后视镜位置上，在驾驶员头部往下看时，从摄像头上获取的图像无法准确判断眼睛开闭状态，相关技术中仅通过图像来分析眼睛开闭状态在精度和鲁棒性上存在明显的不足，大大降低了后续基于眼睛开闭状态所确定的用户疲劳状态的准确度。
而本申请实施例提供了一种驾驶员状态检测方法、装置及存储介质，系统在确定第二图像帧对应的第一眼部状态为闭眼状态时，根据第一图像帧(即在第二图像帧之前采集的图像帧)的第一检测信息和第二图像帧的第二检测信息确定第二图像帧对应的第二眼部状态，从而对第二图像帧对应的眼部状态进行修正得到正确的眼部状态，解决了相关技术中系统在驾驶员低头时直接通过图像无法准确判断眼睛开闭状态的问题，保证了高精度和强鲁棒性的眼睛开闭状态识别效果，有助于后续获取稳定、精准的用户疲劳状态。需要说明的是，第一图像帧和第二图像帧的定义可参考下面实施例中的相关描述。
首先，对本申请涉及的应用场景进行介绍。
本申请实施例涉及到的产品是DMS，分为前装和后装两种，前装DMS一般由主机厂设计开发，仅适配当前车型和安装位置，而后装DMS，主要由相关软硬件供应商设计开发，可方便安装在各种车型和各种位置上。该产品主要用于座舱环境，针对驾驶员的状态进行监控，一方面是为了驾驶员的安全，在危险状态下及时提醒，另一方面，国家规定某些场景必须安装DMS，用于及时监控到危险驾驶行为，及时制止驾驶员可能产生的交通事故。
图2示出根据本申请一个示例性实施例提供的DMS的架构示意图。如图2所示，DMS可以包括车辆21，车辆21可以是具有无线通信功能的车辆，其中，无线通信功能可设置于该车辆21的车载终端、车载模组、车载单元、芯片(系统)或其他部件或组件。本申请的实施例中的车辆21用于监控驾驶员状态，在确定为疲劳和分神状态的情况下发出提醒信息，提醒信息指示驾驶员注意安全驾驶。
车辆21上可以设置有至少一个传感器22，如车载雷达(如毫米波雷达、激光雷达、超声波雷达等)、雨量传感器、摄像头、车姿传感器(如陀螺仪)、惯性测量单元(inertial measurement unit，IMU)、全球导航卫星系统(global navigation satellite system，GNSS)等，车辆21上还可以设置有其他传感器。
其中，摄像头用于捕获静态图像或视频。通常，摄像头可以包括感光元件比如镜头组和图像传感器，其中，镜头组包括多个透镜(凸透镜或凹透镜)，用于采集待拍摄物体反射的光信号，并将采集的光信号传递给图像传感器。图像传感器根据光信号生成待拍摄物体的原始图像。比如，摄像头为DMS红外摄像头。通过车辆21上设置的至少一个摄像头可以采集到驾驶员图像帧，驾驶员图像帧为包括驾驶员人脸的图像帧。
其中，通过车辆21上设置的至少一个车载雷达，还可以采集到路面的点云数据以 及车辆21的垂向加速度(即车辆21在垂直于路面方向上的加速度数据)等数据。
通过车辆21上设置的至少一个传感器22，还可以采集到路面的点云数据以及车辆21的垂向加速度(即车辆21在垂直于路面方向上的加速度数据)等数据。
车辆21上还可以设置有驾驶员监控系统23，驾驶员监控系统23用于监控驾驶员状态。车辆21上还可以设置有自动驾驶系统24，驾驶员监控系统23辅助自动驾驶系统24，当驾驶员监控系统23确定出疲劳和分神状态的情况下，可有自动驾驶系统24接管，自动驾驶系统24可用于根据传感器采集的数据，生成用于应对路面情况的自动驾驶策略，并根据生成的策略实现车辆21的自动驾驶。
车辆21上还可以设置有人机界面(human machine interface，HMI)25，人机界面25可用于通过视觉图标、语音播报方式对当前的路面情况以及自动驾驶系统24对车辆21采取的策略进行播报，以提醒相关驾乘人员。
车辆21上还可以设置有处理器26，比如处理器26为高性能计算处理器。处理器26用于通过摄像头获取第一图像帧和第二图像帧，第一图像帧和第二图像帧均为包括驾驶员人脸的图像帧，第一图像帧是在第二图像帧之前采集的图像帧；处理器26用于通过驾驶员监控系统23获取第一图像帧的第一检测信息和第二图像帧的第二检测信息，第一检测信息和第二检测信息均指示驾驶员的眼部状态和头部姿态；在第二检测信息指示第二图像帧对应的第一眼部状态为闭眼状态时，根据第一检测信息和第二检测信息确定第二图像帧对应的第二眼部状态。
在一种可能的实现方式中，本申请实施例的DMS还可以包括服务器，服务器可以作为车载计算单元位于上述车辆21上，也可以位于云端，可以是实体设备，也可以是虚拟设备如虚拟机、容器等，具有无线通信功能，其中，无线通信功能可设置于该服务器的芯片(系统)或其他部件或组件。服务器和车辆21可以通过无线连接的方式进行通信，例如可以通过2G/3G/4G/5G等移动通信技术，以及Wi-Fi、蓝牙、调频(frequency modulation，FM)、数传电台、卫星通信等无线通信方式进行通信，例如在测试中，服务器可承载于车辆21上并与车辆21通过无线连接的方式进行通信，通过车辆21和服务器之间的通信，服务器可以收集一个或多个车辆21上、或是设置在道路上或其他地方的传感器采集到的数据进行计算，并将计算结果回传给对应的车辆21。
图3示出根据本申请一个示例性实施例提供的驾驶员状态检测装置的架构示意图。该驾驶员状态检测装置可以通过专用硬件电路，或者，软硬件的结合实现成为图2中的DMS的全部或一部分，该驾驶员状态检测装置包括：图像采集模块310、眼部状态检测模块320和头部姿态检测模块330。
图像采集模块310用于采集驾驶员图像帧。眼部状态检测模块320用于检测驾驶员图像帧中的驾驶员的眼部状态，头部姿态检测模块330用于检测驾驶员图像帧中的驾驶员的头部姿态。
该驾驶员状态检测装置还可以包括其他疲劳依赖检测模块340和疲劳状态检测模块350。其他疲劳依赖检测模块340用于检测驾驶员图像帧中的驾驶员的其他指定状态，其他指定状态是与驾驶员疲劳状态相关的生物特征状态，比如其他指定状态为哈欠状态。
眼部状态检测模块320还用于将检测到的驾驶员的眼部状态输入至疲劳状态检测模块350，头部姿态检测模块330还用于将检测到的驾驶员的头部姿态输入至疲劳状态检测模块350，其他疲劳依赖检测模块340还用于将检测到的驾驶员的其他指定状态输入至疲劳状态检测模块350。对应的，疲劳状态检测模块350用于根据输入的状态(比如眼部状态、头部姿态和其他指定状态)综合判断驾驶员的疲劳状态。
以下以图2或图3所提供的DMS(下面简称为系统)的架构为例，对本申请实施例提供的驾驶员状态检测方法的流程进行说明。
图4示出根据本申请一个示例性实施例提供的驾驶员状态检测方法的流程示意图。如图4所示，驾驶员状态检测方法的流程包括：
步骤401，获取第一图像帧和第二图像帧，第一图像帧和第二图像帧均为包括驾驶员人脸的图像帧，第一图像帧是在第二图像帧之前采集的图像帧。
可选地，系统通过摄像头采集图像帧序列，图像帧序列包括至少两个图像帧，即至少一个第一图像帧和一个第二图像帧。
其中，第一图像帧和第二图像帧均为包括驾驶员人脸的图像帧，即第一图像帧和第二图像帧中均包括驾驶员的整个面部的特征。
第一图像帧是在第二图像帧之前采集的图像帧。可选地，第一图像帧是在第二图像帧之前采集的至少两个图像帧。
可选地，至少一个第一图像帧和一个第二图像帧为连续的多个图像帧。
步骤402，获取第一图像帧的第一检测信息和第二图像帧的第二检测信息，第一检测信息和第二检测信息均指示驾驶员的眼部状态和头部姿态。
可选地，系统对第一图像帧进行眼部状态检测和头部姿态检测得到第一检测信息，第一检测信息指示该第一图像帧对应的驾驶员的眼部状态和头部姿态。示意性的，第一检测信息包括第一眼部检测信息和第一头部检测信息，第一眼部检测信息指示该第一图像帧对应的驾驶员的眼部状态，第一头部检测信息指示该第一图像帧对应的驾驶员的头部姿态。其中，眼部状态包括闭眼状态或睁眼状态。
可选地，系统对第二图像帧进行眼部状态检测和头部姿态检测得到第二检测信息，第二检测信息指示该第二图像帧对应的驾驶员的眼部状态和头部姿态。示意性的，第二检测信息包括第二眼部检测信息和第二头部检测信息，第二眼部检测信息指示该第二图像帧对应的驾驶员的眼部状态，第二头部检测信息指示该第二图像帧对应的驾驶员的头部姿态。
需要说明的是，眼部状态检测和头部姿态检测可以是并列执行的，也可以是分先后顺序执行的，本申请实施例对此不加以限定。对眼部状态检测和头部姿态检测的相关细节可参考下面实施例中的相关描述，在此先不介绍。
步骤403，在第二检测信息指示第二图像帧对应的第一眼部状态为闭眼状态时，根据第一检测信息和第二检测信息确定第二图像帧对应的第二眼部状态。
可选地，系统判断第二检测信息是否指示第二图像帧对应的第一眼部状态为闭眼状态，若第二图像帧对应的第一眼部状态为闭眼状态，则根据第一检测信息和第二检测信息确定第二图像帧对应的第二眼部状态；若第二图像帧对应的第一眼部状态为睁 眼状态，则结束进程。
可选地，当第一检测信息和第二检测信息满足第一预设条件时，系统确定第二图像帧对应的第二眼部状态为睁眼状态。其中，第一预设条件为第二检测信息指示第二图像帧对应的头部姿态在俯仰角方向发生跳变，且第一检测信息指示第一图像帧对应的眼部状态为睁眼状态。
示意性的，第一图像帧包括多个第一图像帧，“第一检测信息指示第一图像帧对应的眼部状态为睁眼状态”的含义包括：第一检测信息指示眼部状态为睁眼状态的第一图像帧的数量在第一图像帧的总数量中的比例大于第一预设阈值。
其中，第一预设阈值为默认设置的，或者自定义设置的。比如，第一预设阈值为0.9。本申请实施例对此不加以限定。
可选地，当第一检测信息和第二检测信息满足第二预设条件时，系统确定第二图像帧对应的第二眼部状态为闭眼状态。其中，第二预设条件为第二检测信息指示第二图像帧对应的头部姿态在俯仰角方向发生跳变，且第一检测信息指示第一图像帧对应的眼部状态为闭眼状态。
示意性的，第一图像帧包括多个第一图像帧，“第一检测信息指示第一图像帧对应的眼部状态为闭眼状态”的含义包括：第一检测信息指示眼部状态为闭眼状态的第一图像帧的数量在第一图像帧的总数量中的比例大于第二预设阈值。
其中，第二预设阈值为默认设置的，或者自定义设置的。比如，第二预设阈值为0.95。本申请实施例对此不加以限定。
可选地，当第二检测信息指示第二图像帧对应的头部姿态在俯仰角方向未发生跳变时，系统确定第二图像帧对应的第二眼部状态为闭眼状态。
可选地，系统根据第二图像帧对应的头部姿态和第二眼部状态，确定疲劳状态检测结果。当疲劳状态检测结果满足预设报警条件时，输出报警信息。
可选地，系统获取预设的疲劳检测模型，根据第二图像帧对应的头部姿态和第二眼部状态调用疲劳检测模型，输出得到疲劳状态检测结果。其中，疲劳检测模型指示头部姿态、眼部状态与疲劳状态的相关关系，疲劳检测模型为基于样本图像帧预先训练完成的模型。比如，疲劳检测模型为基于眼睛状态和头部姿态融合得到的模型。或者，疲劳检测模型为基于眼睛状态、头部姿态和哈欠状态融合得到的模型。或者，疲劳检测模型为基于眼睛状态、头部姿态、哈欠状态和其他信息融合得到的模型。本申请实施例对此不加以限定。
在一种可能的实现方式中，疲劳状态检测结果包括第一检测结果和第二检测结果中的一种，第一检测结果指示驾驶员处于疲劳状态，第二检测结果指示驾驶员处于非疲劳状态。
在该实现方式中，当疲劳状态检测结果满足预设报警条件时，输出报警信息，包括：当疲劳状态检测结果为第一检测结果时，输出报警信息。
在另一种可能的实现方式中，疲劳状态检测结果包括疲劳状态等级，疲劳状态等级与预测的驾驶员的疲劳强度相关。示意性的，疲劳状态等级与预测的驾驶员的疲劳强度呈正相关关系，即疲劳状态等级越高，预测的驾驶员的疲劳强度越大。
在该实现方式中，当疲劳状态检测结果满足预设报警条件时，输出报警信息，包 括：当疲劳状态等级大于预设等级阈值时，输出报警信息。
其中，预设等级阈值为默认设置的，或者自定义设置的。本申请实施例对此不加以限定。
可选地，系统按照预设提示形式输出报警信息，预设提示形式包括语音形式、文字形式、图像形式、动画形式中的至少一种。本申请实施例对报警信息的输出方式和输出内容不加以限定。
综上所述，本申请实施例通过获取第一图像帧的第一检测信息和第二图像帧的第二检测信息(比如驾驶员的连续眼部状态和对应的连续头部姿态)，针对不同场景，比如低头往下看、低头打盹等场景，可以在确定第二图像帧对应的第一眼部状态为闭眼状态时，根据第一检测信息和第二检测信息确定第二图像帧对应的第二眼部状态，即对第二图像帧对应的眼部状态进行修正得到正确的眼部状态，解决了相关技术中系统在驾驶员低头时直接通过图像无法准确判断眼睛开闭状态的问题，保证了高精度和强鲁棒性的眼睛开闭状态识别效果，有助于后续获取稳定、精准的用户疲劳状态。
图5示出根据本申请另一个示例性实施例提供的驾驶员状态检测方法的流程示意图。如图5所示，驾驶员状态检测方法的流程包括：
步骤501，通过摄像头进行图像采集得到图像帧序列。
可选地，系统通过车辆内至少一个位置部署的摄像头进行图像采集，得到图像帧序列。其中，摄像头可以是红外摄像头。
可选地，至少一个位置包括以下任意一个或多个位置：转向柱、仪表盘上方或附近位置，中控台上方或附近位置，A柱或附近位置，后视镜或附近位置。其中，当摄像头放置在转向柱上对眼部状态的识别是最优的位置，不容易发生低头事件下无法正确识别眼部状态的情况，但是该位置容易发生方向盘遮挡人脸导致眼部状态无法识别，当摄像头放置在A柱或者后视镜位置时，图像采集不会产生方向盘遮挡，但是一定会发生低头事件下无法正确识别眼部状态的场景。
可选地，通过摄像头进行图像采集得到图像帧序列，包括：在车辆处于行驶状态时通过摄像头进行图像采集，获得图像帧序列；和/或，在车辆的行驶速度超过预设车速时通过摄像头进行图像采集，获得图像帧序列；和/或，在检测到车辆点火后通过摄像头进行图像采集，获得图像帧序列；和/或，在检测到车辆的启动指令时通过摄像头进行图像采集，获得图像帧序列；和/或，在检测到对车辆或车辆中部件或系统的控制指令时通过摄像头进行图像采集，获得图像帧序列。需要说明的是，本申请实施对图像采集的触发条件和采集方式不加以限定。
步骤502，对采集到的图像帧序列进行人脸检测。
人脸检测是人脸其他应用的基础，基于人脸的眼部状态检测和头部姿态算法受前端人脸检测算法的影响，当人脸检测算法定位产生抖动或者不准确导致的误识别或漏检时，导致眼部状态发生误识别和头部姿态发生比较大的抖动，从而导致系统将正常状态误判为低头事件发生，只有获取到了是否有人脸和人脸的精确的位置信息，系统才可能得到较为稳定准确的眼部状态结果。
可选地，系统采用预设人脸检测算法对采集到的图像帧序列进行人脸检测。
在一种可能的实现方式中，预设人脸检测算法为专用的人脸检测算法，比如多任务人脸检测算法，该多任务人脸检测算法用于检测人脸并输出人脸关键点信息，以便后续系统可以通过人脸关键点信息对人脸进行校正，为后端的眼部状态检测算法提供更好的输入，提升单帧眼部状态识别算法的准确率。专用的人脸检测算法具有非常高的准确率，但是该算法端侧部署非常耗时较长，不适合端侧部署。
在另一种可能的实现方式中，预设人脸检测算法为轻量级的人脸检测算法，该算法那适合端侧部署。
在另一种可能的实现方式中，预设人脸检测算法为通用的目标检测算法，比如单阶段目标检测算法，该算法端侧速度快，准确率高，较为适合在车机系统上部署作为人脸检测算法。需要说明的是，本申请实施例对预设人脸检测算法不加以限定。
步骤503，判断是否在图像帧序列中检测到人脸。
若系统在图像帧序列中未检测到人脸，继续执行步骤501；若系统在图像帧序列中检测到人脸，则执行步骤504。
步骤504，若在图像帧序列中检测到人脸，则进行眼部状态检测得到眼部状态序列，并进行头部姿态检测得到头部姿态序列。
可选地，针对图像帧序列中的每帧图像帧，系统进行眼部状态检测和头部姿态检测得到该图像帧对应的眼部状态检测数据和头部姿态检测数据，从而获得该图像帧序列对应的眼部状态序列和头部姿态序列，即该眼部状态序列包括该图像帧序列中多个图像帧各自对应的眼部状态检测数据，头部姿态序列包括该图像帧序列中多个图像帧各自对应的头部姿态检测数据。
针对单帧的眼部状态检测，在一种可能的实现方式中，系统可以采用目标检测算法进行眼部状态检测，基于目标检测算法进行眼部状态识别具有很强的鲁棒性，眼部状态识别受干扰可能性较小，很容易支持带口罩，手部遮挡人脸和化妆的场景，只有当人眼部位被遮挡或者被干扰的场景，才会影响眼部状态的识别。但是，基于目标检测算法，无法区分眯眯眼场景和因为疲劳而眯眼的场景，导致两种状态存在相互误识别的情况。该眼部状态识别算法适合人脸数据量较少，无法全场景覆盖驾驶环境的人脸样本的场景采用。
在另一种可能的实现方式中，由于人脸尺寸和眼睛尺寸变化小，系统可以采用剪枝后通用的目标检测算法进行眼部状态检测，仅用部分网络分支进行眼部状态的预测，可保证准确率的同时，检测速度有较大提升，端侧优势更大。
在另一种可能的实现方式中，系统可以采用预设分类算法极性进行眼部状态检测。基于预设分类算法进行眼部状态检测可以分为两种情况，一种情况是根据包括人脸的图像帧判断眼部状态，由于人眼占比非常小，容易受人脸其他部位的干扰导致误识别，如戴口罩、化妆等，都有可能导致误识别，同时该方法无法区分眼睛是否被遮挡导致的无效状态。另外一种情况是增加人眼检测模型，然后将包括人眼的图像帧输入至人眼检测模型输出得到眼部状态，其中人眼检测模型为预先训练的用于在图像帧中识别眼部状态的神经网络模型，该种方法不受人眼其他部位的干扰，鲁棒性更高，准确率相比上一种情况更高，该方法同样存在无法区分眯眯眼场景和因为疲劳而眯眼的场景，同时该方法新增人眼检测模型，资源消耗增加，端侧性能降低。
在另一种可能的实现方式中，系统可以采用关键点算法进行眼部状态检测。可选地，如图6所示，系统通过检测眼睛上、下眼皮和眼角6个关键点位置“p 43、p 44、p 45、p 46、p 47、p 48”，然后通过如下公式计算归一化的距离EAR：
Figure PCTCN2021137537-appb-000001
其中，归一化的距离作为眼睛开度指标，当眼睛开度指标小于预设开度阈值，则确定眼睛处于闭眼状态，否则确定眼睛处于睁眼状态，其中，预设开度阈值是默认设置的，或者自定义设置的，比如，预设开度阈值为正常眼睛开度值的20％，本申请实施例对此不加以限定。基于关键点算法结合头部姿态校准，可以区分眯眯眼场景和因为疲劳而眯眼的场景，显著提升这两种场景的眼部状态识别的准确性。基于眼睛开度指标的眼睛开闭状态非常适合作为后续判断驾驶员疲劳状态的依据，但是该方法对关键点算法的稳定性要求高，要求关键点位置不会出现较大抖动，也即是相应的眼睛开度要求稳定，为防止眼睛开度出现较大抖动，可以增加滤波方法，即系统对输出的眼睛开度指标进行滤波处理，得到更加稳定的眼睛开度指标。
针对单帧的头部姿态检测，系统将二维的图像帧映射到三维图像帧得到头部检测姿态数据，头部姿态检测数据指示驾驶员人脸姿态的朝向。可选地，头部姿态检测数据采用3个自由度(degrees of freedom，dof)表示，即俯仰角(pitch)，滚动角(roll)和偏航角(yaw)。如图7所示，头部姿态检测数据包括驾驶员的头部与设备坐标系的横轴之间的俯仰角、头部与设备坐标系的纵轴之间的偏航角以及头部与设备坐标系的竖轴之间的滚动角。
对于整个眼部状态识别的系统中，头部姿态检测的准确率和稳定性至关重要，同时由于摄像头的安装位置和低头原因，头部姿态检测算法需要支持大角度头部姿态。
在一种可能的实现方式中，系统基于人脸关键点进行头部姿态检测。该方法可基于已有的大量关键点数据训练；可获得人脸各部位的关键点信息；因为点的扰动对角度影响比较小，头部姿态更稳定。
在另一种可能的实现方式中，系统基于回归算法进行头部姿态检测。该方法可直接回归欧拉角，简洁方便；模型设计简单，不需要经过后处理或者二次算法优化求解。基于回归的头部姿态检测算法，难点在于头部姿态数据采集，头部姿态数据采集的方法包括数据生成的方法和直接采集的方法，直接采集的方法包括：通过光学追踪仪和相机阵列可以直接采集到真实的相机坐标系的头部姿态标签，可以覆盖到360°的头部姿态数据，但是设备昂贵，部署复杂，采集周期长，人力投入大。而数据生成的方法包括：对二维的图像帧中的关键点数据进行三维形变模型(3D Morphable Model，3DMM)拟合，再通过头部姿态增强，得到大角度下的头部姿态标签，该方法可支撑±90°下的头部姿态数据，该方法仅依赖2D关键点标注，可行性和成本都比较合理。
在另一种可能的实现方式中，系统基于基于人脸关键点和回归算法进行头部姿态检测。可选地，系统将图像帧输入至头部姿态检测模型，输出得到头部姿态数据。其中，头部姿态检测模型为基于人脸关键点和回归的方法的多任务神经网络模型，即将欧拉角回归和人脸关键点等任务做在一个网络模型中，通过共享骨架网络，用多个头部网络去做多个不同的任务，通过此方法可以一定程度降低欧拉角的预测误差。
需要说明的是，本申请实施例对眼部状态检测算法和头部姿态检测算法的方式均 不加以限定。
步骤505，判断第二图像帧对应的第一眼部状态是否为闭眼状态。
经过对图像帧序列中的每帧图像帧进行眼部状态检测和头部姿态检测，系统得到该图像帧序列对应的眼部状态队列和头部姿态队列，系统判断该图像帧序列中的最新帧即第二图像帧中的第一眼部状态是否为闭眼状态。
若第二图像帧对应的第一眼部状态为睁眼状态，则结束进程；若第二图像帧对应的第一眼部状态为闭眼状态，则系统继续执行步骤506，通过校验该图像帧序列对应的眼部状态队列和头部姿态队列，区分如下三种场景：正常闭眼场景、低头睁眼往下看场景和低头打盹场景。
步骤506，若第二图像帧对应的第一眼部状态为闭眼状态，则判断第二图像帧对应的头部姿态在俯仰角方向是否发生跳变。
若第二图像帧对应的第一眼部状态为闭眼状态，则系统判断第二图像帧对应的头部姿态在俯仰角方向是否发生跳变，若第二图像帧对应的头部姿态在俯仰角方向未发生跳变，则执行步骤507，若第二图像帧对应的头部姿态在俯仰角方向发生跳变，则执行步骤508。
步骤507，若第二图像帧对应的头部姿态在俯仰角方向未发生跳变，则输出第二图像帧对应的第二眼部状态为闭眼状态。
若第二图像帧对应的头部姿态在俯仰角方向未发生跳变，则确定为正常闭眼场景，输出第二图像帧对应的第二眼部状态为闭眼状态。
在一个示意性的例子中，如图8所示，当眼睛从睁眼状态(open)到闭眼状态(close)的同时，头部姿态在俯仰角方向未发生跳变，系统判定为正常闭眼场景，眼部状态保持原有的闭眼状态，得到修正后的眼部状态。
步骤508，若第二图像帧对应的头部姿态在俯仰角方向发生跳变，则判断第一图像帧对应的眼部状态是否为闭眼状态。
若第二图像帧对应的头部姿态在俯仰角方向发生跳变，则系统判断第一图像帧对应的眼部状态是否为闭眼状态，若第一图像帧对应的眼部状态为睁眼状态，则执行步骤509，若第一图像帧对应的眼部状态为闭眼状态，则执行步骤510。
步骤509，若第一图像帧对应的眼部状态为睁眼状态，则输出第二图像帧对应的第二眼部状态为睁眼状态。
若第一图像帧对应的眼部状态为睁眼状态，则确定为低头睁眼往下看场景，输出第二图像帧对应的第二眼部状态为睁眼状态。
在一个示意性的例子中，如图9所示，当眼睛从睁眼状态(open)到闭眼状态(close)的同时，头部姿态在俯仰角方向发生跳变，系统判定为低头睁眼往下看场景，眼部状态修正为睁眼状态(open)，得到修正后的眼部状态。
步骤510，若第一图像帧对应的眼部状态为闭眼状态，则输出第二图像帧对应的第二眼部状态为闭眼状态。
若第一图像帧对应的眼部状态为闭眼状态，则确定为低头打盹场景，输出第二图像帧对应的第二眼部状态为闭眼状态。
在一个示意性的例子中，如图10所示，当眼睛从睁眼状态(open)到闭眼状态(close) 的同时，头部姿态在俯仰角方向发生跳变，系统判定为低头打盹场景，眼部状态保持原有的闭眼状态(close)，得到修正后的眼部状态。
可选地，当第二图像帧对应的第一眼部状态为闭眼状态时，系统可根据眼部状态序列和头部姿态序列区分正常闭眼、低头睁眼往下看和低头打盹三种场景。当仅用眼部状态序列来作为疲劳状态的输入参数时，会导致在这三种状态下发生误报，影响告警系统的有效性。我们通过增加头部姿态检测模块，可以很容易的区分正常闭眼和低头睁眼往下看两种场景，如果检测到闭眼状态的同时，校验头部姿态是否发生跳变：如果没有发生跳变，则判定为正常的闭眼场景，不修正结果保持原有的眼部状态；如果发生跳变，则判定为低头睁眼往下看场景；并修正原有的闭眼状态为睁眼状态。但是此时会将低头打盹事件误判为低头睁眼的情况，导致眼部状态修正错误。为解决低头打盹被误修正为睁眼的情况，增加了眼部稳定态队列，通过校验历史连续帧的眼部状态即第一图像帧对应的眼部状态，去区分低头睁眼往下看和低头打盹两种场景，如果第一图像帧对应的眼部状态为闭眼状态，则判定为低头打盹场景，不修正结果保持原有闭眼状态，如果第一图像帧对应的眼部状态为睁眼状态，则判定为低头睁眼往下看场景，修正原有的闭眼状态为睁眼状态。
在一个示意性的例子中，区分三种场景和修正过程如图11所示。图11中，第一行表示头部姿态在俯仰角方向的头部姿态序列，第二行表示与头部姿态对应的不做任何后处理的眼部状态序列。第三行表示增加头部姿态检测模块，修正后的眼部状态序列，从图11中可以看到，低头打盹场景的眼部状态被误修正为睁眼状态；第四行表示再增加眼部稳定态队列，进一步修正眼部状态，得到准确的眼部状态。
综上所述，本申请实施例提供的驾驶员状态检测方法，可以准确的区分三种从图像上分析几乎没有区别的正常闭眼、往下一瞥和打盹场景，同时利用稳定态队列准确的将往下一瞥导致的闭眼误检测恢复为睁眼状态，不会错误修改打盹场景下的闭眼状态，并且，通过结合头部姿态序列和眼部状态序列来区分不同场景，容错性更高，显著提高了驾驶场景中眼部状态的识别的准确性。
下述为本申请装置实施例，可以用于执行本申请方法实施例。对于本申请装置实施例中未披露的细节，请参照本申请方法实施例。
请参考图12，其示出了本申请一个示例性实施例提供的驾驶员状态检测装置的框图。该装置可以通过软件、硬件或者两者的结合实现成为图2提供的DMS或者图3提供的驾驶员状态检测装置的全部或者一部分。该装置可以包括：第一获取单元1210、第二获取单元1220和确定单元1230；
第一获取单元1210，用于获取第一图像帧和第二图像帧，第一图像帧和第二图像帧均为包括驾驶员人脸的图像帧，第一图像帧是在第二图像帧之前采集的图像帧；
第二获取单元1220，用于获取第一图像帧的第一检测信息和第二图像帧的第二检测信息，第一检测信息和第二检测信息均指示驾驶员的眼部状态和头部姿态；
确定单元1230，用于在第二检测信息指示第二图像帧对应的第一眼部状态为闭眼状态时，根据第一检测信息和第二检测信息确定第二图像帧对应的第二眼部状态。
在一种可能的实现方式中，确定单元1230，还用于：
当第一检测信息和第二检测信息满足第一预设条件时，确定第二图像帧对应的第二眼部状态为睁眼状态；
其中，第一预设条件为第二检测信息指示第二图像帧对应的头部姿态在俯仰角方向发生跳变，且第一检测信息指示第一图像帧对应的眼部状态为睁眼状态。
在另一种可能的实现方式中，确定单元1230，还用于：
当第一检测信息和第二检测信息满足第二预设条件时，确定第二图像帧对应的第二眼部状态为闭眼状态；
其中，第二预设条件为第二检测信息指示第二图像帧对应的头部姿态在俯仰角方向发生跳变，且第一检测信息指示第一图像帧对应的眼部状态为闭眼状态。
在另一种可能的实现方式中，确定单元1230，还用于：
当第二检测信息指示第二图像帧对应的头部姿态在俯仰角方向未发生跳变时，确定第二图像帧对应的第二眼部状态为闭眼状态。
在另一种可能的实现方式中，该装置还包括：检测单元；
检测单元，用于根据第二图像帧对应的头部姿态和第二眼部状态，确定疲劳状态检测结果。
在另一种可能的实现方式中，该装置还包括：报警单元；
报警单元，用于当疲劳状态检测结果满足预设报警条件时，输出报警信息。
需要说明的是，上述实施例提供的装置，在实现其功能时，仅以上述各功能模块的划分进行举例说明，实际应用中，可以根据需要而将上述功能分配由不同的功能模块完成，即将设备的内部结构划分成不同的功能模块，以完成以上描述的全部或者部分功能。另外，上述实施例提供的装置与方法实施例属于同一构思，其具体实现过程详见方法实施例，这里不再赘述。
本申请实施例提供了一种驾驶员状态检测装置，该驾驶员状态检测装置包括：处理器；用于存储处理器可执行指令的存储器；其中，处理器被配置为执行指令时实现上述实施例中由DMS执行的方法。
本申请实施例提供了一种计算机程序产品，包括计算机可读代码，或者承载有计算机可读代码的非易失性计算机可读存储介质，当计算机可读代码在处理器中运行时，处理器执行上述实施例中由DMS执行的方法。
本申请实施例提供了一种非易失性计算机可读存储介质，其上存储有计算机程序指令，计算机程序指令被处理器执行时实现上述实施例中由DMS执行的方法。
计算机可读存储介质可以是可以保持和存储由指令执行设备使用的指令的有形设备。计算机可读存储介质例如可以是――但不限于――电存储设备、磁存储设备、光存储设备、电磁存储设备、半导体存储设备或者上述的任意合适的组合。计算机可读存储介质的更具体的例子(非穷举的列表)包括：便携式计算机盘、硬盘、随机存取存储器(Random Access Memory，RAM)、只读存储器(Read Only Memory，ROM)、可擦式可编程只读存储器(Electrically Programmable Read-Only-Memory，EPROM或闪存)、静态随机存取存储器(Static Random-Access Memory，SRAM)、便携式压缩 盘只读存储器(Compact Disc Read-Only Memory，CD-ROM)、数字多功能盘(Digital Video Disc，DVD)、记忆棒、软盘、机械编码设备、例如其上存储有指令的打孔卡或凹槽内凸起结构、以及上述的任意合适的组合。
这里所描述的计算机可读程序指令或代码可以从计算机可读存储介质下载到各个计算/处理设备，或者通过网络、例如因特网、局域网、广域网和/或无线网下载到外部计算机或外部存储设备。网络可以包括铜传输电缆、光纤传输、无线传输、路由器、防火墙、交换机、网关计算机和/或边缘服务器。每个计算/处理设备中的网络适配卡或者网络接口从网络接收计算机可读程序指令，并转发该计算机可读程序指令，以供存储在各个计算/处理设备中的计算机可读存储介质中。
用于执行本申请操作的计算机程序指令可以是汇编指令、指令集架构(Instruction Set Architecture，ISA)指令、机器指令、机器相关指令、微代码、固件指令、状态设置数据、或者以一种或多种编程语言的任意组合编写的源代码或目标代码，所述编程语言包括面向对象的编程语言—诸如Smalltalk、C++等，以及常规的过程式编程语言—诸如“C”语言或类似的编程语言。计算机可读程序指令可以完全地在用户计算机上执行、部分地在用户计算机上执行、作为一个独立的软件包执行、部分在用户计算机上部分在远程计算机上执行、或者完全在远程计算机或服务器上执行。在涉及远程计算机的情形中，远程计算机可以通过任意种类的网络—包括局域网(Local Area Network，LAN)或广域网(Wide Area Network，WAN)—连接到用户计算机，或者，可以连接到外部计算机(例如利用因特网服务提供商来通过因特网连接)。在一些实施例中，通过利用计算机可读程序指令的状态信息来个性化定制电子电路，例如可编程逻辑电路、现场可编程门阵列(Field-Programmable Gate Array，FPGA)或可编程逻辑阵列(Programmable Logic Array，PLA)，该电子电路可以执行计算机可读程序指令，从而实现本申请的各个方面。
这里参照根据本申请实施例的方法、装置(系统)和计算机程序产品的流程图和/或框图描述了本申请的各个方面。应当理解，流程图和/或框图的每个方框以及流程图和/或框图中各方框的组合，都可以由计算机可读程序指令实现。
这些计算机可读程序指令可以提供给通用计算机、专用计算机或其它可编程数据处理装置的处理器，从而生产出一种机器，使得这些指令在通过计算机或其它可编程数据处理装置的处理器执行时，产生了实现流程图和/或框图中的一个或多个方框中规定的功能/动作的装置。也可以把这些计算机可读程序指令存储在计算机可读存储介质中，这些指令使得计算机、可编程数据处理装置和/或其他设备以特定方式工作，从而，存储有指令的计算机可读介质则包括一个制造品，其包括实现流程图和/或框图中的一个或多个方框中规定的功能/动作的各个方面的指令。
也可以把计算机可读程序指令加载到计算机、其它可编程数据处理装置、或其它设备上，使得在计算机、其它可编程数据处理装置或其它设备上执行一系列操作步骤，以产生计算机实现的过程，从而使得在计算机、其它可编程数据处理装置、或其它设备上执行的指令实现流程图和/或框图中的一个或多个方框中规定的功能/动作。
附图中的流程图和框图显示了根据本申请的多个实施例的装置、系统、方法和计算机程序产品的可能实现的体系架构、功能和操作。在这点上，流程图或框图中的每 个方框可以代表一个模块、程序段或指令的一部分，所述模块、程序段或指令的一部分包含一个或多个用于实现规定的逻辑功能的可执行指令。在有些作为替换的实现中，方框中所标注的功能也可以以不同于附图中所标注的顺序发生。例如，两个连续的方框实际上可以基本并行地执行，它们有时也可以按相反的顺序执行，这依所涉及的功能而定。
也要注意的是，框图和/或流程图中的每个方框、以及框图和/或流程图中的方框的组合，可以用执行相应的功能或动作的硬件(例如电路或ASIC(Application Specific Integrated Circuit，专用集成电路))来实现，或者可以用硬件和软件的组合，如固件等来实现。
尽管在此结合各实施例对本申请进行了描述，然而，在实施所要求保护的本申请过程中，本领域技术人员通过查看所述附图、公开内容、以及所附权利要求书，可理解并实现所述公开实施例的其它变化。在权利要求中，“包括”(comprising)一词不排除其他组成部分或步骤，“一”或“一个”不排除多个的情况。单个处理器或其它单元可以实现权利要求中列举的若干项功能。相互不同的从属权利要求中记载了某些措施，但这并不表示这些措施不能组合起来产生良好的效果。
以上已经描述了本申请的各实施例，上述说明是示例性的，并非穷尽性的，并且也不限于所披露的各实施例。在不偏离所说明的各实施例的范围和精神的情况下，对于本技术领域的普通技术人员来说许多修改和变更都是显而易见的。本文中所用术语的选择，旨在最好地解释各实施例的原理、实际应用或对市场中的技术的改进，或者使本技术领域的其它普通技术人员能理解本文披露的各实施例。

Claims (16)

  1. 一种驾驶员状态检测方法，其特征在于，所述方法包括：
    获取第一图像帧和第二图像帧，所述第一图像帧和所述第二图像帧均为包括驾驶员人脸的图像帧，所述第一图像帧是在所述第二图像帧之前采集的图像帧；
    获取所述第一图像帧的第一检测信息和所述第二图像帧的第二检测信息，所述第一检测信息和所述第二检测信息均指示所述驾驶员的眼部状态和头部姿态；
    在所述第二检测信息指示所述第二图像帧对应的第一眼部状态为闭眼状态时，根据所述第一检测信息和所述第二检测信息确定所述第二图像帧对应的第二眼部状态。
  2. 根据权利要求1所述的方法，其特征在于，所述根据所述第一检测信息和所述第二检测信息确定所述第二图像帧对应的第二眼部状态，包括：
    当所述第一检测信息和所述第二检测信息满足第一预设条件时，确定所述第二图像帧对应的第二眼部状态为睁眼状态；
    其中，所述第一预设条件为所述第二检测信息指示所述第二图像帧对应的头部姿态在俯仰角方向发生跳变，且所述第一检测信息指示所述第一图像帧对应的眼部状态为睁眼状态。
  3. 根据权利要求1或2所述的方法，其特征在于，所述根据所述第一检测信息和所述第二检测信息确定所述第二图像帧对应的第二眼部状态，包括：
    当所述第一检测信息和所述第二检测信息满足第二预设条件时，确定所述第二图像帧对应的第二眼部状态为闭眼状态；
    其中，所述第二预设条件为所述第二检测信息指示所述第二图像帧对应的头部姿态在俯仰角方向发生跳变，且所述第一检测信息指示所述第一图像帧对应的眼部状态为闭眼状态。
  4. 根据权利要求1至3任一所述的方法，其特征在于，所述方法还包括：
    当所述第二检测信息指示所述第二图像帧对应的头部姿态在俯仰角方向未发生跳变时，确定所述第二图像帧对应的第二眼部状态为闭眼状态。
  5. 根据权利要求1至4任一所述的方法，其特征在于，所述方法还包括：
    根据所述第二图像帧对应的所述头部姿态和所述第二眼部状态，确定疲劳状态检测结果。
  6. 根据权利要求5所述的方法，其特征在于，所述方法还包括：
    当所述疲劳状态检测结果满足预设报警条件时，输出报警信息。
  7. 一种驾驶员状态检测装置，其特征在于，所述装置包括：
    第一获取单元，用于获取第一图像帧和第二图像帧，所述第一图像帧和所述第二 图像帧均为包括驾驶员人脸的图像帧，所述第一图像帧是在所述第二图像帧之前采集的图像帧；
    第二获取单元，用于获取所述第一图像帧的第一检测信息和所述第二图像帧的第二检测信息，所述第一检测信息和所述第二检测信息均指示所述驾驶员的眼部状态和头部姿态；
    确定单元，用于在所述第二检测信息指示所述第二图像帧对应的第一眼部状态为闭眼状态时，根据所述第一检测信息和所述第二检测信息确定所述第二图像帧对应的第二眼部状态。
  8. 根据权利要求7所述的装置，其特征在于，所述确定单元，还用于：
    当所述第一检测信息和所述第二检测信息满足第一预设条件时，确定所述第二图像帧对应的第二眼部状态为睁眼状态；
    其中，所述第一预设条件为所述第二检测信息指示所述第二图像帧对应的头部姿态在俯仰角方向发生跳变，且所述第一检测信息指示所述第一图像帧对应的眼部状态为睁眼状态。
  9. 根据权利要求7或8所述的装置，其特征在于，所述确定单元，还用于：
    当所述第一检测信息和所述第二检测信息满足第二预设条件时，确定所述第二图像帧对应的第二眼部状态为闭眼状态；
    其中，所述第二预设条件为所述第二检测信息指示所述第二图像帧对应的头部姿态在俯仰角方向发生跳变，且所述第一检测信息指示所述第一图像帧对应的眼部状态为闭眼状态。
  10. 根据权利要求7至9任一所述的装置，其特征在于，所述确定单元，还用于：
    当所述第二检测信息指示所述第二图像帧对应的头部姿态在俯仰角方向未发生跳变时，确定所述第二图像帧对应的第二眼部状态为闭眼状态。
  11. 根据权利要求7至10任一所述的装置，其特征在于，所述装置还包括：检测模块；
    所述检测模块，用于根据所述第二图像帧对应的所述头部姿态和所述第二眼部状态，确定疲劳状态检测结果。
  12. 根据权利要求11所述的装置，其特征在于，所述装置还包括：报警模块；
    所述报警模块，用于当所述疲劳状态检测结果满足预设报警条件时，输出报警信息。
  13. 一种驾驶员状态检测装置，其特征在于，所述装置包括：
    处理器；
    用于存储处理器可执行指令的存储器；
    其中，所述处理器被配置为执行所述指令时实现权利要求1-6任意一项所述的方法。
  14. 一种非易失性计算机可读存储介质，其上存储有计算机程序指令，其特征在于，所述计算机程序指令被处理器执行时实现权利要求1-6任意一项所述的方法。
  15. 一种计算机程序产品，其特征在于，所述计算机程序产品在计算机上运行时，所述计算机执行如权利要求1-6任意一项所述的方法。
  16. 一种车辆，其特征在于，所述车辆包含如权利要求7-12任意一项所述的装置。
PCT/CN2021/137537 2021-12-13 2021-12-13 驾驶员状态检测方法、装置及存储介质 WO2023108364A1 (zh)

Priority Applications (2)

Application Number Priority Date Filing Date Title
CN202180022842.5A CN116615765A (zh) 2021-12-13 2021-12-13 驾驶员状态检测方法、装置及存储介质
PCT/CN2021/137537 WO2023108364A1 (zh) 2021-12-13 2021-12-13 驾驶员状态检测方法、装置及存储介质

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/CN2021/137537 WO2023108364A1 (zh) 2021-12-13 2021-12-13 驾驶员状态检测方法、装置及存储介质

Publications (1)

Publication Number Publication Date
WO2023108364A1 true WO2023108364A1 (zh) 2023-06-22

Family

ID=86775273

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2021/137537 WO2023108364A1 (zh) 2021-12-13 2021-12-13 驾驶员状态检测方法、装置及存储介质

Country Status (2)

Country Link
CN (1) CN116615765A (zh)
WO (1) WO2023108364A1 (zh)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117687047A (zh) * 2024-01-31 2024-03-12 中寰星网数字科技(大连)有限公司 基于边缘计算的人工智能gnss高精度位移处理方法

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20170236016A1 (en) * 2016-02-15 2017-08-17 Renesas Electronics Corporation Eye opening degree detection system, doze detection system, automatic shutter system, eye opening degree detection method, and eye opening degree detection program
CN108701214A (zh) * 2017-12-25 2018-10-23 深圳市大疆创新科技有限公司 图像数据处理方法、装置及设备
CN109670421A (zh) * 2018-12-04 2019-04-23 青岛小鸟看看科技有限公司 一种疲劳状态检测方法和装置
US20190122044A1 (en) * 2016-04-07 2019-04-25 Seeing Machines Limited Method and system of distinguishing between a glance event and an eye closure
CN111079476A (zh) * 2018-10-19 2020-04-28 上海商汤智能科技有限公司 驾驶状态分析方法和装置、驾驶员监控系统、车辆
CN112949370A (zh) * 2019-12-10 2021-06-11 托比股份公司 眼睛事件检测
CN113536967A (zh) * 2021-06-25 2021-10-22 武汉极目智能技术有限公司 基于驾驶员头部运动姿态与人眼开合度的驾驶员状态辨识方法、装置、电子设备

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20170236016A1 (en) * 2016-02-15 2017-08-17 Renesas Electronics Corporation Eye opening degree detection system, doze detection system, automatic shutter system, eye opening degree detection method, and eye opening degree detection program
US20190122044A1 (en) * 2016-04-07 2019-04-25 Seeing Machines Limited Method and system of distinguishing between a glance event and an eye closure
CN108701214A (zh) * 2017-12-25 2018-10-23 深圳市大疆创新科技有限公司 图像数据处理方法、装置及设备
CN111079476A (zh) * 2018-10-19 2020-04-28 上海商汤智能科技有限公司 驾驶状态分析方法和装置、驾驶员监控系统、车辆
CN109670421A (zh) * 2018-12-04 2019-04-23 青岛小鸟看看科技有限公司 一种疲劳状态检测方法和装置
CN112949370A (zh) * 2019-12-10 2021-06-11 托比股份公司 眼睛事件检测
CN113536967A (zh) * 2021-06-25 2021-10-22 武汉极目智能技术有限公司 基于驾驶员头部运动姿态与人眼开合度的驾驶员状态辨识方法、装置、电子设备

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117687047A (zh) * 2024-01-31 2024-03-12 中寰星网数字科技(大连)有限公司 基于边缘计算的人工智能gnss高精度位移处理方法

Also Published As

Publication number Publication date
CN116615765A (zh) 2023-08-18

Similar Documents

Publication Publication Date Title
CN109345829B (zh) 无人车的监控方法、装置、设备及存储介质
JP6791198B2 (ja) 自律型車両向けの交通状況認知
US9881221B2 (en) Method and system for estimating gaze direction of vehicle drivers
US10372972B2 (en) Face detection apparatus, face detection method, and program
US9824286B2 (en) Method and apparatus for early detection of dynamic attentive states for providing an inattentive warning
CN112389448B (zh) 一种基于车辆状态和驾驶员状态的异常驾驶行为识别方法
CN110765807B (zh) 驾驶行为分析、处理方法、装置、设备和存储介质
US20200198645A1 (en) System and method for analysis of driver behavior
US20180017799A1 (en) Heads Up Display For Observing Vehicle Perception Activity
US20160159217A1 (en) System and method for determining drowsy state of driver
CN111506057B (zh) 辅助自动驾驶的自动驾驶辅助眼镜
García et al. Driver monitoring based on low-cost 3-D sensors
Zheng et al. Unsupervised driving performance assessment using free-positioned smartphones in vehicles
JPWO2017163514A1 (ja) 眼鏡型ウェアラブル端末、その制御方法および制御プログラム
WO2023108364A1 (zh) 驾驶员状态检测方法、装置及存储介质
JP2017129973A (ja) 運転支援装置および運転支援方法
CN112689587A (zh) 考虑到在要求接管驾驶任务时驾驶员的非驾驶任务活动的可中断性而分类非驾驶任务活动的方法及因要求接管驾驶任务而中断非驾驶任务活动后重新释放非驾驶任务活动的方法
CN116783462A (zh) 一种自动驾驶系统的性能测试方法
Kashevnik et al. Context-based driver support system development: Methodology and case study
JP7298351B2 (ja) 状態判定装置、車載機、運転評価システム、状態判定方法、及びプログラム
JP7068606B2 (ja) 運転評価装置、車載機、運転評価方法、及びコンピュータプログラム
JP7019394B2 (ja) 視認対象検知装置、視認対象検知方法、およびプログラム
RU2703341C1 (ru) Способ определения опасных состояний на дорогах общего пользования на основе мониторинга ситуации в кабине транспортного средства
JP7075048B2 (ja) 安全確認評価装置、車載機、これらを備えた安全確認評価システム、安全確認評価方法、及び安全確認評価プログラム
CN109693631A (zh) 显示视图警报的方法和装置

Legal Events

Date Code Title Description
WWE Wipo information: entry into national phase

Ref document number: 202180022842.5

Country of ref document: CN

121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21967506

Country of ref document: EP

Kind code of ref document: A1