CN111434553B

CN111434553B - Brake system, method and device, and fatigue driving model training method and device

Info

Publication number: CN111434553B
Application number: CN201910037187.5A
Authority: CN
Inventors: 贾思博; 冉旭; 王晋玮
Original assignee: Momenta Suzhou Technology Co Ltd
Current assignee: Momenta Suzhou Technology Co Ltd
Priority date: 2019-01-15
Filing date: 2019-01-15
Publication date: 2021-12-24
Anticipated expiration: 2039-01-15
Also published as: CN111434553A

Abstract

The embodiment of the invention discloses a vehicle braking system, a vehicle braking method, a vehicle braking device, a fatigue driving model training method and a fatigue driving model training device. The braking system of the vehicle comprises an automatic emergency braking device and a driver monitoring device; when the automatic emergency braking device detects a risk area, sending a detection result of the risk area to the driver monitoring device; when receiving a detection result sent by the automatic emergency braking device, the driver monitoring device identifies an image containing a face area of the driver based on a preset fatigue driving detection model to determine the mental state of the driver, judges the reaction state of the driver according to the mental state and sends the judgment result to the automatic emergency braking device; the automatic emergency braking device is configured to receive the judgment result and adjust the braking strategy according to the judgment result. By adopting the technical scheme, the accuracy and the safety of triggering the automatic emergency braking device are improved.

Description

Brake system, method and device, and fatigue driving model training method and device

Technical Field

The invention relates to the technical field of automatic driving, in particular to a vehicle braking system, a vehicle braking method, a vehicle braking device, a fatigue driving model training method and a fatigue driving model training device.

Background

With the development of the times, artificial intelligence is gradually widely applied in various fields, and therefore emerging concepts such as 'automatic driving' and 'unmanned vehicle' are derived. Road scene analysis and environment identification are difficult problems which need to be overcome in the field of automatic driving, and whether correct and rapid reaction and processing can be carried out aiming at a real-time road environment is a key factor for ensuring the safety of automatic driving. Therefore, one of the important points of research in the field of automatic driving is the active safety technology of automobiles.

AEB (automatic Emergency Braking) is an active safety technology for automobiles. AEB is one of the active safety technologies that have been widely used in medium and high end vehicles, and it can automatically perform emergency braking to avoid collision or reduce collision damage when the vehicle is about to collide with a front object and the driver does not react accordingly. The AEB system has the simplest principle that the distance between the AEB system and a front vehicle or an obstacle is measured by adopting a radar, then the measured distance is compared with an alarm distance and respective safety distances by utilizing a data analysis module, an alarm prompt is carried out when the measured distance is smaller than the alarm distance, and the AEB system is started even if a driver does not have to step on a brake pedal when the measured distance is smaller than the safety distance, so that the vehicle is automatically braked, and the safe trip is guaranteed.

However, AEB technology also has significant drawbacks: because it triggers very strong braking, unnecessary emergency braking occurs once the vehicle has false detection of the front target, which not only seriously affects the driving experience, but also may cause serious accidents such as driver injury, rear-end collision and the like.

To reduce false triggers, existing AEB systems set very strict trigger conditions and exit logic. If the front collision barrier meets certain conditions of position and speed to trigger AEB, the driver must continuously step on the accelerator pedal to trigger AEB, and the like. This has resulted in many AEB systems not being able to react to all dangerous situations.

Disclosure of Invention

The embodiment of the invention discloses a vehicle braking system, a vehicle braking method, a vehicle braking device, a fatigue driving model training method and a fatigue driving model training device, and the accuracy and the safety of triggering of an AEB system are improved.

In a first aspect, the embodiment of the invention discloses a braking system of a vehicle, which comprises an automatic emergency braking device and a driver monitoring device; wherein the content of the first and second substances,

the automatic emergency braking device is configured to send a detection result of a risk area to the driver monitoring device when the risk area is detected, wherein the risk area is an area with an obstacle when a set distance is met with a vehicle;

the driver monitoring device is configured to identify an image containing a face area of a driver based on a preset fatigue driving detection model, determine the mental state of the driver, judge the reaction state of the driver according to the mental state and send the judgment result to the automatic emergency braking device when receiving the detection result of the risk area sent by the automatic emergency braking device;

and the automatic emergency braking device is configured to receive the judgment result and adjust the braking strategy according to the judgment result.

As an optional implementation manner, in the first aspect of the embodiment of the present invention, when receiving the detection result of the risk existence area sent by the automatic emergency braking device, the driver monitoring device is specifically configured to:

based on a preset fatigue driving detection model, carrying out key point positioning on a face region, and determining a target region comprising eyes and a mouth;

identifying the eye state of the driver based on the fatigue driving detection model so as to judge whether the driver is in the fatigue state;

and/or the presence of a gas in the gas,

and identifying the mouth state of the driver based on the fatigue driving detection model to judge whether the driver is in a distraction state.

As an optional implementation manner, in the first aspect of the embodiment of the present invention, the driver monitoring apparatus is specifically configured to:

detecting the posture of the eyeball of the driver and the position change of the eyeball of the driver relative to the eye socket, and determining the sight watching direction of the driver;

and judging whether the sight line watching direction is a risk area direction or not, and determining the reaction state of the driver according to the judgment result.

As an alternative implementation, in the first aspect of the embodiment of the present invention, the automatic emergency braking device includes a confidence interval adjusting unit, and/or an intervention time adjusting unit, wherein,

the confidence interval adjusting unit is configured to adjust a confidence interval for obstacle detection according to the determination result;

the intervention time adjusting unit is configured to adjust an intervention time of an intervention braking process according to the determination result.

As an optional implementation manner, in the first aspect of the embodiment of the present invention, the confidence interval adjusting unit is specifically configured to:

if the judgment result is that the driver is in a high-reaction state, the confidence interval for the obstacle detection is increased; alternatively, the first and second electrodes may be,

if the evaluation result is that the driver is in a low reaction state, reducing the confidence interval of the obstacle detection; alternatively, the first and second electrodes may be,

and if the evaluation result is that the driver is in the middle reaction state, adjusting the confidence interval of the obstacle detection to be within the preset confidence interval range.

As an optional implementation manner, in the first aspect of the embodiment of the present invention, the intervention time adjusting unit is specifically configured to:

if the driver is in a high-reaction state as a result of the evaluation, intervening in a braking process of the vehicle when the remaining time TTC of the collision with the obstacle is detected to reach a first time threshold; alternatively, the first and second electrodes may be,

if the driver is in the low reaction state, intervening in the braking process of the vehicle before the remaining time TTC of the collision with the obstacle reaches a second time threshold value;

wherein the second time threshold is greater than the first time threshold.

In a second aspect, an embodiment of the present invention discloses a braking method for a vehicle, which is applied to an automatic emergency braking device, and the method includes:

when a risk area is detected, sending a detection result of the risk area to the driver monitoring device, wherein the risk area is an area with an obstacle when a set distance from a vehicle is met;

receiving a judgment result of a driver reaction state sent by the driver monitoring device based on the detection result;

adjusting the braking strategy according to the judgment result;

the judgment result is determined by the driver monitoring device according to the mental state of the driver, the mental state of the driver comprises a fatigue state and/or a distraction state, and the mental state is obtained by identifying an image containing a face area of the driver by the driver monitoring device based on a preset fatigue driving detection model.

As an optional implementation manner, in the second aspect of the embodiment of the present invention, the adjusting the braking strategy according to the determination result includes:

adjusting a confidence interval for obstacle detection based on the determination, and/or,

and adjusting the intervention time in the intervention braking process according to the judgment result.

As an optional implementation manner, in the second aspect of the embodiment of the present invention, the adjusting the confidence interval for obstacle detection according to the determination result includes:

if the judgment result is that the driver is in a high-reaction state, increasing the confidence interval of the obstacle detection; alternatively, the first and second electrodes may be,

As an optional implementation manner, in the second aspect of the embodiment of the present invention, adjusting the intervention time of the intervention braking process according to the determination result includes:

if the driver is in a high-reaction state as a result of the evaluation, intervening in a braking process of the vehicle when the remaining time TTC of the collision with the obstacle is detected to reach a first time threshold;

alternatively, the first and second electrodes may be,

wherein the second time threshold is greater than the first time threshold.

As an alternative implementation, in the second aspect of the embodiment of the present invention,

the high reaction state is as follows: the driver is not in a distraction state or a fatigue state, and if a risk area exists, the sight line direction of the driver is in a state of watching the direction of the risk area;

the low reaction state is as follows: the driver is in a distraction state and/or in a fatigue state;

the medium reaction state is as follows: the driver is not in a distracted state or in a tired state, but the driver is not looking at the risk area.

In a third aspect, an embodiment of the present invention further discloses an automatic emergency braking device for a vehicle, including:

the risk monitoring device comprises a risk monitoring result sending module, a risk monitoring result sending module and a risk monitoring result sending module, wherein the risk monitoring result sending module is used for sending a detection result of a risk area to the driver monitoring device when the risk area is detected, and the risk area is an area with an obstacle when a set distance is met with a vehicle;

the reaction state receiving module is used for receiving a judgment result of the reaction state of the driver, which is sent by the driver monitoring device based on the detection result;

the braking strategy adjusting module is used for adjusting the braking strategy according to the judgment result;

As an optional implementation manner, in a third aspect of the embodiment of the present invention, the braking strategy adjustment module includes:

a confidence interval adjusting unit for adjusting a confidence interval for obstacle detection according to the judgment result, and/or,

and the intervention time adjusting unit is used for adjusting the intervention time in the intervention braking process according to the judgment result.

As an optional implementation manner, in the third aspect of the embodiment of the present invention, the confidence interval adjusting unit is specifically configured to:

As an optional implementation manner, in the third aspect of the embodiment of the present invention, the intervention time adjusting unit is specifically configured to:

alternatively, the first and second electrodes may be,

if the driver is in a low reaction state, intervening in the braking process of the vehicle when the remaining time TTC of the collision with the obstacle is detected to reach a second time threshold;

wherein the second time threshold is less than the first time threshold.

As an alternative implementation, in a third aspect of an embodiment of the present invention,

the high reaction state is as follows: the driver is not in a distraction state or in a fatigue state, and if a risk area is detected, the sight line direction of the driver is in a state of being in the direction of the risk area;

the low reaction state is that the driver is in a distraction state and/or in a fatigue state;

the intermediate reaction state is a state that the driver is in a distraction state and a fatigue state, and the sight of the driver does not gaze at the risk area.

In a fourth aspect, an embodiment of the present invention further provides a training method for a fatigue driving model, including:

acquiring an image containing a face area of a driver;

carrying out key point positioning on the face region, and determining a target region containing the key points, wherein the target region at least comprises eyes and a mouth;

generating a training sample set based on different eye and mouth features in the plurality of images and mental state categories of drivers corresponding to the different eye and mouth features respectively;

training an initial deep regression network model based on the training sample set to obtain a fatigue driving detection model, wherein the fatigue driving detection model enables eye features and mouth features in each image in the training sample set to be associated with the mental state category of the driver corresponding to the image.

In a fifth aspect, an embodiment of the present invention further provides a training apparatus for a fatigue driving model, including:

the face image acquisition module is configured to acquire an image containing a face area of a driver;

a target region determining module configured to perform key point positioning on the face region, and determine a target region including the key points, wherein the target region at least includes an eye and a mouth;

the training sample set generating module is configured to generate a training sample set based on different eye features and mouth features in the plurality of images and mental state categories of drivers corresponding to the different eye features and mouth features respectively;

and the model training module is configured to train an initial deep regression network model based on the training sample set to obtain a fatigue driving detection model, and the fatigue driving detection model enables the eye features and the mouth features in each image in the training sample set to be associated with the mental state category of the driver corresponding to the image.

In a sixth aspect, embodiments of the present invention further provide a computer-readable storage medium storing a computer program including instructions for executing some or all of the steps of the braking method for a vehicle applied to an automatic emergency braking device provided in any of the embodiments of the present invention.

In a seventh aspect, the present invention further provides a computer-readable storage medium storing a computer program including instructions for executing part or all of the steps of the training method for a fatigue driving model provided in any embodiment of the present invention. In an eighth aspect, embodiments of the present invention further provide a computer program product, which when run on a computer, causes the computer to execute some or all of the steps of the braking method for a vehicle applied to an automatic emergency braking device provided in any of the embodiments of the present invention.

In a ninth aspect, the embodiments of the present invention further provide a computer program product, which when run on a computer, causes the computer to execute part or all of the steps of the training method for a fatigue driving model provided in any embodiment of the present invention.

In the prior art, the automatic emergency braking device can only obtain the knowledge of the state of the driver in a mode similar to stepping on an accelerator pedal, and the information is not accurate enough. The technical scheme of the embodiment of the invention organically combines the automatic emergency braking device and the driver monitoring device, judges the comprehensive reaction state information of the driver by using the driver monitoring device, and sends the judgment result of the reaction state to the automatic emergency braking device. The automatic emergency braking device improves the processing logic of the automatic emergency braking device according to the received judgment result so as to improve the triggering accuracy and safety of the automatic emergency braking device.

The invention of the embodiment of the invention comprises the following steps:

1. the driver monitoring device judges the comprehensive reaction state information of the driver and sends the judgment result of the reaction state to the automatic emergency braking device. The automatic emergency braking device improves the processing logic of the automatic emergency braking device according to the received judgment result so as to improve the triggering accuracy and safety of the automatic emergency braking device, and is one of the invention points of the embodiment of the invention.

2. According to the technical scheme provided by the embodiment of the invention, the driver monitoring device divides the driver into a high-reaction state, a medium-reaction state and a low-reaction state. If the driver is in a high-response state, the higher possibility is shown that the emergency situation can be effectively dealt with, and at the moment, the intervention degree can be reduced by the automatic emergency braking device through improving the confidence coefficient of the obstacle detection, improving the threshold value for triggering the automatic braking function, delaying the intervention time and the like; if the driver is in the middle reaction state, adjusting the confidence threshold value for obstacle detection to be within a preset confidence threshold value range; if the evaluation result shows that the driver is in a low-response state, the higher possibility is that the emergency situation cannot be effectively handled, so that the automatic emergency braking device can intervene in the braking process more by reducing the confidence threshold of the obstacle detection, reducing the threshold for triggering the automatic braking function, or advancing the intervention time, and the like, thereby avoiding danger.

3. In the technical scheme of the embodiment of the invention, when the driver monitoring device judges the reaction state of the driver, the reaction state can be determined according to the fatigue and/or distraction state of the driver, whether the gaze direction of the driver is the direction of a risk area can be determined by detecting the posture of the eyeball of the driver and the change of the eyeball of the driver relative to the direction of the eyepit, and the reaction state of the driver can be judged by judging whether the gaze direction is the direction of the risk area, so that the processing sensitivity and the processing accuracy of the driver monitoring device can be further improved, and the invention is one of the invention points of the embodiment of the invention.

4. In the technical scheme of the embodiment of the invention, the automatic emergency braking device can firstly send the detection result to the driver monitoring device after detecting the risk area. If the driver monitoring device judges that the driver does not watch the risk area, the automatic emergency braking device sends out a collision alarm, so that the driver is further reminded, and the frequent early warning phenomenon that the automatic emergency braking device sends out an alarm once the risk is monitored is avoided.

Drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings needed to be used in the embodiments will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art that other drawings can be obtained according to these drawings without creative efforts.

FIG. 1a is a flowchart of a training method of a fatigue driving model according to an embodiment of the present invention;

FIG. 1b is a flowchart of a fatigue driving detection model execution method based on target detection and deep learning according to an embodiment of the present invention;

FIG. 2 is a schematic flow chart of a braking method for a vehicle according to an embodiment of the present invention;

FIG. 3 is a schematic flow chart of a braking method for a vehicle according to an embodiment of the present invention;

FIG. 4 is a schematic flow chart illustrating a braking method for a vehicle according to an embodiment of the present invention;

FIG. 5 is a schematic flow chart of a braking method for a vehicle according to an embodiment of the present invention;

FIG. 6 is a block diagram of a vehicle braking system according to an embodiment of the present invention;

FIG. 7 is a flow chart illustrating an implementation of an intelligent emergency braking scheme incorporating a driver monitoring device and an automatic emergency braking device in accordance with an embodiment of the present invention;

FIG. 8 is a schematic structural diagram of an automatic emergency braking device for a vehicle according to an embodiment of the present invention;

fig. 9 is a schematic structural diagram of a driver monitoring apparatus according to an embodiment of the present invention.

Fig. 10 is a schematic structural diagram of a training device for a fatigue driving model according to an embodiment of the present invention.

Detailed Description

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

It is to be noted that the terms "comprises" and "comprising" and any variations thereof in the embodiments and drawings of the present invention are intended to cover non-exclusive inclusions. For example, a process, method, system, article, or apparatus that comprises a list of steps or elements is not limited to only those steps or elements listed, but may alternatively include other steps or elements not listed, or inherent to such process, method, article, or apparatus.

For clearly and clearly explaining the contents of the embodiments of the present invention, the following briefly introduces the implementation principle of the embodiments of the present invention: the technical scheme of the embodiment of the invention mainly utilizes a Driver Monitoring System (DMS) to determine the mental state of the Driver, thereby judging the reaction state of the Driver according to the mental state and sending the judgment result to the automatic emergency braking device. And the automatic emergency braking device adjusts the braking strategy of the automatic emergency braking device according to the received judgment result.

The embodiment of the invention mainly utilizes a fatigue driving detection model to determine the mental state of the driver, wherein the identification of the mental state of the driver is the basis. It can be appreciated that before using the polarity of the fatigue driving detection model, the initial deep regression network model needs to be trained using a training sample set to obtain the fatigue driving detection model. For a specific training process, please refer to the content of fig. 1a below.

Example one

Referring to fig. 1a, fig. 1a is a flowchart of a training method of a fatigue driving model according to an embodiment of the present invention, where the method can be applied to the field of automatic driving, and can be generally integrated into a training device of the fatigue driving model, and the training device can be implemented by software and/or hardware. As shown in fig. 1a, the method comprises:

102. an image containing a face region of a driver is acquired.

In this embodiment, the source of the image including the face area of the driver may be an image in the public data set, or an image of the driver collected by a camera of the vehicle may be acquired from a storage device of the vehicle. In some cases, the sample image may also be directly obtained, for example, an image acquired by a camera of a vehicle in real time is directly obtained, after the face is detected and segmented, the face image is labeled, and the labeled image is used as the sample image, that is, a training sample set.

It should be noted that, after the face region is monitored in step 102, an image with a small size and containing only a face can be obtained by means of cutting, which is beneficial to reducing the engineering quantity of feature extraction and network training and improving the accuracy of key point positioning.

104. And carrying out key point positioning on the face region, and determining a target region containing key points, wherein the target region at least comprises eyes and a mouth.

For example, the face region may be subjected to keypoint localization using face keypoint detection techniques. The face key point detection is also called face key point positioning or face alignment, and refers to positioning key region positions of a face, including eyebrows, eyes, a nose, a mouth, a face contour and the like, of a given face image. Currently, representative face key point detection methods include methods for manually designing features such as ASM (Active Shape Model), AAM (Active Appearance Model), CPR (Cascaded Shape regression), and methods for adjusting Convolutional Neural Networks (DAN) (Deep Convolutional Networks) and (Deep Convolutional Networks) using Deep learning, such as DCNN (Deep Convolutional Neural Network), TCDCN (Tasks-Constrained Deep Convolutional Networks), mtcn (Multi-task Cascaded Convolutional Neural Networks), TCNN (threaded Convolutional Networks), DAN (Deep Convolutional Networks), and (Deep Alignment Networks). Therefore, according to the method and the device, a proper method can be selected to realize the positioning of the key points of the human face according to factors such as the actual algorithm running speed and efficiency.

106. And generating a training sample set based on the mental state categories of the drivers corresponding to different eye features and mouth features and different eye features and mouth features in the plurality of images.

In the present embodiment, a supervised training mode is preferably used. Thus, the images in the training sample set can be classified into the following categories:

1. closed-eye (squint) images while the driver is driving; 2. an image of the eyes of the driver being open when normal; 3. an image of the driver's mouth opening (yawning) while driving; 4. the driver normally drives an image of the mouth closing. In this embodiment, the source of the face keypoint sample image may be an image in the public data set, or an image of a driver collected by a camera of the vehicle may be acquired from a storage device of the vehicle. In some cases, the sample image may also be directly obtained, for example, an image acquired by a camera of a vehicle in real time is directly obtained, after the face is detected and segmented, the face image is labeled, and the labeled image is used as the sample image, that is, a training sample set.

108. And training the initial deep regression network model based on the training sample set to obtain a fatigue driving detection model, wherein the fatigue driving detection model enables the eye characteristics and the mouth characteristics in each image in the training sample set to be associated with the mental state category of the driver corresponding to the image.

After the training sample set is obtained, the sample images can be input into a pre-established initial depth regression network model, so that the initial depth regression network model is trained by using the face key point sample images to obtain a fatigue driving detection model, and the eye features and the mouth features in each image in the training sample set are associated with the mental state category of the driver corresponding to the image by the fatigue driving detection model.

In the implementation example of the present application, a deep convolutional neural network can be adopted as the initial deep regression network model. In addition to designing a new deep convolutional neural network by self, a transfer learning method can also be adopted, the existing deep convolutional neural network which obtains a better result in the field of face detection is utilized, the output category number and the structures of other parts which possibly need to be modified are correspondingly modified, the existing fully trained parameters in the original network model are directly adopted as an initial deep convolutional network model, and a fine tuning method is adopted to train the neural network by utilizing a face sample image.

By processing the key point images of the face of the driver by using the deep regression network model, the state of the driver can be preliminarily classified: whether closed-eye napping or open-mouth napping and the like exist; meanwhile, whether the driver blinks slowly or the eyeballs are still can be judged in a mode of continuously collecting facial images of the driver within a plurality of adjacent frames and positioning the positions of the eyeballs of the driver; secondly, whether the face of the driver is inclined to one side or not and whether the driver carries out a phone call or not can be judged by comparing the key features of the face, because the feature extraction result of the driver is different from the feature extraction result when the driver drives normally (the face faces the middle) at the moment.

In conclusion, when the face of the driver is recognized to face to one side, yawning or calling and the like, the driver is determined to be in the distraction state; or if the states of eye closure, slow blinking or long-time stay of sight of the driver and the like are recognized, the driver is judged to be in a fatigue state.

After the training of the fatigue driving model is completed, the mental state of the driver during driving can be identified by using the model. The specific identification process is as follows:

referring to fig. 1b, fig. 1b is a flowchart of a method for executing a fatigue driving detection model based on object detection and deep learning according to an embodiment of the present invention, where the method may be applied to the field of automatic driving and may be generally integrated in a driver monitoring device, as shown in fig. 1b, the method includes:

110. and detecting a human face area by acquiring a driver monitoring picture acquired by an image sensor.

Currently, the commonly used Face detection algorithms are classified into two types, namely, a traditional feature extraction algorithm and a deep learning algorithm, the former includes a HOG (Histogram of Oriented gradient) algorithm, an Adaboost (iterative) algorithm and the like, and the latter represents a network model, such as a Face R-CNN (Region-CNN), an SSH (single stage Face detection algorithm), a pyramid (pyramid network) and the like. In the present embodiment, it is considered that an image including the face of the driver is segmented using a deep learning network model based on U-Net (U-network). Adopting a supervised training mode, taking the sample image of the marked face area as a training set, and training and fine-tuning the network; then, a driver monitoring picture collected by the image sensor is used as input, and the output of the neural network is utilized to mark the area image containing the human face, so that the aim of human face detection is fulfilled.

In addition, in some possible implementation manners of the embodiment of the application, other deep learning methods can be used to implement face detection, so as to improve the accuracy of detection.

120. Determining a target region containing key points by using a face key point detection technology, wherein the target region at least comprises eyes and a mouth.

The face key point detection is also called face key point positioning or face alignment, and refers to positioning key region positions of a face, including eyebrows, eyes, a nose, a mouth, a face contour and the like, of a given face image. Currently, representative face key point detection methods include methods for manually designing features such as ASM (Active Shape Model), AAM (Active Appearance Model), CPR (Cascaded Shape regression), and methods for adjusting Convolutional Neural Networks (DAN) (Deep Convolutional Networks) and (Deep Convolutional Networks) using Deep learning, such as DCNN (Deep Convolutional Neural Network), TCDCN (Tasks-Constrained Deep Convolutional Networks), mtcn (Multi-task Cascaded Convolutional Neural Networks), TCNN (threaded Convolutional Networks), DAN (Deep Convolutional Networks), and (Deep Alignment Networks). Therefore, according to the method and the device, a proper method can be selected to realize the positioning of the key points of the human face according to factors such as the actual algorithm running speed and efficiency.

130. Identifying the eye state of the driver based on a preset fatigue driving detection model so as to judge whether the driver is in a fatigue state; and/or identifying the mouth state of the driver based on a preset fatigue driving detection model to judge whether the driver is in the distraction state.

The distraction state is defined as the driver's occurrence of a number of special actions, such as face facing to one side, yawning or calling, etc. The distraction state is completed by monitoring the facial expression, the action and the like of the driver, and if one or more states occur, the distraction state is judged. In the embodiment, a target area of a plurality of continuous adjacent frame number images is identified by using a volume based on a preset fatigue driving detection model, and if the driver is in a calling state or a yawning state, the driver is determined to be in a distraction state.

In this embodiment, the fatigue state is mainly determined by monitoring the eye state of the driver, and if the target area of a plurality of consecutive images with adjacent frames is identified by using the convolutional neural network, and the eyeball position of the driver is determined to determine that the driver is in a slow blinking state or a state in which the eyeballs are still, the driver is determined to be in the fatigue state.

As can be seen from the above, the present embodiment provides a fatigue driving detection model based on object detection and deep learning. The real-time mental state of the driver can be accurately judged by methods such as face detection, key point positioning, deep learning based on a CNN convolutional network and the like. The algorithm described above may be applied during braking of a vehicle. And sending the judgment result of the reaction state of the driver obtained by the algorithm to an automatic emergency braking device, wherein the judgment result can be used as the input of the automatic emergency braking device and is used for indicating the automatic emergency braking device to adjust the braking strategy.

Next, a specific implementation of a vehicle braking method based on the fatigue driving detection model of the above object detection and deep learning provided in the embodiment of the present invention will be described.

Example two

Referring to fig. 2, fig. 2 is a schematic flowchart illustrating a braking method for a vehicle according to an embodiment of the present invention. The method is applied to automatic driving, can be executed by a driver monitoring device, can be realized in a software and/or hardware mode, and can be generally applied to a braking system of a vehicle such as a vehicle-mounted Computer and a vehicle-mounted Industrial control Computer (IPC), and the embodiment of the invention is not limited. As shown in fig. 2, the braking method for a vehicle provided in this embodiment specifically includes:

210. and identifying the image containing the face area of the driver based on a preset fatigue driving detection model, and determining the mental state of the driver.

The mental state of the driver includes a fatigue state, a distraction state, a non-fatigue state, a non-distraction state and the like. The above-described mental state can be determined by recognizing an image of the face area of the driver. In this embodiment, the driver monitoring device may utilize the image sensor to capture the driver in real time, and then transmit the monitoring image to the processor of the driver monitoring device as the input of the fatigue driving monitoring model. The basic implementation process of the fatigue driving monitoring model is as follows:

firstly, face detection can be carried out, a face area is defined, and whether size normalization is carried out or not is judged according to requirements; then inputting the face region image into a face key point detection model, wherein the model can extract the features of the image to be processed, map the extracted features and finally mark key points such as human eyes, lips, ears and the like; and finally, classifying the key point images by using a depth regression network model, and judging whether fatigue or distraction behaviors such as eye closure, yawning, call making and the like exist in the driver. Specifically, the working principle and the specific implementation manner of each step of the fatigue driving monitoring model can be referred to above, and this embodiment is not described again. In this embodiment, by using the fatigue monitoring model, it can be accurately determined whether the driver is in a distracted or fatigue state.

220. And judging the reaction state of the driver according to the mental state, sending the judgment result to the automatic emergency braking device, and taking the judgment result as the input of the automatic emergency braking device for indicating the automatic emergency braking device to adjust the braking strategy.

The reaction state of the driver can be divided into three levels: a high reaction state, a medium reaction state and a low reaction state. Of course, other quantitative means may be employed to determine the result of the ranking of the mental states. In this example, a high reaction state, a medium reaction state, and a low reaction state are described as examples.

For example, the determination of the reaction state of the driver according to the mental state may specifically include, but is not limited to, the following embodiments:

(1) the reaction state of the driver is determined according to the fatigue state and/or the distraction state.

And if the driver is not in the distraction state or the fatigue state, determining that the reaction state of the driver is a high reaction state. In this state, the driving process is safer; if the driver is in a distraction state or in a fatigue state, or in both a distraction state and a fatigue state, it is determined that the driver is in a low-reaction state. In this state, the driver has no time to react to a possibly dangerous situation and the driving process is risky.

Further, as an optional implementation manner, when determining the reaction state of the driver, the determination may be performed in combination with the gaze direction of the driver, that is:

(2) and determining the reaction state of the driver according to the fatigue state, the distraction state and the sight line direction of the driver.

The method can specifically be as follows: detecting the posture of the eyeball of the driver and the position change of the eyeball of the driver relative to the eye socket, and determining the sight watching direction of the driver; and judging whether the sight line watching direction is the direction of the risk area or not, and determining the reaction state of the driver according to the judgment result.

The risk area refers to an area where collision is possible, the automatic emergency braking device detects the area and sends a detection result to the driver monitoring device, and the driver monitoring device judges whether the driver notices the risk area.

Specifically, the driver monitoring device can shoot images of the driver in real time through an image sensor, and judge the head orientation of the driver through a target detection or deep learning method so as to determine the sight gaze direction of the driver; or, the determination result of the face orientation of the driver in the step 130 may be directly utilized to calculate and estimate the head orientation of the driver more accurately, so as to determine the gaze direction of the driver.

In this embodiment, the gaze direction of the driver can be estimated from the captured result of the posture of the eyeball of the driver and the relative orientation of the eyeball of the driver to the eye socket. Normally, if it is true that there is a risk of collision during the travel of the vehicle, the driver should concentrate his line of sight on the area of potential collision objects. The driver monitoring device can use this a priori information to determine whether the driver can react to the collision in time.

In the actual driving process, when the driver is in a fatigue state or a distraction state, if the sight line does not gaze at a risk area, the possibility of accidents or accidents is greatly increased, so that the embodiment of judging the mental state of the driver by detecting whether the sight line gaze direction of the driver is the direction of the risk area can be used as a supplement to the embodiment of judging the mental state according to whether the driver is in the fatigue state and/or the distraction state, namely, the embodiment (2) can be used as a supplement to the embodiment (1), thereby further improving the sensitivity and the accuracy of the processing of the driver monitoring device.

Specifically, on the basis of the above implementation, if it is determined that the driver is not in the distraction state or in the fatigue state and it is determined that the driver's gaze direction is the direction of the risk area, it may be further determined that the reaction state of the driver is the high reaction state. However, if the driver is determined not to be in the distraction state or the fatigue state, but the driver is determined to be in the distraction or the fatigue state when the driver is determined to be in the non-risk area direction of the sight gaze direction, the driver is determined to be in the neutral reaction state.

(3) If a collision warning from the automatic emergency braking device is detected, the reaction of the driver to the collision warning is judged, and the reaction state of the driver is determined.

It should be noted that, as another alternative embodiment, the above-mentioned embodiment (3) may issue a collision warning to remind the driver of danger when the automatic emergency braking device detects that a risk area exists, and may determine the reaction state of the driver according to the reaction of the user to the collision warning. This embodiment can be regarded as an embodiment in which the reaction state is determined according to the state of mind of the driver, that is, an embodiment in which the embodiment (1) is arranged in parallel, and the execution order of the two embodiments is not sequential, and may be performed simultaneously or sequentially.

It should be noted that, the above-mentioned embodiment (3) may also be implemented as an embodiment in parallel with the embodiment (2), and both may be implemented simultaneously, that is, when the automatic emergency braking device detects that there is a risk area, a collision alarm may be immediately issued to remind the driver of the danger, and the reaction state of the driver may be determined according to the reaction of the user to the collision alarm. Meanwhile, the reaction state of the driver can also be determined according to the fatigue state, the distraction state, and the line-of-sight direction of the driver in embodiment (2).

Preferably, the embodiment (3) may be further modified, and specifically, the modified embodiment may include:

when the detection result of the risk-existing region sent by the automatic emergency braking device is received, it may be determined whether the driver is in distraction or in fatigue, and if the driver is in distraction or in fatigue, or both states exist, that is, the driver is in a low-reaction state, an alarm may be issued to remind the driver, so as to further determine the reaction state of the driver, and the embodiment (3) to be improved is executed after the embodiment (2). The reason for setting like this is that if the driver is in the high reaction state, then need not to send out the warning driver and also can notice the risk area automatically to avoid in case monitoring the risk then send out the warning and the condition of the frequent early warning that leads to takes place, help promoting driver's driving experience.

Specifically, when detecting the reaction of the driver to the collision alarm, the driver monitoring device can capture the action, the head orientation and the sight line direction of the driver in real time through the image sensor, and analyze the behavior of the driver by using the related algorithm mentioned in the above steps. If the driver does not generate any action, sight line change and the like within a short period of time after the alarm is given, the driver is considered not to react to the alarm, and at this time, the reaction state of the driver can be further explained as a low reaction state.

In this embodiment, the driver monitoring device determines the reaction state of the driver, and aims to: after the reaction state is sent to the automatic emergency braking device, the automatic emergency braking device can adjust the braking strategy of the automatic emergency braking device according to the evaluation result. For example, if the driver reaction state is higher, it indicates that there is a higher possibility that the emergency can be effectively dealt with. The automatic emergency braking device may then suitably reduce its intervention level, for example by increasing the confidence interval for obstacle detection, or by increasing the threshold for triggering the automatic braking function, or by delaying the intervention time, etc. For example, if the reaction state of the driver is lower, the possibility that the emergency situation cannot be effectively dealt with is higher. At this time, the automatic emergency braking device should intervene more, that is, danger can be avoided by reducing the confidence threshold of obstacle detection, or reducing the threshold for triggering the automatic braking function, or advancing the intervention time.

The embodiment provides a technical scheme for combining the driver monitoring device with the automatic emergency braking device, and information such as attention, fatigue condition and dangerous actions of a driver can be effectively acquired by using sensors such as a camera in the driver monitoring device. And the reaction level of the driver is determined according to the information, so that the automatic emergency braking device can be indicated according to the reaction level to improve the processing logic of the automatic emergency braking device, and the triggering accuracy and safety of the automatic emergency braking device are improved.

EXAMPLE III

Referring to fig. 3, fig. 3 is a schematic flowchart of a braking method for a vehicle according to an embodiment of the present invention. The method is applied to automatic driving, can be executed by an automatic emergency braking device, can be realized in a software and/or hardware mode, and can be applied to a braking system of a vehicle such as a vehicle-mounted Computer and an Industrial Personal Computer (IPC), and the embodiment of the invention is not limited. As shown in fig. 3, the braking method for a vehicle provided in this embodiment specifically includes:

310. and receiving a judgment result of the reaction state of the driver, which is sent by the driver monitoring device.

The judgment result of the reaction state of the driver sent by the driver monitoring device is determined by the driver monitoring device according to the mental state of the driver, the mental state of the driver is obtained by identifying the image containing the face area of the driver by the driver monitoring device based on a preset fatigue driving detection model, and the mental state comprises the fatigue state or the distraction state or the two states. In this embodiment, the driver may be photographed in real time by an image sensor in the driver monitoring device, and the image may be processed as an input of the fatigue driving monitoring model. The working principle of the fatigue driving monitoring model can be referred to above, and this embodiment is not described again. In this embodiment, the driver monitoring device image sensor is combined with the fatigue monitoring model, so that whether the driver is in a distraction or fatigue state can be accurately judged.

320. And adjusting the braking strategy according to the judgment result.

For example, in the present embodiment, the braking strategy includes, but is not limited to, a confidence interval for obstacle detection by the automatic emergency braking device. In the traditional automatic emergency braking device, a confidence interval is always an invariant, and the influence of the state of a driver on risk judgment is not considered, so that the comfort of driving experience is sacrificed, and the safety of travel is guaranteed; or in order to guarantee the driving experience, the confidence interval is increased, and higher risks are brought to the driving of the vehicle. In the embodiment of the application, the confidence interval can be fed back and adjusted in real time according to the judgment result of the reaction state of the driver.

For example, in the present embodiment, the braking strategy further includes, but is not limited to, Time To Collision (TTC) of the automatic emergency braking device intervening in the braking process, where TTC refers to the remaining time of collision with the obstacle if the vehicle continues the current motion state.

In this embodiment, if the driver reaction state is higher, it indicates that there is a higher possibility that the emergency can be effectively dealt with. In this case, the intervention degree of the automatic emergency braking device may be appropriately reduced, for example, by increasing the confidence of the obstacle detection, increasing the threshold value for triggering the automatic braking function, or delaying the intervention time. On the contrary, if the reaction state of the driver is lower, the possibility that the emergency situation cannot be effectively dealt with is higher. At this time, the automatic emergency braking device should intervene more, that is, danger can be avoided by reducing the confidence threshold of obstacle detection, or reducing the threshold for triggering the automatic braking function, or advancing the intervention time.

The embodiment provides a technical scheme for combining the driver monitoring device with the automatic emergency braking device, and information such as attention, fatigue condition and dangerous actions of a driver can be effectively acquired by using sensors such as a camera in the driver monitoring device system. Based on this information, the driver monitoring device can determine the reaction state of the driver and send it to the automatic emergency braking device. The automatic emergency braking device can improve the processing logic of the automatic emergency braking device according to the reaction state, for example, the confidence threshold value of the automatic emergency braking device for obstacle detection can be adjusted according to the reaction state, or the intervention time can be adjusted, so that the triggering accuracy and safety of the automatic emergency braking device are improved.

Example four

Referring to fig. 4, fig. 4 is a schematic flow chart illustrating a braking method for a vehicle according to an embodiment of the present invention, where the braking method is applied to automatic driving and can be executed by an automatic emergency braking device. The embodiment is optimized on the basis of the embodiment, and different confidence threshold adjustment modes are provided for different reaction states of the driver. As shown in fig. 4, the braking method for a vehicle provided in this embodiment specifically includes:

400. when the risk area is detected, a detection result of the existence of the risk area is sent to the driver monitoring device.

Wherein the risk area is an area where an obstacle exists when a set distance from the vehicle is satisfied.

410. And receiving a judgment result of the reaction state of the driver, which is sent by the driver monitoring device based on the detection result.

420. And if the judgment result is that the driver is in a high reaction state, increasing the confidence interval for the obstacle detection.

In this embodiment, if the driver is in the high-response state, it indicates that there is a high possibility that the driver can deal with the potential driving danger smoothly, and therefore, the confidence interval should be increased, for example, the confidence interval may be increased to 90% -100% or more, so as to avoid the driver monitoring device from being triggered by mistake and generating unnecessary risks.

430. If the driver is in a low reaction state, the confidence threshold for the obstacle detection is adjusted to be low.

In this embodiment, if the driver is in the low-response state, it indicates that the driver has a relatively low possibility to correctly judge and handle the potential risk, and therefore, the confidence interval should be narrowed, for example, the confidence may be decreased to below 50%, specifically, to a range of 30% -50%, so that the automatic emergency braking device can be triggered more easily.

440. And if the evaluation result is that the driver is in the middle reaction state, adjusting the confidence threshold value for the obstacle detection to be within the preset confidence threshold value range.

Wherein, the preset confidence interval can be set as an empirical value of 50% -70%.

It should be noted that, the

above steps

420, 430, and 440 are different braking strategy adjustment manners corresponding to different reaction states of the driver, and the three steps are parallel steps, and there is no difference in execution order.

In the technical scheme of the embodiment, the variable confidence interval adjusting system fully considers the subjective factor of the driving state of the driver and objective factors such as the distance between the vehicle and an obstacle, the driving speed and the like, so that the triggering of the automatic emergency braking device is more intelligent.

EXAMPLE five

Referring to fig. 5, fig. 5 is a schematic flow chart illustrating a braking method for a vehicle according to an embodiment of the present invention, where the braking method is applied to automatic driving and can be executed by an automatic emergency braking device. The embodiment is optimized on the basis of the above embodiment, and different intervention times are given for different reaction states of the driver. As shown in fig. 5, the braking method for a vehicle provided in this embodiment specifically includes:

500. when the risk area is detected, a detection result of the existence of the risk area is sent to the driver monitoring device.

510. And receiving a judgment result of the reaction state of the driver, which is sent by the driver monitoring device based on the detection result.

520. If the driver is in a high-reaction state as a result of the evaluation, a braking process of the vehicle is initiated when the remaining time for detecting a collision with an obstacle reaches a first time threshold.

In this embodiment, if the driver is in a high-response state, which indicates that the driver has a high possibility to smoothly handle the potential driving danger, the automatic emergency braking device may intervene in the automatic emergency braking device again in a shorter time at TTC, that is, in a time closer to the obstacle, thereby preventing the automatic emergency braking device from being triggered by mistake. Wherein the first time threshold is an empirical value indicating the latest time to initiate intervention braking.

530. If the driver is in the low-reaction state as a result of the evaluation, the braking process of the vehicle is initiated before the remaining time for detecting a collision with an obstacle reaches a second time threshold.

In this embodiment, if the driver is in a low-response state, which indicates that the driver has a lower possibility of correctly judging and handling the potential risk, in order to ensure the safety to the maximum extent, the braking process should be involved when the TTC is larger, i.e. farther away from the obstacle and earlier. And the second time threshold is the latest time for intervention braking, and is greater than the first time threshold. In order to increase the driving safety, the automatic emergency braking device should intervene in the automatic process between the arrival of the second time threshold.

It should be noted that, the

above steps

520 and 530 are different braking strategy adjustment manners corresponding to different reaction states of the driver, and the three steps are parallel steps, and there is no difference in the execution order.

According to the technical scheme of the embodiment, the automatic emergency braking device ensures driving safety by adjusting the time of the automatic emergency braking device in the braking process, and avoids false triggering of the automatic emergency braking device.

It should be further noted that, for the two different braking strategy adjustment manners provided in the fifth embodiment and the fourth embodiment, when the automatic emergency braking device performs braking strategy adjustment, the two different strategy adjustment manners may be sampled simultaneously, and any one of the two different strategy adjustment manners may also be adopted by the automatic emergency braking device, which is not specifically limited in this embodiment of the present invention.

EXAMPLE six

Referring to fig. 6, fig. 6 is a block diagram of a braking system of a vehicle according to an embodiment of the present invention, where the braking system of the vehicle can be applied to the field of automatic driving. As shown in fig. 6, the brake system 600 of the vehicle includes: a driver monitoring device 610 and an automatic emergency braking device 620; wherein the content of the first and second substances,

an automatic emergency braking device 620 configured to transmit a detection result of the existence of a risk area to the driver monitoring device when the risk area is detected, wherein the risk area is an area where an obstacle exists when a set distance from the vehicle is satisfied;

the driver monitoring device 610 is configured to recognize an image including a face area of a driver based on a preset fatigue driving detection model, determine a mental state of the driver, judge a reaction state of the driver according to the mental state, and send a judgment result to the automatic emergency braking device when receiving a detection result of the risk area sent by the automatic emergency braking device, wherein the mental state includes a fatigue state and/or a distraction state;

and an automatic emergency braking device 620 configured to receive the judgment result and adjust the braking strategy according to the judgment result.

In this embodiment, the driver monitoring device and the automatic emergency braking device of the vehicle are integrated in the braking system of the same vehicle, but they may be integrated in the braking systems of two vehicles respectively, and there is a communication connection between the braking systems of the two vehicles.

Referring to fig. 7, fig. 7 is a flowchart illustrating an implementation of an intelligent emergency braking scheme combining a driver monitoring device and an automatic emergency braking device, which is applied to the field of automatic driving, according to an embodiment of the present invention, and referring to fig. 7, the method includes:

700. the automatic emergency braking device detects the risk of collision and issues a collision warning, and sends the detection result of the risk zone to the driver monitoring device.

710. The driver monitoring device receives the detection result sent by the automatic emergency braking device and judges whether the driver has a reaction to the collision alarm, if so, step 720 is executed; otherwise, step 760 is performed.

720. The driver monitoring device receives the detection result sent by the automatic emergency braking device, and judges whether the driver is in a distraction and/or fatigue state, if so (judges that the driver is in either or both of the distraction and fatigue states), step 760 is executed; otherwise (if not in distraction or fatigue), then step 730 is performed.

It should be noted that

steps

710 and 720 may be executed sequentially or simultaneously, that is, step 720 may also be executed directly after step 700.

730. Detecting the posture of the eyeball of the driver and the position change of the eyeball of the driver relative to the eyepit by the driver monitoring device, determining the sight watching direction of the driver, judging whether the sight watching direction is the direction of the risk area, and if so, executing a step 740; otherwise, go to step 750;

740. the driver monitoring device determines that the driver is in a high reaction state and proceeds to step 770.

750. The driver monitoring device determines that the driver is in a neutral reaction state and proceeds to step 780.

760. The driver monitoring device determines that the driver is in a low reaction state and proceeds to step 790.

770. The automatic emergency braking device increases the confidence interval for the obstacle detection according to the judgment result of the reaction state sent by the driver monitoring device, and/or intervenes in the braking process of the vehicle when the automatic emergency braking device detects that the remaining time TTC of the collision with the obstacle reaches a first time threshold value.

780. The automatic emergency braking device adjusts the confidence interval of the obstacle detection to be within the preset confidence interval range according to the judgment result of the reaction state sent by the driver monitoring device.

790. The automatic emergency braking device reduces the confidence interval of the system for the obstacle detection according to the judgment result of the reaction state sent by the driver monitoring device, and/or intervenes in the braking process of the vehicle before the remaining time TTC of the collision with the obstacle reaches a second time threshold.

Wherein the second time threshold is greater than the first time threshold.

According to the intelligent emergency braking scheme provided by the embodiment, the driver monitoring device and the automatic emergency braking device are combined to form an effective monitoring model, and the driving state of a driver can be comprehensively and quantitatively graded. The driver monitoring device evaluates the driver into three grades of a high reaction state, a medium reaction state, a low reaction state and the like, and the automatic emergency braking device adjusts the logic judgment of the automatic emergency braking device according to the reaction state of the driver, so that the method is different from the singleness and the harshness of the logic judgment of the traditional method, and improves the processing logic of the automatic emergency braking device.

EXAMPLE seven

Referring to fig. 8, fig. 8 is a schematic structural diagram of an automatic emergency braking device of a vehicle according to an embodiment of the present invention, and as shown in fig. 8, the automatic emergency braking device 800 of the vehicle includes: a risk monitoring result sending module 810, a reaction state receiving module 820 and a braking strategy adjusting module 830; wherein the content of the first and second substances,

a risk monitoring result transmitting module 810 configured to transmit a detection result of the presence of a risk area to the driver monitoring apparatus when the risk area is detected, wherein the risk area is an area where an obstacle exists when a set distance from the vehicle is satisfied;

a reaction state receiving module 820 configured to receive a determination result of the reaction state of the driver transmitted by the driver monitoring apparatus based on the detection result;

and a braking strategy adjusting module 830 configured to adjust the braking strategy according to the determination result.

The embodiment provides a technical scheme for combining the driver monitoring device with the automatic emergency braking device, and information such as attention, fatigue condition and dangerous actions of a driver can be effectively acquired by using sensors such as a camera in the driver monitoring device. Based on this information, the driver monitoring device can determine the reaction state of the driver and send it to the automatic emergency braking device. The automatic emergency braking device can improve the processing logic of the automatic emergency braking device according to the reaction state, for example, the confidence threshold value of the automatic emergency braking device for obstacle detection can be adjusted according to the reaction state, or the intervention time can be adjusted, so that the triggering accuracy and safety of the automatic emergency braking device are improved.

On the basis of the above embodiment, the braking strategy adjusting module 830 includes:

On the basis of the above embodiment, the confidence interval adjustment unit is specifically configured to:

On the basis of the above embodiment, the intervention time adjustment unit is specifically configured to:

alternatively, the first and second electrodes may be,

wherein the second time threshold is less than the first time threshold.

On the basis of the above example, the high reaction state is: the driver is not in a distraction state or in a fatigue state, and if a risk area is detected, the sight line direction of the driver is in a state of being in the direction of the risk area; the risk area is an area with an obstacle when a set distance is met with a vehicle;

the medium reaction state is as follows: the driver is in a distracted state and in a tired state, and the driver is not looking at the risk area.

The vehicle braking system provided by the embodiment of the invention can execute the vehicle braking method applied to the automatic emergency braking device provided by any embodiment of the invention, and has corresponding functional modules and beneficial effects of the execution method. Technical details which are not described in detail in the above embodiments can be referred to a braking method of a vehicle applied to an automatic emergency braking device provided in any embodiment of the present invention.

Embodiments of the present invention also provide a computer-readable storage medium storing a computer program including instructions for performing some or all of the steps of a braking method for a vehicle applied to an automatic emergency braking device provided by any of the embodiments of the present invention.

The embodiment of the invention discloses a computer program product, wherein when the computer program product runs on a computer, the computer is caused to execute part or all of the steps of the braking method applied to the vehicle of the automatic emergency braking device provided by any embodiment of the invention.

Example eight

Referring to fig. 9, fig. 9 is a schematic structural diagram of a driver monitoring device according to an embodiment of the present invention, and as shown in fig. 9, the driver monitoring device 900 includes: a mental state determining module 910 and a judgment result transmitting module 920; wherein the content of the first and second substances,

a mental state determination module 910, configured to identify an image including a face area of a driver based on a preset fatigue driving detection model, and determine a mental state of the driver, where the mental state includes a fatigue state and/or a distraction state;

and a judgment result sending module 920, configured to judge a reaction state of the driver according to the mental state, and send a judgment result to the automatic emergency braking device, where the judgment result is used as an input of the automatic emergency braking device and is used to instruct the automatic emergency braking device to adjust a braking strategy.

On the basis of the above embodiment, the fatigue driving detection model is constructed by:

acquiring an image containing a face area of a driver;

and training the initial deep regression network model based on the training sample set to obtain a fatigue driving detection model, wherein the fatigue driving detection model enables eye features and mouth features in each image in the training sample set to be associated with the mental state category of the driver corresponding to the image.

On the basis of the above embodiment, the apparatus further includes:

the sight line watching direction determining module is used for detecting the posture of the eyeball of the driver and the position change of the eyeball of the driver relative to the eyepit and determining the sight line watching direction of the driver;

and the wind direction area direction judging module is used for judging whether the sight line watching direction is the direction of the risk area or not and determining the reaction state of the driver according to the judgment result.

On the basis of the above embodiment, the apparatus further includes:

and the alarm reaction module is used for judging the reaction of the driver to the collision alarm if the automatic emergency braking device is detected to send out the collision alarm so as to determine the reaction state of the driver.

On the basis of the above embodiment, the mental state determining module 910 includes:

the device comprises a target area determining unit, a judging unit and a judging unit, wherein the target area determining unit is used for positioning key points of a face area based on a preset fatigue driving detection model and determining a target area containing the key points, and the target area at least comprises eyes and a mouth;

a fatigue state determination unit for identifying the eye state of the driver based on the fatigue driving detection model to determine whether the driver is in a fatigue state;

and/or the presence of a gas in the gas,

and the distraction state judging unit is used for identifying the mouth state of the driver based on the fatigue driving detection model so as to judge whether the driver is in the distraction state.

On the basis of the foregoing embodiment, the fatigue state determination unit is specifically configured to:

identifying a target area of a plurality of continuous adjacent frame number images based on the fatigue driving detection model, and determining the eyeball position of the driver so as to judge whether the driver is in a slow blinking state or a state that the eyeball is still;

and if the driver is in the state of slow blinking or the eyeball is still, determining that the driver is in the fatigue state.

On the basis of the above embodiment, the distraction state determination unit is specifically configured to:

identifying a target area of a plurality of continuous adjacent frame number images based on the fatigue driving detection model, and judging whether a driver is in a calling or yawning state;

and if the driver is in the calling or yawning state, determining that the driver is in the distraction state.

The driver monitoring device provided by the embodiment of the invention can execute the vehicle braking method applied to the driver monitoring device provided by any embodiment of the invention, and has corresponding functional modules and beneficial effects of the execution method. Technical details that are not described in detail in the above embodiments may be referred to a braking method of a vehicle applied to a driver monitoring apparatus provided in any embodiment of the present invention.

Embodiments of the present invention also provide a computer-readable storage medium storing a computer program including instructions for performing some or all of the steps of the braking method for a vehicle applied to a driver monitoring apparatus provided in any of the embodiments of the present invention.

Embodiments of the present invention also disclose a computer program product, wherein when the computer program product runs on a computer, the computer is caused to execute part or all of the steps of the braking method for a vehicle applied to a driver monitoring apparatus provided in any embodiment of the present invention.

Example ten

Referring to fig. 10, fig. 10 is a schematic structural diagram of a training device for a fatigue driving model according to an embodiment of the present invention, and as shown in fig. 10, the training device 1000 for a fatigue driving model includes: a face image acquisition module 1010, a target area determination module 1020, a training sample set generation module 1030 and a model training module 1040; wherein the content of the first and second substances,

a face image obtaining module 1010, configured to obtain an image including a face area of a driver;

a target region determining module 1020, configured to perform key point positioning on the face region, and determine a target region including the key points, where the target region at least includes an eye and a mouth;

a training sample set generating module 1030, configured to generate a training sample set based on different eye and mouth features in the multiple images and mental state categories of drivers corresponding to the different eye and mouth features respectively;

the model training module 1040 is configured to train the initial deep regression network model based on the training sample set to obtain a fatigue driving detection model, where the fatigue driving detection model associates an eye feature and a mouth feature in each image in the training sample set with a mental state category of a driver corresponding to the image.

By adopting the technical scheme, the fatigue driving detection model can be obtained, and the mental state of the driver can be accurately classified by using the model.

Embodiments of the present invention further provide a computer-readable storage medium storing a computer program, where the computer program includes some or all of the steps for executing the training method of the fatigue driving model provided in any embodiment of the present invention.

The embodiment of the invention also discloses a computer program product, wherein when the computer program product runs on a computer, the computer is enabled to execute part or all of the steps of the training method of the fatigue driving model provided by any embodiment of the invention.

In summary, the fatigue monitoring and state analysis model in the above embodiments mainly uses the convolutional neural network model as a basis, performs analysis processing on a real-time image of a driver, extracts a face region and face key points, and determines whether the driver has behaviors such as eye closure, mouth opening and yawning or not based on a face key point posture classification network trained in advance, so as to perform quantitative classification on the driving mental state of the driver. With the continuous development of machine learning, the convolutional neural network model used in the present embodiment is also continuously developed. In particular, different types of convolutional neural networks may be employed as the initial neural network based on the function of the model to be trained and the data to be processed by the model. Common convolutional Neural networks for object detection include R-CNN (Regions with convolutional Neural Network), Fast R-CNN (Fast Regions with convolutional Neural Network), R-FCN (Region-based convolutional Neural Network), YOLO (Young detection system), YOLO9000, NAS (single-shot SSD multi-box detector), Network Architecture (Neural array, Network), and Master R-CNN. In some possible implementations, the SSD may be used as an initial neural network model, and after a part of the structure is modified, the SSD is trimmed and then trained. In some possible implementations, other convolutional neural networks as mentioned above may be used, or other networks that achieve better results in this area may be used. The embodiments of the present application are not limited in any way in this respect.

Therefore, the application example provides a fatigue driving monitoring algorithm, a driver fatigue monitoring model based on the algorithm, and an intelligent emergency braking scheme established based on the two models. At present, a scheme of combining an AEB with a driver monitoring device does not exist, so that the key point and the pre-protection point of the invention are that the AEB and the driver monitoring device are organically combined and mutually assisted, and a method for improving AEB processing logic by using the driver monitoring device improves the safety and the accuracy of triggering of an automatic emergency braking device, and simultaneously ensures the safety and the comfort of driving of a driver. It should be noted that the target detection, the deep learning model, and the gesture recognition algorithm involved in the fatigue monitoring algorithm may be changed or improved as necessary according to the actual situation in the specific implementation process; in addition to using the above-mentioned fatigue monitoring algorithm, checking the gaze direction, checking the driver's feedback to the alarm, etc., the driver monitoring device may also use other effective methods to determine and analyze the driving state of the driver, and the embodiments of the present application are not limited in this respect.

In various embodiments of the present invention, it should be understood that the sequence numbers of the above-mentioned processes do not imply an inevitable order of execution, and the execution order of the processes should be determined by their functions and inherent logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

In the embodiments provided herein, it should be understood that "B corresponding to A" means that B is associated with A from which B can be determined. It should also be understood, however, that determining B from a does not mean determining B from a alone, but may also be determined from a and/or other information.

In addition, functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit.

The integrated units, if implemented as software functional units and sold or used as a stand-alone product, may be stored in a computer accessible memory. Based on such understanding, the technical solution of the present invention, which is a part of or contributes to the prior art in essence, or all or part of the technical solution, can be embodied in the form of a software product, which is stored in a memory and includes several requests for causing a computer device (which may be a personal computer, a server, a network device, or the like, and may specifically be a processor in the computer device) to execute part or all of the steps of the above-described method of each embodiment of the present invention.

It will be understood by those skilled in the art that all or part of the steps in the methods of the embodiments described above may be implemented by hardware instructions of a program, and the program may be stored in a computer-readable storage medium, where the storage medium includes Read-Only Memory (ROM), Random Access Memory (RAM), Programmable Read-Only Memory (PROM), Erasable Programmable Read-Only Memory (EPROM), One-time Programmable Read-Only Memory (OTPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Compact Disc Read-Only Memory (CD-ROM), or other Memory, such as a magnetic disk, or a combination thereof, A tape memory, or any other medium readable by a computer that can be used to carry or store data.

The driving strategy generating method and device based on the automatic driving electronic navigation map disclosed by the embodiment of the invention are described in detail, a specific example is applied in the text to explain the principle and the implementation mode of the invention, and the description of the embodiment is only used for helping to understand the method and the core idea of the invention; meanwhile, for a person skilled in the art, according to the idea of the present invention, there may be variations in the specific embodiments and the application scope, and in summary, the content of the present specification should not be construed as a limitation to the present invention.

Claims

1. A braking system for a vehicle, comprising automatic emergency braking means and driver monitoring means; wherein the content of the first and second substances,

the driver monitoring device is configured to identify an image containing a face area of a driver based on a preset fatigue driving detection model when receiving a detection result of a risk area sent by the automatic emergency braking device, determine a mental state of the driver, judge a reaction state of the driver according to the mental state, and send the judgment result to the automatic emergency braking device, wherein the mental state comprises a fatigue state and/or a distraction state;

the automatic emergency braking device is configured to receive the judgment result and adjust a braking strategy according to the judgment result;

wherein the driver monitoring device is specifically configured to: detecting the posture of the eyeball of the driver and the position change of the eyeball of the driver relative to the eye socket, and determining the sight watching direction of the driver; if the driver is determined not to be in the distraction state or the fatigue state and the sight watching direction of the driver is judged to be the direction of the risk area, determining that the reaction state of the driver is the high reaction state;

if the driver is determined not to be in the distraction state or the fatigue state and the sight gaze direction of the driver is judged to be in the direction of the non-risk area, determining that the reaction state of the driver is the middle reaction state;

if the driver is in a distraction state or in a fatigue state, or in both the distraction state and the fatigue state, determining that the driver is in a low-reaction state;

wherein the automatic emergency braking device comprises a confidence interval adjustment unit, and/or an intervention time adjustment unit, wherein,

the intervention time adjusting unit is configured to adjust an intervention time of an intervention braking process according to the judgment result;

wherein the confidence interval adjustment unit is specifically configured to: if the driver is in a high reaction state, the confidence interval is increased to 90% -100%; if the driver is in a low reaction state, the confidence interval is adjusted to be within the range of 30% -50%; if the driver is in the middle reaction state, adjusting the confidence threshold value for the obstacle detection to be in the range of 50% -70%;

the fatigue driving detection model is obtained by training an initial deep regression network model based on a training sample set; wherein the initial deep regression network model is obtained by the following method:

by adopting a transfer learning method, the number of output categories and the structures of other parts needing to be modified are modified by utilizing a deep convolutional neural network with obtained results in the face detection field, and existing training parameters in an original network model are directly adopted as parameters of an initial deep convolutional network model for training.

2. A braking system according to claim 1, characterized in that the driver monitoring device, upon receiving the detection result of the risk area sent by the automatic emergency braking device, is specifically configured to:

and/or the presence of a gas in the gas,

3. A braking system according to claim 1, characterized in that the intervention time adjustment unit is specifically configured to:

wherein the second time threshold is greater than the first time threshold.

4. A braking method of a vehicle, which is applied to an automatic emergency braking device, is characterized by comprising the following steps:

when a risk area is detected, sending a detection result of the risk area to a driver monitoring device, wherein the risk area is an area with an obstacle when a set distance is met with a vehicle;

adjusting the braking strategy according to the judgment result;

the judgment result is determined by the driver monitoring device according to the mental state of the driver, the mental state of the driver comprises a fatigue state and/or a distraction state, and the mental state is obtained by identifying an image containing a face area of the driver by the driver monitoring device based on a preset fatigue driving detection model;

wherein the adjusting the braking strategy according to the judgment result comprises: adjusting a confidence interval for obstacle detection according to the judgment result; and/or adjusting the intervention time of the intervention braking process according to the judgment result;

wherein the adjusting the confidence interval for the obstacle detection according to the judgment result comprises:

if the judgment result is that the driver is in a high reaction state, the confidence interval is increased to 90% -100%; alternatively, the first and second electrodes may be,

if the assessment result is that the driver is in a low reaction state, the confidence interval is adjusted to be within the range of 30% -50%; alternatively, the first and second electrodes may be,

if the assessment result is that the driver is in a middle reaction state, adjusting the confidence threshold value for the obstacle detection to be in the range of 50% -70%;

the fatigue driving detection model is obtained by training an initial deep regression network model based on a training sample set; the initial depth regression network model is obtained through the following method:

5. The method of claim 4, wherein said adjusting an intervention time of an intervention braking process based on said determination comprises:

alternatively, the first and second electrodes may be,

wherein the second time threshold is greater than the first time threshold.

6. An automatic emergency braking device for a vehicle, comprising:

a risk monitoring result transmitting module configured to transmit a detection result of the existence of a risk area to the driver monitoring apparatus when the risk area is detected, wherein the risk area is an area where an obstacle exists when a set distance from the vehicle is satisfied;

a reaction state receiving module configured to receive a determination result of a reaction state of the driver, which is transmitted by the driver monitoring apparatus based on the detection result;

the braking strategy adjusting module is configured to adjust a braking strategy according to the judgment result;

wherein the braking strategy adjustment module comprises:

the intervention time adjusting unit is used for adjusting the intervention time in the intervention braking process according to the judgment result;

wherein the confidence interval adjustment unit is specifically configured to:

wherein the high reaction state is: the driver is not in a distraction state or a fatigue state, and if a risk area exists, the sight line direction of the driver is in a state of watching the direction of the risk area; the risk area is an area with an obstacle when a set distance is met with a vehicle;

the medium reaction state is as follows: the driver is not in a distraction state or a fatigue state, but the sight of the driver does not focus on the risk area;

7. A training method of a fatigue driving model is characterized by comprising the following steps:

acquiring an image containing a face area of a driver;

training an initial deep regression network model based on the training sample set to obtain a fatigue driving detection model, wherein the fatigue driving detection model enables eye features and mouth features in each image in the training sample set to be associated with the mental state category of a driver corresponding to the image;

wherein the initial deep regression network model is obtained by the following method:

by adopting a transfer learning method, the number of output categories and the structures of other parts needing to be modified are modified by utilizing a deep convolutional neural network with obtained results in the face detection field, and existing training parameters in an original network model are directly adopted as parameters of an initial deep convolutional network model for training;

the trained fatigue driving detection model is used for identifying an image containing a face area of a driver and determining the mental state of the driver, so that the driver monitoring device judges the reaction state of the driver according to the mental state and sends the judgment result to the automatic emergency braking device; the automatic emergency braking device is configured to receive the judgment result and adjust a braking strategy according to the judgment result;

wherein the confidence interval adjustment unit is configured to: if the driver is in a high reaction state, the confidence interval is increased to 90% -100%; if the driver is in a low reaction state, the confidence interval is adjusted to be within the range of 30% -50%; if the driver is in the neutral reaction state, the confidence threshold for obstacle detection is adjusted to be in the range of 50% to 70%.

8. A training device for a fatigue driving model, comprising:

a model training module configured to train an initial deep regression network model based on the training sample set to obtain a fatigue driving detection model, wherein the fatigue driving detection model associates an eye feature and a mouth feature in each image in the training sample set with a mental state category of a driver corresponding to the image;