WO2020237664A1

WO2020237664A1 - Driving prompt method, driving state detection method and computing device

Info

Publication number: WO2020237664A1
Application number: PCT/CN2019/089639
Authority: WO
Inventors: 郑睿姣; 叶凌峡
Original assignee: 驭势（上海）汽车科技有限公司
Priority date: 2019-05-31
Filing date: 2019-05-31
Publication date: 2020-12-03
Also published as: CN110582437A

Abstract

Disclosed by the present application is a driving prompt method, a driving state detection method and a computing device. The driving prompt methods comprises: acquiring face image information of a detected subject; acquiring at least one piece of facial feature information of the detected subject from within the face image information; determining the driving state of the detected subject according to the facial feature information; and sending corresponding prompt information on the basis of the driving state when a predetermined condition is met. The method proposed in embodiments of the present application may solve the problem of high false alarm rates of existing driver state detection systems, and ensure that a driver is able to safely take over a vehicle within a specified time range.

Description

Driving reminding method, driving state detection method and computing device

Technical field

This application relates to the field of automatic driving technology, and in particular to a driving reminder method, a driving state detection method, and a computing device.

Background technique

Autonomous vehicles (Autonomous vehicles; Self-piloting automobiles), also known as unmanned vehicles and computer-driven vehicles, are intelligent vehicles that realize unmanned driving through a computer's automatic driving system. In the 20th century, there have been decades of history, and the beginning of the 21st century showed a trend close to practicality.

According to the degree of driver participation, autonomous driving can be divided into several levels:

Level L0: The driver has complete control of the vehicle;

Level L1: The automatic system can sometimes assist the driver to complete certain driving tasks;

L2 assisted driving: The automatic system can complete certain driving tasks, but the driver needs to monitor the driving environment, complete the rest, and ensure that problems occur and take over at any time. At this level, the wrong perception and judgment of the automatic system can be corrected by the driver at any time, and most car companies can provide this system. L2 can be divided into different usage scenarios based on speed and environment, such as low-speed traffic jams on the loop, fast driving on highways and automatic parking by the driver in the car;

L3 semi-autonomous driving: The automatic system can not only complete certain driving tasks, but also monitor the driving environment under certain conditions, but the driver must be ready to regain driving control (when the automatic system makes a request). Therefore, at this level, the driver still cannot sleep or take a deep rest. The difference between L3 and L2 is that the vehicle is responsible for surrounding monitoring, while the human driver only needs to maintain attention for emergencies.

Level 4 highly automated driving: Automated systems can complete driving tasks and monitor the driving environment in certain environments and specific conditions; currently, the deployment of L4 is mostly based on city use, which can be fully automated valet parking. It can also be done directly in conjunction with taxi services. At this stage, within the scope of autonomous driving, all tasks related to driving have nothing to do with the driver and passengers. The perception of external responsibility lies in the autonomous driving system, and there are different design and deployment ideas here;

Level 5 fully automated driving: all driving tasks that the automated system can complete under all conditions

It can be seen that the higher the level, the lower the driver’s participation; the lower the level, the higher the driver’s participation.

For the L0, L1, and L2 levels with high driver participation, the current automatic driving system has developed corresponding driver state detection methods to monitor the driver's state to ensure that the driver can concentrate on driving.

With the development of autopilot technology, autopilot systems can judge more and more complex situations, and the number of situations requiring driver participation is gradually reduced. The existing driver state detection methods are mainly applied to the autonomous driving level with higher driver participation. If you switch to the driving level with lower driver participation, it will easily cause problems such as high false alarm rate and frequent false alarms. , Affect the driver's mood and mental state.

Therefore, it is necessary to propose a driver state detection method suitable for autonomous driving levels with low driver participation.

Summary of the invention

In view of the foregoing problems, an embodiment of the present application proposes a driving reminder method, a driving state detection method, and a computing device to solve the problems in the prior art.

In order to solve the above problem, an embodiment of the present application discloses a driving reminder method, including:

Acquiring face image information of the detected object;

Acquiring at least one facial feature information of the detected object from the facial image information;

Obtaining the driving state of the detected object based on the facial feature information;

When a predetermined condition is met, corresponding reminder information is issued based on the driving state.

An embodiment of the present application also discloses a computing device, including:

One or more processors; and

One or more machine-readable media on which instructions are stored, when executed by the one or more processors, cause the computing device to execute the above-mentioned method.

An embodiment of the present application also discloses one or more machine-readable media, on which instructions are stored, which when executed by one or more processors, cause a computing device to execute the foregoing method.

It can be seen from the foregoing that the embodiments of the present application include the following advantages:

The method proposed in the embodiment of the present application can solve the problem of the high false alarm rate of the existing driver state detection system, and ensure that the driver has the ability to safely take over the vehicle within a specified time range.

Description of the drawings

In order to more clearly describe the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings that need to be used in the description of the embodiments or the prior art. Obviously, the drawings in the following description These are some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative work.

Fig. 1 is a block diagram of an automatic driving system according to an embodiment of the application.

Fig. 2 is a block diagram of visual algorithm processing according to an embodiment of the application.

Fig. 3 shows a flowchart of a driving reminding method according to an embodiment.

4A to 4D are flowcharts of sub-steps of the driving reminding method shown in FIG. 3.

Fig. 5 schematically shows a block diagram of a computing device for executing the method according to the present application.

Fig. 6 schematically shows a storage unit for holding or carrying program codes for implementing the method according to the present application.

Detailed ways

The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art fall within the protection scope of this application.

For the gradually increasing level of automatic driving, the embodiments of this application propose a driving method and device applied to an automatic driving system, which can solve the problem of high false alarm rate of the existing driver state detection system and ensure that the driver has the ability to Safely take over the vehicle within the time frame.

The embodiment of the application proposes a driving reminding method, which is applied to an automatic driving system of a vehicle. The automatic driving system can detect the information inside and outside the car, and this information can be input into the automatic driving system as the basis for the automatic driving system to judge and perform operations.

Information outside the vehicle may include traffic environment information and natural environment information; traffic environment information, for example, road condition information, traffic light information, obstacle information, etc.; natural environment information, for example, temperature, humidity, light, etc. This information can be obtained through detection elements such as sensors, cameras, and radars outside the vehicle.

In-vehicle information includes, for example, in-vehicle environment information, driver status information, and driver's operation information on the vehicle, etc. These information can be acquired through detection elements such as in-vehicle sensors and cameras.

This application mainly aims at information processing in the vehicle, and proposes a driving reminder method and device. Fig. 1 shows a system block diagram of an automatic driving system proposed in an embodiment of the application. As shown in Figure 1, the automatic driving system of a vehicle can be composed of software, hardware, or a combination of software and hardware. The automatic driving system can include a vehicle sensor module 10, a vehicle-mounted camera module 20, a driver state detection module 30, and an automatic driving system master. The control module 40, the wake-up strategy control module 50 and the human-computer interaction interface module 60. In some embodiments, the vehicle sensor module 10 and the vehicle camera module 20 may be hardware devices, which are connected to the vehicle computer through a connection such as a data bus; the driver state detection module 30, the automatic driving system main control module 40, and the wake-up strategy control The module 50 may be a computer program in a vehicle-mounted computer processor; the human-computer interaction interface module 60 may be a software module or a hardware module.

The vehicle sensor module 10 and the vehicle camera module 20 are used to collect information in the vehicle, and the vehicle sensor module 10 is used to detect whether there is a driver at the driving position through sensors. In some embodiments, the sensor module 10 may be a pressure sensor installed in the driver's seat. In some embodiments, the vehicle sensor module 10 may be used to receive related sensor signals of the vehicle, such as driving position pressure sensor signals, seat belt signals, etc., to determine whether the driver is in the driving position.

The vehicle-mounted camera module 20 is used to collect multiple frames of video images of the driving position. In some embodiments, the vehicle-mounted camera module 20 may be one or more of a normal camera, a high-definition camera, and a stereo camera. In some embodiments, the multiple frames of video images may be continuous or discontinuous. In some embodiments, the vehicle-mounted camera module 20 can be installed at the A-pillar position in the vehicle to collect image information of the driver and monitor the status of the driver. The installation position of the vehicle-mounted camera module can be used to obtain a greater degree of driver The facial information is the best and cannot affect the operation of the driver, for example, it cannot obscure the operation of the driver.

In some embodiments, the signal and video image of the sensor module may be sent to the driver state detection module 30 of the onboard computer. In some embodiments, the driver state detection module 30 may use at least one of the sensor signal and the video image to determine the driving state of the driver, and send the driving state to the automatic driving system main control module 40 . In some embodiments, the driver state detection module 30 may also send the driving state to the wake-up strategy control module 50. In some embodiments, the driver state detection module 30 may receive the image information. In some embodiments, the driver state detection module 30 performs driver detection through a face classifier to determine whether there is a human face in the detected frame of video image. When there is a human face, the driver's rectangular area can be determined, and the driver's facial feature points can be located in the rectangular area of the human face to obtain facial feature information, and the driver's state can be determined based on the facial feature information.

In some embodiments, FIG. 2 shows a visual algorithm processing block diagram of the driver state detection module 30. As shown in Figure 2, the processing flow of the driver state detection module 30 includes four parts: video image input, driver detection, facial feature point positioning, and driver state judgment. The video image input process is used to obtain the video image of the on-board camera module 20; the driver detection process is used to determine whether there is a multi-frame video image to determine whether the driver’s facial image information exists; the process of facial feature point positioning is used to determine the The image information determines the facial feature points; the driver state judgment process is used to judge the driver's driving state based on facial features.

Returning to Figure 1, the main control module 40 of the automatic driving system can be used to send system status signals according to the operating conditions of the automatic driving system, such as a signal that the system is malfunctioning, an emergency situation, or the automatic driving system cannot accurately determine the road conditions ahead. Signal etc.

The wake-up strategy control module 50 may be used to send different instructions according to the driver state and/or the state signal of the automatic driving system, and use different reminding methods to remind the driver to take over the vehicle. In some embodiments, upon receiving the system status signal sent by the automatic driving system, for example, when a vehicle failure or system failure occurs, different reminding methods can be adopted to the driver according to the driver status and confidence level to ensure The driver can take over the driving task safely and smoothly within the specified time.

In some embodiments, the wake-up strategy control module 50 formulates a wake-up strategy according to the driving state, and executes the wake-up strategy through the human-computer interaction interface module 60. In some embodiments, the driver state detection module 30 and the wake-up strategy control module 50 may be software modules in the on-board computer processor, and the human-computer interaction interface module 60 may be used to control driving according to the control instructions given by the wake-up strategy control module 50. The hardware module that sends notification information to the operator, and the control command includes different wake-up modes. According to the wake-up strategy control instruction, the human-computer interaction interface module 60 executes a corresponding wake-up mode for the driver to remind the driver to take over the driving task. The human-computer interaction interface module 60 may include, for example, a sound module, a light module, a vibration module, a display module, etc., which are not particularly limited in this application.

It is worth noting that the above description of the automatic driving system is only for convenience of description, and does not limit the present application within the scope of the listed embodiments. It can be understood that for those skilled in the art, after understanding the principle of the system, it is possible to arbitrarily combine various modules, or form subsystems to connect with other modules without departing from this principle, to implement the above Various amendments and changes in the form and details of the application field of the method and system. For example, the above-mentioned driver state detection module 30, automatic driving system main control module 40, wake-up strategy control module 50 and human-computer interaction interface module 60 are separate software modules. For another example, these modules may also be two-by-two integrated or multiple integrated modules, and any deformation or modification belongs to the protection scope of this application. For example, the driver state detection module 30 and the automatic driving system main control module 40 can be integrated together in the form of software; for example, the wake-up strategy control module 50 and the human-computer interaction interface module 60 can be integrated together in the form of software; another example The driver state detection module 30, the automatic driving system main control module 40, and the wake-up strategy control module 50 can be integrated in the form of software, or the driver state detection module 30, the automatic driving system main control module 40, and the wake-up strategy The control module 50 and the human-computer interaction interface module 60 are integrated together in the form of software as a whole, and this application does not particularly limit the implementation of the above modules alone or in combination. Such deformations are all within the protection scope of this application.

FIG. 3 is a flowchart of the steps of the driving reminding method according to the first embodiment of the application. As shown in FIG. 3, the driving reminder method of the embodiment of the present application is applied to an automatic driving system and includes the following steps:

S101: Obtain face image information of a detected object;

In some embodiments, the automatic driving system obtains facial image information of the detected object. Wherein, the automatic driving system may obtain the facial image information of the driver in the driving position of the vehicle where the automatic driving system is located through a sensor or a camera.

In an embodiment, the face image information of the detected object may include a face image recognized through face recognition technology. In one embodiment, the vehicle-mounted camera module 10 in FIG. 1 may be used to shoot video, for example, 30 frames per second of continuous video images. After that, a face classifier can be used to analyze and detect the video image to determine whether there is face image information. In an embodiment, the face classifier may be obtained by extracting MBLBP (Multiscale BlockLBP) features from a training set containing human faces and non-human faces, and then training using a cascaded AdaBoost algorithm.

In some embodiments, after the face classifier determines that there is face image information, the rectangular area of the face can be obtained through an algorithm.

In other embodiments, judging whether there is face image information in the image can also be implemented by means of machine learning. For example, a machine learning model can be used to determine whether there is face image information in the image. The application of the machine learning model includes the training phase and the use phase: in the training phase, multiple images containing face image information and images not containing face image information can be input into the machine learning model, and these pictures are marked as "containing "Or "does not contain", use these images as samples to train the machine learning model; in the use stage, input new images into the training mature machine learning model, and the machine learning model can automatically output whether the image contains face image information critical result.

In some other embodiments, judging whether there is face image information in the image can also be implemented by means of deep learning in machine learning. Deep learning uses a neural network model that contains multiple hidden layers to establish and simulate a neural network that simulates the human brain for analysis and learning, and imitates the mechanism of the human brain to interpret data, such as text, images, and sounds. Deep learning usually requires a larger amount of training data to train the neural network model. These training data are, for example, a large number of images marked with "face image information" or "not containing face image information"; in the use stage after training, Input the new image into the neural network model, the neural network model can automatically output the judgment result of whether the image contains facial image information, and the accuracy of the output information is significantly improved compared with the traditional machine learning model.

It is worth noting that the above is only an example. The method of judging whether there is face image information in the image may not be limited to the above. Those skilled in the art can make any changes to the determined method, and these transformations are included in this article. Within the scope of application.

After step S101 is performed, according to an embodiment of the present application, step S102 can be performed as follows:

S102: Obtain facial feature information and facial feature information of the detected object from the facial image information;

In this step, you can first locate and obtain facial feature points from the aforementioned rectangular area of the face, such as eyes, mouth corners, nose tip, and face contour; then use the position information of the facial feature points as facial feature information for subsequent use Determine driving status.

In some embodiments, further, the position information of the facial feature points is determined, an initial shape is given for the rectangular region of the face, the image features of the key feature points are extracted, and the initial shape is returned to Position close to or even equal to the true shape.

In some embodiments, the location information of the facial feature points can be determined by using the aforementioned Supervised Descent Method (SDM) to solve the problem, and the image feature adopts the Histogram of Oriented Gradient, HOG); The orientation gradient histogram feature is a feature description factor formed by calculating and counting the gradient orientation histogram of the local area of the image, and will not be repeated here.

In some embodiments, determining the position information of the facial feature points may also be obtained through machine learning. For example, a machine learning model can be used to obtain the location information of facial feature points. The application of the machine learning model includes the training phase and the use phase: in the training phase, multiple face images marked with the location information of the facial feature points can be input into the machine learning model to train the machine learning model; in the use phase, The new face image input trains a mature machine learning model, and the machine learning model can automatically output the location information of the facial feature points of the face image.

In some embodiments, determining the position information of the facial feature points can also be obtained by means of deep learning in machine learning. In the training phase, a large amount of training data can be used to train the neural network model. These training data are, for example, face images marked with the location information of facial feature points; in the use phase, new face images are input to the neural network model, neural network model The position information of the facial feature points of the face image can be automatically output, and the accuracy of the output information is significantly improved compared to traditional machine learning models.

It is worth noting that the above is only for example, and the method for determining the position information of the facial feature points may not be limited to the above-mentioned methods. Those skilled in the art can change the determination method arbitrarily, and these transformations are included in this article. Within the scope of application.

After step S102 is executed, according to an embodiment of the present application, step S103 can be executed as follows:

S103: Acquire the driving state of the detected object based on the facial feature information;

In this step, the driving state of the detected object can be obtained based on facial feature information. The corresponding facial feature information can be obtained through facial feature points. The facial feature information includes, for example, position information of facial feature points. The location information can be used to extract feature description factors to determine the driver's state.

For example, in one embodiment, the driver’s eye area can be located from the position information of the facial feature points; the description factor is extracted using the position information of the eye area, such as the aspect ratio of the driver’s eyes, and the SVM algorithm is used to determine the state of the eyes ——The state can include open, closed, half open, etc., for example.

Similarly, in other embodiments, it is also possible to locate the driver’s mouth area from the position information of facial feature points; use the position information of the mouth area to extract descriptive factors, such as the aspect ratio of the driver’s mouth, and use specific The algorithm determines the state of the mouth-the state can include, for example, open, closed, half-open, etc.

Similarly, in other embodiments, the head posture of the driver can be calculated by combining the internal and external parameters of the on-board camera module 20. For example, the head posture of the driver can be calculated according to the driver's current captured image, combined with the deflection angle of the camera relative to the x-y-z three-axis coordinate system. Alternatively, the angle and transformation relationship between the axis of the driver's head and the axis of the body are used to determine the driver's head posture. For example, when the preset head posture is less than a specific angle relative to the body, for example, the angle is between 0-15 degrees, the driver is considered to be in a normal posture; when the driver’s head posture is larger than the body At an angle of 15 degrees, it is considered that the driver may be sleeping.

In some embodiments, when the driving state of the driver is determined by one or more of these methods, it is used to subsequently issue corresponding reminders. In some embodiments, the automatic driving system may set one or more of the postures of the eyes, mouth, and head to correspond to the driving state of the driver. In some embodiments, when the automatic driving system determines one or more of the eyes, mouth, head, etc., the driving state of the driver can be determined.

The driving state can be divided into multiple driving state levels. Taking the use of the driver's eyes to detect the driving state as an example, three different levels of state (0/1/2) can be set, the lower the level, the more sober. For example, if any eye is closed at a certain moment, the consecutive frames with closed eyes increase once. If the number of consecutive frames with closed eyes is greater than the maximum number of consecutive closed eyes, the driver's driving state level is 2; if the consecutive number of closed eyes is between the maximum and minimum consecutive closed eyes, the driver's driving state level is 1; otherwise , The driver’s driving state level is 0.

For another example, when it is detected that the user’s head drooping exceeds the first time period, the driver’s driving state level is 2; when the user’s head drooping time is detected between the first time period and the second time period, the first time period is If the duration is longer than the second duration, the driver's driving state level is 1; otherwise, the driver's driving state level is 0.

The open and closed state of the eyes can be determined based on the SVM classifier by extracting feature information from the eye region. The feature information may include the fusion information of the LBP feature, the HU moment feature, and the histogram feature of the gray-level rotation invariant equivalent mode, and these information are used for feature description of the eye region. In some embodiments, the SVM classifier first extracts the fusion feature information of the image, and then trains the data sample set images based on the support vector machine algorithm to obtain a classifier capable of judging the open and closed state of the eyes.

After step S103 is performed, according to an embodiment of the present application, step S104 can be performed as follows:

S104: When a predetermined condition is met, send corresponding reminder information based on the driving state.

In this step, the predetermined condition is based on at least one of the driver's driving state, the current system state signal of the automatic driving system, and the accuracy of the driver's driving state judgment (for example, system confidence). One or a combination of them.

In some embodiments, the predetermined condition may be: the current system status signal of the automatic driving system is a system error or a vehicle failure; that is, as long as the system status signal contains a signal related to a system error or a vehicle failure, the wake-up shown in FIG. 1 The strategy reminding module 50 determines the reminding mode according to the driving state of the driver, and sends out corresponding reminding information through the human-computer interaction interface module 60.

In other embodiments, the predetermined condition may be: the driving state level is 1 or 2 (for example, the driver is not awake or less awake), and the current system state signal of the automatic driving system is a system error or a vehicle failure ; That is, it is necessary to simultaneously satisfy that the driver is not in a driving state and the system status signal is displayed as a system error or a vehicle failure. At this time, the driver's intervention is required to meet the predetermined conditions; when the predetermined conditions are met, the wake-up strategy control module 50 is According to the driving state of the driver, the reminding method is determined, and the corresponding reminding information is sent out through the human-computer interaction interface module 60.

In other embodiments, the system confidence level, that is, the accuracy of the system judgment, can be introduced to determine whether the predetermined conditions are met or decide to issue different levels of reminder information. The step of determining the confidence of the system may include the following steps, for example:

Count the number of consecutive detections of face image information; and

The system confidence is determined according to the number of times of the facial image information, and the system confidence includes more than two system confidence levels.

In an embodiment, the automatic driving system may use the number of consecutive detections of human faces to divide the confidence of the system result, using three levels for characterization (0/1/2), the higher the level, the higher the confidence. If the number of consecutively detected faces is greater than the maximum threshold for consecutively detected faces, the confidence level is 2; if the number of consecutively detected faces is between the maximum and minimum thresholds, the confidence level is 1; otherwise, the confidence level Is 0.

Taking the system confidence into consideration, the predetermined condition may be: the driving state level is 1 or 2 (for example, the driver is relatively unconscious or very unconscious), and the current system state signal of the automatic driving system is a system error Or vehicle failure, and the aforementioned system confidence is 1 or 2 (that is, the confidence level is high or medium); that is, it is necessary to satisfy that the driver is not in a driving state, and the system status signal shows that the driver is required to intervene, and the system confidence is high When the predetermined conditions are met, the wake-up strategy reminder module 50 of the on-board control system determines the reminder mode according to the driving state of the driver, and sends out the corresponding reminder through the human-computer interaction interface module 60. Reminder information.

The operation of issuing corresponding reminder information based on the driving state of the driver may be, for example, setting reminding methods of different intensities for different levels of driving states, for example, three reminding methods of high, middle and low intensities. Wherein, the reminding methods of different intensities can be realized by one or more of the methods such as volume, light flashing, steering wheel vibration, seat vibration, and the like. Among them, the difference between high-intensity reminder, medium-intensity reminder, and low-intensity reminder lies in the intensity or intensity of the used reminder. For example, when using volume reminder, the high-intensity reminder can use high decibel volume and high light flashing frequency. Or use high-frequency red light flashing, high-frequency steering wheel vibration or seat vibration; medium-intensity reminder can use medium decibel volume, light flashing frequency or yellow light medium-frequency flashing, medium-frequency steering wheel vibration or seat vibration; Low-intensity reminders can use low-decibel volume, low-frequency light flashing or green light flashing, low-frequency steering wheel vibration or seat vibration.

The corresponding reminder information is issued based on the driving state, and the system confidence can also be used as one of the reference factors, that is, the corresponding reminder method can be determined based on the driving state and the system confidence, and the reminder information of the corresponding level can be issued.

Table 1 shows an example of multiple reminding methods set for the driving state when the predetermined conditions are met, as follows:

驾驶状态等级Driving status level	系统置信度System confidence	提醒方式Reminder
1(不清醒)1 (not sober)	1(高)1 (high)	高强度high strength
2(较不清醒)2 (less sober)	1(高)1 (high)	中等强度Medium intensity
3(清醒)3 (awake)	2(中)2 (medium)	低强度Low intensity
2(较不清醒)2 (less sober)	3(低)3 (low)	低强度Low intensity
3(清醒)3 (awake)	3(低)3 (low)	低强度Low intensity

Table 1

It is worth noting that the above-mentioned intensity levels set by different intensity reminding methods and the specific reminding means of each intensity level are all examples, and the same reminding intensity level may correspond to different reminding means, and this application is not particularly limited. Those skilled in the art can set the reminding means by themselves within the scope of the principles disclosed in this application.

In some embodiments, steps S101 to S104 may respectively include the following sub-steps.

In an optional embodiment, further, as shown in FIG. 4A, the step S101, that is, the step of acquiring face image information of the detected object, may include the following sub-steps:

S1011: Use a face classifier to detect whether there is face image information in the collected image;

S1012: When it is determined that there is facial image information in the collected image, extract facial feature information from the facial image information;

In some embodiments, when it is determined that there is a human face in the collected image, an algorithm can be used to obtain a rectangular area of the human face. After the rectangular area of the human face is determined, at least one facial feature information can be obtained from the rectangular area of the human face.

For example, in sub-step S1011, a face classifier may be used to detect whether there is a face image. The face classifier can be obtained by extracting MBLBP features from a training set containing human faces and non-human faces, and training them using a cascaded AdaBoost algorithm.

In some embodiments, in the face detection process, the AdaBoost algorithm is used to select some rectangular features (weak classifiers) that best represent the face, and the weak classifier is constructed into a strong classifier according to the weighted voting method, and then Several strong classifiers obtained by training are connected in series to form a cascaded classifier.

The aforementioned MBLBP feature refers to the Multiscale Block LBP feature. Compared with the LBP feature, the MBLBP feature used in the embodiment of the present application is more robust and characterizes the image more completely.

In sub-step S1012, further, when it is determined that there is face image information in the collected image, the facial feature information of the face image information may be further determined.

After judging that there is face image information in sub-step S1011, the rectangular area of the face can be obtained. After obtaining the rectangular area of the human face, the facial feature points can be obtained, and the location information of the facial feature points can be determined by the positioning method as the facial feature information. That is, the facial feature information may include automatically locating the positions of facial feature points according to the rectangular area of the human face.

The facial feature information may be the positions of various parts that make up the human face, such as the eyes, the corners of the mouth, the tip of the nose, and the contour of the human face. These feature positions can be obtained through algorithm positioning. The position information of the facial feature points is determined, an initial shape is given for the rectangular region of the face, the image features of the key feature points are extracted, and the initial shape is returned to a position close to or even equal to the true shape through continuous iteration.

In an optional embodiment, the facial feature information may include eye information of the detected object; the eye feature information may include, for example, a ratio of eye height to eye width.

Therefore, further, as shown in FIG. 4B and FIG. 4C, step S102 may include the following sub-steps:

S1021: Determine facial feature points according to the facial image information;

S1022: Determine location information of the facial feature point, and use the location information as facial feature information.

Sub-step S1021, that is, the step of determining facial feature points according to the facial image information, may include:

S1021a: Obtain a rectangular area of the face according to the face image information;

S1021b: Extract the initial shape of at least one facial feature point from the rectangular area of the face;

Sub-step S1022, determining the location information of the facial feature points, and using the location information as the facial feature information may include:

S1022a, using the image features of the facial feature points to correct the initial shape based on a supervised descent algorithm;

Wherein, the image features include directional gradient histogram features.

In some embodiments, the facial feature points, such as eyes, mouth corners, nose tip, and face contour, can be obtained from the aforementioned rectangular area of the face first; and then the position information of the facial feature points can be used as facial feature information for subsequent Determine driving status. For the rectangular region of the human face, an initial shape is given, image features of key feature points are extracted, and the initial shape is returned to a position close to or even equal to the true shape through continuous iteration.

Further, determining the position information of the facial feature points may be solved by using a supervised descent algorithm (Supervised Descent Method, SDM), and the image feature uses a histogram of orientation gradient (Histogram of Oriented Gradient, HOG). The supervised descent algorithm is a method used to minimize the non-linear least squares (Non-linear Least Squares) objective function. By learning a series of descent directions and the scales of the directions, the objective function is made at a very fast speed. Convergence to the minimum value, avoiding the problem of solving Jacobian matrix and Hessian matrix. The directional gradient histogram feature is a feature description factor formed by calculating and counting the gradient directional histogram of the local area of the image. Its essence is the statistical information of the image gradient, which can maintain good geometrical and optical deformations in the image. Immutability.

As shown in FIG. 4D, the aforementioned step S103, that is, the step of judging the driving state of the detected object according to the facial feature information, may include the following sub-steps:

S1031: Extract eye feature information according to the eye information, and determine the open and closed state of the eyes;

S1032: Determine the driving state of the detected object by using the eye open and closed states of the multiple continuous images, where the driving state includes more than two driving state levels.

In sub-step S1031, feature information can be extracted according to the eye region, and then the open and closed state of the eyes can be obtained based on the SVM classifier. As mentioned above, the fusion information of the LBP feature, the HU moment feature and the histogram feature of the gray-level rotation invariant equivalent mode can be used to describe the eye area.

In sub-step S1032, the state of the driver is judged based on the open and closed state of the human eyes in consecutive frames. For example, three different levels (0/1/2) may be set, and the lower the level, the more awake. Specifically, if any eye is closed at a certain moment, the consecutive frames with closed eyes increase once. If the number of consecutive frames with closed eyes is greater than the maximum number of consecutive closed eyes, the driver's status level is 2; if the number of consecutive eyes closed is between the maximum and minimum consecutive eye closures, the driver's status level is 1; otherwise, driving The status level of the member is 0.

In an embodiment, the facial feature information may also include the mouth information of the detected object; the mouth information includes the ratio of the height of the mouth of the detected object to the width of the mouth, and at least the area of the mouth. One. In other embodiments, the facial feature information may further include head posture information of the detected object; the head posture information may include, for example, the angle between the current head axis direction and the preset head axis direction. Although the foregoing embodiments are described by taking eye feature information as an example, those skilled in the art can clearly understand that facial feature information is not limited to only including the aforementioned eye feature information.

From the foregoing, it can be seen that the embodiment of the present application proposes a driving reminder method, which has at least the following advantages compared with the prior art:

The driving reminding method proposed in the embodiments of the present application can solve the problem of the high false alarm rate of the existing driver state detection system, and ensure that the driver has the ability to safely take over the vehicle within a specified time range.

For automatic driving systems above the L3 level, for example, the driver does not need to be awake all the time, nor need to pay attention to the road conditions ahead, or even sleep, but needs to be awakened in the event of a system failure. When the automatic driving system fails, the system can adopt different intensities of reminding methods according to the detected driver's status, so that the driver can complete the switching of the driving task subject in a short time.

The driving reminder method proposed in this application can be applied to an automatic driving system above the L3 level, can solve the problem of high false alarm rate of the existing driver state detection system, and ensure that the driver has the ability to safely take over the vehicle within a specified time range.

In addition, the driving reminder method proposed in the optional embodiment of the present application at least includes the following advantages:

First, the driving reminder method proposed in some embodiments of this application uses MBLBP features as the feature descriptor of the face detection process. This feature can more completely characterize image information, is more robust than LBP features, and is more robust than Haar-like features. Efficient.

Second, the embodiment of this application uses the supervised descent method to solve the problem of minimizing the nonlinear least squares (Non-linear Least Squares) objective function. This method has fast processing speed and accurate calculation results, which can overcome the shortcomings of many second-order optimization schemes. , Such as non-differentiable, Hessian matrix is computationally intensive, etc.

The embodiment of the present application also proposes a driving state detection method, which is used to detect the state of the driver of an automatic driving vehicle, including the aforementioned steps S101 to S103.

Corresponding to the aforementioned driving reminder method, an embodiment of the present application also proposes a driving reminder device, including:

A memory in which a computer readable program is stored;

The processor is connected to the memory and is used to execute the computer-readable program to perform the following operations:

Acquiring face image information of the detected object;

Corresponding to the aforementioned driving state detection method, an embodiment of the present application also provides a driving state detection device, including:

A memory in which a computer readable program is stored;

Acquiring face image information of the detected object;

Based on the facial feature information, the driving state of the detected object is acquired.

The embodiment of the present application also proposes an automatic driving system, including:

The vehicle sensor module is used to detect whether the detected object is in the driving position;

Vehicle-mounted camera module, used to obtain images of detected objects;

A memory in which a computer readable program is stored;

The processor is connected to the vehicle sensor module, the vehicle camera module, and the memory, acquires sensor signals and the image, and is used to execute the computer-readable program to perform the following operations:

Acquiring face image information of the detected object from the image;

The embodiment of the present application also proposes an automatic driving system for detecting the driving state of the detected object, including:

Vehicle-mounted camera module, used to obtain images of detected objects;

A memory in which a computer readable program is stored;

Acquiring face image information of the detected object from the image;

Each component embodiment of the present application may be implemented by hardware, or by software modules running on one or more processors, or by a combination of them. Those skilled in the art should understand that a microprocessor or a digital signal processor (DSP) may be used in practice to implement some or all of the functions of some or all components in the computing device according to the embodiments of the present application. This application can also be implemented as a device or device program (for example, a computer program and a computer program product) for executing part or all of the methods described herein. Such a program for implementing the present application may be stored on a computer-readable medium, or may have the form of one or more signals. Such signals can be downloaded from Internet websites, or provided on carrier signals, or provided in any other form.

For example, FIG. 5 shows a computing device that can implement the method according to the present application. The computing device traditionally includes a processor 1010 and a computer program product in the form of a memory 1020 or a computer readable medium. The memory 1020 may be an electronic memory such as flash memory, EEPROM (Electrically Erasable Programmable Read Only Memory), EPROM, hard disk, or ROM. The memory 1020 has a storage space 1030 for executing the program code 1031 of any method step in the above method. For example, the storage space 1030 for program codes may include various program codes 1031 for implementing various steps in the above method. These program codes can be read out from or written into one or more computer program products. These computer program products include program code carriers such as hard disks, compact disks (CDs), memory cards or floppy disks. Such a computer program product is usually a portable or fixed storage unit as described with reference to FIG. 6. The storage unit may have storage segments, storage spaces, etc., arranged similarly to the memory 1020 in the computing device of FIG. 5. The program code can be compressed in an appropriate form, for example. Generally, the storage unit includes computer-readable codes 1031', that is, codes that can be read by, for example, a processor such as 1010. These codes, when run by a computing device, cause the computing device to execute each of the methods described above. step.

The embodiment of the present application provides a computing device, including: one or more processors; and one or more machine-readable media on which instructions are stored. When executed by the one or more processors, The computing device executes the method described in one or more of the embodiments of the present application.

The various embodiments in this specification are described in a progressive manner. Each embodiment focuses on the differences from other embodiments, and the same or similar parts between the various embodiments can be referred to each other.

Although the preferred embodiments of the embodiments of the present application have been described, those skilled in the art can make additional changes and modifications to these embodiments once they learn the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the embodiments of the present application.

Finally, it should be noted that in this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply these entities Or there is any such actual relationship or sequence between operations. Moreover, the terms "include", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or computing device that includes a series of elements includes not only those elements, but also those that are not explicitly listed. Other elements listed, or also include elements inherent to this process, method, article, or computing device. If there are no more restrictions, the element defined by the sentence "including a..." does not exclude the existence of other same elements in the process, method, article or computing device that includes the element.

The above provides a detailed introduction to the driving reminder method and driving reminder device provided by the present application. Specific examples are used in this article to explain the principles and implementation of the present application. The description of the above embodiments is only used to help understand the present application. The method of application and its core idea; meanwhile, for those skilled in the art, according to the idea of this application, there will be changes in the specific implementation and scope of application. In summary, the content of this specification should not be understood It is a restriction on this application.

Industrial applicability

Second, the driving reminder method proposed in some embodiments of this application uses MBLBP features as the feature descriptor of the face detection process. This feature can more completely characterize image information, is more robust than LBP features, and is more robust than Haar-like features. Efficient.

Third, the embodiment of this application uses the supervised descent method to solve the problem of minimizing the nonlinear least squares (Non-linear Least Squares) objective function. This method has fast processing speed and accurate calculation results, which can overcome the shortcomings of many second-order optimization schemes. , Such as non-differentiable, Hessian matrix is computationally intensive

Claims

A driving reminder method, characterized in that it comprises:

Acquiring face image information of the detected object;

Acquiring at least one facial feature information of the detected object from the facial image information;

Obtaining the driving state of the detected object based on the facial feature information;

When a predetermined condition is met, corresponding reminder information is issued based on the driving state.
The driving reminder method according to claim 1, wherein the step of obtaining face image information of the detected object comprises:

Use a face classifier to detect whether there is face image information in the collected images;

When it is determined that there is face image information in the collected image; extracting facial feature information from the face image information.
The driving reminder method according to claim 2, wherein the face classifier is a classifier obtained by extracting MBLBP features from a training set containing human faces and non-human faces, and training by using a cascaded AdaBoost algorithm.
The driving reminder method according to claim 1, wherein the step of obtaining at least one facial feature information of the detected object from the facial image information comprises:

Determining facial feature points according to the facial image information;

The location information of the facial feature point is determined, and the location information is used as the facial feature information.
The driving reminder method according to claim 4, wherein the facial feature points include at least one of eyes, corners of mouth, tip of nose, and face contour.
The driving reminder method according to claim 4, wherein the step of determining facial feature points according to the facial image information comprises:

Obtain the rectangular area of the face according to the face image information;

Extracting the initial shape of at least one facial feature point from the rectangular region of the face;

The step of determining the location information of the facial feature point and using the location information as the facial feature information includes:

The initial shape is corrected by using the image features of the facial feature points.
The driving reminder method according to claim 4, wherein in the step of obtaining the at least one facial feature information from the facial image information according to the facial feature points, the facial feature information comprises: At least one of eye information, mouth information, and head posture information of the detected object.
The driving reminder method of claim 7, wherein the facial feature information includes eye information of the detected object;

The step of judging the driving state of the detected object according to the facial feature information includes:

Extracting eye feature information according to the eye information, and determining the open and closed state of the eye;

Using the eye open and closed states of a plurality of continuous images, the driving state of the detected object is determined, and the driving state includes more than two driving state levels.
The driving reminder method according to claim 8, wherein the eye feature information includes a ratio of eye height to eye width.
The driving reminder method according to claim 7, wherein the facial feature information includes mouth information of the detected object;

The mouth information includes at least one of the ratio of the height of the mouth to the width of the mouth and the area of the mouth of the detected object.
The driving reminder method according to claim 7, wherein the facial feature information includes head posture information of the detected object;

The head posture information includes the angle between the current head axis direction and the preset head axis direction.
The driving reminder method according to claim 1, wherein the predetermined condition comprises:

A system failure signal is received; or

A vehicle failure signal is received.
The driving reminder method of claim 12, wherein the step of issuing corresponding reminder information based on the driving state comprises:

According to the driving state, the system confidence level and the system state signal, corresponding reminder information is sent.
The driving reminder method of claim 13, wherein the method further comprises:

Determine system confidence, including:

Count the number of consecutive detections of face image information;

The system confidence is determined according to the number of times of the facial image information, and the system confidence includes more than two system confidence levels.
A driving state detection method, characterized in that it comprises:

Acquiring face image information of the detected object;

Acquiring at least one facial feature information of the detected object from the facial image information;

The driving state of the detected object is determined based on the facial feature information.
A computing device for executing driving reminders, the computing device includes:

A memory, which stores a computer-readable program; a processor, connected to the memory, for executing the computer-readable program to perform the following operations:

Acquiring face image information of the detected object;

Acquiring at least one facial feature information of the detected object from the facial image information;

Obtaining the driving state of the detected object based on the facial feature information;

When a predetermined condition is met, corresponding reminder information is issued based on the driving state.
The computing device of claim 16, further comprising:

The vehicle sensor module is used to detect whether the detected object is in the driving position;

Vehicle-mounted camera module, used to obtain images of detected objects;

In the operation of obtaining the face image information of the detected object, the face image information is obtained from the image obtained by the vehicle-mounted camera module.
A computing device for performing driving state detection, the computing device includes:

A memory, which stores a computer-readable program; a processor, connected to the memory, for executing the computer-readable program to perform the following operations:

Acquiring face image information of the detected object;

Acquiring at least one facial feature information of the detected object from the facial image information;

Based on the facial feature information, the driving state of the detected object is acquired.
The computing device of claim 18, further comprising:

The vehicle sensor module is used to detect whether the detected object is in the driving position;

Vehicle-mounted camera module, used to obtain images of detected objects;

In the operation of obtaining the face image information of the detected object, the face image information is obtained from the image obtained by the vehicle-mounted camera module.