WO2022094787A1

WO2022094787A1 - Driver data processing system and driver data acquisition method

Info

Publication number: WO2022094787A1
Application number: PCT/CN2020/126437
Authority: WO
Inventors: 高杨; 陈创荣; 陈晓智
Original assignee: 深圳市大疆创新科技有限公司
Priority date: 2020-11-04
Filing date: 2020-11-04
Publication date: 2022-05-12
Also published as: CN114503171A

Abstract

A driver data processing system, comprising: multiple guide markers, provided in front of a driving seat of a vehicle body; a prompting device, used for prompting a driver on the driving seat to look at any target guide marker of the multiple guide markers; multiple image acquisition devices, configured to be mounted at set positions of a driver cabin of the vehicle to at least acquire image data of the driver on the driving seat; and a monitoring data processing device, used for generating a training data set according to the driver image data acquired by the multiple image acquisition devices, three-dimensional position information of the multiple image acquisition devices, and three-dimensional position information of the multiple guide markers, for use in training a driver monitoring model. Performing data acquisition in a real vehicle can obtain driver data closer to a real situation, such that a driver monitoring model for accurately monitoring the state of a driver can be trained.

Description

Driver data processing system and method for collecting driver data

technical field

The invention relates to the technical field of assisted driving, in particular to a driver data processing system, a method for collecting driver data, a driver monitoring model training method, an assisted driving system and a vehicle.

Background technique

In the prior art, the driver's data is collected by using an eye tracker on the hardware to obtain the true value of the gaze, and the position of the center point of the head is obtained through the RGBD camera to obtain the head pose. The appearance of the instrument is removed, so as to obtain the normal image information and the truth information of gaze and head pose.

These solutions are often implemented by building a simulated scene indoors. The indoor simulated scene is quite different from the actual use scene, resulting in a big difference between the imaging effect and the true value distribution of gaze/head pose of the collected two-dimensional image and the actual use scene. If a dataset is built and CNN is trained in this way, the actual performance when running in the cockpit is often not good, because the generalization ability of CNN is limited, in the final analysis, the in-vehicle scene and the simulated scene are not similar.

SUMMARY OF THE INVENTION

Embodiments of the present invention provide a driver data processing system, a method for collecting driver data, a driver monitoring model training method, an assisted driving system and a vehicle, which are used to solve at least one of the above technical problems.

In a first aspect, an embodiment of the present invention provides a driver data processing system, which includes:

A plurality of guide marks are arranged in front of the driver's seat of the vehicle body;

a prompting device for prompting the driver on the driving seat to look at any target guide mark among the plurality of guide marks;

a plurality of image acquisition devices, which are used to be installed at the set positions of the cockpit of the vehicle, so as to collect at least the image data of the driver in the driver's seat;

A monitoring data processing device, configured to generate training data according to the image data of the driver collected by the plurality of image capture devices, the three-dimensional position information of the plurality of image capture devices, and the three-dimensional position information of the plurality of guide marks set for training driver monitoring models.

In a second aspect, an embodiment of the present invention provides a method for collecting driver data, comprising:

A plurality of guide marks are arranged in front of the driving seat of the vehicle body;

Install a plurality of image acquisition devices at the set positions of the cockpit to collect at least the image data of the driver in the driver's seat;

Prompt the driver to look at any target guide mark among the plurality of guide marks by means of a prompting device;

Collect the image data of the driver through the plurality of image collection devices;

The monitoring data processing device generates a training data set according to the image data of the driver collected by the plurality of image collection devices, the three-dimensional position information of the plurality of image collection devices, and the three-dimensional position information of the plurality of guide marks, so as to Used to train driver monitoring models.

In a third aspect, an embodiment of the present invention provides a method for training a driver monitoring model, which includes: collecting driver data by using the method for collecting driver data according to any embodiment of the present invention to construct a training data set; using the training data set to train a driver monitoring model.

In a fourth aspect, an embodiment of the present invention provides an assisted driving system, which is configured with a driver monitoring model trained by the driver monitoring model training method according to any embodiment of the present invention.

In a fifth aspect, an embodiment of the present invention provides a vehicle configured with the driving assistance system according to any embodiment of the present invention.

The beneficial effect of the embodiments of the present invention is that by collecting data in a real vehicle, driver data closer to the real situation can be obtained, which can be used to train a driver monitoring model that accurately monitors the driver's state.

Description of drawings

In order to illustrate the technical solutions of the embodiments of the present invention more clearly, the following briefly introduces the accompanying drawings used in the description of the embodiments. Obviously, the drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can also be obtained from these drawings without any creative effort.

1 is a schematic diagram of an embodiment of an application scenario of a driver data processing system of the present invention;

2 is a schematic flowchart of an embodiment of a method for collecting driver data according to the present invention;

3 is a schematic flowchart of an embodiment of a method for collecting driver data according to the present invention;

4 is a schematic flowchart of an embodiment of a method for collecting driver data according to the present invention;

FIG. 5 is a schematic flowchart of an embodiment of a method for collecting driver data according to the present invention.

Detailed ways

In order to make the purposes, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments These are some embodiments of the present invention, but not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

It should be noted that the embodiments of the present invention and the features of the embodiments may be combined with each other under the condition of no conflict.

The invention may be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, elements, data structures, etc. that perform particular tasks or implement particular abstract data types. The invention may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media including storage devices.

In the present invention, "module", "device", "system", etc. refer to relevant entities applied to a computer, such as hardware, a combination of hardware and software, software or software in execution, and the like. In detail, for example, an element may be, but is not limited to, a process running on a processor, a processor, an object, an executable element, a thread of execution, a program, and/or a computer. Also, an application program or script program running on the server, the server can be a component. One or more elements may be in a process and/or thread of execution, and an element may be localized on one computer and/or distributed between two or more computers and may be executed from various computer readable media . Elements may also pass through a signal having one or more data packets, for example, a signal from one interacting with another element in a local system, in a distributed system, and/or with data interacting with other systems through a network of the Internet local and/or remote processes to communicate.

Finally, it should also be noted that in this document, relational terms such as first and second are used only to distinguish one entity or operation from another, and do not necessarily require or imply these entities or that there is any such actual relationship or sequence between operations. Furthermore, the terms "comprising" and "comprising" include not only those elements, but also other elements not expressly listed, or elements inherent to such a process, method, article or apparatus. Without further limitation, an element defined by the phrase "comprises" does not preclude the presence of additional identical elements in a process, method, article, or device that includes the element.

In order to solve the technical problems existing in the prior art, an embodiment of the present invention provides a driver data processing system, which includes: a plurality of guide marks for guiding the driver to look at each guide mark; a plurality of image acquisition devices , which is used to collect the image data of the driver; the prompting device is used to prompt the driver to look at the target guide mark; the monitoring data processing device is used to process the image data collected by a plurality of influence collecting devices.

As shown in FIG. 1 , it is a schematic diagram of an embodiment of an application scenario of the driver data processing system of the present invention. In this embodiment, the driver data processing system includes:

A plurality of guide marks (1-17) are provided in front of the driver's seat of the vehicle body. Exemplarily, the setting positions of the plurality of guide marks ( 1 - 17 ) are selected from the positions within the line of sight of the driver under normal circumstances during driving. The plurality of guide marks (1-17) may be wholly or partially located inside or outside the vehicle, which is not limited in the present invention. The driver can look at different guidance marks by adjusting the head posture and/or the line of sight of the human eye.

A plurality of image acquisition devices (cam_0-cam_n) are used to be installed at a set position of the cockpit of the vehicle, so as to collect at least image data of the driver in the driver's seat. Illustratively, a plurality of image capture devices are installed in the cockpit or outside the cockpit. For example, the image capture device can be installed on the front windshield (inside or outside the car) through suction cups or double-sided tape, or multiple image capture devices can be installed on the front windshield through brackets erected on the outside of the roof. outside. The present invention does not limit the fixing and installation methods of the plurality of image capturing devices.

Exemplarily, the plurality of image capturing devices may be any device with a camera and/or a photographing function, for example, a camera, or a hardware device with a camera function (for example, a smart phone, etc.), to which the present invention Not limited. Multiple image collection devices can collect image data of drivers from multiple different angles. The plurality of image capturing devices may take pictures according to preset program intervals or take pictures in response to the control of the operator, which is not limited in the present invention.

A prompting device is used for prompting the driver on the driving seat to look at any target guide mark among the plurality of guide marks. Exemplarily, the prompting device may be a voice broadcasting device or an indicating device using visible light. For example, the voice announcement device can prompt the driver participating in the data collection to look at the target guidance point through voice; the indicating device using visible light can guide the driver participating in the data collection to look at the target guidance point by projecting visible light to the target guidance point, for example. The prompting device may be an independent device or a module integrated in a certain terminal device, which is not limited in the present invention.

A monitoring data processing device, configured to generate training data according to the image data of the driver collected by the plurality of image capture devices, the three-dimensional position information of the plurality of image capture devices, and the three-dimensional position information of the plurality of guide marks set for training driver monitoring models. The monitoring data processing device can be an independent device or a module integrated in a terminal device.

Exemplarily, the monitoring data processing device is a mobile control terminal, the mobile control terminal is communicatively connected with the plurality of image capture devices, and the mobile control terminal is further configured to control the plurality of image capture devices in response to the driver's operation. device to take pictures.

Exemplarily, the prompting device and the monitoring data processing device may be separate devices or different modules integrated with the same terminal device. The present invention is not limited to this.

The driver data processing system of the embodiment of the present invention can be directly applied in the car to collect and process driver data, so that the driver data closer to the real situation can be obtained by collecting data in the real car, which can be used for Train a driver monitoring model that accurately monitors driver status.

In some embodiments, a training data set is generated according to the image data of the driver collected by the plurality of image capture devices, the three-dimensional position information of the plurality of image capture devices and the three-dimensional position information of the plurality of guide markers The method includes: determining the driver's eye sight data and head posture data according to the image data, the three-dimensional position information of the plurality of image acquisition devices and the three-dimensional position information of the plurality of guide marks, so as to generate a training data set.

In this embodiment, the true value information of the gaze of the human eye, the head pose, and the corresponding two-dimensional image of the human head can be obtained. The application method is mainly to obtain the true values of gaze and head pose to make a dataset, and build a neural network to train on the corresponding dataset, so that the neural network has the ability to predict gaze and head pose information. It is mainly used in the fields of intelligent driving dms (driver monitoring system), human eye attention detection, etc., which is not limited in the present invention.

In some embodiments, determining the driver's eye sight data and head posture data according to the image data, the three-dimensional position information of the plurality of image acquisition devices, and the three-dimensional position information of the plurality of guide marks includes: according to The image data, the three-dimensional position information of the plurality of image acquisition devices, and the three-dimensional position information of the plurality of guide marks determine the line-of-sight data and the head posture data when the driver looks at the target guide mark.

The present invention can obtain the gaze and head pose truth value information of the head looking at the preset point of the designated position in the cockpit. Other similar schemes are often implemented by building simulated scenes indoors, and the scenes are quite different. If other schemes are used to build a dataset for training CNN, the actual performance is often not good when running in the car, because the generalization ability of CNN is limited. In the final analysis, the in-car scene and the simulated scene are not similar. The present invention solves this problem by obtaining the true value information of gaze and head pose in the car.

In some embodiments, the image data, the three-dimensional position information of the multiple image acquisition devices, and the three-dimensional position information of the multiple guide markers are the input of the driver monitoring model, and the human eye sight data and the head posture data are the input of the driver monitoring model. output. During the training of the driver monitoring model, the image data, the three-dimensional position information of multiple image acquisition devices and the three-dimensional position information of multiple guide markers are obtained from the constructed database as input, and the corresponding human eye sight data and The head pose data is used as the output target for training.

In some embodiments, according to the image data, the three-dimensional position information of the plurality of image acquisition devices, and the three-dimensional position information of the plurality of guide marks, the line of sight data of the human eye when the driver looks at the target guide mark is determined and head pose data including:

Determine the three-dimensional position information of the face key point when the driver looks at the target guide mark according to the image data, and the three-dimensional position information of the face key point includes the three-dimensional position information of the center point of the human eye and the three-dimensional position of the center point of the head information;

According to the three-dimensional position information of the face key points, the three-dimensional position information of the plurality of image acquisition devices, and the three-dimensional position information of the target guide mark, determine the line of sight data of the human eye when the driver looks at the target guide mark and Head pose data.

In this embodiment, the three-dimensional position information of the face key points is determined through the image data collected by multiple cameras, so as to ensure the accuracy of the obtained three-dimensional position information of the face key points; The position information and the three-dimensional position information of the current first preset guide point determine the line-of-sight data and the head posture data. As a result, more realistic and more accurate data can be obtained.

In some embodiments, the number of the multiple image acquisition devices is n, and the corresponding image data includes n images; the three-dimensional position information of the key points of the face when the driver looks at the target guide mark according to the image data includes:

Restore the scene depth map from n images;

Determine the two-dimensional position information of the face key point in the ith image, and the ith image corresponds to the ith image acquisition device;

According to the scene depth map and the two-dimensional position information of the face key points in the ith image, determine the three-dimensional position information of the face key points in the coordinate system of the ith image acquisition device when the driver looks at the target guide mark.

In this embodiment, every time the driver looks at a preset guide point, he simultaneously obtains the three-dimensional position information of multiple sets of face key points in each camera coordinate system. Thus, it is convenient to obtain more abundant human eye sight data and head posture data.

In some embodiments, the plurality of image capture devices include a first image capture device and a second image capture device, and the positions of the first image capture device and the second image capture device are based on the space of the plurality of guide markers distribution is determined.

In this embodiment, the first camera and the second camera are set by determining two optimal positions according to the distribution of the preset guide points, and the requirement of data collection is ensured under the condition of using the least number of cameras.

In some embodiments, the plurality of guide marks includes a plurality of guide marks in front of the cockpit and a plurality of guide marks on the left and right sides of the cockpit. Illustratively, the plurality of guide marks ahead of the cockpit include one or more guide marks set on the windshield, and one or more guide marks set on the instrument panel.

Illustratively, a plurality of guide markings in front of the cockpit are provided on the front windshield and on the instrument panel using discrete patches. By setting guide marks in the form of patches, it is convenient to quickly complete the arrangement of preset points in different models.

In some embodiments, the plurality of guide marks on the left and right sides of the cockpit include one or more guide marks disposed on the left and right front door glass, and one or more guide marks disposed on the left and right rear view mirrors guide marker. Exemplarily, a plurality of guide marks on the left and right sides of the cockpit are arranged on the left and right front door glass and on the left and right rear view mirrors using discrete patches.

In some embodiments, the front windshield of the cockpit is a transparent display screen, and a plurality of guide marks in front of the cockpit are presented on the transparent display screen according to set rules; the left and right sides of the cockpit are displayed on the transparent display screen. The multiple guide marks of the 2000 are arranged in discrete patches on the left and right front door glass and on the left and right rear-view mirrors. Alternatively, a plurality of guide marks in front of the cockpit are presented according to a set rule through a display screen arranged on the front and outer side of the cockpit; the plurality of guide marks on the left and right sides of the cockpit are set on the left and right fronts using discrete patches On the door glass and on the left and right mirrors.

In this embodiment, the preset guide points are presented through a "screen", which can provide sufficiently dense preset guide points, and avoid the problem of data sparseness caused by discrete guide marks of physical marks.

In some embodiments, the three-dimensional position information of the plurality of image capturing devices and the three-dimensional position information of the plurality of guide marks are predetermined.

Exemplarily, the present invention adopts two or more image acquisition devices (eg, cameras), in addition to that, it is not necessary to perform too many modifications on the cockpit of the real vehicle, which is relatively simple and convenient to implement. It mainly includes two major components, a system calibration scheme and a self-labeling scheme, to determine the three-dimensional position information of the multiple image acquisition devices and the three-dimensional position information of the multiple guide marks.

1. The system calibration scheme mainly consists of three components, namely, internal parameter calibration of multiple cameras, external parameter calibration between multiple cameras, and external parameter calibration between cameras and preset points (for example, preset guide marks) . Suppose we use n cameras for data collection, in which the camera internal parameters of cam_1~cam_n and the camera external parameters between cam_1~cam_n can be calculated by the existing camera external parameter calibration method, and the existing mature technology can be used. Program. For example, using the open source solution kalibr can achieve better results. The following will focus on how to calibrate the external parameters between the camera and the preset point.

The following describes how to calibrate the external parameters between the preset point and the camera with reference to Figure 1. The cockpit scene can define several preset points according to requirements, for example, 17 preset points are defined in the schematic diagram of FIG. 1 . We have n cameras for data collection (n is greater than or equal to 2). Taking cam_0 as the reference camera, we need to calculate the coordinates of all preset points in the cam_0 camera coordinate system. Because spatial points do not have any perception ability, we consider placing an additional camera cam_s on each preset point (as shown in the guide point 8 in Figure 1), and then perform pairwise camera extrinsic calibration with cam_0, just You can get the rotation and translation of the preset point position camera cam_s to the reference camera cam_0. We only need to translate to get the spatial coordinates of the preset point in the camera coordinate system of cam_0, and then complete the external parameter calibration between the preset point and the camera.

2. The self-labeling algorithm of this scheme is similar to the existing scheme as a whole, but a unique process design is added to make the whole set of algorithms fast and efficient. The invention has only one guide point, and the spatial coordinates of the center point of the human eye and the center point of the head are obtained through three-dimensional reconstruction and face key point detection. The calibration result of the camera and the preset point in the previous step can convert the center point and the guide point to the same A camera coordinate system, so that the space vector can be obtained, and then converted into pitch angle and yaw angle.

The key to the whole problem is how to obtain the spatial position of the face key points in the camera coordinate system. The present invention adopts a simple and quick way to calculate. When the present invention collects data, there are multiple sets of cameras that can be triggered at the same time, and the external parameters between the cameras can be obtained by means of calibration, so the depth map of the head can be obtained by the method of three-dimensional reconstruction.

Reconstruction using image data from multiple cameras is often time-consuming and increases algorithmic complexity. In theory, the scene depth of the face can be recovered by using two cameras. However, it is necessary to determine the positions of the two cameras with the best observation effect. On the one hand, the larger the overlapping area of the images between the cameras, the better the reconstruction effect; on the other hand, we are concerned with the reproduction effect of the face area. The other parts are not needed, so it is necessary to make sure that the face is facing the two cameras facing each other. In actual use, since the spatial position of the guidance point in the cockpit is known, and the external parameters between each camera are also known, it is best to find two observation positions by comparing the distances from the spatial point to each camera. camera, and then restore the scene depth through stereo matching. At the same time, the face key point detection is performed on the image of one of the cameras, and the two-dimensional position of the key point is found and mapped into a three-dimensional space position. Thereby, the spatial position of the key points of the face is obtained, and then the self-labeling of the gaze gaze and the head pose is performed.

In some embodiments, the present invention also provides a method for collecting driver data based on the system described in any of the foregoing embodiments.

As shown in FIG. 2 , it is a schematic flowchart of an embodiment of a method for collecting driver data according to the present invention. The following steps are included in this embodiment:

System initialization: including the internal parameter calibration of the camera, the external parameter calibration between multiple cameras, and the external parameter calibration between the camera and the cabin preset point.

1. Preset point selection

1.1. From all the preset points in the cockpit, select a point G as a guide point according to a certain strategy (for example, randomly select points);

1.2. The system guides (voice prompts or text prompts) to collect the sight of the person looking at the guide point G (the head is facing the guide point, and the eyes are looking at the guide point at the same time).

2. Acquire face box position

2.1. Multiple cameras (for example, two cameras cam_0 and cam_1) restore the depth map of the scene through a multi-view reconstruction method (such as sgbm);

2.2. Execute the face det and face landmark inference processes to obtain the 2D position of the center point of the eye and the center point of the head area in the camera image;

2.3. Take the 2D position area of the head center point and the eye center point in the depth map, and perform threshold and average operations in turn to roughly obtain the depth value of the relevant center point after filtering, and then obtain the corresponding point in the camera coordinates through coordinate system transformation. the spatial location of the system.

3. Calculate the human eye sight and head posture

3.1. Map the spatial positions of the head center point and the eye center point obtained in the previous step into other camera coordinate systems (external parameters between cameras are required);

If two cameras cam_0 and cam_1 are taken as an example, and the coordinate system originally used is the coordinate system of cam_0, then "other cameras" refers to the camera cam_1.

3.2. Map the guide point G in the cockpit to each camera coordinate system (external parameters between cameras and external parameters between the camera and the preset guide point G are required);

3.3. In each camera coordinate system, calculate gaze vector and head pose vector respectively and convert them to pitch, yaw of gaze and pitch, yaw of head pose;

Take two cameras cam_0 and cam_1 as an example, please confirm whether the following understanding is correct:

Under the coordinate system of camera cam_0:

When the preset point (1) is the guide point G, gaze vector_01 and head pose vector_01 are calculated and converted into pitch_01, yaw_01 of gaze and pitch_01, yaw_01 of head pose;

When the preset point (2) is the guide point G, gaze vector_02 and head pose vector_02 are calculated and converted into pitch_02, yaw_02 of gaze and pitch_02, yaw_02 of head pose;

...

When the preset point (18) is the guide point G, gaze vector_018 and head pose vector_018 are calculated and converted into pitch_018, yaw_018 of gaze and pitch_018, yaw_018 of head pose.

Under the coordinate system of camera cam_1:

When the preset point (1) is the guide point G, gaze vector_11 and head pose vector_11 are calculated and converted into pitch_11, yaw_11 of gaze and pitch_11, yaw_11 of head pose;

When the preset point (2) is the guide point G, gaze vector_12 and head pose vector_12 are calculated and converted into pitch_12, yaw_12 of gaze and pitch_12, yaw_12 of head pose;

...

When the preset point (18) is the guide point G, gaze vector_118 and head pose vector_118 are calculated and converted into pitch_118, yaw_118 of gaze and pitch_118, yaw_118 of head pose.

3.4. The final output multi-channel self-labeling results (pitch, yaw of head pose and pitch, yaw of gaze).

Finally, it is judged whether the time for each person to collect data has reached the predetermined time, and if it exceeds the predetermined time, the collection ends, otherwise the cycle continues.

As shown in FIG. 3, another embodiment of the present invention is a method for collecting driver data. In this embodiment, the method includes:

S100, setting a plurality of guide marks in front of the driving seat of the vehicle body;

S200. Install a plurality of image acquisition devices at set positions of the cockpit to collect at least image data of the driver in the driver's seat;

S300, prompting the driver to look at any target guide mark in the plurality of guide marks through the prompting device;

S400, collecting image data of the driver through the plurality of image collecting devices;

S500. The monitoring data processing device generates a training data set according to the image data of the driver collected by the multiple image collecting devices, the three-dimensional position information of the multiple image collecting devices, and the three-dimensional position information of the multiple guide markers , for training the driver monitoring model.

The method for collecting driver data in the embodiment of the present invention can be directly applied in the car to collect and process driver data, so that the driver data that is closer to the real situation can be obtained by collecting data in the real car, so that it can be used in It is used to train a driver monitoring model that accurately monitors the driver's state.

In some embodiments, a training data set is generated according to the image data of the driver collected by the plurality of image capture devices, the three-dimensional position information of the plurality of image capture devices and the three-dimensional position information of the plurality of guide markers include:

According to the image data, the three-dimensional position information of the plurality of image acquisition devices, and the three-dimensional position information of the plurality of guide marks, the driver's eye line of sight data and head posture data are determined to generate a training data set.

In some embodiments, determining the driver's eye line of sight data and head posture data according to the image data, the three-dimensional position information of the plurality of image acquisition devices, and the three-dimensional position information of the plurality of guide marks includes:

According to the image data, the three-dimensional position information of the plurality of image acquisition devices and the three-dimensional position information of the plurality of guide marks, the line of sight data and the head posture data of the driver when looking at the target guide mark are determined.

In some embodiments, the input of the driver monitoring model is the image data, the three-dimensional position information of the plurality of image acquisition devices and the three-dimensional position information of the plurality of guide marks are the input of the driver monitoring model, The human eye sight data and the head posture data are outputs of the driver monitoring model.

As shown in FIG. 4 , in some embodiments, according to the image data, the three-dimensional position information of the plurality of image acquisition devices, and the three-dimensional position information of the plurality of guide marks, it is determined that the driver is looking at the target guide mark The human eye sight data and head pose data include:

S510. Determine the three-dimensional position information of the face key point when the driver looks at the target guide mark according to the image data, and the three-dimensional position information of the face key point includes the three-dimensional position information of the center point of the human eye and the center point of the head three-dimensional position information;

S520. Determine, according to the three-dimensional position information of the face key points, the three-dimensional position information of the multiple image acquisition devices, and the three-dimensional position information of the target guide mark, the line of sight of the driver when looking at the target guide mark data and head pose data.

In some embodiments, the number of the plurality of image acquisition devices is n, and the corresponding image data includes n images; as shown in FIG. 5 , according to the image data, it is determined when the driver looks at the target guide mark. The three-dimensional position information of face key points includes:

S511, restore the scene depth map according to the n images;

S512, determine the two-dimensional position information of the face key point in the ith image, where the ith image corresponds to the ith image acquisition device;

S513, according to the scene depth map and the two-dimensional position information of the face key points in the i-th image, determine the position of the driver under the coordinate system of the i-th image acquisition device when the driver looks at the target guide mark 3D position information of face key points.

In some embodiments, the plurality of guide marks includes a plurality of guide marks in front of the cockpit and a plurality of guide marks on the left and right sides of the cockpit.

In some embodiments, the plurality of guide marks ahead of the cockpit include one or more guide marks set on the windshield, and one or more guide marks set on the instrument panel.

In some embodiments, the plurality of guide markings on the front of the cockpit are provided in discrete patches on the front windshield and on the instrument panel.

In some embodiments, the plurality of guide marks on the left and right sides of the cockpit include one or more guide marks disposed on the left and right front door glass, and one or more guide marks disposed on the left and right rear view mirrors guide marker.

In some embodiments, a plurality of guide marks on the left and right sides of the cockpit are arranged on the left and right front door glass and on the left and right rear view mirrors using discrete patches.

In some embodiments, the front windshield of the cockpit is a transparent display screen, and a plurality of guide marks in front of the cockpit are presented on the transparent display screen according to set rules; the left and right sides of the cockpit are displayed on the transparent display screen. The multiple guide marks of the 2000 are set in discrete patches on the left and right front door glass and on the left and right rear view mirrors.

In some embodiments, a plurality of guide marks in front of the cockpit are presented according to a set rule through a display screen arranged on the front and outside of the cockpit; the plurality of guide marks on the left and right sides of the cockpit are set using discrete patches On the left and right front door glass and on the left and right mirrors.

In some embodiments, multiple image capture devices are mounted in the cockpit or outside the cockpit.

In some embodiments, the monitoring data processing device is a mobile control terminal, the mobile control terminal is connected in communication with the plurality of image acquisition devices, and the mobile control terminal is further configured to control the plurality of image capture devices in response to the operation of the driver The image acquisition device takes pictures.

In some embodiments, the method of collecting driver data further includes:

Before collecting the image data when the driver looks at the target guide mark through the plurality of image collection devices, calibrate the plurality of image collection devices and the plurality of guide marks in the same coordinate system to obtain a calibration result;

After collecting the image data when the driver looks at the target guide mark through the plurality of image acquisition devices, the monitoring data processing device determines the person when the driver looks at the target guide mark according to the calibration result and the image data Eye gaze data and head pose data.

In some embodiments, the calibration results obtained by calibrating the plurality of image acquisition devices and the plurality of guide markers in the same coordinate system include:

Perform internal parameter calibration on the plurality of image acquisition devices, and perform external parameter calibration among the plurality of image acquisition devices;

selecting one image acquisition device among the plurality of image acquisition devices as a reference image acquisition device;

The plurality of guide marks are calibrated according to the coordinate system where the reference image acquisition device is located.

In some embodiments, calibrating the plurality of guide marks according to the coordinate system where the reference image acquisition device is located includes:

Execute on each of the plurality of guide markers:

Arrange preset points at the guide marks to calibrate the image acquisition device;

Perform external parameter calibration on the reference image acquisition device and the preset point calibration image acquisition device to complete the calibration of the guide marks.

selecting one guide marker from the plurality of guide markers as a reference preset guide point;

Arranging preset points at the reference preset guide points to calibrate the image acquisition device;

performing external parameter calibration on the reference image acquisition device and the preset point calibration image acquisition device to complete the calibration of the reference preset guide point;

Other guide marks in the plurality of guide marks are calibrated according to the calibration result of the reference preset guide point and the relative positional relationship between the plurality of guide marks.

The calibration of the plurality of guide marks in the coordinate system of the reference image acquisition device is completed by means of pre-measurement.

The calibration of the plurality of guide markers in the coordinate system of the reference image acquisition device is completed by constructing a cockpit simulation model.

In some embodiments, the present invention also provides a method for training a driver monitoring model, the method comprising: using the method described in any of the foregoing embodiments to collect driver data to construct a training data set; training using the training data set Driver monitoring model. Exemplarily, the driver monitoring model adopts a convolutional neural network model.

In some embodiments, the present invention also provides an assisted driving system, which is configured with a driver monitoring model trained by the method described in any of the foregoing embodiments.

In some embodiments, the present invention also provides a vehicle equipped with the driving assistance system described in any of the foregoing embodiments.

It should be noted that, for the sake of simple description, the foregoing method embodiments are all expressed as a series of actions combined, but those skilled in the art should know that the present invention is not limited by the described sequence of actions. As in accordance with the present invention, certain steps may be performed in other orders or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present invention. In the above-mentioned embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference may be made to the relevant descriptions of other embodiments.

The device embodiments described above are only illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in One place, or it can be distributed over multiple network elements. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution in this embodiment.

From the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a general hardware platform, and certainly can also be implemented by hardware. Based on this understanding, the above-mentioned technical solutions can be embodied in the form of software products in essence, or the parts that make contributions to related technologies, and the computer software products can be stored in computer-readable storage media, such as ROM/RAM, magnetic disks , optical disc, etc., including several instructions to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to perform the methods described in various embodiments or some parts of the embodiments.

Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, but not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that it can still be The technical solutions described in the foregoing embodiments are modified, or some technical features thereof are equivalently replaced; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

A driver data processing system, comprising:

A plurality of guide marks are arranged in front of the driver's seat of the vehicle body;

a prompting device for prompting the driver on the driving seat to look at any target guide mark among the plurality of guide marks;

a plurality of image acquisition devices, which are used to be installed at the set positions of the cockpit of the vehicle, so as to collect at least the image data of the driver in the driver's seat;

A monitoring data processing device, configured to generate training data according to the image data of the driver collected by the plurality of image capture devices, the three-dimensional position information of the plurality of image capture devices, and the three-dimensional position information of the plurality of guide marks set for training driver monitoring models.
The system according to claim 1, wherein, according to the image data of the driver collected by the plurality of image capture devices, the three-dimensional position information of the plurality of image capture devices and the information of the plurality of guide marks The training dataset for generating 3D position information includes:

According to the image data, the three-dimensional position information of the plurality of image acquisition devices, and the three-dimensional position information of the plurality of guide marks, the driver's eye line of sight data and head posture data are determined to generate a training data set.
The system according to claim 2, wherein the driver's eye sight data and Head pose data includes:

According to the image data, the three-dimensional position information of the plurality of image acquisition devices and the three-dimensional position information of the plurality of guide marks, the line of sight data and the head posture data of the driver when looking at the target guide mark are determined.
The system according to claim 3, wherein the image data, the three-dimensional position information of the plurality of image acquisition devices and the three-dimensional position information of the plurality of guide marks are the input of the driver monitoring model, The human eye sight data and the head posture data are outputs of the driver monitoring model.
The system according to claim 3, wherein, according to the image data, the three-dimensional position information of the plurality of image acquisition devices, and the three-dimensional position information of the plurality of guidance marks, it is determined that the driver is looking at the target guidance The eye sight data and head pose data at the time of marking include:

Determine the three-dimensional position information of the face key point when the driver looks at the target guide mark according to the image data, and the three-dimensional position information of the face key point includes the three-dimensional position information of the center point of the human eye and the three-dimensional position of the center point of the head information;

According to the three-dimensional position information of the face key points, the three-dimensional position information of the plurality of image acquisition devices, and the three-dimensional position information of the target guide mark, determine the line of sight data of the human eye when the driver looks at the target guide mark and Head pose data.
The system according to claim 5, wherein the number of the plurality of image acquisition devices is n, and the corresponding image data includes n images;

Determining, according to the image data, the three-dimensional position information of the face key points when the driver looks at the target guidance mark includes:

Restore the scene depth map according to the n images;

Determine the two-dimensional position information of the face key point in the ith image, and the ith image corresponds to the ith image acquisition device;

According to the scene depth map and the two-dimensional position information of the face key points in the ith image, determine the face of the driver in the coordinate system of the ith image acquisition device when the driver looks at the target guide mark 3D position information of key points.
The system according to claim 1, wherein the plurality of image capturing devices comprises a first image capturing device and a second image capturing device, and the positions of the first image capturing device and the second image capturing device are It is determined according to the spatial distribution of the plurality of guide marks.
The system of claim 1, wherein the plurality of guide marks comprises a plurality of guide marks in front of the cockpit and a plurality of guide marks on left and right sides of the cockpit.
9. The system of claim 8, wherein the plurality of guide markings on the front of the cockpit include one or more guide markings set on a windshield and one or more guide markings set on an instrument panel guide marker.
The system according to claim 9, wherein the plurality of guide marks in front of the cockpit are arranged on the front windshield and on the instrument panel using discrete patches.
The system according to claim 8, wherein the plurality of guide marks on the left and right sides of the cockpit include one or more guide marks arranged on the left and right front door glass, and one or more guide marks arranged on the left and right One or more guide markings on the rear view mirror.
The system according to claim 11, wherein a plurality of guide marks on the left and right sides of the cockpit are arranged on the left and right front door glass and on the left and right rear view mirrors using discrete patches.
The system according to claim 8, wherein the front windshield of the cockpit is a transparent display screen, and a plurality of guide marks in front of the cockpit are presented on the transparent display screen according to set rules; A plurality of guide marks on the left and right sides of the cockpit are arranged on the left and right front door glass and on the left and right rear-view mirrors using discrete patches.
The system according to claim 8, wherein a plurality of guide marks in front of the cockpit are presented according to a set rule through a display screen arranged on the front and outside of the cockpit; Each guide mark is set in discrete patches on the left and right front door glass and on the left and right rear view mirrors.
The system according to any one of claims 1-14, wherein the plurality of image acquisition devices are installed in the cockpit or outside the cockpit.
The system according to any one of claims 1-14, wherein the monitoring data processing device is a mobile control terminal, the mobile control terminal is communicatively connected to the plurality of image acquisition devices, and the mobile control terminal It is also used for controlling the plurality of image capturing devices to take pictures in response to the operation of the driver.
The system according to any one of claims 1-14, wherein the three-dimensional position information of the plurality of image acquisition devices and the three-dimensional position information of the plurality of guide marks are predetermined.
A method for collecting driver data, comprising:

A plurality of guide marks are arranged in front of the driving seat of the vehicle body;

Install a plurality of image acquisition devices at the set positions of the cockpit to collect at least the image data of the driver in the driver's seat;

Prompt the driver to look at any target guide mark among the plurality of guide marks by means of a prompting device;

Collect the image data of the driver through the plurality of image collection devices;

The monitoring data processing device generates a training data set according to the image data of the driver collected by the plurality of image collection devices, the three-dimensional position information of the plurality of image collection devices, and the three-dimensional position information of the plurality of guide marks, so as to Used to train driver monitoring models.
The method according to claim 18, wherein, according to the image data of the driver collected by the plurality of image capture devices, the three-dimensional position information of the plurality of image capture devices and the information of the plurality of guide marks The training dataset for generating 3D position information includes:

According to the image data, the three-dimensional position information of the plurality of image acquisition devices, and the three-dimensional position information of the plurality of guide marks, the driver's eye line of sight data and head posture data are determined to generate a training data set.
The method according to claim 19, wherein the driver's eye sight data and Head pose data includes:

According to the image data, the three-dimensional position information of the plurality of image acquisition devices, and the three-dimensional position information of the plurality of guide marks, the human eye line of sight data and the head posture data when the driver looks at the target guide mark are determined.
The method according to claim 20, wherein the input of the driver monitoring model is the image data, the three-dimensional position information of the plurality of image acquisition devices, and the three-dimensional position information of the plurality of guide marks: The input of the driver monitoring model, the eye sight data and the head posture data are the output of the driver monitoring model.
The method according to claim 20, characterized in that it is determined according to the image data, the three-dimensional position information of the plurality of image acquisition devices, and the three-dimensional position information of the plurality of guide marks to guide the driver looking at the target The eye sight data and head pose data at the time of marking include:

Determine the three-dimensional position information of the face key point when the driver looks at the target guide mark according to the image data, and the three-dimensional position information of the face key point includes the three-dimensional position information of the center point of the human eye and the three-dimensional position of the center point of the head information;

According to the three-dimensional position information of the face key points, the three-dimensional position information of the plurality of image acquisition devices, and the three-dimensional position information of the target guide mark, determine the line of sight data of the human eye when the driver looks at the target guide mark and Head pose data.
The method according to claim 22, wherein the number of the plurality of image capturing devices is n, and the corresponding image data includes n images;

Determining, according to the image data, the three-dimensional position information of the face key points when the driver looks at the target guidance mark includes:

Restore the scene depth map according to the n images;

Determine the two-dimensional position information of the face key point in the ith image, and the ith image corresponds to the ith image acquisition device;

According to the scene depth map and the two-dimensional position information of the face key points in the ith image, determine the face of the driver in the coordinate system of the ith image acquisition device when the driver looks at the target guide mark 3D position information of key points.
The method according to claim 18, wherein the plurality of image capturing devices comprises a first image capturing device and a second image capturing device, and the positions of the first image capturing device and the second image capturing device are It is determined according to the spatial distribution of the plurality of guide marks.
The method of claim 18, wherein the plurality of guide marks comprises a plurality of guide marks in front of the cockpit and a plurality of guide marks on the left and right sides of the cockpit.
26. The method of claim 25, wherein the plurality of guide markings on the front of the cockpit include one or more guide markings set on a windshield and one or more guide markings set on a dashboard guide marker.
The method according to claim 26, wherein the plurality of guide marks in front of the cockpit are arranged on the front windshield and on the instrument panel using discrete patches.
The method according to claim 25, wherein the plurality of guide marks on the left and right sides of the cockpit include one or more guide marks arranged on the left and right front door glass, and a plurality of guide marks arranged on the left and right One or more guide markings on the rear view mirror.
The method according to claim 28, wherein a plurality of guide marks on the left and right sides of the cockpit are arranged on the left and right front door glass and on the left and right rear view mirrors using discrete patches.
The method according to claim 25, wherein the front windshield of the cockpit is a transparent display screen, and a plurality of guide marks in front of the cockpit are presented on the transparent display screen according to a set rule; A plurality of guide marks on the left and right sides of the cockpit are arranged on the left and right front door glass and on the left and right rear-view mirrors using discrete patches.
The method according to claim 25, wherein a plurality of guide marks in front of the cockpit are presented according to a set rule through a display screen arranged on the front and outer side of the cockpit; Each guide mark is set in discrete patches on the left and right front door glass and on the left and right rear view mirrors.
The method according to any one of claims 18-31, wherein the plurality of image capturing devices are installed in the cockpit or outside the cockpit.
The method according to any one of claims 18-31, wherein the monitoring data processing device is a mobile control terminal, the mobile control terminal is communicatively connected to the plurality of image acquisition devices, and the mobile control terminal It is also used for controlling the plurality of image capturing devices to take pictures in response to the operation of the driver.
The method of claim 18, further comprising:

Before collecting the image data when the driver looks at the target guide mark through the plurality of image collection devices, calibrate the plurality of image collection devices and the plurality of guide marks in the same coordinate system to obtain a calibration result;

After collecting the image data when the driver looks at the target guide mark through the plurality of image acquisition devices, the monitoring data processing device determines the person when the driver looks at the target guide mark according to the calibration result and the image data Eye gaze data and head pose data.
The method according to claim 34, wherein the obtaining a calibration result by calibrating the plurality of image acquisition devices and the plurality of guide marks in the same coordinate system comprises:

Perform internal parameter calibration on the plurality of image acquisition devices, and perform external parameter calibration among the plurality of image acquisition devices;

selecting one image acquisition device among the plurality of image acquisition devices as a reference image acquisition device;

The plurality of guide marks are calibrated according to the coordinate system where the reference image acquisition device is located.
The method according to claim 35, wherein the calibrating the plurality of guide marks according to the coordinate system where the reference image acquisition device is located comprises:

Execute on each of the plurality of guide markers:

Arrange preset points at the guide marks to calibrate the image acquisition device;

Perform external parameter calibration on the reference image acquisition device and the preset point calibration image acquisition device to complete the calibration of the guide marks.
The method according to claim 35, wherein the calibrating the plurality of guide marks according to the coordinate system where the reference image acquisition device is located comprises:

selecting one guide marker from the plurality of guide markers as a reference preset guide point;

Arranging preset points at the reference preset guide points to calibrate the image acquisition device;

performing external parameter calibration on the reference image acquisition device and the preset point calibration image acquisition device to complete the calibration of the reference preset guide point;

Other guide marks in the plurality of guide marks are calibrated according to the calibration result of the reference preset guide point and the relative positional relationship between the plurality of guide marks.
The method according to claim 34, wherein the obtaining a calibration result by calibrating the plurality of image acquisition devices and the plurality of guide marks in the same coordinate system comprises:

Perform internal parameter calibration on the plurality of image acquisition devices, and perform external parameter calibration among the plurality of image acquisition devices;

selecting one image acquisition device among the plurality of image acquisition devices as a reference image acquisition device;

The calibration of the plurality of guide marks in the coordinate system of the reference image acquisition device is completed by means of pre-measurement.
The method according to claim 34, wherein the obtaining a calibration result by calibrating the plurality of image acquisition devices and the plurality of guide marks in the same coordinate system comprises:

Perform internal parameter calibration on the plurality of image acquisition devices, and perform external parameter calibration among the plurality of image acquisition devices;

selecting one image acquisition device among the plurality of image acquisition devices as a reference image acquisition device;

The calibration of the plurality of guide markers in the coordinate system of the reference image acquisition device is completed by constructing a cockpit simulation model.
A driver monitoring model training method, comprising:

Using the method of claims 18-39 to collect driver data to construct a training data set;

The driver monitoring model is trained using the training data set.
The method according to claim 40, wherein the driver monitoring model adopts a convolutional neural network model.
An assisted driving system, characterized in that, a driver monitoring model trained according to the method of claim 40 or 41 is configured.
A vehicle equipped with the driving assistance system according to claim 42 .