WO2022110737A1

WO2022110737A1 - Vehicle anticollision early-warning method and apparatus, vehicle-mounted terminal device, and storage medium

Info

Publication number: WO2022110737A1
Application number: PCT/CN2021/097281
Authority: WO
Inventors: 李佳琳; 李昌昊; 王健宗
Original assignee: 平安科技（深圳）有限公司
Priority date: 2020-11-25
Filing date: 2021-05-31
Publication date: 2022-06-02
Also published as: CN112489425A

Abstract

A vehicle anticollision early-warning method and apparatus, a vehicle-mounted terminal device, and a storage medium. The vehicle anticollision early-warning method comprises: acquiring state data of a driver and traveling data of a vehicle (101); inputting the state data and the traveling data into a pre-trained anticollision early-warning model, and predicting, by means of the anticollision early-warning model, whether the vehicle is likely to collide (102), the anticollision early-warning model being a neural network model obtained by training using the corresponding state data of the driver and traveling data of the vehicle at the time of a vehicle collision as a training set; and if the vehicle is likely to collide, determining an early-warning mode according to the state data, and giving an early warning to the driver according to the early-warning mode (103). By means of the vehicle anticollision early-warning method, a suitable early-warning mode can be made according to the state of the driver, thereby further improving the driving safety of the vehicle.

Description

Vehicle collision avoidance warning method, device, vehicle terminal device and storage medium

This application claims the priority of the Chinese patent application filed on November 25, 2020 with the application number 202011337072.7 and the invention titled "Vehicle Collision Avoidance Warning Method, Device, Vehicle Terminal Equipment and Storage Medium", the entire contents of which are Incorporated herein by reference.

technical field

The present application belongs to the technical field of artificial intelligence, and in particular relates to a vehicle collision avoidance warning method, device, vehicle terminal equipment and storage medium.

Background technique

With the continuous increase in the number of vehicles, it is convenient for the public to travel, but also brings huge security risks. The endless traffic accidents seriously threaten the lives and property safety of the people. In response to this problem, many vehicles will install a vehicle collision avoidance warning system on the vehicle terminal equipment. When the driver drives the vehicle, the system can provide services such as driving monitoring and danger warning.

Specifically, through the vehicle collision avoidance warning system, before the vehicle is about to collide, it can issue an audible or visual alarm to the driver according to data such as distance and vehicle speed, thereby reminding the possible collision event. However, the inventors realized that, at some point, the driver may be aware of the danger, and the warning issued by the collision avoidance warning system will distract the driver's attention, leading to the occurrence of a collision accident.

technical problem

In view of this, the present application proposes a vehicle collision avoidance warning method, device, on-board terminal device and storage medium, which can formulate an appropriate warning mode according to the driver's state and further improve the safety of vehicle driving.

technical solutions

In a first aspect, an embodiment of the present application provides a vehicle collision avoidance warning method, including:

obtaining the state data of the driver of the vehicle and the driving data of the vehicle;

Input the state data and the driving data into a pre-trained collision avoidance warning model, and predict whether the vehicle may collide through the collision avoidance warning model. The driver state data and vehicle driving data are used as the training set to train the neural network model;

If the vehicle may collide, an early warning mode is determined according to the state data, and the vehicle driver is warned according to the early warning mode.

In an embodiment of the present application, the acquiring the state data of the vehicle driver may include:

Collect the head image of the driver of the vehicle through a camera;

The head image is input into the pre-trained head pose estimation model, and the head pose data of the vehicle driver is obtained through the head pose estimation model. face key points to determine head pose data;

The eye movement track data of the vehicle driver is collected by an eye tracker.

Further, the determining an early warning mode according to the state data may include:

Determine the gaze area of the vehicle driver according to the head posture data and the eye movement trajectory data;

A corresponding warning mode is determined according to the gaze area and the vehicle collision position predicted by the collision avoidance warning model.

Studies have shown that a person's prediction of gaze comes from a combination of head pose and eye orientation. Since the eye fixation points have been collected by the eye tracker, after the head posture is judged, a higher-precision fixation area judgment can be completed in combination with the eye fixation coordinate data. After the driver's gaze area is determined, a corresponding early warning mode can be determined according to the relative relationship between the vehicle collision position predicted by the model and the gaze area.

Further, determining the gaze area of the vehicle driver according to the head posture data and the eye movement trajectory data may include:

The maximum field of vision area of the vehicle driver in front of the vehicle is obtained by calculating according to the head posture data;

The gaze area is obtained by locating the area of the largest visual field in combination with the eye movement track data.

Further, according to the gaze area and the collision position predicted by the collision avoidance early warning model, determine the corresponding early warning mode, which may include:

If the gaze area continues to cover the collision position within a preset time period, determining that the early warning mode is the first mode;

If the gaze area is switched back and forth between the collision position and other positions within a preset time period, determining that the early warning mode is the second mode;

If the gaze area does not cover the collision position, it is determined that the early warning mode is a third mode.

Further, performing an early warning on the vehicle driver according to the early warning mode may include:

If the warning mode is the first mode, no warning prompt of any form is output;

If the pre-warning mode is the second mode, project the preset pre-warning information to the front glass of the vehicle through the projector;

If the pre-warning mode is the third mode, the projector is used to project the preset pre-warning information to the front glass of the vehicle, and the buzzer of the vehicle is controlled to play a warning sound.

In an embodiment of the present application, the collision avoidance warning model can be obtained by training in the following manner:

obtaining sample data, where the sample data includes driver state data and vehicle driving data corresponding to the collision of the vehicle;

The sample data is input into an automatic machine learning module for model design and training, and the collision avoidance warning model is obtained.

In a second aspect, an embodiment of the present application provides a vehicle collision avoidance warning device, including:

a data acquisition module for acquiring the state data of the vehicle driver and the driving data of the vehicle;

The collision prediction module is used to input the state data and the driving data into a pre-trained collision avoidance warning model, and predict whether the vehicle may collide through the collision avoidance warning model, and the collision avoidance warning model is based on the following: The corresponding driver state data and vehicle driving data when the vehicle collides are used as the neural network model trained by the training set;

An early warning module, configured to determine an early warning mode according to the state data if the vehicle may collide, and give an early warning to the driver of the vehicle according to the early warning mode.

In a third aspect, an embodiment of the present application provides an in-vehicle terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, where the processor executes the computer program When implementing the steps of the vehicle collision avoidance warning method proposed in the first aspect of the embodiments of the present application.

In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, where the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the implementation of the first aspect of the embodiment of the present application is implemented. The steps of the vehicle collision avoidance warning method.

In a fifth aspect, the embodiments of the present application provide a computer program product, which, when the computer program product runs on a terminal device, enables the terminal device to execute the steps of the vehicle collision avoidance warning method described in the first aspect of the embodiments of the present application.

beneficial effect

In the embodiment of the present application, the state data of the vehicle driver and the driving data of the vehicle are input into the designed collision avoidance warning model, which can predict whether the vehicle may collide. Then, according to the determined early warning mode, the vehicle driver is warned to further improve the safety of vehicle driving.

Description of drawings

1 is a flowchart of a first embodiment of a vehicle collision avoidance warning method provided by an embodiment of the present application;

FIG. 2 is a flowchart of a second embodiment of a vehicle collision avoidance warning method provided by an embodiment of the present application;

3 is a structural diagram of an embodiment of a vehicle collision avoidance warning device provided by an embodiment of the present application;

FIG. 4 is a schematic diagram of a vehicle-mounted terminal device provided by an embodiment of the present application.

Embodiments of the present invention

In the following description, for the purpose of illustration rather than limitation, specific details such as a specific system structure and technology are set forth in order to provide a thorough understanding of the embodiments of the present application. However, it will be apparent to those skilled in the art that the present application may be practiced in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, devices, circuits, and methods are omitted so as not to obscure the description of the present application with unnecessary detail. In addition, in the description of the specification of the present application and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the description, and should not be construed as indicating or implying relative importance.

The present application proposes a vehicle collision avoidance warning method, device, on-board terminal device and storage medium, which can formulate an appropriate warning mode according to the driver's state and further improve the safety of vehicle driving. It should be understood that the execution subject of the vehicle collision avoidance warning method proposed by each embodiment of the present application is various types of vehicle-mounted terminal devices.

Referring to FIG. 1, the first embodiment of a vehicle collision avoidance warning method in the embodiment of the present application includes:

101. Acquire state data of a vehicle driver and driving data of the vehicle;

First, the in-vehicle terminal device obtains the status data of the vehicle driver and the driving data of the vehicle. Among them, the state data of the vehicle driver may include head posture data, eye movement trajectory data, hand motion data, human heart rate data and other data; the driving data of the vehicle may include vehicle driving speed, driving direction, vehicle position, horizontal Longitudinal acceleration data, steering wheel status data, vehicle infrared ranging data and other data.

Specifically, the in-vehicle terminal device can acquire the driving data of the vehicle, such as data such as driving speed and acceleration, by docking with an electronic control unit (ECU) of the vehicle. In addition, the driver's head posture can also be obtained through devices such as cameras installed in the carriage, and the eye movement trajectory data can be obtained through an eye tracker (a device used to record the characteristics of human eye movements when processing visual information). The wearable device obtains the driver's human heart rate data, and so on. The data obtained by different devices such as cameras, eye trackers, and wearable devices are sent to the vehicle terminal for processing.

102. Input the state data and the driving data into a pre-trained collision avoidance warning model, and use the collision avoidance warning model to predict whether the vehicle may collide;

After the vehicle terminal device obtains the driver's state data and the vehicle's driving data, it will input this part of the data into a pre-trained anti-collision warning model, and output the result of whether the vehicle may collide through the model. Specifically, the collision avoidance warning model is a neural network model trained by using the driver state data and vehicle driving data corresponding to the collision of the vehicle as a training set, and the model can be trained in the following ways:

(1) Obtain sample data, where the sample data includes driver state data and vehicle driving data corresponding to the collision of the vehicle;

(2) Inputting the sample data into an automatic machine learning module to design and train a model to obtain the collision avoidance warning model.

When obtaining the sample data for training the collision avoidance warning model, the simulator can be used to simulate the real driving environment, and different collision scenarios can be set, such as vehicle rear-end collision, pedestrian crossing the road, side impact and other possible situations, and at the same time record driving under different conditions member's response. When the system predicts that a collision may occur at a later time, the system will record the driver's gaze coordinate data within a certain period of time before the possible collision, as well as the driving data (horizontal and longitudinal speed and acceleration of the vehicle), driver's The heart rate data of the driver can be used to judge the driving status of the driver during this period, including the gaze area and activity level, the degree of self-excitedness, and the driving speed. Specifically, the multi-camera inside the vehicle can be used to obtain the image of the driver's head or facial posture, the eye tracker can be used to obtain the driver's eye movement trajectory data or gaze pattern data, and the wrist wearable device can be used to obtain the driver's heart rate data. Obtain various driving data of the vehicle by docking the vehicle ECU, and so on.

Next, the collected data is fed into the Automatic Machine Learning (AutoML) module for model design and training. The release of traditional anti-collision warning signals mostly relies on certain established formulas. By bringing in the driving speed, the distance data of infrared detection and the driver's gaze coordinate data, a limit braking distance is calculated and compared with the set braking distance. Threshold completes judgment of warning release. The difference between this application is that a variety of different data can be added, including the driver's facial posture (used to assist in judging the gaze area), the coordinates of the gaze point (used to determine the gaze area), and the dwell time of each gaze coordinate (used to determine the gaze area) attention level), heart rate (used to confirm the driver’s excitement, which is roughly proportional to the level of concentration), and data such as driving speed and distance. Therefore, this application introduces the automatic continuous learning (AutoML) technology for model training, so as to solve the problem that a single machine learning algorithm is difficult to adapt to multi-source heterogeneous data. The sample data is normalized in the AutoML layer, and the model is trained based on whether the collision is avoided as a result. By collecting large-scale, multi-type driver simulation training data, the training set and the test set are divided, so as to complete the The training of the collision avoidance warning model.

103. If the vehicle may collide, determine an early warning mode according to the state data, and give an early warning to the driver of the vehicle according to the early warning mode.

Through the collision avoidance warning model, the result of whether the vehicle may collide within a certain period of time can be predicted. If the vehicle is not in danger of collision, the on-board terminal equipment does not need to give an early warning. If it is predicted that the vehicle is in danger of collision, the in-vehicle terminal device will determine a corresponding early warning mode according to the driver's state data, and give an early warning to the vehicle driver according to the determined early warning mode. For example, if it is detected that the driver has been looking at the location where the collision is about to occur, and the driver's heart rate is high, it indicates that the driver is aware of the danger, and a lower level of warning mode can be used at this time (for example, the hazard indicator light is on, or the output If it is detected that the driver does not observe the position where the collision is about to occur, the driver's heart rate is flat, or the driver is in a sleepy state, a higher degree of early warning mode (such as a high-decibel speaker playing warning information) is used.

Referring to FIG. 2 , a second embodiment of a vehicle collision avoidance warning method in the embodiment of the present application includes:

201. Acquire state data of a vehicle driver and driving data of the vehicle;

In the embodiment of the present application, the obtaining the state data of the vehicle driver includes:

(1) Collect the head image of the driver of the vehicle through a camera;

(2) Input the head image into a pre-trained head pose estimation model, obtain the head pose data of the vehicle driver through the head pose estimation model, and the head pose estimation model detects the head face key points in the external image to determine the head pose data;

(3) Collect the eye movement trajectory data of the vehicle driver through an eye tracker.

The head image of the vehicle driver can be captured by the camera installed in the cabin, and then the obtained head image is input into a pre-trained head pose estimation model, which can detect the head image by detecting the head pose estimation model. face key points in to determine the driver's head pose data. The head posture can be divided into relatively static and moving from the motion state, and the movement can be further subdivided into various action modes such as raising the head, turning the head and shaking the head. Its use can also be divided into two aspects here, including the detection of the driver's fatigue state (such as long-term head still state and eye blink frequency to determine whether it is fatigue driving) and assisting to complete higher-precision eye tracking (using for determining the driver's gaze area).

The construction of the head pose estimation model is mainly through the detection of multiple 2D key points in the head image collected in real time (including key points such as the corner of the eye, the tip of the nose, the corner of the mouth, the chin, etc., the number of which varies with the algorithm, and multiple detection points can bring higher accuracy, but also increase the amount of calculation), and based on the face image matching the 3D face model with the highest degree of fit (there are many face models in existing algorithm models that can be used for matching and fitting), this The 3D face model is used as the head attitude judgment model of the driver, and then the conversion relationship between the 3D point and the corresponding collected 2D point is solved to calculate the three different Euler angles of pitch angle, yaw angle and roll angle. Corresponding to looking up, turning head, and shaking head movements, respectively. For example, if the model finds that the yaw angle has a large change in a certain period of time, and exceeds the set change threshold, then it is judged that the driver has a relatively large head turning action, and the corresponding head posture is "turning head in motion". ".

The eye tracker is a device used to record the features of the eye track of a person when processing visual information. The eye tracker can be used to collect the eye track data of the vehicle driver to more accurately determine the vehicle driver's gaze area. In practical applications, the eye tracker can be set at a designated position inside the vehicle. After the eye tracker collects the eye movement trajectory data of the vehicle driver, this part of the data is sent to the vehicle terminal device for subsequent processing.

202. Input the state data and the driving data into a pre-trained collision avoidance warning model, and use the collision avoidance warning model to predict whether the vehicle may collide;

The state data including head posture data and eye movement trajectory data, and the driving data of the vehicle are input into the collision avoidance warning model, and the model is used to predict whether the vehicle may collide. For the training process and working principle of the collision avoidance warning model, reference may be made to the relevant description of the previous embodiment.

203. If the vehicle may collide, determine the gaze area of the vehicle driver according to the head posture data and the eye movement trajectory data;

If it is predicted that the vehicle may collide, the in-vehicle terminal device will determine the current gaze area of the vehicle driver according to the vehicle driver's head posture data and eye movement trajectory data. Studies have shown that a person's prediction of gaze comes from a combination of head pose and eye orientation. Since the eye fixation points have been collected by the eye tracker, after the head posture is judged, a higher-precision fixation area judgment can be completed in combination with the eye fixation coordinate data.

(1) Calculate the maximum field of view area of the vehicle driver in front of the vehicle according to the head posture data;

(2) The gaze area is obtained by locating the gaze area from the largest visual field area in combination with the eye movement trajectory data.

Specifically, the calculation model of the front view range data corresponding to different head postures can be completed according to the pre-collected human visual area data in the real scene, and then each deflection angle calculated based on the head posture, namely the pitch angle, The yaw angle and roll angle are calculated by applying this calculation model to obtain the driver's maximum field of vision area or central gaze area in front of the vehicle (including key areas of interest such as the front and rearview mirrors) under the current head posture. Based on the head posture and the position information of the key points of the eyes, the approximate range of eye gaze of the 3D face model can be obtained, so as to obtain the driver's maximum field of view area. Combined with the eye movement trajectory data for relocation, the driver at this moment can be calculated. gaze area.

204. Determine a corresponding warning mode according to the gaze area and the vehicle collision position predicted by the collision avoidance warning model;

Through the collision avoidance early warning model, it is not only possible to predict whether the vehicle may collide, but also further predict the possible collision position of the vehicle when it is predicted that the vehicle may collide. After the driver's gaze area is determined, a corresponding early warning mode can be determined according to the relative relationship between the vehicle collision position predicted by the model and the gaze area.

Specifically, step 204 may include:

(1) If the gaze area continues to cover the collision position within a preset time period, then determine that the early warning mode is the first mode;

(2) If the gaze area switches back and forth between the collision position and other positions within a preset time period, determine that the early warning mode is the second mode;

(3) If the gaze area does not cover the collision position, determine that the early warning mode is the third mode.

For the above step (1), if the gaze area continues to cover the collision position predicted by the model within the preset time period, it means that the driver's eye gaze point is concentrated on the position where the collision may occur for a long time before the warning, and the driver can be judged at this time. The operator has realized the danger of collision and learned the possible collision position, and then enters the first warning mode. Since the driver is aware of the danger, it is not appropriate to perform a severe warning at this time to avoid distracting the driver, so the first warning mode may be a light warning mode. In addition, when judging the early warning mode, data such as the driver's heart rate and vehicle speed can also be combined. For example, when it is detected that the gaze area continues to cover the collision position, it is further detected that the driver's heart rate is high and the vehicle speed is reduced. The first warning mode.

For the above step (2), if the gaze area switches back and forth between the collision position and other positions within the preset time period, it indicates that the driver may have been aware of the danger, but did not know the possible collision position. Warning mode. The second warning mode can be a moderate warning mode. When judging the warning mode, data such as the driver's heart rate and vehicle speed can also be combined. For example, if it is detected that the driver's head posture changes frequently, the driver's heart rate is high and the vehicle speed is reduced, Then it is determined to enter the second early warning mode.

For the above step (3), if the gaze area does not cover the collision position, it means that the driver is not aware of the danger, and the third warning mode is entered at this time. Since the driver is not aware of the danger and needs to perform a high warning at this time, the third warning mode can be a severe warning mode. In addition, when judging the early warning mode, data such as the driver's heart rate and vehicle speed can also be combined. For example, if it is detected that the driver's head posture is basically unchanged, the driver's heart rate is stable and the vehicle speed has not changed, it is determined to enter the third early warning model.

205. Provide an early warning to the vehicle driver according to the early warning mode.

After the corresponding early warning mode is determined, the vehicle driver can be warned according to the corresponding early warning mode.

Specifically, step 205 may include:

(1) If the warning mode is the first mode, no warning prompts of any form are output;

(2) If the warning mode is the second mode, project the preset warning information to the front glass of the vehicle through the projector;

(3) If the warning mode is the third mode, project the preset warning information to the front glass of the vehicle through the projector, and control the buzzer of the vehicle to play the warning sound.

The first mode is a mild early warning mode. At this time, only simple warning information can be displayed on the display screen of the vehicle terminal, or no warning prompt of any form can be output. The second mode is a moderate early warning mode. At this time, preset warning information can be projected to the front glass through a projector set in the vehicle to indicate the time and location of a possible collision. The third mode is the severe warning mode. At this time, the preset warning information can be projected to the front glass through the projector set in the car, and the designated buzzer can be controlled to play the warning sound to remind the driver of the danger of impending collision.

The embodiment of the present application modifies the traditional vehicle collision avoidance warning system, collects the driver's eye activity data by adding an eye movement detector, collects facial posture through a camera device, collects physiological data through a wearable device, etc., and uses automatic machine learning. The method conducts model training on various types of data collected to realize a more intelligent vehicle early warning system. The system can send out appropriate warning information according to the driver's different gaze conditions, and effectively avoid traffic accidents caused by distraction caused by inappropriate warnings.

It should be understood that the size of the sequence numbers of the steps in the above embodiments does not mean the sequence of execution, and the execution sequence of each process should be determined by its function and internal logic, and should not constitute any limitation to the implementation process of the embodiments of the present application.

Corresponding to the vehicle collision avoidance warning method described in the above embodiment, FIG. 3 shows a structural block diagram of a vehicle collision avoidance warning device provided by the embodiment of the present application. relevant part.

Referring to Figure 3, the device includes:

A data acquisition module 301, configured to acquire the state data of the vehicle driver and the driving data of the vehicle;

A collision prediction module 302, configured to input the state data and the driving data into a pre-trained collision avoidance warning model, and predict whether the vehicle may collide through the collision avoidance warning model, and the collision avoidance warning model is: The neural network model obtained by training the corresponding driver state data and vehicle driving data when the vehicle collided as the training set;

An early warning module 303 is configured to determine an early warning mode according to the state data if the vehicle may collide, and give an early warning to the driver of the vehicle according to the early warning mode.

Further, the data acquisition module may include:

a head image acquisition unit, configured to collect the head image of the driver of the vehicle through a camera;

a head pose estimation unit, configured to input the head image into a pre-trained head pose estimation model, and obtain the head pose data of the vehicle driver through the head pose estimation model, and the head pose The estimation model determines the head pose data by detecting the key points of the face in the head image;

The eye movement trajectory acquisition unit is used for collecting the eye movement trajectory data of the vehicle driver through the eye tracker.

Further, the early warning module may include:

A gaze area determination unit, configured to determine the gaze area of the vehicle driver according to the head posture data and the eye movement trajectory data;

An early warning mode determination unit, configured to determine a corresponding early warning mode according to the gaze area and the vehicle collision position predicted by the collision avoidance early warning model.

Further, the gaze area determination unit may include:

a maximum field of view area determination subunit, configured to calculate and obtain the maximum field of view area of the vehicle driver in front of the vehicle according to the head posture data;

The gaze area determination subunit is configured to locate the gaze area from the largest visual field area in combination with the eye movement track data.

Further, the early warning mode determination unit may include:

a first mode determination subunit, configured to determine that the early warning mode is the first mode if the gaze area continues to cover the collision position within a preset time period;

a second mode determination subunit, configured to determine that the early warning mode is the second mode if the gaze area switches back and forth between the collision position and other positions within a preset time period of the gaze area;

A third mode determination subunit is configured to determine that the early warning mode is a third mode if the gaze area does not cover the collision position.

Further, the early warning module may include:

a first warning unit, configured to not output any form of warning prompt if the warning mode is the first mode;

a second pre-warning unit, configured to project preset pre-warning information to the front glass of the vehicle through a projector if the pre-warning mode is the second mode;

The third warning unit is used for projecting preset warning information to the front glass of the vehicle through the projector if the warning mode is the third mode, and controlling the buzzer of the vehicle to play a warning sound.

Further, the vehicle collision avoidance warning device may further include:

a sample data acquisition module, configured to acquire sample data, where the sample data includes driver state data and vehicle driving data corresponding to the collision of the vehicle;

An early warning model training module is used to input the sample data into an automatic machine learning module for model design and training to obtain the collision avoidance early warning model.

Embodiments of the present application further provide a computer-readable storage medium, where computer-readable instructions are stored in the computer-readable storage medium, and when the computer-readable instructions are executed by a processor, any one of the instructions shown in FIG. 1 or FIG. 2 is implemented. Steps of a vehicle collision avoidance warning method.

Embodiments of the present application also provide a computer program product, which, when the computer program product runs on the server, causes the server to execute the steps of implementing any one of the methods for vehicle collision avoidance warning as shown in FIG. 1 or FIG. 2 .

Embodiments of the present application further provide an in-vehicle terminal device, including a memory, a processor, and computer-readable instructions stored in the memory and executable on the processor, where the processor executes the computer-readable instructions At the same time, the steps of any vehicle collision avoidance warning method shown in FIG. 1 or FIG. 2 are realized.

FIG. 4 is a schematic diagram of a vehicle terminal device provided by an embodiment of the present application. As shown in FIG. 4 , the in-vehicle terminal device 4 of this embodiment includes: a processor 40 , a memory 41 , and computer-readable instructions 42 stored in the memory 41 and executable on the processor 40 . When the processor 40 executes the computer-readable instructions 42 , the steps in each of the foregoing embodiments of the vehicle collision avoidance warning method are implemented, for example, steps 101 to 103 shown in FIG. 1 . Alternatively, when the processor 40 executes the computer-readable instructions 42, the functions of the modules/units in each of the foregoing apparatus embodiments, such as the functions of the modules 301 to 303 shown in FIG. 3, are implemented.

Exemplarily, the computer-readable instructions 42 may be divided into one or more modules/units, and the one or more modules/units are stored in the memory 41 and executed by the processor 40, to complete this application. The one or more modules/units may be a series of computer-readable instruction segments capable of accomplishing specific functions, and the instruction segments are used to describe the execution process of the computer-readable instructions 42 in the vehicle-mounted terminal device 4 .

The in-vehicle terminal device 4 may be a computing device such as a smart phone, a notebook, a palmtop computer, and a cloud in-vehicle terminal device. The in-vehicle terminal device 4 may include, but is not limited to, a processor 40 and a memory 41 . Those skilled in the art can understand that FIG. 4 is only an example of the in-vehicle terminal device 4 , and does not constitute a limitation on the in-vehicle terminal device 4 , and may include more or less components than those shown in the figure, or combine some components, or different For example, the in-vehicle terminal device 4 may also include input and output devices, network access devices, buses, and the like.

The processor 40 may be a central processing unit (CentraL Processing Unit, CPU), or other general-purpose processors, digital signal processors (Digital Signal Processor, DSP), application-specific integrated circuits (Application Specific Integrated Circuit, ASIC), off-the-shelf Programmable Gate Array (FieLd-ProgrammabLe Gate Array, FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.

The memory 41 may be an internal storage unit of the in-vehicle terminal device 4 , such as a hard disk or a memory of the in-vehicle terminal device 4 . The memory 41 may also be an external storage device of the on-board terminal device 4, such as a plug-in hard disk equipped on the on-board terminal device 4, a smart memory card (Smart memory card). Media Card, SMC), Secure Digital (Secure Digital, SD) card, flash memory card (FLash Card), etc. Further, the memory 41 may also include both an internal storage unit of the in-vehicle terminal device 4 and an external storage device. The memory 41 is used to store the computer-readable instructions and other programs and data required by the vehicle-mounted terminal device. The memory 41 can also be used to temporarily store data that has been output or will be output.

It should be noted that the information exchange, execution process and other contents between the above-mentioned devices/units are based on the same concept as the method embodiments of the present application. For specific functions and technical effects, please refer to the method embodiments section. It is not repeated here.

Those skilled in the art can clearly understand that, for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned functions can be allocated to different functional units, Module completion, that is, dividing the internal structure of the device into different functional units or modules to complete all or part of the functions described above. Each functional unit and module in the embodiment may be integrated in one processing unit, or each unit may exist physically alone, or two or more units may be integrated into one unit, and the above-mentioned integrated units may adopt hardware. It can also be realized in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing from each other, and are not used to limit the protection scope of the present application. For the specific working processes of the units and modules in the above-mentioned system, reference may be made to the corresponding processes in the foregoing method embodiments, which will not be repeated here.

The integrated unit, if implemented in the form of a software functional unit and sold or used as an independent product, may be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can be implemented by a computer program to instruct the relevant hardware. The computer program can be stored in a computer-readable storage medium, and the computer program When executed by a processor, the steps of each of the above method embodiments can be implemented. Wherein, the computer program includes computer program code, and the computer program code may be in the form of source code, object code, executable file or some intermediate form, and the like. The computer-readable medium may include at least: any entity or device capable of carrying computer program codes to the photographing device/terminal device, recording medium, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electrical carrier signals, telecommunication signals, and software distribution media. For example, U disk, mobile hard disk, disk or CD, etc.

In the foregoing embodiments, the description of each embodiment has its own emphasis. For parts that are not described or described in detail in a certain embodiment, reference may be made to the relevant descriptions of other embodiments.

The above-mentioned embodiments are only used to illustrate the technical solutions of the present application, but not to limit them; although the present application has been described in detail with reference to the above-mentioned embodiments, those of ordinary skill in the art should understand that: it can still be used for the above-mentioned implementations. The technical solutions described in the examples are modified, or some technical features thereof are equivalently replaced; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions in the embodiments of the application, and should be included in the within the scope of protection of this application.

Claims

A vehicle collision avoidance warning method, comprising:

obtaining the state data of the driver of the vehicle and the driving data of the vehicle;

Input the state data and the driving data into a pre-trained collision avoidance warning model, and predict whether the vehicle may collide through the collision avoidance warning model. The driver state data and vehicle driving data are used as the training set to train the neural network model;

If the vehicle may collide, an early warning mode is determined according to the state data, and the vehicle driver is warned according to the early warning mode.
The vehicle collision avoidance warning method according to claim 1, wherein the acquiring the state data of the vehicle driver comprises:

Collect the head image of the driver of the vehicle through a camera;

The head image is input into the pre-trained head pose estimation model, and the head pose data of the vehicle driver is obtained through the head pose estimation model. face key points to determine head pose data;

The eye movement track data of the vehicle driver is collected by an eye tracker.
The vehicle collision avoidance warning method according to claim 2, wherein the determining an early warning mode according to the state data comprises:

Determine the gaze area of the vehicle driver according to the head posture data and the eye movement trajectory data;

A corresponding warning mode is determined according to the gaze area and the vehicle collision position predicted by the collision avoidance warning model.
The vehicle collision avoidance warning method according to claim 3, wherein determining the gaze area of the vehicle driver according to the head posture data and the eye movement trajectory data, comprising:

The maximum field of vision area of the vehicle driver in front of the vehicle is obtained by calculating according to the head posture data;

The gaze area is obtained by locating the area of the largest visual field in combination with the eye movement track data.
The vehicle collision avoidance warning method according to claim 3, wherein determining the corresponding warning mode according to the gaze area and the collision position predicted by the collision avoidance warning model, comprising:

If the gaze area continues to cover the collision position within a preset time period, determining that the early warning mode is the first mode;

If the gaze area is switched back and forth between the collision position and other positions within a preset time period of the gaze area, determining that the early warning mode is the second mode;

If the gaze area does not cover the collision position, it is determined that the early warning mode is a third mode.
The vehicle collision avoidance warning method according to claim 5, wherein the warning to the vehicle driver according to the warning mode comprises:

If the warning mode is the first mode, no warning prompt of any form is output;

If the pre-warning mode is the second mode, project the preset pre-warning information to the front glass of the vehicle through the projector;

If the pre-warning mode is the third mode, the projector is used to project the preset pre-warning information to the front glass of the vehicle, and the buzzer of the vehicle is controlled to play a warning sound.
The vehicle collision avoidance warning method according to any one of claims 1 to 6, wherein the collision avoidance warning model is obtained by training in the following manner:

obtaining sample data, where the sample data includes driver state data and vehicle driving data corresponding to the collision of the vehicle;

The sample data is input into an automatic machine learning module for model design and training, and the collision avoidance warning model is obtained.
A vehicle collision avoidance warning device, comprising:

a data acquisition module for acquiring the state data of the vehicle driver and the driving data of the vehicle;

The collision prediction module is used to input the state data and the driving data into a pre-trained collision avoidance warning model, and predict whether the vehicle may collide through the collision avoidance warning model, and the collision avoidance warning model is based on the following: The corresponding driver state data and vehicle driving data when the vehicle collides are used as the neural network model trained by the training set;

An early warning module, configured to determine an early warning mode according to the state data if the vehicle may collide, and give an early warning to the driver of the vehicle according to the early warning mode.
A terminal device, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor implements the following steps when executing the computer program:

obtaining the state data of the driver of the vehicle and the driving data of the vehicle;

Input the state data and the driving data into a pre-trained collision avoidance warning model, and predict whether the vehicle may collide through the collision avoidance warning model. The driver state data and vehicle driving data are used as the training set to train the neural network model;

If the vehicle may collide, an early warning mode is determined according to the state data, and the vehicle driver is warned according to the early warning mode.
The terminal device according to claim 9, wherein the acquiring the state data of the vehicle driver comprises:

Collect the head image of the driver of the vehicle through a camera;

The head image is input into the pre-trained head pose estimation model, and the head pose data of the vehicle driver is obtained through the head pose estimation model. face key points to determine head pose data;

The eye movement track data of the vehicle driver is collected by an eye tracker.
The terminal device according to claim 10, wherein the determining an early warning mode according to the state data comprises:

Determine the gaze area of the vehicle driver according to the head posture data and the eye movement trajectory data;

A corresponding warning mode is determined according to the gaze area and the vehicle collision position predicted by the collision avoidance warning model.
The terminal device according to claim 11, wherein determining the gaze area of the vehicle driver according to the head posture data and the eye movement trajectory data comprises:

The maximum field of vision area of the vehicle driver in front of the vehicle is obtained by calculating according to the head posture data;

The gaze area is obtained by locating the area of the largest visual field in combination with the eye movement track data.
The terminal device according to claim 11, wherein determining a corresponding early warning mode according to the gaze area and the collision position predicted by the collision avoidance early warning model, comprising:

If the gaze area continues to cover the collision position within a preset time period, determining that the early warning mode is the first mode;

If the gaze area is switched back and forth between the collision position and other positions within a preset time period of the gaze area, determining that the early warning mode is the second mode;

If the gaze area does not cover the collision position, it is determined that the early warning mode is a third mode.
The terminal device according to any one of claims 9 to 13, wherein the collision avoidance warning model is obtained by training in the following manner:

obtaining sample data, where the sample data includes driver state data and vehicle driving data corresponding to the collision of the vehicle;

The sample data is input into an automatic machine learning module for model design and training, and the collision avoidance warning model is obtained.
A computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, the following steps are implemented:

obtaining the state data of the driver of the vehicle and the driving data of the vehicle;

Input the state data and the driving data into a pre-trained collision avoidance warning model, and predict whether the vehicle may collide through the collision avoidance warning model. The driver state data and vehicle driving data are used as the training set to train the neural network model;

If the vehicle may collide, an early warning mode is determined according to the state data, and the vehicle driver is warned according to the early warning mode.
The computer-readable storage medium of claim 15, wherein said obtaining state data of a vehicle driver comprises:

Collect the head image of the driver of the vehicle through a camera;

The head image is input into the pre-trained head pose estimation model, and the head pose data of the vehicle driver is obtained through the head pose estimation model. face key points to determine head pose data;

The eye movement track data of the vehicle driver is collected by an eye tracker.
The computer-readable storage medium of claim 16, wherein the determining an early warning mode according to the status data comprises:

Determine the gaze area of the vehicle driver according to the head posture data and the eye movement trajectory data;

A corresponding warning mode is determined according to the gaze area and the vehicle collision position predicted by the collision avoidance warning model.
The computer-readable storage medium of claim 17, wherein determining the gaze area of the vehicle driver according to the head posture data and the eye movement trajectory data comprises:

The maximum field of vision area of the vehicle driver in front of the vehicle is obtained by calculating according to the head posture data;

The gaze area is obtained by locating the area of the largest visual field in combination with the eye movement track data.
The computer-readable storage medium of claim 17, wherein determining a corresponding early warning mode according to the gaze area and the collision position predicted by the collision avoidance early warning model comprises:

If the gaze area continues to cover the collision position within a preset time period, determining that the early warning mode is the first mode;

If the gaze area is switched back and forth between the collision position and other positions within a preset time period of the gaze area, determining that the early warning mode is the second mode;

If the gaze area does not cover the collision position, it is determined that the early warning mode is a third mode.
The computer-readable storage medium according to any one of claims 15 to 19, wherein the collision avoidance warning model is obtained by training in the following manner:

obtaining sample data, where the sample data includes driver state data and vehicle driving data corresponding to the collision of the vehicle;

The sample data is input into an automatic machine learning module for model design and training, and the collision avoidance warning model is obtained.