CN114348017A - Driver monitoring method and system based on vehicle-mounted terminal and cloud analysis - Google Patents
Driver monitoring method and system based on vehicle-mounted terminal and cloud analysis Download PDFInfo
- Publication number
- CN114348017A CN114348017A CN202111555459.4A CN202111555459A CN114348017A CN 114348017 A CN114348017 A CN 114348017A CN 202111555459 A CN202111555459 A CN 202111555459A CN 114348017 A CN114348017 A CN 114348017A
- Authority
- CN
- China
- Prior art keywords
- driver
- vehicle
- cloud
- state
- video
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000012544 monitoring process Methods 0.000 title claims abstract description 130
- 238000000034 method Methods 0.000 title claims abstract description 77
- 238000004458 analytical method Methods 0.000 title claims abstract description 62
- 230000005540 biological transmission Effects 0.000 claims abstract description 25
- 238000004891 communication Methods 0.000 claims description 31
- 238000004422 calculation algorithm Methods 0.000 claims description 28
- 238000004590 computer program Methods 0.000 claims description 18
- 238000007405 data analysis Methods 0.000 claims description 17
- 230000002159 abnormal effect Effects 0.000 claims description 16
- 230000000295 complement effect Effects 0.000 abstract description 4
- 238000004364 calculation method Methods 0.000 abstract description 2
- 230000003993 interaction Effects 0.000 description 10
- 239000000243 solution Substances 0.000 description 9
- 230000006870 function Effects 0.000 description 7
- 238000005516 engineering process Methods 0.000 description 6
- 230000008569 process Effects 0.000 description 6
- 238000010586 diagram Methods 0.000 description 5
- 238000012790 confirmation Methods 0.000 description 3
- 238000013461 design Methods 0.000 description 3
- 238000012545 processing Methods 0.000 description 3
- 230000000391 smoking effect Effects 0.000 description 3
- 230000004927 fusion Effects 0.000 description 2
- 238000012806 monitoring device Methods 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 102100038409 T-box transcription factor TBX3 Human genes 0.000 description 1
- 101710167703 T-box transcription factor TBX3 Proteins 0.000 description 1
- 230000006399 behavior Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 239000003086 colorant Substances 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000005286 illumination Methods 0.000 description 1
- 238000002347 injection Methods 0.000 description 1
- 239000007924 injection Substances 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000036651 mood Effects 0.000 description 1
- 230000006855 networking Effects 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 230000035790 physiological processes and functions Effects 0.000 description 1
- 230000011664 signaling Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 238000012795 verification Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Images
Abstract
The invention relates to the technical field of intelligent driving, and provides a driver monitoring method and system based on vehicle-mounted terminal and cloud analysis. The method comprises the following steps: acquiring and analyzing driver video data to obtain a first driver state identification result; and (3) judging: if the bandwidth meets the video transmission requirement, sending the driver video data to a cloud end, and taking a second driver state identification result obtained by analyzing the driver video data by the cloud end as a driver state true value; otherwise, taking the first driver state recognition result as a driver state true value; and executing a preset automatic driving program according to the true value of the state of the driver. The vehicle-mounted terminal and the cloud terminal are used for monitoring and analyzing the video data of the driver, and the real-time state of the driver is obtained by preferentially using the cloud terminal with higher calculation and higher accuracy under the condition of better network environment; the driver monitoring controller is used in poor network environment. The two are complementary, and the monitoring accuracy and precision of the state of the driver can be greatly improved.
Description
Technical Field
The invention relates to the technical field of intelligent driving, in particular to a driver monitoring method and system based on vehicle-mounted terminal and cloud analysis.
Background
The driver state monitoring system DMS (driver Monitor System) belongs to a part of automatic driving man-machine interaction, monitors the driving behavior and physiological state of a driver by using images acquired by a camera and data input by other vehicle body sensors through technologies such as visual tracking, motion recognition and the like, and sends out an alarm or executes other safety strategies to the driver when judging that the driver is absent or in an abnormal driving state (fatigue, distraction and the like) so as to ensure the vehicle running safety.
However, limited by the computing power of the vehicle-mounted terminal, the existing DMS has a disadvantage of insufficient monitoring accuracy, and how to provide a driver monitoring method and system with higher reliability becomes a technical problem to be solved urgently in the industry.
Disclosure of Invention
The invention provides a driver monitoring method and system based on vehicle-mounted terminal and cloud analysis, which are used for overcoming the defect of insufficient monitoring accuracy in the prior art and realizing more reliable driver state monitoring.
The invention provides a driver monitoring method based on a vehicle-mounted terminal and cloud analysis, which is applied to the vehicle-mounted terminal and comprises the following steps:
acquiring and analyzing driver video data to obtain a first driver state identification result;
performing a determination for a network communication condition:
if the bandwidth meets the video transmission requirement, sending the driver video data to a cloud end, and taking a second driver state identification result which is returned by the cloud end and is obtained based on the analysis of the driver video data as a driver state true value;
if the bandwidth does not meet the video transmission requirement, taking the first driver state identification result as a driver state true value;
and executing a preset automatic driving program according to the real value of the driver state.
According to the driver monitoring method based on the vehicle-mounted terminal and the cloud analysis, the step of executing the preset automatic driving program according to the real value of the driver state comprises the following steps:
if the true value of the driver state is determined to be a first value, executing a preset automatic driving program to control the vehicle to run, and controlling a vehicle-mounted audio-video system to execute audio-video prompt;
the first value is a driver state true value when a driver is in an abnormal driving state;
the audio-visual prompt comprises any one or any combination of the following components:
displaying prompt information on an instrument panel;
displaying prompt information on a display screen;
and playing the prompt tone through the IVI.
According to the driver monitoring method based on the vehicle-mounted terminal and the cloud analysis provided by the invention, after the step of executing a preset automatic driving program to control the vehicle to run and controlling a vehicle-mounted audio-video system to execute audio-video prompt if the true value of the driver state is determined to be a first value, the method further comprises the following steps:
and if the driver state true value keeps the first value in the set time period after the vehicle-mounted audio-video system executes the audio-video prompt, executing a preset automatic driving program to control the vehicle to decelerate, and moving to a safe position to stop the vehicle.
According to the driver monitoring method based on the vehicle-mounted terminal and the cloud analysis provided by the invention, after the step of executing a preset automatic driving program to control the vehicle to run and controlling a vehicle-mounted audio-video system to execute audio-video prompt if the true value of the driver state is determined to be a first value, the method further comprises the following steps:
determining that the real value of the driver state keeps a first value in a set time period after the vehicle-mounted audio-video system executes the audio-video prompt, and controlling the vehicle to execute the physical prompt through a preset automatic driving program;
the physical cue comprises any one or any combination of the following:
controlling the vibration of a seat where a driver is located;
controlling a safety belt worn by a driver to be tightened to a set position;
and controlling the atmosphere lamp to light the set take-over color.
According to the driver monitoring method based on the vehicle-mounted terminal and the cloud analysis provided by the invention, after the step of executing a preset automatic driving program to control the vehicle to run and controlling a vehicle-mounted audio-video system to execute audio-video prompt if the true value of the driver state is determined to be a first value, the method further comprises the following steps:
determining that the driver state true value keeps a first value in a set time period after the vehicle-mounted audio-video system executes the audio-video prompt, and controlling the vehicle-mounted audio-video system to execute the take-over prompt through a preset automatic driving program until the driver takes over the vehicle control;
the take-over prompt comprises any one or any combination of the following:
displaying takeover information on an instrument panel;
displaying the takeover information on a display screen;
and playing the takeover voice through the IVI.
According to the driver monitoring method based on the vehicle-mounted terminal and the cloud analysis provided by the invention, in the set time period after the vehicle-mounted audio and video system is determined to execute the audio and video prompt, if the driver state true value keeps the first value, a preset automatic driving program is executed to control the vehicle to decelerate, and the vehicle is moved to a safe position to stop, the method further comprises the following steps:
and if the driver is determined to take over the vehicle control, stopping executing the preset automatic driving program to control the vehicle to decelerate, and moving to a safe position to stop the vehicle.
According to the driver monitoring method based on the vehicle-mounted terminal and the cloud analysis, the step of executing the preset automatic driving program according to the real value of the driver state comprises the following steps:
if the true value of the driver state is determined to be a second value, executing a preset automatic driving program to control the vehicle to run; the second value is a driver state true value when the driver is in a normal driving state.
According to the driver monitoring method based on the vehicle-mounted terminal and the cloud analysis provided by the invention, after the step of executing a preset automatic driving program to control the vehicle to run and controlling a vehicle-mounted audio-video system to execute audio-video prompt if the true value of the driver state is determined to be a first value, the method further comprises the following steps:
if the driver state true value is a second value within a set time period after the vehicle-mounted audio-video system executes the audio-video prompt, the step of controlling the vehicle-mounted audio-video system to execute the audio-video prompt is terminated, and a preset automatic driving program is executed to control the vehicle to run; the second value is a driver state true value when the driver is in a normal driving state.
According to the driver monitoring method based on the vehicle-mounted terminal and the cloud analysis provided by the invention, if the bandwidth meets the video transmission requirement, the steps of sending the video data of the driver to the cloud, and taking a second driver state identification result returned by the cloud and obtained based on the analysis of the video data of the driver as a true value of the driver state comprise:
determining that a second driver state recognition result which is returned by the cloud and obtained based on analysis of the driver video data is received, and taking the second driver state recognition result as a driver state true value;
and determining that a second driver state recognition result which is returned by the cloud and is obtained based on the analysis of the driver video data is not received, and taking the first driver state recognition result as a driver state true value.
The driver monitoring method based on the vehicle-mounted terminal and the cloud analysis further comprises the following steps:
and determining that the vehicle is in a non-running state, and adjusting a driver video data analysis algorithm deployed on the vehicle-mounted terminal according to the first driver state recognition result and the second driver state recognition result.
According to the driver monitoring method based on the vehicle-mounted terminal and the cloud analysis provided by the invention, the step of determining that the vehicle is in the non-running state and adjusting the driver video data analysis algorithm deployed on the vehicle-mounted terminal according to the first driver state identification result and the second driver state identification result comprises the following steps:
uploading the first driver state recognition result to a cloud;
determining that the vehicle is in a non-running state, and receiving a driver video data analysis algorithm update file which is returned by a cloud end through an OTA and is deployed on the vehicle-mounted terminal based on the first driver state recognition result and the second driver state recognition result;
and determining that the vehicle is in a non-running state, running the update file, and updating a driver video data analysis algorithm deployed on the vehicle-mounted terminal.
The invention also provides a driver monitoring system based on the vehicle-mounted terminal and cloud analysis, which is applied to the vehicle-mounted terminal and comprises the following components:
the acquisition module is used for acquiring and analyzing the driver video data to obtain a first driver state identification result;
a judging module, configured to perform judgment for the network communication condition:
if the bandwidth meets the video transmission requirement, sending the driver video data to a cloud end, and taking a second driver state identification result which is returned by the cloud end and is obtained based on the analysis of the driver video data as a driver state true value;
if the bandwidth does not meet the video transmission requirement, taking the first driver state identification result as a driver state true value;
and the automatic driving module is used for executing a preset automatic driving program according to the real value of the driver state.
The invention also provides an electronic device, which comprises a memory, a processor and a computer program which is stored on the memory and can run on the processor, wherein when the processor executes the program on the vehicle-mounted terminal, the steps of the driver monitoring method based on the vehicle-mounted terminal and the cloud analysis are realized.
The present invention also provides a non-transitory computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, implements the steps of the driver monitoring method based on the vehicle-mounted terminal and cloud analytics as described in any of the above.
The invention further provides a computer program product, which comprises a computer program, and the computer program is executed by a processor to realize the steps of any one of the driver monitoring methods based on the vehicle-mounted terminal and the cloud analysis.
According to the driver monitoring method and system based on the vehicle-mounted terminal and the cloud analysis, the vehicle-mounted terminal and the cloud are used for monitoring and analyzing the video data of the driver, and the real-time state of the driver is obtained by preferentially using the cloud monitoring with higher computing power and higher accuracy under the condition of better network environment; the driver monitoring controller is used in poor network environment. The two are complementary, and the monitoring accuracy and precision of the state of the driver can be greatly improved.
Drawings
In order to more clearly illustrate the technical solutions of the present invention or the prior art, the drawings needed for the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and those skilled in the art can also obtain other drawings according to the drawings without creative efforts.
Fig. 1 is a schematic flow chart of a driver monitoring method based on vehicle-mounted terminal and cloud analysis according to the present invention;
FIG. 2 is a diagram of a hardware architecture provided by an embodiment of the present invention;
FIG. 3 is a schematic workflow diagram provided by an embodiment of the present invention;
fig. 4 is a schematic structural diagram of a driver monitoring system based on vehicle-mounted terminal and cloud analysis according to the present invention;
fig. 5 is a schematic structural diagram of an electronic device provided in the present invention.
Reference numerals:
1: a driver monitoring camera;
2: a vehicle driver monitor controller;
3: a telecommunications module (T-Box);
4: a cloud processor;
5: an autonomous driving range controller;
6: a power plant;
7: a steering device;
8: a brake device;
9: vibrating the seat;
10: pre-tightening the safety belt;
11: an atmosphere lamp;
12: an entertainment information controller (IVI);
13: a meter;
14: a central control display screen;
401: an acquisition module;
402: a judgment module;
403: an automatic driving module;
510: a processor;
520: a communication interface;
530: a memory;
540: a communication bus.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings, and it is obvious that the described embodiments are some, but not all embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
The driver monitoring method based on the vehicle-mounted terminal and the cloud end analysis is described below with reference to fig. 1 to 3.
As shown in fig. 1, an embodiment of the present invention provides a driver monitoring method based on a vehicle-mounted terminal and cloud analysis, which is applied to the vehicle-mounted terminal, and includes:
if the bandwidth meets the video transmission requirement, sending the driver video data to a cloud end, and taking a second driver state identification result which is returned by the cloud end and is obtained based on the analysis of the driver video data as a driver state true value;
if the bandwidth does not meet the video transmission requirement, taking the first driver state identification result as a driver state true value;
and step 106, executing a preset automatic driving program according to the real value of the driver state.
The execution subject of the present embodiment is a program running on the in-vehicle terminal, which is capable of calling in-vehicle hardware.
As shown in fig. 2, in a preferred embodiment, the onboard hardware includes a driver monitoring camera 1, an onboard driver monitoring controller 2, a remote communication module (T-Box)3, a cloud processor 4, an autopilot domain controller 5, chassis system parts (power plant 6, steering plant 7, braking plant 8), body control system parts (vibrating seat 9, pretensioned seat belt 10, mood light 11) and onboard entertainment system parts (infotainment controller 12/IVI12, meters 13, center control display 14).
In step 102, the acquisition of the driver video data is realized by calling a driver monitoring camera 1 by a program on the vehicle-mounted terminal; and the analysis of the driver video data is realized by calling the vehicle-mounted driver monitoring controller 2 by a program on the vehicle-mounted terminal.
In step 104, the execution of the judgment is realized by calling a remote communication module (T-Box)3 by a program on the in-vehicle terminal.
In step 106, the automatic driving program is executed by calling the automatic driving area controller 5 by a program on the vehicle-mounted terminal, and the automatic driving area controller 5 further calls any one or any combination of a chassis system part (a power device 6, a steering device 7 and a braking device 8), a vehicle body control system part (a vibration seat 9, a pre-tightening safety belt 10 and an atmosphere lamp 11), a vehicle-mounted entertainment system part (an IVI12, a meter 13 and a central control display screen 14).
The beneficial effect of this embodiment lies in:
monitoring and analyzing the video data of the driver through the vehicle-mounted terminal and the cloud, and preferentially monitoring the video data of the driver through the cloud with higher computing power and higher accuracy to obtain the real-time state of the driver under the condition of better network environment; the driver monitoring controller is used in poor network environment. The two are complementary, and the monitoring accuracy and precision of the state of the driver can be greatly improved.
According to the above embodiment, in the present embodiment:
the step of executing a preset autopilot program according to the driver status truth value includes:
if the true value of the driver state is determined to be a first value, executing a preset automatic driving program to control the vehicle to run, and controlling a vehicle-mounted audio-video system to execute audio-video prompt;
the first value is a driver state true value when a driver is in an abnormal driving state;
the audio-visual prompt comprises any one or any combination of the following components:
displaying prompt information on an instrument panel;
displaying prompt information on a display screen;
and playing the prompt tone through the IVI.
After the step of determining that the true value of the driver state is the first value, executing a preset automatic driving program to control the vehicle to run and controlling the vehicle-mounted audio-video system to execute audio-video prompt, the method further comprises the following steps:
and if the driver state true value keeps the first value in the set time period after the vehicle-mounted audio-video system executes the audio-video prompt, executing a preset automatic driving program to control the vehicle to decelerate, and moving to a safe position to stop the vehicle.
After the step of determining that the true value of the driver state is the first value, executing a preset automatic driving program to control the vehicle to run and controlling the vehicle-mounted audio-video system to execute audio-video prompt, the method further comprises the following steps:
determining that the real value of the driver state keeps a first value in a set time period after the vehicle-mounted audio-video system executes the audio-video prompt, and controlling the vehicle to execute the physical prompt through a preset automatic driving program;
the physical cue comprises any one or any combination of the following:
controlling the vibration of a seat where a driver is located;
controlling a safety belt worn by a driver to be tightened to a set position;
and controlling the atmosphere lamp to light the set take-over color.
After the step of determining that the true value of the driver state is the first value, executing a preset automatic driving program to control the vehicle to run and controlling the vehicle-mounted audio-video system to execute audio-video prompt, the method further comprises the following steps:
determining that the driver state true value keeps a first value in a set time period after the vehicle-mounted audio-video system executes the audio-video prompt, and controlling the vehicle-mounted audio-video system to execute the take-over prompt through a preset automatic driving program until the driver takes over the vehicle control;
the take-over prompt comprises any one or any combination of the following:
displaying takeover information on an instrument panel;
displaying the takeover information on a display screen;
take-over speech is played through the infotainment controller IVI 12.
In the set time interval after confirming that the vehicle-mounted audio-visual system carries out audio-visual suggestion, the driver state true value keeps first value, then carry out the step that preset automatic driving procedure controlled the vehicle to slow down, and move to the safe position and park, still include:
and if the driver is determined to take over the vehicle control, stopping executing the preset automatic driving program to control the vehicle to decelerate, and moving to a safe position to stop the vehicle.
The step of executing a preset autopilot program according to the driver status truth value includes:
if the true value of the driver state is determined to be a second value, executing a preset automatic driving program to control the vehicle to run; the second value is a driver state true value when the driver is in a normal driving state.
After the step of determining that the true value of the driver state is the first value, executing a preset automatic driving program to control the vehicle to run and controlling the vehicle-mounted audio-video system to execute audio-video prompt, the method further comprises the following steps:
if the driver state true value is a second value within a set time period after the vehicle-mounted audio-video system executes the audio-video prompt, the step of controlling the vehicle-mounted audio-video system to execute the audio-video prompt is terminated, and a preset automatic driving program is executed to control the vehicle to run; the second value is a driver state true value when the driver is in a normal driving state.
In this embodiment, different follow-up actions are performed on the first and second values of the driver status true value respectively:
for a first true value, namely that the driver is in an abnormal driving state, sequentially executing two stages of subsequent operations to ensure that the driver is out of the abnormal driving state and has the capability of taking over the vehicle at any time, specifically:
the first stage is as follows: the automatic driving area controller 5 can continue to control the automatic driving of the vehicle, simultaneously sends a reminding instruction to the vehicle-mounted entertainment system, respectively displays characters and images for reminding safe driving on the instrument 13 and the central control display screen 14, and the IVI12 sends out a safe driving reminding sound; and if the driver is in a normal state, the automatic driving is continuously kept without any reminding.
After the first stage lasts for a set duration (the duration can be calibrated according to human factors), if the state of the driver is recovered to be normal, the system keeps automatic driving, and the reminding information is eliminated; if the driver state does not recover the normal driving state, the second stage is carried out: the automatic driving controller needs to take emergency measures to control part of the power device 6 of the chassis system to decelerate and stop in the safe lane. And simultaneously, the vibration of a vibration seat 9 of a vehicle body control system, the pre-tightening of a safety belt 10 for early warning, the illumination of emergency colors by an atmosphere lamp 11 are controlled, and the display of the take-over information by an instrument 13, an IVI12 and a central control display screen 14 of the vehicle-mounted entertainment system is controlled. Waiting for the driver to take over the vehicle.
It should be noted that steps 102, 104, and 106 are repeatedly executed, that is, the real value of the driver state is updated based on a set interval, the set duration of the first stage duration has a corresponding relationship with the set interval, and the set interval is not greater than the set duration.
According to the method, a vehicle-mounted driver state identification method and a cloud driver state identification method are used in a fusion mode, a vehicle-side algorithm model is optimized through data comparison, and the comprehensive identification accuracy is improved. The driver is recognized to be in an abnormal driving state, the first stage of man-machine interaction reminding is carried out, the second stage of taking emergency measures to stop in the safe lane, the driver is reminded to take over the vehicle in a richer man-machine interaction mode, and the safety of the driver and the traffic participants is effectively guaranteed.
According to any of the embodiments described above, in this embodiment:
if the bandwidth meets the video transmission requirement, the driver video data are sent to a cloud end, and a second driver state identification result returned by the cloud end and obtained based on the analysis of the driver video data is used as a driver state true value, and the method comprises the following steps:
determining that a second driver state recognition result which is returned by the cloud and obtained based on analysis of the driver video data is received, and taking the second driver state recognition result as a driver state true value;
and determining that a second driver state recognition result which is returned by the cloud and is obtained based on the analysis of the driver video data is not received, and taking the first driver state recognition result as a driver state true value.
The method of the embodiment further comprises the following steps:
and determining that the vehicle is in a non-running state, and adjusting a driver video data analysis algorithm deployed on the vehicle-mounted terminal according to the first driver state recognition result and the second driver state recognition result.
The step of determining that the vehicle is in a non-running state, and adjusting a driver video data analysis algorithm deployed on the vehicle-mounted terminal according to the first driver state recognition result and the second driver state recognition result comprises the following steps:
uploading the first driver state recognition result to a cloud;
determining that the vehicle is in a non-running state, and receiving a driver video data analysis algorithm update file which is returned by a cloud end through an OTA and is deployed on the vehicle-mounted terminal based on the first driver state recognition result and the second driver state recognition result;
and determining that the vehicle is in a non-running state, running the update file, and updating a driver video data analysis algorithm deployed on the vehicle-mounted terminal.
On the basis of the above embodiments, the embodiment provides a verification process in which the cloud returns the second driver state identification result, and an update method of the driver video data analysis algorithm deployed on the vehicle-mounted terminal, so that the reliability and accuracy of the driver state monitoring can be further improved.
According to any of the above embodiments, embodiments will be provided below from the perspective of on-vehicle hardware.
The technical problem that this embodiment solved lies in:
at present, a driver monitoring system only processes and identifies the state of a driver through a vehicle-end controller, and the identification accuracy is low. The invention can adopt two modes of monitoring the controller by the driver and uploading the video of the driver to the cloud in real time. Under the condition of a better network environment, the real-time state of a driver is obtained by preferentially using cloud monitoring with stronger calculation power and higher accuracy; the driver monitoring controller is used in poor network environment. The two are complementary, and the monitoring accuracy and precision of the state of the driver can be greatly improved.
The updating iteration of the driver monitoring system software algorithm needs to improve the identification accuracy rate by learning a large amount of data and optimizing the state identification algorithm and model. According to the invention, the iterative driver monitoring controller software can be rapidly updated through an Over-the-Air Technology (OTA), so that the identification accuracy of the vehicle-end monitoring system is improved.
In addition, the existing driver monitoring system only provides auxiliary prompts when recognizing the state of the driver, so that the driver can easily ignore the auxiliary prompts and can not participate in vehicle control. The driver monitoring system designed by the invention not only can provide richer man-machine interaction reminding modes, but also can participate in vehicle control to ensure the safety of the driver and other traffic participants. The automatic driving vehicle needs to fully remind the driver in a human-family interaction (such as hearing, vision and touch) mode when the state of the driver is fatigue, distraction and the like; if the driver does not recover the normal state after reminding, the automatic driving vehicle needs to take emergency measures to control the vehicle to turn, brake, a danger warning lamp and the like to decelerate and brake in the safe lane and turn on the danger warning lamp.
The advantages of this embodiment over the prior art are:
1. monitoring driver state by utilizing vehicle-mounted and cloud-end schemes
2. Iterative OTA-based driver monitoring controller software update
3. Starting from the whole vehicle architecture, an automatic driving vehicle driver monitoring system is designed, and the emergency control vehicle safe parking strategy after the driver is reminded and fully reminded is realized.
Therefore, the following can be further brought:
1. the vehicle-mounted scheme and the cloud scheme are used for monitoring the state of a driver, so that the overall recognition rate and accuracy are improved;
2. the OTA upgrades the vehicle-mounted driver monitoring system, and algorithm model optimization and software iteration are completed more efficiently;
3. the safety of the driver and the traffic participants is fully guaranteed by reminding and taking over in an emergency under the condition of abnormal state of the driver.
The embodiment relates to the design of the overall functional architecture scheme of the automatic driving vehicle driver monitoring system, and is one of important contents for the development of the automatic driving system. In the embodiment, two driver state monitoring schemes of a vehicle-mounted driver and a cloud are innovatively designed, and a high-computing-power and high-accuracy cloud monitoring mode is adopted when the network condition is good; and when the network condition is poor, a vehicle-mounted monitoring system is adopted, so that the comprehensive identification rate is greatly improved. And monitoring controller software for the vehicle-mounted driver by an OTA technology, and optimizing an algorithm model. In addition, in the automatic driving process of the vehicle, when the state of the driver is abnormal, the driver is fully reminded in a man-machine interaction (such as hearing, vision and touch); after the warning, the driver does not recover the normal state, and the automatic driving vehicle needs to take emergency measures to control the vehicle to turn, brake, a danger warning lamp and the like to decelerate and brake in the safe lane and turn on the danger warning lamp. Based on the above, an automatic driving driver monitoring system based on an automatic driving function architecture level is provided, which mainly comprises a driver monitoring camera 1, a vehicle-mounted driver monitoring controller 2, a remote communication module (T-Box)3, a cloud processor 4, an automatic driving area controller (ADU)5, a chassis system, a vehicle body control system and a vehicle-mounted entertainment system.
The driver monitoring system suitable for the automatic driving vehicle can identify the state of the driver with higher accuracy, quickly iterate the software of the vehicle-mounted driver monitoring controller, and remind and take over the vehicle in an emergency when the state of the driver is abnormal in the automatic driving process.
To achieve the above function, a driver monitoring system adapted for an autonomous vehicle includes: the vehicle-mounted intelligent monitoring system comprises a driver monitoring camera 1, a vehicle-mounted driver monitoring controller 2, a remote communication module (T-Box)3, a cloud processor 4, an automatic driving area controller (ADU)5, a chassis system (a power device 6, a steering device 7 and a brake device 8), a vehicle body control system (comprising a vibration seat 9, a pre-tightening safety belt 10 and an atmosphere lamp 11), and a vehicle-mounted entertainment system (comprising an IVI12, an instrument 13 and a display screen 14).
The driver monitoring camera 1 collects original video data of a driver in real time and transmits the original video data to the vehicle-mounted driver monitoring controller 2, the vehicle-mounted driver monitoring controller 2 outputs the processed video data to the state of the driver, meanwhile, the video data and the state of the driver identified by the vehicle-mounted controller (namely the vehicle-mounted driver monitoring controller 2) are transmitted to the T-Box3 and transmitted to the cloud processor 4, and the state of the driver identified by the cloud returns to the driver monitoring controller 2 through the T-Box 3.
Under the condition of good network condition, the vehicle-mounted and cloud-side can obtain two groups of driver state data. At this time, the vehicle-mounted driver monitoring controller 2 inputs the driver state identified by the cloud with higher accuracy into the automatic driving area controller 5 as a true value of the subsequent processing; and if the network environment is not good, directly taking the state identification data of the vehicle-mounted end as input.
And comparing two groups of state data obtained by the cloud processor 4 to obtain two groups of data differences, judging whether the state identification is accurate or not through the original driver video data, and adjusting and optimizing the algorithm models of the cloud processor 4 and the vehicle-mounted driver monitoring controller 2. And upgrading the optimized and upgraded software of the vehicle-mounted driver controller by using an OTA technology through a T-Box3 networking function, thereby improving the state identification accuracy of the vehicle-mounted driver monitoring controller 2.
In the system, an automatic driving domain controller 5 is used as an automatic driving control and decision-making module. There are several functions in this system:
function one: when the vehicle-mounted driver monitoring controller 2 inputs the state that the automatic driving area controller 5 is in a normal driving state (no distraction, fatigue, smoking, calling, leaving off duty and the like), the automatic driving area controller 5 controls the vehicle to automatically drive;
and a second function: when the vehicle-mounted driver monitoring controller 2 inputs the abnormal driving state to the automatic driving area controller 5:
the first stage is as follows: the automatic driving area controller 5 can continue to control the automatic driving of the vehicle, simultaneously sends a reminding instruction to the vehicle-mounted entertainment system, respectively displays characters and images for reminding safe driving on the instrument 13 and the central control display screen 14, and the IVI12 sends out a safe driving reminding sound;
and a second stage: after the driver is fully reminded, when the state of the driver does not recover to normal driving, the automatic driving controller 5 needs to take emergency measures to control the chassis system to decelerate and stop when the driver drives to the safe lane. And simultaneously, controlling the vehicle body system to vibrate the seat 9, pre-tightening the safety belt 10 for early warning, lighting the take-over color by the atmosphere lamp 11, and controlling the instruments, the IVI12 and the central control display screen 14 of the vehicle-mounted entertainment system to display take-over information. Waiting for the driver to take over the vehicle.
One of the innovations of this embodiment is: under the condition of good network condition, two groups of driver state data of a vehicle-mounted driver and a cloud driver can be acquired. The driver state obtained by the high computing power cloud processor 4 is selected as a follow-up processing basis, so that the accuracy is higher; even if the network environment is not good, the vehicle-mounted recognition result can be used as the input of the subsequent processing.
The second innovation of this embodiment is: a large amount of actual driver video data can be used as machine learning samples, vehicle-mounted driver monitoring controller software is updated through an OTA technology, iterative updating is fast, and identification accuracy is improved.
The third innovation of this embodiment is: in the driving process of the automatic driving vehicle, when a driver is in an abnormal driving state, the first-stage system reminds the driver of paying attention to safe driving through rich images, characters, sounds and instrument instructions; after the first stage, the driver is reminded that the normal driving state is not recovered, and the system enters the second stage. The vehicle is controlled to slow down and stop in the safe lane, meanwhile, the vehicle body system is controlled to vibrate the seat 9, the safety belt 10 is pre-tightened to give an early warning, the atmosphere lamp 11 lights the take-over color, and the instrument, the IVI12 and the central control display screen 14 of the vehicle-mounted entertainment system are controlled to display take-over information. Waiting for the driver to take over the vehicle and ensuring the safety of the driver and the traffic participants.
As shown in fig. 2, the driver monitoring system suitable for automatic driving provided by this embodiment includes a driver monitoring camera 1, a vehicle-mounted driver monitoring controller 2, a remote communication module (T-Box)3, a cloud processor 4, an automatic driving area controller 5, a chassis system part power device 6, a steering device 7, a braking device 8, a vehicle body control system part vibration seat 9, a pre-tightening safety belt 10, an atmosphere lamp 11, a vehicle-mounted entertainment system part entertainment information controller (IVI)12, an instrument 13, and a central control display screen 14.
The driver monitoring camera 1 is used for acquiring original video image data of a driver in real time and transmitting the original driver data to the vehicle-mounted driver monitoring controller 2 through an LVDS (low voltage differential signaling) line or other special video transmission lines.
The vehicle-mounted driver monitoring controller 2 identifies real-time driver states including distraction, fatigue, call making, smoking, off duty and the like through an identification algorithm. And the recognized state is converted into a CAN bus or other vehicle-mounted common bus signal form.
And the remote communication module (T-Box)3 realizes the communication between the vehicle end and the cloud end by utilizing a 4G/5G communication mode and is connected with the vehicle-mounted driver monitoring controller through a vehicle-mounted Ethernet. The remote communication module (T-Box)3 needs to determine whether the network environment is normal or not and whether it is suitable for video data transmission. If the network environment is normal, the vehicle-mounted driver monitoring controller 2 transmits the driver video data collected by the camera and the recognized driver state data to the remote communication module (T-Box)3, and the driver video data and the recognized driver state data are uploaded to the cloud processor 4.
The cloud processor 4 receives the video data uploaded by the remote communication module 3, recognizes the driver state and transmits the driver state back to the remote communication module 3,
an automatic driving area controller (ADU)5 is connected with the vehicle-mounted driver monitoring controller 2 through a CAN bus. Receives a driver status signal from the driver monitoring controller to the remote communication module 3. In addition, the cloud processor 4 compares two groups of identification data of the vehicle and the cloud and a large amount of external injection data, and optimizes an algorithm model of the vehicle-mounted driver monitoring controller 2.
The automatic driving domain controller 5 receives the bus data of the vehicle-mounted driver monitoring controller 2, and controls the whole vehicle to remind the driver of man-machine interaction or take over the vehicle to stop in the safe lane until the driver takes over the vehicle
The power device 6, the steering device 7, the brake device 8, the vibration seat 9, the pre-tightening safety belt 10, the atmosphere lamp 11, the entertainment information controller 12, the instrument 13 and the central control display screen 14 are all vehicle end actuators
As shown in fig. 3, in this embodiment, the specific steps are as follows:
A. the vehicle enters autonomous driving.
B. The camera collects the state video of the driver in real time.
C. When the network condition is good, the vehicle-mounted driver monitoring controller 2 receives the video data, calculates a result, and uploads the original video data and the vehicle end identification result to the cloud end through the remote communication module 3.
D. The cloud processor 4 recognizes the state of the driver in real time through the uploaded video data, and transmits the state of the driver back to the vehicle-mounted driver monitoring controller 2 through the remote communication module 3, meanwhile, the vehicle end and the cloud data are compared, and an algorithm model of the vehicle end is optimized through a large amount of data learning. When the vehicle is idle, the software of the vehicle-mounted driver monitoring controller 2 is upgraded through OTA.
E. The vehicle-mounted driver monitoring controller 2 inputs the driver state returned by the cloud side to the automatic driving area controller 5.
F. If the automatic driving area controller 5 receives that the driver state is an abnormal driving state (distraction, fatigue, smoking, calling, off duty, etc.), the first stage: the automatic driving area controller 5 can continue to control the automatic driving of the vehicle, simultaneously sends a reminding instruction to the vehicle-mounted entertainment system, respectively displays characters and images for reminding safe driving on an instrument and a display screen, and the IVI controller sends out a safe driving reminding sound; and if the driver is in a normal state, the automatic driving is continuously kept without any reminding.
G. After the first stage lasts for a period of time (the time can be calibrated according to human factors engineering), if the state of the driver is recovered to be normal, the system keeps automatic driving, and the reminding information is eliminated; if the driver state does not recover the normal driving state, the second stage is carried out: the automatic driving controller needs to take emergency measures to control the chassis system to decelerate and stop when the vehicle runs to the safe lane. And meanwhile, the vehicle body system is controlled to vibrate the seat, pre-tighten the safety belt early warning, the atmosphere lamp is lightened to have an emergency color, and the instrument, the IVI controller and the central control display screen of the vehicle-mounted entertainment system are controlled to display the take-over information. Waiting for the driver to take over the vehicle.
According to the method, a vehicle-mounted driver state identification method and a cloud driver state identification method are used in a fusion mode, a vehicle-side algorithm model is optimized through data comparison, and the comprehensive identification accuracy is improved. The driver is recognized to be in an abnormal driving state, the first stage of man-machine interaction reminding is carried out, the second stage of taking emergency measures to stop in the safe lane, the driver is reminded to take over the vehicle in a richer man-machine interaction mode, and the safety of the driver and the traffic participants is effectively guaranteed.
The present embodiment of a driver monitoring system design suitable for autonomous driving is particularly useful in some respects in conjunction with a particular type of vehicle, but the vehicle may be any type of vehicle including, but not limited to, trucks, cars, buses, ships, airplanes, land vehicles, agricultural equipment, trams.
The present embodiment proposes a driver monitoring system suitable for autonomous driving. The vehicle-mounted and cloud-side parallel driver state monitoring method adopts a high-computing-power and high-accuracy cloud-side monitoring mode when the network condition is good; and when the network condition is poor, a vehicle-mounted monitoring system is adopted. The comprehensive recognition rate is greatly improved. And monitoring controller software for the vehicle-mounted driver by using an OTA technology, and quickly optimizing a state recognition algorithm model. In addition, in the automatic driving process of the vehicle, when the state of the driver is abnormal, the driver is fully reminded in a man-machine interaction (such as hearing, vision and touch); after the warning, the driver does not recover the normal state, and the automatic driving vehicle needs to take emergency takeover to control the vehicle to turn, brake, a danger warning lamp and the like to decelerate and brake in the safe lane and start the danger warning lamp. Based on the design content, the automatic driving driver monitoring system based on the automatic driving function architecture level is provided.
The driver monitoring device based on the vehicle-mounted terminal and the cloud analysis provided by the invention is described below, and the driver monitoring device based on the vehicle-mounted terminal and the cloud analysis described below and the driver monitoring method based on the vehicle-mounted terminal and the cloud analysis described above can be referred to in a mutual correspondence manner.
The embodiment of the invention provides a driver monitoring system based on vehicle-mounted terminal and cloud analysis, which is applied to the vehicle-mounted terminal and comprises the following components:
the acquisition module 401 is configured to acquire and analyze driver video data to obtain a first driver state identification result;
a determining module 402, configured to perform determination for network communication conditions:
if the bandwidth meets the video transmission requirement, sending the driver video data to a cloud end, and taking a second driver state identification result which is returned by the cloud end and is obtained based on the analysis of the driver video data as a driver state true value;
if the bandwidth does not meet the video transmission requirement, taking the first driver state identification result as a driver state true value;
and an automatic driving module 403, configured to execute a preset automatic driving program according to the driver status true value.
The autopilot module 403 includes:
the audio-visual prompting submodule is used for executing a preset automatic driving program to control the vehicle to run and controlling the vehicle-mounted audio-visual system to execute audio-visual prompting if the true value of the driver state is determined to be a first value;
the first value is a driver state true value when a driver is in an abnormal driving state;
the audio-visual prompt comprises any one or any combination of the following components:
displaying prompt information on an instrument panel;
displaying prompt information on a display screen;
and playing the prompt tone through the IVI.
And the deceleration parking submodule is used for determining that the driver state true value keeps a first value in a set time period after the vehicle-mounted audio-video system executes the audio-video prompt, executing a preset automatic driving program to control the vehicle to decelerate, and moving the vehicle to a safe position for parking.
The physical prompting submodule is used for determining that the driver state true value keeps a first value in a set time period after the vehicle-mounted audio-video system executes the audio-video prompting, and controlling the vehicle to execute the physical prompting through a preset automatic driving program;
the physical cue comprises any one or any combination of the following:
controlling the vibration of a seat where a driver is located;
controlling a safety belt worn by a driver to be tightened to a set position;
and controlling the atmosphere lamp to light the set take-over color.
The take-over prompting submodule is used for determining that the driver state true value keeps a first value in a set time period after the vehicle-mounted audio-video system executes the audio-video prompting, and controlling the vehicle-mounted audio-video system to execute take-over prompting through a preset automatic driving program until the driver takes over vehicle control;
the take-over prompt comprises any one or any combination of the following:
displaying takeover information on an instrument panel;
displaying the takeover information on a display screen;
and playing the takeover voice through the IVI.
And the take-over confirmation submodule is used for determining that the driver takes over the control of the vehicle, terminating the step of executing the preset automatic driving program to control the vehicle to decelerate, and moving to a safe position to stop the vehicle.
The first driving submodule is used for determining that the real value of the driver state is a second value and executing a preset automatic driving program to control the vehicle to drive; the second value is a driver state true value when the driver is in a normal driving state.
The second driving submodule is used for determining that the real value of the state of the driver is a second value in a set time period after the vehicle-mounted audio-video system executes the audio-video prompt, terminating the step of controlling the vehicle-mounted audio-video system to execute the audio-video prompt and executing a preset automatic driving program to control the vehicle to drive; the second value is a driver state true value when the driver is in a normal driving state.
The determining module 402 comprises:
the receiving confirmation submodule is used for determining that a second driver state recognition result which is returned by the cloud end and is obtained based on analysis of the driver video data is received, and taking the second driver state recognition result as a driver state true value;
and the non-reception confirmation submodule is used for determining that a second driver state recognition result which is returned by the cloud end and is obtained based on the analysis of the driver video data is not received, and then taking the first driver state recognition result as a driver state true value.
Further, the driver monitoring system based on vehicle-mounted terminal and cloud analysis further comprises:
and the algorithm updating module is used for determining that the vehicle is in a non-running state and adjusting a driver video data analysis algorithm deployed on the vehicle-mounted terminal according to the first driver state recognition result and the second driver state recognition result.
The algorithm updating module comprises:
the uploading sub-module is used for uploading the first driver state recognition result to a cloud;
the transmission submodule is used for determining that the vehicle is in a non-running state, and receiving a driver video data analysis algorithm update file which is returned by a cloud end and is deployed on the vehicle-mounted terminal based on the first driver state identification result and the second driver state identification result through an OTA (over the air) module;
and the updating submodule is used for determining that the vehicle is in a non-running state, running the updating file and updating the driver video data analysis algorithm deployed on the vehicle-mounted terminal.
Fig. 5 illustrates a physical structure diagram of an electronic device, which may include, as shown in fig. 5: a processor (processor)510, a communication Interface (Communications Interface)520, a memory (memory)530 and a communication bus 540, wherein the processor 510, the communication Interface 520 and the memory 530 communicate with each other via the communication bus 540. Processor 510 may invoke logic instructions in memory 530 to perform a method for driver monitoring based on vehicle terminal and cloud analytics, the method comprising: acquiring and analyzing driver video data to obtain a first driver state identification result; performing a determination for a network communication condition: if the bandwidth meets the video transmission requirement, sending the driver video data to a cloud end, and taking a second driver state identification result which is returned by the cloud end and is obtained based on the analysis of the driver video data as a driver state true value; if the bandwidth does not meet the video transmission requirement, taking the first driver state identification result as a driver state true value; and executing a preset automatic driving program according to the real value of the driver state.
Furthermore, the logic instructions in the memory 530 may be implemented in the form of software functional units and stored in a computer readable storage medium when the software functional units are sold or used as independent products. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.
In another aspect, the present invention further provides a computer program product, where the computer program product includes a computer program, the computer program may be stored on a non-transitory computer-readable storage medium, and when the computer program is executed by a processor, a computer is capable of executing the driver monitoring method based on the vehicle-mounted terminal and the cloud end analysis provided by the foregoing methods, and the method includes: acquiring and analyzing driver video data to obtain a first driver state identification result; performing a determination for a network communication condition: if the bandwidth meets the video transmission requirement, sending the driver video data to a cloud end, and taking a second driver state identification result which is returned by the cloud end and is obtained based on the analysis of the driver video data as a driver state true value; if the bandwidth does not meet the video transmission requirement, taking the first driver state identification result as a driver state true value; and executing a preset automatic driving program according to the real value of the driver state.
In still another aspect, the present invention further provides a non-transitory computer-readable storage medium, on which a computer program is stored, the computer program being implemented by a processor to perform the in-vehicle terminal and cloud analytics based driver monitoring method provided by the above methods, the method including: acquiring and analyzing driver video data to obtain a first driver state identification result; performing a determination for a network communication condition: if the bandwidth meets the video transmission requirement, sending the driver video data to a cloud end, and taking a second driver state identification result which is returned by the cloud end and is obtained based on the analysis of the driver video data as a driver state true value; if the bandwidth does not meet the video transmission requirement, taking the first driver state identification result as a driver state true value; and executing a preset automatic driving program according to the real value of the driver state.
The above-described embodiments of the apparatus are merely illustrative, and the units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the present embodiment. One of ordinary skill in the art can understand and implement it without inventive effort.
Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware. With this understanding in mind, the above-described technical solutions may be embodied in the form of a software product, which can be stored in a computer-readable storage medium such as ROM/RAM, magnetic disk, optical disk, etc., and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the methods described in the embodiments or some parts of the embodiments.
Finally, it should be noted that: the above examples are only intended to illustrate the technical solution of the present invention, but not to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions of the embodiments of the present invention.
Claims (15)
1. A driver monitoring method based on vehicle-mounted terminal and cloud analysis is characterized by being applied to the vehicle-mounted terminal and comprising the following steps:
acquiring and analyzing driver video data to obtain a first driver state identification result;
performing a determination for a network communication condition:
if the bandwidth meets the video transmission requirement, sending the driver video data to a cloud end, and taking a second driver state identification result which is returned by the cloud end and is obtained based on the analysis of the driver video data as a driver state true value;
if the bandwidth does not meet the video transmission requirement, taking the first driver state identification result as a driver state true value;
and executing a preset automatic driving program according to the real value of the driver state.
2. The driver monitoring method based on the vehicle-mounted terminal and the cloud terminal analysis as claimed in claim 1, wherein the step of executing a preset automatic driving program according to the driver status true value comprises:
if the true value of the driver state is determined to be a first value, executing a preset automatic driving program to control the vehicle to run, and controlling a vehicle-mounted audio-video system to execute audio-video prompt;
the first value is a driver state true value when a driver is in an abnormal driving state;
the audio-visual prompt comprises any one or any combination of the following components:
displaying prompt information on an instrument panel;
displaying prompt information on a display screen;
and playing the prompt tone through the IVI.
3. The driver monitoring method based on the vehicle-mounted terminal and the cloud analysis of claim 2, wherein after the step of determining that the true value of the driver status is the first value, executing a preset automatic driving program to control the vehicle to run and controlling the vehicle-mounted audio-video system to execute audio-video prompt further comprises:
and if the driver state true value keeps the first value in the set time period after the vehicle-mounted audio-video system executes the audio-video prompt, executing a preset automatic driving program to control the vehicle to decelerate, and moving to a safe position to stop the vehicle.
4. The driver monitoring method based on the vehicle-mounted terminal and the cloud analysis of claim 2, wherein after the step of determining that the true value of the driver status is the first value, executing a preset automatic driving program to control the vehicle to run and controlling the vehicle-mounted audio-video system to execute audio-video prompt further comprises:
determining that the real value of the driver state keeps a first value in a set time period after the vehicle-mounted audio-video system executes the audio-video prompt, and controlling the vehicle to execute the physical prompt through a preset automatic driving program;
the physical cue comprises any one or any combination of the following:
controlling the vibration of a seat where a driver is located;
controlling a safety belt worn by a driver to be tightened to a set position;
and controlling the atmosphere lamp to light the set take-over color.
5. The driver monitoring method based on the vehicle-mounted terminal and the cloud analysis of claim 2, wherein after the step of determining that the true value of the driver status is the first value, executing a preset automatic driving program to control the vehicle to run and controlling the vehicle-mounted audio-video system to execute audio-video prompt further comprises:
determining that the driver state true value keeps a first value in a set time period after the vehicle-mounted audio-video system executes the audio-video prompt, and controlling the vehicle-mounted audio-video system to execute the take-over prompt through a preset automatic driving program until the driver takes over the vehicle control;
the take-over prompt comprises any one or any combination of the following:
displaying takeover information on an instrument panel;
displaying the takeover information on a display screen;
and playing the takeover voice through the IVI.
6. The driver monitoring method based on the vehicle-mounted terminal and the cloud analysis of claim 3, wherein in the set time period after the vehicle-mounted audio/video system is determined to execute the audio/video prompt, if the driver status true value keeps a first value, a preset automatic driving program is executed to control the vehicle to decelerate, and the vehicle is moved to a safe position to stop, the method further comprises the following steps:
and if the driver is determined to take over the vehicle control, stopping executing the preset automatic driving program to control the vehicle to decelerate, and moving to a safe position to stop the vehicle.
7. The driver monitoring method based on the vehicle-mounted terminal and the cloud terminal analysis as claimed in claim 1, wherein the step of executing a preset automatic driving program according to the driver status true value comprises:
if the true value of the driver state is determined to be a second value, executing a preset automatic driving program to control the vehicle to run; the second value is a driver state true value when the driver is in a normal driving state.
8. The driver monitoring method based on the vehicle-mounted terminal and the cloud analysis of claim 2, wherein after the step of determining that the true value of the driver status is the first value, executing a preset automatic driving program to control the vehicle to run and controlling the vehicle-mounted audio-video system to execute audio-video prompt further comprises:
if the driver state true value is a second value within a set time period after the vehicle-mounted audio-video system executes the audio-video prompt, the step of controlling the vehicle-mounted audio-video system to execute the audio-video prompt is terminated, and a preset automatic driving program is executed to control the vehicle to run; the second value is a driver state true value when the driver is in a normal driving state.
9. The driver monitoring method based on the vehicle-mounted terminal and the cloud analysis of claim 1, wherein if the bandwidth meets a video transmission requirement, the step of sending the driver video data to a cloud, and taking a second driver state recognition result returned by the cloud and obtained based on the driver video data analysis as a driver state true value comprises the steps of:
determining that a second driver state recognition result which is returned by the cloud and obtained based on analysis of the driver video data is received, and taking the second driver state recognition result as a driver state true value;
and determining that a second driver state recognition result which is returned by the cloud and is obtained based on the analysis of the driver video data is not received, and taking the first driver state recognition result as a driver state true value.
10. The driver monitoring method based on the vehicle-mounted terminal and the cloud analysis as claimed in claim 1, further comprising:
and determining that the vehicle is in a non-running state, and adjusting a driver video data analysis algorithm deployed on the vehicle-mounted terminal according to the first driver state recognition result and the second driver state recognition result.
11. The vehicle-mounted terminal and cloud analysis based driver monitoring method according to claim 10, wherein the step of determining that the vehicle is in a non-running state, and adjusting the driver video data analysis algorithm deployed on the vehicle-mounted terminal according to the first driver state recognition result and the second driver state recognition result comprises:
uploading the first driver state recognition result to a cloud;
determining that the vehicle is in a non-running state, and receiving a driver video data analysis algorithm update file which is returned by a cloud end through an OTA and is deployed on the vehicle-mounted terminal based on the first driver state recognition result and the second driver state recognition result;
and determining that the vehicle is in a non-running state, running the update file, and updating a driver video data analysis algorithm deployed on the vehicle-mounted terminal.
12. The utility model provides a driver monitored control system based on vehicle-mounted terminal and high in the clouds are analytic, its characterized in that is applied to vehicle-mounted terminal, includes:
the acquisition module is used for acquiring and analyzing the driver video data to obtain a first driver state identification result;
a judging module, configured to perform judgment for the network communication condition:
if the bandwidth meets the video transmission requirement, sending the driver video data to a cloud end, and taking a second driver state identification result which is returned by the cloud end and is obtained based on the analysis of the driver video data as a driver state true value;
if the bandwidth does not meet the video transmission requirement, taking the first driver state identification result as a driver state true value;
and the automatic driving module is used for executing a preset automatic driving program according to the real value of the driver state.
13. An electronic device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method for monitoring a driver based on the in-vehicle terminal and cloud end analysis according to any one of claims 1 to 11 when executing the program on the in-vehicle terminal.
14. A non-transitory computer readable storage medium, having a computer program stored thereon, wherein the computer program, when being executed by a processor, implements the steps of the vehicle terminal and cloud analytics based driver monitoring method according to any one of claims 1 to 11.
15. A computer program product comprising a computer program, wherein the computer program when executed by a processor implements the steps of the vehicle terminal and cloud analytics based driver monitoring method according to any one of claims 1 to 11.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111555459.4A CN114348017A (en) | 2021-12-17 | 2021-12-17 | Driver monitoring method and system based on vehicle-mounted terminal and cloud analysis |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111555459.4A CN114348017A (en) | 2021-12-17 | 2021-12-17 | Driver monitoring method and system based on vehicle-mounted terminal and cloud analysis |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114348017A true CN114348017A (en) | 2022-04-15 |
Family
ID=81100214
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111555459.4A Pending CN114348017A (en) | 2021-12-17 | 2021-12-17 | Driver monitoring method and system based on vehicle-mounted terminal and cloud analysis |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114348017A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117041503A (en) * | 2023-10-10 | 2023-11-10 | 四川科泰智能电子有限公司 | High-performance flash memory array video identification equipment and method |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105966405A (en) * | 2015-03-13 | 2016-09-28 | 哈曼国际工业有限公司 | Driver distraction detection system |
| CN108407813A (en) * | 2018-01-25 | 2018-08-17 | 惠州市德赛西威汽车电子股份有限公司 | A kind of antifatigue safe driving method of vehicle based on big data |
| CN109353347A (en) * | 2018-12-04 | 2019-02-19 | 爱驰汽车有限公司 | Vehicle and its driving adapter tube based reminding method, system, electronic equipment and storage medium |
| CN109598174A (en) * | 2017-09-29 | 2019-04-09 | 厦门歌乐电子企业有限公司 | The detection method and its device and system of driver status |
| KR20200086785A (en) * | 2019-01-09 | 2020-07-20 | (주)씨앤아이피 | A method and system for preventing drowsiness driving and keeping vehicle safe |
| KR20210148497A (en) * | 2020-05-28 | 2021-12-08 | 주식회사 씽크아이 | Camera Apparatus Installing at a Car for Detecting Drowsy Driving and Careless Driving and Method thereof |
-
2021
- 2021-12-17 CN CN202111555459.4A patent/CN114348017A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105966405A (en) * | 2015-03-13 | 2016-09-28 | 哈曼国际工业有限公司 | Driver distraction detection system |
| CN109598174A (en) * | 2017-09-29 | 2019-04-09 | 厦门歌乐电子企业有限公司 | The detection method and its device and system of driver status |
| CN108407813A (en) * | 2018-01-25 | 2018-08-17 | 惠州市德赛西威汽车电子股份有限公司 | A kind of antifatigue safe driving method of vehicle based on big data |
| CN109353347A (en) * | 2018-12-04 | 2019-02-19 | 爱驰汽车有限公司 | Vehicle and its driving adapter tube based reminding method, system, electronic equipment and storage medium |
| KR20200086785A (en) * | 2019-01-09 | 2020-07-20 | (주)씨앤아이피 | A method and system for preventing drowsiness driving and keeping vehicle safe |
| KR20210148497A (en) * | 2020-05-28 | 2021-12-08 | 주식회사 씽크아이 | Camera Apparatus Installing at a Car for Detecting Drowsy Driving and Careless Driving and Method thereof |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117041503A (en) * | 2023-10-10 | 2023-11-10 | 四川科泰智能电子有限公司 | High-performance flash memory array video identification equipment and method |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111402925B (en) | Voice adjustment method, device, electronic equipment, vehicle-mounted system and readable medium | |
| CN109229102A (en) | Automatic driving vehicle control system, method and apparatus | |
| CN110085022B (en) | Method and system for interaction between motorcade vehicles and vehicle | |
| CN106218405A (en) | Fatigue driving monitoring method and cloud server | |
| CN109727427A (en) | DMS driver fatigue early warning system | |
| CN110576808B (en) | Vehicle, vehicle machine equipment and scene information pushing method based on artificial intelligence | |
| US20220204020A1 (en) | Toward simulation of driver behavior in driving automation | |
| CN110393540A (en) | Method and device, system based on the prompt safe driving that driver is emotionally stable | |
| CN106471573A (en) | Speech recognition equipment and speech recognition system | |
| CN112937590B (en) | Intelligent vehicle dynamic man-machine interaction system and method | |
| CN114379581B (en) | Algorithm iteration system and method based on automatic driving | |
| CN114348017A (en) | Driver monitoring method and system based on vehicle-mounted terminal and cloud analysis | |
| CN108860157B (en) | Violation risk assessment method, system, equipment and storage medium | |
| CN106904122A (en) | Vehicle speech sound alarming method and apparatus | |
| CN112102584B (en) | Automatic driving alarm method and device for vehicle, vehicle and storage medium | |
| CN109830116A (en) | Information cuing method, device and relevant device based on image recognition | |
| CN113829994B (en) | Early warning method and device based on car external whistling, car and medium | |
| US20210403052A1 (en) | Impaired driver assistance | |
| CN113479214B (en) | Automatic driving system and application method thereof | |
| CN114655241A (en) | Automobile intelligent multimedia central control screen fatigue driving reminding method | |
| CN113771859A (en) | Intelligent driving intervention method, device and equipment and computer readable storage medium | |
| CN109743382B (en) | Cloud service system of vehicle and interaction method thereof | |
| CN111652065B (en) | Multi-mode safe driving method, equipment and system based on vehicle perception and intelligent wearing | |
| CN114286479A (en) | Scene-based in-vehicle light control method and system and readable storage medium | |
| CN109410582B (en) | Traffic condition analysis method and terminal equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20220921 Address after: Room 2528, Building 2, Lane 1800, Xinyang Road, Lingang New Area, Pudong New Area, Shanghai, 201801 Applicant after: Inceptio Star Intelligent Technology (Shanghai) Co.,Ltd. Address before: Room 4, room 001, building 11, Lane 1333, Jiangnan Avenue, Changxing Town, Chongming District, Shanghai 202150 Applicant before: International network technology (Shanghai) Co.,Ltd. |
|
| TA01 | Transfer of patent application right |