CN116129641B - A vehicle safety situation calculation method and system based on multi-terminal collaborative identification - Google Patents

Abstract

The present invention discloses a vehicle safety situation calculation method and system based on multi-terminal collaborative identification, which is used in a system in which a vehicle-mounted sensing device, a vehicle-mounted safety situation calculation device connected to the vehicle-mounted sensing device, multiple roadside sensing devices wirelessly connected to the vehicle-mounted sensing device and the vehicle-mounted safety situation calculation device, and a cloud-based sensing device bidirectionally connected to the roadside sensing device cooperate with each other. By obtaining risk factor information within and beyond visual range, the sensing feature information of the vehicle, roadside, and cloud is integrated, and at the same time, the advantages of different terminals are utilized to interconnect and communicate with and allocate the computing resources of different terminals, and the path selection of the vehicle is planned according to the calculation results of the safety situation. The method and system provided by the present invention realize the driving risk estimation in the intelligent interconnected traffic scene, ensure the real-time perception and the accuracy of the safety situation judgment, and can greatly enhance the driving safety of intelligent vehicles.

Description

A vehicle safety situation calculation method and system based on multi-terminal collaborative identification

Technical Field

The present invention relates to the field of vehicle safety and intelligent interconnected transportation, and in particular to a vehicle safety situation calculation method and system based on multi-terminal collaborative identification.

Background technique

With the advancement of vehicle intelligence, the demand for vehicles to perceive their own and operating environment status is increasing. In the existing advanced driver assistance systems, the focus is mainly on the perception of people's bad conditions and the relatively small operating environment outside the vehicle. However, it is impossible to make an objective judgment on driving safety by relying solely on the perception of the sensors arranged on the vehicle itself; in the existing methods for determining vehicle operation risks, the integration of various characteristic information of the driver, the vehicle itself, and the operating environment is weak, and often several relatively independent systems make judgments and warnings separately. With the development of autonomous driving, the characteristic information obtained by the vehicle through perception will determine the vehicle's operation decision control. Therefore, increasing the richness of the vehicle's perception pathways and effectively integrating perception parameters are essential requirements for the further development of human-machine co-driving autonomous driving. At the same time, during the driving process of current intelligent vehicles, the computing power resources of the on-board computing equipment and other computing pathways in the vehicle's intelligent driving system are unevenly distributed, and the situation of insufficient single-end computing power often occurs, and there are many repeated calculations, which makes it difficult to achieve high-precision safety situation assessment and driving path decision planning during the intelligent driving of the vehicle.

Therefore, while improving the richness of intelligent vehicle perception pathways and effectively integrating perception parameters, it is an urgent technical problem to use auxiliary calculations from other pathways combined with on-board calculations to conduct more accurate safety situation assessments and driving path decision planning.

Summary of the invention

The present invention provides a vehicle safety situation calculation method and system based on multi-terminal collaborative identification, which is used to solve the technical problems of insufficient perception range capability of current intelligent vehicles and uneven distribution and low utilization efficiency of computing resources at each end in the vehicle-road-cloud system.

To achieve the above-mentioned object, the present invention provides a vehicle safety situation calculation method based on multi-terminal collaborative identification, which is used in a system in which a vehicle-mounted sensing device, a vehicle-mounted safety situation calculation device, multiple roadside sensing devices, and a cloud sensing device cooperate with each other, and includes the following steps:

S1. Collecting information: Collect the driver's multimodal information through the on-board sensing device, generate the driver's characteristic parameters through calculation, and send the driver's characteristic parameters to the on-board safety situation calculation device; collect the vehicle's external environment information and vehicle operation parameter information through the on-board sensing device, and send them to the roadside sensing device in the vicinity of the vehicle; collect road network environment information through the roadside sensing device and pre-store road network information in the area; the cloud sensing device pre-stores and updates high-precision map information.

S2. Processing information: After receiving the vehicle's external environment information and vehicle operating parameter information, the roadside sensing device identifies and ignores duplicate information in the road network environment information, road network road information, vehicle's external environment information and vehicle operating parameter information; performs calculations after allocating computing resources, and sends data that exceeds the computing capacity to the cloud sensing device.

S3. Generate risk factor package: The cloud-based sensing device receives information from the roadside sensing device and performs calculations, and returns the calculated data to the roadside sensing device; the roadside sensing device receives information returned by the cloud-based sensing device, generates a risk factor data packet, and sends it to the vehicle-mounted safety situation calculation device.

S4. Conduct safety situation assessment and driving decision planning: After receiving the driver's characteristic parameters and risk factor data packet, the on-board safety situation calculation device conducts a vehicle safety situation assessment based on the driver's characteristic parameters and risk factor data packet. After the assessment is completed, driving decision planning is conducted based on the assessment results and it is determined whether to issue a warning to the driver.

Preferably, in S1, the multimodal information includes the driver's facial expression, head posture, body posture, heart rate, blood oxygen saturation, skin electricity, respiratory rate and voice intonation information; the vehicle's external environment information includes two-dimensional images and three-dimensional radar point cloud information within the detectable range of the on-board sensing device; the vehicle's operating parameter information includes the vehicle's speed, acceleration, steering wheel angle, positioning information, accelerator pedal opening and brake pedal opening information; the road network environment information includes road network image information and road network three-dimensional radar point cloud information.

The road network information in the area pre-stored in the roadside sensing device includes lane information, road connection relationships, intersection information and road section information; the high-precision map information pre-stored and updated by the cloud sensing device includes road congestion conditions, construction conditions, traffic accidents, weather conditions, traffic lights, crosswalks and speed limit conditions.

Preferably, when the roadside perception device calculates the collected data, the data is input into the YOLOv7 deep neural network to obtain the position, speed, and acceleration information of risk factors such as vehicles, pedestrians, and obstacles in the current road network.

After calculating the data sent by the roadside sensing device, the cloud sensing device obtains the risk factor information in each road network area, and returns the coordinates, speed and acceleration parameter information in the risk factor information to the roadside sensing device.

Preferably, in S4, when the vehicle-mounted safety situation calculation device performs vehicle safety situation assessment, the vehicle operation safety situation E is:

Among them, α _i , β _j , and γ _k are preset parameters greater than 0, representing the weight of each risk factor, _Vi represents the external factor risk factor, D _j represents the driver's risk factor in different states, and R _k represents the road risk factor.

When the vehicle-mounted safety situation calculation device performs driving decision planning, it searches for the vehicle path planning set according to the safety situation assessment results, continuously adjusts the best planned path in the set based on the self-learning method and displays it to the driver. The vehicle planned path set P is:

P＝{δ|ε _I ,E _I ,θ,ε _O ,E _O }

Among them, δ represents the path planning set of the vehicle, ε _I represents the historical trajectory of the vehicle, E _I represents the safety situation of the vehicle, θ represents the driving decision of the driver, ε _O represents the historical trajectory of the surrounding vehicles, and E _O represents the safety situation of the surrounding vehicles.

The present invention also provides a vehicle safety situation calculation system based on multi-terminal collaborative identification, including: an on-board sensing device, an on-board safety situation calculation device connected to the on-board sensing device, a roadside sensing device wirelessly connected to the on-board sensing device and the on-board safety situation calculation device, and a cloud-based sensing device bidirectionally connected to the roadside sensing device.

The vehicle-mounted sensing device is used to collect the driver's multimodal information, calculate the driver's characteristic parameters, and send the driver's characteristic parameters to the vehicle-mounted safety situation calculation device.

The roadside sensing device is used to collect road network environment information, process its own information and information sent by other devices, send information beyond its own computing capacity to the cloud sensing device, and send risk factor data packets to the on-board safety situation calculation device.

The cloud sensing device is used to calculate the information from the roadside sensing device and return the result to the roadside sensing device.

The vehicle-mounted safety situation calculation device is used to perform vehicle safety situation assessment and driving decision planning based on driver characteristic parameters and risk factor data packets.

Preferably, the roadside sensing device is also used to pre-store road network information in the area; the cloud sensing device is also used to pre-store and update high-precision map information.

Preferably, the vehicle-mounted sensing device includes a vehicle-mounted driver sensing module, a vehicle-mounted vehicle sensing module, a vehicle-mounted environment sensing module and a vehicle-mounted information processing module.

The vehicle-mounted driver perception module includes: one or more high-definition network cameras installed in front of the driver without blocking his or her line of sight and above the rearview mirror inside the vehicle, for collecting the driver's facial expressions, head posture and body posture; an audio collector for collecting the driver's voiceprint signals and voice intonation; a physiological signal collection bracelet worn on the driver's wrist or embedded in the steering wheel, for collecting the driver's heart rate, respiratory rate, skin electricity and blood oxygen saturation.

The on-board vehicle sensing module includes: a speed sensor for collecting the vehicle's speed; an acceleration sensor for collecting the vehicle's acceleration; an angle sensor for collecting the vehicle's steering wheel angle; a GPS locator for collecting the vehicle's positioning information; an accelerator pedal opening sensor for collecting the vehicle's accelerator pedal opening; and a brake pedal opening sensor for collecting the vehicle's brake pedal opening information.

The on-board environmental perception module includes a high-definition network camera, infrared camera, lidar and millimeter-wave radar installed on the vehicle's exterior body to collect two-dimensional images and three-dimensional radar point cloud information outside the vehicle.

The on-board information processing module includes at least one edge computing device for receiving multimodal information from the on-board driver perception module, processing the multimodal information into driver characteristic parameters through a multi-task neural network, sending the driver characteristic parameters to the on-board safety situation calculation device, and sending the vehicle external environment information and vehicle operation parameter information to the roadside perception device.

Preferably, the vehicle-mounted safety situation calculation device includes a safety situation calculation module and a vehicle path planning module.

The safety situation calculation module is used to receive driver characteristic parameters from the on-board sensing device and risk factor data packets from the roadside sensing device, and then calculate the safety situation.

The vehicle path planning module is used to receive the safety situation information from the safety situation calculation module, and then perform path planning and display and high-risk warning to the driver.

Preferably, the roadside perception device includes a roadside environment perception module and a roadside information processing module.

The roadside environment perception module includes a camera and a detection radar installed on the roadside perception device for collecting road network image information and road network three-dimensional radar point cloud information under the current road network.

The roadside information processing module includes a module installed in the roadside sensing device, which is used to pre-store road network information in the current area, receive road network image information and road network three-dimensional radar point cloud information from the roadside environment perception module, receive vehicle external environment information and vehicle operation parameter information sent by vehicles within the receiving range, check for duplicate data and ignore it, allocate computing resources and then perform calculations through the YOLOv7 deep neural network, send data that exceeds the computing capacity to the cloud sensing device, receive information returned by the cloud sensing device and generate risk factor data packets, and send the risk factor data packets to multiple integrated computing devices of the on-board safety situation computing device.

Preferably, the cloud-based sensing device includes a cloud-based information processing module integrated with multiple high-performance computing clusters for pre-storing and updating high-precision map information, processing information from the roadside sensing device, and returning the processed data together with the road environment status under beyond-visual-range wide-area conditions to the roadside sensing device.

The present invention has the following beneficial effects:

The vehicle safety situation calculation method based on multi-terminal collaborative identification of the present invention integrates the perception capabilities of the vehicle-mounted sensing device, the roadside sensing device, and the cloud-based sensing device, so that the intelligent vehicle can obtain risk factor information within and beyond the visual range, improves the richness of the intelligent vehicle's perception path, and effectively integrates the perception parameters, so that the perception and judgment of risks during driving are more comprehensive and accurate, greatly enhancing the driving safety of intelligent vehicles; the roadside sensing device is used to check the duplicates of each data and ignore the duplicate data, thereby avoiding the waste of computing resources, and dynamically allocating the computing capabilities of the vehicle-mounted sensing device, the roadside sensing device, and the cloud-based sensing device to avoid the situation of insufficient computing power, so that the safety situation assessment and driving path decision planning in the intelligent driving process have higher accuracy and real-time performance. The vehicle safety situation calculation system based on multi-terminal collaborative identification of the present invention provides a hardware foundation for the implementation of the above method, so it has the same beneficial effects as the above method.

In addition to the above-described purposes, features and advantages, the present invention has other purposes, features and advantages. The present invention will be further described in detail below with reference to the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

The drawings constituting a part of this application are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

FIG1 is a flow chart of a method for calculating vehicle safety situation based on multi-terminal collaborative identification according to a preferred embodiment 1 of the present invention;

FIG2 is a vehicle safety situation calculation framework diagram of a preferred embodiment 1 of the present invention;

FIG3 is a structural diagram of a vehicle safety situation calculation system based on multi-terminal collaborative identification according to a preferred embodiment 2 of the present invention.

Detailed ways

The embodiments of the present invention are described in detail below with reference to the accompanying drawings, but the present invention can be implemented in many different ways as defined and covered by the claims.

Embodiment 1:

Referring to FIG. 1 and FIG. 2, in a preferred embodiment of the present invention, a vehicle safety situation calculation method based on multi-terminal collaborative identification is provided, which is used in a system in which a vehicle-mounted sensing device, a vehicle-mounted safety situation calculation device connected to the vehicle-mounted sensing device, multiple roadside sensing devices wirelessly connected to the vehicle-mounted sensing device and the vehicle-mounted safety situation calculation device, and a cloud sensing device bidirectionally connected to the roadside sensing device cooperate with each other, and includes the following steps:

S1. Collecting information: Collect the driver's multimodal information through the on-board sensing device, generate the driver's characteristic parameters through calculation, and send the driver's characteristic parameters to the on-board safety situation calculation device; collect the vehicle's external environment information and vehicle operation parameter information through the on-board sensing device, and send them to the roadside sensing device in the vicinity of the vehicle; collect road network environment information through the roadside sensing device and pre-store road network information in the area; the cloud sensing device pre-stores and updates high-precision map information.

In S1, multimodal information includes the driver's facial expression, head posture, body posture, heart rate, blood oxygen saturation, skin electricity, respiratory rate and voice intonation information; the vehicle's external environment information includes two-dimensional images and three-dimensional radar point cloud information within the detectable range of the on-board sensing device; the vehicle's operating parameter information includes the vehicle's speed, acceleration, steering wheel angle, positioning information, accelerator pedal opening and brake pedal opening information; the road network environment information includes road network image information and road network three-dimensional radar point cloud information.

The road network information in the area pre-stored in the roadside sensing device includes lane information, road connection relationships, intersection information and road section information; the high-precision map information pre-stored and updated by the cloud sensing device includes road congestion conditions, construction conditions, traffic accidents, weather conditions, traffic lights, crosswalks and speed limit conditions.

In a preferred embodiment of the present invention, multiple sensing devices are used to collect various physiological information and external performance information of the driver, and the driver's state behavior is identified through a multi-task convolutional neural network to obtain the driver's characteristic parameters. The vehicle's external environment information, vehicle operation parameter information and road network environment information are collected, that is, the driver, and the driver's risk factor information within and beyond the visual range are collected, thereby ensuring the comprehensiveness of data collection while ensuring the quality of data collection.

The road network information in the area is pre-stored in the roadside sensing device, and the high-precision map information is pre-stored and updated in the cloud sensing device, which ensures the timeliness of the existing data and makes the method real-time when applied.

S2. Processing information: After receiving the vehicle's external environment information and vehicle operating parameter information, the roadside sensing device identifies and ignores duplicate information in the road network environment information, road network road information, vehicle's external environment information and vehicle operating parameter information; performs calculations after allocating computing resources, and sends data that exceeds the computing capacity to the cloud sensing device.

When the roadside perception device calculates the collected data, it inputs the data into the YOLOv7 deep neural network to obtain the position, speed, and acceleration information of risk factors such as vehicles, pedestrians, and obstacles in the current road network.

In a preferred embodiment of the present invention, each data is checked for duplication through a roadside sensing device and duplicate data is ignored, the sensing feature information on the vehicle, roadside and cloud is integrated, the advantages of different ends are utilized, the computing resources of different ends are interconnected and communicated and effectively allocated, and different vehicle-side devices are prevented from calculating duplicate environmental information, thereby reducing the waste of computing resources and avoiding insufficient computing power.

S3. Generate risk factor package: The cloud-based sensing device receives information from the roadside sensing device and performs calculations, and returns the calculated data to the roadside sensing device; the roadside sensing device receives information returned by the cloud-based sensing device, generates a risk factor data packet, and sends it to the vehicle-mounted safety situation calculation device.

After calculating the data sent by the roadside sensing device, the cloud-based sensing device obtains the risk factor information in each road network area, and returns the coordinate, speed and acceleration parameter information in the risk factor information to the roadside sensing device without transmitting large-capacity information such as pictures, videos and point cloud maps.

In a preferred embodiment of the present invention, computing pressure is shared by cloud-based sensing devices to avoid insufficient computing power. Key information on risk factors is transmitted first during data transmission to minimize the amount of data, thereby ensuring the real-time and stability of data interaction between devices.

S4. Conduct safety situation assessment and driving decision planning: After receiving the driver's characteristic parameters and risk factor data packet, the on-board safety situation calculation device conducts a vehicle safety situation assessment based on the driver's characteristic parameters and risk factor data packet. After the assessment is completed, driving decision planning is conducted based on the assessment results and it is determined whether to issue a warning to the driver.

In S4, when the vehicle safety situation calculation device performs vehicle safety situation assessment, the vehicle operation safety situation E is:

Among them, α _i , β _j , and γ _k are preset parameters greater than 0, representing the weights of each risk factor, _Vi represents the risk factor of external factors, such as neighboring vehicles, pedestrians, obstacles, etc., D _j represents the risk factor of different driver states, such as fatigue, emotion, operating behavior, etc., and P _k represents the road risk factor, such as road curvature, road slope, road visibility, road flatness, traffic lights at intersections, etc.

When the on-board safety situation calculation device makes driving decision planning, it searches for the vehicle path planning set according to the safety situation assessment results, continuously adjusts the best planned path in the set based on the self-learning method and displays it to the driver. When it is determined that the risk is high, it will also perform a warning operation; the vehicle planning path set P is:

P＝{δ|ε _I ,E _I ,θ,ε _O ,E _O }

Among them, δ represents the path planning set of the vehicle, ε _I represents the historical trajectory of the vehicle, E _I represents the safety situation of the vehicle, θ represents the driving decision of the driver, ε _O represents the historical trajectory of the surrounding vehicles, and E _O represents the safety situation of the surrounding vehicles.

In the preferred embodiment of the present invention, the "man-machine-environment" information is integrated to make a more comprehensive and effective judgment on driving risks, and the vehicle driving path is planned according to the judgment results, thereby realizing the driving risk estimation in the intelligent interconnected traffic scenario, ensuring the real-time perception and the accuracy of safety situation judgment, and greatly enhancing the safety of intelligent driving.

Embodiment 2:

Referring to Figure 3, in a preferred embodiment of the present invention, a vehicle safety situation calculation system based on multi-terminal collaborative identification is provided, including: an on-board sensing device, an on-board safety situation calculation device connected to the on-board sensing device, a roadside sensing device wirelessly connected to the on-board sensing device and the on-board safety situation calculation device, and a cloud-based sensing device bidirectionally connected to the roadside sensing device.

The vehicle-mounted sensing device is used to collect the driver's multimodal information, calculate the driver's characteristic parameters, and send the driver's characteristic parameters to the vehicle-mounted safety situation calculation device.

The vehicle-mounted sensing device includes a vehicle-mounted driver sensing module, a vehicle-mounted vehicle sensing module, a vehicle-mounted environment sensing module and a vehicle-mounted information processing module.

The vehicle-mounted driver perception module includes: one or more high-definition network cameras installed directly in front of the driver's line of sight and above the rearview mirror inside the vehicle, used to collect the driver's facial expressions, head posture and body posture; an audio collector for collecting the driver's voiceprint signals and voice intonation; a physiological signal collection bracelet worn on the driver's wrist or embedded in the steering wheel, used to collect the driver's heart rate, breathing rate, skin electricity and blood oxygen saturation.

The on-board vehicle sensing module includes: a speed sensor for collecting the vehicle's speed; an acceleration sensor for collecting the vehicle's acceleration; an angle sensor for collecting the vehicle's steering wheel angle; a GPS locator for collecting the vehicle's positioning information; an accelerator pedal opening sensor for collecting the vehicle's accelerator pedal opening; and a brake pedal opening sensor for collecting the vehicle's brake pedal opening information.

The on-board environmental perception module includes a high-definition network camera, infrared camera, lidar and millimeter-wave radar installed on the vehicle's exterior body to collect two-dimensional images and three-dimensional radar point cloud information outside the vehicle.

The on-board information processing module includes at least one edge computing device for receiving multimodal information from the on-board driver perception module, processing the multimodal information into driver characteristic parameters through a multi-task neural network, sending the driver characteristic parameters to the on-board safety situation calculation device, and sending the vehicle external environment information and vehicle operation parameter information to the roadside perception device.

The on-board information processing module receives the feature data of the on-board driver perception module, performs time series alignment and dimensionality reduction processing, and then inputs the data into the multi-task neural network to obtain the driver's fatigue, emotion, action behavior and other status information.

In a preferred embodiment of the present invention, through the mutual cooperation of various modules in the vehicle-mounted sensing device, various physiological information and external performance information of the driver, vehicle external environment information and vehicle operation parameter information are comprehensively collected, ensuring the reliability and richness of data collection.

The vehicle-mounted safety situation calculation device is used to perform vehicle safety situation assessment and driving decision planning based on driver characteristic parameters and risk factor data packets.

The vehicle-mounted safety situation calculation device includes a safety situation calculation module and a vehicle path planning module.

The safety situation calculation module is used to receive driver characteristic parameters from the on-board sensing device and risk factor data packets from the roadside sensing device, and then calculate the safety situation.

The vehicle path planning module is used to receive the safety situation information from the safety situation calculation module, and then perform path planning and display and high-risk warning to the driver.

In a preferred embodiment of the present invention, through the collaboration between the modules within the vehicle-mounted safety situation calculation device, a comprehensive and effective judgment of driving risks is ensured based on the received driver characteristic parameters and risk factor data packets, and then the vehicle driving path is planned, so that the driving risk judgment in the intelligent interconnected traffic scenario is comprehensive and accurate.

The roadside sensing device is used to collect road network environment information, process its own information and information sent by other devices, send information beyond its own computing capacity to the cloud sensing device, and send risk factor data packets to the on-board safety situation calculation device.

The roadside sensing device is also used to pre-store road network information in the area.

The roadside perception device includes a roadside environment perception module and a roadside information processing module.

The roadside environment perception module includes a camera and a detection radar installed on the roadside perception device for collecting road network image information and road network three-dimensional radar point cloud information under the current road network.

The roadside information processing module includes a module installed in the roadside sensing device, which is used to pre-store road network information in the current area, receive road network image information and road network three-dimensional radar point cloud information from the roadside environment perception module, receive vehicle external environment information and vehicle operation parameter information sent by vehicles within the receiving range, check for duplicate data and ignore it, allocate computing resources and then perform calculations through the YOLOv7 deep neural network, send data that exceeds the computing capacity to the cloud sensing device, receive information returned by the cloud sensing device and generate risk factor data packets, and send the risk factor data packets to multiple integrated computing devices of the on-board safety situation computing device.

In a preferred embodiment of the present invention, through the collaboration between the modules within the roadside sensing device, duplicate data is ignored, the sensing feature information on the vehicle, roadside and cloud is integrated, and the advantages of different ends are utilized to interconnect and communicate with the computing resources of different ends and effectively allocate them, thereby preventing different devices from calculating duplicate environmental information, reducing the waste of computing resources and avoiding insufficient computing power.

The cloud sensing device is used to calculate the information from the roadside sensing device and return the result to the roadside sensing device.

Cloud-based sensing devices are also used to pre-store and update high-precision map information.

The cloud sensing device includes a cloud information processing module integrated with multiple high-performance computing clusters for pre-storing and updating high-precision map information, processing information from the roadside sensing device, and returning the processed data together with the road environment status under beyond-visual-range wide-area conditions to the roadside sensing device.

In the preferred embodiment of the present invention, by sending data outside the calculation capacity of the roadside sensing device to the cloud sensing device for calculation, the calculation pressure of the roadside sensing device is effectively reduced, the real-time nature of the data is ensured, and insufficient computing power is avoided; at the same time, the high-precision map information provided by the cloud sensing device in the preferred embodiment of the present invention also provides indispensable data support for intelligent driving.

The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims (8)

1. The vehicle security situation calculation method based on multi-terminal collaborative identification is used in a system of mutual collaboration of a vehicle-mounted sensing device, a vehicle-mounted security situation calculation device, a plurality of road side sensing devices and a cloud sensing device, and is characterized by comprising the following steps:

S1, information is collected: the method comprises the steps of collecting multi-mode information of a driver through a vehicle-mounted sensing device, generating characteristic parameters of the driver through calculation, and sending the characteristic parameters of the driver to a vehicle-mounted security situation calculating device; collecting vehicle external environment information and vehicle operation parameter information through a vehicle-mounted sensing device, and sending the vehicle external environment information and the vehicle operation parameter information to a road side sensing device in a vehicle adjacent range; collecting road network environment information through a road side sensing device and pre-storing road network road information in an area; the cloud sensing device pre-stores and updates high-precision map information;

s2, processing information: after receiving the external environment information of the vehicle and the running parameter information of the vehicle, the road side sensing device recognizes and ignores repeated information in the road network environment information, the road network road information, the external environment information of the vehicle and the running parameter information of the vehicle; after computing resource allocation, computing is executed, and data exceeding the computing tolerance are sent to a cloud sensing device;

S3, generating a risk factor package: the cloud sensing device receives and calculates information from the road side sensing device to obtain risk factor information in each road network area, and returns coordinate, speed and acceleration parameter information in the risk factor information to the road side sensing device; the road side sensing device receives the information returned by the cloud sensing device, generates a risk factor data packet and sends the risk factor data packet to the vehicle-mounted security situation calculating device;

s4, carrying out security situation assessment and driving decision planning: the vehicle-mounted safety situation calculation device receives the driver characteristic parameters and the risk factor data packet, carries out vehicle safety situation assessment according to the driver characteristic parameters and the risk factor data packet, carries out driving decision planning according to assessment results after assessment is finished, and judges whether to pre-warn the driver or not;

In S4, when the vehicle-mounted security situation calculation device performs the vehicle security situation assessment, the vehicle operation security situation E is:

wherein alpha _i、β_j、γ_k is a preset parameter greater than 0, the weight of each risk factor is represented, V _i represents an external factor risk factor, D _j represents a risk factor of different states of a driver, and R _k represents a road risk factor;

when the vehicle-mounted safety situation calculation device performs driving decision planning, a vehicle path planning set is searched according to a safety situation evaluation result, the optimal planning path in the set is continuously adjusted based on a self-learning method and displayed to a driver, and the vehicle planning path set P is as follows:

P＝{δ|ε_I,E_I,θ,ε_O,E_O}

Where δ represents the path planning set of the vehicle, ε _I represents the historical track of the vehicle, E _I represents the safety situation of the vehicle, θ represents the driving decision of the driver, ε _o represents the historical track of the surrounding vehicle, and E _O represents the safety situation of the surrounding vehicle.

2. The vehicle security posture calculation method based on multi-terminal collaborative recognition according to claim 1, wherein in S1, the multi-modal information includes facial expression, head pose, body pose, heart rate, blood oxygen saturation, skin electricity, respiration rate, and voice intonation information of a driver; the vehicle external environment information comprises two-dimensional images and three-dimensional Lei Dadian cloud information in a detectable range of the vehicle-mounted sensing device; the vehicle operation parameter information comprises speed, acceleration, steering wheel rotation angle, positioning information, accelerator pedal opening and brake pedal opening information of the vehicle; the road network environment information comprises road network image information and road network three-dimensional radar point cloud information;

The road network and road information in the area pre-stored by the road side sensing device comprises lane information, road connection relation, intersection information and road section information; the high-precision map information pre-stored and updated by the cloud sensing device comprises road congestion conditions, construction conditions, traffic accident conditions, weather conditions, traffic lights, crosswalk and speed limiting conditions.

3. The vehicle security situation calculating method based on multi-terminal collaborative recognition according to claim 1, wherein when the road side sensing device calculates the collected data, the data is input into YOLOv depth neural network, so as to obtain the position, speed and acceleration information of risk factor vehicles, pedestrians and obstacles in the current road network;

The cloud sensing device calculates the data sent by the road side sensing device to obtain risk factor information in each road network area, and returns coordinate, speed and acceleration parameter information in the risk factor information to the road side sensing device.

4. A vehicle security situation computing system based on multi-terminal collaborative recognition, comprising: the system comprises a vehicle-mounted sensing device, a vehicle-mounted safety situation calculating device connected with the vehicle-mounted sensing device, a road side sensing device connected with the vehicle-mounted sensing device and the vehicle-mounted safety situation calculating device in a wireless communication mode and a cloud sensing device connected with the road side sensing device in a two-way communication mode;

the vehicle-mounted sensing device is used for collecting multi-mode information of a driver, calculating characteristic parameters of the driver and sending the characteristic parameters of the driver to the vehicle-mounted security situation calculating device;

The road side sensing device is used for collecting road network environment information, processing self information and information sent by other devices, sending information outside self-calculation bearing capacity to the cloud sensing device and sending a risk factor data packet to the vehicle-mounted security situation calculating device;

the cloud sensing device is used for calculating information from the road side sensing device and returning a result to the road side sensing device, and after the cloud sensing device obtains risk factor information in each road network area, coordinate, speed and acceleration parameter information in the risk factor information is returned to the road side sensing device, so that large-capacity information is not transmitted;

the vehicle-mounted safety situation calculation device is used for carrying out vehicle safety situation assessment and driving decision planning according to the characteristic parameters of the driver and the risk factor data packet;

The vehicle-mounted safety situation calculation device comprises a safety situation calculation module and a vehicle path planning module;

the safety situation calculation module is used for receiving the characteristic parameters of the driver from the vehicle-mounted sensing device and the risk factor data packet from the road side sensing device, so as to calculate the safety situation;

The vehicle path planning module is used for receiving the safety situation information from the safety situation calculation module, further carrying out path planning, and carrying out display and high risk early warning on a driver;

When the vehicle-mounted safety situation calculation device evaluates the safety situation of the vehicle, the running safety situation E of the vehicle is as follows:

wherein alpha _i、β_j、γ_k is a preset parameter greater than 0, the weight of each risk factor is represented, V _i represents an external factor risk factor, D _j represents a risk factor of different states of a driver, and R _k represents a road risk factor;

when the vehicle-mounted safety situation calculation device performs driving decision planning, a vehicle path planning set is searched according to a safety situation evaluation result, the optimal planning path in the set is continuously adjusted based on a self-learning method and displayed to a driver, and the vehicle planning path set P is as follows:

P＝{δ|ε_I,E_I,θ,ε_O,E_O}

Where δ represents the path planning set of the vehicle, ε _I represents the historical track of the vehicle, E _I represents the safety situation of the vehicle, θ represents the driving decision of the driver, ε _O represents the historical track of the surrounding vehicle, and E _O represents the safety situation of the surrounding vehicle.

5. The vehicle security situation calculating system based on multi-terminal collaborative recognition according to claim 4, wherein the road side sensing device is further configured to pre-store road network road information in an area; the cloud sensing device is also used for pre-storing and updating high-precision map information.

6. The vehicle security situation computing system based on multi-terminal collaborative recognition according to claim 4, wherein the vehicle-mounted sensing device comprises a vehicle-mounted driver sensing module, a vehicle-mounted vehicle sensing module, a vehicle-mounted environment sensing module, and a vehicle-mounted information processing module;

The vehicle-mounted driver perception module comprises: the high-definition network cameras are arranged right in front of the rearview mirror in the vehicle and above the rearview mirror in the vehicle, and are used for collecting facial expressions, head postures and body postures of the driver; an audio collector for collecting voiceprint signals and voice tones of a driver; the physiological signal acquisition bracelet is worn on the wrist of the driver or embedded in the steering wheel and is used for acquiring the heart rate, the respiratory rate, the skin electricity and the blood oxygen saturation of the driver;

The vehicle-mounted vehicle sensing module includes: speed sensors for acquiring the speed of the vehicle, respectively; an acceleration sensor for acquiring acceleration of the vehicle; the steering angle sensor is used for collecting the steering angle of the steering wheel of the vehicle; the GPS locator is used for collecting vehicle positioning information; an accelerator pedal opening sensor for acquiring an opening of an accelerator pedal of a vehicle; a brake pedal opening sensor for acquiring vehicle brake pedal opening information;

The vehicle-mounted environment sensing module comprises a high-definition network camera, an infrared camera, a laser radar and a millimeter wave radar, wherein the high-definition network camera, the infrared camera, the laser radar and the millimeter wave radar are arranged on the outer body of a vehicle and used for acquiring two-dimensional images and three-dimensional Lei Dadian cloud information outside the vehicle;

The vehicle-mounted information processing module comprises at least one edge computing device, wherein the at least one edge computing device is used for receiving the multi-mode information from the vehicle-mounted driver perception module, processing the multi-mode information into driver characteristic parameters through a multi-task neural network, sending the driver characteristic parameters to the vehicle-mounted security situation computing device and sending the vehicle external environment information and the vehicle operation parameter information to the road side perception device;

7. The vehicle security situation computing system based on multi-terminal collaborative recognition according to claim 5, wherein the roadside awareness device comprises a roadside environment awareness module and a roadside information processing module;

The road side environment sensing module comprises a camera and a detection radar, wherein the camera and the detection radar are arranged on the road side sensing device and are used for acquiring road network image information and road network three-dimensional radar point cloud information under the current road network;

The road side information processing module comprises a plurality of integrated computing devices which are arranged in the road side sensing device and used for pre-storing road network information in a current area, receiving road network image information and road network three-dimensional radar point cloud information from the road side environment sensing module, receiving vehicle external environment information and vehicle operation parameter information sent by a vehicle in a range, performing data check and neglect, performing calculation through YOLOv depth neural network after calculating resource allocation, sending data exceeding a calculation tolerance to the cloud sensing device, receiving return information of the cloud sensing device, generating a risk factor data packet and sending the risk factor data packet to the vehicle-mounted security situation computing device.

8. The vehicle security situation computing system based on multi-terminal collaborative recognition according to claim 5, wherein the cloud sensing device comprises a cloud information processing module for pre-storing and updating high-precision map information, processing information from a road side sensing device, and integrating a plurality of high-performance computations for returning processed data to the road environment state over a wide area of beyond-line-of-sight to the road side sensing device.