CN119408566A

CN119408566A - Driving scenario prediction and adaptive strategy generation system based on cloud-based multimodal large model

Info

Publication number: CN119408566A
Application number: CN202411627278.1A
Authority: CN
Inventors: 梁晓辉; 陈荣波; 蔡轶佳; 杨昊
Original assignee: Guangzhou Zhiti Technology Co ltd
Current assignee: Guangzhou Zhiti Technology Co ltd
Priority date: 2024-11-14
Filing date: 2024-11-14
Publication date: 2025-02-11

Abstract

The present invention discloses a driving scenario prediction and adaptive strategy generation system based on a cloud-based multimodal large model, including a multimodal data collection and scenario prediction module, an adaptive strategy generation and remote optimization module, a collaborative driving and multi-vehicle strategy coordination module, a cloud-based driving intelligent monitoring and feedback module, and a dynamic risk assessment and path optimization module; the present invention processes a variety of input data from vehicles and environments through the cloud, and predicts risk factors in autonomous driving scenarios in advance, such as sudden obstacles, weather deterioration, complex traffic conditions, etc. Cloud computing provides powerful data processing capabilities and storage resources. The large model deployed in the cloud can train the data collected by the vehicle, thereby improving the perception and prediction capabilities of the autonomous driving system, and generating a variety of response strategies for these risks. It also helps vehicles make safer and smarter driving decisions in complex scenarios through dynamic generation and remote optimization of adaptive strategies.

Description

Driving situation prediction and self-adaptive strategy generation system based on cloud multi-mode large model

Technical Field

The invention relates to the technical field of automatic driving, in particular to a driving situation prediction and self-adaptive strategy generation system based on a cloud multi-mode large model.

Background

Current multi-Modal Large Language Models (MLLMs) show great potential in improving the interpretability of autopilot systems, which are capable of generating control predictions and natural language interpretations, but at the same time face the challenges of high data labeling costs and large field differences.

The invention discloses an automatic driving decision control method based on road side fusion perception, which comprises the following steps of S1, obtaining road condition information and environment information around a vehicle and state information of the vehicle, S2, determining driving behaviors of the vehicle according to the road condition information and the environment information around the vehicle and the state information of the vehicle, S3, planning an optimal driving path of the vehicle according to the determined driving behaviors, and S4, enabling the vehicle to run according to the optimal driving path by controlling a longitudinal track of the vehicle and a transverse track of the vehicle. The invention can control the driving behavior of the vehicle based on the current scene, plan the optimal path of the vehicle based on the driving behavior, and control the vehicle to run according to the optimal path.

The automatic driving system used in the patent faces the problems of data scarcity, obvious field difference, high training cost, disastrous forgetting and the like when processing complex and changeable traffic environments, and the problems limit the generalization capability and instantaneity of the system, especially in driving situations needing quick response.

Existing autopilot technologies have higher processing capability in conventional environments, but in dynamic and complex driving situations (such as sudden obstacles, extreme weather, traffic jams, accidents, etc.), vehicles may still face mishandling problems, existing systems often rely on local sensors and algorithms to make decisions, and lack advanced perceptibility of larger range traffic, weather, environmental changes.

Disclosure of Invention

The invention aims to provide a driving situation prediction and self-adaptive strategy generation system based on a cloud multi-mode large model, which is used for solving the problems in the prior art.

The application is specifically as follows:

The driving situation prediction and self-adaptive strategy generation system based on the cloud multi-mode large model comprises a multi-mode data acquisition and situation prediction module, a self-adaptive strategy generation and remote optimization module, a collaborative driving and multi-vehicle strategy coordination module, a cloud driving intelligent monitoring and feedback module and a dynamic risk assessment and path optimization module;

The multi-mode data acquisition and situation prediction module acquires vehicle running data, weather data and traffic data through a camera, a laser radar, a sensor and a GPS which are deployed on a vehicle, and performs situation analysis and prediction on the acquired vehicle running data, weather data and traffic data by utilizing a multi-mode large model technology to generate a multi-mode large model analysis result;

The self-adaptive strategy generation and remote optimization module is used for generating multiple strategies for coping with complex situations through the multi-mode large model analysis result, automatically selecting and optimizing multiple strategies aiming at the multiple strategies, and generating an optimal strategy;

The cooperative driving and multi-vehicle strategy coordination module is used for coordinating strategies for coping with complex situations among a plurality of automatic driving vehicles, ensuring that the vehicles can cooperatively work under the same complex situations, avoiding collision among the vehicles and optimizing the overall safety;

the cloud driving intelligent monitoring and feedback module is used for intelligently monitoring the running state, the environmental change and the coping strategy of the vehicle through the cloud and providing feedback in real time;

The dynamic risk assessment and path optimization module is used for carrying out real-time assessment on the dynamic risk in the vehicle driving path through the prediction and analysis of the multi-mode large model and generating an optimal path optimization scheme;

The multi-mode data acquisition and situation prediction module, the self-adaptive strategy generation and remote optimization module, the collaborative driving and multi-vehicle strategy coordination module, the cloud driving intelligent monitoring and feedback module and the dynamic risk assessment and path optimization module are connected with each other.

Further, the multi-mode data acquisition and situation prediction module comprises an image and sensor data fusion sub-module, a situation prediction sub-module and a danger early warning sub-module;

The image and sensor data fusion sub-module is used for fusion analysis of the vehicle running data, the weather data and the traffic data through a multi-mode large model to generate a three-dimensional environment perception result;

the situation prediction submodule predicts a possible risk situation by combining the three-dimensional environment sensing result;

The danger early warning sub-module sends out early warning signals in advance to inform vehicles and passengers to take preventive measures when the situation prediction sub-module predicts the impending danger situation;

The image and sensor data fusion sub-module comprises an image data processing unit, a laser radar data processing unit and a data fusion unit;

The image data processing unit is used for carrying out different levels of image characteristics on two-dimensional image data captured by a camera by using a convolutional neural network, the convolutional neural network comprises ResNet-50 and EFFICIENTNET, the ResNet-50 and the EFFICIENTNET are connected in series, and the output of the ResNet-50 is used as the input of the EFFICIENTNET;

The laser radar data processing unit is used for extracting features of three-dimensional point cloud data captured by a laser radar by adopting a PointNet ++ network to obtain spatial information features and geometric features of each point so as to form three-dimensional point cloud features;

the data fusion unit is used for fusing the image characteristics and the three-dimensional point cloud characteristics to form multi-mode characteristics, and the fusion adopts a EarlyFusion strategy and self-attention mechanism mode, and the formula is as follows:

F_fusion＝SelfAttention(F_img,F_lidar,F_{Other sensors}),

Wherein F _fusion is the fused multi-modal feature, selfAttention is the self-attention function, F _fusion is the image feature, F _lidar is the three-dimensional point cloud feature, and F _{Other sensors} is the other sensor feature;

the specific process of the situation prediction submodule is as follows:

Modeling and prediction of the space around the vehicle using BEVTransformer-based models that can handle long-range dependencies and effectively capture global information in complex scenes:

H=Transformer(F_fusion),

where H is a spatial representation of the surroundings of the vehicle, predictions generated based on the representation will be used for subsequent driving decisions;

estimating probability distribution of potential risk situations based on a Bayesian neural network BNN model:

p (y|h) = ≡p (y|z) P (z|h) dz, where z is an hidden variable to describe a potential risk context.

Further, the self-adaptive strategy generation and remote optimization module comprises a multi-strategy generation sub-module, a real-time remote optimization sub-module and a self-adaptive driving sub-module;

The multi-strategy generation sub-module is used for detecting that the vehicle encounters different situations, and generating a plurality of coping strategies, wherein the coping strategies comprise path re-planning, speed adjustment and emergency braking, and the different situations comprise complex situations or potential hazards encountered by the vehicle;

The real-time remote optimization sub-module optimizes the plurality of coping strategies through the cloud, dynamically adjusts the running scheme of the vehicle, and sends control instructions corresponding to the optimized strategies to the vehicle in real time.

Further, the adaptive driving sub-module generates adaptive driving strategies of different levels based on different situations;

the specific implementation process of the multi-strategy generation sub-module is as follows:

different driving strategies are generated by using the deep Q network in reinforcement learning, different driving scenes are considered in the driving strategy generating process, and the optimal strategy is selected according to the current environment state:

Q(s,a)=E[r+βmax'_aQ(s',a')],

Wherein s represents the current vehicle state, including speed, position, sensor data, a represents the set of actions that can be taken, including braking, steering, accelerating, decelerating, β is a discount factor, r is an instant prize, max '_a Q (s', a ') represents the maximum Q value of all possible actions in the next state s', E represents the learning rate, and the driving scenario includes emergency braking, avoidance of obstacles, and decelerating;

The real-time remote optimization sub-module optimizes the specific implementation process of the plurality of coping strategies through the cloud:

optimizing the generated multiple coping strategy sets through a genetic algorithm GA, and performing multi-objective optimization by using multiple objective functions:

Min{f₁(a),f₂(a),...,f_n(a)},

Where f _n (a) is a loss function for a certain target, min represents the final selection of the optimal strategy;

The real-time remote optimization sub-module optimizes the plurality of coping strategies through a cloud, and the specific implementation process of dynamically adjusting the running scheme of the vehicle is as follows:

and realizing real-time remote strategy optimization through a differential evolution algorithm and a distributed deep learning model, and dynamically adjusting each vehicle according to an optimization strategy provided by a cloud.

Further, the collaborative driving and multi-vehicle strategy coordination module comprises a collaborative situation prediction sub-module, a strategy coordination sub-module and a collective path planning sub-module;

the collaborative situation prediction submodule is used for predicting global driving situation through cloud analysis of multi-mode data of all vehicles when a plurality of vehicles run in the same area and generating a multi-vehicle collaborative strategy;

the strategy coordination sub-module dynamically adjusts the driving strategy of each vehicle according to the state and the driving task of each vehicle, and ensures the mutual coordination among a plurality of vehicles;

The collective path planning sub-module is used for generating a collective path plan when a plurality of vehicles need to jointly pass through different situation road sections.

Further, the specific implementation process of the collaborative situation prediction submodule is as follows:

Global modeling is carried out on multi-mode data of multiple vehicles by using a global situation prediction algorithm based on a graph neural network GNN, the state of each vehicle is taken as a node, and updating is carried out through data of neighbor nodes, so that a global collaborative strategy and a global graph structure are finally formed:

Wherein, The characteristic representation N (v) which is the k-layer node v is a neighbor node set, W ^k is a network weight, and c _vu is a normalization coefficient;

based on the global graph structure, generating a multi-vehicle collaborative path plan by using a distributed optimization Lagrange multiplier algorithm;

the strategy coordination sub-module dynamically adjusts the driving strategy of each vehicle to ensure that the mutual coordination among a plurality of vehicles is realized in the following steps:

And continuously adjusting the multi-vehicle strategy through real-time data feedback, and carrying out multi-vehicle strategy coordination by adopting a model based on a game theory.

Further, the cloud driving intelligent monitoring and feedback module comprises a global monitoring and scheduling sub-module, a feedback optimizing sub-module and an emergency remote receiving sub-module;

The global monitoring and scheduling sub-module continuously monitors the running condition of the vehicle through traffic data and weather data, and automatically triggers a strategy to adjust if a dangerous situation or an abnormal state is detected;

the feedback optimization sub-module is used for feeding back the execution result and data of the global monitoring and scheduling sub-module to the cloud, wherein the execution result and data are used for future situation prediction and strategy optimization generation to form a self-learning closed loop;

The emergency remote access sub-module is used for remotely taking over the control right of the vehicle and implementing emergency operation under the extreme situation, wherein the extreme situation comprises the collision of the vehicle or other serious danger.

Further, the implementation process of continuously monitoring the running condition of the vehicle through traffic data and weather data in the global monitoring and scheduling sub-module is as follows:

The position, the speed and the acceleration of the vehicle are monitored in real time through a Bayesian online learning algorithm and a Kalman filter, abnormal conditions are detected based on a dynamic Bayesian network DBN, and the Kalman filter updates a formula:

x_t+1＝Ax_t+Bu_t+w_t,

P_t+1＝AP_tA^T+Q,

Where x _t is the system state, u _t is the predicted state estimate, P _t is the covariance matrix, A is the state transition matrix, B is the control input matrix, and w _t is the process noise;

the specific implementation process of the feedback optimization sub-module is as follows:

analyzing by adopting a regression analysis method, preprocessing the result and data executed by the global monitoring and scheduling submodule in real time, and carrying out time sequence analysis on historical dangerous situation or abnormal state data of the vehicle by utilizing an autoregressive moving average model to generate feature data of future situation prediction and strategy optimization of the vehicle;

Based on the feature data of the future situation prediction and strategy optimization of the vehicle, combining external environment data, adopting random forest regression to analyze road conditions and weather information, and forming a self-learning closed loop;

the specific implementation process of the emergency remote receiving pipe module is as follows:

when a potential extreme situation is detected, remote take over is performed by a rule-based PID controller, the PID control formula:

where e (t) is the error signal and K _p、K_i、K_d is the proportional, integral and derivative coefficients, respectively.

Further, the dynamic risk assessment and path optimization module comprises a dynamic risk assessment sub-module, a path planning and optimization sub-module and an emergency route generation sub-module;

The dynamic risk assessment sub-module is used for analyzing potential risks on a road in real time and optimizing and adjusting the path of the vehicle according to the severity of the risks;

The path planning and optimizing sub-module generates an optimal running path for the vehicle according to the evaluation result of the dynamic risk evaluation sub-module and simultaneously ensures that the path planning is dynamically updated according to actual conditions in the running process of the vehicle;

The emergency route generation sub-module is used for generating an emergency path planning route when the vehicle encounters a high-risk situation, and optimizing the path planning route in real time.

Further, the specific implementation process of the dynamic risk assessment sub-module is as follows:

Analyzing by adopting a factor analysis method, preprocessing data in real time, extracting the characteristics of the position, the speed and the acceleration of the vehicle, and performing time sequence analysis on the historical driving data of the vehicle by utilizing an autoregressive moving average model to generate real-time characteristic data of the vehicle;

Based on the real-time characteristic data of the vehicle, combining external environment data, adopting a Gaussian Process Regression (GPR) to analyze road conditions and weather information, identifying and classifying the types of roads and road conditions, evaluating risk grades, and generating a risk evaluation result;

road condition and weather information analysis are carried out by adopting a Gaussian process regression GPR, the type and the road surface condition of the road are identified and classified, the risk level is estimated, and the method further comprises the step of predicting the potential danger area on the road through a confidence interval:

f (x) =gpr (m (x), k (x, x '), where m (x) is a mean function and k (x, x') is a kernel function for measuring similarity between road points;

the specific implementation process of the path planning and optimizing sub-module is as follows:

Generating an optimal running path by adopting an A-algorithm and combining dynamic programming DP, calculating an alternative path bypassing a potential risk area through multidimensional analysis of the current traffic condition and a road network, and carrying out predictive evaluation on traffic flow and predicted running time of each path to generate a preferred detour path set;

The goal of dynamic programming DP is to find the optimal route for the set of preferred detour paths with guaranteed security by the shortest path algorithm:

J^*(s)＝min_u{g(s,u)+J^*(f(s,u))},

where J ^*(s) is the optimal cost for state s and g (s, u) is the cost function for taking action u.

Compared with the prior art, the embodiment of the invention achieves the following beneficial effects:

The embodiment of the invention provides a driving situation prediction and self-adaptive strategy generation system based on a cloud multi-mode large model, wherein a multi-mode data acquisition and situation prediction module acquires vehicle driving data, weather data and traffic data through cameras, laser radars, sensors and GPS deployed on vehicles, carries out situation analysis and prediction on the acquired vehicle driving data, weather data and traffic data by utilizing a multi-mode large model technology to generate a multi-mode large model analysis result, a self-adaptive strategy generation and remote optimization module automatically selects and optimizes various strategies aiming at the various strategies and generates an optimal strategy, a collaborative driving and multi-vehicle strategy coordination module is used for coordinating strategies for coping with the complex situations among a plurality of automatic driving vehicles, ensures that the plurality of vehicles can work cooperatively under the same complex situation, avoids collision among the plurality of vehicles and optimizes the whole safety, and carries out intelligent monitoring on the driving state, environmental change and coping strategies of the vehicles by utilizing a multi-mode large model technology, and provides feedback in real time, a dynamic risk assessment and path optimization module automatically selects and optimizes various strategies for coping with the complex situations, and can calculate and process the situation of the data in advance by deploying the cloud large model, and the optimal driving situation, can calculate and store the real-time risk prediction data, and the real-time risk prediction data can be stored by the data of the cloud situation has high risk prediction system, the perception and prediction capability of the automatic driving system is improved, various coping strategies aiming at the risks are generated, dynamic generation and remote optimization are further carried out through the self-adaptive strategies, and the vehicle is helped to make safer and more intelligent driving decisions in complex scenes.

Drawings

Fig. 1 is a system architecture diagram of a driving situation prediction and adaptive strategy generation system based on a cloud multi-mode large model provided by an embodiment of the present invention;

FIG. 2 is a block diagram of a multi-modal data collection and situation prediction module of a cloud-based multi-modal large model driving situation prediction and adaptive strategy generation system according to an embodiment of the present invention;

FIG. 3 is a block diagram of adaptive strategy generation and remote optimization of a cloud-based multi-modal large model driving situation prediction and adaptive strategy generation system provided by an embodiment of the present invention;

Fig. 4 is a schematic diagram of a collaborative driving and multi-vehicle strategy coordination module of a driving situation prediction and adaptive strategy generation system based on a cloud multi-mode large model according to an embodiment of the present invention;

Fig. 5 is a cloud driving intelligent monitoring and feedback module diagram of a cloud multi-mode large model-based driving situation prediction and adaptive strategy generation system provided by an embodiment of the invention;

fig. 6 is a dynamic risk assessment and path optimization block diagram of a driving situation prediction and adaptive strategy generation system based on a cloud multi-mode large model according to an embodiment of the present invention.

Detailed Description

The present invention will be described in detail with reference to the accompanying drawings.

Example 1

The embodiment of the invention provides a driving situation prediction and self-adaptive strategy generation system based on a cloud multi-mode large model, which is shown in fig. 1 and comprises a multi-mode data acquisition and situation prediction module, a self-adaptive strategy generation and remote optimization module, a collaborative driving and multi-vehicle strategy coordination module, a cloud driving intelligent monitoring and feedback module and a dynamic risk assessment and path optimization module;

Further, the multi-mode data acquisition and situation prediction module comprises an image and sensor data fusion sub-module, a situation prediction sub-module and a danger early warning sub-module, as shown in fig. 2;

F_fusion＝SelfAttention(F_img,F_lidar,F_{Other sensors}),

the specific process of the situation prediction submodule is as follows:

H=Transformer(F_fusion),

Further, the adaptive strategy generation and remote optimization module comprises a multi-strategy generation sub-module, a real-time remote optimization sub-module and an adaptive driving sub-module, as shown in fig. 3;

Specifically, the interaction process between the multi-mode data acquisition and situation prediction module and the self-adaptive strategy generation and remote optimization module is as follows:

the multi-mode data acquisition and situation prediction module is responsible for acquiring and processing sensor data of the vehicle and the environment and generating real-time perception results and situation prediction information of the environment. The information is directly input into an adaptive strategy generation module for generating a strategy adapting to the current situation.

Interaction algorithm and model:

Data sharing and transfer mechanism:

the characteristics F _fusion and the situation prediction result H after the multi-mode data fusion are transmitted to the adaptive strategy generation module as input:

Inputto Module:{F_fusion,H},

Wherein F _fusion represents the environmental perception information of the current vehicle, and H represents the situation prediction result in a short time in the future, including sudden obstacle and bad weather;

bayesian network-based risk assessment delivery:

the situation prediction result generated by the multi-mode data acquisition module can be used for modeling the potential risk based on the Bayesian network, and the risk estimation is transmitted to the strategy generation module:

the strategy generation module generates various coping strategies according to the risk probability;

policy generation dependent state input:

The state space S includes prediction information from the context prediction module, such as risk level on the path, environmental complexity, etc. Based on these states, the policy generation module uses deep Q learning or a policy gradient algorithm to generate corresponding countermeasure actions.

Q(s,a)=E[r+βmax'_aQ(s',a')],

Min{f₁(a),f₂(a),...,f_n(a)},

Further, the collaborative driving and multi-vehicle strategy coordination module comprises a collaborative situation prediction sub-module, a strategy coordination sub-module and a collective path planning sub-module, as shown in fig. 4;

Specifically, the interaction process between the adaptive strategy generation and remote optimization module and the collaborative driving and multi-vehicle strategy coordination module is as follows:

interaction description:

In a multi-vehicle cooperative scene, the self-adaptive strategy generation module interacts the generated single-vehicle strategy with the cooperative driving module so as to ensure strategy compatibility among multiple vehicles and avoid conflict.

Interaction algorithm and model:

policy coordination based on game theory:

The multi-vehicle cooperation is optimized through a model based on game theory, the strategy generation of each vehicle needs to consider the behaviors of other vehicles, and the system can find the optimal strategy combination of the plurality of vehicles by using a Nash equilibrium model;

policy adjustment and information exchange between multiple workshops:

The collaborative driving module adjusts the strategy of each vehicle by using a distributed optimization algorithm to ensure global optimization, for example, a distributed consensus algorithm based on Lagrangian multipliers is adopted for optimization, and the system coordinates the strategy of all vehicles through distributed optimization;

Global context prediction and sharing for co-driving:

The situation prediction information H _i of each vehicle is integrated through the cloud to generate global situation prediction information H _global to adjust the collective paths and behaviors of multiple vehicles:

H_global＝Fusion(H₁,H₂,...,H_n),

Through an embedded node aggregation mechanism of the GNN graph neural network, the system can capture the interaction relationship among vehicles and perform global optimization.

Further, the cloud driving intelligent monitoring and feedback module comprises a global monitoring and scheduling sub-module, a feedback optimizing sub-module and an emergency remote receiving sub-module, as shown in fig. 5;

Specifically, the interaction process of the collaborative driving and multi-vehicle strategy coordination module and the cloud driving intelligent monitoring and feedback module is as follows:

interaction description:

Interaction between the collaborative driving module and the cloud intelligent monitoring module ensures timeliness of global monitoring and feedback. The cloud system can conduct remote optimization and emergency adjustment according to the multi-vehicle state information provided by the collaborative driving module.

Interaction algorithm and model:

feedback optimization based on global monitoring:

The cloud monitoring module automatically adjusts the cooperative strategy of multiple vehicles when danger or abnormality is detected by acquiring the state and environment information S _i of each vehicle in real time, wherein the cooperative adjustment based on feedback can be realized through model predictive control;

Cloud global risk assessment and remote takeover:

When the cloud monitoring system detects an extreme dangerous situation, such as a complex situation which is about to collide or cannot be processed, the cloud system can trigger a remote take-over function, and by using an anomaly detection mechanism based on a Kalman filter, the cloud system can rapidly evaluate risks and send out a remote control instruction, and once the anomaly is detected, the cloud monitoring can calculate new control input according to the vehicle state and remotely take over the vehicle.

x_t+1＝Ax_t+Bu_t+w_t,

P_t+1＝AP_tA^T+Q,

Further, the dynamic risk assessment and path optimization module comprises a dynamic risk assessment sub-module, a path planning and optimization sub-module and an emergency route generation sub-module, as shown in fig. 6;

Specifically, the interaction process between the cloud driving intelligent monitoring and the feedback module and between the dynamic risk assessment and path optimization module is as follows:

interaction description:

the dynamic risk assessment and path optimization module is responsible for dynamically adjusting the running path of the vehicle according to the global information provided by the cloud monitoring module, and the real-time performance of the system in the aspects of risk assessment and path selection is ensured through interaction;

Interaction algorithm and model:

Path optimization based on risk assessment:

The cloud monitoring module provides a global risk assessment result R _global, and the dynamic risk assessment module adjusts the vehicle path in real time according to the information. Path optimization is performed through an A-algorithm or dynamic programming DP based on a Belman equation;

real-time generation of emergency paths:

When the dynamic risk assessment module finds that a vehicle path has higher risk, such as an obstacle or an accident in front, the system generates an emergency path and informs the vehicle, a path planning algorithm is combined with risk assessment, an optimal emergency path is generated through a rapid search algorithm such as RRT or PRM, and the algorithm ensures that the vehicle avoids a high risk area through iterative search.

J^*(s)＝min_u{g(s,u)+J^*(f(s,u))},

Embodiment 2, situation prediction of road burst obstacle and adaptive strategy generation

1. Description of the background automated driving vehicles travel on highways with sudden obstacles, such as accidents or temporary road blocks, in front.

The system needs to predict the dangerous situation in advance and automatically generate the coping strategies.

2. The situation prediction and early warning system detects a front obstacle through a multi-mode sensor and predicts that the obstacle can influence the vehicle driving path through cloud multi-mode large model analysis.

The system gives out danger early warning in advance and prepares to generate a plurality of coping strategies.

3. The self-adaptive strategy generation comprises the steps of generating three strategies, namely emergency braking, lane change avoidance, speed reduction and waiting for removing the obstacle, by the system according to road conditions, vehicle speed and the like.

The cloud system selects the optimal strategy, namely lane change avoidance, sends a command to the vehicle, and adjusts the vehicle path.

4. And (3) feeding back and optimizing, namely recording an execution result by the system, feeding back to the cloud end, and optimizing a future situation prediction model.

Embodiment 3 Multi-vehicle coordination handling bursty traffic events

1. Background description when a plurality of autonomous vehicles run in urban areas, sudden traffic accidents occur, and the system needs to coordinate driving strategies of the plurality of vehicles so as to avoid secondary collision and traffic jam.

2. And the cloud system analyzes the multi-mode data of all related vehicles, predicts the influence range of the traffic accident in front, and evaluates the influence of the accident on the driving paths of all vehicles.

3. And the system generates different coping strategies for each vehicle, namely, part of vehicles adopt emergency braking to avoid and part of vehicles bypass in advance.

The system coordinates the speeds and the running routes of all vehicles to ensure smooth traffic at the accident scene.

4. Dynamic adjustment and feedback, wherein the system dynamically adjusts the driving strategy of each vehicle according to the real-time traffic condition, ensures safe passing, and feeds back the result for future collaborative optimization.

The algorithms and displays presented herein are not inherently related to any particular computer, virtual system, or other apparatus. Various general-purpose systems may also be used with the teachings herein. The required structure for a construction of such a system is apparent from the description above. In addition, the present invention is not directed to any particular programming language. It will be appreciated that the teachings of the present invention described herein may be implemented in a variety of programming languages, and the above description of specific languages is provided for disclosure of enablement and best mode of the present invention.

In the description provided herein, numerous specific details are set forth. However, it is understood that embodiments of the invention may be practiced without these specific details. In some instances, well-known methods, structures and techniques have not been shown in detail in order not to obscure an understanding of this description.

Similarly, it should be appreciated that in the foregoing description of exemplary embodiments of the invention, various features of the invention are sometimes grouped together in a single embodiment, figure, or description thereof for the purpose of streamlining the disclosure and aiding in the understanding of one or more of the various inventive aspects. This method of disclosure, however, is not to be interpreted as reflecting an intention that the claimed invention requires more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive aspects lie in less than all features of a single foregoing disclosed embodiment. Thus, the claims following the detailed description are hereby expressly incorporated into this detailed description, with each claim standing on its own as a separate embodiment of this invention.

Those skilled in the art will appreciate that the modules in the apparatus of the embodiments may be adaptively changed and disposed in one or more apparatuses different from the embodiments. The modules or units or components of the embodiments may be combined into one module or unit or component and, furthermore, they may be divided into a plurality of sub-modules or sub-units or sub-components. Any combination of all features disclosed in this specification (including any accompanying claims, abstract and drawings), and all of the processes or units of any method or apparatus so disclosed, may be used in combination, except insofar as at least some of such features and/or processes or units are mutually exclusive. Each feature disclosed in this specification (including any accompanying claims, abstract and drawings), may be replaced by alternative features serving the same, equivalent or similar purpose, unless expressly stated otherwise.

Furthermore, those skilled in the art will appreciate that while some embodiments herein include some features but not others included in other embodiments, combinations of features of different embodiments are meant to be within the scope of the invention and form different embodiments. For example, in the following claims, any of the claimed embodiments can be used in any combination.

Various component embodiments of the invention may be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. Those skilled in the art will appreciate that some or all of the functions of some or all of the components in an apparatus according to embodiments of the present invention may be implemented in practice using a microprocessor or Digital Signal Processor (DSP). The present invention can also be implemented as an apparatus or device program (e.g., a computer program and a computer program product) for performing a portion or all of the methods described herein. Such a program embodying the present invention may be stored on a computer readable medium, or may have the form of one or more signals. Such signals may be downloaded from an internet website, provided on a carrier signal, or provided in any other form.

Claims

1. A driving scenario prediction and adaptive strategy generation system based on a cloud-based multimodal large model, characterized by including a multimodal data acquisition and scenario prediction module, an adaptive strategy generation and remote optimization module, a collaborative driving and multi-vehicle strategy coordination module, a cloud-based driving intelligent monitoring and feedback module, and a dynamic risk assessment and path optimization module;

The multimodal data collection and situation prediction module collects vehicle driving data, weather data and traffic data through cameras, lidars, sensors and GPS deployed on the vehicle, and uses multimodal large model technology to perform situation analysis and prediction on the collected vehicle driving data, weather data and traffic data to generate multimodal large model analysis results;

The adaptive strategy generation and remote optimization module generates multiple strategies for dealing with complex situations through the analysis results of the multimodal large model, automatically selects and optimizes multiple strategies for the multiple strategies, and generates the best strategy;

The collaborative driving and multi-vehicle strategy coordination module is used to coordinate strategies for coping with complex situations among multiple autonomous driving vehicles, ensuring that multiple vehicles can work together in the same complex situation, avoiding conflicts between multiple vehicles and optimizing overall safety;

The cloud-based driving intelligent monitoring and feedback module intelligently monitors the vehicle's driving status, environmental changes and response strategies through the cloud, and provides real-time feedback;

The dynamic risk assessment and path optimization module, through the prediction and analysis of the multi-modal large model, conducts real-time assessment of the dynamic risks in the vehicle's driving path and generates an optimal path optimization solution;

The multimodal data collection and situation prediction module, the adaptive strategy generation and remote optimization module, the collaborative driving and multi-vehicle strategy coordination module, the cloud-based driving intelligent monitoring and feedback module and the dynamic risk assessment and path optimization module are interconnected.

2. The driving situation prediction and adaptive strategy generation system based on cloud multimodal large model according to claim 1, characterized in that the multimodal data acquisition and situation prediction module includes an image and sensor data fusion submodule, a situation prediction submodule and a danger warning submodule;

The image and sensor data fusion submodule generates a three-dimensional environment perception result by fusing and analyzing the vehicle driving data, weather data and traffic data through a multi-modal large model;

The situation prediction submodule predicts possible risk situations in combination with the three-dimensional environment perception results;

The danger warning submodule, when the situation prediction submodule predicts an impending dangerous situation, issues a warning signal in advance to notify the vehicle and passengers to take preventive measures;

The image and sensor data fusion submodule includes an image data processing unit, a lidar data processing unit and a data fusion unit;

The image data processing unit uses a convolutional neural network to perform image features of different levels on the two-dimensional image data captured by the camera, and the convolutional neural network includes ResNet-50 and EfficientNet. The ResNet-50 and the EfficientNet are connected in series, and the output of the ResNet-50 is used as the input of the EfficientNet;

The laser radar data processing unit is used to extract features from the three-dimensional point cloud data captured by the laser radar using the PointNet++ network to obtain spatial information features and geometric features of each point to form three-dimensional point cloud features;

The data fusion unit is used to fuse the image features and the three-dimensional point cloud features to form multimodal features. The fusion adopts the EarlyFusion strategy plus the self-attention mechanism. The formula is:

F _fusion =SelfAttention(F _img , F _lidar , F _{other sensors} ),

Among them, _Ffusion is the fused multimodal feature, SelfAttention is the self-attention function, _Ffusion is the image feature, _Flidar is the 3D point cloud feature, and Fother _sensors is the feature of other sensors;

The specific process implemented by the situation prediction submodule is as follows:

The space around the vehicle is modeled and predicted using the BEVTransformer model, which is able to handle long-range dependencies and effectively capture global information in complex scenes:

H = Transformer (F _fusion ),

Where H is the spatial representation around the vehicle, and the predictions generated based on this representation will be used for subsequent driving decisions;

The probability distribution of potential risk scenarios is evaluated by using the Bayesian neural network (BNN) model:

P(y|H)＝∫P(y|z)P(z|H)dz，

Among them, z is a latent variable used to describe potential risk scenarios.

3. The driving situation prediction and adaptive strategy generation system based on cloud multimodal large model according to claim 2, characterized in that the adaptive strategy generation and remote optimization module includes a multi-strategy generation submodule, a real-time remote optimization submodule and an adaptive driving submodule;

The multi-strategy generation submodule is used to detect that the vehicle encounters different situations and generate multiple response strategies, wherein the multiple response strategies include path replanning, speed adjustment and emergency braking, and the different situations include complex situations or potential dangers encountered by the vehicle;

The real-time remote optimization submodule optimizes the multiple response strategies through the cloud, dynamically adjusts the vehicle's driving plan, and sends control instructions corresponding to the optimized strategies to the vehicle in real time.

4. The driving situation prediction and adaptive strategy generation system based on cloud multimodal large model according to claim 3, characterized in that the adaptive driving submodule generates adaptive driving strategies of different levels based on different situations;

The specific implementation process of the multi-strategy generation submodule is as follows:

Use the deep Q network in reinforcement learning to generate different driving strategies. The driving strategy generation process will consider different driving scenarios and select the optimal strategy based on the current environment state:

Q(s,a)=E[r+βmax' _a Q(s',a')],

Where s represents the current state of the vehicle, including speed, position, and sensor data; a represents a set of actions that can be taken, including braking, steering, acceleration, and deceleration; β is a discount factor; r is an immediate reward; max' _a Q(s', a') represents the maximum Q value of all possible actions in the next state s'; E represents the learning rate; and the driving scenarios include emergency braking, obstacle avoidance, and deceleration;

The specific implementation process of optimizing the multiple response strategies through the cloud in the real-time remote optimization submodule is as follows:

The generated multiple response strategy sets are optimized through genetic algorithm GA, and multiple objective functions are used for multi-objective optimization:

Min{f ₁ (a), f ₂ (a),..., f _n (a)},

Where f _n (a) is the loss function for a certain goal, and Min indicates the final selection of the optimal strategy;

The specific implementation process of the real-time remote optimization submodule to optimize the multiple response strategies through the cloud and dynamically adjust the vehicle's driving plan is as follows:

Real-time remote strategy optimization is achieved through differential evolution algorithm and distributed deep learning model, and each vehicle is dynamically adjusted according to the optimization strategy provided by the cloud.

5. The driving situation prediction and adaptive strategy generation system based on cloud multimodal large model according to claim 4, characterized in that the collaborative driving and multi-vehicle strategy coordination module includes a collaborative situation prediction submodule, a strategy coordination submodule and a collective path planning submodule;

The collaborative situation prediction submodule is used to analyze the multimodal data of all vehicles through the cloud when multiple vehicles are driving in the same area, predict the global driving situation, and generate a multi-vehicle collaborative strategy;

The strategy coordination submodule dynamically adjusts the driving strategy of each vehicle according to the status and driving mission of each vehicle to ensure the mutual coordination among multiple vehicles;

The collective path planning submodule is used to generate a collective path planning when multiple vehicles need to pass through different situational sections together.

6. The driving situation prediction and adaptive strategy generation system based on cloud multimodal large model according to claim 5 is characterized in that the specific implementation process of the collaborative situation prediction submodule is:

Using the global situation prediction algorithm based on the graph neural network GNN, the multi-modal data of multiple vehicles is globally modeled. The status of each vehicle is used as a node, which is updated through the data of neighboring nodes, and finally a global collaborative strategy and global graph structure are formed:

in, is the feature representation of node v in the kth layer, N(v) is the set of neighbor nodes, W ^k is the network weight, and c _vu is the normalization coefficient;

Based on the global graph structure, a distributed optimization Lagrange multiplier algorithm is used to generate multi-vehicle collaborative path planning;

The strategy coordination submodule dynamically adjusts the driving strategy of each vehicle to ensure the mutual coordination between multiple vehicles. The specific implementation process is as follows:

Through real-time data feedback, multi-vehicle strategies are continuously adjusted, and a model based on game theory is used to coordinate multi-vehicle strategies.

7. The driving situation prediction and adaptive strategy generation system based on cloud multimodal large model according to claim 6, characterized in that the cloud driving intelligent monitoring and feedback module includes a global monitoring and scheduling submodule, a feedback optimization submodule and an emergency remote takeover submodule;

The global monitoring and scheduling submodule continuously monitors the operation of vehicles through traffic data and weather data, and automatically triggers strategies for adjustment if dangerous situations or abnormal conditions are detected;

The feedback optimization submodule is used to feed back the execution results and data of the global monitoring and scheduling submodule to the cloud. The execution results and data are used for future scenario prediction and strategy optimization generation, forming a self-learning closed loop;

The emergency remote takeover submodule is used to remotely take over vehicle control via the cloud and implement emergency operations in extreme situations, including vehicle collisions or other serious dangers.

8. The driving situation prediction and adaptive strategy generation system based on cloud multimodal large model according to claim 7 is characterized in that the specific implementation process of continuously monitoring the operation of the vehicle through traffic data and weather data in the global monitoring and scheduling submodule is:

The vehicle's position, speed, and acceleration are monitored in real time through the Bayesian online learning algorithm and Kalman filter, and abnormal conditions are detected based on the dynamic Bayesian network DBN. The Kalman filter update formula is:

_xt+1 ＝ _Axt + _But + _wt ，

^Pt ₊₁ ＝ _APtAT +Q，

Where _xt is the updated state estimate, _ut is the predicted state estimate, _Pt is the covariance matrix, A is the state transfer matrix, B is the control input matrix, and _wt is the process noise;

The specific implementation process of the feedback optimization submodule is as follows:

Regression analysis is used to analyze the results and data preprocessing of the global monitoring and scheduling submodule in real time, and an autoregressive sliding average model is used to perform time series analysis on the vehicle's historical dangerous situations or abnormal state data to generate feature data for vehicle future situation prediction and strategy optimization;

Based on the feature data of the vehicle's future scenario prediction and strategy optimization, combined with external environment data, random forest regression is used to analyze road conditions and weather information to form a self-learning closed loop;

The specific implementation process of the emergency remote takeover submodule is as follows:

When a potential extreme situation is detected, remote control is performed through a rule-based PID controller. The PID control formula is:

Wherein, e(t) is the error signal, _Kp , _Ki , and _Kd are the proportional coefficient, integral coefficient, and differential coefficient, respectively.

9. The driving scenario prediction and adaptive strategy generation system based on cloud multimodal large model according to claim 8, characterized in that the dynamic risk assessment and path optimization module includes a dynamic risk assessment submodule, a path planning and optimization submodule and an emergency route generation submodule;

The dynamic risk assessment submodule is used to analyze potential risks on the road in real time and optimize the vehicle's path according to the severity of the risk;

The path planning and optimization submodule generates an optimal driving path for the vehicle based on the evaluation results of the dynamic risk assessment submodule, while ensuring that the path planning is dynamically updated according to the actual situation during the vehicle driving process;

The emergency route generation submodule is used to generate an emergency path planning route when the vehicle encounters a high-risk situation, and optimize the path planning route in real time.

10. The driving situation prediction and adaptive strategy generation system based on cloud multimodal large model according to claim 9, characterized in that the specific implementation process of the dynamic risk assessment submodule is:

Adopt factor analysis method to pre-process data in real time, extract the characteristics of vehicle position, speed and acceleration, and use autoregressive sliding average model to conduct time series analysis on historical vehicle driving data to generate real-time vehicle characteristic data;

Based on the real-time characteristic data of the vehicle and in combination with external environmental data, Gaussian process regression (GPR) is used to analyze road conditions and weather information, identify and classify road types and road conditions, assess risk levels, and generate risk assessment results;

Gaussian process regression (GPR) is used to analyze road conditions and weather information, identify and classify road types and road conditions, assess risk levels, and predict potential dangerous areas on the road through confidence intervals:

f(x)＝GPR(m(x),k(x,x')),

Among them, m(x) is the mean function, k(x,x') is the kernel function, which is used to measure the similarity between road points;

The specific implementation process of the path planning and optimization submodule is as follows:

The A* algorithm is combined with dynamic programming (DP) to generate the optimal driving route. Through multi-dimensional analysis of the current traffic conditions and road network, alternative routes to bypass potential risk areas are calculated, and the traffic flow and estimated driving time of each route are predicted and evaluated to generate a set of optimal detour routes.

The goal of dynamic programming (DP) is to find the optimal route for the preferred detour path set while ensuring safety through the shortest path algorithm:

J ^* (s) = min _u {g (s, u) + J ^* (f (s, u))},

where J ^* (s) is the optimal cost of state s and g(s,u) is the cost function of taking action u.