CN118163804A - Vehicle control method, device, equipment and medium based on multi-mode sensing - Google Patents
Vehicle control method, device, equipment and medium based on multi-mode sensing Download PDFInfo
- Publication number
- CN118163804A CN118163804A CN202410449662.0A CN202410449662A CN118163804A CN 118163804 A CN118163804 A CN 118163804A CN 202410449662 A CN202410449662 A CN 202410449662A CN 118163804 A CN118163804 A CN 118163804A
- Authority
- CN
- China
- Prior art keywords
- driver
- vehicle
- data
- determining
- agi
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 54
- 230000002996 emotional effect Effects 0.000 claims abstract description 86
- 230000008451 emotion Effects 0.000 claims abstract description 49
- 238000011217 control strategy Methods 0.000 claims abstract description 28
- 238000001514 detection method Methods 0.000 claims abstract description 23
- 230000003993 interaction Effects 0.000 claims abstract description 16
- 230000002159 abnormal effect Effects 0.000 claims abstract description 15
- 238000005516 engineering process Methods 0.000 claims description 18
- 230000009471 action Effects 0.000 claims description 17
- 238000012545 processing Methods 0.000 claims description 17
- 230000006399 behavior Effects 0.000 claims description 15
- 230000008859 change Effects 0.000 claims description 8
- 230000006978 adaptation Effects 0.000 claims description 6
- 238000004891 communication Methods 0.000 claims description 4
- 238000004590 computer program Methods 0.000 claims description 4
- 230000007613 environmental effect Effects 0.000 claims description 3
- 238000012544 monitoring process Methods 0.000 abstract description 27
- 230000000694 effects Effects 0.000 abstract description 9
- 230000006870 function Effects 0.000 description 5
- 238000013135 deep learning Methods 0.000 description 4
- 238000010586 diagram Methods 0.000 description 4
- 230000008878 coupling Effects 0.000 description 3
- 238000010168 coupling process Methods 0.000 description 3
- 238000005859 coupling reaction Methods 0.000 description 3
- 230000002787 reinforcement Effects 0.000 description 3
- 230000004044 response Effects 0.000 description 3
- 241001282135 Poromitra oscitans Species 0.000 description 2
- 206010048232 Yawning Diseases 0.000 description 2
- 230000003542 behavioural effect Effects 0.000 description 2
- 230000008901 benefit Effects 0.000 description 2
- 238000011161 development Methods 0.000 description 2
- 230000004424 eye movement Effects 0.000 description 2
- 230000008921 facial expression Effects 0.000 description 2
- 238000003058 natural language processing Methods 0.000 description 2
- 230000008569 process Effects 0.000 description 2
- 238000006467 substitution reaction Methods 0.000 description 2
- 238000012731 temporal analysis Methods 0.000 description 2
- 238000000700 time series analysis Methods 0.000 description 2
- 230000007704 transition Effects 0.000 description 2
- 201000004569 Blindness Diseases 0.000 description 1
- 230000001133 acceleration Effects 0.000 description 1
- 230000009118 appropriate response Effects 0.000 description 1
- 238000013473 artificial intelligence Methods 0.000 description 1
- 230000015572 biosynthetic process Effects 0.000 description 1
- 230000004397 blinking Effects 0.000 description 1
- 230000004399 eye closure Effects 0.000 description 1
- 230000002452 interceptive effect Effects 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012806 monitoring device Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000008447 perception Effects 0.000 description 1
- 230000002265 prevention Effects 0.000 description 1
- 230000036387 respiratory rate Effects 0.000 description 1
- 231100000430 skin reaction Toxicity 0.000 description 1
- 238000003786 synthesis reaction Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Landscapes
- Control Of Driving Devices And Active Controlling Of Vehicle (AREA)
Abstract
The application provides a vehicle control method, device, equipment and medium based on multi-mode sensing, wherein the method comprises the following steps: based on a multi-modal sensing technique, acquiring a plurality of driver detection data, a plurality of vehicle data and a plurality of environment data detected by a plurality of sensing devices; based on the trained AGI, determining an emotional state of the driver and an optimal control strategy of the vehicle according to the plurality of driver detection data, the plurality of vehicle data and the plurality of environment data; if the emotional state of the driver is low and abnormal, carrying out emotion voice interaction on the driver according to the emotional state of the driver so as to comfort the emotion of the driver; and if the emotional state of the driver is high and abnormal, controlling the vehicle to enter an automatic driving state, and controlling the vehicle to automatically drive according to the optimal control strategy of the vehicle. The method achieves the effects of monitoring the state of the driver and actively intervening the vehicle and the driver according to the state of the driver.
Description
Technical Field
The application relates to the technical field of automatic driving, in particular to a vehicle control method, device, equipment and medium based on multi-mode sensing.
Background
Currently, in the development of automatic driving technology, a driver monitoring system (Driver Monitoring System, DMS) plays a key role, which ensures driving safety by monitoring the behavior and state of the driver. Existing DMS rely primarily on cameras and sensors to monitor driver distraction, fatigue or other uncomfortable conditions. However, these systems have focused mainly on passive monitoring, lack the ability to actively intervene, and cannot take action in the event that the driver has no control over the vehicle.
Disclosure of Invention
In view of the above, the present application aims to provide a vehicle control method, device, equipment and medium based on multi-modal sensing, which can acquire and AGI through multi-modal sensing technology, determine an emotional state of a driver and an optimal control strategy of a vehicle, and comfort the driver or switch in driving of the vehicle according to the emotional state of the driver. The system solves the problems that the existing system in the prior art mainly focuses on passive monitoring, lacks the capability of active intervention and cannot take action under the condition that a driver cannot control a vehicle, and achieves the effects of monitoring the state of the driver and performing active intervention on the vehicle and the driver according to the state of the driver.
In a first aspect, an embodiment of the present application provides a vehicle control method based on multi-modal sensing, the method including: based on a multi-modal sensing technique, acquiring a plurality of driver detection data, a plurality of vehicle data and a plurality of environment data detected by a plurality of sensing devices; based on the trained AGI, determining an emotional state of the driver and an optimal control strategy of the vehicle according to the plurality of driver detection data, the plurality of vehicle data and the plurality of environment data; if the emotional state of the driver is low and abnormal, carrying out emotion voice interaction on the driver according to the emotional state of the driver so as to comfort the emotion of the driver; and if the emotional state of the driver is high and abnormal, controlling the vehicle to enter an automatic driving state, and controlling the vehicle to automatically drive according to the optimal control strategy of the vehicle.
Optionally, the AGI comprises an emotional AGI and an intelligent decision AGI, wherein the driver's emotional state and the vehicle optimal control strategy are determined by: determining an emotional state of the driver according to the plurality of driver detection data based on the trained emotional AGI; based on the trained intelligent decision AGI, a vehicle optimal control strategy is determined according to the plurality of vehicle data and the plurality of environment data.
Optionally, the plurality of driver detection data includes a driver image, a driver infrared image, belt pressure data, driver physiological data, driver behavior data, wherein the step of determining the emotional state of the driver from the plurality of driver detection data comprises: identifying a driver image and a driver infrared image, and determining a first emotional state of the driver according to a plurality of target actions of the driver in the driver image and the driver infrared image; determining a sitting posture change frequency of the driver according to the safety belt pressure data, and determining a second emotional state of the driver according to the sitting posture change frequency of the driver; determining a plurality of physiological indexes of the driver according to the physiological data of the driver, and determining a third emotional state of the driver according to the plurality of physiological indexes of the driver; determining a fourth emotional state of the driver according to the driver behavior data; determining the emotional state of the driver according to the first emotional state, the second emotional state, the third emotional state and the fourth emotional state.
Optionally, the emotional state of the driver includes a plurality of emotional sub-states of the driver, and the emotional speech includes a plurality of emotional sub-voices, wherein the step of performing emotional speech interaction on the driver according to the emotional state of the driver includes: determining at least one emotion sub-voice corresponding to the emotion sub-state of the at least one driver according to the emotion sub-state of the at least one driver; at least one emotion sub-voice is played through the sound of the vehicle to comfort the emotion of the driver.
Optionally, the intelligent decision AGI includes a path planning AGI, a speed control AGI, an obstacle avoidance AGI, an emergency handling AGI, and a driver status adaptation AGI, wherein an optimal control strategy for the vehicle is determined by: based on the path planning AGI, determining an optimal path of the vehicle according to the starting point of the vehicle, traffic environment information, weather environment information and road information in the plurality of environment data; determining an optimal speed of the vehicle according to the road type information, the traffic rule information and the current traffic flow in the plurality of environmental data based on the speed control AGI; based on the obstacle avoidance AGI, determining an avoidance route of the vehicle according to a plurality of obstacle data in a plurality of vehicle data; based on the emergency processing AGI, determining whether the vehicle has an emergency according to a plurality of vehicle data and a plurality of environment data, and controlling the vehicle to take preset emergency protection measures when the vehicle has the emergency so as to protect the safety of passengers and pedestrians; based on the driver state adaptation AGI, determining a driving mode of the vehicle according to the driver's emotional state, the driving mode of the vehicle being used to control a highest driving speed of the vehicle to ensure driving safety.
Optionally, the method further comprises: acquiring a voice signal of a driver; processing the voice signal based on a voice recognition algorithm to obtain a processed voice text; performing intention recognition according to the processed voice text, and determining the voice intention of a driver; determining a reply voice or a control instruction according to the voice intention of the driver based on a predefined decision model; and replying or controlling the vehicle action to the driver according to the replying voice or the control instruction.
In a second aspect, an embodiment of the present application further provides a vehicle control device based on multi-modal sensing, where the device includes:
the data sensing module is used for acquiring a plurality of driver detection data, a plurality of vehicle data and a plurality of environment data which are detected by a plurality of sensing devices based on a multi-mode sensing technology;
The state determining module is used for determining the emotional state of the driver and the optimal control strategy of the vehicle according to the plurality of driver detection data, the plurality of vehicle data and the plurality of environment data based on the trained AGI;
the emotion interaction module is used for carrying out emotion voice interaction on the driver according to the emotion state of the driver if the emotion state of the driver is low and abnormal so as to comfort the emotion of the driver;
And the automatic driving module is used for controlling the vehicle to enter an automatic driving state if the emotion state of the driver is high and abnormal, and controlling the vehicle to automatically drive according to the optimal control strategy of the vehicle.
Optionally, the apparatus further comprises:
the voice signal acquisition module is used for acquiring a voice signal of a driver;
the voice signal processing module is used for processing the voice signal based on a voice recognition algorithm to obtain a processed voice text;
the voice intention determining module is used for carrying out intention recognition according to the processed voice text and determining the voice intention of the driver;
the decision model decision module is used for determining a reply voice or a control instruction according to the voice intention of the driver based on a predefined decision model;
And the vehicle control module is used for replying or controlling the vehicle action to the driver according to the replying voice or the control instruction.
In a third aspect, an embodiment of the present application further provides an electronic device, including: the system comprises a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, the processor and the memory are communicated through the bus when the electronic device is running, and the machine-readable instructions are executed by the processor to perform the steps of the vehicle control method based on multi-mode sensing.
In a fourth aspect, embodiments of the present application also provide a computer readable storage medium having a computer program stored thereon, which when executed by a processor performs the steps of the multimodal perception based vehicle control method as described above.
According to the vehicle control method, device, equipment and medium based on multi-mode sensing, which are provided by the embodiment of the application, the emotional state of the driver and the optimal control strategy of the vehicle can be determined by acquiring and AGI through the multi-mode sensing technology, and the driver is comforted or driven by accessing the vehicle according to the emotional state of the driver. The system solves the problems that the existing system in the prior art mainly focuses on passive monitoring, lacks the capability of active intervention and cannot take action under the condition that a driver cannot control a vehicle, and achieves the effects of monitoring the state of the driver and performing active intervention on the vehicle and the driver according to the state of the driver.
In order to make the above objects, features and advantages of the present application more comprehensible, preferred embodiments accompanied with figures are described in detail below.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings that are needed in the embodiments will be briefly described below, it being understood that the following drawings only illustrate some embodiments of the present application and therefore should not be considered as limiting the scope, and other related drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a flow chart of a vehicle control method based on multi-modal awareness according to an embodiment of the present application;
FIG. 2 is a schematic structural diagram of a vehicle control device based on multi-modal sensing according to an embodiment of the present application;
fig. 3 is a schematic structural diagram of an electronic device according to an embodiment of the present application.
Detailed Description
For the purpose of making the objects, technical solutions and advantages of the embodiments of the present application more apparent, the technical solutions of the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application, and it is apparent that the described embodiments are only some embodiments of the present application, not all embodiments. The components of the embodiments of the present application generally described and illustrated in the figures herein may be arranged and designed in a wide variety of different configurations. Thus, the following detailed description of the embodiments of the application, as presented in the figures, is not intended to limit the scope of the application, as claimed, but is merely representative of selected embodiments of the application. Based on the embodiments of the present application, every other embodiment obtained by a person skilled in the art without making any inventive effort falls within the scope of protection of the present application.
First, an application scenario to which the present application is applicable will be described. The application can be applied to the technical field of automatic driving.
It has been found that, at present, during the development of automatic driving technology, a driver monitoring system (Driver Monitoring System, DMS) plays a key role in ensuring driving safety by monitoring the behavior and state of the driver. Existing DMS rely primarily on cameras and sensors to monitor driver distraction, fatigue or other uncomfortable conditions. However, these systems have focused mainly on passive monitoring, lack the ability to actively intervene, and cannot take action in the event that the driver has no control over the vehicle.
Based on the above, the embodiment of the application provides a vehicle control method based on multi-mode sensing, which can acquire and AGI through the multi-mode sensing technology, determine the emotional state of a driver and the optimal control strategy of the vehicle, and comfort the driver or access the driving of the vehicle according to the emotional state of the driver. The system solves the problems that the existing system in the prior art mainly focuses on passive monitoring, lacks the capability of active intervention and cannot take action under the condition that a driver cannot control a vehicle, and achieves the effects of monitoring the state of the driver and performing active intervention on the vehicle and the driver according to the state of the driver.
Referring to fig. 1, fig. 1 is a flowchart of a vehicle control method based on multi-mode sensing according to an embodiment of the application. As shown in fig. 1, a vehicle control method based on multi-mode sensing provided by an embodiment of the present application includes:
S101, acquiring a plurality of driver detection data, a plurality of vehicle data and a plurality of environment data which are detected by a plurality of sensing devices based on a multi-mode sensing technology.
Here, the plurality of sensing devices may include video surveillance cameras, infrared cameras, pressure sensors, physiological sensors, behavioral monitoring systems, and the like.
For example, video surveillance cameras are typically mounted on an instrument panel, at the front windshield of a vehicle, or at the roof of the vehicle, facing the driver. Facial expressions, eye closure frequency, gaze direction, head pose, etc. of the driver may be detected, and whether the driver is tired or distracted may be assessed by analyzing the facial expressions and eye movement patterns. For example, frequent blinks, yawning, or prolonged periods of road blindness may indicate fatigue driving, etc., are used to apply the video surveillance camera data.
The infrared camera is installed like a video monitoring camera and faces a driver, and eye movement and head posture of the driver can be detected in a dark-light environment. The driver's attention and fatigue can be evaluated at night or under light shortage.
The pressure sensor is mounted on the driver seat and/or the safety belt. The driver's sitting posture change and the tension of the seat belt can be detected. Frequent changes in the passable seat may indicate discomfort or distraction to the driver, and the usage habits of the seat belt may also be part of the assessment.
The physiological sensor may be worn directly on the driver, such as a sensor embedded in a wristwatch or seat. Can detect physiological indexes such as heart rate, skin electric activity, respiratory rate and the like. Physiological data detected by physiological sensors may reflect the driver's emotional state and stress level, such as an abnormal increase in heart rate may indicate tension or excessive concentration.
The behavior monitoring system may incorporate cameras and other sensors within the vehicle. The detected data includes driving behavior such as steering frequency, accelerator and brake pedal usage, vehicle speed, etc. The driving style and possible distraction of the driver can be deduced by analyzing the driving behavior.
S102, determining the emotion state of the driver and the optimal control strategy of the vehicle according to the plurality of driver detection data, the plurality of vehicle data and the plurality of environment data based on the trained AGI.
The AGI comprises emotion AGI and intelligent decision AGI.
Specifically, the emotional state of the driver and the optimal control strategy of the vehicle are determined by the following steps: determining an emotional state of the driver according to the plurality of driver detection data based on the trained emotional AGI; based on the trained intelligent decision AGI, a vehicle optimal control strategy is determined according to the plurality of vehicle data and the plurality of environment data.
Wherein the intelligent decision AGI comprises a path planning AGI, a speed control AGI, an obstacle avoidance AGI, an emergency processing AGI and a driver state adaptation AGI,
Specifically, an optimal control strategy for a vehicle may be determined by: based on the path planning AGI, determining an optimal path of the vehicle according to the starting point of the vehicle, traffic environment information, weather environment information and road information in the plurality of environment data; determining an optimal speed of the vehicle according to the road type information, the traffic rule information and the current traffic flow in the plurality of environmental data based on the speed control AGI; based on the obstacle avoidance AGI, determining an avoidance route of the vehicle according to a plurality of obstacle data in a plurality of vehicle data; based on the emergency processing AGI, determining whether the vehicle has an emergency according to a plurality of vehicle data and a plurality of environment data, and controlling the vehicle to take preset emergency protection measures when the vehicle has the emergency so as to protect the safety of passengers and pedestrians; based on the driver state adaptation AGI, determining a driving mode of the vehicle according to the driver's emotional state, the driving mode of the vehicle being used to control a highest driving speed of the vehicle to ensure driving safety.
The plurality of driver detection data includes driver images, driver infrared images, belt pressure data, driver physiological data, and driver behavior data.
Specifically, the step of determining the emotional state of the driver according to the plurality of driver detection data includes: identifying a driver image and a driver infrared image, and determining a first emotional state of the driver according to a plurality of target actions of the driver in the driver image and the driver infrared image; determining a sitting posture change frequency of the driver according to the safety belt pressure data, and determining a second emotional state of the driver according to the sitting posture change frequency of the driver; determining a plurality of physiological indexes of the driver according to the physiological data of the driver, and determining a third emotional state of the driver according to the plurality of physiological indexes of the driver; determining a fourth emotional state of the driver according to the driver behavior data; determining the emotional state of the driver according to the first emotional state, the second emotional state, the third emotional state and the fourth emotional state.
S103, if the emotion state of the driver is low and abnormal, performing emotion voice interaction on the driver according to the emotion state of the driver so as to comfort the emotion of the driver
Wherein the emotional state of the driver comprises a plurality of emotional sub-states of the driver, and the emotional speech comprises a plurality of emotional sub-speech.
Specifically, according to the emotional state of the driver, the step of performing emotion voice interaction on the driver includes: determining at least one emotion sub-voice corresponding to the emotion sub-state of the at least one driver according to the emotion sub-state of the at least one driver; at least one emotion sub-voice is played through the sound of the vehicle to comfort the emotion of the driver.
And S104, if the emotion state of the driver is high and abnormal, controlling the vehicle to enter an automatic driving state, and controlling the vehicle to automatically drive according to the optimal control strategy of the vehicle.
It should be noted that, in the automatic driving technology, the physical intervention driving of the robot actuator, including automatic adjustment of the steering wheel or control of the brake, is a complex process, and requires accurate judgment and execution. Such interventions are typically based on the output of a series of algorithms and models driven by Artificial Intelligence (AI) or Artificial General Intelligence (AGI) systems. The following is a detailed description of how the intervention timing, the intervention modality, and the stopping of the intervention and the restoration of driver control are determined.
The robotic actuators may intervene in driving in situations that are typically based on data analyzed by the AGI system, such as emergency avoidance: when the system detects an impending collision or dangerous condition (e.g., sudden braking of a lead vehicle, crossing of a road by a pedestrian), it automatically intervenes to avoid or mitigate the collision. Driver status monitoring: if the system judges that the driver is tired, distracted or unsuitable to drive through analyzing the physiological and behavioral data of the driver, the system can automatically intervene to ensure the driving safety. Technical problems detected by the system: if the system of the vehicle fails, the automatic intervention adjustment is performed when the driving safety is affected. Automatic driving mode: in the full-automatic driving mode, the execution mechanism can control the vehicle in the whole course to execute navigation and driving tasks.
Here, the function of the intervention control may be determined by various conditions, such as collision prevention: the emergency operation is carried out by preferentially intervening in a brake system and direction control. Driver fatigue or distraction: speed control (e.g., deceleration) may be interposed as appropriate while the driver is alerted by an audio or visual signal. Technical failure: related vehicle systems are interposed for safety control, such as limiting speed or guiding the vehicle to a safe stop if the engine is problematic. Automatic driving mode: the direction, speed and other functions of the vehicle are comprehensively controlled, and the vehicle is dynamically adjusted according to the running environment and the route.
Here, intervention may be stopped and driver control resumed after a certain condition is met, e.g. if the driver explicitly indicates that control is desired to resume (by operating the steering wheel, the pedal or using a cancel button), the system should be able to respond immediately. Once the system determines that the emergency has been successfully avoided or resolved, it will gradually reduce the intervention, eventually stopping the intervention altogether, and reverting to normal driving mode. If the system detects that technical problems or instability occurs during the intervention, the intervention should be automatically reduced until the intervention is completely stopped, and the driver is prompted to take over. When the system determines that the vehicle has reached a safe state and the external environment allows driver control to resume (e.g., vehicle speed falls within a safe range, no emergency risk avoidance needs are surrounding).
In an alternative embodiment, computer vision techniques, such as face recognition and eye tracking, may be used first to analyze the driver's gaze direction and blink frequency. Heart rate variability, galvanic skin activity, etc. are analyzed based on the application of signal processing techniques to assess the emotional state and fatigue level of the driver. And analyzing information such as vehicle speed change, acceleration, steering wheel rotation angle and the like, and judging whether the driving behavior is standard or not. And extracting the characteristics of the preprocessed data by using a deep learning algorithm, and identifying the characteristic modes of states such as fatigue, distraction and the like of a driver. Based on the extracted features, reinforcement learning or other intelligent algorithms are used to assess the safety risk of the current driving situation. The algorithm comprehensively considers the state of the driver and the behavior of the vehicle, and judges whether intervention is needed, such as giving out a warning or taking automatic driving measures. If the system decides that intervention is required, the manner of intervention (such as sounding or light warning, automatically adjusting the speed of the vehicle, or even taking over vehicle control) will be dynamically determined based on the real-time status of the driver and the driving situation of the vehicle. Meanwhile, the system can continuously monitor the intervention effect, and adjust the intervention strategy according to the real-time data to ensure the optimal safety measure.
For example, frequent blinking and yawning of the driver is captured by the dashboard camera, while the physiological monitoring device displays a slowed heart rate and reduced galvanic skin response, and vehicle sensors register fluctuations in vehicle speed and frequent minor direction adjustments. After the comprehensive data are analyzed by the deep learning algorithm, the fatigue driving of the driver is identified. And then warning or taking automatic driving measures according to the fatigue driving degree of the driver.
Here, a new type of driver monitoring system, called driver robot system (Driver Robot System, DRS), can be formed based on a multi-modal aware vehicle control method, which can not only monitor the state of the driver, but also actively intervene in driving when necessary, ensuring driving safety. The DRS system realizes the transition from monitoring to intelligent interaction by combining a latest Artificial General Intelligent (AGI) technology with a robot technology.
Optionally, the method further comprises: acquiring a voice signal of a driver; processing the voice signal based on a voice recognition algorithm to obtain a processed voice text; performing intention recognition according to the processed voice text, and determining the voice intention of a driver; determining a reply voice or a control instruction according to the voice intention of the driver based on a predefined decision model; and replying or controlling the vehicle action to the driver according to the replying voice or the control instruction.
Natural Language Processing (NLP) allows the system to provide instructions, warnings, or other notifications in natural language form in applications that communicate with the driver, thereby improving the intuitiveness and efficiency of the interaction. This technique involves multiple steps of speech recognition, understanding, generation and synthesis, with the driver's speech input being received first. This is typically done by a microphone built into the vehicle. The speech signal is converted into digital form and then converted into text by a speech recognition algorithm. And relates to denoising, voice slicing, pattern matching and other technologies. And carrying out natural language understanding processing on the converted text, and analyzing the grammar structure, intention and semantics of the text. This may involve entity recognition, intent classification, etc. A specific intent of the driver's request or problem is determined. For example, the driver may ask for a route, need to control a particular function of the vehicle, or express a concern about the state of the vehicle. Based on the output of the understanding stage, the system uses a predefined logic or machine learning model to determine the most appropriate response. This may include performing an action (e.g., adjusting temperature), providing information (e.g., current speed), or generating a follow-up problem that requires further information. A natural language response is generated based on the decision logic. This process may be accomplished by simple template filling or more complex Natural Language Generation (NLG) techniques. Finally, the system converts the generated text response to speech using text-to-speech (TTS) technology. This allows the driver to hear natural, smooth voice feedback. The voice is played to the driver through the speaker system of the vehicle.
In all cases, safety is considered to ensure that the provided instructions or warnings are not distracted by the driver, especially during driving. Adjustments may be made according to driver preferences and behavior patterns to provide a more personalized interactive experience. For example, if the system notices a brief instruction that the driver prefers, it can adjust its response accordingly.
The intelligent driving system not only can monitor the state of a driver, but also can actively intervene in driving when necessary, so that the driving safety is ensured. The DRS system realizes the transition from monitoring to intelligent interaction by combining a latest Artificial General Intelligent (AGI) technology with a robot technology.
A deep learning (DEEP LEARNING) algorithm may be used: the method is used for processing and analyzing complex data such as video streams, physiological signals and the like. Reinforcement learning (Reinforcement Learning, RL) algorithm: for dynamically adjusting system behavior to optimize driving intervention strategies. Time series analysis (TIME SERIES ANALYSIS): algorithms for monitoring and predicting the trend of changes in driver status, etc.
According to the vehicle control method based on multi-mode sensing, the emotional state of the driver and the optimal control strategy of the vehicle can be determined through the multi-mode sensing technology to obtain and AGI, and the driver is comforted or connected with the vehicle to drive according to the emotional state of the driver. The system solves the problems that the existing system in the prior art mainly focuses on passive monitoring, lacks the capability of active intervention and cannot take action under the condition that a driver cannot control a vehicle, and achieves the effects of monitoring the state of the driver and performing active intervention on the vehicle and the driver according to the state of the driver.
Based on the same inventive concept, the embodiment of the application further provides a vehicle control device based on multi-modal sensing corresponding to the vehicle control method based on multi-modal sensing, and since the principle of solving the problem by the device in the embodiment of the application is similar to that of the vehicle control method based on multi-modal sensing in the embodiment of the application, the implementation of the device can refer to the implementation of the method, and the repetition is omitted.
Referring to fig. 2, fig. 2 is a schematic structural diagram of a vehicle control device based on multi-mode sensing according to an embodiment of the application. As shown in fig. 2, the vehicle control apparatus 200 based on multi-modal awareness includes:
A data sensing module 201, configured to acquire a plurality of driver detection data, a plurality of vehicle data, and a plurality of environment data detected by a plurality of sensing devices based on a multi-modal sensing technology;
A state determining module 202, configured to determine an emotional state of a driver and a vehicle optimal control strategy according to the plurality of driver detection data, the plurality of vehicle data, and the plurality of environment data based on the trained AGI;
the emotion interaction module 203 is configured to perform emotion voice interaction on the driver according to the emotional state of the driver if the emotional state of the driver is low and abnormal, so as to comfort the emotion of the driver;
the automatic driving module 204 is configured to control the vehicle to enter an automatic driving state if the emotional state of the driver is high and abnormal, and control the vehicle to automatically drive according to an optimal control strategy of the vehicle.
Optionally, the apparatus further comprises: the voice signal acquisition module is used for acquiring a voice signal of a driver;
the voice signal processing module is used for processing the voice signal based on a voice recognition algorithm to obtain a processed voice text;
the voice intention determining module is used for carrying out intention recognition according to the processed voice text and determining the voice intention of the driver;
the decision model decision module is used for determining a reply voice or a control instruction according to the voice intention of the driver based on a predefined decision model;
And the vehicle control module is used for replying or controlling the vehicle action to the driver according to the replying voice or the control instruction.
The vehicle control device based on the multi-mode sensing provided by the embodiment of the application can acquire and AGI through the multi-mode sensing technology, determine the emotion state of the driver and the optimal control strategy of the vehicle, and comfort or access the driver to drive the vehicle according to the emotion state of the driver. The system solves the problems that the existing system in the prior art mainly focuses on passive monitoring, lacks the capability of active intervention and cannot take action under the condition that a driver cannot control a vehicle, and achieves the effects of monitoring the state of the driver and performing active intervention on the vehicle and the driver according to the state of the driver.
Referring to fig. 3, fig. 3 is a schematic structural diagram of an electronic device according to an embodiment of the application. As shown in fig. 3, the electronic device 300 includes a processor 310, a memory 320, and a bus 330.
The memory 320 stores machine-readable instructions executable by the processor 310, and when the electronic device 300 is running, the processor 310 communicates with the memory 320 through the bus 330, and when the machine-readable instructions are executed by the processor 310, the steps of the method for controlling a vehicle based on multi-modal awareness in the method embodiment shown in fig. 1 can be executed, and detailed description of the method embodiment will be omitted.
The embodiment of the present application further provides a computer readable storage medium, where a computer program is stored on the computer readable storage medium, and when the computer program is executed by a processor, the steps of the vehicle control method based on multi-mode sensing in the method embodiment shown in fig. 1 may be executed, and a specific implementation manner may refer to the method embodiment and will not be described herein.
It will be clear to those skilled in the art that, for convenience and brevity of description, specific working procedures of the above-described systems, apparatuses and units may refer to corresponding procedures in the foregoing method embodiments, and are not repeated herein.
In the several embodiments provided by the present application, it should be understood that the disclosed systems, devices, and methods may be implemented in other manners. The above-described apparatus embodiments are merely illustrative, for example, the division of the units is merely a logical function division, and there may be other manners of division in actual implementation, and for example, multiple units or components may be combined or integrated into another system, or some features may be omitted, or not performed. Alternatively, the coupling or direct coupling or communication connection shown or discussed with each other may be through some communication interface, device or unit indirect coupling or communication connection, which may be in electrical, mechanical or other form.
The units described as separate units may or may not be physically separate, and units shown as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
In addition, each functional unit in the embodiments of the present application may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit.
The functions, if implemented in the form of software functional units and sold or used as a stand-alone product, may be stored in a non-volatile computer readable storage medium executable by a processor. Based on this understanding, the technical solution of the present application may be embodied essentially or in a part contributing to the prior art or in a part of the technical solution, in the form of a software product stored in a storage medium, comprising several instructions for causing a computer device (which may be a personal computer, a server, a network device, etc.) to perform all or part of the steps of the method according to the embodiments of the present application. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a random access Memory (Random Access Memory, RAM), a magnetic disk, or an optical disk, or other various media capable of storing program codes.
Finally, it should be noted that: the above examples are only specific embodiments of the present application, and are not intended to limit the scope of the present application, but it should be understood by those skilled in the art that the present application is not limited thereto, and that the present application is described in detail with reference to the foregoing examples: any person skilled in the art may modify or easily conceive of the technical solution described in the foregoing embodiments, or perform equivalent substitution of some of the technical features, while remaining within the technical scope of the present disclosure; such modifications, changes or substitutions do not depart from the spirit and scope of the technical solutions of the embodiments of the present application, and are intended to be included in the scope of the present application. Therefore, the protection scope of the application is subject to the protection scope of the claims.
Claims (10)
1. A method of vehicle control based on multi-modal awareness, the method comprising:
Based on a multi-modal sensing technique, acquiring a plurality of driver detection data, a plurality of vehicle data and a plurality of environment data detected by a plurality of sensing devices;
based on the trained AGI, determining an emotional state of the driver and an optimal control strategy of the vehicle according to the plurality of driver detection data, the plurality of vehicle data and the plurality of environment data;
If the emotional state of the driver is low and abnormal, carrying out emotion voice interaction on the driver according to the emotional state of the driver so as to comfort the emotion of the driver;
And if the emotional state of the driver is high and abnormal, controlling the vehicle to enter an automatic driving state, and controlling the vehicle to automatically drive according to the optimal control strategy of the vehicle.
2. The method of claim 1, wherein the AGI comprises an emotion AGI and an intelligent decision AGI,
Wherein the driver's emotional state and the vehicle optimal control strategy are determined by:
Determining an emotional state of the driver according to the plurality of driver detection data based on the trained emotional AGI;
based on the trained intelligent decision AGI, a vehicle optimal control strategy is determined according to the plurality of vehicle data and the plurality of environment data.
3. The method of claim 1, wherein the plurality of driver detection data comprises a driver image, a driver infrared image, belt pressure data, driver physiological data, driver behavior data,
Wherein the step of determining the emotional state of the driver based on the plurality of driver detection data comprises:
identifying a driver image and a driver infrared image, and determining a first emotional state of the driver according to a plurality of target actions of the driver in the driver image and the driver infrared image;
determining a sitting posture change frequency of the driver according to the safety belt pressure data, and determining a second emotional state of the driver according to the sitting posture change frequency of the driver;
Determining a plurality of physiological indexes of the driver according to the physiological data of the driver, and determining a third emotional state of the driver according to the plurality of physiological indexes of the driver;
Determining a fourth emotional state of the driver according to the driver behavior data;
determining the emotional state of the driver according to the first emotional state, the second emotional state, the third emotional state and the fourth emotional state.
4. The method of claim 1 wherein the driver's emotional state comprises a plurality of driver emotional sub-states, the emotional speech comprises a plurality of emotional sub-speech,
The step of carrying out emotion voice interaction on the driver according to the emotion state of the driver comprises the following steps:
Determining at least one emotion sub-voice corresponding to the emotion sub-state of the at least one driver according to the emotion sub-state of the at least one driver;
At least one emotion sub-voice is played through the sound of the vehicle to comfort the emotion of the driver.
5. The method of claim 4, wherein the intelligent decision AGI comprises a path planning AGI, a speed control AGI, an obstacle avoidance AGI, an emergency handling AGI, and a driver status adaptation AGI,
Wherein the optimal control strategy for the vehicle is determined by:
Based on the path planning AGI, determining an optimal path of the vehicle according to the starting point of the vehicle, traffic environment information, weather environment information and road information in the plurality of environment data;
determining an optimal speed of the vehicle according to the road type information, the traffic rule information and the current traffic flow in the plurality of environmental data based on the speed control AGI;
based on the obstacle avoidance AGI, determining an avoidance route of the vehicle according to a plurality of obstacle data in a plurality of vehicle data;
Based on the emergency processing AGI, determining whether the vehicle has an emergency according to a plurality of vehicle data and a plurality of environment data, and controlling the vehicle to take preset emergency protection measures when the vehicle has the emergency so as to protect the safety of passengers and pedestrians;
based on the driver state adaptation AGI, determining a driving mode of the vehicle according to the driver's emotional state, the driving mode of the vehicle being used to control a highest driving speed of the vehicle to ensure driving safety.
6. The method according to claim 1, wherein the method further comprises:
Acquiring a voice signal of a driver;
Processing the voice signal based on a voice recognition algorithm to obtain a processed voice text;
performing intention recognition according to the processed voice text, and determining the voice intention of a driver;
determining a reply voice or a control instruction according to the voice intention of the driver based on a predefined decision model;
and replying or controlling the vehicle action to the driver according to the replying voice or the control instruction.
7. A vehicle control apparatus based on multi-modal awareness, the apparatus comprising:
the data sensing module is used for acquiring a plurality of driver detection data, a plurality of vehicle data and a plurality of environment data which are detected by a plurality of sensing devices based on a multi-mode sensing technology;
The state determining module is used for determining the emotional state of the driver and the optimal control strategy of the vehicle according to the plurality of driver detection data, the plurality of vehicle data and the plurality of environment data based on the trained AGI;
the emotion interaction module is used for carrying out emotion voice interaction on the driver according to the emotion state of the driver if the emotion state of the driver is low and abnormal so as to comfort the emotion of the driver;
And the automatic driving module is used for controlling the vehicle to enter an automatic driving state if the emotion state of the driver is high and abnormal, and controlling the vehicle to automatically drive according to the optimal control strategy of the vehicle.
8. The apparatus of claim 7, wherein the apparatus further comprises:
the voice signal acquisition module is used for acquiring a voice signal of a driver;
the voice signal processing module is used for processing the voice signal based on a voice recognition algorithm to obtain a processed voice text;
the voice intention determining module is used for carrying out intention recognition according to the processed voice text and determining the voice intention of the driver;
the decision model decision module is used for determining a reply voice or a control instruction according to the voice intention of the driver based on a predefined decision model;
And the vehicle control module is used for replying or controlling the vehicle action to the driver according to the replying voice or the control instruction.
9. An electronic device, comprising: a processor, a memory and a bus, the memory storing machine-readable instructions executable by the processor, the processor and the memory in communication over the bus when the electronic device is running, the processor executing the machine-readable instructions to perform the steps of the method of any one of claims 1 to 6.
10. A computer-readable storage medium, characterized in that it has stored thereon a computer program which, when executed by a processor, performs the steps of the method according to any of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410449662.0A CN118163804A (en) | 2024-04-15 | 2024-04-15 | Vehicle control method, device, equipment and medium based on multi-mode sensing |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410449662.0A CN118163804A (en) | 2024-04-15 | 2024-04-15 | Vehicle control method, device, equipment and medium based on multi-mode sensing |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN118163804A true CN118163804A (en) | 2024-06-11 |
Family
ID=91352939
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202410449662.0A Pending CN118163804A (en) | 2024-04-15 | 2024-04-15 | Vehicle control method, device, equipment and medium based on multi-mode sensing |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN118163804A (en) |
-
2024
- 2024-04-15 CN CN202410449662.0A patent/CN118163804A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11709488B2 (en) | Manual control re-engagement in an autonomous vehicle | |
| CN106803423B (en) | Man-machine interaction voice control method and device based on user emotion state and vehicle | |
| EP3286058B1 (en) | Apparatus and method for controlling a user situation awareness modification of a user of a vehicle, and a user situation awareness modification processing system | |
| EP3588372B1 (en) | Controlling an autonomous vehicle based on passenger behavior | |
| EP3779921A1 (en) | Information processing device, moving apparatus, method, and program | |
| EP3286057B1 (en) | Apparatus and method for controlling a user situation awareness modification of a user of a vehicle, and a user situation awareness modification processing system | |
| CN112424848B (en) | Warning device and driving tendency analysis method | |
| JP4841425B2 (en) | Method and mechanism for controlling automobile subsystems based on driver behavior interpretation | |
| US9558414B1 (en) | Method for calculating a response time | |
| JP6353383B2 (en) | Driver support system control method | |
| CN111373335A (en) | Method and system for driving mode switching based on self-awareness performance parameters in hybrid driving | |
| US20140125474A1 (en) | Adaptive actuator interface for active driver warning | |
| EP3885220B1 (en) | Automatically estimating skill levels and confidence levels of drivers | |
| EP3889740A1 (en) | Affective-cognitive load based digital assistant | |
| WO2019044427A1 (en) | Assist method, and assist system and assist device using same | |
| US11603104B2 (en) | Driver abnormality determination system, method and computer program | |
| EP3755596B1 (en) | Cognitive state-based stimuli | |
| US11427208B2 (en) | Driver condition determination apparatus, method and computer program product | |
| KR20190134909A (en) | The apparatus and method for Driver Status Recognition based on Driving Status Decision Information | |
| US20210316737A1 (en) | Driver abnormality determination apparatus, method and computer program | |
| US11878714B2 (en) | Techniques for customizing self-driving models | |
| CN118163804A (en) | Vehicle control method, device, equipment and medium based on multi-mode sensing | |
| US20230404456A1 (en) | Adjustment device, adjustment system, and adjustment method | |
| WO2022258150A1 (en) | Computer-implemented method of assisting a driver, computer program product, driving assistance system, and vehicle | |
| CN114469097A (en) | Man-machine common driving takeover state testing method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination |