WO2023092611A1

WO2023092611A1 - Information broadcasting method and apparatus, traffic state prompting method and apparatus, and vehicle

Info

Publication number: WO2023092611A1
Application number: PCT/CN2021/134178
Authority: WO
Inventors: 格拉多·罗萨诺; 周游; 陈晓智; 张谷力; 高健博
Original assignee: 深圳市大疆创新科技有限公司; 上汽大众汽车有限公司
Priority date: 2021-11-29
Filing date: 2021-11-29
Publication date: 2023-06-01
Also published as: CN117836853A

Abstract

Provided in the present application are an information broadcasting method and apparatus, a traffic state prompting method and apparatus, and a vehicle. The information broadcasting method is applied to a vehicle equipped with a camera apparatus, which faces a user seat inside the vehicle. The information broadcasting method comprises: determining, on the basis of image information that is collected by a camera apparatus, expression information of a user who is located in a user seat; acquiring specific state information of a vehicle, wherein the specific state information comprises one or more of the following: alarm information that is outputted by the vehicle, or actual operation state information of an execution module when the vehicle has received a user instruction for instructing the execution module of the vehicle to change an operation state, but has not executed the user instruction; and if an expression feature in the expression information matches a preset expression feature for representing puzzlement, acquiring parsing information on the basis of the specific state information, and broadcasting the parsing information.

Description

An information broadcasting method, a traffic state prompting method, device and vehicle

technical field

The present application relates to the technical field of automobiles, in particular to an information broadcasting method, a traffic state prompting method, a device and a vehicle.

Background technique

With the rapid development of the field of autonomous driving, the intelligence and autonomy of autonomous driving technology are increasing day by day, and its application scenarios are becoming more and more abundant. Based on factors such as the role assignment of the automatic driving system in performing dynamic driving tasks and the presence or absence of design operating conditions, the automatic driving technology can be divided into six levels from manual driving to fully automated driving. There is still a long way to go to realize fully automated driving. In the progressive autonomous driving technology, L2 and L3 belong to the human-vehicle co-driving mode. Authority, how to ensure the safety control of the vehicle, is a technical issue that has been concerned in this field.

Contents of the invention

In view of this, one of the purposes of the present application is to provide an information broadcasting method, a traffic state prompting method, a device, and a vehicle, so as to improve the safety control of the vehicle in the shared driving mode of people and vehicles.

In order to achieve the above technical effects, the embodiments of the present invention disclose the following technical solutions:

In the first aspect, there is provided an information broadcasting method, which is applied to a vehicle equipped with a camera device, and the camera device faces a user seat inside the vehicle, and the method includes:

determining facial expression information of a user located in the user seat based on the image information collected by the camera device;

Acquiring specific status information of the vehicle, the specific status information includes one or more of the following: alarm information output by the vehicle, or the vehicle receives a message indicating that the execution module of the vehicle changes working status The actual working status information of the execution module when the user instruction is not executed, and the user instruction is not executed;

If the expression feature in the expression information matches the preset expression feature representing doubt, the analysis information is obtained based on the working state information, and the analysis information is broadcast.

In a second aspect, a method for prompting traffic status is provided, the method is applied to a vehicle, and the vehicle is equipped with a sensor facing the outside of the vehicle and a camera device facing a user seat inside the vehicle; the method includes :

Based on the road information collected by the sensor, it is determined that the road where the vehicle is located changes from a non-traffic state to a traffic state;

Obtaining the expected motion state of the vehicle corresponding to the passing state;

Obtain the current motion state of the vehicle;

determining the characteristics of the driving state of the user located in the user seat based on the images collected by the camera device;

If the current motion state of the vehicle does not reach the expected motion state, and the feature of the driving state of the user matches the state feature for indicating inattentive driving, output a Status message.

In a third aspect, an information broadcasting method is provided, the method is applied to a vehicle equipped with a camera device, and the camera device faces a user seat inside the vehicle, and the method includes:

Based on the image collected by the camera device, determine the posture and behavior information of the user located in the user seat, the posture and behavior information includes the user's head posture, face information, line of sight direction, hand position, and behavior information. one or more;

determining state index information of the user based on the gesture behavior information, the state index information being used to indicate that the user is in a state of concentrating on driving or not in a state of driving;

Acquiring prompt content information and presentation parameters corresponding to the state indicator information; the presentation parameters include audio presentation parameters and/or visual presentation parameters;

Based on the prompting content information and the presentation parameters of the prompting degree, generating prompting information of a corresponding degree, and outputting the prompting information of a corresponding degree.

In a fourth aspect, an information broadcasting device is provided, including:

processor;

memory for storing processor-executable program instructions;

Wherein, when the processor invokes the executable instruction, the operation of the method described in the first aspect above is implemented.

In the fifth aspect, there is provided a traffic state prompting device, including:

processor;

memory for storing processor-executable program instructions;

Wherein, when the processor invokes the executable instruction, the operation of the method described in the second aspect above is implemented.

In a sixth aspect, an information broadcasting device is provided, including:

processor;

memory for storing processor-executable program instructions;

Wherein, when the processor invokes the executable instruction, the operation of the method described in the third aspect above is implemented.

In the seventh aspect, a vehicle is provided, comprising

body;

Power components;

a user seat towards the interior of the vehicle, and/or a sensor towards the exterior of the vehicle; and

device; the device includes a processor, a memory for storing processor-executable program instructions, wherein, when the processor invokes the executable instructions, the method according to any one of the above-mentioned first aspect to the third aspect is implemented operate.

In an eighth aspect, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the steps of the method described in any one of the above first to third aspects are implemented.

In the ninth aspect, there is provided a machine-readable storage medium, the machine-readable storage medium stores several computer instructions, and when the computer instructions are executed, the method described in any one of the first to third aspects above is performed. method.

The beneficial effects of an information broadcasting method, a traffic state prompting method, a device, and a vehicle provided by the present application will be described below.

Description of drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings that need to be used in the description of the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application. For those skilled in the art, other drawings can also be obtained based on these drawings without any creative effort.

Fig. 1 is a flowchart of an information broadcasting method according to an embodiment of the present application.

Fig. 2 is a flow chart of an information broadcasting method according to another embodiment of the present application.

Fig. 3 is a flowchart of an information broadcasting method according to another embodiment of the present application.

Fig. 4 is a flowchart of an information broadcasting method according to another embodiment of the present application.

Fig. 5 is a flowchart of a method for prompting traffic status according to an embodiment of the present application.

Fig. 6 is a flow chart of a method for prompting traffic status according to another embodiment of the present application.

Fig. 7 is a flowchart of an information broadcasting method according to another embodiment of the present application.

Fig. 8 is a flowchart of a method for prompting traffic status according to another embodiment of the present application.

Fig. 9 is a flowchart of an information broadcasting method according to another embodiment of the present application.

Fig. 10 is a flowchart of an information broadcasting method according to another embodiment of the present application.

Fig. 11 is a schematic structural diagram of an information broadcasting device according to an embodiment of the present application.

Fig. 12 is a schematic structural diagram of a passing state prompting device according to an embodiment of the present application.

Fig. 13 is a schematic structural diagram of an information broadcasting device according to an embodiment of the present application.

Fig. 14 is a schematic structural diagram of a vehicle according to an embodiment of the present application.

Detailed ways

The following will clearly and completely describe the technical solutions in the embodiments of the application with reference to the drawings in the embodiments of the application. Apparently, the described embodiments are only some of the embodiments of the application, not all of them. Based on the embodiments in this application, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the scope of protection of this application.

With the rapid development of the field of autonomous driving, the intelligence and autonomy of autonomous driving technology are increasing day by day, and its application scenarios are becoming more and more abundant. Based on factors such as the role assignment of the automatic driving system in performing dynamic driving tasks and whether there are design operating conditions, the automatic driving technology can be divided into six levels from manual driving to fully automated driving. There is still a long way to go to realize fully automated driving. In the progressive automatic driving technology, L2 and L3 belong to the human-vehicle co-driving mode. Level 2 autonomous driving is partially automated driving, which allows the driver to detach his hands and feet, but still needs to keep his eyes on the direction of travel and concentrate on the road conditions. The L3 level of automatic driving is conditional automatic driving, which can automatically control the vehicle in most road conditions, and the driver needs to be ready to take over the vehicle at any time.

Most of the existing autonomous driving technologies focus on the environmental information outside the vehicle, which is usually collected by various sensors mounted on the vehicle. Vehicle-mounted sensors include, for example, vision sensors, distance sensors, lidar, millimeter-wave radar, and the like. The vehicle's automatic driving module needs to remind the user to take over the vehicle when it finds that the road conditions are complex and exceed the operation design threshold of the automatic driving level through the collected environmental observation data. In this human-vehicle shared driving mode, both the human and the vehicle's automatic driving modules have control over the vehicle, and it is urgent to effectively ensure the safe driving of the vehicle.

When the inventor of the present application solved the problem of vehicle safety control, he found that the safe driving of the vehicle does not only depend on the environment outside the vehicle, but also depends on whether the user in the vehicle, such as the driver, is in a full-time driving state, or is in fatigue driving or state of distraction. Therefore, the present application hopes to propose a solution by focusing on the state of the driver in the vehicle in the scenario of automatic driving.

At present, the vehicle will be equipped with a Driver Monitoring System (DMS), which has functions such as face recognition, fatigue monitoring, and attention monitoring for drivers, including:

1. Status monitoring: including driver fatigue monitoring, distraction monitoring, eye tracking and other dangerous behavior monitoring, such as making phone calls, eating, chatting, etc.

2. Face recognition: including driver identification, feature recognition and emotion recognition.

3. Other functions: including vehicle accident records and emergency rescue accident confirmation, etc.

DMS can collect the driver's image through the camera device configured in the vehicle, and then use different trained neural networks to perform behavior detection and face detection on the image respectively. Wherein, the above-mentioned neural network, behavior detection and face detection can refer to solutions recorded in related technologies, and the present application will not expand the description here. Behavior detection can monitor whether the driver is making phone calls, smoking and other behaviors. Face detection can include head pose detection, eye tracking, eye state detection, etc. Through head posture detection, it can be judged whether the driver has chatted with the passenger in the passenger seat, turned his head to look behind, looked down at the mobile phone and other behaviors. With eye tracking, it is possible to detect whether the driver's gaze is focused on the front. Through the eye state detection, it can be judged whether the driver has closed eyes or the eyes are slack, so as to judge whether the driver is asleep or in a state of fatigue.

The current DMS can detect the status and behavior of the driver, and remind the driver based on the detected information. However, in the related art, the DMS only focuses on the driver itself, and its function is only to remind the driver of the status, and does not Better vehicle control is achieved, but the safe driving of the vehicle cannot be guaranteed. Especially for scenarios in the co-driving mode, the control of the timing when the driver and the car take over directly affects the safety in the co-driving mode. Drivers are required to concentrate on the road conditions and be ready to take over the vehicle at any time. However, in the road section where the vehicle is driving automatically, the driver will have difficulty maintaining concentration for a long time because there is no need for manual driving, or the driver will be distracted due to the interference of other information, and his attention will not be focused on the road conditions. This application scheme is based on the above-mentioned problem of safe driving of vehicles, and provides an information broadcasting scheme that can be used to ensure safe driving of vehicles through the idea of combining DMS with automatic driving technology. Applying DMS to automatic driving scenarios will improve human Safety in car-sharing mode.

However, in the process of combining DMS with automatic driving technology, there are many problems. For example, in some scenarios, when the vehicle enters the automatic driving mode, the built-in automatic driving module of the vehicle will make decisions based on road condition information. These decisions can include Make turns, change lanes, keep going straight, speed up, slow down, turn on lights, and more. However, the user may not understand the decisions made by the automatic driving module, so that the user cannot have a good grasp of the running state of the vehicle. In the shared driving mode of people and vehicles, if the user cannot grasp the running status of the vehicle well, it will make it impossible for the user to take over the vehicle at the right moment, which will affect the safety of the shared driving mode. For this reason, this application combines DMS to propose a kind of information broadcasting method, generally speaking, the vehicle that is equipped with DMS usually carries camera device, and this camera device faces the user's seat inside the vehicle, such as driver's seat, passenger seat, Back seats etc. The camera device is used to collect images, and the images may include images of the user on the facing user's seat. Through the recognition and analysis of the image, some posture and behavior information of the user can be obtained, such as expression information, head posture, line of sight direction, hand position, behavior information and so on. In this way, a method for broadcasting information provided by the present application can be applied to a vehicle equipped with a DMS, the vehicle is equipped with a camera device, and the camera device faces the user's seat inside the vehicle, and the method includes the steps shown in Figure 1 :

Step 110: Determine the facial expression information of the user located in the user seat based on the image information collected by the camera device;

Step 120: Obtain specific state information of the vehicle;

Wherein, the specific state information includes one or more of the following: alarm information output by the vehicle, or the vehicle receives a user instruction for instructing the execution module of the vehicle to change the working state and does not execute the When the user commands, the actual working status information of the execution module;

Step 130: If the expression feature in the expression information matches the preset expression feature representing doubt, obtain analysis information based on the specific state information, and broadcast the analysis information.

Wherein, the above step 110 and step 120 are not executed sequentially, and step 110 and step 120 may also be executed at the same time.

For example, the camera device can face the driver's seat, and the user in the driver's seat is the driver. Of course, the camera device can also be directed towards the passenger seat, the rear seat, etc., so the user in the seat is the passenger. The following takes the user as a driver as an example to expand the description.

The vehicle-specific state information may be vehicle-specific state information related to decisions made by the autonomous driving module. For example, if the automatic driving module makes a decision to output warning information, then the specific state information of the vehicle may include the warning information output by the vehicle. Another example is that the vehicle receives a user instruction, the user instruction is used to instruct the execution module of the vehicle to change the working state, but the automatic driving module makes a decision not to execute the user instruction, then the specific state information of the vehicle may include the actual working state information of the execution module .

In addition, how to determine whether the user has produced a puzzled expression can be implemented in multiple ways in practical applications. As an example, the image information collected by the camera device can be input into the trained neural network, and the neural network can extract the user's expression features in the image information, and match the extracted expression features with the preset expression features representing doubts , according to the output result of the neural network, it can be determined that the user has produced a puzzled expression. Among them, the expression feature can be extracted in many ways. As an example, the expression feature can be obtained according to the position of the facial feature points and the distance between the feature points. For example, the expression feature may include the distance between the feature points representing the left and right eyebrows, and may also include the position of the feature points representing the corners of the mouth. Those skilled in the art can select different expression features to represent puzzled expressions according to actual needs, which is not limited in this application.

The above embodiments combine the application of DMS in the scene of automatic driving, which can monitor the running state of the vehicle and the state of the user. When the user has a puzzled expression, it can output analysis information based on the user's doubts in a timely manner, so that the user can grasp the running status of the vehicle and prevent the user from making wrong decisions. Users can make correct decisions in a timely manner based on the broadcast analysis information, control the safe driving of vehicles, and improve the safety of people and vehicles in the shared driving mode.

Vehicle-specific status information may include alert messages output by the vehicle. The analysis information may be the analysis information of the alarm information, which is used to analyze the cause of the alarm information. In some embodiments, the alarm information output by the vehicle may include an icon alarm displayed on the instrument panel, and the icon alarm may include presentation modes such as icon flashing, constant light, and color change. The alarm information output by the vehicle may also include alarm sounds emitted by vehicle components, such as the alarm sound of "Didi Di". The user may not know clearly the meanings represented by the icons on the instrument panel and the alarm sound or the cause of the alarm information, and it is inconvenient for the user to refer to the manual to understand the above alarm information during the driving of the vehicle. Then the analysis information may be the meaning represented by the alarming icon on the instrument panel, or the meaning represented by the icon in the presentation mode, or the meaning represented by the alarm sound. For example, the parsed information could be "the seat belt icon is on, the passenger is not wearing the seat belt correctly". For another example, the analysis information may be "the alarm sound means that the trunk door is not closed properly". In this way, when the user shows a puzzled expression, the analysis information of the alarm information is acquired and played, which prevents the user from staring at the icons on the instrument panel for a long time while the vehicle is running, and allows the user to know the alarm information in time. At the same time, the user can know the alarm information in time by analyzing the information, and make corresponding processing for the alarm information. For example, in the above example, if the alarm sound from the vehicle indicates that the trunk door is not closed, the user can stop the vehicle in time according to the alarm sound, close the trunk door and continue driving the vehicle. Users can always grasp the running status of the vehicle, make decisions at the right moment, and improve the safety of people and vehicles in the shared driving mode.

The specific state information of the vehicle may also include the actual working state information of the execution module when the vehicle receives a user instruction for instructing the execution module of the vehicle to change the working state and the user instruction is not executed. The analysis information may be the analysis information of the actual working status information of the execution module, and is used to analyze the cause of the actual working status of the execution module. In some embodiments, the actual working state information of the executive module is determined based on the instruction input to the executive module by the vehicle's automatic driving module, and the user instruction is different from the instruction input to the executive module by the automatic driving module. For example, it may include that the instruction input by the user to the power assembly of the vehicle is different from the instruction input by the automatic driving module to the power assembly; it may also include that the driving mode set by the user for the vehicle is different from the driving mode set by the automatic driving module for the vehicle. In this way, there is a conflict between the user and the automatic driving module's instructions to a certain execution module. In this case, the priority of the automatic driving module's instruction is higher than that of the user's instruction, so the user's instruction is not executed, and the vehicle The working status of the execution module has not changed as expected by the user.

For example, in some scenarios, when the user wants the vehicle to travel from the current lane to an adjacent lane, the user's input to the power assembly of the vehicle may include a lane change instruction. However, through the sensors mounted on the vehicle facing the outside of the vehicle, including vision sensors, distance sensors, lidar, millimeter-wave radar, etc., based on the environmental observation data collected by these sensors, it is detected that the distance between the vehicle behind and the vehicle on the next lane is insufficient to allow the vehicle to change lanes. If the vehicle executes the lane-changing command input by the user, it is likely to collide with a vehicle in the next lane. Therefore, based on the above detection results, the automatic driving module of the vehicle makes a decision not to execute the user instruction, and the instructions input by the automatic driving module to the power components include the instruction to keep going straight.

Because the vehicle did not change lanes as the user expected, the user may become confused, which manifests itself as a puzzled expression on the face. When the DMS detects that the user has a puzzled expression, it can broadcast the analysis information of the actual working status information of the execution module, such as the analysis information that the vehicle has not changed lanes, such as "there is a risk of collision when changing lanes" to analyze the actual work of the execution module The cause of the state.

For another example, in some other scenarios, when the user wants the vehicle to enter the automatic driving mode, the driving mode of the vehicle is set to the automatic driving mode. However, through the environmental observation data collected by the vehicle-mounted sensors facing the outside of the vehicle, it is detected that the current road conditions are complex, and it is not suitable for driving in automatic driving mode. Thus, based on the above detection results, the automatic driving module makes a decision not to execute the user instruction, and the automatic driving module sets the driving mode of the vehicle to the manual driving mode.

Because the vehicle does not enter the autonomous driving mode as the user expected, the user may be confused, which can be expressed as a puzzled expression on the facial expression. When the DMS detects that the user has a puzzled expression, it can broadcast the analysis information of the actual working status information of the execution module, such as the analysis information that the vehicle has not entered the automatic driving mode, such as "the road conditions are complicated, please drive in manual driving mode", to Analyze the cause of the actual working state of the execution module.

The above-mentioned embodiment records that when the instrument panel of the vehicle displays an icon alarm, a vehicle component emits an alarm sound, or when the vehicle does not execute the user's instruction, and when the user makes a puzzled expression, the analysis information of the alarm is broadcast or the alarm is not executed. Parsing information for user commands. Therefore, on the one hand, the broadcast of the analysis information can eliminate the user's doubts and improve the user experience; on the other hand, the analysis information is only broadcast when the user shows doubts, so the frequent broadcast will not cause interference to the user.

In some embodiments, the vehicle may include multiple execution modules, and each execution module has its own corresponding actual working state. In this way, when the user has a puzzled expression, it is necessary to determine from the multiple execution modules which actual working state of the execution module makes the user doubt. Then the step of obtaining the analysis information in step 130 above may include steps as shown in FIG. 2:

Step 210: Determine a target execution module among the plurality of execution modules according to the expression information;

Step 220: Obtain analysis information according to the actual working state information of the target execution module.

In some embodiments, the user's gaze direction may be determined based on the expression information, and then the target execution module may be determined from multiple execution modules based on the gaze direction. As an example, if the user inputs a lane change instruction and an instruction to lower the temperature of the air conditioner in the car. However, the automatic driving module did not execute the lane change command based on the fact that the distance between the vehicle behind and the vehicle in the next lane is not enough to change lanes. At the same time, due to the lack of fuel and electricity in the vehicle, if the air conditioner in the car is turned down, the fuel or electricity will not be enough to support the vehicle to reach the destination or the nearest gas station or charging pile. Therefore, the command to lower the temperature of the air conditioner in the vehicle was not carried out. Both of the above two situations may cause confusion to the user. In this case, it is possible to determine the target execution module that causes confusion to the user through the direction of the user's line of sight. For example, if the user's line of sight is looking at the adjacent lane or the rearview mirror, it means that the user may have doubts about the vehicle not changing lanes, and thus broadcast the analysis information that the vehicle did not execute the lane changing instruction. For another example, if the user's line of sight is looking at the air outlet of the air conditioner, it means that the user may have doubts that the temperature of the air conditioner in the car has not been lowered, so the analysis information that the instruction to lower the temperature of the air conditioner in the car has not been executed is broadcast.

In some embodiments, in addition to determining the target execution module from multiple execution modules according to the expression information, when the vehicle outputs multiple warning messages, it is also possible to determine the target warning message from the multiple warning messages according to the user's expression information. Information, obtain the analysis information of the alarm information and broadcast it.

In some embodiments, the gaze direction of the user may be determined based on the expression information, and then target alert information is determined from multiple alert information based on the gaze direction. As an example, when a vehicle's dashboard displays an icon alert while a vehicle component sounds an alert, both alert messages may confuse the user. Therefore, when it is detected that the user has a doubtful expression, the target alarm information that makes the user doubtful can be determined through the user's gaze direction. For example, when the user's line of sight falls on the instrument panel, it means that the user may be confused about the icon alarm displayed on the instrument panel, thereby determining that the target alarm information is the icon alarm. For another example, when the user's line of sight falls on the speaker of the vehicle or other components capable of emitting an alarm sound, it indicates that the user may be confused about the alarm sound emitted by the vehicle component, thereby determining the alarm sound as the target alarm information. After the target alarm information is determined, the analysis information of the alarm information can be acquired and broadcast.

In addition, in some embodiments, the above-mentioned step 130 for obtaining the analysis information may include steps as shown in Figure 3:

Step 310: Determine the user's line of sight direction based on the expression information;

Step 320: Determine a target execution module in the internal environment of the vehicle based on the line-of-sight direction;

Step 330: Obtain analysis information according to the actual working status information of the target execution module.

Based on the image information collected by the camera mounted on the vehicle, the user's expression information can be obtained. According to the three-dimensional position and posture relationship between the camera device and the user, the three-dimensional vector information of the user's line of sight, that is, the direction of the user's line of sight, can be calculated. Based on the user's gaze direction, the target execution module can be determined in the interior environment of the vehicle. For example, if the user's line of sight intersects the instrument panel inside the vehicle, it is determined that the user is looking at the area on the instrument panel. In this way, analysis information can be obtained and broadcasted according to the actual working status information of the dashboard. If there is an icon alarm on the dashboard at this moment, the analysis information of the icon alarm can be broadcast.

Furthermore, if there are multiple icon alerts on the dashboard, the target icon can be further determined from the multiple icons based on the gaze direction, that is, to determine which icon the user's gaze falls on, and then obtain and broadcast the analysis of the target icon information.

In some embodiments, broadcasting the analysis information may include: controlling a playback device in the vehicle to broadcast the analysis information. The playback device can be, for example, a loudspeaker in the vehicle.

In some embodiments, the analysis information of the specific state information may be pre-stored in the memory of the vehicle, or the analysis information of the specific state information may be obtained from a connected server in real time through the Internet.

The above embodiments describe that according to the expression information, the target execution module can be determined from a plurality of row modules, or the target alarm information can be determined from a plurality of alarm information. As an example, the user's gaze direction may be determined based on the expression information, and then the target execution module may be determined from multiple line modules based on the gaze direction, or the target alarm information may be determined from multiple alarm information. In addition, according to the expression information, the target execution module can be determined in the internal environment of the vehicle. Through the method provided by the above embodiments, the analysis information of the target alarm information or the working status information of the target execution module can be broadcast to the user in a targeted and more accurate manner, and useful analysis can be provided for the user more accurately, and the user experience can be improved.

In some scenarios, the DMS equipped with the vehicle can also monitor the status of the user, such as the driver, including user fatigue monitoring, distraction monitoring, eye tracking and other dangerous behavior monitoring, such as calling, eating, chatting, etc. In some embodiments, in addition to combining with DMS to broadcast analysis information when the user has a doubtful expression, the DMS can also be used to detect the user's status and make different levels of reminders according to the user's status. In this way, based on any of the above-mentioned embodiments, an information broadcasting method proposed by this application may further include the steps shown in Figure 4:

Step 410: Determine the gesture and behavior information of the user based on the image collected by the camera device, the gesture and behavior information includes one of the user's head posture, face information, line of sight direction, hand position, and behavior information. item or items;

Step 420: Determine state index information of the user based on the gesture behavior information, the state index information is used to indicate that the user is in a state of concentrating on driving or not in a state of driving;

Step 430: Obtain the prompt content information and the presentation parameters of the prompt level corresponding to the state indicator information; the presentation parameters include audio presentation parameters and/or visual presentation parameters;

Step 440: Based on the prompt content information and the presentation parameters of the prompt level, generate prompt information of a corresponding degree, and output the prompt information of a corresponding degree.

Based on the vehicle-mounted inward-facing camera device, user images can be collected, and user posture and behavior information can be extracted from the user images, including the user's head posture, face information, line of sight direction, hand position, behavior information, etc. Based on the user's head posture, it can be judged whether the user has chatted with the passenger in the passenger seat, turned his head to look behind, or looked down at the mobile phone. Based on the user's face information, the user's eye state can be obtained, so as to determine whether the user has closed eyes or distracted eyes, and thus infers whether the driver is asleep or in a state of fatigue. Based on the user's gaze direction, it may be determined whether the user's gaze is focused on the front. Based on the position of the user's hand, it may be determined whether the user's hand keeps holding the steering wheel. Based on the user's behavior information, it can be judged whether the user is making a phone call, eating, chatting, smoking, etc.

Based on the gesture behavior information, the user's state index information can be determined. For example, according to the user's gesture behavior information, the user's state can be scored, and the user's state index information can be a score value representing the user's state. According to the score value of the user's state, it can be determined that the user is in the state of driving attentively or not. Wherein, the inattentive driving state may include a fatigue state and a distraction state. In addition, if the gesture behavior information includes more than one type of information, such as the direction of sight, hand position, etc., then the score based on the direction of sight can be added to the score based on the hand position, or based on different gesture behavior information The obtained scores are weighted and summed to obtain the user's status indicator information. Of course, in addition to using the score value of the user's state to represent the user's state index information, those skilled in the art can also express the user's state index information in other ways. The application is not limited here.

Different state indicator information corresponds to different presentation parameters of prompting degrees. Take the status indicator information as the score value of the status of the user as an example. Several score areas can be divided, and each score area corresponds to a presentation parameter of a prompting degree. In this way, the prompt content information and the presentation parameters corresponding to the score area where the score value is located can be acquired, and based on the prompt content information and the presentation parameters, corresponding prompt information is generated.

In some embodiments, the presentation parameters include audio presentation parameters, such as pitch, volume, speech rate, playback frequency, and the like. Different score regions can correspond to audio presentation parameters with different prompting levels, and the audio presentation parameters with different prompting levels can be reflected in different pitches, different volumes, different speech speeds, and different playback frequency intervals. The presentation parameters may also include visual presentation parameters, such as brightness, animation type, area occupied by the broadcast content on the interface, and the like. Different score areas can correspond to video presentation parameters with different prompting levels, which can be reflected in different brightness, different animation types, and different sizes of areas occupied by the broadcast content on the interface.

There are many ways to output the prompt information. As an example, the prompt information can be played through a loudspeaker inside the vehicle. Wherein, the prompt information may be played with different frequencies and volumes according to the status indicator information of the user. For example, if the user is distracted, a reminder message can be played every minute at a normal volume. If the user is in a state of fatigue, the prompt information can be played every 30 seconds according to a higher volume.

As an example, the prompt content can also be displayed through the human-computer interaction interface inside the vehicle. Wherein, according to the status indicator information of the user, different colors and font sizes may be used to display the prompt content.

In some embodiments, the visual presentation parameters may include animation types, which may include simulated expressions representing different degrees of severity. In this way, a corresponding degree of simulated expression can be displayed on the human-computer interaction interface of the vehicle, such as a display screen inside the vehicle. Wherein, the simulated expressions representing different degrees of severity may include: one or more of simulated expressions representing normal emotions, simulated expressions representing nervous emotions, and simulated expressions representing painful emotions.

In some embodiments, the highest level of prompt information may also include tightening the seat belt at the user's location or turning on the double flashing lights of the vehicle.

As an example, according to the user's status index information, the vehicle can output different levels of prompts, including first-level prompts, second-level prompts, and third-level prompts. If the user makes a phone call, drinks water, smokes, etc. during driving, the vehicle can output a first-level reminder, such as displaying a simulated expression representing normal emotions on the display screen inside the vehicle, and displaying in a black-and-white prompt box such as " Please concentrate on driving" prompt information, at the same time, it can also broadcast the prompt information with a gentle tone and low frequency. If the user chats with others while driving, the vehicle can output a secondary reminder, such as a simulated expression representing nervousness on the display screen inside the vehicle, and a message such as "Please concentrate on driving" in the orange prompt box To prompt information, at the same time, the prompt information can also be broadcast in a tense tone and at a low frequency. If the user is not in the driving state during driving, including behaviors such as lowering the head to pick up the mobile phone, closing eyes, etc., the vehicle can output a three-level prompt. Prompt information such as "Please concentrate on driving" is displayed in bold font, and at the same time, prompt information can be broadcast at a high frequency in a serious tone. At the same time, tighten the seat belt at the user's location, or turn on the double flashing lights of the vehicle.

The above-mentioned embodiment records that in the automatic driving scene, in addition to combining with DMS to broadcast analysis information when the user has a doubtful expression, the DMS can also be used to detect the user's state and make different levels of reminders according to the user's state. For progressive automatic driving scenarios, reminding users to concentrate on driving at all times can effectively control the timing of switching between people and vehicles in automatic driving, thereby improving safety in the shared driving mode.

In some scenarios, when the road condition is congested or waiting to pass at an intersection, the user, such as a driver, may not be able to concentrate. For example, you might be looking down at your phone or chatting with a passenger while waiting for a traffic light to go. It is very likely that the user will not realize that the road has passed due to inattention, and will not drive the vehicle in time. This will cause trouble for the rear vehicles, and even affect the smooth flow of urban traffic. For this reason, this application proposes a method for prompting traffic status in combination with DMS. Generally speaking, a vehicle equipped with a DMS is usually equipped with a camera device, and the camera device faces user seats inside the vehicle, such as a driver's seat, a passenger seat, a rear seat, and the like. The camera device is used to collect images, and the images may include images of the user on the facing user's seat. Through the recognition and analysis of the image, some posture and behavior information of the user can be obtained, such as expression information, head posture, line of sight direction, hand position, behavior information and so on. In addition, self-driving cars are generally equipped with sensors facing the outside of the vehicle, and the vehicle's self-driving module makes relevant decisions based on the environmental observation data collected by the sensors. Sensors can include vision sensors, distance sensors, lidar, millimeter-wave radar, and more. In this way, a method for prompting traffic status provided in this application can be applied to a vehicle equipped with a DMS, and the vehicle is equipped with a sensor facing the outside of the vehicle and a camera device facing the user's seat inside the vehicle. Described method comprises the steps as shown in Figure 5:

Step 510: Based on the road information collected by the sensor, determine that the road where the vehicle is located changes from a non-traffic state to a traffic state;

Step 520: Obtain the expected motion state of the vehicle corresponding to the passing state;

Step 530: Obtain the current motion state of the vehicle;

Step 540: Based on the images collected by the camera device, determine the characteristics of the driving state of the user in the user seat;

Step 550: If the current motion state of the vehicle does not reach the expected motion state, and the feature of the user's driving state matches the state feature for indicating inattentive driving, output a The prompt information of the passing state.

The road information collected by the sensor may include traffic light information. For example, the sensor could be a color camera that detects the color displayed by the traffic light by taking an image of the traffic light in front of the car. When the traffic light information indicates that vehicles are allowed to pass, if the green light is on, the road is in a state of passing. In this way, based on the traffic light information, it can be determined that the road on which the vehicle is located changes from a non-traffic state to a traffic state.

The road information collected by the sensor may also include information on surrounding objects. The information of the surrounding objects may include the distance between the surrounding objects and the vehicle, and when the distance between the surrounding objects and the vehicle is greater than a distance threshold, the road is in a passing state. For example, surrounding objects may be vehicles ahead. When the vehicle in front drives away, the distance between it and the vehicle will gradually increase. When the distance increases to a certain extent, the car can start to move forward. Therefore, when the distance between the vehicle in front and the vehicle is greater than the distance threshold, the road is in a passing state. In this way, based on the distance between the surrounding objects and the vehicle, it can be determined that the road on which the vehicle is located changes from a non-traffic state to a traffic state.

The information on surrounding objects may also include whether there are obstacles within the preset range of the vehicle. For example, some parts of the road have zebra crossings but no traffic lights. When pedestrians cross the road, vehicles need to give way to pedestrians and stop behind the zebra crossing. The vehicle can detect whether there is an obstacle within the preset range through the sensor. If it does not exist, it means that the pedestrian has crossed the road, the road is in the state of traffic, and the vehicle can move forward. In this way, based on whether there is an obstacle within the preset range of the vehicle, it can be determined that the road on which the vehicle is located changes from a non-traffic state to a traffic state.

After determining that the road where the vehicle is located changes from a non-traffic state to a traffic state, the expected motion state of the vehicle corresponding to the traffic state can be obtained, for example, it can be a driving state. and the current state of motion of the vehicle.

Based on the images captured by the camera device, characteristics of the user's driving state can be determined. In some embodiments, the gesture behavior information of the user can be determined based on the images collected by the camera device, and the gesture behavior information can include one or more of the user's head posture, face information, gaze direction, hand position, and behavior information. multiple. Based on the user's head posture, it can be judged whether the user has chatted with the passenger in the passenger seat, turned his head to look behind, or looked down at the mobile phone. Based on the user's face information, the user's eye state can be obtained, so as to determine whether the user has closed eyes or distracted eyes, and thus infers whether the driver is asleep or in a state of fatigue. Based on the user's gaze direction, it may be determined whether the user's gaze is focused on the front. Based on the position of the user's hand, it may be determined whether the user's hand keeps holding the steering wheel. Based on the user's behavior information, it can be judged whether the user is making a phone call, eating, chatting, smoking, etc. Based on the gesture behavior information, characteristics of the user's driving state may be determined.

If the current motion state of the vehicle does not reach the expected motion state, and the characteristics of the user's driving state match the state characteristics for indicating inattentive driving, output prompt information for indicating that the road is in a passing state. For example, in the above example, when the road changes from a non-traffic state to a traffic state, the vehicle is in a parked state, the expected driving state is not reached, and the user is in a state of inattentive driving, then a prompt message indicating that the road is in a traffic state is output.

The above-mentioned embodiment records that when the road changes from non-traffic to traffic, including the situation that the traffic light changes to green, or the vehicle in front has driven away, if the vehicle does not drive in time and the user is not driving attentively, a prompt message is output to Prompt the user that the road is open. In order to prevent the user from not noticing that the road is already passable due to inattention, causing the vehicle to fail to drive in time and causing road congestion.

In some embodiments, if the current motion state of the vehicle has not reached the expected motion state within a preset period of time after outputting the prompt information, a warning message is sent to surrounding objects. Or when the characteristics of the user's driving state still match the characteristics of the inattentive driving state, that is, the user is still in the inattentive driving state, then a warning message is sent to the surrounding objects. It can also be within a preset period of time after outputting the prompt information, if the current motion state of the vehicle has not yet reached the expected motion state, and the user is still in a state of not concentrating on driving, then a warning message is sent to the surrounding objects to prompt the surrounding objects such as A special situation has occurred between other vehicles and the pedestrian's own vehicle.

In some embodiments, the way of issuing the warning information includes turning on the double flashing lights of the vehicle to warn surrounding objects, such as surrounding vehicles.

In some embodiments, different levels of reminders can also be made according to the driving state of the user. In this way, the above step 550 outputs prompt information, including the steps shown in FIG. 6:

Step 610: Obtain the prompt content information and the presentation parameters of the prompt level corresponding to the driving state index information; the presentation parameters include audio presentation parameters and/or visual presentation parameters;

Step 620: Based on the prompt content information and the presentation parameters of the prompt level, generate prompt information of a corresponding degree, and output the prompt information of a corresponding degree.

The index information of the driving state may be a score value of the state the user is in. The score value can be determined according to the user's posture and behavior information, including the user's head posture, face information, gaze direction, hand position, behavior information, and so on. If the gesture behavior information includes more than one type of information, such as the direction of sight, hand position, etc., then the score based on the direction of sight can be added to the score based on the hand position, or the score based on different gesture behavior information can be added. The scores are weighted and summed to obtain the user's status indicator information. Of course, in addition to representing the user's state index information by the score value of the user's state, those skilled in the art can also express the user's state index information in other ways. The application is not limited here.

Different state indicator information corresponds to different presentation parameters of prompting levels. Take the status indicator information as the score value of the status of the user as an example. Several score areas can be divided, and each score area corresponds to a presentation parameter of a prompting degree. In this way, the prompt content information and the presentation parameters corresponding to the score area where the score value is located can be acquired, and based on the prompt content information and the presentation parameters, corresponding prompt information is generated.

There are many ways to output the prompt information. As an example, the prompt information can be played through a loudspeaker inside the vehicle. Wherein, the prompt information can be played with different frequencies and volumes according to the status index information of the user. For example, if the user is distracted, a reminder message can be played every minute at a normal volume. If the user is in a state of fatigue, the prompt information can be played every 30 seconds according to a higher volume.

As an example, according to the user's state index information, the vehicle can output different levels of prompts, including first-level prompts, second-level prompts, and third-level prompts. If the user makes a phone call, drinks water, smokes, etc. during driving, the vehicle can output a first-level reminder, such as displaying a simulated expression representing normal emotions on the display screen inside the vehicle, and displaying in a black-and-white prompt box such as " Please concentrate on driving" prompt information, at the same time, it can also broadcast the prompt information with a gentle tone and low frequency. If the user chats with others while driving, the vehicle can output a secondary reminder, such as a simulated expression representing nervousness on the display screen inside the vehicle, and a message such as "Please concentrate on driving" in the orange prompt box To prompt information, at the same time, the prompt information can also be broadcast in a tense tone and at a low frequency. If the user is not in the driving state during driving, including behaviors such as lowering the head to pick up the mobile phone, closing eyes, etc., the vehicle can output a three-level prompt. Prompt information such as "Please concentrate on driving" is displayed in bold font, and at the same time, prompt information can be broadcast at a high frequency in a serious tone. At the same time, tighten the seat belt at the user's location, or turn on the double flashing lights of the vehicle.

The above-mentioned embodiment records that when the road changes from non-traffic to traffic, if the vehicle does not drive away in time and the user is not driving attentively, different levels of reminders will be given according to the state of the user. For the scene of progressive automatic driving, reminding the user to concentrate on driving at all times can effectively control the timing of switching between people and vehicles in automatic driving, thereby improving the safety of the human-vehicle shared driving mode. At the same time, it can also prevent users from not noticing that the road is already passable due to inattention, causing the vehicle to fail to drive away in time and causing road congestion.

In some scenarios, in the human-vehicle shared driving mode, both the human and the vehicle's automatic driving modules have control authority over the vehicle. The control of the time when the driver and the car take over directly affects the safety in the shared driving mode. Drivers are required to concentrate on the road conditions and be ready to take over the vehicle at any time. However, in the road section where the vehicle is driving automatically, the driver will have difficulty maintaining concentration for a long time because there is no need for manual driving. The driver may enter a state of fatigue, or lower his head to play with mobile phones, chat with passengers and other distracting behaviors. For this reason, in order to avoid safety accidents caused by the user failing to take over the vehicle at the correct time, the present application also provides an information broadcasting method, which is applied to a vehicle equipped with an imaging device, wherein the imaging device faces the user's seat inside the vehicle, and the The method includes the steps shown in Figure 7:

Step 710: Determine the gesture and behavior information of the user based on the images collected by the camera device, the gesture and behavior information includes one of the user's head posture, face information, line of sight direction, hand position, and behavior information. item or items;

Step 720: Determine state index information of the user based on the gesture behavior information, the state index information is used to indicate that the user is in a state of concentrating on driving or not in a state of driving;

Step 730: Obtain the prompt content information and the presentation parameters of the prompt level corresponding to the state indicator information; the presentation parameters include audio presentation parameters and/or visual presentation parameters;

Step 740: Based on the prompt content information and the presentation parameters of the prompt level, generate prompt information of a corresponding degree, and output the prompt information of a corresponding degree.

Based on the gesture behavior information, the user's state index information can be determined. For example, according to the user's gesture and behavior information, the user's state can be scored, and the user's state index information can be a score representing the user's state. According to the score value of the user's state, it can be determined that the user is in the state of driving attentively or not. Wherein, the inattentive driving state may include a fatigue state and a distraction state. In addition, if the gesture behavior information includes more than one type of information, such as the direction of sight, hand position, etc., then the score based on the direction of sight can be added to the score based on the hand position, or based on different gesture behavior information The obtained scores are weighted and summed to obtain the user's status indicator information. Of course, in addition to representing the user's state index information by the score value of the user's state, those skilled in the art can also express the user's state index information in other ways. The application is not limited here.

The above embodiments describe that in the automatic driving scene, the DMS is used to detect the user's state, and different levels of reminders are made according to the user's state. For progressive automatic driving scenarios, reminding users to concentrate on driving at all times can effectively control the timing of human-vehicle switching during automatic driving, avoiding safety accidents caused by users failing to take over the vehicle at the right time, thereby improving the safety of the human-vehicle shared driving mode.

In addition, the present application also provides a method for prompting traffic status. As shown in FIG. 8 , the method is applied to a vehicle equipped with a 4K color camera facing outside the vehicle to capture images in front of the vehicle. The 4K color camera can continuously output color images, which can be input into Convolutional Neural Networks (CNN) to detect traffic lights in the image, and the color or status of traffic lights can be detected by CNN. In addition, the 4K color camera is also used to detect the surrounding environment, including detecting the surrounding vehicles, estimating the depth information of the surrounding environment, calculating the three-dimensional position of the surrounding objects, and then calculating the motion state of the surrounding vehicles.

In addition to the 4K camera, the vehicle is also equipped with a DMS grayscale camera facing the interior of the vehicle to capture images of the driver. The DMS grayscale camera can continuously output grayscale images that can be fed into another CNN to detect faces in the image, the pose of the head, track gaze, detect behavior, etc. Combining this information can further determine whether the driver is fatigued, inattentive, and whether he is holding the steering wheel.

When it is detected that the color of the traffic light changes from red light to green light, or from a signal indicating no traffic to a signal indicating traffic, and the vehicle in front has gone far away, and the vehicle is in a manual driving state, and the driver is not paying attention , including drinking water, chatting, playing with mobile phones, etc., the vehicle will output prompt information to remind the driver to drive the vehicle as soon as possible.

Among them, the prompt information can be output in a variety of ways, such as audio broadcast, display on the human-computer interaction interface, and icon alarm on the instrument panel. The prompt information can be divided into multi-level prompt information. For example, it can be divided into a first-level prompt, a second-level prompt and a third-level prompt. If it is detected that the driver is talking on the phone, drinking water, smoking, etc., the vehicle can output a first-level reminder, such as displaying a simulated expression representing normal emotions on the display screen inside the vehicle, and displaying the following in a black-and-white prompt box: The reminder message "Please concentrate on driving", at the same time, the reminder information can also be broadcast at a low frequency in a gentle tone. If it is detected that the driver is chatting with others, the vehicle can output a secondary prompt, for example, a simulated expression representing nervousness can be displayed on the display screen inside the vehicle, and a prompt such as "Please concentrate on driving" can be displayed in the orange prompt box Information, at the same time, can also broadcast prompt information in a tense tone and in low frequency. If it is detected that the driver is not driving, including bowing his head to pick up his mobile phone, closing his eyes, etc., the vehicle can output a three-level reminder. Prompt information such as "Please concentrate on driving" is displayed in bold font, and at the same time, prompt information can be broadcast at a high frequency in a serious tone. At the same time, tighten the seat belt at the user's location, or turn on the double flashing lights of the vehicle.

The above prompt information can be output as an icon alarm on the instrument panel. In addition, when the vehicle is in an abnormal state, an icon alarm can also be issued on the instrument panel. However, during the driving process of the vehicle, the driver cannot understand the analysis of the alarm by consulting the manual. If the driver looks at the central control screen while driving, it will bring driving risks. In this way, as shown in FIG. 9 , according to the status information of the vehicle, the icon on the instrument panel will light up to perform an icon alarm. The inward-facing DMS grayscale camera mounted on the vehicle can continuously output grayscale images, and CNN can be used to detect faces and head postures in these images, track line of sight, and conduct behavior detection. If it is detected that the driver does not express doubts, continue image detection; if it is detected that the driver has noticed the icon alarm, and the driver expresses doubts after seeing the icon alarm, and the vehicle is driving at a fast speed, the driver and DMS gray The three-dimensional position and posture relationship between the cameras, calculate the three-dimensional vector information of the driver's sight, locate the icon area the driver is looking at, and then interpret the analysis information of the icon alarm that the driver is looking at through voice broadcast.

In addition, the DMS can also be used to detect the dangerous behavior of the driver during the driving of the vehicle. As shown in Figure 10, the 4K camera mounted on the vehicle facing outside the vehicle can detect the surrounding environment, including detecting surrounding vehicles and estimating the depth of the surrounding environment. Information, calculate the three-dimensional position of the surrounding objects, and then calculate the movement state of the surrounding vehicles and itself. The DMS grayscale camera mounted on the vehicle facing the interior of the vehicle can continuously output grayscale images, and CNN can be used to detect faces and head postures in these images, track sight, behavior detection, etc. If the driver is detected to be driving dangerously, including sleepy eyes closing, turning his head to look behind, lowering his head to play with mobile phones and other dangerous behaviors, and the vehicle is driving fast, a prompt message will be output to remind the driver to drive the vehicle safely. If the prompt information is invalid, for example, the driver is still driving dangerously after the prompt information output lasts for 3 seconds, then further upgrade the prompt level, and actively turn on the double flashing lights of the vehicle to alert the driver with a higher level of prompt information, such as tightening the Seat belts, increase the volume of announcements of prompt information, etc. As mentioned above, the output prompt information can be divided into multi-level prompt information. The specific multi-level prompt information can refer to the above-mentioned embodiments, and the present application will not repeat them here.

The above-mentioned embodiment shows that in the automatic driving scene, the combination of DMS can remind the driver that the road on which the vehicle is located changes from a non-traffic state to a traffic state, thereby allowing the user to drive the vehicle in time to avoid road congestion. At the same time, when the user has doubts about the alarm information inside the vehicle, it can broadcast and analyze the information for the user in time to improve the safety of automatic driving. And in the process of automatic driving, DMS is used to detect the user's driving status at all times, and different levels of prompt information are used to warn the user, so that the user can concentrate on driving and be ready for switching between people and vehicles at all times, effectively controlling the timing of switching between people and vehicles in automatic driving, thereby improving human The safety of car sharing mode.

Based on the information broadcasting method described in any of the above embodiments, the present application also provides a schematic structural diagram of an information broadcasting device as shown in FIG. 11 . As shown in Figure 11, at the hardware level, the information broadcasting device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory, and of course may also include hardware required by other services. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it, so as to realize the information broadcasting method described in any of the above embodiments.

Based on the method for prompting the passing state described in any of the above embodiments, the present application also provides a schematic structural diagram of a passing state prompting device as shown in FIG. 12 . As shown in Figure 12, at the hardware level, the traffic status prompting device includes a processor, an internal bus, a network interface, a memory and a non-volatile memory, and of course may also include hardware required by other services. The processor reads the corresponding computer program from the non-volatile memory into the memory, and then runs it, so as to realize a method for prompting the passing status described in any of the above embodiments.

Based on the information broadcasting method described in any of the above embodiments, the present application also provides a schematic structural diagram of an information broadcasting device as shown in FIG. 13 . As shown in Figure 13, at the hardware level, the information broadcasting device includes a processor, an internal bus, a network interface, a memory and a non-volatile memory, and of course may also include hardware required by other services. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it, so as to realize the information broadcasting method described in any of the above embodiments.

Based on the information broadcasting method, the traffic state reminder and the information broadcasting method described in any of the above embodiments, the present application also provides a schematic structural diagram of a vehicle as shown in FIG. 14 . As shown in Figure 14, at the hardware level, the vehicle includes a vehicle body, a power assembly, a user seat facing the interior of the vehicle, and/or a sensor facing the exterior of the vehicle, and a device, the device includes a processor, an internal bus, a network Interfaces, memory, and non-volatile storage, and of course, possibly other hardware required by the business. The processor reads the corresponding computer program from the non-volatile memory into the memory and runs it, so as to realize the information broadcasting method, the traffic status prompt or the information broadcasting method described in any of the above embodiments.

Based on an information broadcasting method, a traffic status reminder, and an information broadcasting method described in any of the above embodiments, the present application also provides a computer program product, including a computer program, which can be used when the computer program is executed by a processor. Executing an information broadcasting method, a traffic state reminder and an information broadcasting method described in any of the above embodiments.

Based on an information broadcasting method, a traffic status prompt, and an information broadcasting method described in any of the above embodiments, the present application also provides a computer storage medium, the storage medium stores a computer program, and the computer program is executed by a processor It can be used to implement an information broadcasting method, a traffic status reminder and an information broadcasting method described in any of the above embodiments.

As for the device embodiment, since it basically corresponds to the method embodiment, for related parts, please refer to the part description of the method embodiment. The device embodiments described above are only illustrative, and the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in One place, or it can be distributed to multiple network elements. Part or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. It can be understood and implemented by those skilled in the art without creative effort.

It should be noted that in this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is a relationship between these entities or operations. There is no such actual relationship or order between them. The term "comprises", "comprises" or any other variation thereof is intended to cover a non-exclusive inclusion such that a process, method, article or apparatus comprising a set of elements includes not only those elements but also other elements not expressly listed elements, or also elements inherent in such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article or apparatus comprising said element.

The methods and devices provided by the embodiments of the present application have been described in detail above. The principles and implementation methods of the present application have been explained by using specific examples in this paper. The descriptions of the above embodiments are only used to help understand the methods and methods of the present application. core idea; at the same time, for those of ordinary skill in the art, according to the idea of this application, there will be changes in the specific implementation and application scope. In summary, the content of this specification should not be construed as limiting the application .

Claims

A method for broadcasting information, characterized in that it is applied to a vehicle equipped with an imaging device, and the imaging device faces a user seat inside the vehicle, the method comprising:

determining facial expression information of a user located in the user seat based on the image information collected by the camera device;

Acquiring specific status information of the vehicle, the specific status information includes one or more of the following: alarm information output by the vehicle, or the vehicle receives a message indicating that the execution module of the vehicle changes working status The actual working status information of the execution module when the user instruction is not executed, and the user instruction is not executed;

If the expression feature in the expression information matches the preset expression feature representing doubt, analysis information is obtained based on the specific state information, and the analysis information is broadcast.
The method according to claim 1, wherein the alarm information output by the vehicle includes: one or more of alarm icons displayed on the instrument panel, or alarm sounds emitted by vehicle components.
The method according to claim 1, wherein the actual working state information of the execution module is determined based on the instruction input to the execution module by the automatic driving module of the vehicle; the user instruction is related to the automatic driving module The commands input to the execution modules are different.
The method according to claim 3, wherein the vehicle is equipped with a visual sensor facing the outside of the vehicle, and the instruction input by the automatic driving module to the execution module is based on the environmental observation data collected by the visual sensor Sure.
The method according to claim 3, wherein the user instruction is different from the instruction input by the automatic driving module to the execution module and includes:

The user's command input to the power component of the vehicle is different from the command input of the automatic driving module to the power component; or

The driving mode set by the user for the vehicle is different from the driving mode set by the automatic driving module for the vehicle.
The method according to claim 5, characterized in that,

The instruction input by the user to the power assembly includes a lane change instruction; the instruction input by the automatic driving module to the power assembly includes an instruction to keep going straight;

The driving mode set by the user for the vehicle includes an automatic driving mode; the driving mode set by the automatic driving module for the vehicle includes a manual driving mode.
The method according to claim 1, wherein the vehicle includes a plurality of execution modules, and each execution module has its own corresponding actual working state information, and the obtaining analysis information based on the specific state information includes:

determining a target execution module among the plurality of execution modules according to the expression information;

The analysis information is obtained according to the actual working state information of the target execution module.
The method according to claim 1, wherein said acquiring parsing information based on said specific state information comprises:

determining the user's line of sight direction based on the expression information;

determining a target execution module within an interior environment of the vehicle based on the line-of-sight direction;

The analysis information is obtained according to the actual working state information of the target execution module.
The method according to claim 1, wherein the broadcasting the analysis information comprises:

Controlling the playback device in the vehicle to broadcast the analysis information.
The method according to claim 1, further comprising:

Determine the user's gesture and behavior information based on the images collected by the camera device, the gesture and behavior information includes one or more of the user's head posture, face information, line of sight direction, hand position, and behavior information. item;

determining state index information of the user based on the gesture behavior information, the state index information being used to indicate that the user is in a state of concentrating on driving or not in a state of driving;

Acquiring prompt content information and presentation parameters corresponding to the state indicator information; the presentation parameters include audio presentation parameters and/or visual presentation parameters;

Based on the prompting content information and the presentation parameters of the prompting degree, generating prompting information of a corresponding degree, and outputting the prompting information of a corresponding degree.
The method according to claim 10, wherein the audio presentation parameters include pitch, volume, speech rate, and playback frequency; the visual presentation parameters include brightness, animation type, and area occupied by the broadcast content on the interface.
The method according to claim 11, wherein the animation type includes simulated expressions representing different degrees of severity, and the outputting the prompt information of the corresponding degree includes:

A corresponding degree of simulated expression is displayed on the human-computer interaction interface of the vehicle.
The method according to claim 10, wherein the prompt information of the highest level further includes: tightening the seat belt at the user's location or turning on the double flashing lights of the vehicle.
A method for prompting traffic status, characterized in that the method is applied to a vehicle, and the vehicle is equipped with a sensor facing the outside of the vehicle and a camera device facing a user seat inside the vehicle; the method includes:

Based on the road information collected by the sensor, it is determined that the road where the vehicle is located changes from a non-traffic state to a traffic state;

Obtaining the expected motion state of the vehicle corresponding to the passing state;

Acquiring the current motion state of the vehicle;

determining the characteristics of the driving state of the user located in the user seat based on the images collected by the camera device;

If the current motion state of the vehicle does not reach the expected motion state, and the feature of the driving state of the user matches the state feature for indicating inattentive driving, output a Status message.
The method of claim 14 wherein the vehicle is in a manual driving mode.
The method according to claim 14, wherein the road information includes: one or more of traffic light information and surrounding object information.
The method according to claim 16, characterized in that,

In the case where the traffic light information indicates that vehicles are allowed to pass, the road is in a passable state; and/or

The information on the surrounding objects indicates that the distance between the surrounding objects and the vehicle is greater than a distance threshold, or indicates that there is no obstacle within a preset range of the vehicle, and the road is in a passing state.
The method according to claim 14, wherein the determining the features of the driving state of the user located in the user seat based on the image collected by the camera device comprises:

Determine the gesture behavior information of the user based on the images collected by the camera device; the gesture behavior information includes one or more of the user's head posture, face information, line of sight direction, hand position, and behavior information. Multiple;

A feature of the user's driving state is determined based on the gesture behavior information.
The method according to claim 14, characterized in that the method further comprises:

If within a preset period of time after outputting the prompt information, the current motion state of the vehicle has not yet reached the expected motion state, and/or the characteristics of the user's driving state are still consistent with inattentive driving The characteristics of the state are matched, and a warning message is sent to the surrounding objects.
The method according to claim 19, wherein the sending out the warning information comprises: turning on the double flashing lights of the vehicle to warn surrounding objects.
The method according to claim 14, wherein before outputting the prompt information, the method further comprises:

Obtain prompt content information and presentation parameters of prompt levels corresponding to the driving state index information; the presentation parameters include audio presentation parameters and/or visual presentation parameters;

Based on the prompting content information and the presentation parameters of the prompting degree, generating prompting information of a corresponding degree, and outputting the prompting information of a corresponding degree.
The method according to claim 21, wherein the audio presentation parameters include pitch, volume, speech rate, and playback frequency; and the visual presentation parameters include brightness, animation type, and area occupied by the broadcast content on the interface.
The method according to claim 22, wherein the animation type includes simulated expressions representing different degrees of severity, and the outputting prompt information of the corresponding degree includes:

A corresponding degree of simulated expression is displayed on the human-computer interaction interface of the vehicle.
The method according to claim 21, wherein the prompt information of the highest level further includes: tightening the seat belt at the user's location or turning on the double flashing lights of the vehicle.
A method for broadcasting information, characterized in that the method is applied to a vehicle equipped with an imaging device, and the imaging device faces a user seat inside the vehicle, and the method includes:

Based on the images collected by the camera device, determine the posture and behavior information of the user located in the user seat, the posture and behavior information includes the user's head posture, face information, line of sight direction, hand position, and behavior information. one or more;

determining state index information of the user based on the gesture behavior information, the state index information being used to indicate that the user is in a state of concentrating on driving or not in a state of driving;

Obtain prompt content information and presentation parameters of prompt levels corresponding to the state indicator information; the presentation parameters include audio presentation parameters and/or visual presentation parameters;

Based on the prompting content information and the presentation parameters of the prompting degree, generating prompting information of a corresponding degree, and outputting the prompting information of a corresponding degree.
The method according to claim 25, wherein the audio presentation parameters include pitch, volume, speech rate, and playback frequency; the visual presentation parameters include brightness, animation type, and the occupied area of the broadcast content on the interface.
The method according to claim 26, wherein the animation type includes simulated expressions representing different degrees of severity, and the outputting prompt information of the corresponding degree includes:

A corresponding degree of simulated expression is displayed on the human-computer interaction interface of the vehicle.
The method according to claim 27, wherein the simulated expressions representing different degrees of severity include: one or more of simulated expressions representing normal emotions, simulated expressions representing nervous emotions, and simulated expressions representing painful emotions kind.
The method according to claim 25, wherein the prompt information of the highest level further includes: tightening the seat belt at the user's location or turning on the double flashing lights of the vehicle.
An information broadcasting device is characterized in that it comprises:

processor;

memory for storing processor-executable program instructions;

Wherein, when the processor invokes the executable instruction, the operation of the method according to any one of claims 1-13 is implemented.
A passing state prompting device is characterized in that it comprises:

processor;

memory for storing processor-executable program instructions;

Wherein, when the processor invokes the executable instruction, the operation of the method according to any one of claims 14-24 is realized.
An information broadcasting device is characterized in that it comprises:

processor;

memory for storing processor-executable program instructions;

Wherein, when the processor invokes the executable instruction, the operation of the method according to any one of claims 25-29 is realized.
A vehicle, characterized in that it comprises

body;

Power components;

a user seat towards the interior of the vehicle, and/or a sensor towards the exterior of the vehicle; and

A device; the device includes a processor, and a memory for storing processor-executable program instructions, wherein the processor implements the operation of the method according to any one of claims 1-29 when invoking the executable instructions.
A computer program product, comprising a computer program, characterized in that, when the computer program is executed by a processor, the steps of the method according to any one of claims 1-29 are realized.
A machine-readable storage medium, characterized in that several computer instructions are stored on the machine-readable storage medium, and the method according to any one of claims 1-29 is executed when the computer instructions are executed.