WO2024093768A1

WO2024093768A1 - Vehicle alarm method and related device

Info

Publication number: WO2024093768A1
Application number: PCT/CN2023/126656
Authority: WO
Inventors: 卢远志; 吕自波; 王燕
Original assignee: 华为技术有限公司
Priority date: 2022-10-31
Filing date: 2023-10-26
Publication date: 2024-05-10
Also published as: CN115675504A

Abstract

A vehicle alarm method and a related device, applicable to the field of autonomous driving in artificial intelligence. The method comprises: acquiring a line of sight of a driver (301) and environment information around a vehicle (302), and when it is determined, according to the line of sight of the driver, that a first condition is satisfied (304), outputting alarm information (305), wherein the first condition comprises a mismatch between the line of sight of the driver and a driving intention of the vehicle (304), and the driving intention of the vehicle is determined on the basis of the environment information around the vehicle (303). In the scenario of triggering alarm information provided by the method, a line of sight of a driver in a vehicle is considered, and environment information around the vehicle is also considered, thereby facilitating more accurate output of the alarm information.

Description

Vehicle warning method and related equipment

This application claims priority to a Chinese patent application filed with the State Intellectual Property Office on October 31, 2022, with application number 202211366481.9 and invention name “A vehicle alarm method and related equipment”, the entire contents of which are incorporated by reference in this application.

Technical Field

The present application relates to the field of autonomous driving, and in particular to a vehicle warning method and related equipment.

Background technique

During the driving process of the vehicle, the intelligent driving system in the vehicle needs to understand the driver's status and output warning information to the driver in time. The warning information is used to remind the driver to take over the vehicle. The triggering scenarios of the warning information mainly include: periodic triggering and event triggering; for example, the aforementioned event triggering may include: fatigue driving, making phone calls or other abnormal behaviors of the driver.

However, in the triggering scenarios of the above warning information, only the information of the driver inside the vehicle is considered, while the information outside the vehicle is not considered, resulting in an inaccurate determination process of "whether to output the warning information".

Summary of the invention

The present application provides a vehicle alarm method and related equipment. The triggering scenario of the alarm information provided by this solution not only takes into account the driver's line of sight inside the vehicle, but also takes into account the environmental information around the vehicle, which is conducive to more accurate output of alarm information.

In order to solve the above technical problems, this application provides the following technical solutions:

In the first aspect, the present application provides a vehicle warning method, which can be used in the field of autonomous driving in the field of artificial intelligence. The method may include: the vehicle obtains the driver's line of sight and the environmental information around the vehicle; when it is determined that the first condition is met according to the driver's line of sight, the warning information is output. Among them, the first condition includes that the driver's line of sight does not match the vehicle's driving intention, and the vehicle's driving intention is determined based on the environmental information around the vehicle; the vehicle's driving intention may include any one or more of the following combinations: one or more objects that the driver needs to pay attention to during driving, the vehicle's driving direction or other types of driving intentions, etc. The aforementioned vehicle may be a car, truck, motorcycle, bus, ship, airplane, helicopter, lawn mower, recreational vehicle, amusement park vehicle, construction equipment, tram and golf cart, etc.

In the present application, another triggering scenario for warning information is provided. When it is determined that the driver's line of sight does not match the vehicle's driving intention, that is, when it is determined that the object the driver is paying attention to does not match the driving intention determined by the vehicle's intelligent driving system, a warning message is output to the driver; in addition, in the determination process of "whether to output warning information", not only the driver's line of sight inside the vehicle is considered, but also the environmental information around the vehicle is considered, which is conducive to more accurate output of warning information.

Optionally, the method further includes: the vehicle determines one or more objects that the driver needs to pay attention to during driving based on environmental information around the vehicle; wherein the driving intention of the vehicle includes objects that the driver needs to pay attention to during driving, and the situation where the driver's line of sight does not match the vehicle's driving intention includes: the driver's line of sight is outside all objects that need attention. In the present application, the driving behavior needs to be determined based on the surrounding objects during vehicle driving, that is, the intelligent driving system and the driver need to observe the surrounding objects in real time during vehicle driving, and the driver's line of sight being outside the objects that need attention is determined as a situation where the driver's line of sight does not match the vehicle's driving intention, which is consistent with the logic of manual driving, that is, this solution has a high degree of fit with the actual application scenario, which is conducive to accurately determining whether the driver's line of sight matches the vehicle's driving intention.

Optionally, the situation that the driver's line of sight is outside the object that needs attention may include: at a first moment, the driver's line of sight is outside all objects that need attention. Alternatively, the situation that the driver's line of sight is outside the object that needs attention may include: at the first moment, the driver's line of sight is inside the object that needs attention, and the second movement parameter of the driver's line of sight in the first time period does not match the first movement parameter, and the first time period is after the first moment. Alternatively, the situation that the driver's line of sight is outside the object that needs attention may include: at the first moment and multiple consecutive moments after the first moment, the driver's line of sight is outside the object that needs attention.

Optionally, the method may further include: the vehicle determines the first movement parameter of the object requiring attention within the first time period based on the environmental information around the vehicle. The driver's line of sight being outside the object requiring attention also includes: the driver's line of sight being within the object requiring attention at the first moment, and the second movement parameter of the driver's line of sight within the first time period does not match the first movement parameter, and the first time period is after the first moment. The first movement parameter may include the first movement direction of the object requiring attention within the first time period; or, The first movement parameter may also include any one or more of the following information: a first movement distance, a first movement speed, or other movement information of the object of interest within the first time period, etc. Correspondingly, the second movement parameter may include a second movement direction of the driver's line of sight within the first time period; or, the second movement parameter may also include any one or more of the following information: a second movement distance, a second movement speed, or other movement information of the driver's line of sight within the first time period, etc.

In the present application, when considering whether the driver's line of sight is outside the object that needs attention, not only whether the driver's line of sight is outside the object that needs attention at a single first moment is considered, but also whether the movement parameters of the driver's line of sight in a first time period after the first moment are consistent with the movement parameters of the object that needs attention. This is beneficial to improving the accuracy of the judgment process of "whether the driver's line of sight is outside the object that needs attention", and is also beneficial to improving the safety of the vehicle driving process.

Optionally, the first movement parameter includes a first movement direction of an object that needs attention within a first time period, and the second movement parameter includes a second movement direction of the driver's line of sight within the first time period; each object that needs attention can be represented as an area, and the driver's line of sight observation range can also be represented as an area, then the first movement direction can be the movement direction of any point in the object that needs attention within the first time period, and the second movement direction can be the movement direction of any point within the driver's line of sight observation range within the first time period. The situation where the second movement parameter does not match the first movement parameter includes: the difference between the first movement direction and the second movement direction satisfies the second condition; for example, the second condition can include that the angle between the first movement direction and the second movement direction is less than the first angle threshold. In the present application, the first movement parameter and the second movement parameter are both determined as movement directions, that is, an implementation scheme that is easy to implement and has high accuracy is provided.

Optionally, the driving intention of the vehicle includes the driving direction of the vehicle, and the situation where the driver's line of sight does not match the driving intention of the vehicle includes: the driver's line of sight does not match the driving direction of the vehicle. The method may also include: when it is determined based on the environmental information around the vehicle that there is no object that needs attention at the current moment, it can be determined whether the direction of the driver's line of sight matches the driving direction of the vehicle; wherein, the situation where the direction of the driver's line of sight does not match the driving direction of the vehicle may include that the angle between the driver's line of sight and the driving direction of the vehicle is greater than or equal to the second angle threshold.

Optionally, the method may further include: the vehicle displays objects that require attention to the driver. For example, the vehicle may highlight objects that require attention determined by the vehicle to the driver through the head-up display system HUD; optionally, the vehicle may highlight objects that require attention when displaying the navigation route to the driver through the HUD. The highlighting method includes any one or more of the following: adding prompt text next to the object that requires attention, framing the object that requires attention, or other methods of highlighting the object that requires attention, etc. In the present application, objects that require attention can also be displayed to the driver, so as to assist the driver in learning the driving ideas of the intelligent driving system, which is conducive to improving the safety of the driver's driving behavior.

Optionally, the vehicle outputs warning information when it is determined that the first condition is met based on the driver's line of sight, including: when the duration of the mismatch between the driver's line of sight and the vehicle's driving intention reaches a first duration, the vehicle outputs warning information, and the value of the first duration is associated with the safety of the driver's driving behavior. Among them, the factors determining the safety of the driver's driving behavior may include any one or more of the following: the cumulative number of forward warnings FCW, the number of sudden accelerations, the number of sudden decelerations, the distance from the vehicle in front when following the vehicle, the speed of steering, the average number of times the vehicle's intelligent driving system takes over the user's driving, or other information that can reflect the safety of the driver's driving behavior. For example, the higher the safety of the driver's driving behavior, the longer the value of the first duration can be, and the lower the safety of the driver's driving behavior, the shorter the value of the first duration can be.

In the present application, when the duration of the mismatch between the driver's line of sight and the vehicle's driving intention reaches a first duration, a warning message is output to the driver, that is, the value of the first duration can affect the frequency of outputting the warning message, and the safety of the driver's driving behavior will affect the value of the first duration, which is conducive to reducing the frequency of outputting warning messages to drivers with high safety levels, thereby avoiding disturbing the user; it is also conducive to increasing the frequency of outputting warning messages to drivers with low safety levels, thereby improving the safety of the driving process.

On the second aspect, the present application provides a vehicle alarm device that can be used in the field of autonomous driving in the field of artificial intelligence. The vehicle alarm device may include: an acquisition module for acquiring the driver's line of sight and environmental information around the vehicle; an alarm module for outputting an alarm message when a first condition is determined to be met based on the driver's line of sight, wherein the first condition includes that the driver's line of sight does not match the vehicle's driving intention, and the vehicle's driving intention is determined based on the environmental information around the vehicle.

In the second aspect of the present application, the vehicle alarm device can also execute the steps performed by the vehicle in the first aspect and various possible implementation methods of the first aspect. For the specific implementation steps of the second aspect of the present application and various possible implementation methods of the second aspect, as well as the beneficial effects brought about by each possible implementation method, you can refer to the description of the various possible implementation methods in the first aspect, and will not be repeated here one by one.

In a third aspect, the present application provides a vehicle, which may include a memory, a processor, and a bus system, wherein the memory is used to store The processor is used to execute the program in the memory, including the following steps: the bus system is used to connect the memory and the processor so that the memory and the processor communicate with each other. The processor can be used to execute the steps executed by the vehicle in each possible implementation of the first aspect, and the details can be referred to the first aspect, which will not be repeated here.

In a fourth aspect, the present application provides a computer-readable storage medium, in which a computer program is stored. When the computer-readable storage medium is run on a computer, the computer executes the vehicle alarm method described in the first aspect.

In a fifth aspect, the present application provides a circuit system, which includes a processing circuit, and the processing circuit is configured to execute the vehicle alarm method described in the first aspect above.

In a sixth aspect, the present application provides a computer program which, when executed on a computer, enables the computer to execute the vehicle alarm method described in the first aspect above.

In a seventh aspect, the present application provides a chip system, which includes a processor for supporting a server or a vehicle speed generating device to implement the functions involved in the above aspects, for example, sending or processing the data and/or information involved in the above methods. In a possible design, the chip system also includes a memory, which is used to store program instructions and data necessary for the server or communication device. The chip system can be composed of a chip, or it can include a chip and other discrete devices.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is a schematic diagram of a structure of a vehicle provided in an embodiment of the present application;

FIG2 is a flow chart of a vehicle alarm method provided in an embodiment of the present application;

FIG3 is another schematic diagram of a flow chart of a vehicle alarm method provided in an embodiment of the present application;

FIG4 is a schematic diagram of an object that a driver needs to pay attention to provided in an embodiment of the present application;

FIG5 is another schematic diagram of an object that a driver needs to pay attention to provided by an embodiment of the present application;

FIG6 is another schematic diagram of an object that a driver needs to pay attention to provided by an embodiment of the present application;

FIG7a is a schematic diagram of a driver's line of sight being within an object requiring attention and a driver's line of sight being outside an object requiring attention provided by an embodiment of the present application;

FIG7b is a schematic diagram of mapping an object requiring attention in a second coordinate system to a coordinate system corresponding to a virtual camera provided by an embodiment of the present application;

FIG7c is another schematic diagram of mapping an object requiring attention in a second coordinate system to a coordinate system corresponding to a virtual camera provided by an embodiment of the present application;

FIG8 is a schematic diagram of a first movement parameter and a second movement parameter provided in an embodiment of the present application;

FIG9 is a schematic diagram of a structure of a vehicle speed generating device provided in an embodiment of the present application;

FIG10 is another schematic diagram of the structure of a vehicle speed generating device provided in an embodiment of the present application;

FIG11 is another schematic diagram of the structure of a vehicle provided in an embodiment of the present application;

FIG. 12 is a schematic diagram of the structure of a chip provided in an embodiment of the present application.

Detailed ways

The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchanged under appropriate circumstances. This is merely a way of distinguishing objects with the same properties when describing the embodiments of the present application. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, so that a process, method, system, product, or device that includes a series of units is not necessarily limited to those units, but may include other units that are not explicitly listed or inherent to these processes, methods, products, or devices.

The embodiments of the present application are described below in conjunction with the accompanying drawings. It is known to those skilled in the art that with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

The embodiments of the present application can be applied to vehicles, and specifically can be applied to scenarios for determining whether it is necessary to issue a warning message to the driver. Among them, the aforementioned vehicles can be cars, trucks, motorcycles, buses, ships, airplanes, helicopters, lawn mowers, recreational vehicles, amusement park vehicles, construction equipment, trams, and golf carts, etc., and the embodiments of the present application do not make special limitations. For example, when the vehicle adopts the automatic driving mode, the vehicle can periodically obtain the driver's information to determine whether it is necessary to output a warning message to the driver. For another example, when the vehicle adopts the manual driving mode, the vehicle can periodically obtain the driver's information to determine whether it is necessary to output a warning message to the driver. It should be understood that the examples given here are only for the convenience of understanding the application scenarios of the embodiments of the present application, and are not intended to be exhaustive of the application scenarios of the embodiments of the present application.

In order to facilitate understanding of the present solution, the structure of the vehicle is first introduced in conjunction with FIG. 1 in the embodiment of the present application. Please refer to FIG. 1 first. FIG. 1 is a schematic diagram of the structure of a vehicle provided in the embodiment of the present application. The vehicle 100 is configured in a fully or partially automatic driving mode. For example, the vehicle 100 can control itself while being in the automatic driving mode, and can determine the current state of the vehicle and its surrounding environment through human operation, determine the possible behavior of at least one other vehicle in the surrounding environment, and determine the confidence level corresponding to the possibility of other vehicles performing possible behaviors, and control the vehicle 100 based on the determined information. When the vehicle 100 is in the automatic driving mode, the vehicle 100 can also be set to operate without human interaction.

The vehicle 100 may include various subsystems, such as a travel system 102, a sensor system 104, a control system 106, one or more peripheral devices 108, and a power source 110, a computer system 112, and a user interface 116. Optionally, the vehicle 100 may include more or fewer subsystems, and each subsystem may include multiple components. In addition, each subsystem and component of the vehicle 100 may be interconnected by wire or wirelessly.

The travel system 102 may include components that provide powered movement to the vehicle 100. In one embodiment, the travel system 102 may include an engine 118, a power source 119, a transmission 120, and wheels/tires 121.

Among them, the engine 118 can be an internal combustion engine, an electric motor, an air compression engine, or a combination of other types of engines, for example, a hybrid engine consisting of a gasoline engine and an electric motor, and a hybrid engine consisting of an internal combustion engine and an air compression engine. The engine 118 converts the energy source 119 into mechanical energy. Examples of energy sources 119 include gasoline, diesel, other petroleum-based fuels, propane, other compressed gas-based fuels, ethanol, solar panels, batteries, and other sources of electricity. The energy source 119 can also provide energy for other systems of the vehicle 100. The transmission 120 can transmit the mechanical power from the engine 118 to the wheels 121. The transmission 120 may include a gearbox, a differential, and a drive shaft. In one embodiment, the transmission 120 may also include other devices, such as a clutch. Among them, the drive shaft may include one or more shafts that can be coupled to one or more wheels 121.

The sensor system 104 may include several sensors that sense information about the environment surrounding the vehicle 100. For example, the sensor system 104 may include a positioning system 122 (the positioning system may be a global positioning GPS system, or a Beidou system or other positioning systems), an inertial measurement unit (IMU) 124, a radar 126, a laser rangefinder 128, and a camera 130. The sensor system 104 may also include sensors that obtain internal systems of the vehicle 100 (for example, sensors for obtaining in-vehicle air quality, a fuel gauge, an oil temperature gauge, etc.). The sensing data from one or more of these sensors can be used to detect objects and their corresponding characteristics (position, shape, direction, speed, etc.). Such detection and recognition are key functions for the safe operation of the autonomous vehicle 100.

Among them, the positioning system 122 can be used to estimate the geographic location of the vehicle 100. The IMU 124 is used to sense the position and orientation changes of the vehicle 100 based on inertial acceleration. In one embodiment, the IMU 124 can be a combination of an accelerometer and a gyroscope. The radar 126 can use radio signals to sense objects in the surrounding environment of the vehicle 100, and can be specifically manifested as a millimeter wave radar or a laser radar. In some embodiments, in addition to sensing objects, the radar 126 can also be used to sense the speed and/or direction of travel of the object. The laser rangefinder 128 can use lasers to sense objects in the environment where the vehicle 100 is located. In some embodiments, the laser rangefinder 128 may include one or more laser sources, a laser scanner, and one or more detectors, as well as other system components. The camera 130 can be used to capture multiple images of the surrounding environment of the vehicle 100. The camera 130 can be a still camera or a video camera.

The control system 106 controls the operation of the vehicle 100 and its components. The control system 106 may include various components, including a steering system 132 , a throttle 134 , a brake unit 136 , a computer vision system 140 , a lane control system 142 , and an obstacle avoidance system 144 .

Among them, the steering system 132 can be operated to adjust the forward direction of the vehicle 100. For example, in one embodiment, it can be a steering wheel system. The throttle 134 is used to control the operating speed of the engine 118 and thus control the speed of the vehicle 100. The brake unit 136 is used to control the deceleration of the vehicle 100. The brake unit 136 can use friction to slow down the wheel 121. In other embodiments, the brake unit 136 can convert the kinetic energy of the wheel 121 into electric current. The brake unit 136 can also take other forms to slow down the rotation speed of the wheel 121 to control the speed of the vehicle 100. The computer vision system 140 can be operated to process and analyze the images captured by the camera 130 in order to identify objects and/or features in the surrounding environment of the vehicle 100. The objects and/or features may include traffic signals, road boundaries and obstacles. The computer vision system 140 can use object recognition algorithms, Structure from Motion (SFM) algorithms, video tracking and other computer vision technologies. In some embodiments, the computer vision system 140 can be used to map the environment, track objects, estimate the speed of objects, and so on. The route control system 142 is used to determine the route and speed of the vehicle 100. In some embodiments, the route control system 142 may include a lateral planning module 1421 and a longitudinal planning module 1422, which are respectively used to determine the route and speed of the vehicle 100 by combining data from the obstacle avoidance system 144, the GPS 122, and one or more predetermined maps. The avoidance system 144 is used to identify, evaluate and avoid or otherwise cross obstacles in the environment of the vehicle 100, which may be represented by actual obstacles and virtual moving bodies that may collide with the vehicle 100. In one example, the control system 106 may include components other than those shown and described in addition or in place. Alternatively, some of the components shown above may be reduced.

The vehicle 100 interacts with external sensors, other vehicles, other computer systems, or users through the peripheral device 108. The peripheral device 108 may include a wireless communication system 146, an onboard computer 148, a microphone 150, and/or a speaker 152. In some embodiments, the peripheral device 108 provides a means for the user of the vehicle 100 to interact with the user interface 116. For example, the onboard computer 148 may provide information to the user of the vehicle 100. The user interface 116 may also operate the onboard computer 148 to receive user input. The onboard computer 148 may be operated through a touch screen. In other cases, the peripheral device 108 may provide a means for the vehicle 100 to communicate with other devices located in the vehicle. For example, the microphone 150 may receive audio (e.g., voice commands or other audio input) from the user of the vehicle 100. Similarly, the speaker 152 may output audio to the user of the vehicle 100. The wireless communication system 146 may communicate wirelessly with one or more devices directly or via a communication network. For example, the wireless communication system 146 may use 3G cellular communication, such as CDMA, EVDO, GSM/GPRS, or 4G cellular communication, such as LTE. Or 5G cellular communication. The wireless communication system 146 may communicate using a wireless local area network (WLAN). In some embodiments, the wireless communication system 146 may communicate directly with the device using an infrared link, Bluetooth, or ZigBee. Other wireless protocols, such as various vehicle communication systems, for example, the wireless communication system 146 may include one or more dedicated short range communications (DSRC) devices, which may include public and/or private data communications between vehicles and/or roadside stations.

The power source 110 can provide power to various components of the vehicle 100. In one embodiment, the power source 110 can be a rechargeable lithium-ion or lead-acid battery. One or more battery packs of such batteries can be configured as a power source to provide power to various components of the vehicle 100. In some embodiments, the power source 110 and the energy source 119 can be implemented together, such as in some all-electric vehicles.

Some or all functions of the vehicle 100 are controlled by a computer system 112. The computer system 112 may include at least one processor 113 that executes instructions 115 stored in a non-transitory computer-readable medium such as a memory 114. The computer system 112 may also be a plurality of computing devices that control individual components or subsystems of the vehicle 100 in a distributed manner. The processor 113 may be any conventional processor, such as a commercially available central processing unit (CPU). Alternatively, the processor 113 may be a dedicated device such as an application specific integrated circuit (ASIC) or other hardware-based processor. Although FIG. 1 functionally illustrates the processor, memory, and other components of the computer system 112 in the same block, it should be understood by those skilled in the art that the processor, or memory, may actually include multiple processors, or memories that are not stored in the same physical housing. For example, the memory 114 may be a hard drive or other storage medium located in a housing different from the computer system 112. Thus, references to processor 113 or memory 114 will be understood to include references to a collection of processors or memories that may or may not operate in parallel. Rather than using a single processor to perform the steps described herein, some components, such as the steering component and the deceleration component, may each have their own processor that performs only calculations related to the functionality of the component.

In various aspects described herein, the processor 113 may be located remotely from the vehicle 100 and in wireless communication with the vehicle 100. In other aspects, some of the processes described herein are performed on a processor 113 disposed within the vehicle 100 while others are performed by the remote processor 113, including taking the necessary steps to perform a single maneuver.

In some embodiments, the memory 114 may include instructions 115 (e.g., program logic) that can be executed by the processor 113 to perform various functions of the vehicle 100, including those described above. The memory 114 may also include additional instructions, including instructions to send data to, receive data from, interact with, and/or control one or more of the travel system 102, the sensor system 104, the control system 106, and the peripheral devices 108. In addition to the instructions 115, the memory 114 may also store data such as road maps, route information, the vehicle's location, direction, speed, and other such vehicle data, as well as other information. This information can be used by the vehicle 100 and the computer system 112 during the operation of the vehicle 100 in autonomous, semi-autonomous, and/or manual modes. A user interface 116 is used to provide information to or receive information from a user of the vehicle 100. Optionally, the user interface 116 may include one or more input/output devices within the set of peripheral devices 108, such as a wireless communication system 146, an onboard computer 148, a microphone 150, and a speaker 152.

The computer system 112 may control functions of the vehicle 100 based on input received from various subsystems (e.g., the travel system 102, the sensor system 104, and the control system 106) and from the user interface 116. For example, the computer system 112 may utilize input from the control system 106 in order to control the steering system 132 to avoid obstacles detected by the sensor system 104 and the obstacle avoidance system 144. In some embodiments, the computer system 112 may be operable to provide control over many aspects of the vehicle 100 and its subsystems.

Alternatively, one or more of the above-mentioned components may be installed or associated separately from the vehicle 100. For example, the memory 114 may exist partially or completely separately from the vehicle 100. The above-mentioned components may be communicatively coupled together in a wired and/or wireless manner.

Optionally, the above components are just an example. In actual applications, the components in the above modules may be added or deleted according to actual needs, and Figure 1 should not be understood as a limitation on the embodiments of the present application. A vehicle traveling on a road, such as the vehicle 100 above, can identify objects in its surrounding environment to determine the adjustment of the current speed. The object can be another vehicle, a traffic control device, or other types of objects. In some examples, each identified object can be considered independently, and based on the respective characteristics of the object, such as its current speed, acceleration, distance from the vehicle, etc., it can be used to determine the speed to be adjusted for the vehicle.

Optionally, the vehicle 100 or a computing device associated with the vehicle 100, such as the computer system 112, computer vision system 140, and memory 114 of FIG. 1, can predict the behavior of the identified object based on the characteristics of the identified object and the state of the surrounding environment (e.g., traffic, rain, ice on the road, etc.). Optionally, each identified object depends on the behavior of each other, so all the identified objects can also be considered together to predict the behavior of a single identified object. The vehicle 100 can adjust its speed based on the predicted behavior of the identified object. In other words, the vehicle 100 can determine what stable state the vehicle will need to adjust to (e.g., accelerate, decelerate, or stop) based on the predicted behavior of the object. In this process, other factors can also be considered to determine the speed of the vehicle 100, such as the lateral position of the vehicle 100 in the road on which it is traveling, the curvature of the road, the proximity of static and dynamic objects, etc. In addition to providing instructions to adjust the speed of the vehicle, the computing device may also provide instructions to modify the steering angle of vehicle 100 so that vehicle 100 follows a given trajectory and/or maintains a safe lateral and longitudinal distance from objects near vehicle 100 (e.g., cars in adjacent lanes on the road).

In combination with the above description, the embodiment of the present application provides a vehicle alarm method, which can be applied to the vehicle 100 shown in Figure 1. Please refer to Figure 2. Figure 2 is a flow chart of the vehicle alarm method provided by the embodiment of the present application. The vehicle alarm method provided by the embodiment of the present application may include: A1. Obtaining the driver's line of sight and the environmental information around the vehicle. A2. Outputting alarm information when the first condition is determined to be satisfied according to the driver's line of sight, wherein the first condition includes that the driver's line of sight does not match the vehicle's driving intention, and the vehicle's driving intention is determined based on the environmental information around the vehicle. For example, when the driver's line of sight is inconsistent with the driving direction of the vehicle, it can be regarded that the driver's line of sight does not match the vehicle's driving intention; for another example, when the field of view of the driver's line of sight is outside the object determined by the vehicle to be concerned, it can be regarded that the driver's line of sight does not match the vehicle's driving intention, etc. There may also be other "driver's line of sight does not match the vehicle's driving intention" scenes, which are not exhaustive here.

In an embodiment of the present application, another triggering scenario for warning information is provided. When it is determined that the driver's line of sight does not match the vehicle's driving intention, that is, when it is determined that the object the driver is paying attention to does not match the driving intention determined by the vehicle's intelligent driving system, a warning message is output to the driver; in addition, in the determination process of "whether to output warning information", not only the driver's line of sight inside the vehicle is taken into account, but also the environmental information around the vehicle is taken into account, which is conducive to more accurate output of warning information.

The specific implementation of the vehicle alarm method provided by the embodiment of the present application is described in detail below. Specifically, please refer to Figure 3, which is another flow chart of the vehicle alarm method provided by the embodiment of the present application. The vehicle alarm method provided by the embodiment of the present application may include:

301. Obtain the driver's line of sight.

In the embodiment of the present application, in order to obtain the driver's line of sight, one or more first sensors can be deployed inside the vehicle, and the field of view (FOV) of the one or more first sensors must at least cover the driver's head. Among them, the first sensor can be specifically manifested as a camera, a radar or other sensor that can collect video information of the driver; for example, the camera can be an infrared camera or other types of cameras. Exemplarily, the installation position of any first sensor can be any of the following positions: on the steering wheel of the vehicle, on the rearview mirror inside the vehicle, on the A-pillar of the vehicle, or on other positions inside the vehicle, etc. The A-pillar of the vehicle refers to two pillars located on the left front and right front, respectively. The two A-pillars connect the top of the vehicle and the front cabin of the vehicle. It should be noted that the installation position of the first sensor can be flexibly set according to the actual application product. The example here is only for the convenience of understanding this solution and is not used to limit this solution.

The vehicle can collect information about the driver at at least one moment through one or more first sensors inside. The information about the driver at at least one moment can be an image of the driver at one moment; or, it can be a video of the driver at multiple moments; or, it can be at least one set of point cloud data corresponding to at least one moment, and each set of point cloud data reflects the driver's behavior at one moment. The information about the driver at at least one moment includes information about the driver's head at at least one moment. Based on the collected video information about the driver, at least the driver's line of sight can be tracked to obtain the driver's line of sight. For example, the vehicle can input the collected at least one video information of the driver into a video recording device. The first neural network is input into the first neural network, and the driver's line of sight is tracked by the first neural network to obtain first information output by the first neural network, where the first information indicates the driver's line of sight at at least one moment.

Among them, the driver's line of sight at a certain moment may include the position information of the driver's line of sight in the first coordinate system; the first coordinate system may be a coordinate system established with the first sensor as the origin, or a coordinate system established with the center of the vehicle as the origin, or a coordinate system established with the center of the rearview mirror inside the vehicle as the origin, and so on; the first coordinate system may be a two-dimensional coordinate system, a three-dimensional coordinate system, or other types of coordinate systems, and the specific first coordinate system to be adopted may be determined in combination with the actual application scenario, and is not limited here.

Optionally, the vehicle can also obtain any one or more of the following information about the driver based on at least one video information collected of the driver: the driver's fatigue level, whether the driver has abnormal behavior, the driver's user identity number (identity, ID) or other information, etc., which can be flexibly determined in combination with actual application scenarios, and are not exhaustive here. Among them, the driver's abnormal behavior may include: the driver making phone calls, the driver's hands leaving the steering wheel, or other abnormal behaviors of the driver, etc., which are not limited here.

302. Obtain environmental information around the vehicle.

In the embodiment of the present application, one or more second sensors may be deployed on the outside of the vehicle, and the one or more second sensors are used to collect environmental information around the vehicle. The second sensor may be specifically a camera, a radar, or other sensor capable of collecting environmental information around the vehicle, etc. The position of each second sensor may be flexibly set in combination with the actual product form, and is not limited here. The environmental information around the vehicle may be expressed as video information of the environment around the vehicle, or the environmental information around the vehicle may be expressed as at least one set of point cloud data corresponding to at least one moment, each set of point cloud data indicating the environment around the vehicle at a moment, or the environmental information around the vehicle may also be expressed as other types of data, which are not exhaustive here.

303. Determine the driving intention of the vehicle based on the environmental information around the vehicle.

In some embodiments of the present application, the vehicle can determine the driving intention of the vehicle based on the environmental information around the vehicle. The driving intention of the vehicle may include the driving direction of the vehicle and/or one or more objects that the driver needs to pay attention to during driving. Among them, the one or more objects that the driver needs to pay attention to during driving include objects outside the vehicle, such as other vehicles, pedestrians, traffic lights, traffic signs, or other objects around the vehicle; the objects that the driver needs to pay attention to during driving also include objects on the vehicle, such as the rearview mirror inside the vehicle, the rearview mirror on the left front of the vehicle, the rearview mirror on the right front of the vehicle, or other objects on the vehicle, etc. The specific details can be determined in combination with the actual application scenario and are not limited here.

Exemplarily, the vehicle's driving intention may include that the vehicle is going straight at a speed of 50 km/h, and the vehicle in front of the right is about to merge into the lane where the vehicle is located. When the vehicle is going straight, attention should be paid to the vehicle in front of the right. For another example, the vehicle's driving intention may include that the vehicle is turning left at an intersection at a speed of 30 km/h. When the vehicle is turning left, attention should be paid to the vehicle on the left side of the vehicle that is going straight and coming from the opposite side of the intersection. For another example, the vehicle's driving intention may include executing after waiting for the traffic light to turn green at the intersection, and the target that needs to be paid attention to is the traffic light, etc. It should be noted that the examples of "the vehicle's driving intention" here are only for the convenience of understanding this solution and are not used to limit this solution.

Step 303 may include: the vehicle determines the driving behavior of the vehicle based on the surrounding environmental information, and determines one or more objects that the vehicle needs to pay attention to during driving. Specifically, after the vehicle determines at least one object around the vehicle based on the environmental information around the vehicle, it can determine one or more objects that the driver needs to pay most attention to at the current moment based on the driving behavior planned by the vehicle. For a more intuitive understanding of this solution, please refer to Figures 4 to 6, which are three schematic diagrams of objects that the driver needs to pay attention to provided in the embodiments of the present application. First refer to Figure 4, the driving behavior planned by the vehicle is to go straight, and the vehicle determines that there is a vehicle in front of the vehicle based on the surrounding environmental information, and the vehicle on the left side of the vehicle is ready to squeeze in, then the object that the driver needs to pay most attention to at the current moment is the vehicle on the left side of the vehicle.

Please refer to Figure 5 again. The planned driving behavior of the ego vehicle is to turn left. Based on the surrounding environmental information, the ego vehicle determines that there are no pedestrians and non-motor vehicles around the ego vehicle. Therefore, the objects that the driver needs to pay most attention to at the current moment are the traffic lights and the vehicles on the right.

Please refer to Figure 6 again. The vehicle determines that there are pedestrians crossing the road around the vehicle based on the surrounding environmental information. Then the object that the driver needs to pay most attention to at the current moment is the pedestrians crossing the road. It should be understood that the examples in Figures 4 to 6 are only for the convenience of understanding the concept of "one or more objects that the vehicle needs to pay attention to while driving" and are not used to limit this solution.

Optionally, step 303 may also include: when it is determined based on the environmental information around the vehicle that there is no object that needs attention at the current moment, the vehicle may also determine the vehicle's driving direction as the driver's desired direction of sight. For example, when the vehicle is traveling straight on a highway and there are no other objects around the vehicle, it can be considered that there is no object that needs attention, and the vehicle's driving direction can be determined as the driver's desired direction of sight. It should be understood that the examples here are only used to prove the feasibility of this solution and are not used to limit this solution.

It should be noted that the execution order of step 301 and steps 302 to 303 is not limited in the embodiment of the present application. Step 301 may be executed first, Then execute steps 302 to 303; or execute steps 302 to 303 first and then execute step 301; or execute step 301 and steps 302 to 303 simultaneously.

304. Determine whether the driver's line of sight meets the first condition. If the judgment result is yes, proceed to step 305; if the judgment result is no, it can be determined that there is no need to output warning information to the driver. The first condition includes that the driver's line of sight does not match the driving intention of the vehicle.

In the embodiment of the present application, after acquiring the driver's line of sight, the vehicle can determine whether the driver's line of sight meets the first condition. If the judgment result is yes, the process can proceed to step 305; if the judgment result is no, it can be determined that it is not necessary to output warning information to the driver. Optionally, the vehicle can periodically trigger the determination of whether it is necessary to output warning information to the driver, that is, trigger the execution of step 301 every second time period. If the judgment result is no, it can be determined that it is not necessary to output warning information to the driver in the current period.

In one implementation, the length of the second time period may be a preset fixed value, for example, the length of the second time period may be 2 minutes, 5 minutes, 10 minutes or other time lengths. In another implementation, the second time period is variable, and the value factors of the length of the second time period may include any one or more of the following: the current speed of the vehicle, the complexity of the driving behavior planned by the vehicle, the safety of the environment around the vehicle, the safety of the driver's driving behavior or other factors, for example, the slower the current speed of the vehicle, the larger the value of the length of the second time period can be; the faster the current speed of the vehicle, the smaller the value of the second time period can be. The more complex the driving behavior planned by the vehicle, the smaller the value of the second time period can be, and the simpler the driving behavior planned by the vehicle, the larger the value of the second time period can be. The higher the safety of the environment around the vehicle, the larger the value of the second time period can be, and the lower the safety of the environment around the vehicle, the smaller the value of the second time period can be. The higher the safety of the driver's driving behavior, the larger the value of the second time period can be, and the lower the safety of the driver's driving behavior, the smaller the value of the second time period can be. It can be flexibly set in combination with the actual application scenario, and is not limited here.

Among them, the first condition includes that the driver's line of sight does not match the driving intention of the vehicle; the situation that the driver's line of sight does not match the driving intention of the vehicle may include: the driver's line of sight is outside the object that needs attention. In one implementation, the situation that the driver's line of sight is outside the object that needs attention may include: at a first moment, the driver's line of sight is outside the object that needs attention; since at a first moment, the object that needs attention may be one or more, as long as the driver's line of sight is within any one of the one or more objects that need attention, it can be regarded that the driver's line of sight is within the object that needs attention; if the driver's line of sight is outside all objects that need attention, it can be regarded that the driver's line of sight is outside the object that needs attention. Specifically, in one case, step 303 is an optional step. If step 303 is not performed, step 304 may include: the vehicle can obtain second information from the first information; the second information indicates the driver's line of sight at the first moment, and the second information may include the position information of the observation range of the driver's line of sight in the first coordinate system at the first moment. A second neural network is deployed on the vehicle, and the second information and the environmental information around the vehicle at the first moment are input into the second neural network, and the second neural network outputs first prediction information, and the first prediction information indicates whether the driver's line of sight is outside the object that needs attention at the first moment. The second neural network can be a convolutional neural network, a residual neural network, or other types of neural networks, etc.

In another case, if step 303 is executed, and in step 303 the vehicle determines the objects that the driver needs to pay attention to at the first moment in the driving process based on the surrounding environmental information, then step 304 may include: the vehicle may obtain the second information and the third information, the second information may include the position information of the driver's line of sight at the first moment in the first coordinate system, the third information may include the position information of each object that the driver needs to pay attention to at the first moment in the second coordinate system, and the second coordinate system and the first coordinate system may be different coordinate systems or the same coordinate system. For example, the first coordinate system and the second coordinate system are both coordinate systems established with the center of the vehicle as the origin; for another example, the first coordinate system is a coordinate system established with the first camera as the origin, and the second coordinate system is a coordinate system established with the second camera as the origin. It should be noted that the specific coordinate system used by the first coordinate system and the second coordinate system can be flexibly determined in combination with the actual application scenario, and is not limited here.

The vehicle can determine whether the driver's line of sight at the first moment is outside the object that needs attention based on the second information and the third information. Specifically, if the first coordinate system and the second coordinate system are different coordinate systems, in one implementation, the vehicle can input the second information and the third information into a third neural network, and the third neural network outputs second prediction information, and the second prediction information indicates whether the driver's line of sight at the first moment is outside the object that needs attention.

In another implementation, the vehicle may perform a mapping operation based on the second information and/or the third information to map the driver's sight range at the first moment and the object that needs attention at the first moment to the same target coordinate system. The vehicle may determine whether the driver's sight is outside the object that needs attention based on the driver's sight range at the first moment and the object that needs attention at the first moment in the target coordinate system.

Among them, the above-mentioned mapping operation can be to map the position information of the driver's line of sight at the first moment in the first coordinate system to the position information of the driver's line of sight at the first moment in the second coordinate system, that is, the target coordinate system is the second coordinate system; or, map the position information of the object that the driver needs to pay attention to at the first moment in the second coordinate system to the position information of the object that the driver needs to pay attention to at the first moment in the first coordinate system, that is, the target coordinate system is the first coordinate system; or, map the position information of the driver's line of sight at the first moment in the first coordinate system and the position information of the object that the driver needs to pay attention to at the first moment in the second coordinate system to the same third coordinate system, that is, the target coordinate system is the third coordinate system, and so on. The specific mapping operation to be adopted can be flexibly set according to the actual situation and is not limited here.

Specifically, in one implementation, the vehicle can obtain the area (or volume) of the intersection between the area (or volume) of the driver's line of sight at the first moment and the area (or volume) of the object that needs attention at the first moment, and determine whether the area (or volume) of the aforementioned intersection is greater than or equal to the area (or volume) threshold. If the judgment result is yes, it is determined that the driver's line of sight at the first moment is within the object that needs attention; if the judgment result is no, it is determined that the driver's line of sight at the first moment is outside the object that needs attention.

In another implementation, the vehicle can obtain the area (or volume) of the intersection between the area (or volume) of the driver's line of sight at the first moment and the area (or volume) of the object that needs attention at the first moment, and a first ratio between the area (or volume) of the line of sight at the first moment. If the aforementioned first ratio is greater than or equal to a first threshold, it is determined that the driver's line of sight at the first moment is within the object that needs attention; if the first ratio is less than the first threshold, it is determined that the driver's line of sight at the first moment is outside the object that needs attention.

In another implementation, the vehicle can obtain the center of the observation range of the driver's line of sight at the first moment, and T vertices of the object that needs to be paid attention to at the first moment. The vehicle can calculate the angle formed by the above center, the origin and each vertex, and repeat the above operation until T angles corresponding to the T vertices are obtained. If the above multiple angles are all less than the first angle threshold, it is determined that the driver's line of sight at the first moment is within the object that needs attention; if there is at least one angle among the above multiple angles that is greater than the first angle threshold, it is determined that the driver's line of sight at the first moment is outside the object that needs attention.

It should be noted that after the vehicle maps the driver's line of sight at the first moment and the object that needs to be paid attention to at the first moment to the same coordinate system, a variety of methods can be used to determine whether the driver's line of sight at the first moment is outside the object that needs to be paid attention to. The example here is only used to prove the feasibility of this solution and is not used to limit this solution. If the first coordinate system and the second coordinate system are the same coordinate system, the vehicle can determine whether the driver's line of sight is outside the object that needs to be paid attention to based on the driver's line of sight at the first moment and the object that needs to be paid attention to at the first moment under the same coordinate system. The specific implementation method of the above steps can refer to the above description and will not be repeated here.

To understand this solution more intuitively, please refer to FIG. 7a, which is a schematic diagram of the driver's line of sight being within the object that needs attention and the driver's line of sight being outside the object that needs attention provided by the embodiment of the present application. FIG. 7a includes two upper and lower sub-schematic diagrams. The gray elliptical area in FIG. 7a represents the field of vision of the driver's line of sight, and the area in the dotted box represents the object that needs attention. The upper sub-schematic diagram in FIG. 7a represents that the driver's line of sight is within the object that needs attention, and the lower sub-schematic diagram in FIG. 7a represents that the driver's line of sight is outside the object that needs attention. It should be understood that the example in FIG. 7a is only for the convenience of understanding this solution and is not used to limit this solution.

In another implementation, after the vehicle obtains the third information, that is, after obtaining the position information of each object that the driver needs to pay attention to in the second coordinate system at the first moment, the driver's eyes can be used as the location of the virtual camera, and the position information of each object that needs to be paid attention to in the second coordinate system can be mapped to the coordinate system corresponding to the virtual camera, thereby obtaining the position information of the object that needs to be paid attention to in the coordinate system corresponding to the virtual camera. Optionally, the second coordinate system can be a three-dimensional coordinate system, and the coordinate system corresponding to the virtual camera can be a two-dimensional coordinate system. To further understand this solution, the formula used in the above mapping operation can be as follows:

Wherein, _Xw , _Yw and _Zw represent the coordinates of any point in the object of interest (hereinafter referred to as the "target point" for the convenience of description) in the second coordinate system, and u and v represent the coordinates of the target point mapped to the coordinate system corresponding to the virtual camera. represents the intrinsic parameter matrix of the camera, Represents the external parameter matrix of the camera, R represents the rotation matrix, and T represents the translation matrix. It should be understood that the examples here are only for the convenience of understanding this solution and are not used to limit this solution.

To understand the present solution more intuitively, please refer to Figures 7b and 7c, which are two schematic diagrams of mapping objects that require attention in the second coordinate system to the coordinate system corresponding to the virtual camera provided in an embodiment of the present application. In Figure 7b, the object that requires attention in the surrounding environment of the vehicle (that is, the vehicle in Figure 7b) is a three-dimensional vehicle in the second coordinate system. In Figure 7b, the origin of the second coordinate system is taken as the forward-looking sensor of the vehicle as an example; the aforementioned vehicle is mapped to the coordinate system corresponding to the virtual camera, and the virtual camera is determined based on the position of the driver's glasses. The position of each object that requires attention in the coordinate system corresponding to the virtual camera is the expected position of the driver's field of vision. The vehicle can then determine whether the driver's line of sight is outside the object that requires attention based on the position information of each object that requires attention in the coordinate system corresponding to the virtual camera.

Referring to 7c again, FIG. 7c includes two upper and lower sub-schematic diagrams. The upper sub-schematic diagram of FIG. 7c shows the environmental information around the vehicle collected by the forward-looking sensor on the top of the vehicle, that is, the position of the objects around the vehicle in FIG. 7c in the second coordinate system; the lower sub-schematic diagram of FIG. 7c shows the objects that need to be paid attention to around the vehicle are mapped to the position in the coordinate system corresponding to the human eye (that is, the coordinate system corresponding to the virtual camera determined based on the driver's eyes). It should be noted that FIG. 7b and FIG. 7c are examples only for the convenience of understanding the concept of "taking the driver's eyes as the location of the virtual camera, and mapping the position information of each object that needs to be paid attention to in the second coordinate system to the coordinate system corresponding to the virtual camera", and are not used to limit this solution.

The vehicle can determine whether the driver's line of sight at the first moment is outside the object that needs attention based on the position information of each object that needs attention in the coordinate system corresponding to the virtual camera and the position information of the driver's line of sight in the first coordinate system (that is, the second information).

Optionally, the driver's line of sight being outside the object requiring attention may also include: the driver's line of sight at the first moment is within the object requiring attention, and the second movement parameter of the driver's line of sight within the first time period does not match the first movement parameter, and the first time period is after the first moment. Specifically, when the vehicle determines that the driver's line of sight at the first moment is within the object requiring attention, it may also determine the first movement parameter of the object requiring attention within the first time period based on environmental information around the vehicle; obtain the second movement parameter of the driver's line of sight within the first time period; determine whether the first movement parameter and the second movement parameter match, and if the judgment result is yes, determine that the driver's line of sight is within the object requiring attention; if the judgment result is no, determine that the driver's line of sight is outside the object requiring attention.

The first movement parameter may include a first movement direction of the object to be concerned about within the first time period; correspondingly, the second movement parameter may include a second movement direction of the driver's line of sight within the first time period. For example, the first movement direction may be a movement direction of the center of the object to be concerned about within the first time period, and the second movement direction may be a movement direction of the center of the driver's line of sight within the first time period, or the first movement direction may be a movement direction of any point in the object to be concerned about within the first time period, and the second movement direction may be a movement direction of any point within the driver's line of sight within the first time period.

The specific implementation method of the vehicle determining whether the first movement parameter and the second movement parameter match may include: determining whether the angle between the first movement direction and the second movement direction is less than or equal to the first angle threshold, if the determination result is yes, it can be determined that the first movement parameter and the second movement parameter match; if the determination result is no, it can be determined that the first movement parameter and the second movement parameter do not match. For example, the value of the first angle threshold is less than 90 degrees, for example, the value of the first angle threshold can be 30 degrees, 45 degrees, 60 degrees or other values, etc. The examples here are only for the convenience of understanding this solution and are not used to limit this solution.

For a more intuitive understanding of this solution, please refer to FIG8, which is a schematic diagram of the first movement parameter and the second movement parameter provided in the embodiment of the present application. In FIG8, the first movement parameter and the second movement parameter are both movement directions. In FIG8, the movement direction of the object to be concerned in the first time period is to the left, and the movement direction of the driver's line of sight in the first time period is to the upper right. Then, the first movement parameter and the second movement parameter do not match. It should be understood that the example in FIG8 is only for the convenience of understanding this solution and is not used to limit this solution.

Optionally, the first movement parameter may also include any one or more of the following information: a first movement distance, a first movement speed, or other movement information of the object of interest within a first time period, etc., which are not exhaustively listed here. Correspondingly, the second movement parameter may also include any one or more of the following information: a second movement distance, a second movement speed, or other movement information of the driver's line of sight within the first time period, etc., which are not exhaustively listed here.

The specific implementation method of the vehicle determining whether the first movement parameter and the second movement parameter match may include: the vehicle determining whether the angle between the first movement direction and the second movement direction is less than or equal to the first angle threshold, and may also include: the vehicle determining whether the difference between the first movement distance and the second movement distance is less than or equal to the distance threshold, and/or determining whether the difference between the first movement speed and the second movement speed is less than or equal to the speed threshold, and/or determining whether the difference between the object requiring attention and other movement parameters of the driver's line of sight within the first time period is less than the second threshold. If the judgment results of the above judgment operations are all yes, it can be determined that the first movement parameter and the second movement parameter match. The two movement parameters match; if the judgment result of any judgment operation is no, it can be determined that the first movement parameter and the second movement parameter do not match.

In the embodiment of the present application, when considering whether the driver's line of sight is outside the object that needs attention, not only whether the driver's line of sight is outside the object that needs attention in a single first moment is considered, but also whether the movement parameters of the driver's line of sight in a first time period after the first moment are consistent with the movement parameters of the object that needs attention, which is conducive to improving the accuracy of the judgment process of "whether the driver's line of sight is outside the object that needs attention", and is also conducive to improving the safety of the vehicle driving process. In addition, the first movement parameter and the second movement parameter are both determined as the movement direction, which provides an implementation solution that is easy to implement and has high accuracy.

In another implementation, the driver's line of sight being outside the object that needs attention includes: at a first moment and at multiple consecutive moments after the first moment, the driver's line of sight is outside the object that needs attention. Specifically, in one implementation, the vehicle can determine whether the driver's line of sight is outside the object that needs attention for each of the first moment and multiple consecutive moments after the first moment. If the driver's line of sight is outside the object that needs attention at any moment in the first moment and multiple consecutive moments after the first moment, it is determined that the driver's line of sight is outside the object that needs attention; if the driver's line of sight is inside the object that needs attention at each moment in the first moment and multiple consecutive moments after the first moment, it is determined that the driver's line of sight is inside the object that needs attention. The specific implementation method of the vehicle determining whether the driver's line of sight is outside the object that needs attention at any moment can be referred to the above description, which will not be repeated here.

In another implementation, the vehicle obtains third information from the first information, the third information including position information of the driver's line of sight in the first coordinate system at a first moment and multiple consecutive moments after the first moment; a third neural network can be deployed on the vehicle, and the environmental information around the vehicle at the first moment and multiple consecutive moments after the first moment and the third information are input into the third neural network, and the third neural network outputs third prediction information, the third prediction information indicates whether there is any moment in the first moment and multiple consecutive moments after the first moment when the driver's line of sight is outside the object that needs attention, thereby determining whether the driver's line of sight is outside the object that needs attention.

In the embodiment of the present application, it is necessary to determine the driving behavior based on the surrounding objects during the driving of the vehicle, that is, the intelligent driving system and the driver both need to observe the surrounding objects in real time during the driving of the vehicle, and determine that the driver's line of sight is outside the object that needs attention as a situation where the driver's line of sight does not match the driving intention of the vehicle, which is in line with the logic of manual driving, that is, this solution has a high degree of fit with the actual application scenario, which is conducive to accurately determining whether the driver's line of sight matches the driving intention of the vehicle.

Optionally, if step 303 is executed, and in step 303, the vehicle determines that there is no object that the driver needs to pay attention to at the first moment of the driving process according to the surrounding environmental information, the driving direction planned by the vehicle is determined as the expected direction of the driver's line of sight; then step 304 may include: determining whether the direction of the driver's line of sight matches the driving direction of the vehicle; if the judgment result is no, it can be determined that the driver's line of sight meets the first condition, and if the judgment result is yes, it can be determined that the driver's line of sight does not meet the first condition. Optionally, the situation that the direction of the driver's line of sight does not match the driving direction of the vehicle may include that the angle between the driver's line of sight and the driving direction of the vehicle is greater than or equal to the second angle threshold. For example, the value of the second angle threshold may be 45 degrees, 50 degrees or other values. Exemplarily, the direction of vehicle driving is forward, if the direction of the driver's line of sight is to the left, then the vehicle can determine that the direction of the driver's line of sight does not match the driving intention of the vehicle, that is, it is determined that the first condition is met; if the driver's line of sight is forward, then the vehicle can determine that the direction of the driver's line of sight matches the driving intention of the vehicle, that is, it is determined that the first condition is not met. It should be understood that the example here is only for the convenience of understanding this solution and is not used to limit this solution.

Optionally, the first condition may also include abnormal behavior of the driver. In step 304, the vehicle may also determine whether the driver has abnormal behavior based on the information of the driver at at least one time obtained in step 301. If it is determined that the driver has abnormal behavior, it may also trigger to enter step 305, that is, output warning information to the driver. For example, abnormal behavior may include: fatigue driving, making phone calls, or other abnormal behaviors of the driver, etc., which are not exhaustive here.

305. Output warning information.

In the embodiment of the present application, in one implementation, when the vehicle determines that the driver's line of sight does not match the vehicle's driving intention, it can directly trigger to enter step 305, that is, trigger to output warning information to the driver; in another implementation, when the vehicle determines that the driver's line of sight does not match the vehicle's driving intention for a first time, it determines to trigger to enter step 305. The first time can be pre-set in the vehicle, for example, the length of the first time can be 15 seconds, 20 seconds, 25 seconds or other time, etc., which are not exhaustive here.

Optionally, the vehicle may output warning information in at least two ways, and the vehicle may determine which warning method to use based on the accumulated time that the driver's line of sight does not match the vehicle's driving intention. When the cumulative duration of the image mismatch reaches a first duration, it can be determined to use the first type of warning method; when the vehicle determines that the cumulative duration of the mismatch between the driver's line of sight and the vehicle's driving intention reaches a second duration, it can be determined to use the second type of warning method; the value of the second duration can be greater than the first duration.

Optionally, at least two ways in which the vehicle outputs warning information may include: a first type of warning method, a second type of warning method, and a third type of warning method. Correspondingly, when the vehicle determines that the cumulative duration of the mismatch between the driver's line of sight and the vehicle's driving intention reaches a first duration, it may be determined to use the first type of warning method; when the vehicle determines that the cumulative duration of the mismatch between the driver's line of sight and the vehicle's driving intention reaches a second duration, it may be determined to use the second type of warning method; when the vehicle determines that the cumulative duration of the mismatch between the driver's line of sight and the vehicle's driving intention reaches a third duration, it may be determined to use the third type of warning method. The first type of warning method may include visual type warning information, the second type of warning method may include visual type warning information and acoustic type warning information, and the third type of warning method may include visual type warning information, acoustic type warning information, and tactile type warning information.

For example, visual warning information may include: outputting warning information to the driver through a warning light on the dashboard of the vehicle, or outputting warning information to the driver through a head-up display (HUD), or outputting visual warning information through other means, etc., which are not exhaustive here. Tactile warning information may include: tightening the seat belt, vibrating the steering wheel, or other tactile warning information, etc., which are examples here only for the convenience of understanding this solution and are not used to limit this solution.

Optionally, the value of the first duration may be associated with the safety of the driver's driving behavior, and the values of the second duration and/or the third duration may also be associated with the safety of the driver's driving behavior. For example, the higher the safety of the driver's driving behavior, the longer the value of the first duration may be, and the lower the safety of the driver's driving behavior, the shorter the value of the first duration may be. Correspondingly, the higher the safety of the driver's driving behavior, the longer the value of the second duration and/or the third duration may be, and the lower the safety of the driver's driving behavior, the shorter the value of the second duration and/or the third duration may be.

Specifically, the vehicle can determine the safety of the driver's driving behavior based on any one or more of the following driving behavior information: the cumulative number of forward collision warnings (FCW), the number of sudden accelerations, the number of sudden decelerations, the distance to the vehicle in front when following, the speed of steering, the average number of times the vehicle's intelligent driving system takes over the user's driving, or other information that can reflect the safety of the driver's driving behavior, etc., which are not listed exhaustively here.

More specifically, the vehicle may perform dimensionless processing on each type of at least one type of driving behavior information of the driver and then perform weighted summation to obtain a safety index of the driver's driving behavior, which indicates the safety of the driver's driving behavior; optionally, the higher the safety index, the lower the safety of the driver's driving behavior. To more intuitively understand the present solution, the relationship between the safety index of the driver's driving behavior and the first duration, the second duration, and the third duration is shown in Table 1 below.

Table 1

Among them, K1, K2 and K3 in Table 1 represent different safety indexes respectively. For the first row in Table 1, when the safety index is K1, the cumulative time that the driver's line of sight does not match the vehicle's driving intention reaches 20 seconds, triggering the vehicle to output the first type of warning information, the cumulative time that the driver's line of sight does not match the vehicle's driving intention reaches 40 seconds, triggering the vehicle to output the second type of warning information, and the cumulative time that the driver's line of sight does not match the vehicle's driving intention reaches 60 seconds, triggering the vehicle to output the third type of warning information. For the understanding of the second and third rows in Table 1, please refer to the above explanation of the first row of Table 1, which will not be repeated here. It should be understood that the examples in Table 1 are only for the convenience of understanding this solution and are not used to limit this solution.

In an embodiment of the present application, when the duration of the mismatch between the driver's line of sight and the vehicle's driving intention reaches a first duration, a warning message is output to the driver, that is, the value of the first duration can affect the frequency of outputting the warning message, and the safety of the driver's driving behavior will affect the value of the first duration, which is beneficial to reducing the frequency of outputting warning messages to drivers with high safety levels, thereby avoiding disturbing the user; it is also beneficial to increase the frequency of outputting warning messages to drivers with low safety levels, thereby improving the safety of the driving process.

306. Display objects that require attention to the driver.

In some embodiments of the present application, the vehicle can also display objects that need attention to the driver. For example, the vehicle can highlight the objects that the vehicle has determined to need attention to the driver through the HUD; optionally, the vehicle can highlight the objects that need attention when displaying the navigation route to the driver through the HUD. The highlighting method includes any one or more of the following: adding prompt text next to the object that needs attention, framing the object that needs attention, or other methods of highlighting the object that needs attention, etc., which are not exhaustive here. For another example, the vehicle can prompt the user of the objects that need attention in the form of voice playback, etc., and the vehicle can also use other methods to show the user the objects that need attention, which are not exhaustive here. In the embodiment of the present application, the driver can also be shown objects that need attention, so as to assist the driver in learning the driving ideas of the intelligent driving system, which is conducive to improving the safety of the driver's driving behavior.

On the basis of the embodiments corresponding to Figures 1 to 8, in order to better implement the above-mentioned scheme of the embodiment of the present application, the following also provides related equipment for implementing the above-mentioned scheme. Please refer to Figure 9 in detail, which is a structural schematic diagram of a vehicle speed generating device provided in the embodiment of the present application. The vehicle alarm device 900 may include an acquisition module 901 and an alarm module 902, wherein the acquisition module 901 is used to obtain the driver's line of sight and the environmental information around the vehicle; the alarm module 902 is used to output an alarm message when it is determined that the first condition is met according to the driver's line of sight, wherein the first condition includes that the driver's line of sight does not match the vehicle's driving intention, and the vehicle's driving intention is determined based on the environmental information around the vehicle.

Optionally, please refer to Figure 10, which is another structural schematic diagram of the vehicle speed generating device provided in an embodiment of the present application. The vehicle warning device 900 also includes: a processing module 903, which is used to determine the objects that the driver needs to pay attention to during driving according to the environmental information around the vehicle, wherein the driving intention of the vehicle includes the objects that the driver needs to pay attention to during driving, and the situation where the driver's line of sight does not match the driving intention of the vehicle includes: the driver's line of sight is outside the object that needs attention.

Optionally, referring to Figure 10, the processing module 903 is also used to determine the first movement parameter of the object that needs attention within the first time period based on the environmental information around the vehicle, wherein the driver's line of sight is outside the object that needs attention and also includes: the driver's line of sight at the first moment is within the object that needs attention, and the second movement parameter of the driver's line of sight in the first time period does not match the first movement parameter, and the first time period is after the first moment.

Optionally, the first movement parameter includes a first movement direction of the object of interest within a first time period, the second movement parameter includes a second movement direction of the driver's line of sight within the first time period, and the situation where the second movement parameter does not match the first movement parameter includes: the difference between the first movement direction and the second movement direction satisfies the second condition.

Optionally, referring to FIG. 10 , the vehicle warning device 900 further includes: a display module 904 for displaying objects requiring attention to the driver.

Optionally, the warning module 902 is specifically used to output a warning message when the duration of the mismatch between the driver's line of sight and the vehicle's driving intention reaches a first duration, and the value of the first duration is correlated with the safety of the driver's driving behavior.

It should be noted that the information interaction, execution process, etc. between the modules/units in the vehicle alarm device 900 are based on the same concept as the various method embodiments corresponding to Figures 2 to 8 in the present application. The specific contents can be found in the description of the method embodiments shown in the previous part of the present application, and will not be repeated here.

The embodiment of the present application also provides a vehicle. In combination with the above description of FIG. 1 , please refer to FIG. 11 . FIG. 11 is another structural schematic diagram of the vehicle provided in the embodiment of the present application, wherein the vehicle 100 may be deployed with the vehicle alarm device 900 described in the corresponding embodiments of FIG. 9 and FIG. 10 , for realizing the functions of the vehicle in the corresponding embodiments of FIG. 2 to FIG. 8 . Since in some embodiments, the vehicle 100 may also include a communication function, the vehicle 100 may include, in addition to the components shown in FIG. 1 , a receiver 1101 and a transmitter 1102, wherein the processor 113 may include an application processor 1131 and a communication processor 1132. In some embodiments of the present application, the receiver 1101, the transmitter 1102, the processor 113 and the memory 114 may be connected via a bus or other means.

The processor 113 controls the operation of the vehicle. In a specific application, the various components of the vehicle 100 are coupled together through a bus system, wherein the bus system may include a power bus, a control bus, and a status signal bus in addition to a data bus. However, for the sake of clarity, various buses are referred to as bus systems in the figure.

The receiver 1101 can be used to receive input digital or character information and generate signal input related to the relevant settings and function control of the vehicle. The transmitter 1102 can be used to output digital or character information through the first interface; the transmitter 1102 can also be used to send instructions to the disk group through the first interface to modify the data in the disk group; the transmitter 1102 can also include a display device such as a display screen.

In an embodiment of the present application, the application processor 1131 is used to execute the vehicle alarm method executed by the vehicle in the embodiment corresponding to Figure 2. Specifically, the application processor 1131 is used to execute the following steps: obtain the driver's line of sight and the environmental information around the vehicle; when it is determined that the first condition is met based on the driver's line of sight, output an alarm message, wherein the first condition includes that the driver's line of sight does not match the vehicle's driving intention, and the vehicle's driving intention is determined based on the environmental information around the vehicle. It should be noted that for the specific implementation method of the application processor 1131 executing the vehicle alarm method and the beneficial effects brought about, reference can be made to the descriptions in the various method embodiments corresponding to Figures 2 to 8, and they will not be repeated here one by one.

A computer-readable storage medium is also provided in an embodiment of the present application, in which a program for generating a vehicle driving speed is stored. When the program is run on a computer, the computer executes the steps executed by the vehicle in the method described in the embodiments shown in the aforementioned Figures 2 to 8.

Also provided in an embodiment of the present application is a computer program product, which, when executed on a computer, enables the computer to execute the steps executed by the vehicle in the method described in the embodiments shown in the aforementioned Figures 2 to 8.

A circuit system is also provided in an embodiment of the present application, wherein the circuit system includes a processing circuit, and the processing circuit is configured to execute the steps performed by the vehicle in the method described in the embodiments shown in Figures 2 to 8 above.

The vehicle speed generating device or vehicle provided in the embodiment of the present application may be a chip, and the chip includes: a processing unit and a communication unit, wherein the processing unit may be, for example, a processor, and the communication unit may be, for example, an input/output interface, a pin or a circuit, etc. The processing unit may execute the computer execution instructions stored in the storage unit so that the chip in the server executes the vehicle alarm method described in the embodiments shown in Figures 2 to 8 above. Optionally, the storage unit is a storage unit in the chip, such as a register, a cache, etc. The storage unit may also be a storage unit located outside the chip in the wireless access device end, such as a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM), etc.

Specifically, please refer to FIG. 12 , which is a schematic diagram of a structure of a chip provided in an embodiment of the present application, wherein the chip may be a neural network processor NPU 120, which is mounted on the host CPU (Host CPU) as a coprocessor and is assigned tasks by the Host CPU. The core part of the NPU is the operation circuit 120, which controls the operation circuit 1203 through the controller 1205 to extract matrix data in the memory and perform multiplication operations.

In some implementations, the operation circuit 1203 includes multiple processing units (Process Engine, PE) inside. In some implementations, the operation circuit 1203 is a two-dimensional systolic array. The operation circuit 1203 can also be a one-dimensional systolic array or other electronic circuits capable of performing mathematical operations such as multiplication and addition. In some implementations, the operation circuit 1203 is a general-purpose matrix processor.

For example, assume there is an input matrix A, a weight matrix B, and an output matrix C. The operation circuit takes the corresponding data of matrix B from the weight memory 1202 and caches it on each PE in the operation circuit. The operation circuit takes the matrix A data from the input memory 1201 and performs matrix operation with matrix B, and the partial result or final result of the matrix is stored in the accumulator 1208.

The unified memory 1206 is used to store input data and output data. The weight data is directly transferred to the weight memory 1202 through the direct memory access controller (DMAC) 1205. The input data is also transferred to the unified memory 1206 through the DMAC.

BIU stands for Bus Interface Unit, that is, the bus interface unit 1210, which is used for the interaction between AXI bus and DMAC and instruction fetch buffer (IFB) 1209.

The bus interface unit 1210 (Bus Interface Unit, BIU for short) is used for the instruction fetch memory 1209 to obtain instructions from the external memory, and is also used for the storage unit access controller 1205 to obtain the original data of the input matrix A or the weight matrix B from the external memory.

DMAC is mainly used to transfer input data in the external memory DDR to the unified memory 1206 or to transfer weight data to the weight memory 1202 or to transfer input data to the input memory 1201.

The vector calculation unit 1207 includes multiple operation processing units, which further process the output of the operation circuit when necessary, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, etc. It is mainly used for non-convolutional/fully connected layer network calculations in neural networks, such as Batch Normalization, pixel-level summation, upsampling of feature planes, etc.

In some implementations, the vector calculation unit 1207 can store the processed output vector to the unified memory 1206. For example, the vector The calculation unit 1207 can apply a linear function and/or a nonlinear function to the output of the operation circuit 1203, such as linear interpolation of the feature plane extracted by the convolution layer, and then, for example, a vector of accumulated values to generate an activation value. In some implementations, the vector calculation unit 1207 generates a normalized value, a pixel-level summed value, or both. In some implementations, the processed output vector can be used as an activation input to the operation circuit 1203, such as for use in a subsequent layer in a neural network.

An instruction fetch buffer 1209 connected to the controller 1205 is used to store instructions used by the controller 1205;

Unified memory 1206, input memory 1201, weight memory 1202 and instruction fetch memory 1209 are all on-chip memories. External memories are private to the NPU hardware architecture.

Among them, the operations of each layer in the neural network mentioned in each method embodiment shown in Figures 2 to 8 can be performed by the operation circuit 1203 or the vector calculation unit 1207.

The processor mentioned in any of the above places may be a general-purpose central processing unit, a microprocessor, an ASIC, or one or more integrated circuits for controlling the execution of the program of the above-mentioned first aspect method.

It should also be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the drawings of the device embodiments provided by the present application, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines.

Through the description of the above implementation mode, the technicians in the relevant field can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by special hardware including special integrated circuits, special CLUs, special memories, special components, etc. In general, all functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present application, software program implementation is a better implementation mode in more cases. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer floppy disk, U disk, mobile hard disk, ROM, RAM, disk or optical disk, etc., including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, a computer, a server, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server, or data center. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a server or a data center that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.

Claims

A vehicle alarm method, characterized in that the method comprises:

Obtain the driver's line of sight and the environment around the vehicle;

When a first condition is determined to be satisfied according to the driver's line of sight, a warning message is output, wherein the first condition includes that the driver's line of sight does not match the driving intention of the vehicle, and the driving intention of the vehicle is determined based on environmental information around the vehicle.
The method according to claim 1, characterized in that the method further comprises:

Objects that the driver needs to pay attention to during driving are determined based on environmental information around the vehicle, wherein the driving intention of the vehicle includes the objects that the driver needs to pay attention to during driving, and the situation where the driver's line of sight does not match the driving intention of the vehicle includes: the driver's line of sight is outside the objects that need attention.
The method according to claim 2, characterized in that the method further comprises:

Based on the environmental information around the vehicle, a first movement parameter of the object that needs attention within a first time period is determined, wherein the driver's line of sight being outside the object that needs attention also includes: the driver's line of sight at a first moment is within the object that needs attention, and a second movement parameter of the driver's line of sight within the first time period does not match the first movement parameter, and the first time period is located after the first moment.
The method according to claim 3 is characterized in that the first movement parameter includes a first movement direction of the object of concern within the first time period, the second movement parameter includes a second movement direction of the driver's line of sight within the first time period, and the situation where the second movement parameter does not match the first movement parameter includes: the difference between the first movement direction and the second movement direction satisfies a second condition.
The method according to claim 2, characterized in that the method further comprises:

The object requiring attention is displayed to the driver.
The method according to any one of claims 1 to 5, characterized in that, when determining that the first condition is satisfied according to the driver's line of sight, outputting warning information comprises:

When the duration of the mismatch between the driver's line of sight and the vehicle's driving intention reaches a first duration, the warning information is output, and the value of the first duration is correlated with the safety of the driver's driving behavior.
A vehicle warning device, characterized in that the device comprises:

An acquisition module, used to acquire the driver's line of sight and the environmental information around the vehicle;

An alarm module is used to output an alarm message when a first condition is determined to be satisfied according to the driver's line of sight, wherein the first condition includes that the driver's line of sight does not match the driving intention of the vehicle, and the driving intention of the vehicle is determined based on the environmental information around the vehicle.
The device according to claim 7, characterized in that the device further comprises:

A processing module is used to determine objects that the driver needs to pay attention to during driving based on environmental information around the vehicle, wherein the driving intention of the vehicle includes the objects that the driver needs to pay attention to during driving, and the situation where the driver's line of sight does not match the driving intention of the vehicle includes: the driver's line of sight is outside the object that needs attention.
The device according to claim 8, characterized in that

The processing module is further used to determine a first movement parameter of the object that needs attention within a first time period based on environmental information around the vehicle, wherein the driver's line of sight being outside the object that needs attention also includes: the driver's line of sight at a first moment is within the object that needs attention, and a second movement parameter of the driver's line of sight within the first time period does not match the first movement parameter, and the first time period is located after the first moment.
The device according to claim 9 is characterized in that the first movement parameter includes a first movement direction of the object of concern within the first time period, the second movement parameter includes a second movement direction of the driver's line of sight within the first time period, and the situation where the second movement parameter does not match the first movement parameter includes: the difference between the first movement direction and the second movement direction satisfies a second condition.
The device according to claim 8, characterized in that the device further comprises:

The display module is used to display the object requiring attention to the driver.
The device according to any one of claims 7 to 11, characterized in that

The warning module is specifically used to output the warning information when the duration of the mismatch between the driver's line of sight and the vehicle's driving intention reaches a first duration, and the value of the first duration is correlated with the safety of the driver's driving behavior.
A vehicle, characterized in that it comprises a processor, the processor is coupled to a memory, the memory stores program instructions, and when the program instructions stored in the memory are executed by the processor, the method described in any one of claims 1 to 6 is implemented.
A computer-readable storage medium comprises a program, which, when executed on a computer, causes the computer to execute the method according to any one of claims 1 to 6.
A circuit system, characterized in that the circuit system comprises a processing circuit, and the processing circuit is configured to execute the method according to any one of claims 1 to 6.