WO2023178714A1

WO2023178714A1 - Distraction determination method and apparatus, and storage medium, electronic device and vehicle

Info

Publication number: WO2023178714A1
Application number: PCT/CN2022/083706
Authority: WO
Inventors: 龚骏
Original assignee: 北京魔门塔科技有限公司
Priority date: 2022-03-25
Filing date: 2022-03-29
Publication date: 2023-09-28
Also published as: CN114399753A

Abstract

The present application relates to the technical field of automobiles. Disclosed are a distraction determination method and apparatus, and a storage medium, an electronic device and a vehicle, which are used in a distraction determination scenario, and can solve the problem that occurs in distraction determination based on an eye tracker. The method comprises: identifying a line-of-sight direction in a driver image; determining a target gaze area to which the line-of-sight direction belongs; and when a preset distraction condition is met, determining that a driver is in a distraction state, wherein the preset distraction condition comprises at least one of the following: the duration for continuously gazing at a target gaze area, which belongs to a preset distraction area, being greater than or equal to a first preset duration threshold value; a duration, accumulated in a first preset time window, for gazing at a target gaze area which does not belong to the preset concentration area being greater than or equal to a second preset duration threshold value; the duration for continuously gazing at a target gaze area which belongs to another distraction area being greater than or equal to a third preset duration threshold value; and a weighted score for a plurality of target gaze areas that are determined on the basis of a plurality of consecutive frames of driver images being greater than or equal to a preset score threshold value.

Description

Distraction judgment methods, devices, storage media, electronic equipment and vehicles

Cross-references to related applications

This application claims the priority of the Chinese patent application submitted to the China Patent Office on March 25, 2022, with the application number 202210297856.4 and the application name "Distraction Judgment Method, Device, Storage Medium, Electronic Equipment and Vehicle", and its entire content has been approved. This reference is incorporated into this application.

Technical field

The present application relates to the field of automotive technology, specifically, to a distraction judgment method, device, storage medium, electronic equipment and vehicle.

Background technique

With the improvement of living standards and the increasing number of cars, traffic safety problems have become more serious. Distracted driving is a common and potentially risky driving behavior. Distracted driving will cause the driver to lose his attention and leave the road, unable to observe in advance or naturally ignore the information in road traffic. It will also cause changes in driving posture or operating actions, which is not conducive to safe and smooth operation of the vehicle, and can easily cause traffic accidents. ACCIDENT.

In related technologies, the driver's eye movement trajectory characteristics can be recorded based on an eye tracker, and whether the driver is in a distracted state can be determined by analyzing the driver's eye movement trajectory. However, eye trackers are not only expensive and not suitable for widespread use, but the driver's head movements can easily lead to inaccurate data collection, and the driver may even be distracted from driving due to the discomfort of wearing the eye tracker.

Contents of the invention

This application provides a distraction judgment method, device, storage medium, electronic equipment and vehicle, which can solve the problems existing in related technologies of driver distraction judgment based on eye trackers: (1) Expensive and not suitable for wide application , and the driver’s head movement can easily lead to inaccurate data collection; (2) The driver may be distracted by driving due to discomfort with the wearable distraction detection device.

The specific technical solutions are as follows:

In the first aspect, embodiments of the present application provide a distraction judgment method, which includes:

Identify the direction of gaze in the driver's image based on the gaze recognition model;

Determine the target gaze area to which the gaze direction belongs;

When it is determined that the preset distraction conditions are met, it is determined that the driver is in a distracted state;

Wherein, the category of the target gaze area includes at least one of a preset concentration area, a preset distraction area and other distraction areas;

The preset distraction conditions include at least one of the following:

The duration of the driver's continuous gaze on the first target gaze area is greater than or equal to the first preset duration threshold, and the first target gaze area belongs to the preset distraction area;

The cumulative duration of the driver's gaze on the second target gaze area within the first preset time window is greater than or equal to the second preset duration threshold, and the second target gaze area does not belong to the preset concentration area;

The duration of the driver's continuous gaze on the third target gaze area is greater than or equal to the third preset duration threshold, and the third target gaze area belongs to the other distraction areas;

A plurality of weighted scores of the target gaze areas determined based on the driver images of multiple consecutive frames are greater than or equal to a preset score threshold.

It can be seen from the above solution that compared with related technologies that require the use of wearable distraction detection equipment, such as eye trackers, to determine distraction, the embodiment of the present application can first identify the direction of the driver's gaze in the image based on the gaze recognition model, Then determine the target gaze area to which the gaze direction belongs, and finally determine that the driver is distracted when the preset distraction conditions are met (distraction conditions set based on the duration of gazing at the target gaze area and/or weighted scores, etc.) state. It can be seen from this that the embodiment of the present application combines the inward-looking camera necessary for intelligent driving with the distraction algorithm of the software to make distraction judgments without requiring the driver to wear additional equipment, thereby not only not affecting the driver's driving comfort degree, and the accuracy of distraction judgment will not be reduced due to inaccurate data collection.

In a first possible implementation manner of the first aspect, determining the target gaze area to which the line of sight direction belongs includes:

Calculate respectively whether the line of sight direction intersects with each of the plurality of preset gaze areas, where the preset gaze areas include a preset concentration area and/or a preset distraction area;

When the line of sight direction intersects only one preset gaze area, the intersecting preset gaze area is determined as the target gaze area; or,

When the line of sight direction intersects with at least two preset gaze areas, select the preset gaze area with the highest priority from the at least two preset gaze areas according to the preset area priority, and assign the preset gaze area with the highest priority to The area is determined as the target gaze area; or,

When the gaze direction represents that the driver is in an eyes-open state and the gaze direction does not intersect with multiple preset gaze areas, or the gaze direction represents that the driver is in a closed-eyes state, determine that the gaze area to which the gaze direction belongs is other distractions area, and determine other distracting areas as target fixation areas.

It can be seen from the above solution that the embodiment of the present application can not only determine whether the line of sight direction intersects with each of the plurality of preset attention areas, but also determine the target gaze area in the line of sight direction and at least two preset gaze areas. If the gaze areas intersect, the preset gaze area that the driver is really looking at can be accurately and quickly determined based on the preset region priority. It can be understood that in the embodiment of the present application, when the line of sight direction intersects two preset gaze areas, the preset gaze area with the highest priority is the preset gaze area with higher priority among the two preset gaze areas. .

In a second possible implementation of the first aspect, identifying the gaze direction in the driver's image based on the gaze recognition model includes:

Based on the sight recognition model, the driver's image is recognized and the sight direction in the camera coordinate system is obtained.

In a third possible implementation of the first aspect, separately calculating whether the line of sight direction intersects with each of the plurality of preset gaze areas includes:

When the external parameter calibration of the internal camera is valid, the line of sight direction is converted from the camera coordinate system to the body coordinate system, and the line of sight direction in the body coordinate system and the multiple preset gaze areas in the body coordinate system are calculated respectively. Whether each preset gaze area intersects.

In the fourth possible implementation manner of the first aspect, the method further includes:

When the external parameter calibration of the internal camera fails, calculate the angle between the line of sight direction and the line of sight reference vector in the camera coordinate system;

When the included angle is continuously greater than or equal to the preset included angle threshold, the duration is greater than or equal to the fourth preset included angle threshold, or, within the second preset time window, the cumulative duration for which the included angle is greater than or equal to the preset included angle threshold is greater than or equal to the preset included angle threshold. When equal to the fifth preset duration threshold, it is determined that the driver is in a distracted state.

It can be seen from the above solution that when the external parameter calibration of the internal view camera fails, distraction judgment can be made by using the angle between the line of sight direction and the line of sight reference vector in the camera coordinate system that does not rely on the external parameter calibration, thereby avoiding distraction. Interruption of distraction judgments due to inability to determine the target fixation area.

In the fifth possible implementation of the first aspect, before calculating the angle between the line of sight direction in the camera coordinate system and the line of sight reference vector, the method further includes:

Calculate the variance of N historical sight directions based on N consecutive frames of historical driver image recognition, where N is a positive integer;

The average of the N historical sight directions with the smallest variance is selected as the sight reference vector, or any historical sight direction among the N historical sight directions with the smallest variance is selected as the sight reference vector.

In the sixth possible implementation manner of the first aspect, the method further includes:

When the external parameter calibration of the internal camera fails, the driver's head angle in the driver image is recognized based on the head angle recognition model;

The duration during which the head rotation angle is continuously greater than or equal to the preset angle threshold is greater than or equal to the sixth preset duration threshold, or, within the third preset time window, the cumulative duration during which the head rotation angle is greater than or equal to the preset angle threshold is greater than or equal to the preset angle threshold. When equal to the seventh preset duration threshold, it is determined that the driver is in a distracted state.

It can be seen from the above solution that when the external parameter calibration of the internal-view camera fails, distraction judgment can be made through the head rotation angle in the camera coordinate system that does not rely on external parameter calibration, thereby avoiding the inability to determine the target gaze area and causing distraction. Mental judgment is interrupted.

In a seventh possible implementation manner of the first aspect, before determining that the driver is in a distracted state when it is determined that the preset distraction condition is met, the method further includes:

Determine whether the vehicle meets the preset preconditions;

When the vehicle meets the preset preconditions, it is executed to determine that the driver is in a distracted state when it is judged that the preset distraction conditions are met;

If the vehicle does not meet the preset preconditions, it will not be executed. If it is determined that the preset distraction conditions are met, it will be determined that the driver is in a distracted state;

Among them, the preset preconditions include at least one of the following:

The internal camera is in an unobstructed state;

Vehicle speed is greater than 0;

The vehicle is in a non-traffic jam state;

The vehicle is not in special condition;

The special status includes: when the vehicle is in a preset scene and the driver is in a non-fatigued state, the body posture is the same as the preset steering and the vehicle speed is within the preset safe range. The preset scene includes at least 100% in steering, parking, and reversing. One item.

It can be seen from the above solution that before judging whether the preset distraction conditions are met, the preset preconditions are used to exclude special scenarios in which whether the driver is distracted and has nothing to do with vehicle safety. That is, the analysis will only be performed when the preset preconditions are met. It can save resources and improve the accuracy of distracted judgment.

In the eighth possible implementation manner of the first aspect, the method further includes:

When the gaze direction indicates that the driver is in a closed-eyes state, and the driver's continuous eye-closure duration is greater than or equal to the eighth preset duration threshold, the driving fatigue prompt information is output, and the target statistical information for distraction judgment is cleared, where, Goal statistics include timing information and/or weighted scoring information.

It can be seen from the above solution that the embodiments of the present application can not only detect fatigue driving and output driving fatigue prompt information, but also distinguish fatigue driving from distracted driving. When the driver is driving fatigue, clear the target statistics for distraction judgment. information, so that different states can be analyzed and managed in a targeted manner.

In the ninth possible implementation manner of the first aspect, the method further includes:

When the gaze direction indicates that the driver's eyes are closed, and the continuous closing duration of both eyes is greater than or equal to the ninth preset duration threshold and then one eye is opened, the target statistical information is re-calculated;

Or, when the sight direction indicates that the driver is in a closed state of one eye, and the continuous closing duration of one eye is greater than or equal to the ninth preset duration threshold, the target statistical information is re-calculated.

It can be seen from the above solution that the embodiment of the present application can detect the driver's high-frequency blinking, and in this case, re-calculate the target statistical information, thereby eliminating the impact of high-frequency blinking on distraction judgment.

In a tenth possible implementation manner of the first aspect, the preset distraction area corresponds to at least one distraction level, and the distraction level of the preset distraction area has a negative correlation with the first preset duration threshold.

It can be seen from the above solution that the embodiment of the present application divides the preset distraction area into at least one distraction level, so that distraction judgments can be made in a more fine-grained manner based on the gaze conditions of the preset distraction areas at different distraction levels, thereby improving the accuracy of the analysis. The accuracy of mental judgment.

In an eleventh possible implementation manner of the first aspect, the preset distraction areas corresponding to the first distraction level include the interior rearview mirror, central control screen, left side window, left rearview mirror, right side At least one of the car window, the right rearview mirror, and the control panel, the preset distraction area corresponding to the second distraction level includes the left door and/or the right door, and the preset concentration area includes the front windshield.

In the second aspect, embodiments of the present application provide a distraction judgment device, which includes:

A recognition unit for identifying the gaze direction in the driver's image based on the gaze recognition model;

The area determination unit is used to determine the target gaze area to which the line of sight direction belongs;

A distraction determination unit, used to determine that the driver is in a distracted state when it is judged that the preset distraction conditions are met;

Preset distraction conditions include at least one of the following:

The driver's continuous gaze on the third target gaze area is greater than or equal to the third preset duration threshold, and the third target gaze area belongs to other distraction areas;

The weighted scores of multiple target gaze areas determined based on multiple consecutive frames of driver images are greater than or equal to a preset score threshold.

In a first possible implementation manner of the second aspect, the area determining unit includes:

a calculation module, configured to respectively calculate whether the line of sight direction intersects with each of the plurality of preset gaze areas, wherein the preset gaze areas include a preset concentration area and/or a preset distraction area;

A determination module configured to determine the intersecting preset gaze area as the target gaze area when the line of sight direction intersects with only one preset gaze area; or, when the line of sight direction intersects with at least two preset gaze areas , select the highest priority preset gaze area from at least two preset gaze areas according to the preset area priority, and determine the highest priority preset gaze area as the target gaze area; or, characterize the driver in the line of sight direction When the driver is in the eyes-open state and the gaze direction does not intersect with multiple preset gaze areas, or the gaze direction indicates that the driver is in the eyes-closed state, the gaze area to which the gaze direction belongs is determined to be other distraction areas, and the other distraction areas are classified. The heart area is determined as the target fixation area.

In a second possible implementation of the second aspect, the recognition unit is used to recognize the driver's image based on the sight recognition model and obtain the sight direction in the camera coordinate system.

In a third possible implementation of the second aspect, the calculation module is used to convert the line of sight direction from the camera coordinate system to the body coordinate system when the external parameter calibration of the internal view camera is valid, and calculate the body coordinates respectively. Whether the line of sight direction under the system intersects with each of the multiple preset gaze areas under the body coordinate system.

In a fourth possible implementation manner of the second aspect, the device further includes:

A calculation unit used to calculate the angle between the line of sight direction and the line of sight reference vector in the camera coordinate system when the external parameter calibration of the internal view camera fails;

The distraction determination unit is also configured to be greater than or equal to the fourth preset duration threshold when the included angle is continuously greater than or equal to the preset included angle threshold, or, within the second preset time window, the included angle is greater than or equal to the preset When the cumulative duration of the angle threshold is greater than or equal to the fifth preset duration threshold, it is determined that the driver is in a distracted state.

In a fifth possible implementation manner of the second aspect, the calculation unit is also used to calculate the historical driver image recognition based on N consecutive frames before calculating the angle between the line of sight direction and the line of sight reference vector in the camera coordinate system. The variance of N historical sight directions, where N is a positive integer;

The installation also includes:

The selection unit is used to select the average of N historical sight directions with the smallest variance as the sight reference vector, or select any historical sight direction among the N historical sight directions with the smallest variance as the sight reference vector.

In a sixth possible implementation manner of the second aspect, the recognition unit is also used to identify the driver's head rotation angle in the driver image based on the head rotation angle recognition model when the external parameter calibration of the internal view camera fails. ;

The distraction determination unit is also configured to be greater than or equal to the sixth preset duration threshold when the head rotation angle is continuously greater than or equal to the preset angle threshold, or, within the third preset time window, the head rotation angle is greater than or equal to the preset angle threshold. If the cumulative duration of the angle threshold is greater than or equal to the seventh preset duration threshold, it is determined that the driver is in a distracted state.

In a seventh possible implementation manner of the second aspect, the device further includes:

A judgment unit, used to judge whether the vehicle meets the preset preconditions before determining that the driver is in a distracted state when it is judged that the preset distraction conditions are met;

The distraction determination unit is also used to determine that the driver is in a distracted state when the vehicle meets the preset preconditions; when the vehicle does not meet the preset preconditions, it determines that the driver is in a distracted state. In the case where it is determined that the driver is in a distracted state, it is not determined that the preset distraction conditions are met;

Among them, the preset preconditions include at least one of the following:

The internal camera is in an unobstructed state;

Vehicle speed is greater than 0;

The vehicle is in a non-traffic jam state;

The vehicle is not in special condition;

In an eighth possible implementation manner of the second aspect, the device further includes:

The output unit is used to output driving fatigue prompt information when the direction of vision indicates that the driver is in a closed-eyes state, and the driver's continuous eye-closure duration is greater than or equal to the eighth preset duration threshold;

A clearing unit configured to clear target statistical information for distraction judgment, where the target statistical information includes timing information and/or weighted score information.

In a ninth possible implementation manner of the second aspect, the device further includes:

The statistical unit is used to represent the driver's eyes in a closed state in the direction of the gaze, and when the continuous closing of both eyes is greater than or equal to the ninth preset duration threshold and then one eye is opened, re-statistical target statistical information; or, to represent the driver in the direction of the gaze. When the single eye is in a closed state and the continuous closing duration of the single eye is greater than or equal to the ninth preset duration threshold, the target statistical information is re-calculated.

In a tenth possible implementation manner of the second aspect, the preset distraction area corresponds to at least one distraction level, and the distraction level of the preset distraction area has a negative correlation with the first preset duration threshold.

In an eleventh possible implementation manner of the second aspect, the preset distraction areas corresponding to the first distraction level include the interior rearview mirror, the central control screen, the left window, the left rearview mirror, the right side At least one of the car window, the right rearview mirror, and the control panel, the preset distraction area corresponding to the second distraction level includes the left door and/or the right door, and the preset concentration area includes the front windshield.

In a third aspect, embodiments of the present application provide a storage medium on which a computer program is stored. When the program is executed by a processor, the method described in any possible implementation manner of the first aspect is implemented.

In a fourth aspect, embodiments of the present application provide an electronic device. The electronic device includes:

one or more processors;

a storage device for storing one or more programs,

When one or more programs are executed by one or more processors, the electronic device is caused to implement the method described in any possible implementation manner of the first aspect.

In a fifth aspect, embodiments of the present application provide a vehicle, which includes the device as described in any possible implementation of the second aspect, or the electronic device as described in the fourth aspect.

Description of the drawings

In order to more clearly explain the embodiments of the present application or the technical solutions in the prior art, the following will briefly introduce the drawings that need to be used in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can also be obtained based on these drawings without exerting creative efforts.

Figure 1 is a schematic flowchart of a distraction judgment method provided by an embodiment of the present application;

Figure 2 is a schematic flow chart of another distraction judgment method provided by an embodiment of the present application;

Figure 3 is a schematic flowchart of yet another distraction judgment method provided by an embodiment of the present application;

Figure 4 is a schematic flowchart of yet another distraction judgment method provided by an embodiment of the present application;

Figure 5 is a schematic flowchart of yet another distraction judgment method provided by an embodiment of the present application;

Figure 6 is a block diagram of a distraction judgment device provided by an embodiment of the present application;

Figure 7 is a schematic diagram of a vehicle architecture provided by an embodiment of the present application;

Detailed ways

The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the scope of protection of this application.

It should be noted that the terms "including" and "having" and any variations thereof in the embodiments and drawings of this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally also includes steps or units that are not listed, or optionally also includes Other steps or units inherent to such processes, methods, products or devices.

Figure 1 is a schematic flow chart of a distraction judgment method. This method can be applied to vehicles or servers. The method can include the following steps:

S110: Identify the gaze direction in the driver's image based on the gaze recognition model.

While the vehicle is driving, the internal camera installed inside the vehicle can collect the driver's image and transmit the collected driver's image to the electronic device for distraction judgment, so that the electronic device can extract the eye part from the driver's image. Features, and input the eye features into the gaze recognition model to identify the gaze direction. Among them, the electronic devices used for distraction judgment include ECU (Electronic Control Unit, Electronic Control Unit) in the vehicle and other controllers or servers used for distraction judgment. The gaze recognition model can be trained based on the eye features of a large number of driver images collected in advance, and the gaze recognition model can be a linear regression model or other models.

In addition, this step recognizes the driver's image based on the sight recognition model, and obtains the sight direction in the camera coordinate system.

S120: Determine the target gaze area to which the sight direction belongs.

Wherein, the category of the target gaze area includes at least one of a preset concentration area, a preset distraction area and other distraction areas. The preset concentration area is the gaze area where the driver is definitely driving attentively. The preset distraction area is the preset gaze area where the driver may be distracted. The other distraction areas are the unknown gaze where the driver may be distracted. area.

The default concentration area includes the front windshield, and the default distraction area includes the interior rearview mirror, central control screen, left window, left rearview mirror, right side window, right rearview mirror, control panel, and left door and at least one item in the right door, and other distraction areas are gaze areas other than the preset concentration area and the preset distraction area.

In one implementation, in order to improve the accuracy of subsequent distraction judgments, the preset distraction area can be divided into at least one distraction level, and different distraction judgment criteria can be configured for different distraction levels. Specifically, the preset distraction area corresponds to at least one distraction level, and the distraction level of the preset distraction area is negatively correlated with the first preset duration threshold. The preset distraction area corresponding to the first distraction level includes at least one of the interior rearview mirror, central control screen, left window, left rearview mirror, right window, right rearview mirror, and control panel. The preset distraction area corresponding to the second distraction level includes the left door and/or the right door, and the preset concentration area includes the front windshield. For details about the first preset duration threshold, please refer to the detailed explanation of S130 and will not be described again here.

It should be added that the target gaze area in the embodiment of the present application is not limited to the physical area in the vehicle, and can also be a logical area. A logical area can include one or more complete physical areas, or partial areas of one or more physical areas. For example, three logical areas can be divided into left, middle and right based on the driver looking straight ahead. , where the middle area is the preset distraction area, the left and right areas are the preset distraction areas, and the remaining areas are other distraction areas.

The specific implementation process of step S120 is introduced in detail below, as shown in Figure 2. The process includes:

S121: Calculate whether the line of sight direction intersects with each of the plurality of preset gaze areas.

The preset attention area includes a preset concentration area and/or a preset distraction area.

When the external parameter calibration of the internal camera is valid, the line of sight direction is converted from the camera coordinate system to the body coordinate system, and the line of sight direction in the body coordinate system and the multiple preset gaze areas in the body coordinate system are calculated respectively. Whether each preset gaze area intersects. In addition, you can also convert multiple preset gaze areas in the body coordinate system from the body coordinate system to the camera coordinate system, and then calculate the line of sight direction in the camera coordinate system and the multiple preset gaze areas in the camera coordinate system respectively. Whether each preset gaze area intersects. In other words, just convert the line of sight direction and the preset gaze area to the same coordinate system for intersection calculation.

S122: When the line of sight direction intersects only one preset gaze area, determine the intersecting preset gaze area as the target gaze area.

S123: When the sight direction intersects with at least two preset gaze areas, select the preset gaze area with the highest priority from the at least two preset gaze areas according to the priority of the preset area, and add the preset gaze area with the highest priority to the preset gaze area. Let the gaze area be determined as the target gaze area. It can be understood that in the embodiment of the present application, when the line of sight direction intersects two preset gaze areas, the preset gaze area with the highest priority is the preset gaze area with higher priority among the two preset gaze areas. .

When the driver looks at a certain preset gaze area, the line of sight may pass through other preset gaze areas and intersect with at least two preset gaze areas. For example, when the driver looks at the right rearview mirror, The view will be through the right side window. In order to solve this problem, a configuration file including the priority of each preset gaze area (ie, the preset area priority) can be set in the electronic device in advance. When the line of sight direction intersects with at least two preset gaze areas, the configuration file can be set from Read the preset area priority from the configuration file, determine the preset gaze area with the highest priority among at least two preset gaze areas based on the preset area priority, and determine the preset gaze area with the highest priority as the target gaze. area. Among them, the preset area priority can be determined based on experimental experience, for example, right rearview mirror > right window.

S124: When the gaze direction represents that the driver is in an eyes-open state and the gaze direction does not intersect with multiple preset gaze areas, or the gaze direction represents that the driver is in a closed-eyes state, determine that the gaze area to which the gaze direction belongs is other Distracted areas, and other distracted areas are determined as target fixation areas.

When the driver is not looking at any of the above preset gaze areas, but the driver is in the eyes-open state, the driver's gaze direction can be divided into other distraction areas. When the driver is in the eyes-closed state, it can also be explained that the driver is not paying attention. Focus on any of the above preset gaze areas, so the driver's gaze direction can also be divided into other distraction areas.

It can be seen from S121 to S124 that the embodiment of the present application can not only determine the target gaze area by determining whether the line of sight direction intersects with each of the plurality of preset gaze areas, but also determine the target gaze area in the line of sight direction with at least two preset gaze areas. When the preset gaze areas intersect, the preset gaze area that the driver is really looking at is determined accurately and quickly based on the priority of the preset area.

S130: When it is determined that the preset distraction conditions are met, it is determined that the driver is in a distracted state.

Preset distraction conditions include at least one of the following:

The meaning of continuous gaze includes not only literal continuous gaze, but also includes the total duration of gaze actions exceeding a preset value within a predetermined time period. In addition to literal continuous frames, the meaning of continuous multiple frames can also include that the total number of frames of driver images collected exceeds a preset value within a predetermined time period. The driver's continuous gaze on the first target gaze area includes the driver's continuous gaze on the same first target gaze area; the driver's cumulative gaze on the second target gaze area within the first preset time window includes the driver's gaze on the second target gaze area within the first preset time window. The cumulative gaze on at least one second target gaze area; the driver's continuous gaze on the third target gaze area includes the driver's continuous gaze on at least one third target gaze area.

The first preset duration threshold, the second preset duration threshold, the third preset duration threshold, the first preset time window and the preset score threshold are actual empirical values. For example, the first preset duration threshold can be 2.5s, The second preset duration threshold may be 10s, the third preset duration threshold may be 2s, the first preset time window may be 30s, and the preset score threshold may be 80. When the preset distraction area includes at least one distraction level, the distraction level of the preset distraction area is negatively correlated with the first preset duration threshold, that is, the higher the distraction level, the smaller the first preset duration threshold. , for example, the first preset duration threshold corresponding to the first distraction level is 2.5s, and the first preset duration threshold corresponding to the second distraction level is 2s.

In addition, when performing weighted calculations on multiple target gaze areas, the weight of each target gaze area involved can be distributed according to the safety factor of different target gaze areas, and the weight is an actual empirical value. For example, when assigning weights, the weight of the preset concentration area < the weight of the preset distraction area of the first distraction level < the weight of the preset distraction area of the second distraction level < the weight of other distraction areas.

By way of example, the preset distraction conditions include at least one of the following:

The driver continuously looks at the same preset distraction area of the first distraction level for longer than or equal to 2.5 seconds;

The driver continuously looks at the same preset distraction area of the second distraction level for longer than or equal to 2 seconds;

The cumulative duration of the driver's gaze on non-preset concentration areas (i.e. areas that do not belong to the preset concentration areas) within 30 seconds is greater than or equal to 10 seconds;

The driver continuously looks at other distracting areas for more than or equal to 2 seconds;

The weighted scores of multiple target gaze areas determined based on multiple consecutive frames of driver images are greater than or equal to 80 points.

It should be added that when the driver is distracted, distraction reminder information is output. The distraction reminder information can be output in text form or voice form, and can be output on the vehicle (such as the central control screen, Based on HUD (Head Up Display, head-up display) output on the front windshield), it can also be output on the user's mobile terminal (such as a mobile phone). When outputting on the user's mobile terminal, the electronic device that needs to be used for distraction judgment generates the distraction reminder information, and then sends the distraction reminder information to the user's mobile terminal.

Compared with related technologies that require the use of wearable distraction detection devices, such as eye trackers, for distraction judgment, the distraction judgment method provided by the embodiments of the present application can first identify the line of sight in the driver's image based on the line of sight recognition model. direction, and then determine the target gaze area to which the gaze direction belongs. Finally, when the preset distraction conditions are met (distraction conditions set based on the duration of gazing at the target gaze area and/or weighted scores, etc.), it is determined that the driver is in state of distraction. It can be seen from this that the embodiment of the present application combines the inward-looking camera necessary for intelligent driving with the distraction algorithm of the software to make distraction judgments without requiring the driver to wear additional equipment, thereby not only not affecting the driver's driving comfort degree, and the accuracy of distraction judgment will not be reduced due to inaccurate data collection.

Based on the above method embodiment, another embodiment of the present application also provides a distraction judgment method, as shown in Figure 3, the method includes:

S210: Identify the gaze direction in the driver's image based on the gaze recognition model.

The internal camera includes extrinsic parameter calibration and intrinsic parameter calibration. The calculation in the camera coordinate system can be determined based on the intrinsic parameter calibration, while the rotation and translation from the camera coordinate system to the body coordinate system need to rely on the extrinsic parameter calibration. Therefore, when the external parameter calibration is valid, the method described in S220-S230 can be used to perform distraction judgment; when the external parameter calibration is invalid, the method described in S240-S250 can be used to perform the distraction judgment.

S220: When the external parameter calibration of the inner-view camera is valid, determine the target gaze area to which the line of sight direction belongs.

S230: When it is determined that the preset distraction conditions are met, it is determined that the driver is in a distracted state.

S240: When the external parameter calibration of the internal view camera fails, calculate the angle between the line of sight direction and the line of sight reference vector in the camera coordinate system.

Among them, the calculation process of the gaze reference vector includes: calculating the variance of N historical gaze directions based on the recognition of N consecutive frames of historical driver images, where N is a positive integer; selecting the average of the N historical gaze directions with the smallest variance as the gaze Reference vector, or select any historical line of sight direction among the N historical line of sight directions with the smallest variance as the line of sight reference vector. The value of N can be determined based on actual experience, for example, it can be 10.

It should be noted that all N consecutive frames of historical driver images calculated in different batches can be different, or some frames can be different and some frames can be the same.

S250: When the included angle is continuously greater than or equal to the preset included angle threshold, the duration is greater than or equal to the fourth preset included angle threshold, or, within the second preset time window, the included angle is greater than or equal to the cumulative duration of the preset included angle threshold. If it is greater than or equal to the fifth preset duration threshold, it is determined that the driver is in a distracted state.

Among them, the preset angle threshold, the fourth preset duration threshold, the second preset time window and the fifth preset duration threshold can be determined based on actual experience. For example, the preset angle threshold can be 20 degrees, and the fourth preset duration The threshold may be 2.5s, the second preset time window may be 28s, and the fifth preset duration threshold may be 8s.

The distraction judgment method provided by the embodiment of the present application can perform distraction judgment based on the target gaze area to which the driver's line of sight direction belongs when the internal camera's external parameter calibration is valid, and when the internal camera's external parameter calibration fails Distraction judgment is made by using the angle between the line of sight direction and the line of sight reference vector in the camera coordinate system that does not rely on external parameter calibration. This not only eliminates the need for the driver to wear additional equipment, but also avoids the inability to determine the target gaze area. And lead to distraction and interruption of judgment.

Based on the above method embodiment, another embodiment of the present application also provides a distraction judgment method, as shown in Figure 4. This method can use S310-S330 when the external parameter calibration of the internal camera is valid. When the external parameter calibration of the internal camera is valid, the distraction judgment method described in S340-S350 is used. The specific process includes:

S310: Determine whether the external parameter calibration of the internal camera is valid.

S320: When the external parameter calibration of the internal view camera is valid, identify the gaze direction in the driver's image based on the gaze recognition model.

S330: Determine the target gaze area to which the sight direction belongs.

S340: When it is determined that the preset distraction conditions are met, it is determined that the driver is in a distracted state.

S350: When the external parameter calibration of the internal camera fails, identify the driver's head angle in the driver image based on the head angle recognition model.

The electronic device extracts head features from the driver's image and inputs the head features into the head angle recognition model to identify the head angle. Among them, the head angle recognition model can be trained based on the head features of a large number of driver images collected in advance, and the head angle recognition model can be a linear regression model or other models.

S360: The duration in which the head rotation angle is continuously greater than or equal to the preset angle threshold is greater than or equal to the sixth preset duration threshold, or, within the third preset time window, the cumulative duration in which the head rotation angle is greater than or equal to the preset angle threshold If it is greater than or equal to the seventh preset duration threshold, it is determined that the driver is in a distracted state.

The preset angle threshold, the sixth preset duration threshold, the third preset time window and the seventh preset duration threshold can be determined based on actual experience. For example, the preset angle threshold can be 45 degrees, and the sixth preset duration threshold can be 2.5 s, the third preset time window may be 29s, and the seventh preset duration threshold may be 9s.

The distraction judgment method provided by the embodiment of the present application can perform distraction judgment based on the target gaze area to which the driver's line of sight direction belongs when the internal camera's external parameter calibration is valid, and when the internal camera's external parameter calibration fails Under this method, distraction judgment is made by not relying on the head rotation angle in the camera coordinate system calibrated by external parameters. This not only eliminates the need for the driver to wear additional equipment, but also avoids the interruption of distraction judgment due to the inability to determine the target gaze area.

Based on the above method embodiment, another embodiment of the present application also provides a distraction judgment method, as shown in Figure 5, the method includes:

S410: Determine whether the vehicle meets the preset preconditions. If the vehicle meets the preset preconditions, step S420 is executed; otherwise, step S410 is continued.

Preset preconditions can be used to exclude special scenarios in which driver distraction has nothing to do with vehicle safety, thereby saving resources and improving distraction judgment accuracy.

Among them, the preset preconditions include at least one of the following:

The internal camera is in an unobstructed state;

Vehicle speed is greater than 0;

The vehicle is in a non-traffic jam state;

The vehicle is not in special condition;

The methods used in the embodiments of this application to detect whether the driver is in a fatigue state include but are not limited to the following: (1) The duration of continuous eye closure is greater than a certain threshold; (2) Drowsy Driver Detection System based on DDS (The Drowsy Driver Detection System) ) Carry out fatigue detection, that is, scan the pupil condition through radar and determine the driver's fatigue state through data analysis; (3) By detecting the driver's heart rate variability, and determine the driver's fatigue state based on the heart rate variability.

The preset steering includes left turn, right turn, and straight drive, and the body posture also includes left turn, right turn, and straight drive. When the driver is in a non-fatigued state, the average speed of the vehicle in the recent period can be used as the preset safety range, and the preset safety ranges of different vehicles can be different in different time periods.

S420: Identify the gaze direction in the driver's image based on the gaze recognition model.

S430: Determine the target gaze area to which the sight direction belongs.

S440: When it is determined that the preset distraction conditions are met, determine that the driver is in a distracted state.

It should be added that the embodiment of the present application starts with the execution timing of "judging whether the vehicle satisfies the preset preconditions" as an example. In fact, the execution timing of "judging whether the vehicle satisfies the preset preconditions" only needs to be "in When it is determined that the preset distraction conditions are met, it can be any time before it is determined that the driver is in a distracted state. That is, the execution timing of S410 can be before, after, or even at the same time as S420 (or S430).

The distraction determination method provided by the embodiments of the present application first uses the preset preconditions to exclude special scenarios in which whether the driver is distracted and has nothing to do with vehicle safety before determining whether the preset distraction conditions are met, that is, only if the preset preconditions are met. Distraction judgment will only be made when conditions exist, which can save resources and improve the accuracy of distraction judgment.

In one implementation, in order to distinguish fatigue driving from distracted driving, so as to conduct targeted analysis and management of different states, embodiments of the present application can also represent that the driver is in a closed-eyes state in the direction of the line of sight, and the driving state is When the driver's continuous eye-closure duration is greater than or equal to the eighth preset duration threshold, driving fatigue prompt information is output, and target statistical information for distraction judgment is cleared, where the target statistical information includes timing information and/or weighted score information.

The eighth preset duration threshold can be determined based on actual experience, and can be, for example, 2.5 seconds.

In one implementation, in order to eliminate the impact of high-frequency blinking on distraction judgment, thereby improving the accuracy of distraction judgment, the following solution may be adopted in the embodiment of the present application:

When the gaze direction indicates that the driver's eyes are closed, and the continuous closing duration of both eyes is greater than or equal to the ninth preset duration threshold and then one eye is opened, the target statistical information is re-calculated; or,

When the sight direction indicates that the driver is in a closed state of one eye, and the continuous closing duration of one eye is greater than or equal to the ninth preset duration threshold, the target statistical information is re-calculated.

Corresponding to the above method embodiments, embodiments of the present application provide a distraction judgment device, as shown in Figure 6. The device includes:

The recognition unit 50 is used to identify the gaze direction in the driver's image based on the gaze recognition model;

The area determination unit 52 is used to determine the target gaze area to which the line of sight direction belongs;

The distraction determination unit 54 is used to determine that the driver is in a distracted state when it is determined that the preset distraction conditions are met;

Preset distraction conditions include at least one of the following:

In one implementation, the area determination unit 52 includes:

In one embodiment, the recognition unit 50 is used to recognize the driver's image based on the sight recognition model and obtain the sight direction in the camera coordinate system.

In one embodiment, the calculation module is used to convert the line of sight direction from the camera coordinate system to the body coordinate system when the external parameter calibration of the internal view camera is valid, and calculate the line of sight direction and the body line in the body coordinate system respectively. Whether each of the multiple preset gaze areas under the coordinate system intersects.

In one embodiment, the device further includes:

The distraction determination unit 54 is also configured to be greater than or equal to the fourth preset duration threshold when the included angle is continuously greater than or equal to the preset included angle threshold, or, within the second preset time window, the included angle is greater than or equal to the preset included angle threshold. If the cumulative duration of the angle threshold is greater than or equal to the fifth preset duration threshold, it is determined that the driver is in a distracted state.

In one embodiment, the calculation unit is also configured to calculate the N historical sight directions based on the recognition of N consecutive frames of historical driver images before calculating the angle between the sight direction in the camera coordinate system and the sight reference vector. Variance, where N is a positive integer;

The installation also includes:

In one embodiment, the identification unit 50 is also used to identify the driver's head angle in the driver's image based on the head angle recognition model when the external parameter calibration of the internal view camera fails;

The distraction determination unit 54 is also configured to be greater than or equal to the sixth preset duration threshold when the head rotation angle is continuously greater than or equal to the preset angle threshold, or, within the third preset time window, the head rotation angle is greater than or equal to When the cumulative duration of the preset angle threshold is greater than or equal to the seventh preset duration threshold, it is determined that the driver is in a distracted state.

In one embodiment, the device further includes:

The distraction determination unit 54 is also used to determine that the driver is in a distracted state when the vehicle satisfies the preset preconditions; when the vehicle does not satisfy the preset preconditions, If the conditions are met, it will not be executed and it will be determined that the driver is in a distracted state if it is determined that the preset distraction conditions are met;

Among them, the preset preconditions include at least one of the following:

The internal camera is in an unobstructed state;

Vehicle speed is greater than 0;

The vehicle is in a non-traffic jam state;

The vehicle is not in special condition;

In one embodiment, the device further includes:

In one implementation, the preset distraction area corresponds to at least one distraction level, and the distraction level of the preset distraction area is negatively correlated with the first preset duration threshold.

In one implementation, the preset distraction areas corresponding to the first distraction level include interior rearview mirror, central control screen, left side window, left rearview mirror, right side window, right rearview mirror, At least one item in the control panel, the preset distraction area corresponding to the second distraction level includes the left door and/or the right door, and the preset concentration area includes the front windshield.

Based on the above method embodiments, another embodiment of the present application provides a storage medium on which executable instructions are stored. When executed by a processor, the instructions cause the processor to implement the method described in any of the above embodiments.

Based on the above method embodiment, another embodiment of the present application provides an electronic device or computer device, including:

one or more processors;

a storage device for storing one or more programs,

Wherein, when the one or more programs are executed by the one or more processors, the electronic device or computer device is caused to implement the method described in any of the above embodiments.

Based on the above method embodiments, another embodiment of the present application provides a vehicle, which includes the device as described in any of the above embodiments, or includes the electronic device as described above.

As shown in Figure 7, the vehicle includes an interior rearview mirror 61, a central control screen 62, a left window 63, a left rearview mirror 64, a right window 65, a right rearview mirror 66, a control panel 67, and a left door 68. Right door 69, front windshield 610 and interior camera 611. In some embodiments, the interior rearview mirror 61, the central control screen 62, the left window 63, the left rearview mirror 64, the right window 65, the right rearview mirror 66, the control panel 67, the left door 68, The right door 69 belongs to the preset distraction area, the front windshield 610 belongs to the preset concentration area, and other areas in the vehicle belong to other distraction areas. The internal camera 611 is used to collect the driver's image information and transmit the driver's image information to the ECU and the like for distraction determination. The driver's sight direction is represented by a dotted line, and the target gaze area corresponding to the sight direction is the front windshield 610, so the driver is in a focused state. It can be understood that the above methods of dividing the concentration area or the distraction area are exemplary and can also be divided according to the application scenario of the vehicle or other indicators, which is not limited in this application.

The vehicle also includes GPS (Global Positioning System, Global Positioning System) positioning equipment, V2X (Vehicle-to-Everything, Internet of Vehicles), T-Box (TelematicsBox, telematics processor), radar, and exterior cameras. Among them, GPS positioning equipment is used to obtain vehicle location information; V2X is used to communicate with other vehicles, roadside equipment, etc.; radar or external cameras are used to sense road environment information ahead, and radar and/or external cameras can be configured on the vehicle body The front and/or rear of the car body; T-Box can be used as a wireless gateway to provide remote communication interfaces for the entire vehicle through functions such as 4G/5G and other long-range wireless communications, GPS satellite positioning, acceleration sensing and CAN communications, including driving data. Collection, driving trajectory recording, vehicle fault monitoring, vehicle remote query and control (opening and closing, air conditioning control, window control, engine torque limit, engine start and stop), driving behavior analysis and other services.

The above device embodiments correspond to the method embodiments and have the same technical effects as the method embodiments. For detailed description, please refer to the method embodiments. The device embodiment is obtained based on the method embodiment. For specific description, please refer to the method embodiment section and will not be described again here. Those of ordinary skill in the art can understand that the accompanying drawing is only a schematic diagram of an embodiment, and the modules or processes in the accompanying drawing are not necessarily necessary for implementing the present application.

Those of ordinary skill in the art can understand that the modules in the device in the embodiment may be distributed in the device in the embodiment according to the description of the embodiment, or may be correspondingly changed and located in one or more devices different from this embodiment. The modules of the above embodiments can be combined into one module, or further divided into multiple sub-modules.

Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present application, but not to limit it; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that it can still be Modifications are made to the technical solutions described in the foregoing embodiments, or equivalent substitutions are made to some of the technical features; however, these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions in the embodiments of the present application.

Claims

A distraction judgment method, characterized in that the method includes:

Identify the direction of gaze in the driver's image based on the gaze recognition model;

Determine the target gaze area to which the gaze direction belongs;

When it is determined that the preset distraction conditions are met, it is determined that the driver is in a distracted state;

Wherein, the category of the target gaze area includes at least one of a preset concentration area, a preset distraction area and other distraction areas;

The preset distraction conditions include at least one of the following:

The duration of the driver's continuous gaze on the first target gaze area is greater than or equal to the first preset duration threshold, and the first target gaze area belongs to the preset distraction area;

The cumulative duration of the driver's gaze on the second target gaze area within the first preset time window is greater than or equal to the second preset duration threshold, and the second target gaze area does not belong to the preset concentration area;

The duration of the driver's continuous gaze on the third target gaze area is greater than or equal to the third preset duration threshold, and the third target gaze area belongs to the other distraction areas;

A plurality of weighted scores of the target gaze areas determined based on the driver images of multiple consecutive frames are greater than or equal to a preset score threshold.
The method according to claim 1, wherein determining the target gaze area to which the line of sight direction belongs includes:

Calculate respectively whether the line of sight direction intersects with each of the preset gaze areas in a plurality of preset gaze areas, wherein the preset gaze areas include the preset concentration area and/or the preset distraction area area;

In the case where the line of sight direction only intersects one of the preset gaze areas, the intersecting preset gaze areas are determined as the target gaze areas; or,

When the sight direction intersects with at least two of the preset gaze areas, select the preset gaze area with the highest priority from the at least two preset gaze areas according to the preset area priority, and Determine the preset gaze area with the highest priority as the target gaze area; or,

When the line of sight direction represents that the driver is in an eye-opening state and the line of sight direction does not intersect with any of the plurality of preset gaze areas, or the line of sight direction represents that the driver is in a state of closing eyes. , determine the gaze area to which the line of sight direction belongs as the other distraction area, and determine the other distraction area as the target gaze area.
The method according to claim 2, characterized in that identifying the gaze direction in the driver's image based on the gaze recognition model includes:

The driver image is recognized based on the sight recognition model to obtain the sight direction in the camera coordinate system.
The method according to claim 3, wherein the separately calculating whether the line of sight direction intersects with each of the plurality of preset gaze areas includes:

When the external parameter calibration of the internal camera is valid, the line of sight direction is converted from the camera coordinate system to the body coordinate system, and the line of sight direction in the body coordinate system and multiple coordinates in the body coordinate system are calculated respectively. Whether each of the preset gaze areas intersects.
The method of claim 4, further comprising:

When the external parameter calibration of the inner-view camera fails, calculate the angle between the line of sight direction and the line of sight reference vector in the camera coordinate system;

When the included angle is continuously greater than or equal to the preset included angle threshold, the duration is greater than or equal to the fourth preset included angle threshold, or, within the second preset time window, the included angle is greater than or equal to the preset included angle threshold. If the cumulative duration is greater than or equal to the fifth preset duration threshold, it is determined that the driver is in a distracted state.
The method according to claim 5, characterized in that before calculating the angle between the line of sight direction and the line of sight reference vector in the camera coordinate system, the method further includes:

Calculate the variance of N historical sight directions based on N consecutive frames of historical driver image recognition, where N is a positive integer;

The average value of the N historical sight directions with the smallest variance is selected as the sight reference vector, or any historical sight direction among the N historical sight directions with the smallest variance is selected as the sight reference vector.
The method of claim 1, further comprising:

When the external parameter calibration of the internal view camera fails, identify the driver's head rotation angle in the driver image based on the head rotation angle recognition model;

When the head rotation angle is continuously greater than or equal to the preset angle threshold, the duration is greater than or equal to the sixth preset duration threshold, or, within the third preset time window, the head rotation angle is greater than or equal to the preset angle threshold. If the cumulative duration is greater than or equal to the seventh preset duration threshold, it is determined that the driver is in a distracted state.
The method according to claim 1, characterized in that, before determining that the driver is in a distracted state when it is determined that the preset distraction condition is met, the method further includes:

Determine whether the vehicle meets the preset preconditions;

When the vehicle satisfies the preset precondition, perform the step of determining that the driver is in a distracted state if it is determined that the preset distraction condition is met;

If the vehicle does not meet the preset precondition, do not perform the step of determining that the driver is in a distracted state if it is determined that the preset distraction condition is met;

Wherein, the preset preconditions include at least one of the following:

The internal camera is in an unobstructed state;

Vehicle speed is greater than 0;

The vehicle is in a non-traffic jam state;

The vehicle is not in a special condition;

The special state includes: when the vehicle is in a preset scene and the driver is in a non-fatigue state, the body posture is the same as the preset steering and the vehicle speed is within the preset safe range, where the preset scene includes steering, At least one of parking and reversing.
The method of claim 1, further comprising:

When the gaze direction indicates that the driver is in a closed-eyes state, and the driver's continuous eye-closed duration is greater than or equal to the eighth preset duration threshold, the driving fatigue prompt information is output, and the distraction judgment information is cleared. Target statistical information, wherein the target statistical information includes timing information and/or weighted score information.
The method of claim 9, further comprising:

When the sight direction indicates that the driver is in a state with both eyes closed, and the continuous closing duration of both eyes is greater than or equal to the ninth preset duration threshold and then one eye is opened, the target statistical information is re-calculated;

Alternatively, when the line of sight direction indicates that the driver is in a closed state of one eye, and the continuous closed period of one eye is greater than or equal to the ninth preset duration threshold, the target statistical information is re-calculated.
The method according to any one of claims 1 to 10, wherein the preset distraction area corresponds to at least one distraction level, and the distraction level of the preset distraction area is the same as the first distraction level. The preset duration thresholds are negatively correlated.
The method according to claim 11, characterized in that the preset distraction area corresponding to the first distraction level includes an interior rearview mirror, a central control screen, a left side window, a left rearview mirror, a right side At least one of the car window, right rearview mirror, and control panel, the preset distraction area corresponding to the second distraction level includes the left door and/or the right door, and the preset concentration area includes the front windshield. Glass.
A distraction judgment device, characterized in that the device includes:

A recognition unit for identifying the gaze direction in the driver's image based on the gaze recognition model;

an area determination unit, used to determine the target gaze area to which the line of sight direction belongs;

A distraction determination unit, configured to determine that the driver is in a distracted state when it is determined that the preset distraction conditions are met;

Wherein, the category of the target gaze area includes at least one of a preset concentration area, a preset distraction area and other distraction areas;

The preset distraction conditions include at least one of the following:

The duration of the driver's continuous gaze on the first target gaze area is greater than or equal to the first preset duration threshold, and the first target gaze area belongs to the preset distraction area;

The cumulative duration of the driver's gaze on the second target gaze area within the first preset time window is greater than or equal to the second preset duration threshold, and the second target gaze area does not belong to the preset concentration area;

The duration of the driver's continuous gaze on the third target gaze area is greater than or equal to the third preset duration threshold, and the third target gaze area belongs to the other distraction areas;

A plurality of weighted scores of the target gaze areas determined based on the driver images of multiple consecutive frames are greater than or equal to a preset score threshold.
The device according to claim 13, characterized in that the area determining unit includes:

A calculation module configured to respectively calculate whether the line of sight direction intersects with each of the preset gaze areas in a plurality of preset gaze areas, wherein the preset gaze areas include the preset concentration area and/or the The above-mentioned default distraction area;

a determination module, configured to determine the intersecting preset gaze area as the target gaze area when the line of sight direction intersects with only one of the preset gaze areas; or, when the line of sight direction intersects with at least one of the preset gaze areas; When two of the preset gaze areas intersect, the preset gaze area with the highest priority is selected from the at least two preset gaze areas according to the preset area priority, and the highest priority preset gaze area is The preset gaze area is determined as the target gaze area; or, the sight direction indicates that the driver is in an eye-opening state and the sight direction does not intersect with any of the multiple preset gaze areas, or, the When the gaze direction indicates that the driver is in a closed-eyes state, the gaze area to which the gaze direction belongs is determined to be the other distraction area, and the other distraction area is determined to be the target gaze area.
The device according to claim 14, characterized in that the recognition unit is used to recognize the driver image based on the sight recognition model and obtain the sight direction in the camera coordinate system.
The device according to claim 15, characterized in that the calculation module is used to convert the line of sight direction from the camera coordinate system to the vehicle body coordinate system when the external parameter calibration of the internal view camera is valid, respectively. Calculate whether the line of sight direction in the vehicle body coordinate system intersects with each of the plurality of preset gaze areas in the vehicle body coordinate system.
The device according to claim 16, characterized in that the device further includes:

A calculation unit configured to calculate the angle between the line of sight direction and the line of sight reference vector in the camera coordinate system when the external parameter calibration of the internal view camera fails;

The distraction determination unit is also configured to: when the time period during which the included angle is continuously greater than or equal to the preset included angle threshold is greater than or equal to the fourth preset time length threshold, or, within the second preset time window, the included angle is continuously greater than or equal to the preset included angle threshold. If the cumulative duration of the angle is greater than or equal to the preset angle threshold is greater than or equal to the fifth preset duration threshold, it is determined that the driver is in a distracted state.
The device according to claim 17, characterized in that the calculation unit is further configured to calculate the angle between the line of sight direction and the line of sight reference vector in the camera coordinate system based on consecutive N frames of historical driving. The variance of N historical sight directions recognized by the actor's image, where N is a positive integer;

The device also includes:

The selection unit is used to select the average of N historical sight directions with the smallest variance as the sight reference vector, or select any historical sight direction among the N historical sight directions with the smallest variance as the sight reference vector.
The device according to claim 13, wherein the identification unit is further configured to identify the driver in the driver image based on a head angle recognition model when the external parameter calibration of the internal view camera fails. the player’s head angle;

The distraction determination unit is also configured to: when the head rotation angle is continuously greater than or equal to the preset angle threshold, the duration is greater than or equal to the sixth preset duration threshold, or, within the third preset time window, the head When the cumulative duration of the partial rotation angle is greater than or equal to the preset angle threshold and is greater than or equal to the seventh preset duration threshold, it is determined that the driver is in a distracted state.
The device according to claim 13, characterized in that the device further includes:

A judgment unit configured to judge whether the vehicle satisfies the preset preconditions before determining that the driver is in a distracted state when it is judged that the preset distraction conditions are met;

The distraction determination unit is also configured to determine that the driver is in a distracted state when the vehicle satisfies the preset preconditions. ; If the vehicle does not meet the preset preconditions, do not perform the step of determining that the driver is in a distracted state if it is determined that the preset distraction conditions are met;

Wherein, the preset preconditions include at least one of the following:

The internal camera is in an unobstructed state;

Vehicle speed is greater than 0;

The vehicle is in a non-traffic jam state;

The vehicle is not in a special condition;

The special state includes: when the vehicle is in a preset scene and the driver is in a non-fatigue state, the body posture is the same as the preset steering and the vehicle speed is within the preset safe range, where the preset scene includes steering, At least one of parking and reversing.
The device according to claim 13, characterized in that the device further includes:

An output unit configured to output driving fatigue prompt information when the sight direction indicates that the driver is in a closed-eyes state, and the driver's continuous eye-closure duration is greater than or equal to an eighth preset duration threshold;

A clearing unit configured to clear target statistical information for distraction judgment, wherein the target statistical information includes timing information and/or weighted score information.
The device of claim 21, further comprising:

A statistical unit configured to re-statisticate the target statistical information when the sight direction indicates that the driver is in a state of closing both eyes, and the continuous closing time of both eyes is greater than or equal to the ninth preset time threshold and then one eye is opened; or, When the sight direction indicates that the driver is in a closed state of one eye, and the continuous closing duration of one eye is greater than or equal to the ninth preset duration threshold, the target statistical information is re-calculated.
The device according to any one of claims 13-22, wherein the preset distraction area corresponds to at least one distraction level, and the distraction level of the preset distraction area is the same as the first distraction level. The preset duration thresholds are negatively correlated.
The device according to claim 23, wherein the preset distraction area corresponding to the first distraction level includes an interior rearview mirror, a central control screen, a left side window, a left rearview mirror, a right side At least one of the car window, right rearview mirror, and control panel, the preset distraction area corresponding to the second distraction level includes the left door and/or the right door, and the preset concentration area includes the front windshield. Glass.
A storage medium with a computer program stored thereon, characterized in that when the program is executed by a processor, the method according to any one of claims 1-12 is implemented.
An electronic device, characterized in that the electronic device includes:

one or more processors;

a storage device for storing one or more programs,

When the one or more programs are executed by the one or more processors, the electronic device is caused to implement the method as described in any one of claims 1-12.
A vehicle, characterized in that the vehicle includes the device according to any one of claims 13-24, or the electronic device according to claim 26.