CN112036314A

CN112036314A - Steering wheel hands-off detection method and device, electronic equipment and storage medium

Info

Publication number: CN112036314A
Application number: CN202010899671.1A
Authority: CN
Inventors: 祁凯悦; 伍俊; 范亦卿; 陶莹; 许亮
Original assignee: Shanghai Sensetime Lingang Intelligent Technology Co Ltd
Current assignee: Shanghai Sensetime Lingang Intelligent Technology Co Ltd
Priority date: 2020-08-31
Filing date: 2020-08-31
Publication date: 2020-12-04
Also published as: WO2022041671A1

Abstract

The disclosure relates to a steering wheel hands-off detection method and device, electronic equipment and a storage medium. The method comprises the following steps: acquiring a video stream of a driving area; performing steering wheel hands-off detection on the driver based on the video stream of the driving area to obtain a steering wheel hands-off detection result; and generating reminding information based on the steering wheel hands-off detection result.

Description

Steering wheel hands-off detection method and device, electronic equipment and storage medium

Technical Field

The present disclosure relates to the field of computer vision technologies, and in particular, to a method and an apparatus for detecting steering wheel hands-off, an electronic device, and a storage medium.

Background

In the driving process, the hand is separated from the steering wheel, which is a very dangerous driving behavior and is easy to cause traffic accidents. Therefore, the detection of the hands-off of the steering wheel is carried out on the driver, and the detection method has important significance for safe driving.

Disclosure of Invention

The present disclosure provides a steering wheel hands-off detection technical scheme.

According to an aspect of the present disclosure, there is provided a steering wheel hands-off detection method, including:

acquiring a video stream of a driving area;

performing steering wheel hands-off detection on the driver based on the video stream of the driving area to obtain a steering wheel hands-off detection result;

and generating reminding information based on the steering wheel hands-off detection result.

In a possible implementation manner, the performing steering wheel hands-off detection on the driver based on the video stream of the driving area to obtain a steering wheel hands-off detection result includes:

detecting whether the hands of the driver are separated from a steering wheel or not based on the video stream of the driving area, and obtaining a steering wheel separation type corresponding to the video stream of the driving area;

and determining the steering wheel hands-off type as a steering wheel hands-off detection result.

detecting whether the hands of the driver are separated from a steering wheel and/or the hand motions of the driver based on the video stream of the driving area, and obtaining a steering wheel separation type and/or a hand motion type of the driver corresponding to the video stream of the driving area;

and determining a steering wheel hands-off detection result according to the steering wheel hands-off type and/or the hand action type of the driver.

In one possible implementation, the steering wheel hands-off category is that hands are not off the steering wheel, that one hand is off the steering wheel, or that two hands are off the steering wheel.

In a possible implementation manner, the generating a warning message based on the detection result of the steering wheel out-of-hand includes:

and generating reminding information in response to the fact that the hand-off detection result of the steering wheel is that the hands of the driver are off the steering wheel and the hand action of the driver belongs to dangerous action.

determining a driving state based on the steering wheel hands-off detection result when the automatic driving function is in an on state;

and generating reminding information in response to the driving state being a dangerous state.

In one possible implementation, the determining a driving state based on the steering wheel hands-off detection result includes:

determining a duration of time for which the driver's hand is disengaged from the steering wheel based on the steering wheel disengagement detection result;

determining a driving state based on a duration of time for which the driver's hand is disengaged from the steering wheel; and/or

The determining a driving state based on the steering wheel hands-off detection result includes:

determining a driving state based on the steering wheel hands-off detection result and the driver's attention detection result.

In one possible implementation, the determining the driving state based on a length of time that the driver's hand is disengaged from the steering wheel includes:

determining that the driving state is a dangerous state in response to the time length for which the hands of the driver are disengaged from the steering wheel reaching a first time length threshold;

the determining a driving state based on the steering wheel hands-off detection result and the driver's attention detection result includes:

and determining that the driving state is a dangerous state in response to the steering wheel hands-off detection result being that the hands of the driver are off the steering wheel and the attention detection result of the driver being that the driver is in a distracted state.

and under the condition that the automatic driving function is in a non-opening state, responding to the detection result that the hands of the driver are separated from the steering wheel and the hand action of the driver belongs to dangerous action, and generating reminding information.

In a possible implementation manner, the generating the reminding information includes:

and generating reminding information according to at least one of the number of hands which are separated from the steering wheel, the time length of the hands which are separated from the steering wheel and the hand action type of the driver.

In one possible implementation manner, the generating the reminding information according to at least one of the number of hands disengaged from the steering wheel, the length of time for which the hands are disengaged from the steering wheel, and the hand action category of the driver includes:

acquiring reminding animations and/or voices corresponding to the preset action types under the condition that the hand action types of the drivers are the preset action types;

and generating control information for controlling a display screen to display the reminding animation and/or generating control information for controlling a loudspeaker to play the voice.

In one possible implementation, the steering wheel hands-off detection result includes a steering wheel hands-off category;

the generating of the reminding information based on the result of the detection of the steering wheel out of hand comprises:

generating corresponding warning level reminding information according to the steering wheel hands-off category, wherein the steering wheel hands-off category comprises: the hands are not disengaged from the steering wheel, are disengaged from the steering wheel with one hand, or are disengaged from the steering wheel with both hands.

determining a dangerous state level corresponding to the driving state of the driver according to the detection result of the hands-off of the steering wheel;

and generating reminding information corresponding to the dangerous state grade.

In one possible implementation form of the method,

the video stream based on the driving area is used for detecting the hands-off of the steering wheel of the driver to obtain the detection result of the hands-off of the steering wheel, and the detection method comprises the following steps: responding to the fact that the running speed of the vehicle to which the driving area belongs is larger than a first preset speed threshold value, and carrying out steering wheel hands-off detection on the driver on the basis of the video stream of the driving area to obtain a steering wheel hands-off detection result;

and/or the presence of a gas in the gas,

generating reminding information based on the steering wheel hands-off detection result, comprising: responding to the steering wheel hands-off detection result that the steering wheel is separated from the steering wheel by one hand, wherein the hand action of the driver belongs to dangerous action, and the running speed reaches a second preset speed threshold value, and generating reminding information;

and/or the presence of a gas in the gas,

generating reminding information based on the steering wheel hands-off detection result, comprising: and responding to the steering wheel hands-off detection result that the two hands are separated from the steering wheel, generating reminding information when the running speed reaches a third preset speed threshold value, wherein the third preset speed threshold value is not larger than the second preset speed threshold value.

In one possible implementation, the method further includes:

according to the video stream of the driving area, carrying out fatigue detection on the driver to obtain a fatigue detection result;

and generating reminding information based on the fatigue detection result.

inputting at least one frame of image in the video stream of the driving area into a neural network, carrying out steering wheel hands-off detection on a driver through the neural network to obtain a steering wheel hands-off type corresponding to the video stream of the driving area, and determining the steering wheel hands-off type as a steering wheel hands-off detection result;

alternatively, the first and second electrodes may be,

inputting at least one frame of image in the video stream of the driving area into a neural network, carrying out steering wheel hands-off detection on a driver through the neural network to obtain a steering wheel hands-off type corresponding to the video stream of the driving area and a hand action type of the driver, and determining a steering wheel hands-off detection result according to the steering wheel hands-off type and the hand action type of the driver.

According to an aspect of the present disclosure, there is provided a steering wheel hands-off detection apparatus including:

the acquisition module is used for acquiring a video stream of a driving area;

the first detection module is used for detecting the hands-off of the steering wheel of the driver based on the video stream of the driving area to obtain a detection result of the hands-off of the steering wheel;

and the first generation module is used for generating reminding information based on the detection result of the steering wheel out of hand.

In one possible implementation manner, the first detection module is configured to:

In one possible implementation, the first generating module is configured to:

The first generation module is used for:

In one possible implementation, the first generating module is configured to:

determining that the driving state is a dangerous state in response to the time length for which the hands of the driver are disengaged from the steering wheel reaching a first time length threshold; and/or

In one possible implementation, the first generating module is configured to:

the first generation module is used for:

In one possible implementation, the first generating module is configured to:

In one possible implementation form of the method,

the first detection module is configured to: responding to the fact that the running speed of the vehicle to which the driving area belongs is larger than a first preset speed threshold value, and carrying out steering wheel hands-off detection on the driver on the basis of the video stream of the driving area to obtain a steering wheel hands-off detection result;

and/or the presence of a gas in the gas,

the first generation module is used for: responding to the steering wheel hands-off detection result that the steering wheel is separated from the steering wheel by one hand, wherein the hand action of the driver belongs to dangerous action, and the running speed reaches a second preset speed threshold value, and generating reminding information;

and/or the presence of a gas in the gas,

the first generation module is used for: and responding to the steering wheel hands-off detection result that the two hands are separated from the steering wheel, generating reminding information when the running speed reaches a third preset speed threshold value, wherein the third preset speed threshold value is not larger than the second preset speed threshold value.

In one possible implementation, the apparatus further includes:

the second detection module is used for carrying out fatigue detection on the driver according to the video stream of the driving area to obtain a fatigue detection result;

and the second generation module is used for generating reminding information based on the fatigue detection result.

In one possible implementation, the detection module is configured to:

alternatively, the first and second electrodes may be,

According to an aspect of the present disclosure, there is provided an electronic device including: one or more processors; a memory for storing executable instructions; wherein the one or more processors are configured to invoke the memory-stored executable instructions to perform the above-described method.

According to an aspect of the present disclosure, there is provided a computer readable storage medium having stored thereon computer program instructions which, when executed by a processor, implement the above-described method.

In the embodiment of the disclosure, a scheme of acquiring a video stream of a driving area, detecting the hands-off of a steering wheel of a driver based on the video stream of the driving area to obtain a detection result of the hands-off of the steering wheel, and generating a reminding message based on the detection result of the hands-off of the steering wheel is adopted, which is helpful for reducing potential safety hazards in the driving process. In addition, the method of computer vision is used for detecting the steering wheel hands-off of the video stream in the driving area, a reliable detection result can be obtained without adding extra hardware, and the cost of detecting the steering wheel hands-off can be reduced.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.

Other features and aspects of the present disclosure will become apparent from the following detailed description of exemplary embodiments, which proceeds with reference to the accompanying drawings.

Drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the disclosure.

Fig. 1 shows a flowchart of a method for detecting hands-off of a steering wheel according to an embodiment of the present disclosure.

Fig. 2 shows a block diagram of a steering wheel hands-off detection apparatus provided in an embodiment of the present disclosure.

Fig. 3 illustrates a block diagram of an electronic device 800 provided by an embodiment of the disclosure.

Fig. 4 shows a block diagram of an electronic device 1900 provided by an embodiment of the disclosure.

Detailed Description

Various exemplary embodiments, features and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. In the drawings, like reference numbers can indicate functionally identical or similar elements. While the various aspects of the embodiments are presented in drawings, the drawings are not necessarily drawn to scale unless specifically indicated.

The word "exemplary" is used exclusively herein to mean "serving as an example, embodiment, or illustration. Any embodiment described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments.

The term "and/or" herein is merely an association describing an associated object, meaning that three relationships may exist, e.g., a and/or B, may mean: a exists alone, A and B exist simultaneously, and B exists alone. In addition, the term "at least one" herein means any one of a plurality or any combination of at least two of a plurality, for example, including at least one of A, B, C, and may mean including any one or more elements selected from the group consisting of A, B and C.

Furthermore, in the following detailed description, numerous specific details are set forth in order to provide a better understanding of the present disclosure. It will be understood by those skilled in the art that the present disclosure may be practiced without some of these specific details. In some instances, methods, means, elements and circuits that are well known to those skilled in the art have not been described in detail so as not to obscure the present disclosure.

In the embodiment of the disclosure, the driving safety is improved by acquiring the video stream of the driving area, detecting the hands-off of the steering wheel of the driver based on the video stream of the driving area to obtain the detection result of the hands-off of the steering wheel, and generating the reminding information based on the detection result of the hands-off of the steering wheel. In addition, the steering wheel hands-off detection is carried out by utilizing the video stream of the driving area, additional hardware does not need to be added, and therefore the cost of the steering wheel hands-off detection can be reduced.

Fig. 1 shows a flowchart of a method for detecting hands-off of a steering wheel according to an embodiment of the present disclosure. The execution main body of the steering wheel hands-off detection method can be a steering wheel hands-off detection device. In one possible implementation, the steering wheel hands-off detection method may be performed by a terminal device or a server or other processing device. The terminal device may be a vehicle-mounted device, a User Equipment (UE), a mobile device, a User terminal, a cellular phone, a cordless phone, a Personal Digital Assistant (PDA), a handheld device, a computing device, or a wearable device. The on-board device may be a vehicle, a domain controller or a processor in a vehicle cabin, and may also be a device host for performing data processing operations such as images in a DMS (Driver Monitoring System) or an OMS (Occupant Monitoring System). In some possible implementations, the steering wheel hands-off detection method may be implemented by a processor invoking computer-readable instructions stored in a memory. As shown in fig. 1, the steering wheel hands-off detection method includes steps S11 to S13.

In step S11, a video stream of the driving area is acquired.

The steering wheel hands-off detection method provided by the embodiment of the disclosure can be applied to various types of vehicles. For example, the vehicle may be a vehicle, or may be a vehicle for carrying people or cargo, such as a ship or an airplane. Wherein the vehicle may be a private car or a commercial vehicle, for example, the commercial vehicle may be a shared automobile, a net appointment car, a taxi, a bus, a school bus, a long-haul van, a short-haul van, a long-haul bus, a short-haul bus, a train, a subway, a tram, etc.

In one possible implementation, the video stream of the driving area may be captured by a camera disposed in a cabin of the vehicle. For example, the cameras disposed within the chamber may include DMS cameras, OMS cameras, and the like. The DMS may also perform other driver detection functions, such as fatigue detection, attention detection, etc., among others. The OMS may also perform other occupant detection functions, such as detection of occupant attributes (e.g., at least one of age, gender, emotional state, physical state, etc.), detection of speaking actions of the occupant, and so forth.

In one possible implementation, the video stream of the driving area may be acquired in real time by a camera disposed in the cabin. That is, the acquired video stream of the driving area may be a real-time video stream of the driving area. Wherein the driving area may comprise an area of a main driver's seat.

In step S12, the driver is detected for the hands-off steering wheel based on the video stream of the driving area, and a result of the hands-off steering wheel detection is obtained.

In the embodiment of the present disclosure, the hands-off detection of the steering wheel may be performed on the driver based on a single-frame or multi-frame image in the video stream of the driving area, so as to obtain a result of the hands-off detection of the steering wheel. The steering wheel hands-off detection may include detecting whether the driver's hands are off the steering wheel.

In a possible implementation manner, the performing steering wheel hands-off detection on the driver based on the video stream of the driving area to obtain a steering wheel hands-off detection result may include: detecting a position of a hand of a driver based on a video stream of the driving area; and determining whether the hands of the driver are separated from the steering wheel according to the position of the steering wheel and the positions of the hands of the driver.

As an example of this implementation, the position information of the steering wheel may be stored in advance, and the position information of the steering wheel stored in advance may be acquired to determine whether the hand of the driver is detached from the steering wheel. As another example of this implementation, the position information of the steering wheel may be detected based on a video stream of the driving area.

As one example of this implementation, the position of the steering wheel may include a three-dimensional coordinate range of the steering wheel, or a two-dimensional coordinate range in an image captured by a fixed position camera (e.g., a DMS camera), and the position of the driver's hand may include a three-dimensional or two-dimensional coordinate range of the driver's hand; the determining whether the hand of the driver is separated from the steering wheel according to the position of the steering wheel and the position of the hand of the driver may include: and determining whether the hand of the driver is separated from the steering wheel according to the coordinate range of the steering wheel and the coordinate range of the hand of the driver.

In one example, the coordinate range of the steering wheel may include an image coordinate range of the steering wheel in an image coordinate system corresponding to an image of the steering wheel in the video stream of the driving area, and the coordinate range of the driver's hand may include an image coordinate range of the driver's hand in the image coordinate system.

In another example, the coordinate range of the steering wheel may include a spatial coordinate range of the steering wheel in a spatial coordinate system within a cabin of a vehicle to which the driving area belongs, and the coordinate range of the driver's hand may include a spatial coordinate range of the driver's hand in the spatial coordinate system.

In one example, the determining whether the hand of the driver is detached from the steering wheel according to the coordinate range of the steering wheel and the coordinate range of the hand of the driver may include: and determining whether the hand of the driver is separated from the steering wheel according to the coordinate range of the steering wheel, the coordinate range of the left hand of the driver and the coordinate range of the right hand of the driver. For example, it may be determined whether the left hand of the driver is off the steering wheel according to the size of the overlapping area of the coordinate range of the steering wheel and the coordinate range of the left hand of the driver; whether the right hand of the driver is detached from the steering wheel may be determined according to the size of the overlapping area of the coordinate range of the steering wheel and the coordinate range of the right hand of the driver. For example, it may be determined that the left hand of the driver does not come off the steering wheel in a case where an overlapping area of the coordinate range of the steering wheel and the coordinate range of the left hand of the driver reaches a first preset area threshold; it may be determined that the left hand of the driver is out of the steering wheel in a case where an overlapping area of the coordinate range of the steering wheel and the coordinate range of the left hand of the driver does not reach a first preset area threshold value. For another example, it may be determined that the right hand of the driver does not disengage from the steering wheel when an overlapping area of the coordinate range of the steering wheel and the coordinate range of the right hand of the driver reaches a second preset area threshold; it may be determined that the right hand of the driver is out of the steering wheel in a case where an overlapping area of the coordinate range of the steering wheel and the coordinate range of the right hand of the driver does not reach a second preset area threshold. The second preset area threshold may be equal to the first preset area threshold, or may not be equal to the first preset area threshold. For another example, whether the driver's hand is off the steering wheel may be determined by determining whether the distance exceeds a predetermined distance threshold based on the distance between the center coordinates of the steering wheel position and the center coordinates of the hand position, and determining whether the driver's hand is off the steering wheel accordingly

In one possible implementation, in a case where it is detected that the left hand of the driver is disengaged from the steering wheel and the right hand is not disengaged from the steering wheel, the steering wheel disengagement detection result may include a one-handed disengagement steering wheel and/or a left-handed disengagement steering wheel; in the case that the right hand of the driver is detected to be disengaged from the steering wheel and the left hand of the driver is detected not to be disengaged from the steering wheel, the steering wheel disengagement detection result may include a single-hand disengagement steering wheel and/or a right-hand disengagement steering wheel; in a case where it is detected that both the left hand and the right hand of the driver are disengaged from the steering wheel, the steering wheel disengagement detection result may include both hands being disengaged from the steering wheel; in the case where it is detected that neither the left hand nor the right hand of the driver is disengaged from the steering wheel, the steering wheel disengagement detection result may include that the hand is not disengaged from the steering wheel.

In step S13, a warning message is generated based on the result of the detection of the steering wheel being out of hand.

In the embodiment of the present disclosure, in response to the detection result that the steering wheel is out of hand including the hand of the driver out of the steering wheel, a warning message for warning the driver to replace the hand on the steering wheel may be generated.

In a possible implementation manner, the reminding information may be generated according to at least one of a steering wheel hands-off category corresponding to the steering wheel hands-off detection result, a hand action category of the driver, and a driving state.

As an example of this implementation manner, the reminding information may be generated according to the steering wheel hands-off category corresponding to the steering wheel hands-off detection result. For example, if the type of the steering wheel out-of-hand corresponding to the result of the detection of the steering wheel out-of-hand is one-handed out of the steering wheel or two-handed out of the steering wheel, the reminding information may include information of "please place two hands on the steering wheel".

As an example of this implementation manner, the reminding information may be generated according to the hand motion type of the driver corresponding to the steering wheel hands-off detection result. For example, if the hand motion type of the driver corresponding to the detection result of the hands-off of the steering wheel is a call, the reminding information may include information of "do not call". For another example, if the hand motion type of the driver corresponding to the detection result of the steering wheel out-of-hand is smoking, the reminding information may include information of "do not smoke".

As an example of this implementation, the reminding information may be generated according to the driving state of the driver. For example, when the automatic driving function is in the activated state, if the driving state is a non-dangerous state, a warning message "driving automatically" may be displayed through the vehicle-mounted device and/or the center console. For another example, when the automatic driving function is in the turned-on state, if the driving state is a dangerous state, the reminding information may include information such as "please concentrate attention," please place both hands on the steering wheel, "and the like.

In the embodiment of the disclosure, the driving safety is improved by acquiring the video stream of the driving area, detecting the hands-off of the steering wheel of the driver in a computer vision mode based on the video stream of the driving area to obtain the detection result of the hands-off of the steering wheel, and generating the reminding information based on the detection result of the hands-off of the steering wheel. In addition, by using the video stream of the driving area to detect the hands-off of the steering wheel, a reliable detection result can be obtained without adding extra hardware, for example, hardware such as a steering column torque sensor, a steering wheel pressure sensor and a steering wheel touch sensor, so that the cost of detecting the hands-off of the steering wheel can be reduced.

In a possible implementation manner, the performing steering wheel hands-off detection on the driver based on the video stream of the driving area to obtain a steering wheel hands-off detection result includes: detecting whether the hands of the driver are separated from a steering wheel or not based on the video stream of the driving area, and obtaining a steering wheel separation type corresponding to the video stream of the driving area; and determining the steering wheel hands-off type as a steering wheel hands-off detection result.

Specifically, the steering wheel hands-off condition may be classified in advance, for example, setting the steering wheel hands-off category includes: the steering wheel is not released, the single hand is released from the steering wheel and the two hands are released from the steering wheel; optionally, the category of one-hand off-steering wheel can be further subdivided into: left hand single back disengagement steering wheel and right hand single hand disengagement steering wheel. Then the steering wheel hands-off category of the driver is detected by means of image classification/video classification. For example, if it is detected that the left hand of the driver is detached from the steering wheel and the right hand of the driver is not detached from the steering wheel based on the video stream of the driving area, it may be determined that the type of the steering wheel detachment corresponding to the video stream of the driving area is the one-hand detached steering wheel, and it may be determined that the result of the detection of the steering wheel detachment includes the one-hand detached steering wheel. For another example, if it is detected that the right hand of the driver is disengaged from the steering wheel and the left hand of the driver is not disengaged from the steering wheel based on the video stream of the driving area, it may be obtained that the type of the steering wheel disengagement corresponding to the video stream of the driving area is a single-hand disengagement steering wheel, and it may be determined that the result of the steering wheel disengagement detection includes the single-hand disengagement steering wheel. For another example, if it is detected that both the left hand and the right hand of the driver are disengaged from the steering wheel based on the video stream of the driving area, it may be obtained that the steering wheel disengagement category corresponding to the video stream of the driving area is that both hands are disengaged from the steering wheel, and it may be determined that the steering wheel disengagement detection result includes that both hands are disengaged from the steering wheel. For another example, if it is detected that both the left hand and the right hand of the driver do not disengage from the steering wheel based on the video stream of the driving area, it may be obtained that the steering wheel disengagement type corresponding to the video stream of the driving area is that the hand does not disengage from the steering wheel, and it may be determined that the steering wheel disengagement detection result includes that the hand does not disengage from the steering wheel.

In this implementation, the steering wheel out-of-hand detection result can be accurately obtained based on the steering wheel out-of-hand category.

As an example of this implementation, the steering wheel hands-off category is either hands not off the steering wheel, one-handed off the steering wheel, or two-handed off the steering wheel. In this example, the one-handed disengaged steering wheel may be a left-handed disengaged steering wheel, a right-handed un-disengaged steering wheel, or may be a right-handed disengaged steering wheel, a left-handed un-disengaged steering wheel. In this example, the hand-off steering wheel is further divided into a one-hand-off steering wheel and a two-hand-off steering wheel, so that the type of the hand-off of the steering wheel is further refined, different types of the hand-off of the steering wheel can be detected, and the reminding of the hand-off of the steering wheel can be more accurately performed.

As an example of an implementation manner, after the obtaining of the steering wheel hands-off category corresponding to the video stream of the driving area, the method may further include: and detecting the hand motion of the driver based on the video stream of the driving area to obtain the hand motion type of the driver corresponding to the video stream of the driving area. In this example, the hand motion of the driver may be further detected based on the video stream of the driving area in response to the steering wheel hands-off category being a hands-off steering wheel (e.g., one-handed off steering wheel and/or two-handed off steering wheel) to further analyze the risk of the driver's hand motion when the driver is one-handed off the steering wheel or both hands are off the steering wheel. Of course, in another example, after the steering wheel hands-off category corresponding to the video stream of the driving area is obtained, the hand motion of the driver may not be detected.

In a possible implementation manner, the performing steering wheel hands-off detection on the driver based on the video stream of the driving area to obtain a steering wheel hands-off detection result includes: detecting whether the hands of the driver are separated from a steering wheel and/or the hand motions of the driver based on the video stream of the driving area, and obtaining a steering wheel separation type and/or a hand motion type of the driver corresponding to the video stream of the driving area; and determining a steering wheel hands-off detection result according to the steering wheel hands-off type and/or the hand action type of the driver.

As an example of this implementation, the steering wheel hands-off category is either hands not off the steering wheel, one-handed off the steering wheel, or two-handed off the steering wheel.

As an example of this implementation manner, whether the hands of the driver are off the steering wheel and the hand motion of the driver may be detected in parallel based on the video stream of the driving area, and a steering wheel off-hand type and a hand motion type of the driver corresponding to the video stream of the driving area may be obtained. That is, in this example, detecting whether the driver's hand is off the steering wheel and detecting the driver's hand motion may be performed simultaneously.

As another example of this implementation, it may be configured to detect whether the hands of the driver are off the steering wheel based on the video stream of the driving area to obtain a steering wheel off-hand type corresponding to the video stream of the driving area, and then detect the hand motion of the driver based on the video stream of the driving area to obtain a hand motion type of the driver corresponding to the video stream of the driving area.

As another example of this implementation manner, it may be configured to first detect whether the hand of the driver is off the steering wheel based on the video stream of the driving area, and obtain a steering wheel off-hand type corresponding to the video stream of the driving area. When the steering wheel hands-off type is that hands are not off the steering wheel, the hand motion of the driver can not be detected any more; when the steering wheel hands-off type is hands-off steering wheel, the hand motion of the driver may be detected, and the hand motion type of the driver corresponding to the video stream of the driving area may be obtained.

As another example of this implementation manner, based on the video stream of the driving area, the hand motion of the driver is detected first, the hand motion type of the driver corresponding to the video stream of the driving area is obtained, and then whether the hand of the driver is out of the steering wheel is detected, and the steering wheel out-of-hand type corresponding to the video stream of the driving area is obtained.

As another example of this implementation manner, the hand motion of the driver may be detected first based on the video stream of the driving area, and the hand motion category of the driver corresponding to the video stream of the driving area may be obtained. Under the condition that the hand action category of the driver belongs to a preset dangerous action category, determining a detection result of the hands-off of the steering wheel according to the hand action category of the driver; under the condition that the hand action category of the driver does not belong to a preset dangerous action category, whether the hand of the driver is separated from a steering wheel or not can be detected based on the video stream of the driving area, a steering wheel hands-off category corresponding to the video stream of the driving area is obtained, and a steering wheel hands-off detection result is determined according to the steering wheel hands-off category.

The preset dangerous motion categories may include motion categories that may cause driver distraction or the like to affect driving safety. For example, the preset dangerous motion category may include at least one of one-handed telephone call, one-handed water drinking, one-handed smoking, two-handed mobile phone operation, and the like. For example, if the hand motion type of the driver is one-handed call making, it may be determined that the steering wheel hands off detection result is "one-handed off steering wheel call making"; if the hand action category of the driver is that the mobile phone is operated by two hands, it can be determined that the steering wheel hands-off detection result is that the mobile phone is operated by two hands off the steering wheel. If the hand motion category of the driver does not belong to the preset dangerous motion category, whether the hand of the driver is separated from a steering wheel or not can be detected based on the video stream of the driving area, and the steering wheel separation category corresponding to the video stream of the driving area is obtained. If the steering wheel hands-off type is that the hands are not detached from the steering wheel, the steering wheel hands-off detection result may include that the hands are not detached from the steering wheel; if the steering wheel hands-off category is a single-hand hands-off steering wheel, the steering wheel hands-off detection result may include a single-hand hands-off steering wheel; if the steering wheel hands-off category is that both hands are off the steering wheel, the steering wheel hands-off detection result may include that both hands are off the steering wheel.

According to the implementation mode, a more accurate hand action type detection result can be obtained, whether the hand action type is safe or not is further judged, more targeted reminding can be performed according to the more accurate hand action type detection result, for example, reminding information of 'please do not make a call when driving' is generated according to actions of a driver holding a steering wheel with one hand and holding a phone with the other hand, the driver is assisted in safe driving, and potential safety hazards in driving are reduced.

In a possible implementation manner, the generating a warning message based on the detection result of the steering wheel out-of-hand includes: and generating reminding information in response to the fact that the hand-off detection result of the steering wheel is that the hands of the driver are off the steering wheel and the hand action of the driver belongs to dangerous action.

For example, the hazardous action may include at least one of drinking water, making a call, operating a cell phone, smoking a cigarette, and the like.

As an example of this implementation manner, the generating a warning message in response to the detection result that the hand of the driver is disengaged from the steering wheel and the hand action of the driver belongs to a dangerous action may include: and generating reminding information in response to the fact that the steering wheel is separated from the steering wheel by one hand and the hand action of the driver belongs to dangerous action. For example, the reminding information may be generated in response to the detection result that the steering wheel is detached from the steering wheel by one hand and the hand action of the driver is drinking, making a call or smoking.

As another example of this implementation, the generating of the reminding information in response to the detection result that the hand of the driver is disengaged from the steering wheel and the hand action of the driver belongs to a dangerous action may include: and responding to the steering wheel hands-off detection result that the two hands are separated from the steering wheel, and generating reminding information when the hand action of the driver belongs to dangerous action. For example, the reminding information may be generated in response to that the steering wheel hands are disengaged from the steering wheel and the driver's hand motion is a two-handed mobile phone.

This implementation mode can judge whether the driver takes off the hand and carries out normal control or dangerous action under the condition, reminds the driver not carry out dangerous action at the in-process of driving to help reducing the potential safety hazard in the driving.

In a possible implementation manner, the detection result of the hands-off of the steering wheel is that the hands are separated from the steering wheel by one hand, the hand action of the driver belongs to a non-dangerous action, and the time length that the hands are separated from the steering wheel by one hand does not reach the second time length threshold value, and the reminding information is not generated. For example, the non-hazardous action may be at least one of shifting gears, operating an in-vehicle entertainment system, pulling a handbrake, etc. According to the implementation mode, when a driver disengages from the steering wheel by one hand to perform short-time non-dangerous actions, reminding is not performed, so that unnecessary reminding can be reduced.

In a possible implementation manner, the generating a warning message based on the detection result of the steering wheel out-of-hand includes: and responding to the steering wheel hands-off detection result that the two hands are off the steering wheel, and generating reminding information.

In a possible implementation manner, the generating a warning message based on the detection result of the steering wheel out-of-hand includes: determining a driving state based on the steering wheel hands-off detection result when the automatic driving function is in an on state; and generating reminding information in response to the driving state being a dangerous state.

In this implementation, the autopilot function may represent a function by which the vehicle (e.g., vehicle) can reach a safe driving state without driver manipulation. In some vehicle products or application scenarios, the automatic driving function may also be referred to as a driving assistance function. The automatic driving function may be turned on or off based on an operation by the driver, and may also be turned on or off automatically.

In this implementation, when the automatic driving function is in the activated state, if it is determined that the driving state is a dangerous state based on the detection result of the steering wheel hands off, a warning message may be generated; if the driving state is determined to be a non-dangerous state based on the detection result of the hands-off of the steering wheel under the condition that the automatic driving function is in the turned-on state, the reminding information does not need to be generated.

In the implementation mode, whether the driving state is a dangerous state or not can be judged by combining the starting state of the automatic driving function, so that when the steering wheel hands-off detection method is applied to a vehicle with the automatic driving function, only necessary reminding can be performed under the condition that the automatic driving function is started, and the interference of excessive reminding on a driver is reduced.

As an example of this implementation, the determining a driving state based on the steering wheel hands-off detection result includes: determining a duration of time for which the driver's hand is disengaged from the steering wheel based on the steering wheel disengagement detection result; determining a driving state based on a length of time that the driver's hand is disengaged from the steering wheel.

In this example, the length of time that the driver's hands are disengaged from the steering wheel may be detected based on the video stream of the driving area. The time length for the driver's hand to be separated from the steering wheel can be determined based on the time difference between the first frame image and the last frame image for detecting that the driver's hand is separated from the steering wheel in the video stream of the continuous video clips of the driver's hand separated from the steering wheel state.

As another example of this implementation, the determining a driving state based on the steering wheel hands-off detection result includes: determining a driving state based on the steering wheel hands-off detection result and the driver's attention detection result.

In this example, attention detection may be performed on the driver based on a video stream of the driving area, and a result of the attention detection of the driver may be obtained. Wherein the attention detection may be used to detect whether the driver's attention is focused on driving, i.e. whether the driver is in a distracted state. In this example, the driving state is determined based on the length of time that the driver's hand is disengaged from the steering wheel and/or based on the driver's steering wheel disengagement detection result in combination with the driver's attention detection result, and the driving state thus determined can accurately reflect the current degree of risk.

In one example, the determining the driving state based on a length of time that the driver's hand is disengaged from the steering wheel includes: determining that the driving state is a dangerous state in response to a duration of disengagement of the driver's hand from the steering wheel reaching a first duration threshold. And determining a driving state based on the steering wheel hands-off detection result and the driver's attention detection result, including: and determining that the driving state is a dangerous state in response to the steering wheel hands-off detection result being that the hands of the driver are off the steering wheel and the attention detection result of the driver being that the driver is in a distracted state. For example, the first duration threshold may be 10 seconds. In this example, by determining that the driving state is a dangerous state in response to that the duration of time that the hand of the driver is disengaged from the steering wheel reaches the first duration threshold and/or the driver is in a distracted state as a result of the detection of the attention of the driver in the hands-off state of the steering wheel, and generating the warning information in response to that the driving state is the dangerous state, the warning information can be generated in a dangerous scene in which the hand of the driver is disengaged from the steering wheel for an excessively long time or the driver is distracted in the state in which the automatic driving function is turned on, so as to help reduce the potential safety hazard during driving.

In another example, the determining the driving state based on a length of time that the driver's hand is disengaged from the steering wheel and/or the driver's attention detection result includes: and determining that the driving state is a non-dangerous state in response to that the time length for which the hands of the driver are separated from the steering wheel does not reach a first time length threshold value and the attention detection result of the driver is attention concentration. According to this example, it is possible to allow the driver to leave the hand from the steering wheel for a short time with the automatic driving function turned on, but to request the driver to keep the attention, i.e., to keep the view of the driving environment. By determining that the driving state is a non-dangerous state in response to the duration of time that the driver's hands are disengaged from the steering wheel not reaching the first duration threshold and the driver's attention detection result being that the driver is attentive to driving (i.e., the driver is not distracted), unnecessary reminders can be reduced.

In one example, the automatic driving function may be turned off and reminder information may be generated to remind the driver to resume manual driving in response to the driving status being a dangerous status.

In a possible implementation manner, the generating a warning message based on the detection result of the steering wheel out-of-hand includes: and under the condition that the automatic driving function is in a non-opening state, responding to the detection result that the hands of the driver are separated from the steering wheel and the hand action of the driver belongs to dangerous action, and generating reminding information. In this implementation, the safety of the driver's operation is determined based on the hand operation in the automatic driving function off state, and the safety of the driving is further improved.

In a possible implementation manner, the generating the reminding information includes: and generating reminding information according to at least one of the number of hands which are separated from the steering wheel, the time length of the hands which are separated from the steering wheel and the hand action type of the driver.

As an example of this implementation, the reminder information may be generated according to the number of hands that are off the steering wheel. For example, the voice content in the reminder information corresponding to the two-handed disengagement from the steering wheel may include reminder content with a more urgent feeling than the voice content in the reminder information corresponding to the one-handed disengagement from the steering wheel.

As an example of this implementation, the reminder information may be generated according to a length of time that the hand is disengaged from the steering wheel. For example, the warning level of the reminder information may be positively correlated with the time length for which the hand is separated from the steering wheel, and the longer the time length for which the hand is separated from the steering wheel, the higher the warning level of the reminder information is.

As an example of this implementation, the reminding information may be generated according to the hand motion category of the driver. For example, if the hand motion category of the driver is to make a call, the reminder information may include "do not make a call while driving".

In this implementation, the driver can generate the warning information according to at least one of the number of hands that are separated from the steering wheel, the length of time that the hands are separated from the steering wheel, and the type of hand motion of the driver, so that the driver can be warned more specifically, which contributes to further improvement of the safety of driving.

As an example of this implementation, the generating of the reminder information according to at least one of the number of hands off the steering wheel, the length of time for which the hands are off the steering wheel, and the hand motion category of the driver includes: acquiring reminding animations and/or voices corresponding to the preset action types under the condition that the hand action types of the drivers are the preset action types; and generating control information for controlling a display screen to display the reminding animation, and/or generating reminding information for controlling a loudspeaker to play the voice.

In one example, the preset action category may include at least one of making a call, drinking water, smoking a cigarette, and the like. For example, the call-making corresponding reminding animation may include an animation of making a call while driving a car, the drink-drinking corresponding reminding animation may include an animation of drinking water while driving a car, the smoke-drawing corresponding reminding animation may include an animation of smoking while driving a car, and so on. For another example, the voice corresponding to making a call may include a voice of "do not call during driving", the voice corresponding to drinking may include a voice of "do not drink during driving", the voice corresponding to smoking may include a voice of "do not smoke during driving", and so on.

In this example, when the hand motion type of the driver is a preset motion type, a reminding animation and/or voice corresponding to the preset motion type is obtained, and control information for controlling a display screen to display the reminding animation is generated, and/or control information for controlling a loudspeaker to play the voice is generated, so that the driver can be reminded specifically visually and/or auditorily based on the hand motion of the driver, and therefore the reminding effect can be improved, and the driving safety can be improved.

In one possible implementation, the steering wheel hands-off detection result includes a steering wheel hands-off category; the generating of the reminding information based on the result of the detection of the steering wheel out of hand comprises: generating corresponding warning level reminding information according to the steering wheel hands-off category, wherein the steering wheel hands-off category comprises: the hands are not disengaged from the steering wheel, are disengaged from the steering wheel with one hand, or are disengaged from the steering wheel with both hands.

Because the danger degree that both hands break away from the steering wheel is usually higher than that one hand breaks away from the steering wheel, the warning level of the reminding information corresponding to the situation that both hands break away from the steering wheel can be higher than that of the reminding information corresponding to the situation that one hand breaks away from the steering wheel. For example, the frequency of the alert tone in the alert information corresponding to the two hands being off the steering wheel may be higher than the frequency of the alert tone in the alert information corresponding to the one hand being off the steering wheel. For another example, the duration of the alert sound in the alert information corresponding to the detachment of both hands from the steering wheel may be longer than the duration of the alert sound in the alert information corresponding to the detachment of one hand from the steering wheel. For another example, the repetition times of the prompt voice in the prompt information corresponding to the two-hand separation from the steering wheel may be greater than the repetition times of the prompt voice in the prompt information corresponding to the one-hand separation from the steering wheel. For example, the prompt voice in the prompt information corresponding to the two hands separating from the steering wheel may be repeated 3 times, and the prompt voice in the prompt information corresponding to the one hand separating from the steering wheel may be repeated 2 times. For another example, the blinking frequency of the indicator lights in the reminding information corresponding to the two hands being separated from the steering wheel may be higher than the blinking frequency of the indicator lights in the reminding information corresponding to the one hand being separated from the steering wheel.

In the implementation mode, the warning device can remind based on the danger degree corresponding to the steering wheel hands-off category, so that the reminded personnel can accurately know the warning level of the steering wheel hands-off category, and the driving safety is further improved.

In a possible implementation manner, the generating a warning message based on the detection result of the steering wheel out-of-hand includes: determining a dangerous state level corresponding to the driving state of the driver according to the detection result of the hands-off of the steering wheel; and generating reminding information corresponding to the dangerous state grade.

As one example of this implementation, the dangerous state levels may include a first dangerous state level and a second dangerous state level, wherein the first dangerous state level corresponds to a driving state having a higher degree of danger than the second dangerous state level.

In one example, the reminder information corresponding to the first dangerous state level may include a first warning tone, and the second dangerous state level may include a second warning tone, wherein the frequency of the first warning tone may be higher than the frequency of the second warning tone, and/or the duration of the first warning tone may be longer than the duration of the second warning tone.

In another example, the reminder information corresponding to the first dangerous state level may include a first voice, and the second dangerous state level may include a second voice, wherein the voice content of the first voice may be more urgent than the voice content of the second voice.

In another example, the reminder information corresponding to the first dangerous state level may include an instruction to control the indicator light to blink at a first frequency, and the reminder information corresponding to the second dangerous state level may include an instruction to control the indicator light to blink at a second frequency, wherein the first frequency may be higher than the second frequency.

Of course, in other examples, the number of the dangerous state levels may be more, for example, 3 or 4 dangerous state levels may be included, and the like, which is not limited herein.

In this implementation, the dangerous state level corresponding to the driving state of the driver is determined according to the detection result of the hands-off of the steering wheel, and the warning information corresponding to the dangerous state level is generated, so that the driver can be warned based on the dangerous degree corresponding to the driving state of the driver, and the safety of driving can be further improved.

In a possible implementation manner, the performing steering wheel hands-off detection on the driver based on the video stream of the driving area to obtain a steering wheel hands-off detection result includes: responding to the fact that the running speed of the vehicle to which the driving area belongs is larger than a first preset speed threshold value, and carrying out steering wheel hands-off detection on the driver on the basis of the video stream of the driving area to obtain a steering wheel hands-off detection result; and/or generating reminding information based on the detection result of the steering wheel out of hand, comprising the following steps: responding to the steering wheel hands-off detection result that the steering wheel is separated from the steering wheel by one hand, wherein the hand action of the driver belongs to dangerous action, and the running speed reaches a second preset speed threshold value, and generating reminding information; and/or generating reminding information based on the detection result of the steering wheel out of hand, comprising the following steps: and responding to the steering wheel hands-off detection result that the two hands are separated from the steering wheel, generating reminding information when the running speed reaches a third preset speed threshold value, wherein the third preset speed threshold value is not larger than the second preset speed threshold value.

In this implementation, when the running speed of the vehicle to which the driving area belongs is greater than a first preset speed threshold, the driver may be detected to be out of hand based on the video stream of the driving area, and a result of detecting out of hand of the steering wheel is obtained, so that the running safety can be improved; and under the condition that the running speed of the vehicle to which the driving area belongs is not greater than a first preset speed threshold value, the steering wheel hands-off detection is not carried out, so that unnecessary detection and reminding can be reduced. As an example of this implementation, the first preset speed threshold may be 0, and of course, the first preset speed threshold may also be flexibly set according to the actual application scenario requirement, which is not limited herein. In this example, in response to the traveling speed of the vehicle to which the driving region belongs being greater than 0, performing steering wheel hands-off detection on the driver based on the video stream of the driving region to obtain a steering wheel hands-off detection result; and under the condition that the running speed of the vehicle to which the driving area belongs is 0, the steering wheel hands-off detection is not carried out, so that unnecessary detection and occupation of redundant driving energy consumption and processing resources are avoided.

In this implementation manner, in response to that the detection result of the steering wheel hands off is that the steering wheel is left with one hand, the hand action of the driver belongs to a dangerous action, and the driving speed reaches a second preset speed threshold, a prompt message may be generated; and the detection result of the hand-off of the steering wheel is that the hand-off of the steering wheel is carried out by one hand, the hand action of the driver belongs to dangerous action, and the reminding information is not generated under the condition that the running speed does not reach the second preset speed threshold value. For example, the second preset speed threshold may be 30 km/h.

In this implementation manner, in response to that the detection result of the hands off the steering wheel is that both hands are off the steering wheel, and the driving speed reaches a third preset speed threshold, generating a reminding message; and the reminding information is not generated under the condition that the detection result of the hands-off of the steering wheel is that the two hands are off the steering wheel and the running speed does not reach a third preset speed threshold value. For example, the third preset speed threshold may be 10 km/h.

According to the implementation mode, the reminding can be reduced when the vehicle is in a low risk state with low driving speed, the reminding is pertinently started based on different speed thresholds when one hand is out of hand and two hands are out of hand, the potential safety hazard in driving is reduced, and meanwhile unnecessary detection and/or reminding are reduced.

In one possible implementation, the method further includes: according to the video stream of the driving area, carrying out fatigue detection on the driver to obtain a fatigue detection result; and generating reminding information based on the fatigue detection result. In this implementation, the fatigue detection result may include a fatigue state or a non-fatigue state. In this implementation, the reminder information may be generated in response to the fatigue state result being a fatigue state.

In this implementation, fatigue detection is performed on the driver according to the video stream of the driving area, so that steering wheel hands-off detection and fatigue detection can be performed respectively based on the video streams of the driving area acquired by the same or a same group of cameras, and thus the cost for arranging the cameras in the cabin of the vehicle can be reduced. For example, based on a set of drivers deployed in a cabin, the behavior detection system can realize the functions of detecting fatigue of the drivers and detecting hands off of a steering wheel, and different driver behavior detection functions are integrated in the same system, so that the detection cost is reduced. Further, by generating the warning information based on the fatigue detection result and the steering wheel release detection result, the safety of driving can be further improved.

In a possible implementation manner, the performing steering wheel hands-off detection on the driver based on the video stream of the driving area to obtain a steering wheel hands-off detection result includes: inputting at least one frame of image in the video stream of the driving area into a neural network, carrying out steering wheel hands-off detection on a driver through the neural network to obtain a steering wheel hands-off type corresponding to the video stream of the driving area, and determining the steering wheel hands-off type as a steering wheel hands-off detection result; or inputting at least one frame of image in the video stream of the driving area into a neural network, performing steering wheel hands-off detection on the driver through the neural network to obtain a steering wheel hands-off type corresponding to the video stream of the driving area and a hand action type of the driver, and determining a steering wheel hands-off detection result according to the steering wheel hands-off type and the hand action type of the driver. In the implementation mode, the steering wheel hands-off detection is carried out through the neural network, and the accuracy and the speed of the steering wheel hands-off detection can be improved. And the neural network can be trained and deployed in advance, and can quickly realize the detection of images or video streams with larger data volume, so that the neural network can be applied to the video stream detection of real-time driving scenes.

An application scenario of the embodiment of the present disclosure is explained below. In the application scene, a video stream of a driving area can be acquired through a DMS camera, a processor can acquire the video stream of the driving area from the DMS camera, detect whether the hand of the driver is separated from a steering wheel and the hand action of the driver based on the video stream of the driving area, obtain a steering wheel hands-off type and a hand action type of the driver corresponding to the video stream of the driving area, and determine a steering wheel hands-off detection result according to the steering wheel hands-off type and the hand action type of the driver. If the steering wheel hands-off detection result is that the single hand is away from the steering wheel to make a call, a reminding animation for making a call while driving can be displayed through the center console, and the voice of 'asking not to make a call in the driving process' is played.

It is understood that the above-mentioned method embodiments of the present disclosure can be combined with each other to form a combined embodiment without departing from the logic of the principle, which is limited by the space, and the detailed description of the present disclosure is omitted. Those skilled in the art will appreciate that in the above methods of the specific embodiments, the specific order of execution of the steps should be determined by their function and possibly their inherent logic.

In addition, the present disclosure also provides a steering wheel hands-off detection device, an electronic device, a computer-readable storage medium, and a program, which can be used to implement any one of the steering wheel hands-off detection methods provided by the present disclosure, and corresponding technical solutions and technical effects can be referred to in corresponding descriptions of the method sections and are not described again.

Fig. 2 shows a block diagram of a steering wheel hands-off detection apparatus provided in an embodiment of the present disclosure. As shown in fig. 2, the steering wheel hands-off detection device includes: the acquisition module 21 is used for acquiring a video stream of a driving area; the first detection module 22 is configured to perform steering wheel hands-off detection on the driver based on the video stream of the driving area to obtain a steering wheel hands-off detection result; and the first generating module 23 is configured to generate reminding information based on the detection result of the steering wheel out of hand.

In one possible implementation, the first detection module 22 is configured to: detecting whether the hands of the driver are separated from a steering wheel or not based on the video stream of the driving area, and obtaining a steering wheel separation type corresponding to the video stream of the driving area; and determining the steering wheel hands-off type as a steering wheel hands-off detection result.

In one possible implementation, the first detection module 22 is configured to: detecting whether the hands of the driver are separated from a steering wheel and/or the hand motions of the driver based on the video stream of the driving area, and obtaining a steering wheel separation type and/or a hand motion type of the driver corresponding to the video stream of the driving area; and determining a steering wheel hands-off detection result according to the steering wheel hands-off type and/or the hand action type of the driver.

In a possible implementation manner, the first generating module 23 is configured to: and generating reminding information in response to the fact that the hand-off detection result of the steering wheel is that the hands of the driver are off the steering wheel and the hand action of the driver belongs to dangerous action.

In a possible implementation manner, the first generating module 23 is configured to: determining a driving state based on the steering wheel hands-off detection result when the automatic driving function is in an on state; and generating reminding information in response to the driving state being a dangerous state.

In a possible implementation manner, the first generating module 23 is configured to: determining a duration of time for which the driver's hand is disengaged from the steering wheel based on the steering wheel disengagement detection result; determining a driving state based on a duration of time for which the driver's hand is disengaged from the steering wheel; and/or the first generating module is used for: determining a driving state based on the steering wheel hands-off detection result and the driver's attention detection result.

In a possible implementation manner, the first generating module 23 is configured to: determining that the driving state is a dangerous state in response to the time length for which the hands of the driver are disengaged from the steering wheel reaching a first time length threshold; and/or determining that the driving state is a dangerous state in response to the detection result that the hands of the steering wheel are disengaged from the steering wheel and the detection result of the attention of the driver is that the driver is in a distracted state.

In a possible implementation manner, the first generating module 23 is configured to: and under the condition that the automatic driving function is in a non-opening state, responding to the detection result that the hands of the driver are separated from the steering wheel and the hand action of the driver belongs to dangerous action, and generating reminding information.

In a possible implementation manner, the first generating module 23 is configured to: and generating reminding information according to at least one of the number of hands which are separated from the steering wheel, the time length of the hands which are separated from the steering wheel and the hand action type of the driver.

In a possible implementation manner, the first generating module 23 is configured to: acquiring reminding animation and/or voice of the preset action type under the condition that the hand action type of the driver is the preset action type; and generating control information for controlling a display screen to display the reminding animation and/or generating control information for controlling a loudspeaker to play the voice.

In one possible implementation, the steering wheel hands-off detection result includes a steering wheel hands-off category; the first generating module 23 is configured to: generating corresponding warning level reminding information according to the steering wheel hands-off category, wherein the steering wheel hands-off category comprises: the hands are not disengaged from the steering wheel, are disengaged from the steering wheel with one hand, or are disengaged from the steering wheel with both hands.

In a possible implementation manner, the first generating module 23 is configured to: determining a dangerous state level corresponding to the driving state of the driver according to the detection result of the hands-off of the steering wheel; and generating reminding information corresponding to the dangerous state grade.

In one possible implementation, the first detection module 22 is configured to: responding to the fact that the running speed of the vehicle to which the driving area belongs is larger than a first preset speed threshold value, and carrying out steering wheel hands-off detection on the driver on the basis of the video stream of the driving area to obtain a steering wheel hands-off detection result; and/or the first generating module 23 is configured to: responding to the steering wheel hands-off detection result that the steering wheel is separated from the steering wheel by one hand, wherein the hand action of the driver belongs to dangerous action, and the running speed reaches a second preset speed threshold value, and generating reminding information; and/or the first generating module 23 is configured to: and responding to the steering wheel hands-off detection result that the two hands are separated from the steering wheel, generating reminding information when the running speed reaches a third preset speed threshold value, wherein the third preset speed threshold value is not larger than the second preset speed threshold value.

In one possible implementation, the apparatus further includes: the second detection module is used for carrying out fatigue detection on the driver according to the video stream of the driving area to obtain a fatigue detection result; and the second generation module is used for generating reminding information based on the fatigue detection result.

In one possible implementation, the detection module is configured to: inputting at least one frame of image in the video stream of the driving area into a neural network, carrying out steering wheel hands-off detection on a driver through the neural network to obtain a steering wheel hands-off type corresponding to the video stream of the driving area, and determining the steering wheel hands-off type as a steering wheel hands-off detection result; or inputting at least one frame of image in the video stream of the driving area into a neural network, performing steering wheel hands-off detection on the driver through the neural network to obtain a steering wheel hands-off type corresponding to the video stream of the driving area and a hand action type of the driver, and determining a steering wheel hands-off detection result according to the steering wheel hands-off type and the hand action type of the driver.

In some embodiments, functions or modules included in the apparatus provided in the embodiments of the present disclosure may be used to execute the method described in the above method embodiments, and specific implementations and technical effects thereof may refer to the description of the above method embodiments, which are not described herein again for brevity.

Embodiments of the present disclosure also provide a computer-readable storage medium having stored thereon computer program instructions, which when executed by a processor, implement the above-described method. The computer-readable storage medium may be a non-volatile computer-readable storage medium, or may be a volatile computer-readable storage medium.

The embodiment of the present disclosure also provides a computer program, which includes computer readable code, and when the computer readable code runs in an electronic device, a processor in the electronic device executes the computer program to implement the method described above.

The disclosed embodiments also provide another computer program product for storing computer readable instructions, which when executed, cause a computer to perform the operations of the steering wheel hands-off detection method provided in any of the above embodiments.

An embodiment of the present disclosure further provides an electronic device, including: one or more processors; a memory for storing executable instructions; wherein the one or more processors are configured to invoke the memory-stored executable instructions to perform the above-described method.

The electronic device may be provided as a terminal, server, or other form of device. The electronic device may be a vehicle machine, a domain controller or a processor in a vehicle cabin, and may also be a device host for performing data processing operations such as images in a DMS or OMS.

Fig. 3 illustrates a block diagram of an electronic device 800 provided by an embodiment of the disclosure. For example, the electronic device 800 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, or the like terminal.

Referring to fig. 3, electronic device 800 may include one or more of the following components: processing component 802, memory 804, power component 806, multimedia component 808, audio component 810, input/output (I/O) interface 812, sensor component 814, and communication component 816.

The processing component 802 generally controls overall operation of the electronic device 800, such as operations associated with display, telephone calls, data communications, camera operations, and recording operations. The processing components 802 may include one or more processors 820 to execute instructions to perform all or a portion of the steps of the methods described above. Further, the processing component 802 can include one or more modules that facilitate interaction between the processing component 802 and other components. For example, the processing component 802 can include a multimedia module to facilitate interaction between the multimedia component 808 and the processing component 802.

The memory 804 is configured to store various types of data to support operations at the electronic device 800. Examples of such data include instructions for any application or method operating on the electronic device 800, contact data, phonebook data, messages, pictures, videos, and so forth. The memory 804 may be implemented by any type or combination of volatile or non-volatile memory devices such as Static Random Access Memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic or optical disks.

The power supply component 806 provides power to the various components of the electronic device 800. The power components 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the electronic device 800.

The multimedia component 808 includes a screen that provides an output interface between the electronic device 800 and a user. In some embodiments, the screen may include a Liquid Crystal Display (LCD) and a Touch Panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive an input signal from a user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundary of a touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 808 includes a front facing camera and/or a rear facing camera. The front camera and/or the rear camera may receive external multimedia data when the electronic device 800 is in an operation mode, such as a shooting mode or a video mode. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

The audio component 810 is configured to output and/or input audio signals. For example, the audio component 810 includes a Microphone (MIC) configured to receive external audio signals when the electronic device 800 is in an operational mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signals may further be stored in the memory 804 or transmitted via the communication component 816. In some embodiments, audio component 810 also includes a speaker for outputting audio signals.

The I/O interface 812 provides an interface between the processing component 802 and peripheral interface modules, which may be keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to: a home button, a volume button, a start button, and a lock button.

The sensor assembly 814 includes one or more sensors for providing various aspects of state assessment for the electronic device 800. For example, the sensor assembly 814 may detect an open/closed state of the electronic device 800, the relative positioning of components, such as a display and keypad of the electronic device 800, the sensor assembly 814 may also detect a change in the position of the electronic device 800 or a component of the electronic device 800, the presence or absence of user contact with the electronic device 800, orientation or acceleration/deceleration of the electronic device 800, and a change in the temperature of the electronic device 800. Sensor assembly 814 may include a proximity sensor configured to detect the presence of a nearby object without any physical contact. The sensor assembly 814 may also include a light sensor, such as a Complementary Metal Oxide Semiconductor (CMOS) or Charge Coupled Device (CCD) image sensor, for use in imaging applications. In some embodiments, the sensor assembly 814 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

The communication component 816 is configured to facilitate wired or wireless communication between the electronic device 800 and other devices. The electronic device 800 may access a wireless network based on a communication standard, such as a wireless network (Wi-Fi), a second generation mobile communication technology (2G), a third generation mobile communication technology (3G), a fourth generation mobile communication technology (4G)/long term evolution of universal mobile communication technology (LTE), a fifth generation mobile communication technology (5G), or a combination thereof. In an exemplary embodiment, the communication component 816 receives a broadcast signal or broadcast related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 further includes a Near Field Communication (NFC) module to facilitate short-range communications. For example, the NFC module may be implemented based on Radio Frequency Identification (RFID) technology, infrared data association (IrDA) technology, Ultra Wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

In an exemplary embodiment, the electronic device 800 may be implemented by one or more Application Specific Integrated Circuits (ASICs), Digital Signal Processors (DSPs), Digital Signal Processing Devices (DSPDs), Programmable Logic Devices (PLDs), Field Programmable Gate Arrays (FPGAs), controllers, micro-controllers, microprocessors or other electronic components for performing the above-described methods.

In an exemplary embodiment, a non-transitory computer-readable storage medium, such as the memory 804, is also provided that includes computer program instructions executable by the processor 820 of the electronic device 800 to perform the above-described methods.

Fig. 4 shows a block diagram of an electronic device 1900 provided by an embodiment of the disclosure. For example, the electronic device 1900 may be provided as a server. Referring to fig. 4, electronic device 1900 includes a processing component 1922 further including one or more processors and memory resources, represented by memory 1932, for storing instructions, e.g., applications, executable by processing component 1922. The application programs stored in memory 1932 may include one or more modules that each correspond to a set of instructions. Further, the processing component 1922 is configured to execute instructions to perform the above-described method.

The electronic device 1900 may also include a power component 1926 configured to perform power management of the electronic device 1900, a wired or wireless network interface 1950 configured to connect the electronic device 1900 to a network, and an input/output (I/O) interface 1958. The electronic device 1900 may operate based on an operating system, such as the Microsoft Server operating system (Windows Server), stored in the memory 1932^TM) Apple Inc. of the present application based on the graphic user interface operating System (Mac OS X)^TM) Multi-user, multi-process computer operating system (Unix)^TM) Free and open native code Unix-like operating System (Linux)^TM) Open native code Unix-like operating System (FreeBSD)^TM) Or the like.

In an exemplary embodiment, a non-transitory computer readable storage medium, such as the memory 1932, is also provided that includes computer program instructions executable by the processing component 1922 of the electronic device 1900 to perform the above-described methods.

The present disclosure may be systems, methods, and/or computer program products. The computer program product may include a computer-readable storage medium having computer-readable program instructions embodied thereon for causing a processor to implement various aspects of the present disclosure.

The computer readable storage medium may be a tangible device that can hold and store the instructions for use by the instruction execution device. The computer readable storage medium may be, for example, but not limited to, an electronic memory device, a magnetic memory device, an optical memory device, an electromagnetic memory device, a semiconductor memory device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium would include the following: a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a Static Random Access Memory (SRAM), a portable compact disc read-only memory (CD-ROM), a Digital Versatile Disc (DVD), a memory stick, a floppy disk, a mechanical coding device, such as punch cards or in-groove projection structures having instructions stored thereon, and any suitable combination of the foregoing. Computer-readable storage media as used herein is not to be construed as transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission medium (e.g., optical pulses through a fiber optic cable), or electrical signals transmitted through electrical wires.

The computer-readable program instructions described herein may be downloaded from a computer-readable storage medium to a respective computing/processing device, or to an external computer or external storage device via a network, such as the internet, a local area network, a wide area network, and/or a wireless network. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers and/or edge servers. The network adapter card or network interface in each computing/processing device receives computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in a computer-readable storage medium in the respective computing/processing device.

The computer program instructions for carrying out operations of the present disclosure may be assembler instructions, Instruction Set Architecture (ISA) instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider). In some embodiments, the electronic circuitry that can execute the computer-readable program instructions implements aspects of the present disclosure by utilizing the state information of the computer-readable program instructions to personalize the electronic circuitry, such as a programmable logic circuit, a Field Programmable Gate Array (FPGA), or a Programmable Logic Array (PLA).

Various aspects of the present disclosure are described herein with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the disclosure. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer-readable program instructions.

These computer-readable program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks. These computer-readable program instructions may also be stored in a computer-readable storage medium that can direct a computer, programmable data processing apparatus, and/or other devices to function in a particular manner, such that the computer-readable medium storing the instructions comprises an article of manufacture including instructions which implement the function/act specified in the flowchart and/or block diagram block or blocks.

The computer readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other devices to cause a series of operational steps to be performed on the computer, other programmable apparatus or other devices to produce a computer implemented process such that the instructions which execute on the computer, other programmable apparatus or other devices implement the functions/acts specified in the flowchart and/or block diagram block or blocks.

The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of instructions, which comprises one or more executable instructions for implementing the specified logical function(s). In some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

The computer program product may be embodied in hardware, software or a combination thereof. In an alternative embodiment, the computer program product is embodied in a computer storage medium, and in another alternative embodiment, the computer program product is embodied in a Software product, such as a Software Development Kit (SDK), or the like.

Having described embodiments of the present disclosure, the foregoing description is intended to be exemplary, not exhaustive, and not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen in order to best explain the principles of the embodiments, the practical application, or improvements made to the technology in the marketplace, or to enable others of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A steering wheel hands-off detection method is characterized by comprising the following steps:

acquiring a video stream of a driving area;

2. The method according to claim 1, wherein the detecting the hands-off steering wheel of the driver based on the video stream of the driving area to obtain a detection result of hands-off steering wheel comprises:

3. The method according to claim 1, wherein the detecting the hands-off steering wheel of the driver based on the video stream of the driving area to obtain a detection result of hands-off steering wheel comprises:

4. The method of claim 2 or 3, wherein the steering wheel hands-off category is hands-off steering wheel, one-handed off steering wheel, or two-handed off steering wheel.

5. The method according to any one of claims 1 to 4, wherein the generating of the reminding information based on the steering wheel hands-off detection result comprises:

6. The method according to any one of claims 1 to 4, wherein the generating of the reminding information based on the steering wheel hands-off detection result comprises:

7. The method of claim 6, wherein determining a driving state based on the steering wheel hands-off detection comprises:

8. The method of claim 7, wherein determining a driving state based on a length of time that the driver's hand is disengaged from a steering wheel comprises:

9. The method according to any one of claims 1 to 8, wherein generating a reminder message based on the steering wheel hands-off detection result comprises:

10. The method of any one of claims 1 to 9, wherein the generating of the reminder information comprises:

11. The method of claim 10, wherein generating the reminder information based on at least one of a number of hands off a steering wheel, a length of time a hand is off a steering wheel, and a hand action category of the driver comprises:

12. The method according to any one of claims 1 to 11, wherein the steering wheel hands-off detection result comprises a steering wheel hands-off category;

13. The method according to any one of claims 1 to 12, wherein generating a reminder message based on the steering wheel hands-off detection result comprises:

14. The method according to any one of claims 1 to 13,

and/or the presence of a gas in the gas,

15. The method according to any one of claims 1 to 14, further comprising:

and generating reminding information based on the fatigue detection result.

16. The method according to any one of claims 1 to 15, wherein the performing a hands-off steering wheel detection on the driver based on the video stream of the driving area to obtain a hands-off steering wheel detection result comprises:

alternatively, the first and second electrodes may be,

17. A steering wheel hands-off detection device, comprising:

the acquisition module is used for acquiring a video stream of a driving area;

18. An electronic device, comprising:

one or more processors;

a memory for storing executable instructions;

wherein the one or more processors are configured to invoke the memory-stored executable instructions to perform the method of any one of claims 1 to 16.

19. A computer readable storage medium having computer program instructions stored thereon, which when executed by a processor implement the method of any one of claims 1 to 16.