WO2022142786A1

WO2022142786A1 - Driving behavior recognition method, and device and storage medium

Info

Publication number: WO2022142786A1
Application number: PCT/CN2021/130751
Authority: WO
Inventors: 慕晨; 王春利; 郭红星; 周金星; 孙靖; 陈茹梦; 周萍萍; 赵璐; 张勇; 朱明明
Original assignee: 中兴通讯股份有限公司; 长安大学
Priority date: 2020-12-30
Filing date: 2021-11-15
Publication date: 2022-07-07
Also published as: CN114764912A

Abstract

A driving behavior recognition method, and a device and a storage medium. The driving behavior recognition method comprises: acquiring a plurality of frames of skeleton images corresponding to a driver (S10); according to the plurality of frames of skeleton images, determining action feature parameters corresponding to the driver (S20); and determining the current driving behavior state of the driver according to the action feature parameters (S30).

Description

Driving behavior recognition method, device and storage medium

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is based on the Chinese patent application with the application number of 202011631342.5 and the filing date of December 30, 2020, and claims the priority of the Chinese patent application. The entire content of the Chinese patent application is incorporated herein by reference.

technical field

The present application relates to the technical field of image processing, and in particular, to a driving behavior recognition method, device and storage medium.

Background technique

With the continuous development of the economic level, automobiles have become a common means of transportation, but a large number of traffic accidents have followed. Through the analysis of these accidents, it can be known that the driver's dangerous behavior has become the main cause of traffic accidents. Therefore, the behavioral norms of drivers are very important, and the demand for their behavior detection is also increasing. Existing methods for detecting driver behavior generally detect driver behavior by collecting and recognizing images, but this detection method is often affected by environmental factors such as lights, and the recognition accuracy is not high.

Therefore, how to improve the accuracy of recognizing the driving behavior of the driver has become an urgent problem to be solved.

SUMMARY OF THE INVENTION

The present application provides a driving behavior recognition method, device and storage medium.

In a first aspect, an embodiment of the present application provides a driving behavior recognition method, the method includes: acquiring multiple frames of skeletal images corresponding to the current posture of a driver; determining action features corresponding to the driver according to the multiple frames of the skeletal images parameter; according to the action characteristic parameter, determine the current driving behavior state of the driver.

In a second aspect, an embodiment of the present application further provides a driving behavior recognition device, the driving behavior recognition device includes a processor and a memory; the memory is used for storing a program; the processor is used for executing the program and The driving behavior recognition method as described above is realized when the program is executed.

In a third aspect, an embodiment of the present application further provides a storage medium for readable storage, where the storage medium stores one or more programs, and the one or more programs can be executed by one or more processors , in order to realize the above-mentioned driving behavior recognition method.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not limiting of the present application.

Description of drawings

In order to explain the technical solutions of the embodiments of the present application more clearly, the following briefly introduces the accompanying drawings used in the description of the embodiments. For those of ordinary skill, other drawings can also be obtained from these drawings without any creative effort.

1 is a schematic diagram of a driving behavior recognition system provided by an embodiment of the present application;

2 is a schematic diagram of part segmentation of a human body region provided by an embodiment of the present application;

3 is a schematic structural diagram of a driving behavior recognition device provided by an embodiment of the present application;

4 is a schematic flowchart of a driving behavior recognition method provided by an embodiment of the present application;

5 is a schematic diagram of a skeleton image provided by an embodiment of the present application;

6 is a schematic flowchart of a sub-step of determining action characteristic parameters corresponding to a driver provided by an embodiment of the present application;

7 is a schematic flowchart of a sub-step of determining a target distance and a target angle provided by an embodiment of the present application;

8 is a schematic diagram of a target distance provided by an embodiment of the present application;

9 is a schematic diagram of a target angle provided by an embodiment of the present application;

FIG. 10 is a schematic diagram of a state in which a driver is in a standard driving posture according to an embodiment of the present application.

Detailed ways

The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

The flowcharts shown in the figures are for illustration only, and do not necessarily include all contents and operations/steps, nor do they have to be performed in the order described. For example, some operations/steps can also be decomposed, combined or partially combined, so the actual execution order may be changed according to the actual situation.

It should be understood that the terms used in the specification of the present application herein are for the purpose of describing particular embodiments only and are not intended to limit the present application. As used in this specification and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural unless the context clearly dictates otherwise.

It will also be understood that, as used in this specification and the appended claims, the term "and/or" refers to and including any and all possible combinations of one or more of the associated listed items.

It should be understood that the specific embodiments described herein are only used to explain the present application, but not to limit the present application.

In the following description, suffixes such as 'module', 'component' or 'unit' used to represent elements are used only to facilitate the description of the present application, and have no specific meaning per se. Thus, "module", "component" or "unit" may be used interchangeably.

Embodiments of the present application provide a driving behavior recognition method, device, system, and storage medium. The driving behavior recognition method can be applied to a driving behavior recognition device. By recognizing the current driving behavior state of the driver according to the skeleton image, the influence of external environmental factors can be avoided, and the accuracy of detecting the driving behavior state of the driver can be improved.

In some examples, the driving behavior recognition device may include a server or a terminal. The server may be an independent server or a server cluster; the terminal may be an electronic device such as a smart phone, a tablet computer, a notebook computer, and a desktop computer.

Please refer to FIG. 1. FIG. 1 is a schematic diagram of a driving behavior recognition system provided by an embodiment of the present application. The behavior driving recognition system includes a driving behavior recognition device 10 , an image acquisition device 20 and a target device 30 .

Wherein, the driving behavior recognition device 10 may be connected with the image acquisition device 20 in wired or wireless communication, and the driving behavior recognition device 10 may be connected with the target device 30 in wireless communication. The image acquisition device 20 is configured to acquire a depth image including a driver, generate a skeleton image, and send the skeleton image to the driving behavior recognition device 10 . The driving behavior recognition device 10 is used to recognize and process the skeleton image sent by the image acquisition device 20 to determine the current driving behavior state corresponding to the driver; the driving behavior recognition device 10 can also send an early warning message to the target device 30 according to the current driving behavior state . The target device 30 is configured to receive the early warning message sent by the driving behavior recognition device 10, and issue an alarm according to the early warning message.

In some embodiments, the image collection device 20 may collect multiple frames of depth images including drivers, and perform human body recognition, body part recognition, and bone joint point positioning on each frame of depth image to obtain multiple frames of bone images. The driving behavior recognition device 10 can receive the multi-frame skeleton images sent by the image acquisition device 20, and determine the action feature parameters corresponding to the driver according to the multi-frame skeleton images; then, determine the current driving behavior state of the driver according to the action feature parameters.

In some examples, image acquisition device 20 may include a camera or a sensing device for acquiring depth images. For example, the image capturing device 20 may be a somatosensory sensor. In this embodiment of the present application, a somatosensory sensor may be used to acquire a depth image including a driver.

It should be noted that the somatosensory sensor may include a depth camera, a color camera, and a light source emitter, which can achieve depth information such as a depth image, a color image, and three-dimensional data information of a target object. In some examples, the working principle of obtaining depth information: the light emitted by the light source emitter is projected into the real scene. Since the emitted light will change due to the different surface shape of the target object, this light can be converted into After collecting and encoding, the distance difference between each pixel in the scene and the depth camera can be obtained, and then the position and depth information of the target object can be obtained.

In some embodiments, after collecting the depth image including the driver, the somatosensory can perform human body recognition on the depth image. For example, the background and people in the depth image are segmented according to a preset segmentation strategy to determine the body region or body contour information corresponding to the driver, and the obtained depth image includes the body region or body contour information corresponding to the driver. Then, the body sensor performs human body part recognition on the depth image obtained by the human body recognition.

Please refer to FIG. 2 . FIG. 2 is a schematic diagram of part segmentation of a human body region provided by an embodiment of the present application. As shown in Figure 2, the body region in the depth image is segmented to obtain multiple part images, such as head, arm, leg, limb, and torso images; The body parts corresponding to each part image. Finally, the body sensor locates the bone joint points of the depth image according to the body part, and obtains the bone image. In some examples, the identified body parts are added to the virtual skeleton model, and adjusted according to the position information of the body parts to obtain a skeleton image including multiple joint points.

In some examples, skeletal images may include, but are not limited to, joint points such as head joint points, neck joint points, hand joint points, elbow joint points, wrist joint points, shoulder joint points, and spine joint points.

Please refer to FIG. 3 , which is a schematic structural diagram of a driving behavior recognition device 10 provided by an embodiment of the present application. The driving behavior recognition device 10 may include a processor 11 and a memory 12, wherein the processor 11 and the memory 12 may be connected by a bus, such as an I2C (Inter-integrated Circuit) bus or any other suitable bus.

Wherein, the memory 12 may include a non-volatile storage medium and an internal memory. The nonvolatile storage medium can store operating systems and computer programs. The computer program includes program instructions that, when executed, can cause the processor to execute any driving behavior recognition method.

Wherein, the processor 11 is used to provide computing and control capabilities, and support the operation of the entire driving behavior recognition device 10 .

In one embodiment, the processor 11 is configured to run a computer program stored in the memory 12, and implement the following steps when executing the computer program:

Obtaining multiple frames of skeleton images corresponding to the current posture of the driver; determining action feature parameters corresponding to the driver according to the multiple frames of the skeleton images; determining the current driving behavior state of the driver according to the action feature parameters.

In one embodiment, when the processor 11 realizes the acquisition of multi-frame skeleton images corresponding to the current posture of the driver, it is used to realize:

Acquiring multiple frames of the skeleton images sent by the image acquisition device, wherein the multiple frames of the skeleton images are for the image acquisition device to perform human body recognition, human body part recognition and skeletal joint point positioning on the multiple frames of depth images including the driver generate.

In one embodiment, the action feature parameters include the target distance between the driver's head and the wrist, the target angle between the forearm and the rear arm, and the action duration of the driver's current posture; processing When the device 11 determines the action feature parameters corresponding to the driver according to the multiple frames of the skeleton images, it is used to realize:

The target distance, the target included angle and the action duration corresponding to the driver are determined according to the multiple frames of the skeleton images.

In one embodiment, when the processor 11 determines the current driving behavior state of the driver according to the action characteristic parameter corresponding to the driver, the processor 11 is configured to:

The current driving behavior state of the driver is determined according to the target distance, the target included angle and the action duration.

In one embodiment, before determining the target distance, the target angle and the action duration corresponding to the driver according to the multiple frames of the skeleton images, the processor 11 is further configured to:

According to a preset smoothing processing strategy, smoothing is performed on the initial multiple frames of the skeleton images to obtain the smoothed multiple frames of the skeleton images.

In one embodiment, when the processor 11 determines the target distance, the target angle and the action duration corresponding to the driver according to the multiple frames of the skeleton images, the processor 11 is configured to:

Extract the joint point information in each frame of the skeleton image after smoothing, and determine the target distance and the target angle corresponding to the driver according to the joint point information; determine the corresponding target distance and the target distance. The continuous multiple frames of the skeletal images where the target included angle satisfies a preset condition are used, and the action duration is determined according to the frame number and frame rate corresponding to the multiple continuous frames of the skeletal images.

In one embodiment, the skeleton image includes a head joint point, a neck joint point, an elbow joint point, a wrist joint point, a shoulder joint point, and a spine joint point; When determining the target distance and the target angle corresponding to the driver, it is used to achieve:

A three-dimensional space coordinate system is established according to the spine joint points and the neck joint points, and the head joint points, elbow joint points, wrist joint points, and shoulder joint points in the three-dimensional space coordinate system are respectively determined. joint point coordinates, elbow joint point coordinates, wrist joint point coordinates and shoulder joint point coordinates; determine the target distance according to the head joint point coordinates and the wrist joint point coordinates; The coordinates, the wrist joint point coordinates and the shoulder joint point coordinates determine the target angle.

In one embodiment, when determining the target angle according to the elbow joint point coordinates, the wrist joint point coordinates, and the shoulder joint point coordinates, the processor 11 is configured to:

Determine the first vector between the elbow joint point and the shoulder joint point according to the elbow joint point coordinates and the shoulder joint point coordinates; according to the elbow joint point coordinates and the wrist joint point joint point coordinates, determine the second vector between the elbow joint point and the wrist joint point; based on a preset vector dot product formula, determine the target according to the first vector and the second vector angle.

In one embodiment, before determining the current driving behavior state of the driver according to the target distance, the target angle and the action duration, the processor 11 is further configured to:

Obtain the reference distance between the head and the wrist and the reference angle between the forearm and the rear arm when the driver is in a standard driving posture; when the reference distance is in a preset distance range value, determine the The reference distance is a standard distance; when the reference included angle is within a preset first included angle range value, the reference included angle is determined to be a standard included angle.

In one embodiment, the current driving behavior state includes a dangerous driving behavior state, and the dangerous driving behavior state includes at least one of smoking while driving, making a phone call while driving, making a sharp turn, taking both hands off the steering wheel, and picking up objects; the processor 11 is in When determining the current driving behavior state of the driver according to the target distance, the target angle and the action duration, it is used to achieve:

If the target distance is less than the standard distance and less than a preset distance threshold, and the action duration is greater than or equal to a first time threshold, then according to the target angle, it is determined whether the current driving behavior state is driving smoke or drive a phone, wherein, the preset distance threshold is less than the standard distance; if the target distance is greater than the standard distance, according to the target distance, the standard distance, the target angle, The standard included angle and the action duration determine whether the current driving behavior state is one of sharp turning, hands off the steering wheel, and picking up items.

In one embodiment, after determining the current driving behavior state of the driver, the processor 11 is further configured to:

If the current driving behavior state is a dangerous driving behavior state, performing an early warning according to the dangerous driving behavior state, including sending an early warning message to a target device within a preset range and/or displaying the current vehicle early warning state on a navigation map, Wherein, the target device includes a vehicle and a mobile terminal carried by a user.

The processor 11 may be a central processing unit (Central Processing Unit, CPU), and the processor may also be other general-purpose processors, digital signal processors (Digital Signal Processors, DSP), application specific integrated circuits (application specific integrated circuits) circuit, ASIC), Field-Programmable Gate Array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.

Some embodiments of the present application will be described in detail below with reference to the accompanying drawings. The embodiments described below and features in the embodiments may be combined with each other without conflict.

As shown in FIG. 4 , FIG. 4 is a schematic flowchart of a driving behavior recognition method provided by an embodiment of the present application. The driving behavior recognition method can be applied to a driving behavior recognition device, realizes the recognition of the current driving behavior state of the driver according to the skeleton image, can avoid the influence of external environmental factors, and improves the accuracy of detecting the driving behavior state of the driver. The driving behavior recognition method includes steps S10 to S30.

Step S10 , acquiring multiple frames of skeleton images corresponding to the driver's current posture.

It should be noted that the driving behavior identification method provided by the embodiment of the present application can be applied to scenarios such as detecting whether a driver is driving safely, simulated driving in a virtual game, and an athlete's posture training. The embodiments of the present application will be described in detail by taking the detection of whether the driver drives safely as an example.

In some embodiments, acquiring multiple frames of skeletal images corresponding to the current posture of the driver may include: acquiring multiple frames of skeletal images sent by an image acquisition device, where the multi-frame skeletal images are depths of multiple frames including the driver from the image acquisition device. The image is used for human body recognition, human body part recognition and skeletal joint point location generation.

Wherein, the image acquisition device may be a somatosensory sensor.

It should be noted that, in the embodiment of the present application, the skeleton image corresponding to the driver may be acquired through the body sensor. In some examples, somatosensory sensors can be installed in the cab of a car to monitor the cab. For example, after the car is started, the somatosensory captures a depth image that includes the driver. The somatosensory sensor can perform human body recognition, human body part recognition and bone joint point positioning on the depth image through the built-in software development kit, and obtain multi-frame bone images corresponding to the driver's current posture.

In some examples, the body sensor includes a half-body mode and a full-body mode; wherein, the shooting distance corresponding to the half-body mode is 0.4 to 3 meters, and the shooting distance corresponding to the full-body mode is 0.8 to 4 meters. In the cab, since most of the driver's movements are performed on the upper body, the somatosensory sensor can be set to the half-body mode.

In some embodiments, after collecting the depth image including the driver, the somatosensory sensor may further include: performing format conversion on the depth data in the collected depth image according to a preset data format to obtain a format-converted depth image.

It should be noted that, in order to make the depth image have a better display effect, it is necessary to perform format conversion on the depth data in the acquired depth image.

In some examples, the preset data format may be Mat format. For example, the depth data corresponding to the acquired depth image is format-converted according to the Mat format, and the depth data corresponding to the format-converted depth image is in the Mat format. Then, the depth image after format conversion is processed such as human body recognition, human body part recognition and bone joint point positioning.

In some examples, the driving behavior recognition device may receive multiple frames of skeletal images sent by the somatosensory sensor.

Please refer to FIG. 5 , which is a schematic diagram of a skeleton image provided by an embodiment of the present application. As shown in Fig. 5, the skeleton image includes the skeleton information corresponding to the driver. The skeleton information may include different joint points and connection relationships between the joint points, and the like. In some examples, the joints may include head joints, neck joints, hand joints, elbow joints, wrist joints, shoulder joints, spine joints, and the like.

By receiving the multi-frame skeleton images sent by the somatosensory sensor, the dynamic actions of the driver can be detected and recognized, and the accuracy of recognizing the current driving behavior of the driver can be improved.

Step S20: Determine action characteristic parameters corresponding to the driver according to the multiple frames of the skeleton images.

It should be noted that, in this embodiment of the present application, the action feature parameters may include the target distance between the driver's head and the wrist, the target angle between the forearm and the rear arm, and the action duration of the driver's current posture.

Among them, the target distance and the target angle can be determined by the positional relationship corresponding to the joint points in the skeleton image; the action duration refers to the time the driver maintains the current posture.

By determining the action feature parameters corresponding to the driver according to the multi-frame skeleton images, since the skeleton images are not easily disturbed by the external environment and can be realized in a scene without lights, the accuracy of identifying the action feature parameters corresponding to the driver can be improved, and the Privacy leakage can be avoided.

In some embodiments, determining the action feature parameter corresponding to the driver according to the multi-frame skeleton images may include: determining the target distance, target angle and action duration corresponding to the driver according to the multi-frame skeleton images.

Please refer to FIG. 6 . FIG. 6 is a schematic flowchart of a sub-step of determining action characteristic parameters corresponding to a driver provided by an embodiment of the present application, which may specifically include steps S201 to S203 .

Step S201 , performing smoothing processing on the initial multiple frames of the skeleton images according to a preset smoothing processing strategy to obtain the smoothed multiple frames of the skeleton images.

It should be noted that step S201 is a step performed before determining the target distance, target angle and action duration corresponding to the driver according to the multi-frame skeleton images.

It should be noted that, in the embodiment of the present application, the computer vision platform needs to detect and recognize the skeleton image, so as to determine the current driving behavior state of the driver. In the process of detection and recognition, there are high real-time requirements for the driver's action feature parameters. If the skeleton image is not smoothed, it may cause the computer vision platform to shake or even collapse.

It can be understood that the initial multi-frame skeleton image refers to the skeleton image sent by the somatosensory sensor. Smoothing is also called filtering. By smoothing the skeleton image, not only the noise or distortion in the skeleton image can be reduced, but also the recognition efficiency can be improved.

In some examples, the preset smoothing processing strategy may include, but is not limited to, a mean filtering algorithm, a median filtering algorithm, a Gaussian filtering algorithm, a bilateral filtering algorithm, and the like.

For example, according to the mean filtering algorithm, the initial multi-frame skeleton image is smoothed to obtain the smoothed multi-frame skeleton image.

Step S202: Extract joint point information in each frame of the skeleton image after smoothing, and determine the target distance and the target angle corresponding to the driver according to the joint point information.

It should be noted that determining the target distance, target angle and action duration corresponding to the driver according to the multiple frames of skeleton images includes steps S202 and S203.

Please refer to FIG. 7 . FIG. 7 is a schematic flowchart of a sub-step of determining a target distance and a target angle corresponding to a driver according to an embodiment of the present application. Step S202 may include steps S2021 to S2023 .

Step S2021: Establish a three-dimensional space coordinate system according to the spine joint point and the neck joint point, and determine, respectively, that the head joint point, the elbow joint point, the wrist joint point, and the shoulder joint point are in the three-dimensional space coordinate system. The head joint point coordinates, the elbow joint point coordinates, the wrist joint point coordinates and the shoulder joint point coordinates.

Among them, the wrist joint point can be replaced with the hand joint point.

In some examples, the connection line between the spine joint point and the neck joint point can be used as the Z axis of the three-dimensional space coordinate system, the spine joint point can be used as the origin, and then the X axis and the Y axis are determined based on the Z axis, so as to establish the Three-dimensional space coordinate system.

In some embodiments, the head joint point, elbow joint point, wrist joint point, The head joint point coordinate, the elbow joint point coordinate, the wrist joint point coordinate and the shoulder joint point coordinate in the three-dimensional space coordinate system of the shoulder joint point.

In other embodiments, the coordinates corresponding to each joint point may be determined according to the projection of each joint point on the X axis, the Y axis, and the Z axis. The specific process of determining the coordinates of each joint point will not be repeated here.

Step S2022: Determine the target distance according to the head joint point coordinates and the wrist joint point coordinates.

It should be noted that the target distance refers to the distance between the head joint point and the wrist joint point. Please refer to FIG. 8 , which is a schematic diagram of a target distance provided by an embodiment of the present application.

In some examples, the head joint point coordinates can be represented as (x ₁ , y ₁ , z ₁ ); the wrist joint point coordinates can be represented as (x ₂ , y ₂ , z ₂ ); the target distance can be represented as B.

In this embodiment of the present application, the target distance B between the coordinates of the joint point of the head and the coordinates of the joint point of the wrist can be calculated by the Euclidean distance formula, as shown below:

Step S2023: Determine the target angle according to the coordinates of the elbow joint point, the wrist joint point coordinate and the shoulder joint point coordinate.

It should be noted that the target included angle refers to the angle between the line connecting the elbow joint point and the wrist joint point and the line connecting the elbow joint point and the shoulder joint point. Please refer to FIG. 9. FIG. 9 is a schematic diagram of a target angle provided by an embodiment of the present application.

In some embodiments, determining the target angle according to the coordinates of the elbow joint point, the coordinates of the wrist joint point, and the coordinates of the shoulder joint point may include: determining the elbow joint point according to the coordinates of the elbow joint point and the shoulder joint point coordinates. The first vector between the shoulder joint point and the elbow joint point; the second vector between the elbow joint point and the wrist joint point is determined according to the coordinates of the elbow joint point and the wrist joint point; based on the preset vector dot product formula, according to The first vector and the second vector determine the target angle.

In some examples, the preset vector dot product formula is:

In the formula,

and

represents two vectors; θ represents a vector

with vector

the angle between.

In some examples, the first vector of the line connecting the elbow joint point and the shoulder joint point can be determined according to the coordinates of the elbow joint point and the shoulder joint point, where the first vector can be expressed as

According to the coordinates of the elbow joint point and the wrist joint point, the second vector of the line connecting the elbow joint point and the wrist joint point is determined, wherein the second vector can be expressed as

In some examples, the target angle may be denoted D. Based on the above vector dot product formula, according to the first vector

with the second vector

The cosine value corresponding to the target angle D can be obtained:

Then the target angle D can be obtained as:

According to the head joint point coordinates, the elbow joint point coordinates, the wrist joint point coordinates and the shoulder joint point coordinates, the calculation is simple, and the target distance and target angle corresponding to the driver's current posture can be accurately determined.

Step S203: Determine the continuous multiple frames of the skeletal images whose corresponding target distance and the included angle between the targets satisfy a preset condition, and determine that the action continues according to the frame number and frame rate corresponding to the continuous multiple frames of the skeletal images. time.

It should be noted that the preset condition means that the target distance satisfies the set distance condition and the target angle satisfies the set angle condition. It can be understood that when the target distance corresponding to the driver's current posture satisfies the set distance condition and the target angle satisfies the set angle condition, the action duration corresponding to the driver's current posture is started to be recorded; when the action continues When the time meets the set time threshold, the current driving behavior state of the driver is determined according to the current posture.

In this embodiment of the present application, the action duration can be determined by using consecutive multiple frames of skeleton images.

In some embodiments, when it is determined that the corresponding target distance and the included angle between the target and the target meet the predetermined condition of the continuous multi-frame skeleton image, the action duration is determined according to the frame number and frame rate corresponding to the continuous multi-frame skeleton image.

In some examples, Frame rate=Frames/Time, and the unit of frame rate is frames per second. The frame rate can be set according to the actual situation, and the specific value is not limited here. The action duration can be denoted as T.

For example, if the number of frames corresponding to the consecutive multi-frame skeleton images that meet the preset conditions is 10 frames, and the frame rate is 50 frames per second, the action duration T can be determined to be 0.2 seconds.

By determining the continuous multi-frame skeleton images whose corresponding target distance and target angle meet the preset conditions, the action duration can be determined according to the frame number and frame rate corresponding to the continuous multi-frame skeleton images, so as to realize the real-time monitoring of the driver's current posture. monitor.

Step S30: Determine the current driving behavior state of the driver according to the action characteristic parameter.

In some embodiments, before determining the current driving behavior state of the driver according to the target distance, the target angle and the action duration, the method may further include: acquiring a reference between the head and the wrist of the driver when the driver is in a standard driving posture state Distance and reference angle between forearm and rear arm.

It should be noted that since different drivers have differences in body shape, it is necessary to set the standard distance and standard angle according to the actual body shape of each driver, so that the target distance and target angle can be compared with the standard distance and standard angle respectively. The included angles are compared, thereby improving the accuracy of identifying the current driving behavior state.

Please refer to FIG. 10 . FIG. 10 is a schematic diagram of a state in which a driver is in a standard driving posture according to an embodiment of the present application. As shown in FIG. 10 , before identifying the current driving behavior state of the driver, the driver may be prompted to maintain a standard driving posture. For example, prompt the driver to hold the steering wheel with both hands, lean on the seat with their upper body, and so on. Then, a skeleton image of the driver's standard driving posture state is obtained, and the reference distance between the driver's head and the wrist and the reference angle between the forearm and the rear arm are determined according to the skeleton image.

In some examples, the reference distance may be determined from the distance between the head joint point coordinates and the wrist joint point coordinates.

In some examples, the reference angle between the forearm and the rear arm may be determined according to the coordinates of the elbow joint point, the wrist joint point coordinate and the shoulder joint point coordinate. For the specific process, please refer to the detailed description of the above embodiment. The process is not repeated here.

In some examples, when the reference distance is within the preset distance range value, the reference distance is determined as the standard distance; when the reference included angle is within the preset first included angle range value, the reference included angle is determined as the standard included angle.

Wherein, the preset distance range value and the preset first included angle range value can be set according to the actual situation, and the specific value is not limited here.

In this embodiment of the present application, the distance measurement between the wrist and the head and the joint angle measurement may be performed on a preset number of test personnel. When the tester is in the standard driving posture, measure the distance between the wrist and the head, and obtain the distance range value (41.5cm-60.2cm); measure the angle between the line connecting the elbow and the shoulder and the line connecting the elbow and the wrist range, the value of the first included angle range is (100.5°-167.8°).

In some examples, when the reference distance is in a distance range value (41.5cm-60.2cm), the reference distance may be determined to be a standard distance. When the reference included angle is in the first included angle range value (100.5°-167.8°), it can be determined that the reference included angle is the standard included angle.

It can be understood that when the reference distance of the driver is within the distance range value and the reference included angle is within the first included angle range value, it means that the current posture of the driver is standard and safe, so the reference distance can be determined as Standard distance, the reference angle is determined as the standard included angle.

In some examples, the standard distance may be denoted as A, and the standard included angle may be denoted as C.

By acquiring the skeleton image of the driver's standard driving posture, and determining the reference distance between the driver's head and the wrist and the reference angle between the forearm and the rear arm according to the skeleton image, the standard distance and the standard clip can be determined. When determining the current driving behavior state of the driver, the target distance can be compared with the standard distance and the target angle can be compared with the standard angle, thereby improving the accuracy of identifying the current driving behavior state.

In some embodiments, determining the current driving behavior state of the driver according to the action characteristic parameters may include: determining the current driving behavior state of the driver according to the target distance, the target angle and the action duration.

It should be noted that the current driving behavior state may include a dangerous driving behavior state, wherein the dangerous driving behavior state includes at least one of smoking while driving, making a phone call while driving, turning sharply, taking hands off the steering wheel, and picking up objects.

By comprehensively identifying the current driving behavior state of the driver according to the target distance, the target angle and the action duration, the recognition accuracy can be improved.

In some embodiments, determining the current driving behavior state of the driver according to the target distance, the target angle, and the action duration may include: if the target distance is less than a standard distance and less than a preset distance threshold, and the action duration is greater than or is equal to the first time threshold, then according to the target angle, it is determined whether the current driving behavior state is smoking while driving or making a phone call while driving, wherein the preset distance threshold is smaller than the standard distance.

It should be noted that when the target distance B is less than the standard distance A, the driver's current driving behavior state may be smoking or talking while driving; when the target distance B is greater than or equal to the standard distance A, the driver's current driving behavior state It could be one of sharp turns, hands off the steering wheel, picking up items.

In some examples, when the target distance B is less than the standard distance A and less than a preset distance threshold, and the action duration T is greater than or equal to the first time threshold, then according to the target angle D, it is determined whether the current driving behavior state is driving and smoking Or make a phone call while driving.

The preset distance threshold and the first time threshold may be set according to actual conditions, and the specific values are not limited herein.

In some examples, the preset distance threshold may be 0.1m, and the first time threshold may be 0.8s, wherein the standard distance A is greater than the preset distance threshold of 0.1m.

In the embodiment of the present application, when the target distance B is smaller than the standard distance A and smaller than the preset distance threshold of 0.1m, and the action duration T is greater than or equal to the first time threshold of 0.8s, then according to the target angle D, determine the current Whether the driving behavior status is smoking while driving or making phone calls while driving.

It can be understood that, in this embodiment of the present application, the target distance may include the first target distance corresponding to the driver's left hand and the head and the second target distance corresponding to the right hand and the head of the driver, and the target angle may include The first target included angle corresponding to the driver's left arm and the second target included angle corresponding to the right arm.

In some examples, the target distance B may include a first target distance B1 and a second target distance B2; the target included angle D may include a first target included angle D1 and a second target included angle D2.

In some embodiments, when the first target angle or the second target angle is within a preset second angle range value, it is determined that the current driving behavior state is to make a phone call while driving.

Wherein, the preset second included angle range value can be set according to the actual situation, which is not uniquely limited here, for example, the second included angle range value is [0°, 5°). In some examples, when the first target angle D1 or the second target angle D2 is [0°, 5°), it may be determined that the current driving behavior state is to make a phone call while driving.

In other embodiments, when the first target angle or the second target angle is within a preset third angle range value, and the first target angle or the second target angle increases multiple times within the preset time period It is determined that the current driving behavior state is smoking while driving.

The preset third included angle range value may be set according to actual conditions, and is not uniquely limited here. For example, the third included angle range value is [5°, 10°]. The preset duration can be set according to the actual situation, and there is no unique limitation here. For example, the preset duration can be 10s.

In some examples, when the first target angle D1 or the second target angle D2 is [5°, 10°], and the first target angle D1 or the second target angle D2 is multiple times within the preset duration of 10s Increase and decrease, or when the first target distance B1 or the second target distance B2 increases and decreases multiple times, it may be determined that the current driving behavior state is smoking while driving.

It can be understood that when the driver is smoking, repeated actions generally occur within 10s. For example, the first target angle D1 or the second target angle D2 increases from [5°, 10°] to the standard angle. angle C, and then reduce from the standard angle C to [5°, 10°]; or, the first target distance B1 or the second target distance B2 is increased from the distance threshold 0.1m to the standard distance A, and then from the standard distance A Reduce to a distance threshold of 0.1m.

In other embodiments, determining the current driving behavior state of the driver according to the target distance, the target angle and the action duration may include: if the target distance is greater than the standard distance, The standard angle and action duration determine whether the current driving behavior is one of sharp turning, hands off the steering wheel, and picking up items.

In the embodiment of the present application, when it is determined that the target distance B is greater than the standard distance A, it can be determined whether the current driving behavior state is based on the target distance B, the standard distance A, the target angle D, the standard angle C and the action duration T. It is one of sharp turns, hands off the steering wheel, and picking up items.

In some embodiments, if the first target angle or the second target angle is greater than the standard angle and less than or equal to the preset angle threshold, and the action duration is less than or equal to the second time threshold, the current driving behavior state is determined to be Sharp turn.

Wherein, the second time threshold may be set according to the actual situation, which is not uniquely limited here. For example, the second time threshold may be 1s. The preset angle threshold may be set according to the actual situation, and the specific value is not limited herein. For example, the preset angle threshold may be 180°.

In some examples, when the target angle D is greater than the standard angle C and less than or equal to a preset angle threshold of 180°, and the action duration T is less than or equal to the second time threshold 1s, the current driving behavior state of the driver may be determined for sharp turns.

In other embodiments, if both the first target distance and the second target distance are greater than the standard distance, the first target angle and the second target angle are both greater than the standard angle, and the action duration is greater than or equal to the first time threshold , then it is determined that the current driving behavior state is that both hands are off the steering wheel.

In some examples, when the first target distance B1 and the second target distance B2 are both greater than the standard distance A, the first target angle D1 and the second target angle D2 are both greater than the standard angle C, and the action duration T is greater than or When it is equal to the first time threshold of 0.8s, it can be determined that the current driving behavior state of the driver is that both hands are off the steering wheel.

In other embodiments, if the first target distance or the second target distance is greater than the standard distance, the first target angle or the second target angle is equal to a preset angle threshold, and the action duration is greater than or equal to the first time threshold, Then it is determined that the current driving behavior state is picking up items.

In some examples, when the first target distance B1 or the second target distance B2 is greater than the standard distance A, the first target angle D1 or the second target angle D2 is equal to a preset angle threshold, and the action duration T is greater than or equal to the first When a time threshold is 0.8s, it can be determined that the current driving behavior state of the driver is picking up items.

In some embodiments, after determining the current driving behavior state of the driver, the method further includes: if the current driving behavior state is a dangerous driving behavior state, performing an early warning according to the dangerous driving behavior state, including sending an early warning to a target device within a preset range The message and/or the current warning state of the vehicle is displayed on the navigation map, wherein the target device includes the vehicle and the mobile terminal carried by the user.

In some examples, the mobile terminal may include, but is not limited to, electronic devices such as smart phones, tablet computers, notebook computers, personal digital assistants, and wearable devices.

In some examples, if the current driving behavior state is at least one of smoking while driving, making a phone call while driving, turning sharply, taking hands off the steering wheel, and picking up objects, an early warning is performed.

In some implementations, an alert message may be sent to target devices within a preset range. The preset range can be set according to the actual situation, and the specific value is not limited here. For example, the preset range can be 20 meters or 50 meters.

For example, if the current driving behavior state is smoking while driving, an early warning message is sent to a target device within a preset range.

In some examples, when sending an early warning message, a communication connection can be established with a vehicle and a mobile terminal carried by a user based on communication methods such as 4G, 5G, Bluetooth, Zigbee, and Wifi, and an early warning message is sent to the mobile terminal. The methods of sending the warning message may include but are not limited to text messages, phone calls, WeChat, emails, and the like. After receiving the warning message, the mobile terminal can give an alarm by means of light, sound and vibration to remind the user to pay attention to driving safety.

In some implementations, the current vehicle warning status is displayed on the navigation map.

It should be noted that the navigation map refers to the electronic map being used by vehicles or pedestrians at the current time.

In some examples, the warning status of the current vehicle may be marked on the navigation map and updated in real time. For example, the location of the current vehicle can be highlighted to remind other vehicles and pedestrians to pay attention to safety.

By sending a warning message to a target device within a preset range and/or displaying the warning status of the current vehicle on a navigation map when it is determined that the current driving behavior state is a dangerous driving behavior state, the occurrence of traffic accidents can be reduced.

The driving behavior recognition method, device, system and storage medium provided by the above embodiments can detect and recognize the dynamic actions of the driver by receiving the multi-frame skeleton images sent by the somatosensory sensor, thereby improving the recognition of the current driving behavior of the driver. The accuracy of the state; by determining the action feature parameters corresponding to the driver according to the multi-frame skeleton images, since the skeleton image is not easily disturbed by the external environment, and can be realized in the scene without lights, it can improve the recognition of the action feature parameters corresponding to the driver. It can also avoid privacy leakage; by smoothing the skeleton image, it can not only reduce the noise or distortion in the skeleton image, but also improve the recognition efficiency; The wrist joint point coordinates and shoulder joint point coordinates are simple to calculate, and can accurately determine the target distance and target angle corresponding to the driver's current posture; by determining the corresponding target distance and target angle to meet the preset conditions for multiple consecutive frames Skeletal images, the duration of the action can be determined according to the number of frames and frame rates corresponding to consecutive multi-frame skeletal images, and the current posture of the driver can be monitored in real time; The image determines the reference distance between the driver's head and the wrist and the reference angle between the forearm and the rear arm, so that the standard distance and standard angle can be determined. When determining the driver's current driving behavior, the The target distance is compared with the standard distance and the target angle is compared with the standard angle, so as to improve the accuracy of identifying the current driving behavior; by comprehensively identifying the driver's current driving according to the target distance, target angle and action duration Behavior status, which can improve the accuracy of recognition; when it is determined that the current driving behavior status is a dangerous driving behavior status, sending an early warning message to the target device within a preset range and/or displaying the current vehicle early warning status on the navigation map, you can Reduce traffic accidents.

The embodiments of the present application further provide a storage medium for readable storage, the storage medium stores a program, the program includes program instructions, and the processor executes the program instructions to implement the embodiments of the present application Any of the provided driving behavior recognition methods.

For example, the program is loaded by the processor and can perform the following steps:

Wherein, the storage medium may be an internal storage unit of the driving behavior recognition device described in the foregoing embodiments, such as a hard disk or a memory of the driving behavior recognition device. The storage medium may also be an external storage device of the driving behavior recognition device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital card (Secure Digital Card) equipped on the driving behavior recognition device. Card, SD Card), Flash Card (Flash Card), etc.

The embodiments of the present application disclose a driving behavior recognition method, device, and storage medium. By acquiring multiple frames of skeleton images corresponding to the driver's current posture, and determining action feature parameters corresponding to the driver according to the multiple frames of skeleton images, external environmental factors can be avoided. The action characteristic parameters can be determined more accurately; by determining the current driving behavior state of the driver according to the action characteristic parameters, the accuracy of detecting the driving behavior state of the driver can be improved.

Those of ordinary skill in the art can understand that all or some of the steps in the methods disclosed above, functional modules/units in the systems, and devices can be implemented as software, firmware, hardware, and appropriate combinations thereof.

In a hardware implementation, the division between functional modules/units mentioned in the above description does not necessarily correspond to the division of physical components; for example, one physical component may have multiple functions, or one function or step may be composed of several physical components Components execute cooperatively. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, digital signal processor or microprocessor, or as hardware, or as an integrated circuit, such as an application specific integrated circuit . Such software may be distributed on storable media, which may include computer storage media (or non-transitory media) and communication media (or transitory media). As is known to those of ordinary skill in the art, the term storage medium includes both volatile and nonvolatile implemented in any method or technology for storage of information, such as computer readable instructions, data structures, program modules or other data , removable and non-removable media. Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical disk storage, magnetic cartridges, magnetic tape, magnetic disk storage or other magnetic storage devices, or may Any other medium used to store desired information and which can be accessed by a computer. In addition, communication media typically embodies computer readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and can include any information delivery media, as is well known to those of ordinary skill in the art .

Some embodiments of the present application have been described above with reference to the accompanying drawings, which are not intended to limit the scope of the right of the present application. Any modifications, equivalent substitutions and improvements made by those skilled in the art without departing from the scope and essence of the present application shall fall within the scope of the right of the present application.

Claims

A driving behavior recognition method, comprising:

Obtain multi-frame skeleton images corresponding to the driver's current posture;

Determine the action feature parameters corresponding to the driver according to the multiple frames of the skeleton images;

According to the action characteristic parameter, the current driving behavior state of the driver is determined.
The driving behavior recognition method according to claim 1, wherein the acquiring the multi-frame skeleton images corresponding to the current posture of the driver comprises:

Acquiring multiple frames of the skeleton images sent by the image acquisition device, wherein the multiple frames of the skeleton images are for the image acquisition device to perform human body recognition, human body part recognition and skeletal joint point positioning on the multiple frames of depth images including the driver generate.
The driving behavior recognition method according to claim 1, wherein the action characteristic parameters include the target distance between the driver's head and the wrist, the target angle between the forearm and the rear arm, and the driver's The action duration of the current posture; the determining the action feature parameters corresponding to the driver according to the multiple frames of the skeleton images, including:

Determine the target distance, the target angle and the action duration corresponding to the driver according to the multiple frames of the skeleton images;

The determining of the current driving behavior state of the driver according to the action characteristic parameters corresponding to the driver includes:

The current driving behavior state of the driver is determined according to the target distance, the target included angle and the action duration.
The driving behavior recognition method according to claim 3, wherein before the target distance, the target included angle and the action duration corresponding to the driver are determined according to the multiple frames of the skeleton images, the method further comprises: :

According to a preset smoothing processing strategy, smoothing the initial multiple frames of the skeleton images to obtain the smoothed multiple frames of the skeleton images;

The determining of the target distance, the target included angle and the action duration corresponding to the driver according to the multiple frames of the skeleton images includes:

extracting joint point information in each frame of the skeleton image after smoothing, and determining the target distance and the target angle corresponding to the driver according to the joint point information;

Determining that the corresponding target distance and the included angle between the target meet a preset condition of the continuous multiple frames of the skeleton image, and determine the action duration according to the frame number and frame rate corresponding to the multiple continuous frames of the skeleton image.
The driving behavior recognition method according to claim 4, wherein the skeleton image includes head joint points, neck joint points, elbow joint points, wrist joint points, shoulder joint points and spine joint points; The joint point information determines the target distance and the target angle corresponding to the driver, including:

A three-dimensional space coordinate system is established according to the spine joint points and the neck joint points, and the head joint points, elbow joint points, wrist joint points, and shoulder joint points in the three-dimensional space coordinate system are respectively determined. Joint point coordinates, elbow joint point coordinates, wrist joint point coordinates and shoulder joint point coordinates;

Determine the target distance according to the head joint point coordinates and the wrist joint point coordinates;

The target included angle is determined according to the coordinate of the elbow joint point, the coordinate of the wrist joint point and the coordinate of the shoulder joint point.
The driving behavior recognition method according to claim 5, wherein the determining the target angle according to the coordinates of the elbow joint point, the wrist joint point coordinate and the shoulder joint point coordinate comprises:

Determine the first vector between the elbow joint point and the shoulder joint point according to the elbow joint point coordinate and the shoulder joint point coordinate;

Determine a second vector between the elbow joint point and the wrist joint point according to the elbow joint point coordinate and the wrist joint point coordinate;

Based on a preset vector dot product formula, the target angle is determined according to the first vector and the second vector.
The driving behavior identification method according to claim 3, wherein before determining the current driving behavior state of the driver according to the target distance, the target angle and the action duration, the method further comprises: :

obtaining the reference distance between the head and the wrist and the reference included angle between the forearm and the rear arm when the driver is in a standard driving posture;

When the reference distance is within a preset distance range value, determine that the reference distance is a standard distance;

When the reference included angle is within a preset first included angle range value, the reference included angle is determined to be a standard included angle.
The driving behavior identification method according to claim 7, wherein the current driving behavior state includes a dangerous driving behavior state, and the dangerous driving behavior state includes smoking while driving, making a phone call while driving, turning sharply, taking hands off the steering wheel, picking up items At least one; the determining the current driving behavior state of the driver according to the target distance, the target angle and the action duration, including:

If the target distance is less than the standard distance and less than a preset distance threshold, and the action duration is greater than or equal to a first time threshold, then according to the target angle, it is determined whether the current driving behavior state is driving smoking or making a phone call while driving, wherein the preset distance threshold is less than the standard distance;

If the target distance is greater than the standard distance, then according to the target distance, the standard distance, the target angle, the standard angle and the action duration, determine whether the current driving behavior state is One of sharp turns, hands off the steering wheel, and picking up items.
The driving behavior identification method according to any one of claims 1-8, wherein after the determining the current driving behavior state of the driver, the method further comprises:

If the current driving behavior state is a dangerous driving behavior state, performing an early warning according to the dangerous driving behavior state, including sending an early warning message to a target device within a preset range and/or displaying the current vehicle early warning state on a navigation map, Wherein, the target device includes a vehicle and a mobile terminal carried by a user.
A driving behavior recognition device, comprising:

memory for storing programs;

The processor is configured to execute the program and implement the driving behavior recognition method according to any one of claims 1 to 9 when the program is executed.
A storage medium for readable storage, wherein the storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to realize the invention as claimed in claims 1 to 9 The driving behavior recognition method of any one.