CN115063849A - Dynamic gesture vehicle control system and method based on deep learning - Google Patents

Abstract

A dynamic gesture vehicle control system and method based on deep learning relates to the technical field of deep learning, solves the problems of low accuracy and slow recognition speed of existing dynamic gesture recognition, and can be applied to mid-to-high-end vehicle control systems. The system includes a vehicle-mounted terminal and a cloud platform; the vehicle-mounted terminal includes a face recognition module, a gesture recognition module and a vehicle control module, and each module realizes communication connection through Ethernet; the cloud platform includes an identity authentication module, and the identity authentication module is connected to the The vehicle terminal is connected through the ACP protocol. The dynamic gesture car control method includes: collecting and recognizing a static face image through a high-precision camera, and uploading the face signal to the cloud through the ACP protocol for identity verification by the vehicle communication terminal; capturing the user's dynamic gesture image, and analyzing and calculating in real time through the gesture recognition module Accurate gesture signal is generated; it is transmitted to the on-board computing unit through Ethernet for fusion processing, and sent to the wire-controlled unit to control the car inside and outside the car.

Description

A deep learning-based dynamic gesture car control system and method

technical field

The invention relates to the technical field of deep learning, in particular to a dynamic gesture car control technology based on deep learning.

Background technique

With economic development and the continuous improvement of people's living standards, the number of vehicle sales continues to rise, and the parking space resources are intensified. Important configuration. Therefore, the gesture car control method has become one of the important directions for major car manufacturers to pursue innovation.

At present, for the vehicle external control vehicle system, it only stays at the level of intelligent terminals, such as mobile APP, etc. Terminal control often relies on the cloud to issue control commands. Under the premise of low system performance optimization, there are long remote control links and resources. Defects such as high consumption, poor user experience, timeout, and abnormal instructions. In addition, for in-vehicle control, compared with the touch button method, the gesture recognition method can bring users a better human-computer interaction experience, ensure that users concentrate on driving the vehicle, and achieve three-dimensional spatial control instead of two-dimensional plane control. However, the current dynamic gesture emptying technology has low accuracy in gesture recognition, and the recognition speed is relatively slow, which seriously affects the user experience.

SUMMARY OF THE INVENTION

In order to solve the problems of low accuracy and slow recognition speed of dynamic gesture recognition in the prior art, the present invention proposes a deep learning-based dynamic gesture vehicle control system and method.

The technical scheme of the present invention is as follows:

A dynamic gesture vehicle control system based on deep learning, comprising a vehicle terminal and a cloud platform; the vehicle terminal includes a face recognition module, a gesture recognition module and a vehicle control module, and each module is connected through Ethernet for communication; the cloud platform Including an identity authentication module, the identity authentication module is connected with the vehicle terminal through the ACP protocol;

The gesture recognition module includes: a depth separable convolutional neural network module, a gesture action positioning module, a loss function calculation module, a channel attention module and a dynamic gesture database;

The dynamic gesture database is used to capture and store the user's common gesture information. The loss function calculation module cooperates with the gesture action positioning module to make the prediction frame fit the real frame more accurately. The channel attention module is used to output feature detection results, and the depthwise separable convolution The neural network module performs in-depth training on user gesture information to achieve fast feature extraction and recognition of user gesture operations.

Preferably, the face recognition module includes: a cascaded convolutional neural network, a deep convolutional neural network, a metric module, a normalization calculation module and a loss function calculation module;

The face recognition module extracts the image information of the target face from the static face image collected by the camera through the cascaded convolutional neural network, and then extracts the depth feature vector in the face image through the deep convolutional neural network module, The measurement module judges the correlation degree of the feature vector, and finally calculates and integrates the accurate face data through the normalization calculation module and the loss function calculation module of the face feature data, and compares and authenticates with the cloud face database, and finally realizes the face purpose of identification.

Preferably, the depthwise separable convolutional neural network module uses depthwise convolution and 1×1 convolution to perform deep fusion, and the first and last steps of the depthwise separable convolutional neural network module both use 1×1 convolution The intermediate step uses the ResNet feature fusion concept to fuse shallow features, and reduces the amount of network parameters by compressing the number of channels.

Preferably, the loss function calculation module is used to calculate the similarity loss of the respective aspect ratios of the detection frame and the real frame, and the specific calculation method is:

in,

b and b ₁ represent the center point of the detection frame and the real frame, respectively, ρ is the Euclidean distance, c is the distance between the farthest vertices between the detection frame and the real frame, IoU represents the intersection ratio of b and b ₁ , x and y represent the width and height of the detection frame, respectively, and x ₁ and y ₁ represent the width and height of the real frame, respectively;

The gesture action localization module loss replaces the bounding box prediction loss term in the traditional localization algorithm. The improved loss function L consists of three parts: error, confidence error and classification error:

L=L _C +L _con +L _s ,

in,

代表目标物是否落入第i个网格第j个边界框，

代表没有落入，

代表已经落入；该网络采用13★13和26★26尺度特征融合，因此s²＝13²和s²＝26²，B＝2。where s ² represents the number of grids, knife represents the number of bounding boxes,

Represents whether the target falls into the jth bounding box of the ith grid,

represents not falling,

The representatives have fallen in; the network uses 13★13 and 26★26 scale feature fusion, so s ² =13 ² and s ² =26 ² , B=2.

Preferably, the channel attention module adopts dual channels. After the output of two different scales of the dual channels, the channel attention module is added. By assigning different weights, the feature outputs of the two different scales are suppressed by non-maximum value. the final test results.

Preferably, the identity authentication module is a digital signature authentication system based on a public key cryptosystem. Face images are collected through high-precision cameras inside and outside the vehicle. After preprocessing, the image data is transmitted to the vehicle-mounted communication terminal TBOX via a CAN line. The data is sent to the cloud platform TSP for initial face image storage and a unique identification code is generated, and the initial digital certificate is applied to the authentication system PKI. The module obtains and issues a digital certificate for the user from the certificate store, and then binds the user's identity information and the user's public key in the form of a digital certificate, thereby realizing user identity authentication.

Preferably, the vehicle control module includes a vehicle-mounted high-precision camera, a network communication module, a vehicle-mounted computing unit and a wire-controlled unit; the vehicle-mounted high-precision camera extracts an accurate gesture image signal through a gesture recognition algorithm, and transmits it to the real-time through the communication network module. The on-board computing unit filters and calculates the gesture signal, and integrates other anti-collision signals perceived by the sensor, and sends it to the wire control unit to realize the gesture control of the vehicle.

Preferably, the control of the vehicle by the gesture includes an external control of the vehicle and an in-vehicle control, the external control includes reversing, braking and steering, and the in-vehicle control includes an in-vehicle electronic seat, an in-vehicle screen, and an in-vehicle control. Internal audio and ambient lighting controls.

A deep learning-based dynamic gesture car control method, applying the above dynamic gesture car control system, the dynamic gesture car control method includes the following steps:

S1. A static face image is collected and recognized by a high-precision camera, and the in-vehicle communication terminal uploads the face signal to the cloud through the ACP protocol for identity verification;

S2. After the identity verification is passed, the dynamic gesture image of the user is captured, and the accurate gesture signal is calculated and analyzed in real time by the gesture recognition module.

S3. The gesture signal is transmitted to the in-vehicle computing unit through the Ethernet for fusion processing, and then sent to the wire-controlled unit to control the vehicle inside and outside the vehicle.

Compared with the prior art, the present invention solves the problems of low dynamic gesture recognition accuracy and slow recognition speed, and the specific beneficial effects are:

1. The deep learning-based dynamic gesture car control system provided by the present invention, in view of the current shortage of parking space resources, there is a scene where the user cannot enter the car in the effective space, and solves the problem that the user realizes the function of parking in and out of the vehicle by gestures outside the car, In the narrow parking scene, the vehicle can be controlled to park in and out; and in the in-vehicle scene, the dynamic gesture recognition technology is used to combine the gesture recognition ability and the in-vehicle wire control ability, without touching the instrument or the car screen to achieve control, which ensures the owner of the car. While concentrating on driving the vehicle, the human-computer interaction experience is greatly improved.

2. The deep learning-based dynamic gesture car control system provided by the present invention, in a complex background, through the convolutional neural network algorithm, the network model overhead is reduced at the terminal, and the deep learning-based video gesture action positioning algorithm is used to achieve dynamic The accuracy of gesture recognition is improved, and a dynamic gesture database is established for the training of gesture models. Through algorithms and means such as convolutional neural networks, loss functions, and channel attention, the premise of ensuring real-time performance, terminal capabilities, and user scenarios is guaranteed. In the next step, the accuracy of gesture recognition is improved, and the user-customized dynamic gesture database is entered. After in-depth training, the user's simple gesture operations can be recognized more quickly, so as to fit the user's gesture-controlled car scene.

3. The deep learning-based dynamic gesture car control method provided by the present invention combines face recognition and dynamic gesture recognition to provide low-complexity and accurate image signals, and also combines a cloud authentication system and a vehicle-end car control system , realizes the whole gesture control system in the true sense, improves the whole human-computer interaction performance, and ensures the safe driving of users.

Description of drawings

1 is a schematic structural diagram of the dynamic gesture vehicle control system according to Embodiment 1;

2 is a schematic structural diagram of a face recognition module according to Embodiment 2 of the present invention;

3 is a schematic structural diagram of gesture recognition according to Embodiment 1 of the present invention;

4 is a schematic structural diagram of a separate convolutional neural network module according to Embodiment 3 of the present invention;

5 is a schematic structural diagram of the channel attention module according to Embodiment 5 of the present invention;

6 is a schematic structural diagram of an identity authentication module according to Embodiment 6 of the present invention;

FIG. 7 is a schematic structural diagram of a vehicle control module according to Embodiment 7 of the present invention.

Detailed ways

In order to make the technical solutions of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings of the description of the present invention. It should be noted that the following embodiments are only used for better understanding The technical solutions of the present invention should not be construed as limitations of the present invention.

Example 1.

This embodiment provides a deep learning-based dynamic gesture vehicle control system, the schematic diagram of which is shown in Figure 1, including a vehicle-mounted terminal and a cloud platform; the vehicle-mounted terminal includes a face recognition module, a gesture recognition module, and a vehicle control module. The module realizes communication connection through Ethernet; the cloud platform includes an identity authentication module, and the identity authentication module is connected with the vehicle terminal through ACP protocol.

This embodiment provides a dynamic gesture car control system. The system combines face recognition and dynamic gesture recognition to provide low-complexity and accurate image signals. In addition, it combines a cloud authentication system and a car-end car control system to achieve real In the sense of the whole gesture control system, the whole human-computer interaction performance is improved, and the user can drive safely.

The gesture recognition module includes: a depth separable convolutional neural network module, a gesture action positioning module, a loss function calculation module, a channel attention module and a dynamic gesture database;

The dynamic gesture database is used to capture and store the user's common gesture information. The loss function calculation module cooperates with the gesture action positioning module to make the prediction frame fit the real frame more accurately. The channel attention module models the relationship between different feature channels, and the depth The separable convolutional neural network module reduces network overhead, and realizes fast feature extraction and recognition of user gesture operations after deep training.

FIG. 3 is a schematic structural diagram of the gesture recognition module provided in this embodiment, wherein the depth-separable convolutional neural network module reduces network expenses, greatly improves network processing speed and recognition accuracy, ensures system real-time performance, and is beneficial to mobile terminals. and embedded devices. The gesture action localization module solves the defects of the traditional localization method that the gradient is 0 and different objects cannot be aligned. It considers the distance information between the center point of the real frame and the detection frame, and adds the similarity of the respective aspect ratios of the detection frame and the real frame. The loss term is used to make the predicted box fit the real box more accurately.

The dynamic gesture database can better fit the actual gesture control scene, and use the camera to collect gesture images and background data, where the background data includes data such as light intensity and angle. In addition, the dynamic gesture database is differentiated in the time dimension to meet the operation habits of different users. The user's common gesture information is captured through the database, and the recognition accuracy can be improved through in-depth training. When the network is abnormal, the recognition can still be extracted from the library. This embodiment uses algorithms and means such as convolutional neural network, loss function, channel attention, etc., under the premise of ensuring real-time performance, terminal capability, and conformity to user scenarios, etc., to improve gesture recognition accuracy, and enter a user-customized dynamic gesture database. After in-depth training, the user's simple gesture operations can be recognized more quickly to fit the user's gesture-controlled car scene.

Example 2.

This embodiment is a further illustration of Embodiment 1, and the face recognition module includes: a cascaded convolutional neural network, a deep convolutional neural network, a metric module, a normalization calculation module and a loss function calculation module;

2 is a schematic structural diagram of a face recognition module provided by the present embodiment. The face recognition module is based on the facenet Pearson discriminant network. The network extracts the image information of the target face, and then the deep convolutional neural network module extracts the deep feature vector in the face image, and the metric module determines the correlation degree of the feature vector, and finally calculates the face feature data by normalizing it. The module and the loss function calculation module calculate and integrate the accurate face data, compare and authenticate with the cloud face database, and finally realize the purpose of face recognition.

Example 3.

This embodiment is a further illustration of Embodiment 1. The depthwise separable convolutional neural network module uses depthwise convolution and 1×1 convolution to perform depth fusion. Both the first step and the last step use 1×1 convolution, and the intermediate step uses the ResNet feature fusion concept to fuse shallow features, and reduces the amount of network parameters by compressing the number of channels.

The schematic diagram of the deeply separable convolutional neural network module in this embodiment is shown in Figure 4. The first step is to increase the dimension of the input features to avoid feature loss due to nonlinear activation. In fusion, the number of network parameters is reduced by compressing the number of channels, and the number of features to be fused is reduced in the last step. This embodiment reduces the computational degree of the system on the premise of not affecting the network performance.

Example 4.

This embodiment is a further illustration of Embodiment 1. The loss function calculation module is used to calculate the similarity loss of the respective aspect ratios of the detection frame and the real frame. The specific calculation method is as follows:

in,

b and b ₁ represent the center point of the detection frame and the real frame, respectively, ρ is the Euclidean distance, c is the distance between the farthest vertices between the detection frame and the real frame, IoU represents the intersection ratio of b and b ₁ , x and y represent the width and height of the detection frame, respectively, and x ₁ and y ₁ represent the width and height of the real frame, respectively;

The gesture action localization module loss replaces the bounding box prediction loss term in the traditional localization algorithm. The improved loss function L consists of three parts: error, confidence error and classification error:

L=L _C +L _con +L _s ,

in,

代表目标物是否落入第i个网格第j个边界框，

代表没有落入，

代表已经落入；该网络采用13★13和26★26尺度特征融合，因此s²＝13²和s²＝26²，B＝2。where s ² represents the number of grids, knife represents the number of bounding boxes,

Represents whether the target falls into the jth bounding box of the ith grid,

represents not falling,

The representatives have fallen in; the network uses 13★13 and 26★26 scale feature fusion, so s ² =13 ² and s ² =26 ² , B=2.

Example 5.

This embodiment is a further illustration of Embodiment 1. The channel attention module adopts dual channels. After the output of two different scales of the dual channels, a channel attention module is added. Feature output, the final detection result is obtained by non-maximum suppression.

Since the information of different channels has different feature expression capabilities for gesture targets, the channel attention module models the relationship between different feature channels, highlights the weight of key information and removes irrelevant information to improve the accuracy of gesture detection. In order to improve the network performance, this embodiment proposes a dual-channel attention module. As shown in Figure 5, the gesture image is passed through the preprocessing module, and the channel attention module is added after the output of two different scales of the dual-channel. The feature outputs of two different scales, s1 and s2, are subjected to non-maximum suppression to obtain the final detection result.

Example 6.

This embodiment is a further illustration of Embodiment 1, and FIG. 6 is a schematic structural diagram of an identity authentication module provided by an embodiment of the present invention. The identity authentication module is a digital signature authentication system based on public key cryptography. The camera collects the face image, and the image data is preprocessed and transmitted to the in-vehicle communication terminal TBOX through the CAN line. The TBOX transmits the data to the cloud platform TSP for initial face image storage and generates a unique identification code, and applies for the initial digital certificate. The authentication system PKI, in the PKI system, firstly checks the identity of the user's identity registration request through the RA module, and then the CA module obtains and issues a digital certificate for the user from the certificate store, and then the user's identity information and the user's public key are passed through the digital certificate. The binding is completed in the form of a certificate, thereby realizing user identity authentication.

Example 7.

This embodiment is a further illustration of Embodiment 1. FIG. 7 is a schematic structural diagram of a vehicle control module provided in this embodiment. The vehicle control module includes an on-board high-precision camera, a network communication module, an on-board computing unit, and a wire-control unit. The vehicle high-precision camera extracts the accurate gesture image signal through the gesture recognition algorithm, transmits it to the vehicle computing unit in real time through the communication network module, filters the gesture signal, and fuses other anti-collision signals perceived by the sensor. Send it to the wire control unit to control the vehicle with gestures.

Example 8.

This embodiment is a further illustration of Embodiment 7, the control of the vehicle by gestures includes external control of the vehicle and in-vehicle control, the external control includes reversing, braking and steering, and the in-vehicle control includes control of the vehicle Control of electronic seats, car screen, car audio and ambient lighting.

Example 9.

This embodiment provides a deep learning-based dynamic gesture car control method, applying the dynamic gesture car control system in any one of Embodiments 1-8, and the dynamic gesture car control method includes the following steps:

S1. A static face image is collected and recognized by a high-precision camera, and the in-vehicle communication terminal uploads the face signal to the cloud through the ACP protocol for identity verification;

S2. After the identity verification is passed, the dynamic gesture image of the user is captured, and the accurate gesture signal is calculated and analyzed in real time by the gesture recognition module.

S3. The gesture signal is transmitted to the in-vehicle computing unit through the Ethernet for fusion processing, and then sent to the wire-controlled unit to control the vehicle inside and outside the vehicle.

The deep learning-based dynamic gesture car control method provided in this embodiment combines face recognition and dynamic gesture recognition to provide low-complexity and accurate image signals, and also combines a cloud authentication system and a vehicle-end car control system. Realize the whole gesture car control system in the true sense, improve the whole human-computer interaction performance, and ensure the safe driving of users.

Claims (9)

1. a kind of dynamic gesture vehicle control system based on deep learning, is characterized in that, comprises vehicle-mounted terminal and cloud platform; Described vehicle-mounted terminal comprises face recognition module, gesture recognition module and vehicle control module, and each module realizes communication by Ethernet connection; the cloud platform includes an identity authentication module, and the identity authentication module is connected with the vehicle terminal through the ACP protocol; The gesture recognition module includes: a depth separable convolutional neural network module, a gesture action positioning module, a loss function calculation module, a channel attention module and a dynamic gesture database; The dynamic gesture database is used to capture and store the user's common gesture information. The loss function calculation module cooperates with the gesture action positioning module to make the prediction frame fit the real frame more accurately. The channel attention module is used to output feature detection results, and the depthwise separable convolution The neural network module performs in-depth training on user gesture information to achieve fast feature extraction and recognition of user gesture operations. 2. The dynamic gesture car control system based on deep learning according to claim 1, wherein the face recognition module comprises: cascaded convolutional neural network, deep convolutional neural network, metric module, normalization Calculation module and loss function calculation module; The face recognition module extracts the image information of the target face from the static face image collected by the camera through the cascaded convolutional neural network, and then extracts the depth feature vector in the face image through the deep convolutional neural network module, The measurement module judges the correlation degree of the feature vector, and finally calculates and integrates the accurate face data through the normalization calculation module and the loss function calculation module of the face feature data, and compares and authenticates with the cloud face database, and finally realizes the face purpose of identification. 3 . The deep learning-based dynamic gesture car control system according to claim 1 , wherein the depthwise separable convolutional neural network module utilizes depthwise convolution and 1×1 convolution to perform depth fusion, and the depth The first and last steps of the separable convolutional neural network module use 1×1 convolution, and the intermediate step uses the ResNet feature fusion concept to fuse shallow features, and reduces the amount of network parameters by compressing the number of channels. 4. The deep learning-based dynamic gesture car control system according to claim 1, wherein the loss function calculation module is used to calculate the similarity loss of the respective aspect ratios of the detection frame and the real frame, and the specific calculation method is :

in,

b and b ₁ represent the center point of the detection frame and the real frame, respectively, ρ is the Euclidean distance, c is the distance between the farthest vertices between the detection frame and the real frame, IoU represents the intersection ratio of b and b ₁ , x and y represent the width and height of the detection frame, respectively, and x ₁ and y ₁ represent the width and height of the real frame, respectively; The gesture action localization module loss replaces the bounding box prediction loss term in the traditional localization algorithm. The improved loss function L consists of three parts: error, confidence error and classification error: L=L _c +L _con +L _s , in,

where s ² represents the number of grids, B represents the number of bounding boxes,

Represents whether the target falls into the jth bounding box of the ith grid,

represents not falling,

The representatives have fallen in; the network uses 13★13 and 26★26 scale feature fusion, so s ² =13 ² and s ² =26 ² , B=2. 5. The deep learning-based dynamic gesture car control system according to claim 1, wherein the channel attention module adopts dual channels, and the channel attention module is added after the outputs of two different scales of the dual channels, through Different weights are assigned, and the final detection results are obtained by non-maximum suppression for feature outputs of two different scales. 6. The deep learning-based dynamic gesture car control system according to claim 1, wherein the identity authentication module is a digital signature authentication system based on public key cryptography, and collects human faces through high-precision cameras inside and outside the vehicle The image and image data are preprocessed and transmitted to the in-vehicle communication terminal TBOX through the CAN line. The TBOX transmits the data to the cloud platform TSP for initial face image storage and generates a unique identification code, and applies for the initial digital certificate to the authentication system PKI. In the PKI system, the RA module firstly checks the identity of the user's identity registration request, and then the CA module obtains and issues a digital certificate for the user from the certificate store, and then binds the user's identity information and the user's public key in the form of a digital certificate. to achieve user identity authentication. 7. The deep learning-based dynamic gesture vehicle control system according to claim 1, wherein the vehicle control module comprises a vehicle-mounted high-precision camera, a network communication module, a vehicle-mounted computing unit and a wire-controlled unit; the vehicle-mounted high-precision camera The camera extracts the accurate gesture image signal through the gesture recognition algorithm, transmits it to the vehicle-mounted computing unit in real time through the communication network module, filters and calculates the gesture signal, and integrates other anti-collision signals perceived by the sensor, and sends it to the wire control unit uniformly Implement gesture control of the vehicle. 8 . The deep learning-based dynamic gesture vehicle control system according to claim 7 , wherein the control of the vehicle by the gesture includes external control and interior control of the vehicle, and the external control includes reversing, braking and Steering, the in-vehicle control includes the control of the in-vehicle electronic seat, the in-vehicle screen, the in-vehicle audio and the ambient light. 9. A deep learning-based dynamic gesture vehicle control method, wherein the dynamic gesture vehicle control system according to any one of claims 1-8 is applied, and the dynamic gesture vehicle control method comprises the following steps: S1. A static face image is collected and recognized by a high-precision camera, and the in-vehicle communication terminal uploads the face signal to the cloud through the ACP protocol for identity verification; S2. After the identity verification is passed, the dynamic gesture image of the user is captured, and the accurate gesture signal is calculated and analyzed in real time by the gesture recognition module. S3. The gesture signal is transmitted to the in-vehicle computing unit through the Ethernet for fusion processing, and then sent to the wire-controlled unit to control the vehicle inside and outside the vehicle.

Images