CN107463873B - A real-time gesture analysis and evaluation method and system based on RGBD depth sensor - Google Patents

Abstract

The invention discloses a method and system for real-time gesture analysis and evaluation based on RGBD depth sensors, including a static gesture recognition and evaluation system for the palm of a train driver and a dynamic gesture recognition and evaluation system for the arm of the train driver. The static gesture recognition and evaluation system includes the palm position determination module, the palm area image extraction module, the denoising module, the gesture recognition and evaluation module; the dynamic gesture recognition and evaluation system of the train driver's arm includes the arm skeleton node motion sequence extraction module, Dynamic gesture optimal matching module, arm dynamic gesture evaluation module. The present invention has strong robustness to environmental background and lighting, and adopts gesture pixel search based on palm nodes in palm gesture detection, which improves the detection effect of palm gestures; supervises the driver's gestures in real time to ensure the safety of trains, and also It can avoid artificial gesture monitoring of train drivers and reduce the consumption of human resources.

Description

A real-time gesture analysis and evaluation method and system based on RGBD depth sensor

technical field

The invention belongs to the technical field of image processing, and in particular relates to a real-time gesture analysis and evaluation method and system based on an RGBD depth sensor.

Background technique

As one of the key technologies of future human-computer interaction systems, gesture recognition technology not only has important research value, but also has broad application prospects. At present, the traditional gesture recognition method usually detects and recognizes the gesture of the input two-dimensional image. However, this method is more sensitive to the input image, and the gesture detection and recognition effect is better when the background is simple and the influence of ambient light is small. However, when the background is complex and the illumination changes greatly, the effect of gesture detection and recognition drops sharply, and the application scope is limited. In recent years, in order to overcome the shortcomings of traditional gesture recognition methods, 3D image sensing devices have become more and more popular. Such devices can not only obtain RGB images, but also obtain depth data of images, so as to avoid complicated The influence of environmental background, lighting changes, etc. on gesture recognition.

At present, gestures are widely used in the field of transportation. For example, the train driver needs to use gestures to demonstrate the situation before the train runs, while the train is running, and the detection of the in-vehicle instruments, so as to ensure the safe running of the train and prevent accidents. The traffic police use a series of gestures to ensure the safety and smooth flow of road traffic. However, due to the complex working environment of train drivers, traffic police and other personnel, and the great changes in light, traditional methods cannot effectively identify and evaluate driver gestures, and there are many standard driver gestures, including not only dynamic gestures on the arm, but also dynamic gestures on the arm. Including gestures in the palm area further increases the difficulty of gesture recognition. The traditional gesture recognition method mainly includes two steps, the first is to detect the palm and arm on the two-dimensional image, and the second is to recognize the detected gesture. Generally speaking, the quality of gesture detection directly affects the effect of gesture recognition. Usually the background environment and lighting will have an impact on gesture detection. Moreover, it is also difficult to detect palms and arms on two-dimensional images. The traditional method is to train a large number of Gesture samples are used to obtain a classifier. However, the human hand is a complex deformation body, and gestures have the characteristics of diversity, ambiguity, and time difference. It is difficult to train an ideal gesture classifier, so traditional gesture recognition methods cannot be applied to driving. During the gesture recognition of the staff and so on.

SUMMARY OF THE INVENTION

The purpose of the present invention is to provide a real-time gesture analysis and evaluation system based on RGBD depth sensor, which solves the problem of detecting palms and arms under complex background and lighting conditions, and is able to detect various problems of train drivers, traffic police and other staff. Gestures are analyzed and evaluated, which has broad application prospects.

The technical scheme adopted in the present invention is,

A real-time palm gesture analysis and evaluation method based on RGBD depth sensor, comprising the following steps:

Step 1, using an RGBD sensor to acquire an initial image of T frames within a period of time, each frame of initial image in the initial T frame image includes a palm node, a wrist node and an elbow node, and determine the palm node coordinates of the T frame initial image. ;

include:

Step 11, select a frame of initial image from the initial image of T frames as the initial image of the current frame, and obtain the coordinates of the palm node P in the initial image of the current frame through the initial palm node

Among them, M represents the number of white pixels in the area circle, M is a natural number greater than or equal to 1, _xi represents the abscissa of the ith pixel, _yi represents the ordinate of the ith pixel, and _zi represents the ith pixel The distance between i pixels and the RGBD sensor;

The area circle is _a circle whose center is the initial palm node P1, and the radius is the distance between the initial palm node and the wrist node;

The palm node coordinate P is in a coordinate system with the center of the RGBD sensor as the origin, the horizontal direction as the X axis, the vertical direction as the Y axis, and the direction of the sensor pointing to the driver as the Z axis;

Step 12, repeating step 11, until each frame of the initial image in the T frame initial image is used as the current frame initial image, to obtain the palm node coordinates of the T frame initial image;

Step 2, according to the palm node coordinates of the initial image of the T frame, extract the palm area image of the initial image of the T frame, including:

Step 21, select one frame of initial image as the current initial image from the T frame initial images, and the method for extracting the current initial image includes:

Taking the palm node of the current initial image as the center, search for palm pixels in a rectangular area with a width of W and a height of H, and put the palm pixels that satisfy the formula (2) into the palm pixel set S _k , that is, we get Current palm area image;

In formula (2), k=1,2,...,W×H, k represents the number of searches, d _p represents the distance between the palm node of the current initial image and the RGBD sensor, and _gi represents the i-th Pixel point, d _i represents the distance from the i-th pixel in the rectangular area to the RGBD sensor, abs(d _p -d _i,j ) represents the palm node of the current initial image and the i-th pixel in the rectangular area to the RGBD sensor The absolute value of the difference between the distances between the sensors, threshold represents the threshold, 25≤threshold≤35, S _k represents the set of gesture pixels searched for the kth time, and S _k-1 represents the gesture pixels searched for the k-1th time point set;

其中(x_w,y_w)和(x_e,y_e)分别表示当前初始图像的手掌节点中手腕节点以及手肘节点的坐标；

where (x _w , y _w ) and (x _e , y _e ) represent the coordinates of the wrist node and the elbow node in the palm node of the current initial image, respectively;

Step 22, repeat step 22, until the palm area image of the T frame initial image is extracted;

Step 3, performing expansion and erosion operations on each frame of the palm area image in the palm area image of the T frame initial image, to obtain a denoised palm area image of the T frame;

Step 4: Identify the gestures in the palm area in the denoised palm area image of the T frame through the neural network, and obtain the recognized gesture score P _palm in the palm area by formula (1):

为将T帧去噪后的手掌区域图像输入到神经网络中后神经网络第t帧的输出结果，T为去噪后的手掌区域图像的总帧数，round(·)表示取整数。In formula (1),

It is the output result of the t-th frame of the neural network after inputting the denoised palm region image of T frame into the neural network, where T is the total number of frames of the denoised palm region image, and round( ) represents an integer.

A real-time arm gesture analysis and evaluation method based on RGBD depth sensor, comprising the following steps:

Step 1, use the RGBD sensor to obtain the initial image of T frames within a period of time, select a frame of initial image from the initial image of the T frame as the initial image of the t frame, and extract the arm bone node motion sequence of the initial image of the t frame;

include:

The initial image of the t-th frame includes the initial palm node P ₁ ^t , the wrist node P ₂ ^t , the elbow node P ₃ ^t , the shoulder node P ₄ ^t and the shoulder center node P _s ^t , and the node P _n is obtained by formula (3). Distance D _sn ^t from ^t to shoulder center node P _s ^t :

In formula (3), n=1, 2, 3, 4, t=1, 2,..., T, D _sn ^t represents the distance from node P _n ^t to the shoulder center node P _s ^t in the initial image of the t-th frame. distance, T is the total number of frames of the initial image, x _n ^t , y _n ^t , and z _n ^t respectively represent the coordinate values of the palm node, wrist node, elbow node, and shoulder node in the initial image of the t-th frame; x _s ^t , y _s ^t , z _s ^t represent the coordinates of the shoulder center node in the initial image of the t-th frame;

The motion sequence D _sn =(D _sn ¹ ,D _sn ² ,...,D _sn ^t ,...,D _sn ^T ) of the arm skeleton nodes in the initial image can be obtained, where the arm skeleton nodes include palm nodes, Wrist node, elbow node, shoulder node;

Step 2, find the motion sequence of the driver's standard dynamic gesture sample with the smallest sum of the corresponding point distances between the motion sequences of the arm skeleton nodes in the initial image in the motion sequence library of the driver's standard dynamic gesture samples;

Step 3, the score P _arm of the arm dynamic gesture is obtained by formula (4):

D_a，D_b表示驾驶员标准动态手势样本的运动序列库中的任一运动序列，a＝1,2,...,N，b＝1,2,...,N，a≠b，N为驾驶员标准动态手势样本的运动序列库中运动序列的总数。In formula (4), α is the average DTW distance between standard gesture sequence samples,

D _a , D _b represent any motion sequence in the motion sequence library of driver's standard dynamic gesture samples, a=1,2,...,N, b=1,2,...,N, a≠b , N is the total number of motion sequences in the motion sequence library of standard dynamic gesture samples of drivers.

A real-time gesture analysis and evaluation method based on RGBD depth sensor, including the real-time palm gesture analysis and evaluation method of claim 1 and the real-time arm gesture analysis and evaluation method of claim 2.

A real-time palm gesture analysis and evaluation system based on RGBD depth sensor, including:

The module for determining the palm position of the palm is used to obtain an initial T frame image by using the RGBD sensor within a period of time, and each initial image in the T frame initial image includes a palm node, a wrist node and an elbow node, and determines the initial T frame image. The palm node coordinates of the image;

include:

Step 11, select a frame of initial image from the initial image of T frames as the initial image of the current frame, and obtain the coordinates of the palm node P in the initial image of the current frame through the initial palm node

Among them, M represents the number of white pixels in the area circle, M is a natural number greater than or equal to 1, _xi represents the abscissa of the ith pixel, _yi represents the ordinate of the ith pixel, and _zi represents the ith pixel The distance between i pixels and the RGBD sensor;

The area circle is _a circle whose center is the initial palm node P1, and the radius is the distance between the initial palm node and the wrist node;

The palm node coordinate P is in a coordinate system with the center of the RGBD sensor as the origin, the horizontal direction as the X axis, the vertical direction as the Y axis, and the direction of the sensor pointing to the driver as the Z axis;

Step 12, repeating step 11, until each frame of the initial image in the T frame initial image is used as the current frame initial image, to obtain the palm node coordinates of the T frame initial image;

The palm area image extraction module is used to extract the palm area image of the T frame initial image according to the palm node coordinates of the T frame initial image:

include:

Step 21, select one frame of initial image as the current initial image from the T frame initial images, and the method for extracting the current initial image includes:

Taking the palm node of the current initial image as the center, search for palm pixels in a rectangular area with a width of W and a height of H, and put the palm pixels that satisfy the formula (2) into the palm pixel set S _k , that is, we get Current palm area image;

In formula (2), k=1,2,...,W×H, k represents the number of searches, d _p represents the distance between the palm node of the current initial image and the RGBD sensor, and _gi represents the i-th Pixel point, d _i represents the distance from the i-th pixel in the rectangular area to the RGBD sensor, abs(d _p -d _i,j ) represents the palm node of the current initial image and the i-th pixel in the rectangular area to the RGBD sensor The absolute value of the difference between the distances between the sensors, threshold represents the threshold, 25≤threshold≤35, S _k represents the set of gesture pixels searched for the kth time, and S _k-1 represents the gesture pixels searched for the k-1th time point set;

其中(x_w,y_w)和(x_e,y_e)分别表示当前初始图像的手掌节点中手腕节点以及手肘节点的坐标；

where (x _w , y _w ) and (x _e , y _e ) represent the coordinates of the wrist node and the elbow node in the palm node of the current initial image, respectively;

Step 22, repeat step 22, until the palm area image of the T frame initial image is extracted;

Denoising module: It is used to dilate and corrode each palm area image in the palm area image of the T frame initial image, and obtain the palm area image after T frame denoising;

Gesture recognition and evaluation module: used to recognize the gesture of the palm area in the palm area image after T frame denoising through the neural network, and obtain the recognized gesture score P _palm of the palm area by formula (1):

为将T帧去噪后的手掌区域图像输入到神经网络中后神经网络第t帧的输出结果，T为去噪后的手掌区域图像的总帧数，round(·)表示取整数。In formula (1),

It is the output result of the t-th frame of the neural network after inputting the denoised palm region image of T frame into the neural network, where T is the total number of frames of the denoised palm region image, and round( ) represents an integer.

A real-time arm gesture analysis and evaluation system based on RGBD depth sensor, including:

The arm bone node motion sequence extraction module uses the RGBD sensor to obtain the T frame initial image within a period of time, selects one frame of the initial image from the T frame initial image as the t frame initial image, and extracts the arm bone node of the t frame initial image. motion sequence;

include:

The initial image of the t-th frame includes the initial palm node P ₁ ^t , the wrist node P ₂ ^t , the elbow node P ₃ ^t , the shoulder node P ₄ ^t and the shoulder center node P _s ^t , and the node P _n is obtained by formula (3). Distance D _sn ^t from ^t to shoulder center node P _s ^t :

In formula (3), n=1, 2, 3, 4, t=1, 2,..., T, D _sn ^t represents the distance from node P _n ^t to the shoulder center node P _s ^t in the initial image of the t-th frame. distance, T is the total number of frames of the initial image, x _n ^t , y _n ^t , and z _n ^t respectively represent the coordinate values of the palm node, wrist node, elbow node, and shoulder node in the initial image of the t-th frame; x _s ^t , y _s ^t , z _s ^t represent the coordinates of the shoulder center node in the initial image of the t-th frame;

The motion sequence D _sn =(D _sn ¹ ,D _sn ² ,...,D _sn ^t ,...,D _sn ^T ) of the arm skeleton nodes in the initial image can be obtained, where the arm skeleton nodes include palm nodes, Wrist node, elbow node, shoulder node;

The dynamic gesture optimal matching module is used to find the motion of the driver's standard dynamic gesture sample with the smallest sum of the corresponding point distances between the motion sequences of the arm skeleton nodes in the initial image in the motion sequence library of the driver's standard dynamic gesture samples sequence;

The arm dynamic gesture evaluation module is used to obtain the arm dynamic gesture score P _arm by formula (4):

D_a，D_b表示驾驶员标准动态手势样本的运动序列库中的任一运动序列，a＝1,2,...,N，b＝1,2,...,N，a≠b，N为驾驶员标准动态手势样本的运动序列库中运动序列的总数。In formula (4), α is the average DTW distance between standard gesture sequence samples,

D _a , D _b represent any motion sequence in the motion sequence library of driver's standard dynamic gesture samples, a=1,2,...,N, b=1,2,...,N, a≠b , N is the total number of motion sequences in the motion sequence library of standard dynamic gesture samples of drivers.

A real-time gesture analysis and evaluation system based on RGBD depth sensor, including the real-time palm gesture analysis and evaluation system of claim 4 and the real-time arm gesture analysis and evaluation system of claim 5.

The beneficial effects of the present invention are

The invention adopts the RGBD depth sensor to obtain the depth data of the two-dimensional image, and the key data such as the palm and the arm can be well extracted through the corresponding algorithm. Compared with the traditional method, the invention has strong robustness to the environmental background and illumination. , and the gesture pixel search based on the palm node is used in the palm gesture detection, which further improves the detection effect of the palm gesture. Secondly, the present invention can recognize the palm gesture of the driver and the dynamic arm gesture at the same time, and can perform normative evaluation on the driver's gesture according to the output result of the recognition algorithm and give the score of the palm gesture and the dynamic arm gesture, which can not only supervise the driver in real time Gestures can ensure the safety of trains, and can also avoid artificial monitoring of train drivers' gestures, reducing human resource consumption.

Description of drawings

Fig. 1 is the flow chart of gesture recognition and evaluation in the palm area of the train driver;

Fig. 2 is a schematic diagram of each node of the train driver's palm and arm;

Fig. 3 is the flow chart of dynamic gesture recognition and evaluation of the train driver's arm;

FIG. 4 is an application scenario diagram of the present invention.

Detailed ways

The gesture of the train driver usually includes the gesture of the palm area and the dynamic gesture of the arm part. The following is a detailed introduction to the recognition process of the palm area gesture and the dynamic arm gesture.

Example 1

A real-time palm gesture analysis and evaluation method based on RGBD depth sensor, is characterized in that, comprises the following steps:

Step 1, using an RGBD sensor to acquire an initial image of T frames within a period of time, each frame of initial image in the initial T frame image includes a palm node, a wrist node and an elbow node, and determine the palm node coordinates of the T frame initial image. ;

include:

Step 11, select a frame of initial image from the initial image of T frames as the initial image of the current frame, and obtain the coordinates of the palm node P in the initial image of the current frame through the initial palm node

Among them, M represents the number of white pixels in the area circle, M is a natural number greater than or equal to 1, _xi represents the abscissa of the ith pixel, _yi represents the ordinate of the ith pixel, and _zi represents the ith pixel The distance between i pixels and the RGBD sensor;

As shown in Figure 2, the area circle is _a circle with the initial palm node P1 as the center and the distance between the initial palm node and the wrist node as the radius;

As shown in Figure 4, the palm node coordinate P is in the coordinate system with the center of the RGBD sensor as the origin, the horizontal direction as the X axis, the vertical direction as the Y axis, and the direction of the sensor pointing to the driver as the Z axis;

Since the RGBD sensor is prone to node drift when tracking human skeleton nodes, resulting in a deviation in the distance between the palm and the sensor, in order to reduce the deviation, it is necessary to correct the initial palm node to obtain the accurate coordinates of the palm node P:

Step 12, repeating step 11, until each frame of the initial image in the T frame initial image is used as the current frame initial image, to obtain the palm node coordinates of the T frame initial image;

Step 2, according to the palm node coordinates of the initial image of the T frame, extract the palm area image of the initial image of the T frame, including:

Step 21, select one frame of initial image as the current initial image from the T frame initial images, and the method for extracting the current initial image includes:

Taking the palm node of the current initial image as the center, search for palm pixels in a rectangular area with a width of W and a height of H, and put the palm pixels that satisfy the formula (2) into the palm pixel set S _k , that is, we get Current palm area image;

In formula (2), k=1,2,...,W×H, k represents the number of searches, d _p represents the distance between the palm node of the current initial image and the RGBD sensor, and _gi represents the i-th Pixel point, d _i represents the distance from the i-th pixel in the rectangular area to the RGBD sensor, abs(d _p -d _i,j ) represents the palm node of the current initial image and the i-th pixel in the rectangular area to the RGBD sensor The absolute value of the difference between the distances between the sensors, threshold represents the threshold, 25≤threshold≤35, S _k represents the set of gesture pixels searched for the kth time, and S _k-1 represents the gesture pixels searched for the k-1th time point set;

其中(x_w,y_w)和(x_e,y_e)分别表示当前初始图像的手掌节点中手腕节点以及手肘节点的坐标；Since the W and H of the rectangular search area cannot be set too small to prevent the gesture detection from being incomplete due to the change of the size of the gesture area, the height and width of the rectangular search area are:

where (x _w , y _w ) and (x _e , y _e ) represent the coordinates of the wrist node and the elbow node in the palm node of the current initial image, respectively;

Step 22, repeat step 22, until the palm area image of the T frame initial image is extracted;

Step 3, performing expansion and erosion operations on each frame of the palm area image in the palm area image of the T frame initial image, to obtain a denoised palm area image of the T frame;

The palm area image usually contains some noise, including some burrs on the edge of the gesture and some holes inside the image, etc. In order to obtain a more accurate gesture image, dilation and erosion operations are required. Dilation can remove some burrs on the edge of the binary gesture image. And scattered noise points, etc., while erosion can fill in some holes inside the image.

Step 4: Identify the gestures in the palm area in the denoised palm area image of the T frame through the neural network, and obtain the recognized gesture score P _palm in the palm area by formula (1):

In formula (1), It is the output result of the t-th frame of the neural network after inputting the denoised palm region image of T frame into the neural network, where T is the total number of frames of the denoised palm region image, and round( ) represents an integer.

Example 2

A real-time arm gesture analysis and evaluation method based on RGBD depth sensor, as shown in Figure 3, includes the following steps:

Step 1, use the RGBD sensor to obtain the initial image of T frames within a period of time, select a frame of initial image from the initial image of the T frame as the initial image of the t frame, and extract the arm bone node motion sequence of the initial image of the t frame;

include:

The initial image of the t-th frame includes the initial palm node P ₁ ^t , the wrist node P ₂ ^t , the elbow node P ₃ ^t , the shoulder node P ₄ ^t and the shoulder center node P _s ^t , and the node P _n is obtained by formula (3). Distance D _sn ^t from ^t to shoulder center node P _s ^t :

In formula (3), n=1, 2, 3, 4, t=1, 2,..., T, D _sn ^t represents the distance from node P _n ^t to the shoulder center node P _s ^t in the initial image of the t-th frame. distance, T is the total number of frames of the initial image, x _n ^t , y _n ^t , and z _n ^t respectively represent the coordinate values of the palm node, wrist node, elbow node, and shoulder node in the initial image of the t-th frame; x _s ^t , y _s ^t , z _s ^t represent the coordinates of the shoulder center node in the initial image of the t-th frame;

The motion sequence D _sn =(D _sn ¹ ,D _sn ² ,...,D _sn ^t ,...,D _sn ^T ) of the arm skeleton nodes in the initial image can be obtained, where the arm skeleton nodes include palm nodes, Wrist node, elbow node, shoulder node;

Step 2, find the motion sequence of the driver's standard dynamic gesture sample with the smallest sum of the corresponding point distances between the motion sequences of the arm skeleton nodes in the initial image in the motion sequence library of the driver's standard dynamic gesture samples;

In this embodiment, the DTW algorithm is used to solve the problem of different lengths of the two motion sequences. Assume that the motion sequence D _a = (D _a ¹ , D _a ² , . . . , D _a ^T′ ) of the standard dynamic gesture sample of the driver, the initial The motion sequence D _sn = (D _sn ¹ , D _sn ² ,..., D _sn ^T ) of the arm skeleton nodes in the image, and the point-to-point relationship between the two sequences is φ(k)=(φ _s (k ), φ _a (k)), where 1≤φ _s (k)≤T, 1≤φ _a (k)≤T′, max(T,T′)≤k≤T+T′, the purpose of the DTW algorithm is Find the best point-to-point relationship φ(k) between the two sequences such that the sum of the distances between the corresponding points DTW(D _a , D _sn ) is the smallest:

Step 3, the score P _arm of the arm dynamic gesture is obtained by formula (4):

D_a，D_b表示驾驶员标准动态手势样本的运动序列库中的任一运动序列，a＝1,2,...,N，b＝1,2,...,N，a≠b，N为驾驶员标准动态手势样本的运动序列库中运动序列的总数。In formula (4), α is the average DTW distance between standard gesture sequence samples,

D _a , D _b represent any motion sequence in the motion sequence library of driver's standard dynamic gesture samples, a=1,2,...,N, b=1,2,...,N, a≠b , N is the total number of motion sequences in the motion sequence library of standard dynamic gesture samples of drivers.

Example 3

Based on Embodiments 1 and 2, this embodiment provides a real-time gesture analysis and evaluation method based on an RGBD depth sensor, including the real-time palm gesture analysis and evaluation method provided in Embodiment 1 and the method provided in Embodiment 2. A real-time arm gesture analysis and evaluation method. This embodiment can recognize the driver's palm gesture and arm dynamic gesture at the same time, and can perform normative evaluation on the driver's gesture according to the output result of the recognition algorithm, and give the score of the palm gesture and the arm dynamic gesture, which can not only monitor the driver's gesture in real time In order to ensure the safety of the train, it can also avoid artificial gesture monitoring of the train driver and reduce the consumption of human resources.

Example 4

This embodiment provides a static gesture recognition and evaluation system for the palm of a train driver, as shown in Figure 1, including:

The module for determining the palm position of the palm is used to obtain an initial T frame image by using the RGBD sensor within a period of time, and each initial image in the T frame initial image includes a palm node, a wrist node and an elbow node, and determines the initial T frame image. The palm node coordinates of the image;

include:

Step 11, select a frame of initial image from the initial image of T frames as the initial image of the current frame, and obtain the coordinates of the palm node P in the initial image of the current frame through the initial palm node

Among them, M represents the number of white pixels in the area circle, M is a natural number greater than or equal to 1, _xi represents the abscissa of the ith pixel, _yi represents the ordinate of the ith pixel, and _zi represents the ith pixel The distance between i pixels and the RGBD sensor;

As shown in Figure 2, the area circle is _a circle with the initial palm node P1 as the center and the distance between the initial palm node and the wrist node as the radius;

As shown in Figure 4, the palm node coordinate P is in the coordinate system with the center of the RGBD sensor as the origin, the horizontal direction as the X axis, the vertical direction as the Y axis, and the direction of the sensor pointing to the driver as the Z axis;

Since the RGBD sensor is prone to node drift when tracking human skeleton nodes, resulting in a deviation in the distance between the palm and the sensor, in order to reduce the deviation, it is necessary to correct the initial palm node to obtain the accurate coordinates of the palm node P:

The palm area image module is extracted, and the palm area image module is extracted, which is used to extract the palm area image of the T frame initial image according to the palm node coordinates of the T frame initial image:

include:

Step 21, select one frame of initial image as the current initial image from the T frame initial images, and the method for extracting the current initial image includes:

Taking the palm node of the current initial image as the center, search for palm pixels in a rectangular area with a width of W and a height of H, and put the palm pixels that satisfy the formula (2) into the palm pixel set S _k , that is, we get Current palm area image;

In formula (2), k=1,2,...,W×H, k represents the number of searches, d _p represents the distance between the palm node of the current initial image and the RGBD sensor, and _gi represents the i-th Pixel point, d _i represents the distance from the i-th pixel in the rectangular area to the RGBD sensor, abs(d _p -d _i,j ) represents the palm node of the current initial image and the i-th pixel in the rectangular area to the RGBD sensor The absolute value of the difference between the distances between the sensors, threshold represents the threshold, 25≤threshold≤35, S _k represents the set of gesture pixels searched for the kth time, and S _k-1 represents the gesture pixels searched for the k-1th time point set;

Since the W and H of the rectangular search area cannot be set too small to prevent the gesture detection from being incomplete due to the change of the size of the gesture area, the height and width of the rectangular search area are: where (x _w , y _w ) and (x _e , y _e ) represent the coordinates of the wrist node and the elbow node in the palm node of the current initial image, respectively;

Step 22, repeat step 22, until the palm area image of the T frame initial image is extracted;

Denoising module: It is used to dilate and corrode each palm area image in the palm area image of the T frame initial image, and obtain the palm area image after T frame denoising;

The palm area image usually contains some noise, including some burrs on the edge of the gesture and some holes inside the image, etc. In order to obtain a more accurate gesture image, dilation and erosion operations are required. Dilation can remove some burrs on the edge of the binary gesture image. And scattered noise points, etc., while erosion can fill in some holes inside the image.

Gesture recognition and evaluation module: used to recognize the gesture of the palm area in the palm area image after T frame denoising through the neural network, and obtain the recognized gesture score P _palm of the palm area by formula (1):

为将T帧去噪后的手掌区域图像输入到神经网络中后神经网络第t帧的输出结果，T为去噪后的手掌区域图像的总帧数，round(·)表示取整数。In formula (1),

It is the output result of the t-th frame of the neural network after inputting the denoised palm region image of T frame into the neural network, where T is the total number of frames of the denoised palm region image, and round( ) represents an integer.

Example 5

This embodiment provides a dynamic gesture recognition and evaluation system for the train driver's arm, as shown in Figure 3, including:

The arm bone node motion sequence extraction module uses the RGBD sensor to obtain the T frame initial image within a period of time, selects one frame of the initial image from the T frame initial image as the t frame initial image, and extracts the arm bone node of the t frame initial image. motion sequence;

include:

The initial image of the t-th frame includes the initial palm node P ₁ ^t , the wrist node P ₂ ^t , the elbow node P ₃ ^t , the shoulder node P ₄ ^t and the shoulder center node P _s ^t , and the node P _n is obtained by formula (3). Distance D _sn ^t from ^t to shoulder center node P _s ^t :

In formula (3), n=1, 2, 3, 4, t=1, 2,..., T, D _sn ^t represents the distance from node P _n ^t to the shoulder center node P _s ^t in the initial image of the t-th frame. distance, T is the total number of frames of the initial image, x _n ^t , y _n ^t , and z _n ^t respectively represent the coordinate values of the palm node, wrist node, elbow node, and shoulder node in the initial image of the t-th frame; x _s ^t , y _s ^t , z _s ^t represent the coordinates of the shoulder center node in the initial image of the t-th frame;

The motion sequence D _sn =(D _sn ¹ ,D _sn ² ,...,D _sn ^t ,...,D _sn ^T ) of the arm skeleton nodes in the initial image can be obtained, where the arm skeleton nodes include palm nodes, Wrist node, elbow node, shoulder node;

The dynamic gesture optimal matching module is used to find the motion of the driver's standard dynamic gesture sample with the smallest sum of the corresponding point distances between the motion sequences of the arm skeleton nodes in the initial image in the motion sequence library of the driver's standard dynamic gesture samples sequence;

In this embodiment, the DTW algorithm is used to solve the problem of different lengths of the two motion sequences. Assume that the motion sequence D _a = (D _a ¹ , D _a ² , . . . , D _a ^T′ ) of the standard dynamic gesture sample of the driver, the initial The motion sequence D _sn = (D _sn ¹ , D _sn ² ,..., D _sn ^T ) of the arm skeleton nodes in the image, and the point-to-point relationship between the two sequences is φ(k)=(φ _s (k ), φ _a (k)), where 1≤φ _s (k)≤T, 1≤φ _a (k)≤T′, max(T,T′)≤k≤T+T′, the purpose of the DTW algorithm is Find the best point-to-point relationship φ(k) between the two sequences such that the sum of the distances between the corresponding points DTW(D _a , D _sn ) is the smallest:

The arm dynamic gesture evaluation module is used to obtain the arm dynamic gesture score P _arm by formula (4):

D_a，D_b表示驾驶员标准动态手势样本的运动序列库中的任一运动序列，a＝1,2,...,N，b＝1,2,...,N，a≠b，N为驾驶员标准动态手势样本的运动序列库中运动序列的总数；D_sn＝(D_sn ¹,D_sn ²,...,D_sn ^t,...,D_sn ^T)，D_sn为初始图像中手臂骨骼节点的运动序列。In formula (4), α is the average DTW distance between standard gesture sequence samples,

D _a , D _b represent any motion sequence in the motion sequence library of driver's standard dynamic gesture samples, a=1,2,...,N, b=1,2,...,N, a≠b , N is the total number of motion sequences in the motion sequence library of the driver's standard dynamic gesture samples; D _sn =(D _sn ¹ ,D _sn ² ,...,D _sn ^t ,...,D _sn ^T ), D _sn is the motion sequence of the arm bone node in the initial image.

Example 6

Based on Embodiments 4 and 5, this embodiment provides a real-time gesture analysis and evaluation system based on an RGBD depth sensor, including the real-time palm gesture analysis and evaluation system provided in Embodiment 4 and the system provided in Embodiment 5. Real-time arm gesture analysis and evaluation system. This embodiment can recognize the driver's palm gesture and arm dynamic gesture at the same time, and can perform normative evaluation on the driver's gesture according to the output result of the recognition algorithm, and give the score of the palm gesture and the arm dynamic gesture, which can not only monitor the driver's gesture in real time In order to ensure the safety of the train, it can also avoid artificial gesture monitoring of the train driver and reduce the consumption of human resources.

Claims (4)

1. A real-time palm gesture analysis and evaluation method based on an RGBD depth sensor is characterized by comprising the following steps:

step 1, acquiring T frames of initial images within a period of time by using an RGBD sensor, wherein each frame of initial image in the T frames of initial images comprises a palm node, a wrist node and an elbow node, and determining the coordinates of the palm node of the T frames of initial images;

the method comprises the following steps:

step 11, selecting one frame of initial image from the T frame of initial image as the current frame of initial image, and obtaining the coordinates of the palm node P in the current frame of initial image through the initial palm node

Wherein M represents the number of white pixel points in the region circle, M is a natural number greater than or equal to 1, and x_iAbscissa, y, representing the ith pixel_iThe ordinate, z, representing the ith pixel_iRepresenting the distance from the ith pixel point to the RGBD sensor;

the region circle is an initial palm node P₁A circle with the distance between the initial palm node and the wrist node as the radius and the center of the circle;

the palm node coordinate P is in a coordinate system which takes the center of the RGBD sensor as an origin, takes the horizontal direction as an X axis, takes the vertical direction as a Y axis, and takes the direction of the sensor pointing to the driver as a Z axis;

step 12, repeating step 11 until each frame initial image in the T frame initial images is used as a current frame initial image, and obtaining palm node coordinates of the T frame initial images;

step 2, extracting a palm region image of the T frame initial image according to the palm node coordinates of the T frame initial image, wherein the method comprises the following steps:

step 21, selecting one frame of initial image from the T frames of initial images as a current initial image, and the method for extracting the current initial image includes:

taking a palm node of the current initial image as a center, searching for palm pixel points in a rectangular region with the width of W and the height of H, and putting the palm pixel points satisfying the formula (2) into a palm pixel point set S_kObtaining a current palm area image;

in formula (2), k is 1,2Indicating the number of searches, d_pRepresents the distance, g, between the palm node of the current initial image and the RGBD sensor_iRepresents the ith pixel point, d_iRepresenting the distance between the ith pixel point in the rectangular area and the RGBD sensor, abs (d)_p-d_i) Representing the absolute value of the difference between the distance from the palm node of the current initial image and the ith pixel point in the rectangular region to the RGBD sensor, wherein threshold represents a threshold value, is more than or equal to 25 and less than or equal to 35, and S_kRepresenting the gesture pixel point set searched for the k time, S_k-1Representing the gesture pixel point set searched at the k-1 st time;

wherein (x)_w,y_w) And (x)_e,y_e) Respectively representing the coordinates of a wrist node and an elbow node in a palm node of the current initial image;

step 22, repeating the step 22 until a palm area image of the T-frame initial image is extracted;

step 3, performing expansion and corrosion operations on each frame of palm area image in the palm area image of the T frame initial image to obtain a T frame denoised palm area image;

step 4, recognizing the gesture of the palm area in the T-frame de-noised palm area image through a neural network, and obtaining the fraction P of the recognized gesture of the palm area through the formula (1)_palm：

In the formula (1), the reaction mixture is,

inputting the denoised palm region image of the T frame into the output result of the T frame of the neural network, wherein T is the denoised output result of the T frame of the neural networkThe total frame number of the palm region image of (1), round (·) represents an integer.

2. A real-time gesture analysis and evaluation method based on an RGBD depth sensor is characterized by comprising the real-time palm gesture analysis and evaluation method and the real-time arm gesture analysis and evaluation method of claim 1;

the real-time arm gesture analysis and evaluation method comprises the following steps:

step 1, acquiring a T frame initial image within a period of time by using an RGBD sensor, selecting one frame initial image from the T frame initial image as a T frame initial image, and extracting an arm skeleton node motion sequence of the T frame initial image;

the method comprises the following steps:

the t-th frame initial image comprises an initial palm node P₁ ^tWrist node P₂ ^tElbow joint P₃ ^tShoulder node P₄ ^tAnd shoulder center node P_s ^tObtaining a node P by the formula (3)_n ^tTo shoulder center node P_s ^tDistance D of_sn ^t：

In formula (3), n is 1,2,3,4, T is 1,2_sn ^tIndicating the node P in the initial image of the t-th frame_n ^tTo shoulder center node P_s ^tT is the total frame number of the initial image, x_n ^t，y_n ^t，z_n ^tRespectively representing coordinate values of a palm node, a wrist node, an elbow node and a shoulder node in the t-th frame initial image; x is the number of_s ^t，y_s ^t，z_s ^tRepresenting the coordinates of the shoulder center node in the t-th frame initial image;

obtaining the motion sequence D of the arm skeleton node in the initial image_sn＝(D_sn ¹,D_sn ²,...,D_sn ^t,...,D_sn ^T) The arm skeleton nodes comprise palm nodes, wrist nodes, elbow nodes and shoulder nodes;

step 2, finding the motion sequence of the driver standard dynamic gesture sample with the minimum sum of the distances of corresponding points between the motion sequence of the arm skeleton nodes in the initial image and the motion sequence of the driver standard dynamic gesture sample in the motion sequence library of the driver standard dynamic gesture sample;

step 3, obtaining the score P of the dynamic gesture of the arm through the formula (4)_arm：

In equation (4), α is the average of the DTW distances between the standard gesture sequence samples,

D_a，D_brepresenting any motion sequence in the motion sequence library of the driver standard dynamic gesture sample, wherein a is 1,2, and N, b is 1, 2.

3. A real-time palm gesture analysis and evaluation system based on an RGBD depth sensor is characterized by comprising:

the device comprises a palm center position determining module, a palm image acquiring module and a palm image acquiring module, wherein the palm center position determining module is used for acquiring T frames of initial images in a period of time by using an RGBD sensor, each frame of initial image in the T frames of initial images comprises a palm node, a wrist node and an elbow node, and the palm node coordinates of the T frames of initial images are determined;

the method comprises the following steps:

step 11, selecting one frame of initial image from the T frame of initial image as the current frame of initial image, and obtaining the coordinates of the palm node P in the current frame of initial image through the initial palm node

Wherein M represents the number of white pixel points in the region circle, M is a natural number greater than or equal to 1, and x_iAbscissa, y, representing the ith pixel_iThe ordinate, z, representing the ith pixel_iRepresenting the distance from the ith pixel point to the RGBD sensor;

the region circle is an initial palm node P₁A circle with the distance between the initial palm node and the wrist node as the radius and the center of the circle;

the palm node coordinate P is in a coordinate system which takes the center of the RGBD sensor as an origin, takes the horizontal direction as an X axis, takes the vertical direction as a Y axis, and takes the direction of the sensor pointing to the driver as a Z axis;

step 12, repeating step 11 until each frame initial image in the T frame initial images is used as a current frame initial image, and obtaining palm node coordinates of the T frame initial images;

the palm region image extracting module is used for extracting a palm region image of the T frame initial image according to the palm node coordinates of the T frame initial image:

the method comprises the following steps:

step 21, selecting one frame of initial image from the T frames of initial images as a current initial image, and the method for extracting the current initial image includes:

taking a palm node of the current initial image as a center, searching for palm pixel points in a rectangular region with the width of W and the height of H, and putting the palm pixel points satisfying the formula (2) into a palm pixel point set S_kObtaining a current palm area image;

in formula (2), k is 1,2_pRepresents the distance, g, between the palm node of the current initial image and the RGBD sensor_iRepresents the ith pixel point, d_iRepresenting the distance between the ith pixel point in the rectangular area and the RGBD sensor, abs (d)_p-d_i) Representing the palm node of the current initial image and the ith image in the rectangular areaThe absolute value of the difference between the distances from the prime point to the RGBD sensor, where threshold represents a threshold, 25 ≦ threshold ≦ 35, S_kRepresenting the gesture pixel point set searched for the k time, S_k-1Representing the gesture pixel point set searched at the k-1 st time;

wherein (x)_w,y_w) And (x)_e,y_e) Respectively representing the coordinates of a wrist node and an elbow node in a palm node of the current initial image;

step 22, repeating the step 22 until a palm area image of the T-frame initial image is extracted;

a denoising module: the image processing device is used for performing expansion and corrosion operations on each frame of palm area image in the palm area image of the T frame initial image to obtain a T frame denoised palm area image;

gesture recognition and evaluation module: the method is used for recognizing the gesture of the palm region in the T-frame de-noised palm region image through a neural network and obtaining the fraction P of the recognized gesture of the palm region through the formula (1)_palm：

In the formula (1), the reaction mixture is,

and (3) inputting the denoised palm region image of the T frame into an output result of the T frame of the neural network, wherein T is the total frame number of the denoised palm region image, and round (·) represents an integer.

4. A real-time gesture analysis and evaluation system based on an RGBD depth sensor, comprising the real-time palm gesture analysis and evaluation system and the real-time arm gesture analysis and evaluation system of claim 3;

the real-time arm gesture analysis and evaluation system comprises:

the arm skeleton node motion sequence extraction module is used for acquiring T frame initial images within a period of time by using an RGBD sensor, selecting one frame initial image from the T frame initial images as a T frame initial image, and extracting an arm skeleton node motion sequence of the T frame initial image;

the method comprises the following steps:

the t-th frame initial image comprises an initial palm node P₁ ^tWrist node P₂ ^tElbow joint P₃ ^tShoulder node P₄ ^tAnd shoulder center node P_s ^tObtaining a node P by the formula (3)_n ^tTo shoulder center node P_s ^tDistance D of_sn ^t：

In formula (3), n is 1,2,3,4, T is 1,2_sn ^tIndicating the node P in the initial image of the t-th frame_n ^tTo shoulder center node P_s ^tT is the total frame number of the initial image, x_n ^t，y_n ^t，z_n ^tRespectively representing coordinate values of a palm node, a wrist node, an elbow node and a shoulder node in the t-th frame initial image; x is the number of_s ^t，y_s ^t，z_s ^tRepresenting the coordinates of the shoulder center node in the t-th frame initial image;

obtaining the motion sequence D of the arm skeleton node in the initial image_sn＝(D_sn ¹,D_sn ²,...,D_sn ^t,...,D_sn ^T) The arm skeleton nodes comprise palm nodes, wrist nodes, elbow nodes and shoulder nodes;

the dynamic gesture optimal matching module is used for finding the motion sequence of the driver standard dynamic gesture sample with the minimum sum of corresponding point distances between the motion sequence and the arm skeleton nodes in the initial image in the motion sequence library of the driver standard dynamic gesture sample;

an arm dynamic gesture evaluation module for obtaining the score P of the arm dynamic gesture through the formula (4)_arm：

In equation (4), α is the average of the DTW distances between the standard gesture sequence samples,

D_a，D_brepresenting any motion sequence in the motion sequence library of the driver standard dynamic gesture sample, wherein a is 1,2, and N, b is 1, 2.

Images