CN114639115A - 3D pedestrian detection method based on fusion of human body key points and laser radar - Google Patents

Abstract

The invention discloses a 3D pedestrian detection method fused with human body key points and laser radar, including key point detection, 3D feature extraction and pedestrian position prediction; the invention makes full use of pedestrian key points in images and depth features in point cloud data , enhances the effect of point cloud and image information for improving pedestrian target recognition, effectively improves the accuracy of 3D pedestrian detection, and makes up for the lack of image color information in the point cloud feature and the lack of the three-dimensional position of the image missing target; the method of the present invention is used in intelligent robots. , augmented reality, autonomous driving and many other fields have great significance and application value.

Description

A 3D pedestrian detection method based on fusion of human key points and lidar

technical field

The invention belongs to the technical field of 3D target detection, and relates to a 3D pedestrian detection method integrating human key points and laser radar.

Background technique

3D pedestrian detection tasks rely on application scenarios such as autonomous driving, augmented reality, and intelligent robots, and are currently one of the research hotspots in the field of computer vision. In the above scenarios, people, as the main behavioral subject, are one of the most common detection targets. Especially in the context of autonomous vehicles, uncertainties often come from pedestrians or cyclists. The flexibility of people in the traffic environment and their important position make pedestrians need high detection accuracy. However, due to the small pedestrian target, insufficient features and background interference, it brings great challenges to 3D pedestrian detection.

Lidar is an optical remote sensing technology that obtains target-related information by detecting laser scattering from distant objects. It is a technical product that combines traditional radar and modern lasers. It obtains information by detecting the laser scattering on the surface of the target object, and has a wide range of applications in ranging, speed measurement, scanning, target detection and other fields. In autonomous driving technology, the perception of the surrounding space environment is mainly through lidar scanners to plan the vehicle's travel route and control the vehicle to safely reach the predetermined destination. Compared with traditional measurement technologies, lidar data collectors have the advantages of high measurement accuracy, high detection efficiency, all-weather detection, and non-contact detection.

Human key point detection is a basic task in computer vision, and it is a pre-task of human action recognition, behavior analysis, and human-computer interaction. Human skeleton key points are crucial for describing human posture and predicting human behavior, and are the basis for many computer vision tasks, such as action classification, abnormal behavior detection, and automatic driving. Human key point detection has an accuracy of 80% in human body recognition, and it also has good results in human behavior prediction. Therefore, it is of great significance and application value in the field of automatic driving to study 3D pedestrian detection based on point cloud and image in combination with human key points.

Existing 3D target detection methods are mainly based on point cloud data for target recognition. Such algorithms have the characteristics of good detection accuracy and performance, but due to the lack of image data and lack of color information, some background features are easily misdetected as pedestrians. The human body key point detection is mainly used in the research of human target detection and corresponding behavior prediction in 2D scenes, and lacks characteristics such as the position and size of pedestrians in three-dimensional space.

SUMMARY OF THE INVENTION

In order to solve the above-mentioned deficiencies in the prior art, the present invention proposes a 3D pedestrian detection method fused with human body key points and lidar, which combines the advantages of high distance measurement accuracy of point clouds and high pedestrian recognition ability of human key points to realize three-dimensional pedestrian detection in three-dimensional space. 3D pedestrian object detection. The concrete technical scheme of the present invention is as follows:

A 3D pedestrian detection method fused with human key points and lidar. A fisheye camera and lidar are installed at the front of the vehicle to obtain visible light images of the front and side areas of the vehicle, and 3D space radar points of the front and side areas of the vehicle. Cloud data, including the following steps:

S1: Human body key point detection based on visible light images; extract the position of human body key points from the image, infer the connection relationship between key points, and trace back to the detection frame position of each pedestrian;

S2: Build a radar 3D point cloud feature extraction network based on human key points for feature extraction; based on a voxel-based radar signal detection network, perform feature dimensionality reduction and two-dimensional image registration, and accurately introduce key points according to the registration results. point position, so that the network can perform 3D feature extraction around the key point position;

S3: Training and prediction of the pedestrian 3D position detection network; the neural network is used to perform regression prediction on the pedestrian's center point position and length, width and height information in 3D space, and use the loss function for training to obtain the final pedestrian 3D detection network; and after detection Post-box processing to give the final prediction result.

Further, the specific process of the step S1 is:

S1-1: Annotate training data; train the OpenPose key point recognition algorithm according to the key point data and annotation information of the MSCOCO dataset, and perform human key point detection on the input image information, including 14 key points: head, neck, left and right shoulders, Left and right elbows, left and right wrists, left and right waists, left and right knees, left and right ankles;

S1-2: Using the OpenPose key point detection algorithm, use the training data in step S1-1 for training to obtain a pedestrian key point detection network, input the visible light image obtained by the fisheye camera to the trained network, and obtain the key points of all pedestrians According to the detection results, the pedestrians to which each key point belongs is obtained through the Hungarian algorithm, and finally the 2D key points of everyone in the image are given;

S1-3: According to the pedestrian information to which the key points belong, the maximum and minimum position coordinates of the pedestrian are obtained, and the human body candidate area is generated from the bottom to the top, and the result of the pedestrian candidate frame is obtained.

Further, the radar 3D point cloud feature extraction network based on human key points in the step S2 includes a series connected voxel division module, a feature map matching module, a feature enhancement module and a prediction module, wherein,

① In the voxel division module, the 3D space radar point cloud contains three-dimensional space information. The width along the X, Y, and Z axes is W, the height is H, and the depth is D, and the voxels are divided into uniform and equal-sized cuboid blocks. The minimum units of width, height, and depth used for point cloud voxel division are ν _W , ν _H , and ν _D respectively. The size of the 3D voxel network generated after segmentation is: W′=W/ν _W , H′=H/ ν _H , D′=D/ν _D ; after the voxel is divided, the voxel grid contains radar points, and the voxel grid containing more than T radar points is divided into non-empty voxels, otherwise it is empty voxels , randomly sample T points in each non-empty voxel grid as the feature of the voxel;

(2) The feature map matching module extracts point cloud features from the output of the voxel division module through a three-layer convolutional network, and divides it into two channels to perform dimensionality reduction processing on the 3D space radar point cloud using the feature summation layer, respectively. The dimensionality reduction direction corresponds to the radar front view and the two-dimensional bird's-eye view respectively. The radar front view direction is consistent with the visible light image direction. In this direction, the signal registration is performed based on the radar front view and the visible light image, and the steps are introduced in the radar front view. Human key points obtained in S1;

3. The feature enhancement module, based on the processing result of the feature map matching module, performs feature stacking and enhancement from the direction of the two-dimensional bird's-eye view, and introduces the layer before the feature summation layer, that is, the three-layer volume in the feature map matching module. The features of the product network build a feature pyramid, and the features are summed along the 3*3 pixel area around the point where the local extreme value is located in the radar front view and the two-dimensional bird's-eye view, and then concatenated with the two-dimensional bird's-eye view features to obtain the enhanced features;

④ The prediction module, based on the enhanced features, builds a prediction module to predict the position of the pedestrian's 3D detection frame, including three fully connected layers and two prediction branches for feature abstraction, wherein each fully connected layer is downsampled by 1 /2, the two prediction branches correspond to the prediction of pedestrian category and pedestrian 3D coordinates respectively.

Further, the specific process of the step S3 is:

S3-1: Unified training on the 3D pedestrian detection data set. In step S2, a radar point cloud feature extraction network based on human key point information is built, and the focal loss loss function is used to optimize the prediction results. Its mathematical expression is:

FL(p _t )=-(1-p _t ) ^γ log(p _t )

Among them, y represents the label. Since it is a binary classification, the value of y is {+1,-1}, and p represents the probability that the predicted sample belongs to 1, ranging from 0 to 1; for convenience, p _t is used instead of p, p _t represents the probability that the sample belongs to the positive sample, γ focus parameter, γ≥0, (1-p _t ) ^γ is the modulation parameter. By increasing the weight of the difficult-to-classify samples, the small targets that are not easy to classify can be paid attention to during model training, and finally Obtain a high-precision pedestrian 3D detection network;

S3-2: Step S3-1 trains a 3D pedestrian detection network. During the detection process, the key point image and radar point cloud are directly input into the network. The network predicts a 3D pedestrian detection frame, and outputs the final output after non-maximum suppression. result.

The beneficial effects of the present invention are:

1. The method of the present invention realizes the 3D detection of pedestrian targets by combining the human key points in the image and the depth information of the lidar point cloud. Make full use of the pedestrian key points in the image and the depth features in the point cloud data, enhance the effect of the point cloud and image information for pedestrian target recognition, effectively improve the accuracy of 3D pedestrian detection, and make up for the missing image color information in the point cloud feature. And the lack of low accuracy of 3D target recognition in the image;

2. The method of the present invention has very important significance and application value in the fields of automatic driving and the like.

Description of drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the accompanying drawings required in the embodiments will be briefly introduced below, and the features and advantages of the present invention will be more clearly understood by referring to the drawings. , the accompanying drawings are schematic and should not be construed as any limitation to the present invention. For those of ordinary skill in the art, other drawings can be obtained from these drawings without creative effort. in:

1 is a schematic diagram of the main flow of the 3D pedestrian detection of the present invention;

2 is a schematic diagram of the overall design framework of the method of the present invention;

FIG. 3 is a schematic diagram of pyramid feature enhancement in the region generation network of the present invention.

Detailed ways

In order to understand the above objects, features and advantages of the present invention more clearly, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that the embodiments of the present invention and the features in the embodiments may be combined with each other under the condition of no conflict.

Many specific details are set forth in the following description to facilitate a full understanding of the present invention. However, the present invention can also be implemented in other ways different from those described herein. Therefore, the protection scope of the present invention is not limited by the specific details disclosed below. Example limitations.

In the research of 3D pedestrian detection based on point cloud, it is found that although point cloud data has outstanding performance in the detection of large targets such as vehicles, its accuracy is not high for 3D pedestrian detection tasks. The reason is that pedestrians are small targets in the entire road scene and are easily disturbed by the background; because of the non-rigid structure of pedestrians themselves, the point cloud information obtained by radar scanning is less than that of vehicles, and even some features are missing; due to the lack of color information in the image, As a result, there is a lack of reference when distinguishing pedestrian categories, so that the predicted detection results cannot be further calibrated, resulting in low pedestrian target detection accuracy.

The invention provides a method for 3D pedestrian target detection by fusion of human body key points and laser radar point cloud, identifying pedestrian 3D features with depth information through point cloud data, and combining the pedestrian targets identified by human body key points in the image. The pedestrians identified by the point cloud are calibrated to achieve 3D pedestrian target detection.

In the present invention, a method for realizing 3D pedestrian target detection by fusion of human key points and lidar point cloud is proposed. Figure 1 is a schematic diagram of the 3D pedestrian detection process of the present invention. The overall idea is that the 3D target detection algorithm of radar point cloud recognizes the depth of pedestrians Information and preliminary category prediction, the human body key point detection scheme realizes the identification of the key point features of the human body from the image information, and then realizes the detection of pedestrian objects through the correlation of the limbs. The 3D pedestrian detection result predicted by the radar point cloud is calibrated through the pedestrian information predicted by the human key points, and the final prediction result is output. The overall algorithm is model trained, tested and verified, and the detection results are analyzed.

Specifically, as shown in Figure 1-2, a 3D pedestrian detection method fused with human key points and lidar, a fisheye camera and lidar are installed at the front of the vehicle to obtain visible light images of the front and side areas of the vehicle, 3D spatial radar point cloud data for the area in front of and to the side of the vehicle, including the following steps:

S1: Human body key point detection based on visible light images; extract the position of human body key points from the image, infer the connection relationship between key points, and trace back to the detection frame position of each pedestrian; the specific process is:

S1-1: Annotate training data; train the OpenPose key point recognition algorithm according to the key point data and annotation information of the MSCOCO dataset, and perform human key point detection on the input image information, including 14 key points: head, neck, left and right shoulders, Left and right elbows, left and right wrists, left and right waists, left and right knees, left and right ankles;

S1-2: Using the OpenPose key point detection algorithm, use the training data in step S1-1 for training to obtain a pedestrian key point detection network, input the visible light image obtained by the fisheye camera to the trained network, and obtain the key points of all pedestrians According to the detection results, the pedestrians to which each key point belongs is obtained through the Hungarian algorithm, and finally the 2D key points of everyone in the image are given;

S1-3: According to the pedestrian information to which the key points belong, the maximum and minimum position coordinates of the pedestrian are obtained, and the human body candidate area is generated from the bottom to the top, and the result of the pedestrian candidate frame is obtained.

S2: Build a radar 3D point cloud feature extraction network based on human key points for feature extraction; based on a voxel-based radar signal detection network, perform feature dimensionality reduction and two-dimensional image registration, and accurately introduce key points according to the registration results. point position, so that the network can perform 3D feature extraction around the key point position;

S3: Training and prediction of the pedestrian 3D position detection network; the neural network is used to perform regression prediction on the pedestrian's center point position and length, width and height information in 3D space, and use the loss function for training to obtain the final pedestrian 3D detection network; and after detection Post-box processing to give the final prediction result.

In some embodiments, the radar 3D point cloud feature extraction network based on human key points in step S2 includes a concatenated voxel division module, a feature map matching module, a feature enhancement module and a prediction module, wherein,

①Voxel division module. The 3D space radar point cloud contains three-dimensional space information. The width along the X, Y, and Z axes is W, the height is H, and the depth is D. The voxels are divided into uniform and equal-sized cuboid blocks. The point cloud The minimum units of width, height and depth used for voxel division are ν _W , ν _H , and ν _D respectively. The size of the 3D voxel network generated after segmentation is: W′=W/ν _W , H′=H/ν _H , D′=D/ν _D ; After the voxel is divided, the voxel grid contains radar points, and the voxel grid containing more than T radar points is divided into non-empty voxels, otherwise it is an empty voxel, in Randomly sample T points in each non-empty voxel grid as the feature of the voxel;

② The feature map matching module extracts point cloud features from the output of the voxel division module through a three-layer convolutional network, and divides it into two channels to use the feature summation layer to reduce the dimension of the 3D space radar point cloud. The dimension reduction directions are respectively Corresponding to the radar front view and the two-dimensional bird's-eye view, the direction of the radar front view is consistent with the direction of the visible light image, and the signal registration is performed based on the radar front view and the visible light image in this direction, and the radar front view is introduced in step S1. key points of the human body;

Taking the KITTI data acquisition vehicle as an example, the spatial point m in the lidar coordinate system is converted to n in the camera coordinate system. The specific conversion relationship is:

表示矫正后相机旋转矩阵，使图像在一个平面，在实际计算时，将其扩展成

具体如下：

Represents the camera rotation matrix after rectification, so that the image is in a plane, and in the actual calculation, it is expanded into

details as follows:

代表从激光雷达到相机的转化矩阵，表达如下：

Represents the transformation matrix from lidar to camera, expressed as follows:

表示旋转矩阵，

表示平移矩阵，为激光雷达到0号灰度相机坐标系的转换矩阵。

表示矫正后的相机投影矩阵，表达如下：in the formula

represents the rotation matrix,

Represents the translation matrix, which is the transformation matrix from the lidar to the 0th grayscale camera coordinate system.

Represents the corrected camera projection matrix, expressed as follows:

是第i个相机到0号灰度相机在X轴方向上的偏移量，要将激光雷达点云坐标系下的点投影到左侧的彩色图像中，则i取值为2。

和

是指相机焦距，

和

是指主点的偏移。in the formula

is the offset from the i-th camera to the 0th grayscale camera in the X-axis direction. To project the point under the lidar point cloud coordinate system into the color image on the left, the i value is 2.

and

is the focal length of the camera,

and

is the offset of the principal point.

③ Feature enhancement module, based on the processing results of the feature map matching module, stack and enhance features from the direction of the two-dimensional bird's-eye view, and introduce the layer before the feature summation layer, that is, the feature building feature of the three-layer convolutional network in the feature map matching module. Pyramid, add features along the 3*3 pixel area around the point where the local extreme value is located in the radar front view and the two-dimensional bird's-eye view, and then concatenate with the two-dimensional bird's-eye view features to obtain the enhanced features;

④Prediction module, build a prediction module based on the enhanced features to predict the 3D detection frame position of pedestrians, including three fully connected layers and two prediction branches for feature abstraction, where each fully connected layer is downsampled by 1/2 , the two prediction branches correspond to the prediction of pedestrian category and pedestrian 3D coordinates respectively.

In some embodiments, the specific process of step S3 is:

S3-1: Unified training on the 3D pedestrian detection data set. In step S2, a radar point cloud feature extraction network based on human key point information is built, and the focal loss loss function is used to optimize the prediction results. Its mathematical expression is:

FL(p _t )=-(1-p _t ) ^γ log(p _t )

Among them, y represents the label. Since it is a binary classification, the value of y is {+1,-1}, and p represents the probability that the predicted sample belongs to 1, ranging from 0 to 1; for convenience, p _t is used instead of p, p _t represents the probability that the sample belongs to the positive sample, γ focus parameter, γ≥0, (1-p _t ) ^γ is the modulation parameter. By increasing the weight of the difficult-to-classify samples, the small targets that are not easy to classify can be paid attention to during model training, and finally Obtain a high-precision pedestrian 3D detection network;

S3-2: Step S3-1 trains a 3D pedestrian detection network. During the detection process, the key point image and radar point cloud are directly input into the network. The network predicts a 3D pedestrian detection frame, and outputs the final output after non-maximum suppression. result.

To sum up, the present invention provides an implementation method for fusion of human key points and lidar for 3D pedestrian detection, which can be applied to many fields such as intelligent robots, augmented reality, and automatic driving. The optical image and the point cloud scanned by the laser radar, the method of the present invention is applied to realize the 3D pedestrian detection method of multi-sensor fusion. The method of the present invention has the characteristics of comprehensive sensing information and accurate detection accuracy, which is the development trend of subsequent 3D target detection. Therefore, it has high promotion value.

In the present invention, the terms "first", "second", "third", and "fourth" are used for descriptive purposes only and should not be construed as indicating or implying relative importance. The term "plurality" refers to two or more, unless expressly limited otherwise.

The above descriptions are only preferred embodiments of the present invention, and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and changes. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims (4)

1. A3D pedestrian detection method integrating human body key points and a laser radar is characterized in that a fisheye camera and the laser radar are mounted at the front end of a vehicle and used for acquiring visible light images of the front area and the side area of the vehicle and 3D space radar point cloud data of the front area and the side area of the vehicle, and the method comprises the following steps:

s1: detecting key points of the human body based on the visible light image; extracting the positions of key points of the human body from the image, deducing the connection relation between the key points, and tracing to the position of a detection frame of each pedestrian;

s2: building a radar 3D point cloud feature extraction network based on human body key points to extract features; based on a voxel-based radar signal detection network, registering through feature dimension reduction and a two-dimensional image, and accurately introducing key point positions according to a registration result so that the network can extract three-dimensional features around the key point positions;

s3: training and predicting a pedestrian 3D position detection network; carrying out regression prediction on the central point position and the length, width and height information of the pedestrian in the 3D space through a neural network, and training by using a loss function to obtain a final 3D detection network of the pedestrian; and the final prediction result is given after the detection frame post-processing.

2. The method for detecting pedestrian key points in visible light images according to claim 1, wherein the specific process of step S1 is as follows:

s1-1: marking training data; training an OpenPose key point recognition algorithm according to key point data and annotation information of an MSCOCO data set, and carrying out human body key point detection on input image information, wherein the human body key point detection comprises 14 key points: head, neck, left and right shoulders, left and right elbows, left and right wrists, left and right waists, left and right knees, left and right ankles;

s1-2: training by using the OpenPose key point detection algorithm and the training data in the step S1-1 to obtain a pedestrian key point detection network, inputting visible light images acquired by the fisheye camera to the trained network to obtain key point detection results of all pedestrians, obtaining pedestrians to which each key point belongs by the Hungary algorithm, and finally giving 2D key points of all people in the images;

s1-3: and obtaining the maximum and minimum position coordinates of the pedestrian according to the information of the pedestrian to which the key point belongs, and generating a human body candidate region from bottom to top to obtain a pedestrian candidate frame result.

3. The construction of the pedestrian key point-based 3D point cloud feature extraction network according to claim 1 or 2, wherein the human body key point-based radar 3D point cloud feature extraction network in the step S2 comprises a voxel division module, a feature mapping matching module, a feature enhancement module and a prediction module which are connected in series,

the voxel division module is characterized in that 3D space radar point cloud comprises three-dimensional space information, the width along an X, Y, Z axis is W, the height is H, and the depth is D, voxels are divided into cuboid blocks with uniform equal size, and the minimum units of the width, the height and the depth adopted by point cloud voxel division are respectively v_W、ν_H、ν_DThe size of the three-dimensional voxel network generated after segmentation is as follows: w' ═ W/v_W、H′＝H/ν_H、D′＝D/ν_D(ii) a After the voxels are divided, radar points are contained in the voxel grid, non-empty voxels are divided into the voxels containing more than T radar points in the voxel grid, otherwise the voxels are empty voxels, and T points are randomly sampled in each non-empty voxel grid to serve as the characteristics of the voxels;

the feature mapping matching module extracts point cloud features from the output of the voxel division module through a three-layer convolution network, divides the point cloud features into two paths, respectively uses a feature addition layer to perform dimensionality reduction processing on the 3D space radar point cloud, and enables dimensionality reduction directions to respectively correspond to a radar front view and a two-dimensional aerial view, wherein the radar front view direction is consistent with the visible light image direction, signal registration is performed in the direction based on the radar front view and the visible light image, and human body key points obtained in the step S1 are introduced into the radar front view;

the feature enhancement module is used for stacking and enhancing features from the direction of the two-dimensional aerial view based on the processing result of the feature mapping matching module, introducing layers before feature summation layers, namely feature building feature pyramids of three layers of convolution networks in the feature mapping matching module, carrying out feature summation along 3 pixel areas around the points where local extrema in the radar front view and the two-dimensional aerial view are located, and then connecting the feature summation with the features of the two-dimensional aerial view in series to obtain enhanced features;

and fourthly, the prediction module is used for building a prediction module to predict the position of the 3D detection frame of the pedestrian based on the enhanced features, and comprises three full connection layers and two prediction branches for feature abstraction, wherein each full connection layer carries out downsampling 1/2, and the two prediction branches respectively correspond to the prediction of the pedestrian category and the 3D coordinate of the pedestrian.

4. The network training and predicting method according to any one of claims 1-3, wherein said step S3 comprises the following steps:

s3-1: a radar point cloud characteristic extraction network based on human body key point information is built in a step S2 of training on a 3D pedestrian detection data set in a unified mode, a focal loss function is introduced to optimize a prediction result, and the mathematical expression form of the method is as follows:

FL(p_t)＝-(1-p_t)^γlog(p_t)

wherein y represents a label, the value of y is { +1, -1} due to the binary classification, p represents the probability that the prediction sample belongs to 1, and the range is 0 to 1; for convenience of presentation, p is used_tIn place of p, p_tRepresenting the probability that the sample belongs to a positive sample, a gamma focusing parameter, gamma ≧ 0, (1-p)_t)^γThe method is characterized in that the parameters are modulated, and small targets which are difficult to classify are paid attention to during model training by increasing the weight of samples which are difficult to classify, so that a high-precision pedestrian 3D detection network is obtained finally;

s3-2: and S3-1, training to obtain a 3D pedestrian detection network, directly inputting the key point image and the radar point cloud into the network in the detection process, predicting a 3D pedestrian detection frame by the network, and outputting a final result after non-maximum value suppression.

Images