CN110879994A - 3D visual detection method, system and device based on shape attention mechanism - Google Patents

Abstract

The invention belongs to the field of computer deep reinforcement learning and pattern recognition, and particularly relates to a three-dimensional visual inspection detection method, system and device based on a shape attention mechanism, aiming at solving the problems that the precision of a single-stage detector is lower than that of a two-stage detector, and the two-stage detector consumes much time and is not suitable for a real-time system. The invention comprises the following steps: representing the point cloud data by three-dimensional grid voxels; extracting features and coding a space sparse feature map; extracting different scale features after projecting to a top view; adopting a deconvolution layer merging characteristic; extracting a shape attention feature map through attention weight and a convolution coding layer; and acquiring the target category, the target position, the target size and the target direction through a target classification network and a regression positioning network. The invention uses a sampling strategy based on distance constraint and an attention mechanism based on shape prior, relieves the instability caused by uneven data distribution, improves the problem that a single-stage detector lacks shape prior, and has high precision, short time consumption, strong real-time performance and good robustness.

Description

3D visual detection method, system and device based on shape attention mechanism

technical field

The invention belongs to the fields of deep reinforcement learning, computer vision, pattern recognition and machine learning, and particularly relates to a three-dimensional visual detection method, system and device based on a shape attention mechanism.

Background technique

3D object detectors need to output reliable spatial and semantic information, namely 3D position, orientation, occupied volume and category. Compared with 2D object detection, 3D objects provide more detailed information, but the modeling is more difficult. 3D object detection generally uses distance sensors, such as lidar, TOF cameras, stereo cameras, etc., to predict more meaningful and accurate results. 3D object detection has become a key technology in areas such as self-driving cars, UVA, and robotics. Most accurate 3D object detection algorithms in traffic scenes are based on radar sensors, which have become the basic sensor for outdoor scene perception. Target perception in traffic scenes is a key technology for unmanned vehicles to locate surrounding targets.

Lidar-based 3D object detection involves two important issues. The first problem is how to generate descriptive underlying features for sparse non-uniform point clouds sampled from lidar sensors. The sampling points of the lidar are more sampling points close to the sensor, and less sampling points are farther away. The diverse distribution of point clouds will reduce the detection performance of the detector and lead to unstable detection results. Many methods rely on manual feature extraction methods. However, the detection algorithm is not stable because the hand-crafted features are not well considered and dealt with the uneven distribution of laser point clouds. Object detection and segmentation play an extremely important role in both understanding and perception of visual data. Another problem is how to efficiently encode 3D shape information for better discriminative embeddings. There are two main frameworks for 3D object detection: single-stage detectors and two-stage detectors. Single-stage detectors are more efficient, while two-stage detectors have higher detection accuracy. Since the region candidate network outputs the cropped region of interest ROI, the two-stage detector is inefficient. However, these cropped ROIs provide shape priors for each detected object, and higher detection accuracy can be obtained through subsequent optimization of the network. One-stage detectors outperform two-stage detectors due to the lack of shape priors and subsequent optimization networks. However, for real-time systems, the two-stage detector is very time-consuming. In addition, the 3D shape prior is more suitable for the detection of 3D objects.

SUMMARY OF THE INVENTION

In order to solve the above problem in the prior art, that is, the accuracy of the single-stage three-dimensional object detector is lower than that of the two-stage detector, and the two-stage detector is time-consuming and not suitable for real-time systems, the present invention provides a shape-based A three-dimensional visual detection method of attention mechanism, the three-dimensional target detection method includes:

Step S10, acquiring laser point cloud data containing the target as the data to be detected, and characterizing the data to be detected by voxels based on a three-dimensional network;

Step S20, obtaining the feature expression of the voxel through a feature extractor and performing sparse convolution encoding to obtain a spatial sparse feature map corresponding to the data to be processed;

Step S30, projecting the spatial sparse feature map to a two-dimensional top-view plane, and after obtaining features of different scales through a feature pyramid convolution network, combining the features of different scales through a deconvolution layer to obtain a top-view feature map;

Step S40, obtaining the attention weight feature map of the top-view feature map through an attention weight layer; obtaining an encoding feature map of the top-view feature map through a convolutional coding layer;

Step S50, multiply the attention weight feature map to the corresponding area of the encoding feature map, and perform feature splicing to obtain the attention feature map;

Step S60, based on the attention feature map, obtain the target category in the data to be detected through the trained target classification network; based on the attention feature map, obtain the target position in the data to be detected through the trained target regression positioning network , size, orientation.

In some preferred embodiments, in step S10, "characterize the data to be detected by a three-dimensional network-based voxel", the method is as follows:

Among them, D represents the voxel representation of the laser point cloud data, x _i , y _i , and z _i represent the three-dimensional position information of the ith point in the laser point cloud data relative to the lidar, and R _i represents the laser point cloud data. The reflectivity of the i-th point.

In some preferred embodiments, in step S20, "acquiring the feature expression of the voxel through a feature extractor and performing sparse convolution encoding to obtain a spatial sparse feature map corresponding to the data to be processed", the method is as follows:

Among them, F() represents the feature expression of voxels obtained by the feature extractor, D represents the voxel representation of the laser point cloud data, and (x, y, z) represents the spatial coordinates of the spatially sparse feature map.

In some preferred embodiments, in step S40, "obtaining the attention weight feature map of the top-view feature map through the attention weight layer", the method is as follows:

F _att (u,v)=Conv _att (F _FPN (u,v))

Among them, F _att (u, v) represents the attention weight feature map corresponding to the top-view feature map, F _FPN (u, v) represents the top-view feature map, and Conv _att () represents the attention weight layer convolution operation.

In some preferred embodiments, in step S40, "obtaining the encoded feature map of the top-view feature map through a convolutional encoding layer", the method is as follows:

F _en (u,v)= _Conven (F _FPN (u,v))

Among them, F _en (u, v) represents the encoded feature map corresponding to the top-view feature map, F _FPN (u, v) represents the top-view feature map, and _Conven () represents the convolutional encoding layer convolution operation.

In some preferred embodiments, in step S50, "multiply the attention weight feature map to the corresponding region of the encoding feature map, and perform feature splicing to obtain the attention feature map", the method is as follows:

F _op (u,v)=F _en (u,v)Repeat(Reshape(F _att (u,v)))

Among them, Reshape() represents the deformation operation, and Repeat() represents the copy operation;

Among them, [ ] represents the feature concatenation operation.

In some preferred embodiments, the target classification network is trained through a cross-entropy loss function; the cross-entropy loss function is:

Among them, N represents the number of samples for calculating the loss; y _i represents the positive and negative samples, 0 represents the negative sample, and 1 represents the positive sample; _xi represents the network output value of the sample.

In some preferred embodiments, the target regression positioning network is trained through the Smooth L1 loss function; the Smooth L1 loss function is:

where x represents the residual that needs to be regressed.

In another aspect of the present invention, a 3D visual detection system based on shape attention mechanism is proposed. The 3D object detection system includes an input module, a sparse convolution coding module, a feature pyramid module, an attention weight convolution module, and a coding volume. Product module, feature fusion module, target classification module, target positioning module, output module;

The input module is configured to obtain laser point cloud data containing the target as the data to be detected, and to characterize the data to be detected by voxels based on a three-dimensional network;

The sparse convolution coding module is configured to obtain the feature expression of the voxel through a feature extractor and perform sparse convolution coding to obtain a spatial sparse feature map corresponding to the data to be processed;

The feature pyramid module is configured to project the spatial sparse feature map to a two-dimensional top-view plane, obtain features of different scales through a feature pyramid convolutional network, and combine the features of different scales through a deconvolution layer to obtain Top view feature map;

The attention weight convolution module is configured to obtain the attention weight feature map of the top-view feature map through the attention weight layer;

The coding convolution module is configured to obtain the coding feature map of the top-view feature map through a convolution coding layer;

The feature fusion module is configured to multiply the attention weight feature map to the corresponding region of the encoding feature map, and perform feature splicing to obtain the attention feature map;

The target classification module is configured to obtain target categories in the data to be detected through the trained target classification network based on the attention feature map;

The target positioning module is configured to obtain the target position, size and direction in the data to be detected through the trained target regression positioning network based on the attention feature map;

The output module is configured to output the acquired target category and target position, size, and direction.

In a third aspect of the present invention, a storage device is provided, wherein a plurality of programs are stored, and the programs are adapted to be loaded and executed by a processor to realize the above-mentioned three-dimensional visual inspection method based on the shape attention mechanism.

In a fourth aspect of the present invention, a processing device is provided, including a processor and a storage device; the processor is suitable for executing various programs; the storage device is suitable for storing multiple programs; the program is suitable for Loaded and executed by the processor to realize the above-mentioned three-dimensional visual detection method based on the shape attention mechanism.

Beneficial effects of the present invention:

The three-dimensional visual detection method based on the shape attention mechanism of the present invention uses the sampling strategy based on distance constraints, which can effectively alleviate the unstable results caused by the uneven distribution of radar sampling point cloud data, and solve the problem through the attention mechanism based on shape prior. This method can improve the detection performance of the current single-stage 3D object detector, especially for objects with obvious shape characteristics, with high detection accuracy, short detection time, and suitable for real-time systems. , The model has good robustness.

Description of drawings

Other features, objects and advantages of the present application will become more apparent by reading the detailed description of non-limiting embodiments made with reference to the following drawings:

1 is a schematic flowchart of a three-dimensional visual detection method based on a shape attention mechanism of the present invention;

2 is a schematic diagram of the algorithm structure of an embodiment of the three-dimensional visual detection method based on the shape attention mechanism of the present invention;

3 is an example diagram of a data set and a detection result of an embodiment of the three-dimensional visual detection method based on the shape attention mechanism of the present invention;

FIG. 4 is a comparison diagram of the detection results of the method of the present invention and other methods according to an embodiment of the three-dimensional visual detection method based on the shape attention mechanism of the present invention.

Detailed ways

The present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the related invention, but not to limit the invention. In addition, it should be noted that, for the convenience of description, only the parts related to the related invention are shown in the drawings.

It should be noted that the embodiments in the present application and the features of the embodiments may be combined with each other in the case of no conflict. The present application will be described in detail below with reference to the accompanying drawings and in conjunction with the embodiments.

A three-dimensional visual detection method based on a shape attention mechanism of the present invention, the three-dimensional target detection method includes:

Step S10, acquiring laser point cloud data containing the target as the data to be detected, and characterizing the data to be detected by voxels based on a three-dimensional network;

Step S20, obtaining the feature expression of the voxel through a feature extractor and performing sparse convolution encoding to obtain a spatial sparse feature map corresponding to the data to be processed;

Step S30, projecting the spatial sparse feature map to a two-dimensional top-view plane, and after obtaining features of different scales through a feature pyramid convolution network, combining the features of different scales through a deconvolution layer to obtain a top-view feature map;

Step S40, obtaining the attention weight feature map of the top-view feature map through an attention weight layer; obtaining an encoding feature map of the top-view feature map through a convolutional coding layer;

Step S50, multiply the attention weight feature map to the corresponding area of the encoding feature map, and perform feature splicing to obtain the attention feature map;

Step S60, based on the attention feature map, obtain the target category in the data to be detected through the trained target classification network; based on the attention feature map, obtain the target position in the data to be detected through the trained target regression positioning network , size, orientation.

In order to more clearly describe the three-dimensional visual detection method based on the shape attention mechanism of the present invention, each step in the embodiment of the method of the present invention will be described in detail below with reference to FIG. 1 .

A three-dimensional visual detection method based on a shape attention mechanism according to an embodiment of the present invention includes steps S10 to S60, and each step is described in detail as follows:

In step S10, the laser point cloud data containing the target object is obtained as the data to be detected, and the data to be detected is represented by voxels based on a three-dimensional network, as shown in formula (1):

Among them, D represents the voxel representation of the laser point cloud data, x _i , y _i , and z _i represent the three-dimensional position information of the ith point in the laser point cloud data in the lidar point cloud, and R _i represents the laser point cloud data The reflectivity of the ith point in .

Assume that the lidar point cloud contains a three-dimensional space with the range of H, W, and D, representing the height in the vertical direction, the position and distance in the horizontal direction, and the size of each voxel is ΔH×ΔW×ΔD, ΔH=0.4 m, ΔW=0.2m, ΔD=0.2m. The size of the voxel grid in the entire three-dimensional space can be calculated by H/ΔH, W/ΔW, D/ΔD. Each voxel is then characterized by a feature encoding layer (VFE). In one embodiment of the present invention, the feature extractor uses 7-dimensional vectors (3-dimensional coordinates, reflectivity, and relative 3-dimensional coordinates of voxels, respectively) to describe the sample points in each voxel, and adds the current column center for each sample. Coordinates (P _x , P _y ). At this point, the description vector of the sample points in each voxel becomes 9-dimensional. In one embodiment of the present invention, the feature encoding layer (VFE) includes a linear layer, a batch normalization layer (BN), and a corrected linear unit layer (ReLU) to extract vector features of points.

In step S20, the feature expression of the voxel is obtained through a feature extractor and sparse convolution coding is performed to obtain a spatial sparse feature map corresponding to the data to be processed, as shown in formula (2):

Among them, F() represents the feature expression of voxels obtained by the feature extractor, D represents the voxel representation of the laser point cloud data, and (x, y, z) represents the spatial coordinates of the spatially sparse feature map.

Step S30, project the spatial sparse feature map to a two-dimensional top-view plane, obtain features of different scales through a feature pyramid convolution network, and combine the features of different scales through a deconvolution layer to obtain a top-view feature map.

Projecting the spatially sparse feature map f _s (x, y, z) to the top view (that is, a bird’s-eye view) is to compress the vertical dimension of the spatial sparse feature map f _s (x, y, z) to obtain the top-view feature map f _2D (u,v). Specifically, assuming that the original feature is (C, D, H, W), the height feature is merged into the feature channel to become (C×D, H, W), and the 2D convolution feature is the top-view feature map. The features of different scales of f _2D (u, v) are obtained through the feature pyramid convolutional network, and the features of different scales are combined through the deconvolution layer to obtain the feature map f _FPN (u, v). In an embodiment of the present invention, the feature pyramid convolutional layer includes three convolutional groups, which respectively have (3, 5, 5) convolutional layers, and each convolutional layer is followed by a batch normalization layer (BN), a corrected linear Unit Layer (ReLU).

In step S40, the attention weight feature map of the top-view feature map is obtained through an attention weight layer; the coding feature map of the top-view feature map is obtained through a convolutional coding layer.

The attention weight feature map of the top-view feature map is obtained through the attention weight layer, as shown in formula (3):

F _att (u,v)=Conv _att (F _FPN (u,v)) Equation (3)

Among them, F _att (u, v) represents the attention weight feature map corresponding to the top-view feature map, F _FPN (u, v) represents the top-view feature map, and Conv _att () represents the attention weight layer convolution operation.

The coding feature map of the top-view feature map is obtained through the convolutional coding layer, as shown in equation (4):

F _en (u,v)= _Conven (F _FPN (u,v)) Equation (4)

Among them, F _en (u, v) represents the encoded feature map corresponding to the top-view feature map, F _FPN (u, v) represents the top-view feature map, and _Conven () represents the convolutional encoding layer convolution operation.

Step S50, multiply the attention weight feature map to the corresponding area of the encoding feature map, and perform feature splicing to obtain the attention feature map, as shown in equations (5) and (6):

F _op (u,v)=F _en (u,v)Repeat(Reshape(F _att (u,v))) Equation (5)

Among them, Reshape() represents the deformation operation, and Repeat() represents the copy operation;

Among them, [ ] represents the feature concatenation operation.

Step S60, based on the attention feature map, obtain the target category in the data to be detected through the trained target classification network; based on the attention feature map, obtain the target position in the data to be detected through the trained target regression positioning network , size, orientation.

As shown in FIG. 2, it is a schematic diagram of the algorithm structure of an embodiment of the three-dimensional visual detection method based on the shape attention mechanism of the present invention, which is divided into three parts: the first part is a distance constraint-based voxel generator (Distance -based Voxel Generator), which converts the input lidar point cloud into voxels; the second part is the feature extraction layer, which encodes voxel features and encodes 3D spatial features; the third part is the attention region recommendation network (Attention RPN), inject the attention mechanism to output the detection result.

The target classification network is trained through the cross-entropy loss function, and the cross-entropy loss function is shown in formula (7):

Among them, N represents the number of samples for calculating the loss; y _i represents the positive and negative samples, 0 represents the negative sample, and 1 represents the positive sample; _xi represents the network output value of the sample.

The target regression positioning network is trained through the Smooth L1 loss function. The Smooth L1 loss function is shown in formula (8):

where x represents the residual of the regression.

The attention feature map F _hybrid (u, v) is connected to the target classification network and the target regression positioning network respectively. The target classification network is used to determine whether the detection object is a target, and the target regression positioning network is used to obtain the position, size and direction of the detection object. .

In an embodiment of the present invention, for the class of the car in the target classification task, the intersection between the anchor point and the target (IOU) greater than 0.6 is set as a positive sample, and the intersection ratio is less than 0.45 as a negative sample; For the categories of pedestrians and cyclists, the intersection ratio (IOU) of anchor and target greater than 0.5 is set as a positive sample, and the intersection ratio less than 0.35 is set as a negative sample. For the regression positioning task, the width × length × height of the predefined anchor point corresponding to the target vehicle is set as (1.6 × 3.9 × 1.5) meters; for the target pedestrian, the width × length × height of the predefined anchor point is set as (0.6 × 0.8 × 1.73) meters; the predefined anchor point for the target cyclist has a width x length x height of (0.6 x 1.76 x 1.73) meters. Define a three-dimensional real bounding box as x _g , y _g , z _g , l _g , w _g , h _g , θ _g , where x, y, z are the center positions of the bounding box, and l, w, h represent the three-dimensional The length, width and height of the target, θ is the rotation angle of the target in the Z-axis direction, * _g represents the real value, * _a represents the anchor point of the positive sample, and Δ* represents the corresponding residual. Through network learning, predict the real three-dimensional target location, size and orientation. The residuals of the center position of the bounding box (Δx, Δy, Δz), the residuals of the length, width, and height of the three-dimensional target (Δl, Δw, Δh), and the residuals of the target’s turning angle in the Z-axis direction (Δθ) are shown in formula (9), Formulas (10) and (11) are shown:

Δθ=sin(θ _g -θ _a ) Equation (11)

In order to illustrate the effectiveness of the present invention in detail, the method proposed by the present invention is applied to the public unmanned driving dataset KITTI, which contains 3 verification categories. As shown in FIG. 3, it is an example diagram of the data set and detection result of an embodiment of the 3D visual detection method based on the shape attention mechanism of the present invention. The first column Car represents the detection result of the vehicle, and the second column Pedestrian represents the detection of pedestrians. As a result, the third column Cyclist represents the detection results of cyclists. There are three sets of experimental results in each column, each set includes an RGB image and the top view of the radar, and the detection results are projected onto the graph.

In an embodiment of the present invention, for the KITTI data set, the train data set is used for training, and the test data set is used for testing. As shown in Figure 4, it is a comparison chart of the detection results of the method of the present invention and other methods of an embodiment of the three-dimensional visual detection method based on the shape attention mechanism of the present invention. The data set is divided into three levels for each type of test target: easy, Moderate and difficult. The difficulty is divided according to the height of each object in the camera image, the occlusion level and the degree of truncation. For samples with an easy difficulty, the height of the bounding box is greater than or equal to 40 pixels, the maximum truncation is 15%, and the occlusion level is fully visible; the height of the sample bounding box for the difficulty is greater than or equal to 25 pixels, the maximum truncation is 30%, and the occlusion level is Partial occlusion; samples with difficulty difficulty have a bounding box height of 25 pixels or more, a maximum truncation of 50%, and an occlusion level of hard to see. BEV represents the top view detection result, and 3D represents the detection result of the 3D bounding box. 3D object detection performance is evaluated using the PASCAL criterion (AP, average precision). In the comparison method, ARPNET represents the method of the present invention, MV3D represents the multi-view 3D object detection method, ContFuse represents the deep continuous fusion multi-sensor 3D object detection method, and AOVD represents the multi-view aggregated data to realize the real-time detection method of 3D objects in the unmanned scene. , F-PointNet stands for RGB-D data 3D object detection method of frustum point cloud network, SECOND stands for sparse embedding convolution object detection method, and Voxelnet stands for point cloud data 3D object detection method based on end-to-end learning.

The 3D visual detection system based on the shape attention mechanism according to the second embodiment of the present invention, the 3D target detection system includes an input module, a sparse convolution coding module, a feature pyramid module, an attention weight convolution module, a coding convolution module, a feature Fusion module, target classification module, target positioning module, output module;

The input module is configured to obtain laser point cloud data containing the target as the data to be detected, and to characterize the data to be detected by voxels based on a three-dimensional network;

The sparse convolution coding module is configured to obtain the feature expression of the voxel through a feature extractor and perform sparse convolution coding to obtain a spatial sparse feature map corresponding to the data to be processed;

The feature pyramid module is configured to project the spatial sparse feature map to a two-dimensional top-view plane, obtain features of different scales through a feature pyramid convolutional network, and combine the features of different scales through a deconvolution layer to obtain Top view feature map;

The attention weight convolution module is configured to obtain the attention weight feature map of the top-view feature map through the attention weight layer;

The coding convolution module is configured to obtain the coding feature map of the top-view feature map through a convolution coding layer;

The feature fusion module is configured to multiply the attention weight feature map to the corresponding region of the encoding feature map, and perform feature splicing to obtain the attention feature map;

The target classification module is configured to obtain target categories in the data to be detected through the trained target classification network based on the attention feature map;

The target positioning module is configured to obtain the target position, size and direction in the data to be detected through the trained target regression positioning network based on the attention feature map;

The output module is configured to output the acquired target category and target position, size, and direction.

Those skilled in the art can clearly understand that, for the convenience and brevity of description, for the specific working process and related description of the system described above, reference may be made to the corresponding process in the foregoing method embodiments, which will not be repeated here.

It should be noted that, the three-dimensional visual inspection system based on the shape attention mechanism provided by the above embodiments is only illustrated by the division of the above functional modules. In practical applications, the above functions can be assigned to different functions module, that is, the modules or steps in the embodiments of the present invention are decomposed or combined. For example, the modules in the above embodiments can be combined into one module, and can also be further split into multiple sub-modules to complete all or part of the above description. Function. The names of the modules and steps involved in the embodiments of the present invention are only for distinguishing each module or step, and should not be regarded as an improper limitation of the present invention.

A storage device according to a third embodiment of the present invention stores a plurality of programs, and the programs are adapted to be loaded and executed by a processor to implement the above-mentioned three-dimensional visual inspection method based on a shape attention mechanism.

A processing device according to a fourth embodiment of the present invention includes a processor and a storage device; the processor is adapted to execute various programs; the storage device is adapted to store multiple programs; the programs are adapted to be loaded and executed by the processor In order to realize the above-mentioned 3D visual detection method based on shape attention mechanism.

Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process and relevant description of the storage device and processing device described above can refer to the corresponding process in the foregoing method embodiments, which is not repeated here. Repeat.

Those skilled in the art should be aware that the modules and method steps of each example described in conjunction with the embodiments disclosed herein can be implemented by electronic hardware, computer software or a combination of the two, and the programs corresponding to the software modules and method steps Can be placed in random access memory (RAM), internal memory, read only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or as known in the art in any other form of storage medium. In order to clearly illustrate the interchangeability of electronic hardware and software, the components and steps of each example have been described generally in terms of functionality in the foregoing description. Whether these functions are performed in electronic hardware or software depends on the specific application and design constraints of the technical solution. Skilled artisans may use different methods of implementing the described functionality for each particular application, but such implementations should not be considered beyond the scope of the present invention.

The terms "first," "second," etc. are used to distinguish between similar objects, and are not used to describe or indicate a particular order or sequence.

The term "comprising" or any other similar term is intended to encompass a non-exclusive inclusion such that a process, method, article or device/means comprising a list of elements includes not only those elements but also other elements not expressly listed, or Also included are elements inherent to these processes, methods, articles or devices/devices.

So far, the technical solutions of the present invention have been described with reference to the preferred embodiments shown in the accompanying drawings, however, those skilled in the art can easily understand that the protection scope of the present invention is obviously not limited to these specific embodiments. Without departing from the principle of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will fall within the protection scope of the present invention.

Claims (11)

1. A three-dimensional visual inspection method based on a shape attention mechanism is characterized by comprising the following steps:

step S10, laser point cloud data containing a target object are obtained to serve as data to be detected, and the data to be detected are represented through voxels based on a three-dimensional network;

step S20, acquiring the feature expression of the voxel through a feature extractor and performing sparse convolution coding to obtain a space sparse feature map corresponding to the data to be processed;

step S30, projecting the space sparse feature map to a two-dimensional top view plane, acquiring features of different scales through a feature pyramid convolution network, and then combining the features of different scales through deconvolution lamination to obtain a top view feature map;

step S40, acquiring an attention weight feature map of the top view feature map through an attention weight layer; acquiring a coding feature map of the top view feature map through a convolution coding layer;

step S50, multiplying the attention weight feature map to the corresponding area of the coding feature map, and performing feature splicing to obtain an attention feature map;

step S60, acquiring target categories in the data to be detected through a trained target classification network based on the attention feature map; and acquiring the position, the size and the direction of the target in the data to be detected through the trained target regression positioning network based on the attention feature map.

2. The three-dimensional visual inspection method based on shape attention mechanism according to claim 1, wherein in step S10, "the data to be inspected is characterized by voxels based on three-dimensional network", which is performed by:

wherein D represents the voxel representation of the laser point cloud data, x_i、y_i、z_iRepresenting the three-dimensional position information of the ith point in the laser point cloud data relative to the laser radar, R_iRepresenting the reflectivity of the ith point in the laser point cloud data.

3. A three-dimensional visual inspection method based on a shape attention mechanism according to claim 1, wherein in step S20, "obtaining the feature expression of the voxel by a feature extractor and performing sparse convolution coding to obtain a spatial sparse feature map corresponding to the data to be processed" includes:

wherein, F () represents the feature representation of the voxel obtained by the feature extractor, D represents the voxel representation of the laser point cloud data, and (x, y, z) represents the spatial coordinates of the spatial sparse feature map.

4. A three-dimensional visual inspection method based on a shape attention mechanism according to claim 1, wherein in step S40, "obtaining the attention weight feature map of the top view feature map through the attention weight layer" includes:

F_att(u,v)＝Conv_att(F_FPN(u,v))

wherein, F_att(u, v) represents the attention weight feature map corresponding to the top view feature map, F_FPN(u, v) represents a top view feature map, Conv_att() Representing the attention weight layer convolution operation.

5. A three-dimensional visual inspection method based on shape attention mechanism according to claim 1, wherein in step S40, "obtaining the encoding feature map of the top view feature map by convolution encoding layer" comprises:

F_en(u,v)＝Conv_en(F_FPN(u,v))

wherein, F_en(u, v) represents the coding feature map corresponding to the top view feature map, F_FPN(u, v) represents a top view feature map, Conv_en() Representing a convolutional encoding layer convolution operation.

6. A three-dimensional visual inspection method based on a shape attention mechanism according to claim 1, wherein in step S50, the method comprises the steps of multiplying the attention weight feature map to the corresponding region of the coding feature map and performing feature matching to obtain the attention feature map, and comprises the steps of:

F_op(u,v)＝F_en(u,v)Repeat(Reshape(F_att(u,v)))

wherein, resume () represents the deformation operation, and Repeat () represents the copy operation;

wherein [ ] represents a characteristic splicing operation.

7. The three-dimensional visual inspection method based on shape attention mechanism according to any one of claims 1-6, characterized in that the object classification network is trained by cross entropy loss function; the cross entropy loss function is:

wherein N represents the number of samples for which loss is calculated; y is_iRepresents positive and negative samples, with 0 representing a negative sample and 1 representing a positive sample; x is the number of_iA network output value representing a sample.

8. The three-dimensional visual inspection method based on shape attention mechanism according to any one of claims 1-6, characterized in that the target regression positioning network is trained by Smooth L1 loss function; the Smooth L1 loss function is:

where x represents the residual of the regression.

9. A three-dimensional visual inspection detection system based on a shape attention mechanism is characterized by comprising an input module, a sparse convolution coding module, a characteristic pyramid module, an attention weight convolution module, a coding convolution module, a characteristic fusion module, a target classification module, a target positioning module and an output module;

the input module is configured to acquire laser point cloud containing target object data as to-be-detected data, and the to-be-detected data is represented by voxels based on a three-dimensional network;

the sparse convolution coding module is configured to obtain the characteristic expression of the voxel through a characteristic extractor and carry out sparse convolution coding to obtain a spatial sparse characteristic diagram corresponding to the data to be processed;

the characteristic pyramid module is configured to project the space sparse characteristic diagram to a two-dimensional top view plane, obtain characteristics of different scales through a characteristic pyramid convolution network, and then combine the characteristics of different scales through deconvolution lamination to obtain a top view characteristic diagram;

the attention weight convolution module is configured to acquire an attention weight feature map of the top view feature map through an attention weight layer;

the coding convolution module is configured to acquire a coding feature map of the top view feature map through a convolution coding layer;

the feature fusion module is configured to multiply the attention weight feature map to a corresponding region of the coding feature map, and perform feature splicing to obtain an attention feature map;

the target classification module is configured to obtain a target class in the data to be detected through a trained target classification network based on the attention feature map;

the target positioning module is configured to obtain the position, the size and the direction of a target in the data to be detected through a trained target regression positioning network based on the attention feature map;

the output module is configured to output the acquired object type, and the object position, size and direction.

10. A storage device having stored therein a plurality of programs, wherein the programs are adapted to be loaded and executed by a processor to implement the method for three-dimensional visual inspection based on the shape attention mechanism of any one of claims 1 to 8.

11. A treatment apparatus comprises

A processor adapted to execute various programs; and

a storage device adapted to store a plurality of programs;

wherein the program is adapted to be loaded and executed by a processor to perform:

the three-dimensional visual inspection method based on the shape attention mechanism as set forth in any one of claims 1 to 8.

Images