CN117333676B - Point cloud feature extraction method and point cloud visual detection method based on graph expression - Google Patents

Point cloud feature extraction method and point cloud visual detection method based on graph expression Download PDF

Info

Publication number
CN117333676B
CN117333676B CN202311628039.3A CN202311628039A CN117333676B CN 117333676 B CN117333676 B CN 117333676B CN 202311628039 A CN202311628039 A CN 202311628039A CN 117333676 B CN117333676 B CN 117333676B
Authority
CN
China
Prior art keywords
point cloud
graph
feature vector
branch
layer
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202311628039.3A
Other languages
Chinese (zh)
Other versions
CN117333676A (en
Inventor
汪洋
窦文豪
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Shenzhen Graduate School Harbin Institute of Technology
Original Assignee
Shenzhen Graduate School Harbin Institute of Technology
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Shenzhen Graduate School Harbin Institute of Technology filed Critical Shenzhen Graduate School Harbin Institute of Technology
Priority to CN202311628039.3A priority Critical patent/CN117333676B/en
Publication of CN117333676A publication Critical patent/CN117333676A/en
Application granted granted Critical
Publication of CN117333676B publication Critical patent/CN117333676B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features
    • G06V10/44—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
    • G06V10/443—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components by matching or filtering
    • G06V10/449—Biologically inspired filters, e.g. difference of Gaussians [DoG] or Gabor filters
    • G06V10/451—Biologically inspired filters, e.g. difference of Gaussians [DoG] or Gabor filters with interaction between the filter responses, e.g. cortical complex cells
    • G06V10/454—Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/084—Backpropagation, e.g. using gradient descent
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
    • G06V10/80—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level
    • G06V10/809—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level of classification results, e.g. where the classifiers operate on the same input data
    • G06V10/811—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level of classification results, e.g. where the classifiers operate on the same input data the classifiers operating on different input data, e.g. multi-modal recognition
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02A—TECHNOLOGIES FOR ADAPTATION TO CLIMATE CHANGE
    • Y02A90/00—Technologies having an indirect contribution to adaptation to climate change
    • Y02A90/10—Information and communication technologies [ICT] supporting adaptation to climate change, e.g. for weather forecasting or climate simulation

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Artificial Intelligence (AREA)
  • Evolutionary Computation (AREA)
  • Health & Medical Sciences (AREA)
  • General Physics & Mathematics (AREA)
  • General Health & Medical Sciences (AREA)
  • Multimedia (AREA)
  • Molecular Biology (AREA)
  • Biomedical Technology (AREA)
  • Software Systems (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Computing Systems (AREA)
  • Medical Informatics (AREA)
  • Databases & Information Systems (AREA)
  • Biodiversity & Conservation Biology (AREA)
  • Biophysics (AREA)
  • Computational Linguistics (AREA)
  • Data Mining & Analysis (AREA)
  • General Engineering & Computer Science (AREA)
  • Mathematical Physics (AREA)
  • Image Analysis (AREA)

Abstract

The point cloud feature extraction method and the point cloud visual detection method based on graph expression comprise the following steps: acquiring a graph expression of an original point cloud set and an index matrix P2G and an inverse index matrix G2P of the graph expression of the original point cloud set; converting the original point cloud set into a non-graph representation form to obtain a non-graph representation of the original point cloud set, and an index matrix P2X and an inverse index matrix X2P from the original point cloud set to the non-graph representation; inputting the non-graph representation into a non-graph feature extraction network branch, and inputting the graph representation into a graph neural network branch to obtain a non-graph branch feature vector and a graph branch feature vector respectively, wherein the non-graph feature extraction network branch is the first branchkLayer output eigenvector and graph neural network branchk-layer 1 output feature vector fusion input graph neural network branchingkA layer; and respectively restoring the non-graph branch feature vector and the graph branch feature vector according to the inverse index matrixes X2P and G2P. The invention can enhance the learning capability of the fine granularity information of the point cloud by virtue of the graph neural network based on graph expression in the graph neural network branches.

Description

Point cloud feature extraction method and point cloud visual detection method based on graph expression
Technical Field
The invention relates to the field of computer three-dimensional vision, in particular to a point cloud feature extraction method and a point cloud vision detection method based on graph expression.
Background
In the fields of automatic driving, three-dimensional measurement and the like, point cloud data of the surface of an object in a scene is often required to be acquired, and various visual detection tasks are performed, such as semantic segmentation or target detection based on point cloud. For point clouds, it may be converted into other expressions, such as voxel expressions, projection expressions of different projection directions, etc. For outdoor and sparsely distributed point cloud scenes such as automatic driving, the graph expression of the point cloud is seriously dependent on the calculation of Euclidean distance, and the calculation amount in the process is linearly increased along with the increase of the number of the point clouds; meanwhile, the neighbor point search based on Euclidean distance, the determination of directed edges and the establishment of a graph are directly related to the distribution density of point clouds, and the distribution of the point clouds of an outdoor scene tends to be sparse along with the increase of the object distance. For the above reasons, studies based on the expression of point cloud images have been very rare in the automatic driving scene so far, and the image creation is generally carried out in a local area.
In the process of executing the above-mentioned point cloud visual detection task, an important step is to perform feature extraction on the point cloud first. At the current stage, the point cloud feature extraction is generally carried out by 3D convolution with relatively good effect, and the 3D convolution process based on the point cloud voxel expression cannot avoid the influence of losing original point cloud information due to the ground voxel accuracy, and the calculated amount is extremely large.
Disclosure of Invention
The method mainly solves the technical problem that the existing point cloud feature extraction method is easy to lose the point cloud detail information.
According to a first aspect, in one embodiment, a point cloud feature extraction method based on graph expression is provided, including:
acquiring an original point cloud set P acquired by a laser radar;
converting the original point cloud set P to obtain a graph representation G of the original point cloud set, and obtaining an index matrix P2G and an inverse index matrix G2P from the original point cloud set P to the graph representation G; converting the original point cloud set P into a target expression form X of a non-graph expression form to obtain a non-graph expression of the original point cloud set P, and obtaining an index matrix P2X and an inverse index matrix X2P from the original point cloud set P to the non-graph expression;
inputting the non-graph representation G of the original point cloud set into a non-graph feature extraction network branch of a preset point cloud feature extraction network to obtain a non-graph branch feature vector, and inputting the graph representation G of the original point cloud set into a graph neural network branch of the point cloud feature extraction network to obtain a graph branch feature vector, wherein the non-graph feature extraction network branch is the first branch of the non-graph feature extraction networkkLayer output eigenvector and the graph neural network branch kThe output characteristic vector of the-1 layer is fused and then used as the branch of the graphic neural networkkAn input feature vector of a layer, whereink∈[1,L-1]，LRepresenting the non-graphical feature extraction netThe total layer number of the network branches and the graph neural network branches, and the output characteristic vector of the 0 th layer of the graph neural network branches refers to the graph expression of the original point cloud set;
and converting the non-image branch feature vector according to the inverse index matrix X2P to obtain a non-image expression feature vector of the original point cloud set, and converting the image branch feature vector according to the inverse index matrix G2P to obtain an image expression feature vector of the original point cloud set.
According to a second aspect, in one embodiment, there is provided a point cloud visual detection method based on graph expression, including:
obtaining non-graph expression feature vectors and graph expression feature vectors of the original point cloud set by the point cloud feature extraction method;
inputting the non-graph representation feature vector and/or the graph representation feature vector of the original point cloud set into a target detection head network or a semantic segmentation head network to obtain a visual detection result, wherein the visual detection result comprises a prediction result of the category of each vertex in the graph representation of the original point cloud set.
According to a third aspect, an embodiment provides a computer readable storage medium having a program stored thereon, the program being executable by a processor to implement the above-described point cloud feature extraction method or point cloud visual detection method.
According to the point cloud feature extraction method and the point cloud visual detection method based on graph expression, an original point cloud set P is converted to obtain a graph representation G of the original point cloud set, and an index matrix P2G and an inverse index matrix G2P of the graph representation G of the original point cloud set P; converting the original point cloud set P into a target expression form X of a non-graph expression form to obtain a non-graph expression of the original point cloud set P, and obtaining an index matrix P2X and an inverse index matrix X2P from the original point cloud set P to the non-graph expression; then, carrying out feature extraction on a non-graph feature extraction network branch of a non-graph expression input point cloud feature extraction network and a graph neural network branch of a graph expression G input point cloud feature extraction network to respectively obtain a non-graph branch feature vector and a graph branch feature vector, wherein the non-graph feature extraction network branch is the first branchkLayer output characteristicsVector and graph neural network branchingkThe output characteristic vectors of the layer-1 are fused and then used as the branch of the graphic neural network kAn input feature vector of the layer; and finally, converting the non-graph branch feature vector according to the inverse index matrix X2P to obtain a non-graph expression feature vector of the original point cloud set, and converting the graph branch feature vector according to the inverse index matrix G2P to obtain a graph expression feature vector of the original point cloud set, thereby realizing extraction of point cloud features. It can be seen that the invention forms a graph expression feature extraction branch facing the feature extraction link on the basis of graph expression of the point cloud, can be directly matched with feature extraction networks of other expression forms of the point cloud for use, and has extremely strong compatibility with the existing method; meanwhile, due to the graph neural network based on graph expression in the graph neural network branches, the feature extraction capability of the point cloud fine granularity information can be enhanced, and the calculation amount is smaller compared with 3D convolution.
Drawings
FIG. 1 is a flow chart of a point cloud feature extraction method based on graph representation of one embodiment;
FIG. 2 is a schematic diagram of a point cloud feature extraction network according to an embodiment;
FIG. 3 is a non-image feature extraction network branch of one embodimentkLayer output eigenvector and graph neural network branchk-a flow chart of fusion of the output feature vectors of layer 1;
FIG. 4 is a flow diagram of the processing of the multiple expression channel gated attention module in one embodiment;
FIG. 5 is a schematic diagram of a multiple expression channel gated attention module versus a branch of a graph neural network in one embodimentkOutput feature vector and index conversion feature vector of-1 layerX_in k Performing a flow chart of feature fusion with a channel-gated attention mechanism;
FIG. 6 is a flow chart of a point cloud visual detection method based on graph representation of one embodiment;
FIG. 7 is a schematic diagram of a point cloud visual inspection network according to an embodiment;
FIG. 8 is a schematic diagram of a point cloud visual inspection network according to another embodiment;
FIG. 9 is a schematic diagram showing the relationship between the three-dimensional coordinates of the point cloud and the horizontal azimuth angle and vertical pitch angle of the laser beam corresponding to the point cloud;
FIG. 10 is a schematic diagram of a lidar pixel matrix of an embodiment;
FIG. 11 is a schematic diagram of finding a set of neighbor points of a point cloud and establishing a set of directed edges in one embodiment;
FIG. 12 is a flow diagram of a graph representation G of an original point cloud P converted to obtain the original point cloud in one embodiment;
FIG. 13 is a graph showing the horizontal azimuth and vertical pitch profiles of a purely mechanical or semi-solid lidar;
fig. 14 is a schematic diagram of the basic imaging principle of a biprism semi-solid laser radar.
Detailed Description
The invention will be described in further detail below with reference to the drawings by means of specific embodiments. Wherein like elements in different embodiments are numbered alike in association. In the following embodiments, numerous specific details are set forth in order to provide a better understanding of the present application. However, one skilled in the art will readily recognize that some of the features may be omitted, or replaced by other elements, materials, or methods in different situations. In some instances, some operations associated with the present application have not been shown or described in the specification to avoid obscuring the core portions of the present application, and may not be necessary for a person skilled in the art to describe in detail the relevant operations based on the description herein and the general knowledge of one skilled in the art.
Furthermore, the described features, operations, or characteristics of the description may be combined in any suitable manner in various embodiments. Also, various steps or acts in the method descriptions may be interchanged or modified in a manner apparent to those of ordinary skill in the art. Thus, the various orders in the description and drawings are for clarity of description of only certain embodiments, and are not meant to be required orders unless otherwise indicated.
The numbering of the components itself, e.g. "first", "second", etc., is used herein merely to distinguish between the described objects and does not have any sequential or technical meaning. The terms "coupled" and "connected," as used herein, are intended to encompass both direct and indirect coupling (coupling), unless otherwise indicated. "lidar", "lidar sensor" herein refer to the same thing, and vertices in the graph representation are also referred to as "nodes", "graph nodes".
The existing point cloud semantic segmentation method can be divided into the following types according to the point cloud expression form of the input deep learning network.
1. The method is characterized in that the method is input in a discrete point form, the characteristics of global points are generally aggregated and learned by combining a multi-layer perceptron and maximum pooling, the aggregation of the characteristics of the points in local neighborhood is enhanced by using a sampling method based on Euclidean distance, and the semantics of the points are directly classified. Defects: the discrete points require a network model with a symmetrical structure, and the ordering of the input points has characteristic invariance, which means that the extraction of the characteristics must be realized through a pooling process; the point-to-point operation depends on judgment under the Euclidean distance, and the calculation cost is high.
2. The method comprises the steps of inputting in a voxel expression form of point cloud, converting discrete points into voxel blocks by three-dimensional voxelization of three-dimensional unordered point cloud, aggregating and extracting point cloud features under 3D convolution, finally carrying out semantic classification of the voxel blocks, and finally mapping the voxel blocks into semantic information of the discrete points. Defects: the computation and storage costs of 3D convolution are extremely high; the size of the voxels directly determines the thickness of the point cloud expression, and the large-size voxels are very easy to cause blurring of the point cloud detail information.
3. The three-dimensional point cloud problem is converted into a two-dimensional image problem by inputting the three-dimensional point cloud in a projection mode, and dividing the three-dimensional point cloud into a bird's eye view projection and a spherical projection for front view according to different projection directions. Defects: the conversion of three-dimensional information into two-dimensional information is believed to result in loss of information, but the calculation cost is low, but the information loss is large, so that the accuracy is not high.
4. And (3) inputting the graph representation of the point cloud, and extracting graph node characteristics through a neural network by establishing graph structure data of the point cloud to realize semantic classification of the graph nodes. Defects: the method is only suitable for dense objects or indoor point cloud model data, and large outdoor scenes are difficult to perform semantic segmentation by using graph representation of point clouds.
The invention provides a point cloud feature extraction method and a point cloud visual detection method based on graph expression.
The point cloud feature extraction network provides a feature extraction branch based on point cloud image expression, and the feature extraction branch can be combined with a point cloud segmentation or target detection deep learning model of the existing voxel point cloud or projection point cloud and other expression modes to form a multi-mode fusion segmentation or detection method for carrying out point cloud feature extraction in other point cloud expression modes and carrying out point cloud feature aggregation and propagation in a graph expression mode. Besides feature information obtained from feature extraction branches of other expression forms, feature extraction branches of the graph expression also support point cloud graph nodes to learn neighbor features along directed edges, and feature propagation of the features of the other expression forms in the original resolution graph expression is formed.
Referring to fig. 1, a point cloud feature extraction method based on graph expression in an embodiment of the invention includes steps 100 to 400, which are described in detail below.
Step 100: and acquiring an original point cloud set P acquired by the laser radar.
The laser radar can be a pure solid laser radar, a mechanical laser radar or a semi-solid laser radar and other laser radars. The original point cloud P can be expressed asWherein, the method comprises the steps of, wherein,Nfor the number of point clouds of the original point cloud set, (3 +C+L) Representing the information dimension of each point cloud, and 3 represents #x,y,z) Coordinates of，CRepresenting characteristic information such as the reflection intensity of the point cloud,Ltag information representing the point cloud. In general terms, the process is carried out,Cthe reflected intensity information expressed is located at [0,1 ]]The intensity signal value in the interval and other possibly related characteristic information are also attribute information expressed by a certain numerical value. Label (Label)LGenerally, the tree is also an integer value, and in practical application, information such as "tree", "building" and the like can be represented by an integer value map greater than 0 according to different application scenes.
Step 200: converting the original point cloud set P to obtain a graph representation G of the original point cloud set, and obtaining an index matrix P2G and an inverse index matrix G2P of the original point cloud set P to the graph representation G; and converting the original point cloud set P into a target expression form X of a non-graph expression form to obtain a non-graph expression of the original point cloud set, and obtaining an index matrix P2X and an inverse index matrix X2P of the original point cloud set P to the non-graph expression.
The method of obtaining the graph representation G of the original point cloud set may use various prior arts, and may also use a new graph representation conversion method provided below. The graph representation G of the original point cloud can be expressed asWhereinVRepresenting a set of vertices that are to be processed,Erepresenting a set of edges,p i representing the first point of the original point cloud PiThe point cloud of the object is a point cloud,PN i representing a point cloudp i Is set of neighbor points of the (c),p k representing a point cloudp i Neighbor point of->Representation ofp k Pointing top i Is a directional edge of (a). It will be appreciated that the index matrix P2G and the inverse index matrix G2P of the original point cloud P to the graph representation G may be obtained after the graph representation G of the original point cloud is obtained.
The target expression form X includes, but is not limited to, voxelization of the point cloud, projection compression of the point cloud in a bird's eye view or a front view, etc., and various prior arts can be used for the specific conversion method, which is not described herein. Similarly, for any target expression form X, an index matrix P2X and an inverse index matrix X2P of the original point cloud P converted to the point cloud sequence under the expression form X can be obtained.
Step 300: inputting the non-graph representation of the original point cloud set into a non-graph feature extraction network branch of a preset point cloud feature extraction network to obtain a non-graph branch feature vector, and branching the graph neural network of the G input point cloud feature extraction network of the original point cloud set into a graph neural network branch of the G input point cloud feature extraction network to obtain a graph branch feature vector, wherein the non-graph feature extraction network branch is the first branch kLayer output eigenvector and graph neural network branchkThe output characteristic vectors of the layer-1 are fused and then used as the branch of the graphic neural networkkAn input feature vector of a layer, whereink∈[1,L-1]，LThe output feature vector of the 0 th layer of the graph neural network branch refers to the graph representation of the original point cloud set.
Referring to fig. 2, the point cloud feature extraction network provided by the present invention includes two branches, namely, a non-graph feature extraction network branch and a graph neural network branch, wherein the input of the non-graph feature extraction network branch is the non-graph representation of the original point cloud set, the input of the graph neural network branch is the graph representation of the original point cloud set, and the layers of the two areL，LIs a positive integer greater than or equal to 2, in FIG. 2X k Representing non-graph feature extraction network branchingkThe layer of the material is formed from a layer,G k representing the first branch of the graph neural networkkA layer. The non-graph feature extraction network branch is mainly responsible for the learning and extraction of features, the learned and extracted features are transferred to the graph neural network for fusion, and the graph neural network branch is mainly responsible for the aggregation and propagation of the features. The non-graph feature extraction network branch can adopt various existing feature extraction networks such as UNet and the like, and the graph neural network branch can also be realized by using any type of graph neural network.
The point cloud feature extraction method based on graph expression of the present invention supports the co-implementation of one or more non-graph expressions and graph expressions, i.e. one or more non-graph expressions obtained in step 200 may be provided, and correspondingly, one or more non-graph feature extraction network branches are provided, which are respectively used for extracting features of one non-graph expression.
In practical application, the number of characteristic channels output by each layer of the graph neural network branch is the same as the number of input characteristic channels, and the number of characteristic channels output by each layer of the graph neural network branch is usually different, so that the non-graph characteristic extraction network branch is branched to the firstkLayer output eigenvector and graph neural network branchkWhen the output eigenvectors of the layer-1 are fused, the output eigenvectors need to be branched with the graphic neural networkkThe number of characteristic channels of the output characteristic vector of the layer is aligned so as to input the first branch of the graphic neural networkkA layer.
Step 400: and converting the non-graph branch feature vector according to the inverse index matrix X2P to obtain a non-graph representation feature vector of the original point cloud set, and converting the graph branch feature vector according to the inverse index matrix G2P to obtain a graph representation feature vector of the original point cloud set.
The non-graph branch feature vector obtained in step 300 is still in the non-graph representation form, and the graph representation feature vector is still in the graph representation form, so that in this step, the features extracted from the original point cloud set P are obtained by respectively restoring according to the inverted index matrices X2P and G2P, and thus, feature extraction of the original point cloud set P is completed. After obtaining the non-graph representation feature vector and the graph representation feature vector, the non-graph representation feature vector and the graph representation feature vector can be input into a network for performing subsequent visual detection tasks, for example, a target detection head network or a semantic segmentation head network and the like to execute corresponding tasks.
In some embodiments, the point cloud feature extraction network further includes a multi-expression channel-gating attention module, configured to implement feature fusion of a channel-gating attention mechanism by using multiple expression modes; non-graph feature extraction network branchingkLayer output eigenvector and graph neural network branchkThe output feature vectors of layer-1 are fused according to the procedure shown in fig. 3, comprising the steps of:
step 310, obtaining an index matrix P2G of the original point cloud set P-graph representation;
step 320, non-graph feature extraction network branchingkLayer output feature vector X k Upsampling to the same resolution as the original point cloud PRate and get non-graph feature extraction network branchingkLayer output feature vector X k Index matrix X2P to original point cloud P k ；
Wherein the up-sampling can adopt linear interpolation, deconvolution and other methods, and the output characteristic vector X can be obtained according to the up-sampling result k Index matrix X2P to original point cloud P k ；
Step 330, according to the index matrix X2P k And indexing the matrix P2G to obtain the non-image feature extraction network branchkIndex matrix X2G of layer output eigenvector-to-graph representation k Wherein X2G k =X2P k [P2G]；
Step 340, indexing matrix X2G k Extracting network branches for non-graph featureskLayer output feature vector X k Converting to obtain index conversion feature vectorX_in k WhereinX_in k =X k [X2G k ]；
Step 350, branching the graph neural networkkOutput feature vector and index conversion feature vector of-1 layerX_in k Inputting the multi-expression channel-gating attention module to perform feature fusion with a channel-gating attention mechanism to obtain a graph neural network branchkThe input feature vector of the layer.
Index conversion feature vector obtained by any layer from non-graph feature extraction network branchX_in k Still representing point cloud features under non-graph representation, so that the point cloud features are combined with non-graph representation branch features for better fusion before being input into the graph neural network branch, and are connected with the graph neural network branchkThe characteristics of the layer-1 are subjected to multi-expression characteristic fusion, and the multi-expression characteristic fusion is realized by a multi-expression channel gating attention module.
In some embodiments, the processing procedure of the multiple expression channel-gated attention module is shown in fig. 4, and the multiple expression channel-gated attention module branches the neural network of the graph by the flow shown in fig. 5kOutput feature vector of-1 layerG k-1 And index conversion feature vectorX_in k Feature fusion with a channel-gated attention mechanism is performed to obtain the branch of the graph neural networkkThe input feature vector of the layer is described in detail below.
Step 351: opposite-graph neural network branchkOutput feature vector of-1 layerG k-1 And index conversion feature vectorX_in k Adding or splicing and fusing to obtain approximate fusion feature vector。
The fusion can be a common multi-feature fusion method such as splicing fusion. The invention provides another fusion method: expressing features for a given point cloudAnd Point cloud non-graph representation feature +.>Wherein->Represents a real number and is used to represent a real number,Nfor the number of point clouds,Cfor the feature dimension of the current layer, an approximate fusion feature is obtained according to the following equation (1)：
， （1）
Thus, the graph neural network can be branched to the fifthkOutput feature vector of-1 layer asf G Converting index into feature vectorX_in k As a means off X Substituting the vector into the formula (1) to obtain an approximate fusion feature vector。
Step 352: based on approximate fusion feature vectorCalculation map neural network BranchkRelative channel attention score for output feature vector of layer-1score G And index conversion feature vectorX_in k Relative channel attention score of (2)score X 。
Relative channel attention score represents the first branch of the neural networkkOutput feature vector and index conversion feature vector of-1 layerX_in k The importance of each channel or dimension. In some embodiments, the neural network may be branched k-layer 1 output feature vector or index conversion feature vectorX_in k Channel-by-channel features in an approximation fused feature vectorThe ratio of the channel is used as an importance evaluation basis to calculate the relative channel attention score.
In some embodiments, the index converts feature vectorsX_in k Characteristic channel number and graph neural network branch numberkThe number of feature channels of the output feature vector of the layer is the same, and the relative channel attention scorescore G Andscore X is determined by the following calculation formula:
， （2）
wherein the method comprises the steps ofG k-1 Representing the branch of the graph neural networkkThe output feature vector of layer-1,represents an L2 norm;for trainable coefficient parameters, i.e. parameters determined by the training process, +.>Represents a real number and is used to represent a real number,C k representing the branch of the graph neural networkkOutput feature vector and index conversion feature vector of layerX_in k Is a characteristic channel number of (a); MLP (Multi-layer Programming protocol) G Representing the branch of the graph neural networkkSingle layer perceptron for relative channel attention scoring of output feature vectors of layer-1, MLP G Representing index-to-index conversion feature vectorsX_in k A single layer perceptron that performs relative channel attention scoring; MLP represents a linear transformation implemented by a single-layer perceptron to align output feature vectorsG k-1 And index conversion feature vectorX_in k Because of the number of characteristic channelsG k-1 Often the number of characteristic channels of (a) is equal to X_in k Different.
Step 353: scoring relative channel attentionscore G Andscore X gating activation is carried out to respectively obtain the graph neural network branchkGating coefficients and index conversion feature vector of output feature vector of-1 layerX_in k Gate control coefficient of (2), according to gate control coefficient, branching the graph neural networkkOutput feature vector and index conversion feature vector of-1 layerX_in k Weighted and added to obtain the branch of the graph neural networkkThe input feature vector of the layer.
In general, features are expressed for a point cloud of inputsAnd point cloud non-graph representation featuresThe invention relates to the output characteristics of a multi-expression channel gating attention modulefThe calculation formula of (2) is as follows:
f=σ 1 (score G )×f G +σ 1 (score X )×f X 。 （3）
in particular to the present embodiment, the graph neural network branches tokThe calculation formula of the input feature vector of the layer is:
G_in k =σ 1 (score G )×G k-1 +σ 1 (score X )×X_in k ， （4）
wherein the method comprises the steps ofG_in k Representing the branch of the graph neural networkkThe input feature vector of the layer is used,σ 1 in order to gate the activation function,σ 1 (score G ) Then representG k-1 Is used for the control of the gate coefficient of (c),σ 1 (score X ) Representation ofX_in k Is used for the gating coefficient of the (c). The gating activation function may be a Tanh function, a sigmoid function, an exponential function, or the like.
The graph neural network branches update the characteristics of each vertex in the graph representation of the original point cloud set. The method comprises updating the characteristic value or characteristic vector of each vertex in the original point cloud image expression, and inputting the first layer kThe graph representation of the original point cloud of the layer is the firstkGraph and non-graph feature extraction network branching of original point clouds of layer-1 outputkThe non-graph of the layer output represents the result of the fusion.
In one embodiment of the invention, the characteristics of the vertexes in the graph expression of the original point cloud set are updated in a graph attention network-based mode, and the process carries out weighted weight calculation according to each neighbor characteristic of the target node (namely the updated vertexes) under the guidance of the directed edges, and the updating of the characteristics of the target node is realized by the weighted weight calculation. In this embodiment, each layer of the graph neural network branch updates the eigenvalue of any vertex in the graph expression of the current layer original point cloud set, and the updated vertex updates the eigenvalue according to the following formula according to the neighbor nodes under the space geometrical distribution:
， （5）
wherein the method comprises the steps of、G i Respectively, the first of the map representations of the underlying original point cloudsiOutput characteristic values and input characteristic values of the vertexes,σ 2 in order to activate the function,Wfor a weight matrix of a trainable input linear transformation,PN i is the firstiA set of all neighbor vertices of the vertices,α ji is the firstiThe first of the verticesjThe neighbor vertices point to the firstiThe weighted value of the directed edges of the vertices. In one embodiment, the weighting values α ji The formula of (2) is as follows:
，
where I represents the concatenation operation, vectoraIs a trainable weight vector.
On the basis of the point cloud feature extraction method based on graph expression, the invention also provides a point cloud visual detection method based on graph expression, and referring to fig. 6, in one embodiment, the method comprises steps 10-50.
Step 10: and acquiring an original point cloud set P acquired by the laser radar.
Step 20: converting the original point cloud set P to obtain a graph representation G of the original point cloud set, and obtaining an index matrix P2G and an inverse index matrix G2P of the original point cloud set P to the graph representation G; and converting the original point cloud set P into a target expression form X of a non-graph expression form to obtain a non-graph expression of the original point cloud set, and obtaining an index matrix P2X and an inverse index matrix X2P of the original point cloud set P to the non-graph expression.
Step 30: branching a non-graph characteristic extraction network of a non-graph representation input point cloud characteristic extraction network of an original point cloud set to obtain a non-graph branch characteristic vector, and taking a graph of the original point cloud set as a graph of a G input point cloud characteristic extraction networkBranching the neural network to obtain a graph branch feature vector, wherein the non-graph feature extraction network brancheskLayer output eigenvector and graph neural network branch kThe output characteristic vectors of the layer-1 are fused and then used as the branch of the graphic neural networkkAn input feature vector of a layer, whereink∈[1,L-1]。
Step 40: and converting the non-graph branch feature vector according to the inverse index matrix X2P to obtain a non-graph representation feature vector of the original point cloud set, and converting the graph branch feature vector according to the inverse index matrix G2P to obtain a graph representation feature vector of the original point cloud set.
The step 10-40 is to obtain the non-graph representation feature vector and the graph representation feature vector of the original point cloud set by the point cloud feature extraction method according to any of the above embodiments, and the description of the point cloud feature extraction method can be referred to above, and will not be repeated here.
Step 50: inputting the non-graph representation feature vector and/or the graph representation feature vector of the original point cloud set into a target detection head network or a semantic segmentation head network to obtain a visual detection result, wherein the visual detection result comprises a prediction result of the category of each vertex in the graph representation of the original point cloud set.
The point cloud feature extraction network and the target detection head network or the semantic segmentation head network form a point cloud visual detection network, and the structure of the point cloud visual detection network is shown in fig. 7.
The object detection header network and the semantic segmentation header network may employ any existing object detection header network and semantic segmentation header network. According to different actual tasks, a user can select and input one or both of the non-graph expression feature vector and the graph expression feature vector according to requirements. The general non-graph representation feature vector can be used as a supplementary information of the graph representation feature vector, for example, when the loss function is calculated in the training process, the output value of the non-graph representation feature vector can be used as an additional item to accelerate the convergence of the network. The class of vertices is used to identify whether the vertex belongs to a target or to what object, etc.
The point cloud feature extraction method and the point cloud visual inspection method based on graph expression according to the present invention will be described below with reference to fig. 8 by way of a specific example in which a non-graph feature extraction network branches into UNet networks.
Firstly, obtaining a graph expression G of an original point cloud set P, an index matrix P2G and an inverse index matrix G2P; and converting the original point cloud set P into a non-graph representation form such as voxel representation or any projection representation and the like, and simultaneously obtaining an index matrix P2X and an inverse index matrix X2P of the original point cloud set P in the non-graph representation form under the initial resolution.
Representing non-graph as input depthlIs extracted by UNet network of (1) to obtain a set { { of feature vectors (hereinafter referred to as "features") of each layer of non-graph expression of the original point cloud set PX encoder },{X decoder }, whereinlIs a positive integer, depthlRepresenting the depth of the encoder and decoder in UNet, i.e. the number of layers of the encoder and decoder, the set of encoded layer feature vectorsSet of decoding layer feature vectors +.>，X i AndX j respectively represent encoder firstiLayer and decoding layerjThe feature vector of the layer output is used,C i andC j respectively represent encoder firstiLayer and decoding layerjNumber of characteristic channels of the layer.
Up-sampling each layer of features to the same resolution as the original point cloud set P by linear interpolation, deconvolution and other methods, thereby obtaining an index matrix X2P with layer-by-layer features pointing to the original point cloud set P k Whereink∈[1,L-1]. By combining the index matrix P2G, an index matrix X2G of the characteristic vector representation G of each layer can be obtained k =X2P k [P2G]. Non-graphically expressed feature set X of each layer k ∈{{X encoder },{X decoder ' via an index matrix X2G k Can obtain corresponding index conversion characteristicsX_in k ∈{{X_in encoder },{X_in decoder }, whereinX_in k =X k [X2G k ]。
Index conversion feature due to any layer derived from UNet networkX_in k Still representing the point cloud characteristics under the non-graph expression, the point cloud characteristics are better combined with graph branch characteristics and non-graph expression branch characteristics before being input into the graph neural network branch, and the point cloud characteristics are required to be branched with the graph neural networkkCharacteristics of layer-1G k-1 Performing multi-expression feature fusion to obtain a graph neural network branchkThe input characteristics of the layers are realized by a multi-expression channel gating attention module.
Index conversion characteristics obtained by each layer of the UNet network are input into a multi-expression channel gating attention module, and the graph neural network branches to the thk-layer 1 output feature G k-1 Entry multiple expression channel gated attention moduleX_in k Input graph neural network branch after fusionkThe layers are updated to obtain the output characteristic G k Then continue to enter the multi-expression channel gating attention module and the multi-expression channel gating attention moduleX_in k+1 Fusion is performed, and so on. The fusion mode can adopt the formulas (1) - (4), and the graph neural network branches update the characteristic values of the vertexes in the graph expression according to the formula (5).
Finish 2 in the above-described mannerlAfter the alternative learning and updating of the network branch characteristics of the layer-1 graph neural network branch and the non-graph characteristic extraction network branch, the 2 nd can be processed according to different taskslThe graph branch characteristics and the non-graph branch characteristics of the layer-1 are respectively restored according to an inverse index matrix G2P, X P to obtain the extraction characteristics of the original point cloud set P, and the extraction characteristics are input into a semantic segmentation head network or a target detection head network to realize semantic segmentation or target detection of the point cloud.
The above is a specific introduction of the point cloud feature extraction method and the point cloud visual detection method based on graph expression.
The invention also provides a method for calculating the semantic variability of the edge graph nodes based on the point cloud graph expression, and provides a loss function based on the method for calculating the semantic variability of the edge graph nodes to improve the training accuracy of the point cloud visual detection network. The semantic variability calculation method has strong identification capability for the edge graph nodes, and the semantic variability calculation method can enhance the identification capability of the point cloud feature extraction network on the object edges by introducing the semantic variability calculation method into the loss function. The edge graph nodes refer to graph nodes which are determined through label judgment and do not belong to the same object or the same semantic meaning with neighbor nodes in graph expression of the point cloud.
In the training stage, for the graph representation G of the original point cloud set P, true value labels are marked on each vertex of the graph representation G to represent the true category of the graph representation G, and any vertex in the graph representation G is markedp i And directed edge set connected with the sameECan determine the vertexp i Semantic variability with neighbor vertices is
，
Wherein the method comprises the steps ofρ i The first of the graph representations for the original point cloudiMultiple verticesp i Is used for the semantic variability of the (c) in the image,PN i is the firstiMultiple verticesp i Is a function of computing the total number of neighbor vertices,L i 、L j respectively the firstiMultiple verticesp i And the first thereofjThe true value labels of the adjacent vertexes, namely exclusive OR operation, returns a 1 value when two label values are different, otherwise returns a 0 value.
According to the definition of the semantic variability, the invention provides an edge graph node loss functionLoss edge In some embodiments, the point cloud visual inspection network training loss function includes the edge map node loss functionLoss edge Edge graph node loss functionLoss edge The expression of (2) is:
Loss edge =ρLoss global ，
wherein the method comprises the steps ofLoss global As a global loss function of the original point cloud,ρsemantic variability of vertices expressed for a graph of the original point cloud. Global loss functionLoss global Any existing loss function such as cross entropy loss function, logarithmic loss function, KL divergence, etc. may be used. When global loss function Loss global When the edge graph node loss function is the cross entropy loss functionLoss edge The expression of (2) is:
，
wherein the method comprises the steps ofNThe total number of vertices expressed for the graph of the original point cloud,lthe category is indicated as such,Las a total number of categories,andy ic the first of the graph representations of the original point clouds, respectivelyiThe predictions of the class of vertices and the corresponding one-hot vectors of the truth labels,ρ i the first of the graph representations for the original point cloudiSemantic variability of individual vertices.
Edge graph node loss functionLoss edge The method is a loss function oriented to the edge graph nodes, the convergence of the point cloud vision detection network calculates the gradient according to the loss function value during training, and the edge graph nodes loss functionLoss edge Is an additional term of the existing network training loss function, and can be added with the existing network training loss function to form the loss function of the point cloud vision detection network.
According to the point cloud feature extraction method and the point cloud visual detection method based on graph expression in the embodiment, graph expression is introduced into feature extraction networks of other expression forms of the existing point cloud, a graph expression feature extraction branch facing a feature extraction link is formed, the graph expression feature extraction branch can be combined with a feature extraction network model of the existing expression forms such as voxel point cloud or projection point cloud, point cloud feature extraction is carried out in other point cloud expression forms, the extracted features are fused with the features of the point cloud graph expression, point cloud feature aggregation and propagation are carried out, and the multi-mode fused point cloud feature extraction method and the multi-mode fused point cloud visual detection method are formed.
The point cloud feature extraction method and the point cloud visual detection method based on graph expression have extremely strong compatibility with the existing method, and meanwhile, due to the graph neural network based on graph expression in the graph neural network branch, the feature extraction capability of the point cloud fine granularity information can be enhanced, and compared with 3D convolution, the calculation amount is smaller.
According to the method, semantic variability is obtained based on graph expression, an edge graph node loss function is provided based on the semantic variability, and an additional gradient can be provided for a point cloud feature extraction network according to object edge information to enhance the distinguishing capability of the object edge and enhance the semantic consistency in the object by combining the existing loss function; for non-graph expression of the point cloud, for example, the point cloud voxel expression based on a certain voxel size can effectively reduce the influence of edge detail information blurring caused by the point cloud in the expression conversion process on the segmentation or detection precision.
Step 200 above involves obtaining a graph representation G of an original point cloud, and in one embodiment of the present invention, a method for converting a point cloud graph representation is provided, where the original point cloud is first ordered, and the graph representation G of the point cloud is obtained based on the ordering. The embodiment of the invention also improves the ordering of the point clouds, realizes the ordered reconstruction by converting the original point cloud set into ordered arrangement under two-dimensional angular distribution, and realizes the graph expression of the point clouds based on the calculation of directed edges among the ordered point clouds on the basis. The concept of the conversion method expressed by the point cloud image of the invention is described below.
The point cloud data generated by the laser radar describe the surface information of an object in a scanning area, the laser radar projects laser light to the surface of the object, and the object surface reflects the laser light to a laser radar sensor for imaging to generate the point cloud data. The inventors realized that since current lidar equipment must rely on reciprocating (mechanical or semi-solid lidar) or area arrays (solid state lidar) to emit laser beams, it can be considered that the reflection points of an object to the laser beams describe to some extent not only the surface structure distribution of the object, but also that adjacent reflection points have a certain proximity relationship in the lidar sensor. Therefore, an embodiment of the invention provides an ordered reconstruction method suitable for original point cloud data of any type of laser radar sensor (mechanical, semi-solid, solid) or other expression forms (such as voxelization, rasterization and the like) of point cloud, and aims to reconstruct sensor coordinates of three-dimensional point cloud only according to a laser radar imaging principle and space coordinate data of the point cloud on the premise of deleting original measurement sequence information and distribution information of the laser radar sensor, and accurately obtain absolute positions of the point cloud in a sensor coordinate system, thereby realizing ordering of the original point cloud. After the absolute position coordinates of each space point are defined, the neighbor point cloud of each point cloud under the sensor coordinate system can be clearly found according to the requirements according to the coordinate information, and meanwhile, the directed edges between the point clouds can be defined so as to enable the discrete point clouds to be converted into graph data.
According to the laser radar measurement principle, the space three-dimensional coordinates of any normal imaging pointx,y,z) Distance calculated from time of flight of reflected laser beamdAnd the horizontal rotation angle (i.e. horizontal azimuth angle) of the corresponding laser beamθAnd vertical angle (i.e. vertical pitch angle)γThe determination is as shown in the following formula (6):
。 （6）
under ideal measurement conditions, each three-dimensional space measurement point should be unique to a laser beam at the same time, thus for point cloudsPReferring to FIG. 9, any point can be calculated reversely according to the three-dimensional coordinatesp i Horizontal azimuth angle of corresponding laser beamθ i From vertical pitch angleγ i As shown in equation (7). Point cloud setPThe ordering problem of (2) can be converted into a set { of horizontal azimuth angle and vertical pitch angle of the laser beam corresponding to the point cloud }θ,γOrdering problem of }.
。 （7）
The embodiment of the invention { sets of the horizontal azimuth angle and the vertical pitch angle of the laser beamθ,γThe problem of ordering is that a specific positioning of the point cloud in the laser radar field of view is calculated according to an ordering arrangement method of the laser radar sensor attribute, namely according to the horizontal azimuth angle and vertical pitch angle distribution of the point cloud corresponding to the laser beam. The invention refers to the visual imaging principle, and defines the laser radar field of view as the target size W×HIs called a lidar pixel matrix, characterizes a pixel plane of the lidar imaging, and forms a sensor pixel coordinate system based thereonOuv(i.e., lidar field of view coordinate system), as shown in FIG. 10, in whichWAndHis a positive integer. Under the laser radar visual field coordinate system, the point cloud is positioned to the pixel coordinate through the horizontal azimuth angle and vertical pitch angle informationu,v) WhereinuIs in the abscissa,vIs the ordinate. The process reconstructs the coordinates of the three-dimensional point cloud in the two-dimensional pixel matrix, realizes ordered arrangement of the three-dimensional point cloud under two-dimensional angle distribution, and is defined as shown in a formula (8).
。 （8）
Wherein the method comprises the steps ofF rebuild I.e. the process of ordered reconstruction.
In addition, the inventors have noted that for multi-beam lidar, complex imaging procedures often fail to ensure that the laser receiving component receives the specified laser beam at the desired setting, which often results in a horizontal azimuth angle calculated in reverse according to equation (7)θFrom vertical pitch angleγAnd the range of the included angle of the laser radar field of view under the theoretical condition is exceeded. Thus target size of the lidar pixel matrixWAnd (3) withHIt is preferred that the composition is,Wgreater than the theoretical value of the width of the pixel plane of the lidar, HGreater than the excitationTheoretical value of the height of the pixel plane of the lidar.
Coordinates of each point cloud in original point cloud P on laser radar pixel matrixu i ,v i ) Sensor pixel coordinate set constituting original point cloud set PT{(u,v) And the transformation of the original point cloud set to the graph representation can be realized on the basis. First, a sensor pixel coordinate set based on an original point cloudT{(u,v) "as any non-empty pixelsu i ,v i ) Corresponding point cloudp i Searching for neighbor point setPN i And sets up directed edges in the manner shown in fig. 11. In dotsp i Corresponding sensor pixel coordinates [ ]u i ,v i ) Centering on, a given target size isbox_w×box_hReturning a point cloud set corresponding to non-empty pixels in the neighbor region bounding box as pointsp i Neighbor point set of (a)PN i ={p k ,k=1,2……,N p }, whereinp k Representing the first within a neighbor region bounding boxkA point cloud corresponding to a non-empty pixel,N p representing the total number of non-empty pixels within the neighbor region bounding box. At the same time, a set of neighbor points can be obtainedPN i Pointing to the target pointp i Directed edge set of (a). According to the definition of the graph, target points are formedp i Set of neighbor pointsPN i As vertices, with directed edge setsE i Is of a directed edge structure, can be obtainedp i Local directed graph structure for key points +.>. Sensor pixel coordinate set by traversing original point clouds T{(u,v) Directed graph from original point cloud setA transition to G.
Referring to fig. 12, in the above description, in the above step 200, the conversion of the original point cloud P to obtain the graph representation G of the original point cloud includes steps 210 to 270, which are described below.
Step 210: if the laser radar is a solid-state laser radar, calculating the horizontal azimuth angle and the vertical pitch angle of the laser beams corresponding to each point cloud according to the three-dimensional coordinates of each point cloud in the original point cloud set, and calculating the coordinates of each point cloud on a preset laser radar pixel matrix according to the horizontal azimuth angle and the vertical pitch angle of the laser beams corresponding to each point cloudu i ,v i ) Wherein%u i ,v i ) Represent the firstiPersonal point cloudp i Coordinates on the lidar pixel matrix.
Solid-state lidar is referred to herein as solid-state lidar. According to the three-dimensional coordinates of each point cloud in the original point cloud set, calculating the horizontal azimuth angle and the vertical pitch angle of the laser beam corresponding to each point cloud, wherein the horizontal azimuth angle and the vertical pitch angle can be seen in a formula (7). For the pure solid-state laser radar, the imaging principle is similar to classical visual imaging, so that any point cloud can be calculated by referring to the classical visual imaging principlep i Pixel coordinates of [ ]u i ,v i )。
The laser radar pixel matrix is preset before ordered reconstruction, wherein WAnd (3) withHCan be set according to the actual needs of the user,Wand (3) withHThe choice of (c) determines the accuracy of the ordered reconstruction method of the present invention.
Step 220: if the laser radar is a mechanical laser radar or a semi-solid laser radar, calculating the horizontal azimuth angle and the vertical pitch angle of the laser beams corresponding to each point cloud according to the three-dimensional coordinates of each point cloud in the original point cloud set, calculating the actual line number and the actual horizontal accumulated step number of the laser beams corresponding to each point cloud according to the horizontal azimuth angle and the vertical pitch angle of the laser beams corresponding to each point cloud, and calculating the coordinates of each point cloud on the laser radar pixel matrix according to the actual line number and the actual horizontal accumulated step number of the laser beams corresponding to each point cloudu i ,v i )。
Similarly, the calculation of the horizontal azimuth angle and the vertical pitch angle of the laser beam corresponding to each point cloud according to the three-dimensional coordinates of each point cloud in the original point cloud set can be seen in the formula (7). The actual line number of the laser beams corresponding to the point cloud refers to the number of the laser beams, and the pure mechanical or semi-solid laser radar needs to realize area array or looking around laser scanning through a microelectronic galvanometer or mechanical rotation, namely the actual horizontal accumulated step number of the laser beams refers to the total step number of horizontal rotation or vibration of the laser radar when the laser beams are emitted.
Step 230: if the laser radar is a biprism semi-solid laser radar, the laser radar is used forCentrifugal dip function and->Calculating the horizontal azimuth angle and the vertical pitch angle of the laser beams corresponding to each point cloud according to the laser deflection angle function, calculating the actual line number and the actual horizontal accumulated step number of the laser beams corresponding to each point cloud according to the horizontal azimuth angle and the vertical pitch angle of the laser beams corresponding to each point cloud, and calculating the coordinates of each point cloud on a laser radar pixel matrix according to the actual line number and the actual horizontal accumulated step number of the laser beams corresponding to each point cloudu i ,v i )。
The biprism semi-solid laser radar refers to a semi-solid laser radar based on a rotating biprism, and the horizontal azimuth angle and the vertical pitch angle of a laser beam refracted by the rotating biprism depend on the rotation angular velocity of the biprism due to the unique laser deflection modeω 1 Andω 2 rotation time of biprismtAnd refraction angle of biprismβ 1 Andβ 2 . The invention therefore proposes a semi-solid laser radar according to a biprismCentrifugal dip function and->Calculating horizontal azimuth angle and vertical pitch angle of laser beams corresponding to the point cloud by using a laser deflection angle function, and then calculating the coordinates of each point cloud on a laser radar pixel matrix by referring to a semi-solid laser radar mode u i ,v i )。
For different types of lidar, the azimuth angle is horizontalθVertical pitch angleγTo pixel coordinates [ ]u,v) Some differences exist in the calculation method of (a), and the ordered calculation method is described in detail below with respect to the three laser radar measurement principles described in the above steps.
For the pure solid-state lidar in step 210, in one embodiment the coordinates of the point cloud on the lidar pixel matrix are calculated according to the following formulau i ,v i )：
， （9）
Wherein the method comprises the steps ofθ i Represent the firstiThe horizontal azimuth of the laser beam corresponding to the point cloud,γ i represent the firstiThe vertical pitch angle of the laser beam corresponding to each point cloud,Ang Horizontal is the horizontal azimuth angle of the laser radar,Ang Vertical is the vertical pitch angle of the laser radar.
Ang Horizontal AndAng Vertical expressed are horizontal and vertical viewing angle ranges, respectively.Ang Horizontal AndAng Vertical the values of (a) can be selected for the sensor performance or calculated according to the actual value distribution of the point cloud. The theoretical values generally have plain text comments in a laser radar equipment manual, such as 'horizontal 360 degrees', or 'horizontal 90 degrees', or 'vertical viewing angles +2 degrees to-24 degrees', and the like, and the values of different equipment are generally different. Whereas calculation from the actual value distribution refers to: for a given point cloudPAvailable point cloud setPThe horizontal azimuth angle and the vertical pitch angle corresponding to all the points below, according to the visual angle The absolute value of the maximum minus the minimum in the horizontal or vertical pitch angle can be defined as the point cloudPExpressed visual angle range, i.eAng Horizontal Or (b)Ang Vertical 。
For the purely mechanical lidar or the semi-solid lidar in step 220, the coordinates of each point cloud on the lidar pixel matrix are calculated according to the visual imaging principle, and in one embodiment, the coordinates of each point cloud on the lidar pixel matrix are specifically calculated according to the following formula:
， （10）
wherein the method comprises the steps ofo i Represent the firstiThe actual level of the laser beam corresponding to the point cloud is accumulated,q i represent the firstiThe actual number of lines of the laser beam corresponding to the point cloud,Ofor the total number of horizontal rotations or oscillations of the lidar,Qis the number of laser beam buses of the laser radar.And->Scaling factors of horizontal and vertical view fields respectively, and are used for adjusting point clouds in practical applicationp i Pixel coordinates [ ]u i ,v i ) Is a superposition loss of (2).
In one embodiment, the actual line number and the actual horizontal accumulated step number of the laser beam corresponding to the point cloud are determined by the following formula:
， （11）
wherein,indicating whenθ=θ i Time function->Value of->Indicating whenγ=γ i Time function->Wherein>Distribution function representing horizontal azimuth angle of laser beam of laser radar I n (o) Is an inverse function of +.>Distribution function representing vertical pitch angle of laser beam of laser radarL m (q) Is an inverse function of (c).
It can be appreciated by those skilled in the art that the distribution function of the horizontal azimuth angle of the laser beam can be summarized or designed according to the technical manual, mechanical structure, optical characteristics, etc. of the laser radar in combination with the actual requirementsI n (o) And a vertical pitch angle distribution functionL m (q)。
For pure mechanical lidar or semi-solid lidar, imaging needs to realize area array or looking around laser scanning through microelectronic galvanometer or mechanical rotation, and the sensor of the type can also reach specific laser line number or measuring point density through stacking multiple groups of laser transmitting/receiving or rotating/oscillating mirrors, the inventor realizes that the laser beam of the sensor of the type generally has a sectional uniform distribution lattice in a horizontal or vertical view angle. An embodiment of the present invention therefore proposes a general method of ordered reconstruction based on imaging principles for such lidar sensors and their principles.
Is in a vertical view angleMIs uniformly distributed in sections and is in a horizontal visual angleNUniformly distributed segments, the number of laser lines beingQ、The total number of horizontal rotation or vibration steps isOThe horizontal azimuth and vertical elevation distribution characteristics of the pure mechanical or semi-solid lidar of (c) are shown in fig. 13. In one embodiment, the laser bus number is due to the laser module distribution characteristics QLaser radar sensor of (1)qThe vertical pitch angle of the line laser beam follows the formula (12)MSegment piecewise linear functionL m (q) I.e. the distribution function of the vertical pitch angle of the laser beamL m (q) The expression of (2) is shown in the formula (12).
。 （12）
Wherein the vertical pitch angle of each section of laser beam is fitted by adopting a linear fitting curve,γ m representing the first in the vertical viewing anglemThe vertical pitch angle of the segment laser beam,qindicating the number of lines of the laser beam,q m-1 representing the first in the vertical viewing anglemThe number of start lines of the segment laser beam,q m representing the first in the vertical viewing anglemThe number of termination lines of the segment laser beam,q M =Q。K m andB m respectively represent the first in the vertical viewing anglemSlope and intercept of a linear fit curve of the vertical pitch angle of the segment laser beam.
Those skilled in the art will appreciate the number of laser linesqThe values of the segments may be determined in conjunction with the device design manual and the lidar structure. In general terms, the process is carried out,K m andB m the value of (2) is determined by the ideal or actual vertical pitch angle distribution, and the specific value can be estimated by linear fitting method according to the ideal or actual measured vertical pitch angle distribution. The presence of a well-defined scaling factor in the vertical resolution distribution of some lidars allows for the determination ofTherein {k m ,m∈MIs a known parameter combined with ideal or actual vertical Angle of pitch angleAng Vertical Can be calculated according to the formula (13)K m Whereinunit_kIs a slope scaling factor.
。（13）
The horizontal azimuth distribution of a mechanical or semi-solid lidar is typically determined by a mechanical rotating or microelectronic galvanometer system. In one embodiment, the horizontal azimuth distribution function of the laser beam of the laser radarI n (o) The expression of (2) is shown in the formula (14).
。 （14）
Wherein the horizontal azimuth angle of each section of laser beam is fitted by adopting a quadratic fit curve,θ n representing the first view in the horizontal viewnThe horizontal azimuth angle of the segment laser beam,oindicating the number of horizontal accumulated steps of the laser beam,o n-1 representing the first view in the horizontal viewnThe starting level of the segment laser beam is accumulated for steps,o n representing the first view in the horizontal viewnThe termination level of the segment laser beam accumulates the number of steps,o N =O。a n 、b n andc n respectively represent the first in the horizontal viewing anglenThe quadratic term coefficient, the first order coefficient and the constant term of the quadratic fit curve of the horizontal azimuth angle of the segment laser beam.
As will be appreciated by those skilled in the art, the number of steps is accumulated horizontallyoThe value of the segments can be determined by the rotation or vibration motion characteristics of the laser radar.a n 、b n Andc n the values of (2) are determined by the ideal or actual horizontal azimuth distribution, and specific numerical values can be subjected to quadratic function fitting estimation according to the ideal or actual measured horizontal azimuth distribution. Generally, for a purely mechanical lidar rotated at a constant speed 360 °, the distribution function of the horizontal azimuth angle Number of digitsParameters at this timea n 、c n Respectively, is degenerated to be a constant of 0,b n degradation to constant resolution->。
The inverse function can be further obtained from the equation (12) and the equation (14)And->And combining the formula (11), reversely pushing the horizontal azimuth angle and the vertical pitch angle of the point cloud to obtain the actual line number and the actual horizontal accumulated step number of the laser beam for generating the point cloud, and finally obtaining the coordinates of the point cloud on the laser radar pixel matrix according to the formula (10). The two-dimensional positioning of the point cloud in the laser radar receptive field is obtained by the reverse thrust of the horizontal azimuth angle and the vertical pitch angle of the point cloud according to the basic technical properties of the mechanical or semi-solid laser radar.
For the biprism semi-solid lidar in step 230, one embodiment of the present invention gives the horizontal azimuth and vertical pitch determination formulas as shown in equation (15) according to its basic imaging principle (as shown in fig. 14):
， （15）/>
wherein the method comprises the steps ofRepresenting +.>Centrifugal dip function>Representing +.>A function of the deflection angle of the laser,β 1 andβ 2 is the refraction angle of the biprism of the lidar,ω 1 andω 2 is the rotational angular velocity of the biprism of the lidar,tis the rotation time of the biprism. Combining the formula (11) and the formula (10) to finally obtain the coordinates of the point cloud on the laser radar pixel matrix u i ,v i ) The general expression is shown in formula (16):
。 （16）
thus, an original point cloud set is obtainedPCoordinates of each point cloud on the laser radar pixel matrixu i ,v i ) Thereby forming the original point cloud setPSensor pixel coordinate set of (2)T{(u,v) }, finish the original point cloud setPIs a sequential transition of (a). According to the ordered reconstruction process, the original point cloud set can be obtained simultaneouslyPConversion to a sensor pixel coordinate setTBy which dynamic query of any point in three-dimensional distribution or ordered two-dimensional distribution can be realized, and the time complexity of the query or indexing process of any point or its adjacent point on the pixel matrix is O (1).
Steps 240 to 270 are further described below.
Step 240: sensor pixel coordinate set based on original point cloudT{(u,v) ' for any non-empty pixelsu i ,v i ) Acquiring the corresponding point cloudp i And obtain the non-empty pixelu i ,v i ) The point cloud set corresponding to the non-empty pixels in the surrounding frame of the neighbor area with the preset target size and serving as the point cloudp i Neighbor point set of (a)PN i ={p k ,k=1,2……,N p }。
Step 250: from a set of neighbor pointsPN i Middle point cloud pointing to point cloudp i Directed edges of (1) constitute a directed edge set。
Step 260: in the form of point cloudp i Set of neighbor points PN i Each point cloud of the three points is taken as a vertex, and a directed edge set is taken as a vertexE i Is edge formed with point cloudp i Local directed graph as key point。
Step 270: sensor pixel coordinate set traversing original point cloud setT{(u,v) And obtaining a local directed graph taking the point cloud corresponding to each non-empty pixel as a key point, so that the conversion from the original point cloud set to the directed graph expression G is realized.
According to the method for converting point cloud image expression in the embodiment, the problem of ordering of the laser radar point cloud is converted into a set { of horizontal azimuth angle and vertical pitch angle of laser beams corresponding to the laser radar point cloudθ,γThe ordering problem of the point cloud is used for calculating pixel coordinates of the point cloud on a laser radar pixel matrix according to the horizontal azimuth angle and the vertical pitch angle of laser beams corresponding to the point cloud based on the imaging principle of the point cloud aiming at various laser radarsu,v) The ordered arrangement of the three-dimensional point cloud under the two-dimensional angle distribution is realized, and then the neighbor point search is carried out based on the ordered point cloud to construct the graph representation of the point cloud, so that the method has the following beneficial effects compared with the prior art:
(1) The method for ordering point clouds and expressing the point cloud images in the non-object model and the non-indoor intensive point cloud scene is provided, and the technical blank is filled;
(2) The ordered reconstruction method of the embodiment of the invention can be executed in the preprocessing stage of the point cloud, and is always applicable to one-time processing of each frame of the point cloud;
(3) The invention provides a point cloud ordered reconstruction method which depends on a laser radar sensor measurement principle, and further provides a directed edge and graph expression construction method based on ordered point clouds;
(4) After the ordered reconstruction method is adopted to order the original point cloud set, dynamic inquiry of any point in three-dimensional distribution or ordered two-dimensional distribution can be realized by means of the index matrix from the original point cloud set to the pixel coordinate set, and the time complexity of the inquiry or index process of any point or the adjacent point on the pixel matrix is only O (1); for frequent expression mode switching or point cloud neighbor indexing in part of the target detection model, the invention can always provide a rapid index with the time complexity of O (1) level;
(5) The invention rebuilds from the laser radar measurement principle, avoids the geometric deformation brought by different point cloud density distribution to the spherical projection method, avoids the calculation cost of the tree structure compared with the tree building method such as KDTE and the like, and reduces the index or query cost.
Those skilled in the art will appreciate that all or part of the functions of the various methods in the above embodiments may be implemented by hardware, or may be implemented by a computer program. When all or part of the functions in the above embodiments are implemented by means of a computer program, the program may be stored in a computer readable storage medium, and the storage medium may include: read-only memory, random access memory, magnetic disk, optical disk, hard disk, etc., and the program is executed by a computer to realize the above-mentioned functions. For example, the program is stored in the memory of the device, and when the program in the memory is executed by the processor, all or part of the functions described above can be realized. In addition, when all or part of the functions in the above embodiments are implemented by means of a computer program, the program may be stored in a storage medium such as a server, another computer, a magnetic disk, an optical disk, a flash disk, or a removable hard disk, and the program in the above embodiments may be implemented by downloading or copying the program into a memory of a local device or updating a version of a system of the local device, and when the program in the memory is executed by a processor.
The foregoing description of the invention has been presented for purposes of illustration and description, and is not intended to be limiting. Several simple deductions, modifications or substitutions may also be made by a person skilled in the art to which the invention pertains, based on the idea of the invention.

Claims (10)

1. The point cloud feature extraction method based on graph expression is characterized by comprising the following steps of:
acquiring an original point cloud set P acquired by a laser radar;
converting the original point cloud set P to obtain a graph representation G of the original point cloud set, and obtaining an index matrix P2G and an inverse index matrix G2P from the original point cloud set P to the graph representation G; converting the original point cloud set P into a target expression form X of a non-graph expression form to obtain a non-graph expression of the original point cloud set P, and obtaining an index matrix P2X and an inverse index matrix X2P from the original point cloud set P to the non-graph expression;
inputting the non-graph representation G of the original point cloud set into a non-graph feature extraction network branch of a preset point cloud feature extraction network to obtain a non-graph branch feature vector, and inputting the graph representation G of the original point cloud set into a graph neural network branch of the point cloud feature extraction network to obtain a graph branch feature vector, wherein the non-graph feature extraction network branch is the first branch of the non-graph feature extraction network kLayer output eigenvector and the graph neural network branchkThe output characteristic vector of the-1 layer is fused and then used as the branch of the graphic neural networkkAn input feature vector of a layer, whereink∈[1, L-1]，LRepresenting the total layer number of the non-graph feature extraction network branch and the graph neural network branch, wherein the output feature vector of the 0 th layer of the graph neural network branch refers to the graph expression of the original point cloud set;
and converting the non-image branch feature vector according to the inverse index matrix X2P to obtain a non-image expression feature vector of the original point cloud set, and converting the image branch feature vector according to the inverse index matrix G2P to obtain an image expression feature vector of the original point cloud set.
2. The point cloud feature extraction method of claim 1, wherein said point cloud feature extraction network further comprises a multi-expression channel-gated attention module, said non-graph feature extraction network branching off a first pointkLayer output eigenvector and the graph neural network branchkThe output feature vectors of layer-1 are fused as follows:
acquiring an index matrix P2G expressed from the original point cloud set P to the graph;
branching the non-graph feature extraction networkkLayer output feature vector X k Upsampling to the same resolution as the original point cloud P and obtaining a non-image feature extraction network branchkLayer output feature vector X k Index matrix X2P to original point cloud P k ；
According to an index matrix X2P k And indexing matrix P2G to obtain the non-graph feature extraction network branchkLayer outputting eigenvectors to index matrix X2G of the graph representation k Wherein X2G k =X2P k [P2G]；
By indexing matrix X2G k Extracting network branch from the non-graph featureskLayer output feature vector X k Converting to obtain index conversion feature vectorX_in k ；
Branching the graph neural network to the firstk-layer 1 output feature vector and said index conversion feature vectorX_in k Inputting the multi-expression channel gating attention module to perform feature fusion with a channel gating attention mechanism to obtain the branch th of the graph neural networkkThe input feature vector of the layer.
3. The point cloud feature extraction method of claim 2, wherein said multiple expression channel gated attention module branches said graph neural network byk-layer 1 output feature vector and said index conversion feature vectorX_in k Performing a gate-gated implant with a channelFeature fusion of the Italian mechanism to obtain the graph neural network branch kInput feature vector of layer:
branching the graph neural networkkOutput feature vector of-1 layerG k-1 And the index conversion feature vectorX_in k Adding or splicing and fusing to obtain approximate fusion feature vector；
Based on the approximate fusion feature vectorCalculating the branch of the graph neural networkkRelative channel attention score for output feature vector of layer-1score G And the index conversion feature vectorX_in k Relative channel attention score of (2)score X ；
Scoring relative channel attentionscore G Andscore X gating activation is carried out to respectively obtain the graph neural network branch No.k-gating coefficients of the output feature vector of layer 1 and said index conversion feature vectorX_in k Gate control coefficient of the graph neural network branch according to the gate control coefficientk-layer 1 output feature vector and said index conversion feature vectorX_in k Weighted and added to obtain the branch of the graph neural networkkThe input feature vector of the layer.
4. The point cloud feature extraction method of claim 3, wherein said graph neural network branches to a thirdkRelative channel attention score for output feature vector of layer-1score G And the index conversion feature vectorX_in k Relative channel attention score of (2)score X Is determined by the following calculation formula:
，
wherein the method comprises the steps ofG k-1 Representing the branch of the graph neural network kThe output feature vector of layer-1,the L2 norm is represented by the number,for trainable coefficient parameters, +.>Represents a real number and is used to represent a real number,C k representing the branch of the graph neural networkkOutput feature vector of layer and index conversion feature vectorX_in k Is the number of characteristic channels, MLP G Representing the branching of the graph neural networkkSingle layer perceptron for relative channel attention scoring of output feature vectors of layer-1, MLP G Representing the index transformed feature vectorX_in k Single layer perceptron for relative channel attention scoring, with MLP representing linear transformations implemented by the single layer perceptron to align output feature vectorsG k-1 And index conversion feature vectorX_in k Is a characteristic channel number of (a).
5. The point cloud feature extraction method of claim 3, wherein said graph neural network branches to a thirdkThe calculation formula of the input feature vector of the layer is:
，
wherein the method comprises the steps ofG_in k Representing the branch of the graph neural networkkThe input feature vector of the layer is used,σ 1 the function is activated for gating.
6. The point cloud feature extraction method according to any one of claims 1 to 5, wherein each layer of the graph neural network branches updates feature values of any vertex in the graph representation of the original point cloud set of the current layer, and the updated vertex updates feature values according to neighbor nodes under the spatial geometrical distribution according to the following formula:
，
Wherein the method comprises the steps of、G i Respectively the first of the map representations of the original point clouds under the layeriOutput characteristic values and input characteristic values of the vertexes,σ 2 in order to activate the function,Wfor a weight matrix of a trainable input linear transformation,PN i is the firstiA set of all neighbor vertices of the vertices,α ji is the firstiThe first of the verticesjThe neighbor vertices point to the firstiThe weighted value of the directed edges of the vertices.
7. The point cloud feature extraction method of claim 6, wherein the weighting values areα ji The calculation formula of (2) is as follows:
，
where I represents the concatenation operation, vectoraIs a trainable weight vector.
8. The point cloud vision detection method based on graph expression is characterized by comprising the following steps of:
obtaining a non-graph representation feature vector and a graph representation feature vector of an original point cloud set by the point cloud feature extraction method according to any one of claims 1 to 7;
inputting the non-graph representation feature vector and/or the graph representation feature vector of the original point cloud set into a target detection head network or a semantic segmentation head network to obtain a visual detection result, wherein the visual detection result comprises a prediction result of the category of each vertex in the graph representation of the original point cloud set.
9. The point cloud visual inspection method of claim 8, wherein said point cloud feature extraction network and said target inspection head network or said semantic segmentation head network form a point cloud visual inspection network, and wherein said point cloud visual inspection network training loss function comprises an edge map node loss function Loss edge The edge graph node loss functionLoss edge The expression of (2) is:
Loss edge =ρLoss global ，
wherein the method comprises the steps ofLoss global As a global loss function of the original point cloud,ρsemantic variability of vertices expressed for the graph of the original point cloud set; when the global loss functionLoss global When the edge graph node loss function is a cross entropy loss function, the edge graph node loss functionLoss edge The expression of (2) is:
，
wherein the method comprises the steps ofNThe total number of vertices expressed for the graph of the original point cloud,lthe category is indicated as such,Las a total number of categories,andy ic respectively the first of the graph expressions of the original point cloudsiThe predictions of the class of vertices and the corresponding one-hot vectors of the truth labels,ρ i the first of the graph representations for the original point cloud setiThe semantic variability of each vertex is calculated as follows:
，
wherein the method comprises the steps ofPN i Is the firstiA set of all neighbor vertices of the vertices, the card representing a function of the total number of neighbor vertices,L i 、L j respectively the firstiVertex number and the firstjTrue labels of the neighboring vertices, denoted exclusive or operation.
10. A computer-readable storage medium, characterized in that the medium has stored thereon a program executable by a processor to implement the point cloud feature extraction method according to any one of claims 1 to 7 or the point cloud visual detection method according to any one of claims 8 to 9.
CN202311628039.3A 2023-12-01 2023-12-01 Point cloud feature extraction method and point cloud visual detection method based on graph expression Active CN117333676B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202311628039.3A CN117333676B (en) 2023-12-01 2023-12-01 Point cloud feature extraction method and point cloud visual detection method based on graph expression

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202311628039.3A CN117333676B (en) 2023-12-01 2023-12-01 Point cloud feature extraction method and point cloud visual detection method based on graph expression

Publications (2)

Publication Number Publication Date
CN117333676A CN117333676A (en) 2024-01-02
CN117333676B true CN117333676B (en) 2024-04-02

Family

ID=89283438

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202311628039.3A Active CN117333676B (en) 2023-12-01 2023-12-01 Point cloud feature extraction method and point cloud visual detection method based on graph expression

Country Status (1)

Country Link
CN (1) CN117333676B (en)

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111553859A (en) * 2020-04-29 2020-08-18 清华大学 Laser radar point cloud reflection intensity completion method and system
CN113128348A (en) * 2021-03-25 2021-07-16 西安电子科技大学 Laser radar target detection method and system fusing semantic information
CN114529727A (en) * 2022-04-25 2022-05-24 武汉图科智能科技有限公司 Street scene semantic segmentation method based on LiDAR and image fusion
CN116229057A (en) * 2022-12-22 2023-06-06 之江实验室 Method and device for three-dimensional laser radar point cloud semantic segmentation based on deep learning

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111553859A (en) * 2020-04-29 2020-08-18 清华大学 Laser radar point cloud reflection intensity completion method and system
CN113128348A (en) * 2021-03-25 2021-07-16 西安电子科技大学 Laser radar target detection method and system fusing semantic information
CN114529727A (en) * 2022-04-25 2022-05-24 武汉图科智能科技有限公司 Street scene semantic segmentation method based on LiDAR and image fusion
CN116229057A (en) * 2022-12-22 2023-06-06 之江实验室 Method and device for three-dimensional laser radar point cloud semantic segmentation based on deep learning

Also Published As

Publication number Publication date
CN117333676A (en) 2024-01-02

Similar Documents

Publication Publication Date Title
Ost et al. Neural scene graphs for dynamic scenes
Zhu et al. Vpfnet: Improving 3d object detection with virtual point based lidar and stereo data fusion
CN109655019B (en) Cargo volume measurement method based on deep learning and three-dimensional reconstruction
CN110853075B (en) Visual tracking positioning method based on dense point cloud and synthetic view
CN111681212B (en) Three-dimensional target detection method based on laser radar point cloud data
Hoppe et al. Incremental Surface Extraction from Sparse Structure-from-Motion Point Clouds.
CN113450396A (en) Three-dimensional/two-dimensional image registration method and device based on bone features
US20240013479A1 (en) Methods and Systems for Training Quantized Neural Radiance Field
Ouyang et al. A cgans-based scene reconstruction model using lidar point cloud
CN114782628A (en) Indoor real-time three-dimensional reconstruction method based on depth camera
CN115205463A (en) New visual angle image generation method, device and equipment based on multi-spherical scene expression
CN113421217A (en) Method and device for detecting travelable area
Tao et al. LiDAR-NeRF: Novel lidar view synthesis via neural radiance fields
KR20230150867A (en) Multi-view neural person prediction using implicit discriminative renderer to capture facial expressions, body posture geometry, and clothing performance
Beltrán et al. A method for synthetic LiDAR generation to create annotated datasets for autonomous vehicles perception
Hu et al. Point2pix: Photo-realistic point cloud rendering via neural radiance fields
CN113536920A (en) Semi-supervised three-dimensional point cloud target detection method
CN112950786A (en) Vehicle three-dimensional reconstruction method based on neural network
Kniaz et al. Deep learning a single photo voxel model prediction from real and synthetic images
CN117333676B (en) Point cloud feature extraction method and point cloud visual detection method based on graph expression
CN116246119A (en) 3D target detection method, electronic device and storage medium
Ervan et al. Downsampling of a 3D LiDAR point cloud by a tensor voting based method
Pu et al. Sdf-gan: Semi-supervised depth fusion with multi-scale adversarial networks
CN113281779A (en) 3D object rapid detection method, device, equipment and medium
Kunert et al. Neural network adaption for depth sensor replication

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant