CN114881850A - Point cloud super-resolution method and device, electronic equipment and storage medium - Google Patents
Point cloud super-resolution method and device, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN114881850A CN114881850A CN202210082353.5A CN202210082353A CN114881850A CN 114881850 A CN114881850 A CN 114881850A CN 202210082353 A CN202210082353 A CN 202210082353A CN 114881850 A CN114881850 A CN 114881850A
- Authority
- CN
- China
- Prior art keywords
- point cloud
- super
- resolution
- point
- generator
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 57
- 238000012549 training Methods 0.000 claims abstract description 36
- 238000005070 sampling Methods 0.000 claims description 20
- 239000012634 fragment Substances 0.000 claims description 15
- 230000008447 perception Effects 0.000 claims description 15
- 238000004590 computer program Methods 0.000 claims description 12
- 239000002131 composite material Substances 0.000 claims description 9
- 238000009826 distribution Methods 0.000 claims description 8
- 238000000638 solvent extraction Methods 0.000 claims description 6
- 238000010276 construction Methods 0.000 claims description 3
- 239000003086 colorant Substances 0.000 abstract description 4
- 238000012545 processing Methods 0.000 description 9
- 238000010586 diagram Methods 0.000 description 5
- 238000002474 experimental method Methods 0.000 description 5
- 230000009286 beneficial effect Effects 0.000 description 4
- 238000004891 communication Methods 0.000 description 4
- 238000011156 evaluation Methods 0.000 description 4
- 238000000605 extraction Methods 0.000 description 3
- 230000004927 fusion Effects 0.000 description 3
- 238000011161 development Methods 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 230000008569 process Effects 0.000 description 2
- 238000011144 upstream manufacturing Methods 0.000 description 2
- 230000004913 activation Effects 0.000 description 1
- 230000004931 aggregating effect Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 230000000295 complement effect Effects 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 230000009977 dual effect Effects 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000000802 evaporation-induced self-assembly Methods 0.000 description 1
- 239000000284 extract Substances 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 239000000203 mixture Substances 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 238000011524 similarity measure Methods 0.000 description 1
- 238000004088 simulation Methods 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 239000000126 substance Substances 0.000 description 1
- 238000009827 uniform distribution Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T3/4053—Scaling of whole images or parts thereof, e.g. expanding or contracting based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/70—Determining position or orientation of objects or cameras
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/90—Determination of colour characteristics
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10028—Range image; Depth image; 3D point clouds
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20084—Artificial neural networks [ANN]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30168—Image quality inspection
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Image Analysis (AREA)
Abstract
The application discloses a point cloud super-resolution method, a point cloud super-resolution device, electronic equipment and a storage medium. The method comprises the following steps: constructing and training a point cloud initial super-resolution model to obtain a point cloud super-resolution model meeting a precision threshold; dividing the point cloud to be processed into segmented point clouds, inputting the segmented point clouds into a point cloud super-resolution model, extracting the geometric information features and the color information features of the segmented point clouds through a generator, and reconstructing the super-resolution point cloud through the generator; and judging whether the reconstructed super-resolution point cloud reaches a preset confidence level threshold value or not through a discriminator, and outputting the super-resolution point cloud reaching the preset confidence level threshold value. According to the method, the geometrical and color information of the dense point cloud with high quality can be reconstructed from the sparse point cloud coordinates and colors, the point cloud is modeled by graph convolution, the problem of point cloud disorder is solved, the context information can be effectively utilized, the point cloud characteristics are integrated, and the high-resolution point cloud with accurate outline and clear details can be obtained.
Description
Technical Field
The application relates to the technical field of video image processing, in particular to a point cloud super-resolution method, a point cloud super-resolution device, electronic equipment and a storage medium.
Background
Super-resolution refers to a technique for improving the resolution of an original image by hardware or software. With the rapid development of 3D acquisition technology, 3D sensors (such as LiDAR) become more and more easily available, and point cloud data acquired thereby becomes more and more, and more researchers are gradually invested in the research of three-dimensional point cloud data. However, processing 3D point clouds instead of 2D pixel grids presents new challenges. Unlike the image space represented by a regular grid, the point cloud does not have any spatial order and regular structure. The generated points should describe the basic geometry of the potential target object, which means that they should lie approximately on the surface of the target object. The generated points should be informative and should not be cluttered. The disorder and irregularity of the point cloud data and the above problems limit the development of the deep learning network on the point cloud super-resolution task. The 3D convolution network for analog image processing greatly limits the resolution of point cloud output, point-by-point convolution cannot effectively model disordered unstructured data such as point cloud, and point cloud context information is lacked.
Disclosure of Invention
The application aims to provide a point cloud super-resolution method, a point cloud super-resolution device, an electronic device and a storage medium. The following presents a simplified summary in order to provide a basic understanding of some aspects of the disclosed embodiments. This summary is not an extensive overview and is intended to neither identify key/critical elements nor delineate the scope of such embodiments. Its sole purpose is to present some concepts in a simplified form as a prelude to the more detailed description that is presented later.
According to an aspect of an embodiment of the present application, there is provided a point cloud super-resolution method, including:
constructing and training a point cloud initial super-resolution model to obtain a point cloud super-resolution model meeting a precision threshold; the point cloud initial super-resolution model comprises a generator and a discriminator;
dividing point clouds to be processed into fragment point clouds, and inputting the fragment point clouds into the point cloud super-resolution model;
extracting the geometric information features and the color information features of the fragmented point cloud through the generator, and reconstructing a super-resolution point cloud by using the geometric information features and the color information features through the generator;
and judging whether the reconstructed super-resolution point cloud reaches a preset confidence level threshold value or not through the discriminator, and outputting the super-resolution point cloud reaching the preset confidence level threshold value.
In some embodiments of the present application, the generator comprises a parallel point cloud coordinate generation channel and a point cloud color generation channel; the point cloud coordinate generation channel is used for extracting geometric information features of the segmented point cloud; the point cloud color generation channel is used for extracting color information characteristics of the sliced point cloud.
In some embodiments of the present application, the training point cloud initial super-resolution model comprises:
inputting the point cloud in the training set into the generator, and performing continuous two-time up-sampling operation on the point cloud coordinate generation channel and the point cloud color generation channel to obtain predicted point cloud;
calculating a prediction error of the predicted point cloud, reversely updating parameters in the generator according to the prediction error, and iterating until the prediction error reaches a preset threshold value to obtain a generator which is trained for the first time;
and inputting the point clouds in the training set and the corresponding predicted point clouds into the discriminator to judge the confidence coefficient of the predicted point clouds, updating the whole initial super-resolution model of the point clouds according to the confidence coefficient judgment result, and iterating until the confidence coefficient reaches a preset confidence coefficient threshold value.
In some embodiments of the present application, the training point cloud initial super-resolution model comprises: using composite losses as part of super-resolution task training losses to train the point cloud initial super-resolution model; wherein the composite loss comprises a countermeasure loss, a shape perception loss, and a geometric location-based color loss.
In some embodiments of the present application, the shape perception loss includes an overall constraint and a detail constraint; the integral constraint uses EMD to calculate the distance between the data distribution of the predicted point cloud and the data distribution of the real point cloud for constraint; the detail constraints are constrained using average distance of point-to-point pairs between the predicted point cloud and the real point cloud and a density metric of the generated point cloud.
In some embodiments of the present application, the dividing the point cloud to be processed into sliced point clouds includes:
uniformly dividing the point cloud to be processed into small cubes with the same volume, performing fragment sampling by taking the small cubes as units, and acquiring points of adjacent cubes at a preset overlapping rate during sampling to obtain the fragment point cloud.
In some embodiments of the present application, said extracting, by said generator, geometric information features and color information features of said segmented point cloud comprises:
extracting, by the generator, geometric information features and color information features of the sliced point cloud using a graph convolution operation based on k-nearest neighbors of a three-dimensional space.
According to another aspect of the embodiments of the present application, there is provided a point cloud super-resolution device, including:
the construction training module is used for constructing and training the initial point cloud super-resolution model to obtain a point cloud super-resolution model meeting the precision threshold; the point cloud initial super-resolution model comprises a generator and a discriminator;
the partitioning module is used for partitioning the point cloud to be processed into partitioned point clouds and inputting the partitioned point clouds into the point cloud super-resolution model;
the reconstruction module is used for extracting the geometric information features and the color information features of the fragmented point cloud through the generator and reconstructing the super-resolution point cloud by using the geometric information features and the color information features through the generator;
and the judgment output module is used for judging whether the reconstructed super-resolution point cloud reaches a preset confidence level threshold value or not through the discriminator and outputting the super-resolution point cloud reaching the preset confidence level threshold value.
According to another aspect of embodiments of the present application, there is provided an electronic device including a memory, a processor, and a computer program stored on the memory and executable on the processor, the processor executing the program to implement the point cloud super-resolution method of any one of the above.
According to another aspect of embodiments of the present application, there is provided a computer-readable storage medium having a computer program stored thereon, the program being executed by a processor to implement the point cloud super-resolution method of any one of the above.
The technical scheme provided by one aspect of the embodiment of the application can have the following beneficial effects:
the point cloud super-resolution method provided by the embodiment of the application processes the segmented point cloud by using the point cloud super-resolution model, extracts the geometric information characteristic and the color information characteristic of the segmented point cloud through the generator, reconstructs the super-resolution point cloud by using the geometric information characteristic and the color information characteristic through the generator, judges whether the reconstructed super-resolution point cloud reaches the preset confidence level threshold value through the discriminator, and outputs the super-resolution point cloud reaching the preset confidence level threshold value, can meet the requirements of an upstream task of point cloud processing, effectively solves the problem of point cloud disorder, can effectively use context information to integrate the point cloud characteristics, and can obtain the high-resolution point cloud with accurate outline and clear details.
Additional features and advantages of the application will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by the practice of the embodiments of the application, or may be learned by the practice of the embodiments.
Drawings
In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings needed to be used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments described in the present application, and other drawings can be obtained by those skilled in the art without creative efforts.
FIG. 1 illustrates a flow diagram of a point cloud super-resolution method according to some embodiments of the present application;
FIG. 2 illustrates a flow diagram for training an initial super-resolution model of a point cloud in some embodiments of the present application;
FIG. 3 is a block diagram illustrating a structure of a point cloud super-resolution device according to an embodiment of the present application;
FIG. 4 is a block diagram of an electronic device according to an embodiment of the present application;
FIG. 5 shows a schematic diagram of a computer-readable storage medium of one embodiment of the present application.
The implementation, functional features and advantages of the objects of the present application will be further explained with reference to the accompanying drawings in conjunction with the embodiments.
Detailed Description
In order to make the objects, technical solutions and advantages of the present application more apparent, the present application is further described with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present application and are not intended to limit the present application. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
It will be understood by those within the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. It will be further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of the prior art and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein.
The original point cloud data is unprocessed large complete point cloud, the input point cloud of a generator in the model is sparse fragment point cloud, and the output of the generator is dense fragment point cloud. The complete point cloud is obtained by aggregating all the sliced point clouds.
As shown in fig. 1, one embodiment of the present application provides a point cloud super-resolution method, which in some embodiments of the present embodiment includes steps S10 to S40:
and S10, constructing and training the initial super-resolution point cloud model to obtain the point cloud super-resolution model meeting the precision threshold. The initial super-resolution model of the point cloud comprises a generator and a discriminator.
The generator comprises a parallel point cloud coordinate generation channel and a point cloud color generation channel; the point cloud coordinate generating channel is used for extracting the geometric information characteristics of the segmented point cloud; the point cloud color generation channel is used for extracting color information characteristics of the sliced point cloud.
In some embodiments, parallel channels of the generator, namely a point cloud coordinate generation channel and a point cloud color generation channel, are used, and two channels with different attributes each operate based on structure-aware graph convolution to form channel sub-modules with different layer structures. And when the characteristics of the two parallel channels are extracted, the characteristics of different attributes are fused, a high-dimensional characteristic structure after fusion is used for searching a similar point set to form a graph structure, and the next step of characteristic extraction is carried out.
As shown in fig. 2, in some embodiments, the training point cloud initial super-resolution model comprises:
s101, inputting point clouds in a training set into the generator, and performing continuous twice up-sampling operation on the point cloud coordinate generation channel and the point cloud color generation channel to obtain predicted point clouds;
s102, calculating a prediction error of the predicted point cloud, reversely updating parameters in the generator according to the prediction error, and repeatedly iterating until the prediction error reaches a preset threshold value to obtain a generator which is trained for the first time;
s103, inputting the point clouds in the training set and the corresponding predicted point clouds into the discriminator to judge the confidence coefficient of the predicted point clouds, updating the whole initial super-resolution model of the point clouds according to the confidence coefficient judgment result, and performing iterative operation until the confidence coefficient reaches a preset confidence coefficient threshold value.
In some embodiments, the training point cloud initial super-resolution model comprises: using composite losses as part of super-resolution task training losses to train the point cloud initial super-resolution model; wherein the composite loss comprises a countermeasure loss, a shape perception loss, and a geometric location-based color loss.
In some embodiments, the shape perception loss includes an overall constraint and a detail constraint; the integral constraint uses EMD to calculate the distance between the data distribution of the predicted point cloud and the data distribution of the real point cloud for constraint; the detail constraints are constrained using average distance of point-to-point pairs between the predicted point cloud and the real point cloud and a density metric of the generated point cloud.
In some embodiments, the initial super-resolution model of the training point cloud is divided into two parts, namely training the generator and co-training the generator and the joint arbiter.
The first part trains the generator, inputting the fragmented point cloud (including geometric coordinates and colors), continuously performing up-sampling module operation twice by two parallel channels to obtain high-resolution color point cloud with 4 times of up-sampling rate, calculating the error between the predicted point cloud and the real point cloud by the generator, reversely updating the parameters of a feature extraction module and an up-sampling module in the generator, and repeating iteration until the training of the generator is completed.
The second part trains a generator and a discriminator at the same time, the generator takes point clouds with the same size as the first part as input, and two parallel channels carry out two times of up-sampling operation to obtain high-resolution point clouds; the discriminator takes the real point cloud and the predicted point cloud of the generator as input, continuously samples the same point cloud and judges the truth of the generated point cloud, the whole network is updated at the same time, the generator is updated according to the error between the real point cloud and the predicted point cloud, the discriminator is updated according to the judgment of the correctness of the predicted point cloud, and iteration is carried out repeatedly until the training of the whole model is completed.
And S20, dividing the point cloud to be processed into fragment point clouds, and inputting the fragment point clouds into the point cloud super-resolution model.
In some embodiments, the dividing the point cloud to be processed into sliced point clouds includes:
uniformly dividing the point cloud to be processed into small cubes with the same volume, performing fragment sampling by taking the small cubes as units, and acquiring points of adjacent cubes at a preset overlapping rate during sampling to obtain the fragment point cloud.
For example, the parameters of the minimum cube space occupied by the complete point cloud are counted, the original point cloud is uniformly divided into small cubes with the same volume according to the parameters, the divided small cubes are used as units for sampling in a slicing mode, points of adjacent cubes are collected at a certain overlapping rate during sampling, and therefore loss of edge information of the small cubes is prevented.
In some embodiments, the input data is obtained by sampling the original data in a slicing manner in such a way that cubes are overlapped with each other, and the input point cloud coordinates and color information are normalized correspondingly, so that the model learning is facilitated.
And S30, extracting the geometric information features and the color information features of the fragmented point cloud through the generator, and reconstructing the super-resolution point cloud by the generator through the geometric information features and the color information features.
In some embodiments, the extracting, by the generator, geometric information features and color information features of the segmented point cloud comprises:
extracting, by the generator, geometric information features and color information features of the sliced point cloud using a graph convolution operation based on k-nearest neighbors of a three-dimensional space.
For example, the point cloud attribute information (coordinates and colors) input by the model is firstly subjected to preliminary feature processing on the geometric and color attributes of the point cloud by using the traditional graph convolution operation based on the k-nearest neighbor of the three-dimensional space, and high-dimensional space features related to the two attributes are obtained. This feature serves as the input to the convolution of the structure-aware map.
In some embodiments, the geometric coordinates and color information of the patch point cloud are used as input, the generator uses convolution of a structure perception graph as basic operation to perform feature extraction on the point cloud, and the generator adopts a parallel network framework and fuses multi-attribute information to jointly complete super-resolution of geometry and color.
The structural perception graph convolution method used by the generator utilizes the similarity of high-dimensional space to search point sets with similar structures in the high-dimensional space, the point sets are used for forming a local graph to carry out graph convolution, residual errors are added among the convolutions, more relevant and effective features are extracted, and feature loss is prevented. The convolution of the structure perception graph is as defined in formula (1.1)
Wherein the content of the first and second substances,
is the center point of the ith partial map of the l +1 th layer,
is that
Sorting the jth similar points according to the similarity,
is about a center
Is a set of points that are all structurally similar, σ is an activation function. W is a group of l And U l Learnable parameters of layer I center points and local maps, b l Is the bias term for the l-th layer.
The similarity measure of the high-dimensional structure adopts cosine similarity. As shown in the definition formula (1.2),
wherein x is i Representing the i-dimension characteristic value, y, of a point feature in a point cloud i Representing the ith dimension characteristic value of another point in the point cloud.
And a discriminator in the generative countermeasure network determines the sampling times according to the number of the point clouds, and performs continuous down-sampling and judgment to obtain the final true and false value of the input point clouds.
In some embodiments, the present invention uses the counter-penalty as part of the super-resolution task training penalty, which is defined as shown in equation (1.3) and equation (1.4),
wherein, y represents the true value,
which represents the predicted value of the generator and,
representing the confidence value evaluated by the discriminator. The purpose of the generator is to minimize L gan (G) A prediction value is generated to fool the arbiter.
The geometric loss in the training process uses the shape perception loss proposed by the invention. The shape perception loss comprises integral and detail constraints, and the integral uses EMD to calculate the distance between two data distributions of the predicted point cloud and the real point cloud for constraint; the details are constrained using the average distance of point-to-point pairs between the predicted point cloud and the real point cloud-the CD distance and the density metric of the generated point cloud. Through integral and detail dual constraints, high-quality dense point cloud with clear outline, complete details and uniform distribution can be obtained. The EMD distance and the CD distance are shown in formula (1.5) and formula (1.6),
wherein S is 1 ,S 2 Respectively a real point cloud and a predicted point cloud,
the mapping relation between the real point cloud and the predicted point cloud is obtained according to the specific point cloud.
The shape perception loss is defined as shown in equation (1.7),
L shape (S 1 ,S 2 )＝λ CD ·L CD (S 1 ,S 2 )+λ EMD ·L EMD (S 1 ,S 2 )+λ uni ·L uni , (1.7)
wherein L is uni To distribute losses uniformly, λ CD ，λ EMD And λ uni Is a hyper-parameter of the corresponding item. These values of the over-parameter are set according to specific experiments.
In some embodiments, the color attributes of the point cloud are constrained in combination with the geometric attributes of the point cloud to achieve an accurate constraint effect. Constraint of Point cloud color Using L from the geometric correspondence of the Point cloud 2 The distance calculation corresponds to the distance of the point color rgb to constrain the generation of the point cloud color. The color loss based on geometric position is defined as shown in equation (1.8),
wherein, c 1 Is a value of the prediction RGB, c 2 Is the corresponding point cloud true RGB color value.
In some embodiments, the composite loss includes oppositional loss, shape perception loss, and geometric position-based color loss, and the composite relationship is shown in equation (1.9), L G The composition is shown in formula (1.10),
L total ＝L gan (G)+L gan (D)+λ shape L shape +λ color L color , (1.9)
L G ＝λ gan L gan (G)+λ shape L shape +λ color L color , (1.10)
wherein λ is gan ，λ shape And λ color Are the corresponding weight terms.
In some embodiments, the coordinates, color, and preliminarily extracted geometric and color information features of the point cloud are used as input. Basic features of different attributes are obtained by connecting original data and preliminary features, and the convolution of the structure perception graph in the parallel channel is used for further learning and extracting the basic features. And performing convolution operation on the structure perception graph by taking the high-dimensional features of the point cloud attributes preliminarily extracted in the previous step as basis to calculate to obtain a structure similar local graph required by the structure similar point set to form the convolution, performing convolution operation, and repeating the operation in the same way. And the two attribute channels perform feature fusion only when calculating and calculating the structure-similar local graph to obtain the structure-similar local graph with complementary attributes, and the two attributes help to remove structure deviation points in the local graph so as to obtain the most relevant structure-similar graph and finish graph convolution.
And S40, judging whether the reconstructed super-resolution point cloud reaches a preset confidence level threshold value through the discriminator, and outputting the super-resolution point cloud reaching the preset confidence level threshold value.
In some embodiments, the discriminator takes the predicted segmented point cloud as input, performs continuous multiple down-sampling on the unified segmented point cloud, judges whether the point cloud under different conditions is true or false, and finally determines the output of the confidence coefficient by using multiple values.
Experiments are carried out aiming at the method provided by the embodiment and various methods in the prior art, and the experiments are carried out under the hardware environment of NVIDIA GeForce-GTX 2080GPU and the software environment of Python.
The dataset used in this experiment was JPEG plenum Database. JPEG plenum Database is a public point cloud dataset, and the dynamic voxelized point cloud sequence in this dataset is called 8i voxelized complete entity (8 iVFB). Four sequences were included in the data set, one for the skirt, one for the war, one for the red and black soldier and one for the soldier. In each sequence, 42 RGB cameras in 30 frames are configured into 14 clusters at a speed of 30fps over a time of 10s to capture the entire body of the human body. One spatial resolution is provided for each sequence: a cube of 1024 x 1024 voxels, called depth 10. Each character has 300 frames of point cloud presentations with different poses.
The simulation experiment adopts the method to compare the super-resolution results with a plurality of super-resolution methods on the same data set.
The following table is a statistical table comparing various evaluation indexes by the method of the present embodiment with 7 other geometric super-resolution methods.
In the following table, CD is the chamfer distance, EMD is the earth mover's distance, and HD is the Hausdorffdistance. The JSD measures the degree of similarity of the two distributions, and the PRE → GT error computes the average squared distance from each point in the prediction to the nearest point in the ground truth. GT → PRE error computes the average squared distance of each point in the ground truth to the closest point in the prediction. F-score is a comprehensive evaluation index calculated by accuracy and recall rate.
Table 1 super-resolution distance index result comparison summary table
| Method | CD | EMD | HD |
| PCL-Upsample | 0.0175 | 0.0220 | 0.1296 |
| Fc | 0.1005 | 0.2128 | 1.3662 |
| PCN-CD | 0.0801 | 0.1808 | 0.5499 |
| PCN-EMD | 0.1579 | 0.1869 | 0.5572 |
| Folding | 0.1443 | 0.2220 | 0.9099 |
| PU-Net | 0.1742 | 0.1314 | 1.6710 |
| AR-GCN | 0.0208 | 0.0349 | 0.1527 |
| The method of the invention | 0.0057 | 0.0233 | 0.0368 |
TABLE 2 super resolution non-distance index result comparison List
Since the method applied to geometric super-resolution is generally not applicable to color super-resolution and relatively few tasks are directly aimed at color super-resolution, the following experiments are designed according to experimental requirements.
The following table is a statistical table comparing various evaluation indexes with the method of the present application and 3 other color super-resolution methods of the prior art.
TABLE 3
TABLE 4
As can be seen from the results table, the evaluation indexes of the method of the embodiment are superior to those of other existing methods.
According to the point cloud super-resolution method provided by the embodiment of the application, the generator is combined with the multi-attribute information of the point cloud, and the complementarity between the attributes is utilized to construct the local graph with a similar structure, so that more effective point cloud high-order characteristics are obtained, and the dense reconstruction of the geometric coordinates and the color of the point cloud is facilitated; the discriminator adopts a form of single-chip point cloud multi-time sampling, improves the judgment accuracy of the discriminator, and is beneficial to the overall training of the network model; after the model training is finished, the generator can reconstruct the sparse point cloud coordinates and colors into high-quality dense point cloud geometric and color information, and the requirements of upstream tasks of point cloud processing can be met; the graph convolution is used for modeling point cloud by a graph, so that the problem of point cloud disorder is effectively solved, context information can be effectively utilized, point cloud features are integrated, and the graph convolution operation method is an effective convolution operation method in point cloud processing; according to the complementarity among the attributes, through information fusion, more relevant graph information is used, and locally consistent and relevant high-dimensional features are obtained; and (3) constraining the training of the super-resolution network by using the composite loss to obtain the high-resolution point cloud with accurate contour and clear details.
As shown in fig. 3, another embodiment of the present application provides a point cloud super-resolution device, including:
the construction training module is used for constructing and training the initial point cloud super-resolution model to obtain a point cloud super-resolution model meeting the precision threshold; the point cloud initial super-resolution model comprises a generator and a discriminator;
the partitioning module is used for partitioning the point cloud to be processed into partitioned point clouds and inputting the partitioned point clouds into the point cloud super-resolution model;
the reconstruction module is used for extracting the geometric information features and the color information features of the fragmented point cloud through the generator and reconstructing the super-resolution point cloud by using the geometric information features and the color information features through the generator;
and the judgment output module is used for judging whether the reconstructed super-resolution point cloud reaches a preset confidence level threshold value or not through the discriminator and outputting the super-resolution point cloud reaching the preset confidence level threshold value.
The point cloud super-resolution device provided by the embodiment of the application and the point cloud super-resolution method provided by the embodiment of the application have the same inventive concept and have the same beneficial effects as the method adopted, operated or realized by the device.
Another embodiment of the present application provides an electronic device, which includes a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor executes the program to implement the point cloud super-resolution method of any of the above embodiments.
As shown in fig. 4, the electronic device 10 may include: the system comprises a processor 100, a memory 101, a bus 102 and a communication interface 103, wherein the processor 100, the communication interface 103 and the memory 101 are connected through the bus 102; the memory 101 stores a computer program that can be executed on the processor 100, and the processor 100 executes the computer program to perform the method provided by any of the foregoing embodiments of the present application.
The Memory 101 may include a high-speed Random Access Memory (RAM), and may also include a non-volatile Memory (non-volatile Memory), such as at least one disk Memory. The communication connection between the network element of the system and at least one other network element is realized through at least one communication interface 103 (which may be wired or wireless), and the internet, a wide area network, a local network, a metropolitan area network, and the like can be used.
The bus 102 may be an ISA bus, PCI bus, EISA bus, or the like. The bus may be divided into an address bus, a data bus, a control bus, etc. The memory 101 is used for storing a program, and the processor 100 executes the program after receiving an execution instruction, and the method disclosed in any of the foregoing embodiments of the present application may be applied to the processor 100, or implemented by the processor 100.
The electronic device provided by the embodiment of the application and the method provided by the embodiment of the application have the same inventive concept and have the same beneficial effects as the method adopted, operated or realized by the electronic device.
Another embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, where the computer program is executed by a processor to implement the point cloud super-resolution method of any one of the above embodiments.
Referring to fig. 5, the computer-readable storage medium is an optical disc 20, on which a computer program (i.e., a program product) is stored, and when the computer program is executed by a processor, the computer program performs the method of any of the foregoing embodiments.
It should be noted that examples of the computer-readable storage medium may also include, but are not limited to, phase change memory (PRAM), Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), Read Only Memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), flash memory, or other optical and magnetic storage media, which are not described in detail herein.
The computer-readable storage medium provided by the above-mentioned embodiments of the present application and the method provided by the embodiments of the present application have the same advantages as the method adopted, executed or implemented by the application program stored in the computer-readable storage medium.
It should be noted that:
the term "module" is not intended to be limited to a particular physical form. Depending on the particular application, a module may be implemented as hardware, firmware, software, and/or combinations thereof. Furthermore, different modules may share common components or even be implemented by the same component. There may or may not be clear boundaries between the various modules.
The algorithms and displays presented herein are not inherently related to any particular computer, virtual machine, or other apparatus. Various general purpose devices may also be used with examples based on this disclosure. The required structure for constructing such a device will be apparent from the description above. In addition, this application is not directed to any particular programming language. It will be appreciated that a variety of programming languages may be used to implement the teachings of the present application as described herein, and any descriptions of specific languages are provided above to disclose the best modes of the present application.
It should be understood that, although the steps in the flowcharts of the figures are shown in order as indicated by the arrows, the steps are not necessarily performed in order as indicated by the arrows. The steps are not performed in the exact order shown and may be performed in other orders unless explicitly stated herein. Moreover, at least a portion of the steps in the flow chart of the figure may include multiple sub-steps or multiple stages, which are not necessarily performed at the same time, but may be performed at different times, which are not necessarily performed in sequence, but may be performed alternately or alternately with other steps or at least a portion of the sub-steps or stages of other steps.
The above-mentioned embodiments only express the embodiments of the present application, and the description thereof is more specific and detailed, but not construed as limiting the scope of the present application. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the concept of the present application, which falls within the scope of protection of the present application. Therefore, the protection scope of the present application shall be subject to the appended claims.
Claims (10)
1. A point cloud super-resolution method is characterized by comprising the following steps:
constructing and training a point cloud initial super-resolution model to obtain a point cloud super-resolution model meeting a precision threshold; the point cloud initial super-resolution model comprises a generator and a discriminator;
dividing the point cloud to be processed into fragment point clouds, and inputting the fragment point clouds into the point cloud super-resolution model;
extracting the geometric information features and the color information features of the fragmented point cloud through the generator, and reconstructing a super-resolution point cloud by using the geometric information features and the color information features through the generator;
and judging whether the reconstructed super-resolution point cloud reaches a preset confidence level threshold value or not through the discriminator, and outputting the super-resolution point cloud reaching the preset confidence level threshold value.
2. The point cloud super-resolution method of claim 1,
the generator comprises a parallel point cloud coordinate generation channel and a point cloud color generation channel; the point cloud coordinate generation channel is used for extracting geometric information features of the segmented point cloud; the point cloud color generation channel is used for extracting color information characteristics of the sliced point cloud.
3. The point cloud super-resolution method of claim 2, wherein the training of the initial super-resolution model of the point cloud comprises:
inputting the point cloud in the training set into the generator, and performing continuous two-time up-sampling operation on the point cloud coordinate generation channel and the point cloud color generation channel to obtain predicted point cloud;
calculating a prediction error of the predicted point cloud, reversely updating parameters in the generator according to the prediction error, and iterating until the prediction error reaches a preset threshold value to obtain a generator which is trained for the first time;
and inputting the point clouds in the training set and the corresponding predicted point clouds into the discriminator to judge the confidence coefficient of the predicted point clouds, updating the whole initial super-resolution model of the point clouds according to the confidence coefficient judgment result, and iterating until the confidence coefficient reaches a preset confidence coefficient threshold value.
4. The point cloud super-resolution method of claim 1, wherein the training of the initial super-resolution model of the point cloud comprises: using composite losses as part of super-resolution task training losses to train the point cloud initial super-resolution model; wherein the composite penalty comprises a penalty on confrontation, a penalty on shape perception, and a penalty on color based on geometric position.
5. The point cloud super-resolution method of claim 4, wherein the shape perception loss includes an overall constraint and a detail constraint; the overall constraint uses EMD to calculate the distance between the data distribution of the predicted point cloud and the data distribution of the real point cloud for constraint; the detail constraints are constrained using average distance of point-to-point pairs between the predicted point cloud and the real point cloud and a density metric of the generated point cloud.
6. The point cloud super-resolution method of claim 1, wherein the dividing the point cloud to be processed into patch point clouds comprises:
uniformly dividing the point cloud to be processed into small cubes with the same volume, performing fragment sampling by taking the small cubes as units, and acquiring points of adjacent cubes at a preset overlapping rate during sampling to obtain the fragment point cloud.
7. The point cloud super-resolution method of claim 1, wherein the extracting, by the generator, geometric information features and color information features of the segmented point cloud comprises:
extracting, by the generator, geometric information features and color information features of the sliced point cloud using a graph convolution operation based on k-nearest neighbors of a three-dimensional space.
8. A point cloud super-resolution device is characterized by comprising:
the construction training module is used for constructing and training the initial point cloud super-resolution model to obtain a point cloud super-resolution model meeting the precision threshold; the point cloud initial super-resolution model comprises a generator and a discriminator;
the partitioning module is used for partitioning the point cloud to be processed into partitioned point clouds and inputting the partitioned point clouds into the point cloud super-resolution model;
the reconstruction module is used for extracting the geometric information characteristics and the color information characteristics of the fragment point cloud through the generator and reconstructing the super-resolution point cloud by using the geometric information characteristics and the color information characteristics through the generator;
and the judgment output module is used for judging whether the reconstructed super-resolution point cloud reaches a preset confidence level threshold value or not through the discriminator and outputting the super-resolution point cloud reaching the preset confidence level threshold value.
9. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor executing the program to implement the method of any one of claims 1-7.
10. A computer-readable storage medium, on which a computer program is stored, characterized in that the program is executed by a processor to implement the method according to any of claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210082353.5A CN114881850A (en) | 2022-01-24 | 2022-01-24 | Point cloud super-resolution method and device, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210082353.5A CN114881850A (en) | 2022-01-24 | 2022-01-24 | Point cloud super-resolution method and device, electronic equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114881850A true CN114881850A (en) | 2022-08-09 |
Family
ID=82667112
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210082353.5A Pending CN114881850A (en) | 2022-01-24 | 2022-01-24 | Point cloud super-resolution method and device, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114881850A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115393533A (en) * | 2022-10-27 | 2022-11-25 | 科大讯飞股份有限公司 | Three-dimensional point cloud construction method and device, electronic equipment and storage medium |
-
2022
- 2022-01-24 CN CN202210082353.5A patent/CN114881850A/en active Pending
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115393533A (en) * | 2022-10-27 | 2022-11-25 | 科大讯飞股份有限公司 | Three-dimensional point cloud construction method and device, electronic equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Fei et al. | Comprehensive review of deep learning-based 3d point cloud completion processing and analysis | |
| Xie et al. | Point clouds learning with attention-based graph convolution networks | |
| WO2023025030A1 (en) | Three-dimensional point cloud up-sampling method and system, device, and medium | |
| Wang et al. | Voxel-based network for shape completion by leveraging edge generation | |
| Ye et al. | Drinet: A dual-representation iterative learning network for point cloud segmentation | |
| CN105118090B (en) | A kind of point cloud filtering method of adaptive complicated landform structure | |
| CN113781667B (en) | Three-dimensional structure simplified reconstruction method and device, computer equipment and storage medium | |
| CN113436211B (en) | Medical image active contour segmentation method based on deep learning | |
| CN112347987A (en) | Multimode data fusion three-dimensional target detection method | |
| Wang et al. | Cascaded refinement network for point cloud completion with self-supervision | |
| Li et al. | ADR-MVSNet: A cascade network for 3D point cloud reconstruction with pixel occlusion | |
| CN113052955A (en) | Point cloud completion method, system and application | |
| Liu et al. | High-quality textured 3D shape reconstruction with cascaded fully convolutional networks | |
| CN113378112A (en) | Point cloud completion method and device based on anisotropic convolution | |
| CN115272599A (en) | Three-dimensional semantic map construction method oriented to city information model | |
| CN106611030A (en) | Object similarity comparison method and object search method based on video, and object similarity comparison system and object search system based on video | |
| CN113436237B (en) | High-efficient measurement system of complicated curved surface based on gaussian process migration learning | |
| CN117115359B (en) | Multi-view power grid three-dimensional space data reconstruction method based on depth map fusion | |
| CN114881850A (en) | Point cloud super-resolution method and device, electronic equipment and storage medium | |
| Van Nguyen et al. | Geometric modeling: background for processing the 3d objects | |
| CN104463962A (en) | Three-dimensional scene reconstruction method based on GPS information video | |
| CN116758219A (en) | Region-aware multi-view stereo matching three-dimensional reconstruction method based on neural network | |
| Zhang et al. | PS-Net: Point shift network for 3-D point cloud completion | |
| Lin et al. | A-SATMVSNet: An attention-aware multi-view stereo matching network based on satellite imagery | |
| Chen et al. | Ground 3D Object Reconstruction Based on Multi-View 3D Occupancy Network using Satellite Remote Sensing Image |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |