CN114637880A - Cross-dimensional data retrieval method based on multi-view projection - Google Patents

Abstract

The invention provides a cross-dimensional data retrieval method based on multi-view projection, which comprises the following steps: acquiring two-dimensional image data and a corresponding matched original three-dimensional point cloud; carrying out voxelization processing on the corresponding matched original three-dimensional point cloud to obtain a corresponding voxel; projecting the corresponding voxels to a two-dimensional space to generate a point cloud multi-view projection image corresponding to each two-dimensional image; constructing a deep learning model according to the twin network, and inputting two-dimensional image data and the correspondingly matched point cloud multi-view projection image into the deep learning model for training; acquiring a plurality of two-dimensional images and three-dimensional point clouds to be retrieved, and retrieving the three-dimensional point clouds from the two-dimensional images based on the trained deep learning model to obtain the three-dimensional point clouds which are most matched in all the three-dimensional point clouds of each two-dimensional image to be retrieved; therefore, the data difference between the point cloud data and the two-dimensional image in the cross-dimensional matching process can be reduced, and the retrieval accuracy rate from the two-dimensional image to the three-dimensional point cloud is improved.

Description

Cross-dimensional data retrieval method based on multi-view projection

technical field

The present invention relates to the technical field of augmented reality, and in particular, to a cross-dimensional data retrieval method based on multi-view projection, a computer-readable storage medium and a computer device.

Background technique

In the related art, the retrieval-based pose estimation method is divided into two ways: retrieval from 2D images to 2D images and retrieval from 2D images to 3D models; among them, retrieval from 2D images to 2D images requires Massive 2D images are 3D reconstructed into 3D space, and then the target image is retrieved and matched with all the images in the 3D space, and the camera pose is estimated by PnP. Due to the limitation of 2D images, this method is easily affected by the angle. It is difficult to apply to complex scenes; and the retrieval of 2D images to 3D models is to match the 2D images with the pre-established 3D models. Since the 3D models are rotationally invariant, they are more robust; however, However, the research on cross-modal matching of two-dimensional images and three-dimensional point clouds is relatively scarce, and there are differences in data dimensions and data structures between different data, making cross-dimensional matching difficult to complete, resulting in low accuracy of cross-dimensional data retrieval.

SUMMARY OF THE INVENTION

The present invention aims to solve one of the technical problems in the above technologies at least to a certain extent. To this end, an object of the present invention is to propose a cross-dimensional data retrieval method based on multi-view projection, through the point cloud processing method of multi-view projection, to reduce the data difference between point cloud data and two-dimensional images in cross-dimensional matching, Thus, the retrieval accuracy of 2D image to 3D point cloud can be improved.

A second object of the present invention is to provide a computer-readable storage medium.

The third object of the present invention is to propose a computer device.

In order to achieve the above object, the embodiment of the first aspect of the present invention proposes a cross-dimensional data retrieval method based on multi-view projection, which includes the following steps: acquiring two-dimensional image data and corresponding to each two-dimensional image in the two-dimensional image data. Correspondingly matched original three-dimensional point cloud; performing voxelization processing on the corresponding matched original three-dimensional point cloud of each two-dimensional image in the two-dimensional image data to obtain corresponding voxels; projecting the corresponding voxels to Two-dimensional space to generate a corresponding matching point cloud multi-view projection image of each two-dimensional image; build a deep learning model according to the twin network, and combine the two-dimensional image data with each two-dimensional image in the two-dimensional image data. The corresponding matching point cloud multi-view projection images are input into the deep learning model for training; a plurality of 2D images and 3D point clouds to be retrieved are obtained, and the plurality of 3D point clouds to be retrieved are input into the trained depth In the learning model to obtain the point cloud feature description, and input the plurality of two-dimensional images to be retrieved into the trained deep learning model to obtain the image feature description, and according to the point cloud feature description and the image feature description Description Retrieves the 3D point cloud that best matches among all 3D point clouds for each 2D image to be retrieved.

According to the cross-dimensional data retrieval method based on multi-view projection according to the embodiment of the present invention, firstly obtain two-dimensional image data and an original three-dimensional point cloud corresponding to each two-dimensional image in the two-dimensional image data; Each 2D image corresponds to the matched original 3D point cloud to be voxelized to obtain the corresponding voxel; then the corresponding voxel is projected into the 2D space to generate a multi-view point cloud corresponding to each 2D image. projection image; then build a deep learning model according to the twin network, and input the two-dimensional image data and the point cloud multi-view projection image corresponding to each two-dimensional image in the two-dimensional image data into the deep learning model for training; finally obtain There are multiple 2D images and 3D point clouds to be retrieved, and each 2D image to be retrieved is retrieved based on the trained deep learning model, so that each 2D image to be retrieved is the most among all 3D point clouds. Matching 3D point cloud; thus, through the point cloud processing method of multi-view projection, the data difference between the point cloud data and the 2D image in the cross-dimensional matching is reduced, thereby improving the retrieval accuracy of the 2D image to the 3D point cloud.

In addition, the cross-dimensional data retrieval method based on multi-view projection proposed according to the above-mentioned embodiments of the present invention may also have the following additional technical features:

Optionally, performing voxelization processing on the original three-dimensional point cloud corresponding to each two-dimensional image in the two-dimensional image data to obtain corresponding voxels, including: taking the cubic bounding box of the original three-dimensional point cloud as the boundary Perform voxelized space division; evenly divide multiple cubes in the divided voxelized space, and treat each cube as a voxel; define the voxel value of each voxel as The mean of the point cloud RGB values.

Optionally, before performing the voxelization process on the original three-dimensional point cloud corresponding to each two-dimensional image in the two-dimensional image data, it also includes: performing random angle rotation on the original three-dimensional point cloud, so that the obtained 3D voxels have rotational randomness.

Optionally, build a deep learning model based on the Siamese network, including:

The deep learning twin network structure framework is used to build, and an asymmetric structure dual-branch network with image branch and point cloud branch is designed;

Wherein, the image branch includes an image feature extraction network based on a convolutional network, so as to process two-dimensional image data through the image feature extraction network to obtain a corresponding image feature description; wherein, the point cloud branch includes a point cloud-based The point cloud feature extraction network of multi-view projection and the fusion network that fuses point cloud texture features and point cloud structure features, the point cloud feature extraction network contains texture perceptron and structure perceptron, so that the point cloud The data is processed to obtain point cloud texture features and point cloud structure features; a fusion network that fuses point cloud texture features and point cloud structure features receives the point cloud texture features and point cloud structure features at the same time, and fuses them to obtain corresponding points Cloud feature description; local feature loss function design, the loss function is based on the 2D image negative samples and 3D point cloud negative samples sampled in the training process, so that the distance between the image feature description and the point cloud feature description is shortened, so that the image feature description and the 3D point cloud feature description are closer. The distance between the point cloud negative samples and the point cloud feature description is estranged from the two-dimensional image negative samples.

Optionally, the texture perceptron is used for processing point cloud multi-view projection images, and perceiving texture information contained in the point cloud; the texture perceptron includes a convolutional network and a feature fusion function.

Optionally, the convolutional network is used to process multi-angle projections to obtain n one-dimensional feature descriptions with a fixed length d; the feature fusion function is a symmetrical function that is insensitive to the input order, and n one-dimensional feature descriptions with a fixed length d. It is transformed into a one-dimensional feature description t with a fixed length d by a feature fusion function, and the one-dimensional feature description t with a fixed length d is used as a point cloud texture feature.

Optionally, the structure perceptron is used to process the original point cloud data, and perceive the structure information contained in the original point cloud data, wherein, the structure perceptron adopts a feature extraction method based on the PointNet network structure to obtain a fixed length d. The one-dimensional feature description s is the one-dimensional feature description s with a fixed length d as the point cloud structure feature.

In order to achieve the above object, a second aspect of the present invention provides a computer-readable storage medium on which a multi-view projection-based cross-dimensional data retrieval program is stored, and the multi-view projection-based cross-dimensional data retrieval program is processed. The multi-view projection-based cross-dimensional data retrieval method as described above is implemented when the processor is executed.

According to the computer-readable storage medium of the embodiment of the present invention, the multi-view projection-based cross-dimensional data retrieval program is stored, so that the processor implements the multi-view projection-based multi-view projection-based data retrieval program as described above when the multi-view projection-based cross-dimensional data retrieval program is executed. The cross-dimensional data retrieval method can improve the retrieval accuracy of 2D image to 3D point cloud.

In order to achieve the above object, a third aspect of the present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and running on the processor, when the processor executes the computer program , to realize the cross-dimensional data retrieval method based on multi-view projection as described above.

According to the computer device of the embodiment of the present invention, a computer program that can be run on the processor is stored in the memory, so that when the processor executes the computer program, the multi-view projection-based cross-dimensional data retrieval method as described above is implemented, thereby improving the two-dimensionality. Retrieval accuracy from 3D image to 3D point cloud.

Description of drawings

1 is a schematic flowchart of a method for retrieving cross-dimensional data based on multi-view projection according to an embodiment of the present invention;

2 is a schematic diagram of a two-dimensional and three-dimensional common feature description network structure according to an embodiment of the present invention;

3 is a schematic structural diagram of a network model of a point cloud branch according to an embodiment of the present invention;

4 is a schematic diagram of a plane point cloud visualization and its projection according to an embodiment of the present invention;

5 is a paired matching 2D image-3D point cloud according to an embodiment of the present invention;

6 is a schematic diagram of a negative sample sampling strategy based on difficult samples according to an embodiment of the present invention;

7 is a schematic diagram of a retrieval result from a two-dimensional image to a three-dimensional point cloud according to an embodiment of the present invention;

FIG. 8 is a schematic diagram of visualization of two-dimensional and three-dimensional common feature descriptors according to an embodiment of the present invention.

Detailed ways

The following describes in detail the embodiments of the present invention, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals refer to the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary, and are intended to explain the present invention and should not be construed as limiting the present invention.

For better understanding of the above technical solutions, exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. While exemplary embodiments of the present invention are shown in the drawings, it should be understood that the present invention may be embodied in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided so that the present invention will be more thoroughly understood, and will fully convey the scope of the present invention to those skilled in the art.

In order to better understand the above technical solutions, the above technical solutions will be described in detail below with reference to the accompanying drawings and specific embodiments.

1 is a schematic flowchart of a multi-view projection-based cross-dimensional data retrieval method according to an embodiment of the present invention. As shown in FIG. 1 , the multi-view projection-based cross-dimensional data retrieval method according to an embodiment of the present invention includes the following steps:

Step 101: Acquire two-dimensional image data and an original three-dimensional point cloud corresponding to each two-dimensional image in the two-dimensional image data.

As an embodiment, the two-dimensional image data may be acquired by a camera, and the original three-dimensional point cloud may be acquired by a lidar, which is not specifically limited in the present invention.

Step 102: Perform voxelization processing on the original three-dimensional point cloud corresponding to each two-dimensional image in the two-dimensional image data, so as to obtain corresponding voxels.

As an embodiment, performing voxelization processing on the original 3D point cloud corresponding to each 2D image in the 2D image data to obtain corresponding voxels, including: using the cube bounding box of the original 3D point cloud as a boundary Voxelized space division; evenly divides multiple cubes in the divided voxelized space, and treats each cube as a voxel; defines the voxel value of each voxel as the point cloud RGB contained in each cube space the mean of the values.

It should be noted that the 3D point cloud data is voxelized to prepare for the subsequent projection.

As a specific embodiment, the following steps are included:

S21 , using the cube bounding box of the original three-dimensional point cloud as a boundary to divide the voxelized space.

S22. In the voxelized space divided by S21, the cubes of 32x32x32 are evenly divided, and each small cube is used as a voxel; each small cube is defined as V _i,j,k , where i, j, k represent parallel to each other The number of cubes in three vertical directions.

S23. For the point cloud containing 1024 points in the dataset, define it as P={p ₀ , p ₁ ,...,p ₁₀₂₃ }, construct a zero matrix M _{32×32×32×1024} , and record whether the spatial point cloud falls on V _{i ,j,k} ; define the voxel value of each voxel as the mean of the RGB values of the point cloud contained in a single small cube space:

where p _v represents the RGB value of the point in the point cloud, and Mi _,j,k represent the voxel value in the voxel space.

As an embodiment, before performing voxelization processing on the original 3D point cloud corresponding to each 2D image in the 2D image data, the method further includes: performing random angle rotation on the original 3D point cloud, so that the obtained 3D voxel Has rotational randomness.

As a specific embodiment, the original three-dimensional point cloud is randomly rotated by an angle (α, β, γ), so that the three-dimensional voxels obtained in step 102 have rotational randomness:

M=R _x (α)R _y (β)R _z (γ)

P'=MP

Among them, R represents the rotation transformation with the coordinate axis as the rotation axis, and the point cloud P is completely transformed by M to obtain the point cloud P′.

Step 103: Project the corresponding voxels into a two-dimensional space to generate a corresponding matching point cloud multi-view projection image of each two-dimensional image.

That is to say, the voxels obtained in step 102 are projected into three planes xOy, yOz, and xOz in a two-dimensional space, wherein the three planes are perpendicular to each other, and the projection is saved as a 64x64 image format as shown in Figure 4 .

In step 104, a deep learning model is constructed according to the twin network, and the two-dimensional image data and the point cloud multi-view projection image corresponding to each two-dimensional image in the two-dimensional image data are input into the deep learning model for training.

That is to say, two-dimensional image blocks and three-dimensional point cloud blocks are collected in the scene, and then input to the constructed deep learning model 2D3D-MVPNet to extract common feature descriptors, in which the deep learning model consists of two-dimensional image branches and three-dimensional point clouds. The 3D point cloud adopts a new network model that integrates multi-view views, and the 2D3D-MVPNet network structure diagram is shown in Figure 3.

As an embodiment, constructing a deep learning model according to a twin network includes: building a deep learning twin network structure framework, and designing an asymmetric structure double-branch network with image branches and point cloud branches; wherein, the image branch includes a convolutional network-based The image feature extraction network is used to process the two-dimensional image data through the image feature extraction network to obtain the corresponding image feature description; wherein, the point cloud branch includes a point cloud feature extraction network based on point cloud multi-view projection and fusion point cloud texture features and The fusion network of point cloud structure features, the point cloud feature extraction network includes texture perceptron and structure perceptron, so that point cloud data can be processed by texture perceptron and structure perceptron to obtain point cloud texture features and point cloud structure features; A fusion network that fuses point cloud texture features and point cloud structure features, simultaneously receives point cloud texture features and point cloud structure features, and fuses them to obtain the corresponding point cloud feature description; the local feature loss function is designed, and the loss function is based on the sampling in the training process. 2D image negative samples and 3D point cloud negative samples make the distance between image feature description and point cloud feature description closer, and make the distance between image feature description and 3D point cloud negative sample and point cloud feature description and 2D image negative sample distance away .

As an embodiment, the texture perceptron is used to process the multi-view projection image of the point cloud, and perceive the texture information contained in the point cloud; the texture perceptron includes a convolutional network and a feature fusion function.

As an embodiment, a convolutional network is used to process multi-angle projections to obtain n one-dimensional feature descriptions with a fixed length d; the feature fusion function is a symmetric function that is insensitive to the input order, and the n one-dimensional feature descriptions with a fixed length d are represented by The feature fusion function is transformed into a one-dimensional feature description t with a fixed length d, and a one-dimensional feature description t with a fixed length d is used as the point cloud texture feature.

As an embodiment, the structure perceptron is used to process the original point cloud data and perceive the structure information contained in the original point cloud data, wherein the structure perceptron adopts a feature extraction method based on the PointNet network structure to obtain a one-dimensional feature description with a fixed length d s, a one-dimensional feature description s with a fixed length d is used as the point cloud structure feature.

That is to say, as shown in Figure 2, the construction of a deep learning model includes the following steps:

S11. The deep learning twin network structure framework is built, and an asymmetric structure dual-branch network with image branch and point cloud branch is designed.

S12. The network structure design of image feature extraction based on convolutional network, the two-dimensional image data is processed through the image feature extraction network, and the network parameters of the image feature extraction network are C(32,4,2)-BN-ReLU-C(64, 4,2)-BN-ReLU-C(128,4,2)-BN-ReLU-C(256,4,2)-BN-ReLU-C(256,4,4), Note: C(n, k, s) represents the volume network parameters with n filters, the convolution kernel is k*k, and the stride is s, BN represents bundle regularization, and RELU represents activation function. Finally, the design network obtains a one-dimensional feature description p with a fixed length of 256, that is, the image feature description;

S13. Network structure design of point cloud feature extraction based on point cloud multi-perspective projection. The point cloud feature extraction network includes a texture perceptron and a structure perceptron: the texture perceptron is used to process the multi-perspective projection view of the point cloud and perceive the texture information contained in the point cloud ; It consists of a convolutional network and a feature fusion function. The parameters of the convolutional network are the same as those of the convolutional network in S12. The convolutional network processes multi-angle projection to obtain n one-dimensional feature descriptions with a fixed length of 256 {f ₁ , f ₂ , ...,f _n }; The feature fusion function is essentially a symmetric function that is insensitive to the input order. In this example, the method selects the sum function as the feature fusion function, so n one-dimensional feature descriptions with a fixed length of 256 are transformed by the feature fusion function. Describe t for a one-dimensional feature of fixed length 256:

sum{f ₁ ,f ₂ ,...,f _n }=t

The structure perceptron is used to process the original point cloud data and perceive the structure information contained in the point cloud data; specifically, the structure perceptron adopts the feature extraction method based on the PointNet network structure to obtain a one-dimensional feature description s with a fixed length of 256.

S14, fusion network structure design of fusion point cloud texture features and structural features; in S13, point cloud data are input into texture sensor and structure sensor respectively to obtain texture feature t and structural feature s of length 256; fusion network design is 2 layers The fully connected layer of , whose specific parameters are FC(512,256)-ReLU-FC(256,256), Note: FC(p,q) represents mapping a one-dimensional vector of length p to a one-dimensional vector of length q through a neural network ; The fusion network simultaneously receives texture feature t and structural feature s, connects t and s as a vector of length 512 and inputs it into the fully connected layer, and obtains a point cloud one-dimensional feature description v of fixed length d, that is, point cloud feature description;

S15. Design of local feature loss function. In S12 and S14, the image feature description p and the point cloud feature description v are obtained respectively. The loss function is based on the negative samples p _n and v _n sampled during the training process, so that the distance between the features p and v is shortened, and the features p and v _n , and The distance distance between feature v and p _n :

和

分别表示与p_i最相近的非匹配点云特征和与v_i最相近的非匹配图像特征。In the formula, d( _pi , _vi ) represents the _Euclidean distance between the image feature _pi and the point cloud feature vi,

and

represent the non-matching point cloud features closest to _pi and the non-matching image features closest to _vi , respectively.

As an embodiment, the data set shown in FIG. 5 is used as training data in the training process, with a total of 280,000 pairs of two-dimensional image-3D matching pairs; based on this, the training extracts network parameters described by common features, and this step is specifically performed through the following steps accomplish:

S41. Training parameter setting, wherein each batch is defined as having a size of 64, and 64 pairs of 2D image-3D point cloud matching pairs are input to the network each time.

和

其分别表示与图像p_i最相近的非匹配点云 特征和与v_i最相近的非匹配图像特征；然后比较

和

的大小，选择较小 的样本作为匹配对(p_i,v_j)的负样本。S42, a negative sample sampling strategy. The output image feature description p and point cloud feature description v are used to construct negative samples using the difficult sample strategy. Each time, the two-dimensional images and three-dimensional networks that are the most difficult to distinguish are selected as negative samples, and the input two-dimensional images and 3D point clouds are used as positive samples. Sample, construct ternary loss. Difficult sample sampling is shown in Figure 6. In a bundle of training data, for matching pairs ( _{pi , v j )} , find and define

and

which respectively represent the non-matching point cloud features closest to the image _pi and the non-matching image features closest to _vi ; then compare

and

, select the smaller sample as the negative sample of the matching pair ( _{pi ,v j )} .

Step 105: Acquire multiple 2D images and 3D point clouds to be retrieved, input multiple 3D point clouds to be retrieved into the trained deep learning model to obtain point cloud feature descriptions, and combine the multiple 2D images to be retrieved. The 2D image is input into the trained deep learning model to obtain the image feature description, and according to the point cloud feature description and the image feature description, each 2D image to be retrieved is the most matching 3D point cloud among all the 3D point clouds.

That is to say, the trained network has the ability to extract common feature descriptors, and quantifies the quality of its feature descriptors through retrieval tasks; the retrieval results are shown in Figure 7, and the specific steps are as follows:

S51. Select the network model constructed by S1 with the best effect as the engineering model.

S52. Input all two-dimensional pictures {P ₁ , P ₂ , P ₃ ......P _n } into the picture branch of the model to obtain two-dimensional picture features {p ₁ , p ₂ , p ₃ ...... p _n }.

S53. Input all the three-dimensional point clouds {V ₁ , V ₂ , V ₃ ......V _n } into the point cloud branches of the model to obtain the three-dimensional point cloud features {v ₁ , v ₂ , v ₃ ...... v _n }.

S54. For a single two-dimensional image feature p _i , retrieve the nearest feature description v _j from all features {v ₁ , v ₂ , v ₃ ...... v _n } of the three-dimensional point cloud. If i is equal to j, for the TOP1 retrieval accuracy, it belongs to a group of successful retrievals from 2D images to 3D point clouds. If j belongs to one of the five retrieved nearest neighbors of i, for TOP5 retrieval accuracy, it belongs to a set of successful retrievals from 2D images to 3D point clouds. The visual retrieval results of the feature descriptors of the TOP1 results that have been successfully retrieved are shown in Figure 8. Taking the retrieval accuracy of TOP1 and TOP5 as the criterion, the formula is as follows:

Among them, TP represents the number of successful retrieval samples, TN represents the number of failed retrieval samples, and n represents the total number of samples.

It should be noted that the present invention proposes a common feature description network framework for two-dimensional images and three-dimensional point clouds, and proposes a method idea for pose estimation; the point cloud data is reduced by a point cloud feature extractor based on point cloud multi-angle projection. The data difference between the two-dimensional image and the two-dimensional image in the cross-dimensional matching improves the retrieval accuracy of the two-dimensional image to the three-dimensional point cloud; the unordered feature input is fused by the symmetric function, and the feature fusion problem of the point cloud multi-projection technology is solved; the use of a large number of Data training 2D and 3D common feature description, automatic deep learning method replaces manual design feature description, saves labor costs, and improves the efficiency and accuracy of machine pose estimation.

To sum up, according to the cross-dimensional data retrieval method based on multi-view projection of the embodiment of the present invention, firstly obtain the two-dimensional image data and the original three-dimensional point cloud corresponding to each two-dimensional image in the two-dimensional image data; Each two-dimensional image in the two-dimensional image data corresponds to the matched original three-dimensional point cloud for voxelization to obtain the corresponding voxels; then the corresponding voxels are projected into the two-dimensional space to generate the corresponding matching of each two-dimensional image point cloud multi-perspective projection image; then build a deep learning model according to the twin network, and input the two-dimensional image data and the point cloud multi-perspective projection image corresponding to each two-dimensional image in the two-dimensional image data into the deep learning model Carry out training; finally obtain multiple 2D images and 3D point clouds to be retrieved, and retrieve each 2D image to be retrieved based on the trained deep learning model, so as to obtain each 2D image to be retrieved in all The most matching 3D point cloud in the 3D point cloud; thus, through the point cloud processing method of multi-view projection, the data difference between the point cloud data and the 2D image in the cross-dimensional matching is reduced, thereby improving the 2D image to the 3D point cloud. retrieval accuracy.

In addition, an embodiment of the present invention also provides a computer-readable storage medium on which a multi-view projection-based cross-dimensional data retrieval program is stored, and when the multi-view projection-based cross-dimensional data retrieval program is executed by a processor, the following The above-mentioned cross-dimensional data retrieval method based on multi-view projection.

According to the computer-readable storage medium of the embodiment of the present invention, the multi-view projection-based cross-dimensional data retrieval program is stored, so that the processor implements the multi-view projection-based multi-view projection-based data retrieval program as described above when the multi-view projection-based cross-dimensional data retrieval program is executed. The cross-dimensional data retrieval method can improve the retrieval accuracy of 2D image to 3D point cloud.

In addition, an embodiment of the present invention also proposes a computer device, including a memory, a processor, and a computer program stored in the memory and running on the processor. When the processor executes the computer program, the above-mentioned computer program is implemented. A cross-dimensional data retrieval method based on multi-view projection.

According to the computer device of the embodiment of the present invention, a computer program that can be run on the processor is stored in the memory, so that when the processor executes the computer program, the multi-view projection-based cross-dimensional data retrieval method as described above is implemented, thereby improving the two-dimensionality. Retrieval accuracy from 3D image to 3D point cloud.

As will be appreciated by one skilled in the art, embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein.

The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block in the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to the processor of a general purpose computer, special purpose computer, embedded processor or other programmable data processing device to produce a machine such that the instructions executed by the processor of the computer or other programmable data processing device produce Means for implementing the functions specified in a flow or flow of a flowchart and/or a block or blocks of a block diagram.

These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory result in an article of manufacture comprising instruction means, the instructions The apparatus implements the functions specified in the flow or flow of the flowcharts and/or the block or blocks of the block diagrams.

These computer program instructions can also be loaded on a computer or other programmable data processing device to cause a series of operational steps to be performed on the computer or other programmable device to produce a computer-implemented process such that The instructions provide steps for implementing the functions specified in the flow or blocks of the flowcharts and/or the block or blocks of the block diagrams.

It should be noted that, in the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. The word "comprising" does not exclude the presence of elements or steps not listed in a claim. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The invention can be implemented by means of hardware comprising several different components and by means of a suitably programmed computer. In a unit claim enumerating several means, several of these means may be embodied by one and the same item of hardware. The use of the words first, second, and third, etc. do not denote any order. These words can be interpreted as names.

Although the preferred embodiments of the present invention have been described, additional changes and modifications to these embodiments may occur to those skilled in the art once the basic inventive concepts are known. Therefore, the appended claims are intended to be construed to include the preferred embodiment and all changes and modifications that fall within the scope of the present invention.

It will be apparent to those skilled in the art that various modifications and variations can be made in the present invention without departing from the spirit and scope of the invention. Thus, provided that these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include such modifications and variations.

In the description of the present invention, it should be understood that the terms "first" and "second" are only used for description purposes, and cannot be interpreted as indicating or implying relative importance or implying the number of indicated technical features. Thus, a feature defined as "first", "second" may expressly or implicitly include one or more of that feature. In the description of the present invention, "plurality" means two or more, unless otherwise expressly and specifically defined.

In the present invention, unless otherwise expressly specified and limited, the terms "installation", "connection", "connection", "fixation" and other terms should be understood in a broad sense, for example, it may be a fixed connection or a detachable connection , or integrated; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, and it can be the internal connection of the two elements or the interaction relationship between the two elements. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific situations.

In the present invention, unless otherwise expressly specified and limited, a first feature "on" or "under" a second feature may be in direct contact between the first and second features, or the first and second features indirectly through an intermediary touch. Also, a first feature being "above", "over" and "above" a second feature may mean that the first feature is directly above or obliquely above the second feature, or simply means that the first feature is level higher than the second feature. A first feature being "below", "below" and "below" a second feature may mean that the first feature is directly below or diagonally below the second feature, or simply means that the first feature is level less than the second feature.

In the description of this specification, description with reference to the terms "one embodiment," "some embodiments," "example," "specific example," or "some examples", etc., mean specific features described in connection with the embodiment or example , structure, material or feature is included in at least one embodiment or example of the present invention. In this specification, schematic representations of the above terms should not be construed as necessarily referring to the same embodiment or example. Furthermore, the particular features, structures, materials or characteristics described may be combined in any suitable manner in any one or more embodiments or examples. Furthermore, those skilled in the art may combine and combine the different embodiments or examples described in this specification, as well as the features of the different embodiments or examples, without conflicting each other.

Although the embodiments of the present invention have been shown and described above, it should be understood that the above-mentioned embodiments are exemplary and should not be construed as limiting the present invention. Embodiments are subject to variations, modifications, substitutions and variations.

Claims (9)

1. A cross-dimensional data retrieval method based on multi-view projection is characterized by comprising the following steps:

acquiring two-dimensional image data and an original three-dimensional point cloud correspondingly matched with each two-dimensional image in the two-dimensional image data;

performing voxelization processing on the original three-dimensional point cloud correspondingly matched with each two-dimensional image in the two-dimensional image data to obtain corresponding voxels;

projecting the corresponding voxels to a two-dimensional space to generate a point cloud multi-view projection image corresponding to each two-dimensional image;

constructing a deep learning model according to a twin network, and inputting the two-dimensional image data and the point cloud multi-view projection image which is correspondingly matched with each two-dimensional image in the two-dimensional image data into the deep learning model for training;

the method comprises the steps of obtaining a plurality of two-dimensional images to be retrieved and three-dimensional point clouds, inputting the three-dimensional point clouds to be retrieved into a trained deep learning model to obtain point cloud feature description, inputting the two-dimensional images to be retrieved into the trained deep learning model to obtain image feature description, and retrieving the three-dimensional point clouds, which are most matched in all the three-dimensional point clouds, of each two-dimensional image to be retrieved according to the point cloud feature description and the image feature description.

2. The multi-view projection-based cross-dimensional data retrieval method of claim 1, wherein performing voxelization processing on the original three-dimensional point cloud corresponding to each two-dimensional image in the two-dimensional image data to obtain corresponding voxels comprises:

performing voxelization space division by taking a cubic boundary frame of the original three-dimensional point cloud as a boundary;

uniformly dividing a plurality of cubes in the divided voxelized space, and taking each cube as a voxel;

the voxel value of each voxel is defined as the mean of the point cloud RGB values contained in each cube space.

3. The multi-view projection-based cross-dimensional data retrieval method of claim 2, wherein before performing the voxelization processing on the original three-dimensional point cloud corresponding to each two-dimensional image in the two-dimensional image data, the method further comprises: and carrying out angle random rotation on the original three-dimensional point cloud so as to enable the obtained three-dimensional voxel to have rotation randomness.

4. The multi-view projection-based cross-dimensional data retrieval method of claim 2, wherein constructing a deep learning model from a twin network comprises:

constructing by adopting a deep learning twin network structure framework, and designing an asymmetric structure double-branch network with image branches and point cloud branches;

wherein the image branch comprises an image feature extraction network based on a convolution network, so that two-dimensional image data is processed through the image feature extraction network to obtain corresponding image feature description;

the point cloud branch comprises a point cloud characteristic extraction network based on point cloud multi-view projection and a fusion network fusing point cloud texture characteristics and point cloud structural characteristics, wherein the point cloud characteristic extraction network comprises a texture sensor and a structure sensor so as to process point cloud data through the texture sensor and the structure sensor to obtain point cloud texture characteristics and point cloud structural characteristics; fusing a fusion network of the point cloud texture features and the point cloud structural features, receiving the point cloud texture features and the point cloud structural features at the same time, and fusing to obtain corresponding point cloud characteristic descriptions;

and designing a local characteristic loss function, wherein the loss function enables the distance between the image characteristic description and the point cloud characteristic description to be shortened according to the two-dimensional image negative sample and the three-dimensional point cloud negative sample sampled in the training process, and enables the distance between the image characteristic description and the three-dimensional point cloud negative sample and the distance between the point cloud characteristic description and the two-dimensional image negative sample to be distant.

5. The multi-view projection-based cross-dimensional data retrieval method of claim 4, wherein the texture perceptron is configured to process the point cloud multi-view projection image and to perceive texture information contained in the point cloud; the texture perceptron includes a convolutional network and a feature fusion function.

6. The multi-view projection-based cross-dimensional data retrieval method of claim 5, wherein the convolutional network is used for processing multi-angle projection to obtain n one-dimensional feature descriptions with fixed length d; the feature fusion function is a symmetric function insensitive to an input sequence, the one-dimensional feature description of n fixed lengths d is converted into one-dimensional feature description t of one fixed length d by the feature fusion function, and the one-dimensional feature description t of one fixed length d is used as a point cloud texture feature.

7. The method as claimed in claim 5, wherein the structure sensor is configured to process original point cloud data and sense structure information contained in the original point cloud data, wherein the structure sensor employs a feature extraction method based on a PointNet network structure to obtain a one-dimensional feature description s with a fixed length d, and the one-dimensional feature description s with the fixed length d is used as the point cloud structure feature.

8. A computer-readable storage medium, on which a multi-view projection-based cross-dimensional data retrieval program is stored, which, when executed by a processor, implements the multi-view projection-based cross-dimensional data retrieval method according to any one of claims 1 to 7.

9. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor, when executing the computer program, implements the multi-perspective projection based cross-dimensional data retrieval method according to any one of claims 1-7.

Images