CN113256793A - Three-dimensional data processing method and system - Google Patents
Three-dimensional data processing method and system Download PDFInfo
- Publication number
- CN113256793A CN113256793A CN202110599023.9A CN202110599023A CN113256793A CN 113256793 A CN113256793 A CN 113256793A CN 202110599023 A CN202110599023 A CN 202110599023A CN 113256793 A CN113256793 A CN 113256793A
- Authority
- CN
- China
- Prior art keywords
- dimensional
- point cloud
- network
- cloud data
- data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 26
- 238000012545 processing Methods 0.000 claims abstract description 65
- 230000011218 segmentation Effects 0.000 claims abstract description 38
- 238000000034 method Methods 0.000 claims abstract description 28
- 238000005070 sampling Methods 0.000 claims abstract description 11
- 238000010276 construction Methods 0.000 claims abstract description 9
- 238000007781 pre-processing Methods 0.000 claims description 16
- 238000007405 data analysis Methods 0.000 claims description 12
- 239000000284 extract Substances 0.000 claims description 6
- 238000010606 normalization Methods 0.000 claims description 6
- 238000013519 translation Methods 0.000 claims description 5
- 230000000903 blocking effect Effects 0.000 claims description 4
- 238000004891 communication Methods 0.000 claims description 4
- 238000000605 extraction Methods 0.000 abstract description 15
- 238000005259 measurement Methods 0.000 abstract description 14
- 238000012549 training Methods 0.000 description 27
- 230000008569 process Effects 0.000 description 9
- 230000000295 complement effect Effects 0.000 description 6
- 230000006870 function Effects 0.000 description 6
- 238000013528 artificial neural network Methods 0.000 description 5
- 238000010586 diagram Methods 0.000 description 5
- 238000013461 design Methods 0.000 description 4
- 238000004458 analytical method Methods 0.000 description 3
- 230000002035 prolonged effect Effects 0.000 description 3
- ORILYTVJVMAKLC-UHFFFAOYSA-N Adamantane Natural products C1C(C2)CC3CC1CC2C3 ORILYTVJVMAKLC-UHFFFAOYSA-N 0.000 description 2
- 230000009467 reduction Effects 0.000 description 2
- 238000011160 research Methods 0.000 description 2
- 238000013473 artificial intelligence Methods 0.000 description 1
- 238000013145 classification model Methods 0.000 description 1
- 238000004590 computer program Methods 0.000 description 1
- 238000013527 convolutional neural network Methods 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000018109 developmental process Effects 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
- G06T17/005—Tree description, e.g. octree, quadtree
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/60—Type of objects
- G06V20/64—Three-dimensional objects
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Multimedia (AREA)
- Computer Graphics (AREA)
- Geometry (AREA)
- Software Systems (AREA)
- Image Analysis (AREA)
Abstract
The invention relates to the technical field of image retrieval, in particular to a three-dimensional data processing method and a three-dimensional data processing system. Firstly, three-dimensional point cloud data of an image is obtained, local area set construction is carried out on the three-dimensional point cloud data, and features of the three-dimensional point cloud data in the local area set are extracted through a P o intn et + + network. And establishing a classification network and/or a segmentation network according to the extracted features, and analyzing the three-dimensional data of the image according to the classification network and/or the segmentation network. The problem of uneven sampling of the three-dimensional point cloud data can be solved by adopting the P o intn et + + network to extract the characteristics of the three-dimensional point cloud data. Meanwhile, the distance measurement between a point and a middle point in a three-dimensional space is also considered during extraction of the P o internet + + network, and local features can be learned by using the increase of context scale by using the distance of the measurement space, so that the network structure is more effective and more robust. By constructing a local area set of the three-dimensional point cloud data, the field query time of the P o intn et + + network can be greatly shortened, and the method has higher practicability and popularization value.
Description
[ technical field ] A method for producing a semiconductor device
The invention relates to the technical field of image retrieval, in particular to a three-dimensional data processing method and a three-dimensional data processing system.
[ background of the invention ]
With the rapid development of artificial intelligence, a great deal of research on one-dimensional data and two-dimensional data is currently performed, but the processing of three-dimensional data is still in the early stage of research.
Currently, stereo images are usually used to make the views more visually impactful, and when these stereo images are retrieved, analysis processing is usually required for three-dimensional data in the stereo images. The common processing method is to perform convolution on the voxelized three-dimensional data by adopting a point cloud segmentation method to complete segmentation, and then perform the next processing. However, the processing method has a large data volume and high time and space complexity, which results in a long segmentation time and a low accuracy, and cannot realize rapid and accurate processing of three-dimensional data.
[ summary of the invention ]
In order to solve the problem that the existing point cloud segmentation cannot realize rapid and accurate processing of three-dimensional data, the embodiment of the invention provides a three-dimensional data processing method and a three-dimensional data processing system.
In order to solve the above technical problem, an embodiment of the present invention provides a three-dimensional data processing method, which is used in image recognition and retrieval, and includes the following steps: acquiring three-dimensional point cloud data of an image; carrying out local region set construction on the three-dimensional point cloud data; extracting the characteristics of the three-dimensional point cloud data in the local area set through a Pointnet + + network; and establishing a classification network and/or a segmentation network according to the extracted features, and analyzing the three-dimensional data of the image according to the classification network and/or the segmentation network.
Preferably, the three-dimensional point cloud data is locally region set constructed by octree.
Preferably, the specific steps of constructing the local region set of the three-dimensional point cloud data through the octree are as follows: setting a maximum recursion depth; determining the maximum size of a scene in the three-dimensional data, and establishing a first cube according to the size; sequentially placing the unit elements in the first cube into a second cube without sub-nodes; the second cube is subdivided continuously until a set maximum recursion depth is reached.
Preferably, the specific steps of continuously subdividing the second cube are: the second cube is subdivided into eight third cubes and the unit cell elements in the second cube are assigned to the eight third cubes.
Preferably, after acquiring the three-dimensional point cloud data of the image, the method further comprises: preprocessing three-dimensional point cloud data; the preprocessing comprises blocking, sampling, translation and normalization processing.
Preferably, after extracting the three-dimensional point cloud data in the local area set, obtaining N scores from the characteristics of the three-dimensional point cloud data in the local area set through a full connection layer to establish a classification network; wherein, N scores correspond to N categories, and N is a natural number greater than zero.
Preferably, Dropout processing is performed on the output of the fully connected layer before N scores are obtained.
Preferably, the ratio of Dropout is 0.3 to 0.8.
In order to solve the above technical problems, the present invention provides another technical solution as follows: a three-dimensional data processing system is used for image recognition and retrieval, and comprises a data processing module and a data analysis module, wherein the data processing module is in communication connection with the data analysis module; the data processing module acquires three-dimensional point cloud data of an image, constructs a local area set of the three-dimensional point cloud data, and extracts the characteristics of the three-dimensional point cloud data in the local area set through a Pointernet + + network so as to establish a classification network and/or a segmentation network; the data analysis module analyzes the three-dimensional data of the image through a classification network and/or a segmentation network.
Preferably, the data processing module constructs a local region set of the three-dimensional point cloud data through an octree, specifically: setting a maximum recursion depth; determining the maximum size of a scene in the three-dimensional data, and establishing a first cube according to the size; sequentially placing the unit elements in the first cube into a second cube without sub-nodes; the second cube is subdivided continuously until a set maximum recursion depth is reached.
Compared with the prior art, the three-dimensional data processing method and the three-dimensional data processing system provided by the embodiment of the invention have the following advantages:
1. the density of the three-dimensional point cloud data acquired by the Pointernet + + network at different positions is different, and the multi-scale features can be combined in a self-adaptive manner, so that the problem of uneven sampling of the three-dimensional point cloud data can be solved by adopting the characteristics of the Pointernet + + network for extracting the three-dimensional point cloud data. Meanwhile, the distance measurement between a point and a middle point in a three-dimensional space is also considered in the extraction of the Pointnet + + network, and the local features can be learned by using the increase of context scale by using the distance of the measurement space, so that the network structure is more effective and more robust. Before the characteristics are extracted through the Pointnet + + network, the local area set of the three-dimensional point cloud data is constructed, so that the field query time of the Pointnet + + network can be greatly shortened, and the method has high practicability and popularization value.
2. The design carries out local area set construction on the three-dimensional point cloud data through the octree, the octree algorithm is simple to realize, the set operation of three-dimensional targets, such as intersection, combination, complement, difference and the like, can be rapidly carried out, and the search of the nearest area or point can be rapidly carried out. Therefore, a local area set is constructed through the octree, the feature extraction of the local area on a packet layer in the Pointernet + + network is completed, the classification accuracy of the Pointernet + + network can be improved, the time for extracting the three-dimensional point cloud data features by the Pointernet + + network is greatly shortened, and the three-dimensional data is efficiently and accurately processed.
3. Noise and foreign points in the three-dimensional point cloud data can be effectively removed by preprocessing the three-dimensional point cloud data, the three-dimensional point cloud data is simplified on the basis of keeping geometric characteristics, and the efficiency and the accuracy of processing the three-dimensional data are further improved.
4. According to the design, Dropout processing is carried out on the output of the full connection layer, so that the phenomenon of overfitting in the training process of the Pointnet + + network can be avoided, and the network running time is further prolonged. By preventing overfitting through Dropout, convergence time during network training can be reduced, and efficiency during processing of three-dimensional data is further improved.
5. The ratio of Dropout is set to be 0.3-0.8, and the Dropout ratio is limited in the interval, so that Dropout can randomly generate more network structures, and the occurrence of an overfitting phenomenon can be well reduced.
6. The data processing module in the three-dimensional data processing system builds a local area set of the three-dimensional point cloud data and extracts the characteristics of the three-dimensional point cloud data in the local area set through a Pointnet + + network so as to establish a classification network and/or a segmentation network. The density of three-dimensional point cloud data acquired by the Pointernet + + network at different positions is different, and multi-scale features can be combined in a self-adaptive manner, so that the problem of uneven sampling of the three-dimensional point cloud data can be solved by adopting the characteristics of the Pointernet + + network for extracting the three-dimensional point cloud data. Meanwhile, the distance measurement between a point and a middle point in a three-dimensional space is also considered in the extraction of the Pointnet + + network, and the local features can be learned by using the increase of context scale by using the distance of the measurement space, so that the network structure is more effective and more robust. Before the characteristics are extracted through the Pointnet + + network, the neighborhood query time of the Pointnet + + network can be greatly shortened through the constructed local area set, and the method has high practicability and popularization value.
7. According to the method, the octree is used for carrying out local area set construction on the three-dimensional point cloud data, the algorithm of the octree is simple to realize, the set operation of a three-dimensional target can be carried out rapidly, such as intersection, combination, complement, difference and the like, and the most adjacent area or point can be searched rapidly, so that the local area set is constructed by constructing the octree, the feature extraction of the local area on a packet layer in a Pointernet + + network is completed, the classification accuracy of the Pointernet + + network can be improved, the time for extracting the three-dimensional point cloud data features by the Pointernet + + network is greatly shortened, the three-dimensional data can be processed efficiently and accurately, and the method has high practicability and popularization value.
[ description of the drawings ]
Fig. 1 is a first schematic flowchart illustrating a three-dimensional data processing method according to a first embodiment of the present invention.
Fig. 2 is a schematic flow chart illustrating a step of a three-dimensional data processing method according to a first embodiment of the present invention.
Fig. 3 is a flowchart illustrating steps of constructing a local area set in a three-dimensional data processing method according to a first embodiment of the present invention.
Fig. 4 is a flowchart illustrating steps of establishing a classification network in a three-dimensional data processing method according to a first embodiment of the invention.
Fig. 5 is a functional block diagram of a three-dimensional data processing system according to a second embodiment of the present invention.
The attached drawings indicate the following:
1. a three-dimensional data processing system;
11. a data processing module; 12. and a data analysis module.
[ detailed description ] embodiments
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
Referring to fig. 1, a three-dimensional data processing method for image recognition and retrieval according to a first embodiment of the present invention includes the following steps:
acquiring three-dimensional point cloud data of an image;
carrying out local region set construction on the three-dimensional point cloud data;
extracting the characteristics of three-dimensional point cloud data concentrated in a local area through a Pointnet + + network;
and establishing a classification network and/or a segmentation network according to the extracted features, and analyzing the three-dimensional data of the image according to the classification network and/or the segmentation network.
It is understood that the three-dimensional data of the stereoscopic image has various expression forms such as point cloud, voxel, Mesh, and the like. When the three-dimensional data of the image is expressed by the point cloud, the three-dimensional point cloud data is a set of the three-dimensional data of the image in a three-dimensional coordinate system. When analyzing the three-dimensional data of the image, firstly, the three-dimensional point cloud data is obtained through a device for storing the three-dimensional data of the image, and the three-dimensional point cloud data is input into a Pointernet + + network for the next analysis. The Pointnet + + network can learn each point in the input three-dimensional point cloud data to obtain a corresponding spatial code, and then obtains a global point cloud feature by using the features of all the points.
It can be understood that the densities of the three-dimensional point cloud data acquired at different positions are different, and the Pointernet + + network can be adaptively combined with multi-scale features, so that the problem of uneven sampling of the three-dimensional point cloud data can be solved by adopting the Pointernet + + network to extract the features of the three-dimensional point cloud data in the embodiment of the invention. Meanwhile, the distance measurement between a point and a middle point in a three-dimensional space is also considered in the extraction of the Pointnet + + network, and the local features can be learned by using the increase of context scale by using the distance of the measurement space, so that the network structure is more effective and more robust.
Furthermore, because a large amount of point cloud data without topological relation is faced in the Pointernet + + network, the point cloud neighborhood search of the grouping layer is a time-consuming link, so that when the characteristics of the three-dimensional point cloud data are extracted through the Pointernet + + network, the three-dimensional point cloud data can be divided firstly to form an effective data storage structure.
As an implementation manner, in the embodiment of the present invention, a local area set of three-dimensional point cloud data is constructed through an octree, and then features of the three-dimensional point cloud data in the local area set are extracted through a pointent + + network. It will be appreciated that an octree, as a tree-like data structure describing a three-dimensional space, has eight child nodes each representing a cubic volume element, and that adding together the volume elements represented by the eight child nodes is equivalent to the volume of the parent node. Therefore, the three-dimensional space can be divided into several cubes by the octree, and the three-dimensional point cloud data are placed in different cubes, respectively. At this time, the set of the plurality of cubes is a local area set of the three-dimensional point cloud data. The Pointnet + + network can quickly find the position of a target point in the three-dimensional point cloud data through the local area set, and can quickly extract the characteristics of the three-dimensional point cloud data in the local area set through the Pointnet + + network. In addition, the local area set constructed by the octree can greatly shorten the field query time of Pointnet + +, the algorithm of the octree is simple to realize, the set operation of three-dimensional objects can be quickly carried out, such as intersection, union, complement, difference and the like, and the most adjacent area or point can also be quickly searched.
After the Pointnet + + finishes the feature extraction of the three-dimensional point cloud data, a classification network and/or a segmentation network is established according to the extracted features, and the three-dimensional data of the image is analyzed according to the classification network and/or the segmentation network. It can be understood that the classification network can obtain a trained classification model and output a classification result, and the segmentation network can obtain a trained segmentation model and output segmented point cloud data. When analyzing the three-dimensional data of the image according to the classification network and/or the segmentation network, the three-dimensional data of the currently acquired image may be analyzed, and the three-dimensional data of other input images may also be analyzed, which is not limited in the present invention.
Referring to fig. 2, as an embodiment, before inputting the three-dimensional point cloud data into the pointent + + network, the method further includes:
preprocessing three-dimensional point cloud data;
the preprocessing comprises blocking, sampling, translation, normalization processing and the like.
It can be understood that, in order to improve the accuracy of extracting the three-dimensional point cloud data features by the pointet + +, the three-dimensional point cloud data may be preprocessed before being input to the pointet + + network. Noise and foreign points in the three-dimensional point cloud data can be effectively removed by preprocessing the three-dimensional point cloud data, the three-dimensional point cloud data is simplified on the basis of keeping geometric characteristics, and the efficiency and the accuracy of processing the three-dimensional data are further improved. It is understood that, in other embodiments, the three-dimensional point cloud data may be directly input into the pointent + + network without performing a preprocessing operation on the three-dimensional point cloud data.
It can be understood that the specific formula for translating XYZ of the three-dimensional point cloud is as follows:
X＝X-Xmin，Y＝Y-Ymin，Z＝Z-Zmin
the specific formula for normalization processing of XYZ of the point cloud is as follows:
referring to fig. 3, as an embodiment, the specific steps of constructing the local region set of the three-dimensional point cloud data by using the octree include:
setting a maximum recursion depth;
determining the maximum size of a scene in the three-dimensional data, and establishing a first cube according to the size;
sequentially placing the unit elements in the first cube into a second cube without sub-nodes;
the second cube is subdivided continuously until a set maximum recursion depth is reached.
It can be understood that when constructing a local region set of three-dimensional point cloud data by using octree, the maximum recursion depth needs to be determined first, that is, the segmentation needs to be stopped when determining the degree to which the space is segmented. And then determining the maximum size of the scene in the three-dimensional data, establishing a first cube according to the size, wherein the established first cube is a cube space containing all unit elements. Wherein, the unit element is three-dimensional point cloud data. The unit-cell elements in the first cube are then placed sequentially within a second cube without child nodes. Wherein the second cube is a sub-cube divided on the first cube. If the maximum recursion depth is not reached, the second cube is continuously subdivided until the set maximum recursion depth is reached.
As an embodiment, the specific steps of continuously subdividing the second cube are:
the second cube is subdivided into eight third cubes and the unit cell elements in the second cube are assigned to the eight third cubes. If the maximum recursion depth has not been reached at this point, the subdivision of the third cube continues.
It is understood that a cube may be divided into eight equally sized subcubes, and in the embodiment of the present invention, eight third cubes are divided into one subcube by the second cube. By dividing the second cube into eight third cubes, the unit cell elements in the second cube can be further assigned to different third cubes, and further subdivision of the unit cell elements is achieved.
Further, if the number of unit elements in any third cube is not zero and is equal to the number of unit elements in the second cube, the subdivision of the third cube is stopped. It can be understood that, according to the space division theory, the number of unit element elements allocated to the subdivided space is necessarily small, and if the number of unit element elements allocated to the subdivided space is consistent with the number of unit element elements in the space before subdivision, the subdivided space is divided into a plurality of spaces, and the number of unit element elements in the subdivided space remains unchanged. According to the embodiment of the invention, when the number of the unit elements in any third cube is not zero and is equal to that of the unit elements in the second cube, the subdivision of the third cube is stopped, so that the situation that the constructed octree is infinitely subdivided can be ensured, and the working efficiency in constructing the octree is further improved.
Referring to fig. 4, as an embodiment, when a classification network is established, the characteristics of three-dimensional point cloud data concentrated in a local area are extracted through a pointet + + network, and N scores are obtained through a full connection layer. Wherein, N scores correspond to N categories, and N is a natural number greater than zero.
It will be appreciated that the fully connected layers (FC) act as "classifiers" throughout the convolutional neural network, which can map the learned "distributed feature representation" into the sample label space. After three-dimensional point cloud data in a local area set are extracted through a Pointnet + + network, the extracted data are trained and classified through a full connection layer, N scores can be obtained, and then the establishment of a classification network is achieved.
Further, as an implementation manner, in order to avoid that the network running time is long due to overfitting of the pointet + + network in the training process, in the embodiment of the present invention, before N scores are obtained, Dropout processing is performed on the output of the full connection layer.
It can be understood that overfitting is easily generated during the pointet + + training process. While Dropout temporarily discards neural network units from the network with a certain probability during the training process of the deep learning network, the overfitting phenomenon can be obviously reduced by neglecting some neural network units. According to the embodiment of the invention, Dropout processing is carried out on the output of the full connection layer, so that the phenomenon of overfitting in the training process of the Pointnet + + network can be avoided, and the network running time is further prolonged. Meanwhile, overfitting is prevented through Dropout, convergence time during network training can be shortened, and efficiency of processing three-dimensional data is further improved.
In the embodiment of the present invention, the ratio of Dropout when Dropout processing is performed on the output of the all-connected layer is not particularly limited, but is preferably 0.3 to 0.8. In particular, the ratio Dropout for the inventive examples is 0.3, 0.4, 0.5, 0.6, 0.7 or 0.8. According to the embodiment of the invention, the Dropout ratio is limited in the interval, so that Dropout can randomly generate more network structures, and the occurrence of the overfitting phenomenon can be well reduced.
As another embodiment, in establishing the segmentation network, feature transfer is performed by stacking the MLP with a linear difference. When the characteristics of the points of the corresponding levels are spliced, the following formula is adopted:
wherein x isi∈R3As the point cloud coordinates of the previous level, x ∈ R3Is the point coordinate of the last sample set,
in order to be a feature of the interpolation,
is the point cloud feature of the previous level.
Furthermore, a trained model can be obtained by training the point cloud data, and other point cloud data can be analyzed through the trained model. Specifically, the whole neural network is trained through the Pythrch during training, the batch training network with the size of 32 is used, and the training parameters are reserved to serve as a pre-training model to carry out parameter initialization on the network. And an ADAM algorithm with a driving quantity parameter of 0.9 is adopted to optimize a loss function of the whole system network, so that the loss is reduced to the minimum value of the network. During training, the learning rate is changed to carry out learning, the initial learning rate is 0.001, the learning rate is reduced to 0.5 time of the original learning rate every 20 training cycles, and the reduction is stopped until the learning rate is less than 0.0001. And an early stopping strategy is adopted during training, and 200 periods are trained in total.
Specifically, nllls is used for convergence in training the network and log _ softmax is used in the last layer of the network, where the formula of log _ softmax is as follows:
after the training is finished, the cluster merging module is used for deleting, denoising and de-duplicating the segmentation example combination so as to segment the complete example object.
It is understood that the three-dimensional data of the image can be analyzed in the above manner. The embodiment of the present invention is illustrated by an indoor scene, but is not limited to processing three-dimensional data of the indoor scene. Specifically, when the three-dimensional data of the indoor scene is processed, the three-dimensional data of the indoor scene is set to be a 3D scanned image including 6 scene areas and 271 rooms, the three-dimensional point cloud data is a set of data in a three-dimensional coordinate system of the indoor scene, and each point cloud data in the three-dimensional data of the indoor scene is set to be one of 13 semantic tags (tables, chairs, walls, floors, sundries and the like). When three-dimensional data of an indoor scene is processed, 4096 points are randomly extracted from each block as point cloud data, and then preprocessing operations are performed on the point cloud data, wherein the preprocessing operations comprise translation and normalization. Inputting the point cloud data into a Pointnet + + neural network, selecting a central point of each region in the point cloud data through FPS (frames Per second), constructing a local region set through an octree by using the central point, introducing the octree in a grouping layer for field query, constructing a point cloud index structure, quickly creating sub-point clouds, and then extracting features through Pointnet + +. When creating the classification network, Dropout processing of 0.5 is performed on the output of the fully-connected layer, and 13 scores can be obtained through the fully-connected layer. Wherein 13 scores correspond to 13 categories. Further, a trained model can be obtained by training point cloud data, the whole neural network is trained through a Pythrch during training, a 32-sized batch training network is used, and training parameters are reserved to serve as a pre-training model to initialize parameters of the network. And an ADAM algorithm with a driving quantity parameter of 0.9 is adopted to optimize a loss function of the whole system network, so that the loss is reduced to the minimum value of the network. During training, the learning rate is changed to carry out learning, the initial learning rate is 0.001, the learning rate is reduced to 0.5 time of the original learning rate every 20 training cycles, and the reduction is stopped until the learning rate is less than 0.0001. And an early stopping strategy is adopted during training, and 200 periods are trained. The three-dimensional data of other indoor scenes can be analyzed through the trained model, and the analysis result is predicted.
In summary, in the three-dimensional data processing method provided in the first embodiment of the present invention, when processing three-dimensional data, first three-dimensional point cloud data is obtained, and the three-dimensional point cloud data is preprocessed. And then inputting the three-dimensional point cloud data into a Pointnet + + network, constructing a local area set of the three-dimensional point cloud data through an octree, and extracting the characteristics of the three-dimensional point cloud data in the local area set through the Pointnet + + network. And establishing a classification network and/or a segmentation network according to the extracted features, and analyzing the three-dimensional data according to the classification network and/or the segmentation network. According to the embodiment of the invention, before the characteristics of the three-dimensional point cloud data are extracted by the Pointernet + +, the local area set of the three-dimensional point cloud data is constructed by the octree, and the octree can quickly perform the set operation of the three-dimensional target and the search of the nearest area or point, so that the local area set is constructed by the octree, the characteristic extraction of the local area on the Pointernet + + packet layer is completed, the classification accuracy of the Pointernet + + network can be improved, the time for extracting the characteristics of the three-dimensional point cloud data by the Pointernet + + is greatly shortened, the three-dimensional data can be efficiently and accurately processed, and the method has higher practicability and popularization value.
Referring to fig. 5, a three-dimensional data processing system 1 is provided in a second embodiment of the present invention, which implements processing of three-dimensional data by using the three-dimensional data processing method in the first embodiment. The three-dimensional data processing system 1 comprises a data processing module 11 and a data analysis module 12, wherein the data processing module 11 is in communication connection with the data analysis module 12.
Specifically, the data processing module 11 is in communication connection with a device in which three-dimensional point cloud data is stored to acquire the three-dimensional point cloud data, and inputs the three-dimensional point cloud data into a pointent + + network. And then constructing a local area set of the three-dimensional point cloud data through the octree, and extracting the characteristics of the three-dimensional point cloud data in the local area set through a Pointnet + + network so as to establish a classification network and/or a segmentation network. It can be understood that the data processing module 11 stores a pointent + + network and related software for constructing an octree. After the classification network and/or the segmentation network is established, the data analysis module 12 obtains the classification network and/or the segmentation network, and analyzes the three-dimensional data through the classification network and/or the segmentation network. It can be understood that, when analyzing the three-dimensional data according to the classification network and/or the segmentation network, the data analysis module 12 may analyze the currently acquired three-dimensional data, or may analyze other input three-dimensional data, which is not limited in the present invention.
The data processing module 11 of the embodiment of the invention constructs a local area set of three-dimensional point cloud data through an octree, and extracts the characteristics of the three-dimensional point cloud data in the local area set through a Pointnet + + network so as to establish a classification network and/or a segmentation network. It can be understood that the densities of the three-dimensional point cloud data collected at different positions are different, and the problem of uneven sampling of the three-dimensional point cloud data can be solved by adopting the characteristics of the three-dimensional point cloud data extracted by the Pointernet + + network because the Pointernet + + network can adaptively combine with multi-scale characteristics. Meanwhile, the distance measurement between a point and a middle point in a three-dimensional space is also considered in the extraction of the Pointnet + + network, and the local features can be learned by using the increase of context scale by using the distance of the measurement space, so that the network structure is more effective and more robust. Before the characteristics are extracted through the Pointnet + + network, a local area set of three-dimensional point cloud data is constructed through the octree, and the neighborhood query time of the Pointnet + + network can be greatly shortened through the local area set constructed through the octree. The octree algorithm is simple to implement, can quickly perform set operation of three-dimensional objects, such as intersection, union, complement, difference and the like, and can also quickly perform search of nearest regions or points, so that a local region set is constructed by constructing the octree, the feature extraction of local regions on a Pointernet + + grouping layer is completed, the classification accuracy of the Pointernet + + network can be improved, the time for extracting three-dimensional point cloud data features by Pointernet + + is greatly shortened, three-dimensional data can be efficiently and accurately processed, and the method has high practicability and popularization value.
Further, the data processing module 11 is configured to perform preprocessing on the three-dimensional point cloud data before inputting the three-dimensional point cloud data into the pointent + + network. The preprocessing comprises blocking, sampling, translation, normalization processing and the like. It can be understood that, in order to improve the accuracy of extracting the three-dimensional point cloud data features by the pointet + + network, the three-dimensional point cloud data may be preprocessed before being input to the pointet + + network. The data processing module 11 is used for preprocessing the three-dimensional point cloud data, so that noise and foreign points in the three-dimensional point cloud data can be effectively eliminated, the three-dimensional point cloud data is simplified on the basis of keeping geometric characteristics, and the efficiency and the accuracy of the system 1 in processing the three-dimensional data are further improved. It is understood that, in other embodiments, the data processing module 11 may also directly input the three-dimensional point cloud data into the pointet + + network without performing a preprocessing operation on the three-dimensional point cloud data.
Further, the way of constructing the local area set of the three-dimensional point cloud data by the data processing module 11 through the octree specifically is as follows:
setting a maximum recursion depth;
determining the maximum size of a scene in the three-dimensional data, and establishing a first cube according to the size;
sequentially placing the unit elements in the first cube into a second cube without sub-nodes;
the second cube is subdivided continuously until a set maximum recursion depth is reached.
In summary, when analyzing three-dimensional data, the three-dimensional data processing system 1 according to the second embodiment of the present invention first obtains the three-dimensional point cloud data by communicating with a device storing the three-dimensional point cloud data through the data processing module 11, and inputs the three-dimensional point cloud data into the pointent + + network. And then constructing a local area set of the three-dimensional point cloud data through the octree, and extracting the characteristics of the three-dimensional point cloud data in the local area set through a Pointnet + + network so as to establish a classification network and/or a segmentation network. After the classification network and/or the segmentation network is established, the data analysis module 12 obtains the classification network and/or the segmentation network, and analyzes the three-dimensional data through the classification network and/or the segmentation network. According to the embodiment of the invention, before the characteristics of the three-dimensional point cloud data are extracted by the Pointernet + +, the local area set of the three-dimensional point cloud data is constructed by the octree, and the octree can quickly perform the set operation of the three-dimensional target and the search of the nearest area or point, so that the local area set is constructed by the octree, the characteristic extraction of the local area on the Pointernet + + packet layer is completed, the classification accuracy of the Pointernet + + network can be improved, the time for extracting the characteristics of the three-dimensional point cloud data by the Pointernet + + is greatly shortened, the three-dimensional data can be efficiently and accurately processed, and the method has higher practicability and popularization value.
In the embodiments provided herein, it should be understood that "B corresponding to a" means that B is associated with a from which B can be determined. It should also be understood, however, that determining B from a does not mean determining B from a alone, but may also be determined from a and/or other information.
It should be appreciated that reference throughout this specification to "one embodiment" or "an embodiment" means that a particular feature, structure or characteristic described in connection with the embodiment is included in at least one embodiment of the present invention. Thus, the appearances of the phrases "in one embodiment" or "in an embodiment" in various places throughout this specification are not necessarily all referring to the same embodiment. Furthermore, the particular features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. Those skilled in the art should also appreciate that the embodiments described in this specification are exemplary and alternative embodiments, and that the acts and modules illustrated are not required in order to practice the invention.
In various embodiments of the present invention, it should be understood that the sequence numbers of the above-mentioned processes do not imply an inevitable order of execution, and the execution order of the processes should be determined by their functions and inherent logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.
The flowchart and block diagrams in the figures of the present application illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present application. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will be understood that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
Compared with the prior art, the three-dimensional data processing method and the three-dimensional data processing system provided by the embodiment of the invention have the following advantages:
1. the density of the three-dimensional point cloud data acquired by the Pointernet + + network at different positions is different, and the multi-scale features can be combined in a self-adaptive manner, so that the problem of uneven sampling of the three-dimensional point cloud data can be solved by adopting the characteristics of the Pointernet + + network for extracting the three-dimensional point cloud data. Meanwhile, the distance measurement between a point and a middle point in a three-dimensional space is also considered in the extraction of the Pointnet + + network, and the local features can be learned by using the increase of context scale by using the distance of the measurement space, so that the network structure is more effective and more robust. Before the characteristics are extracted through the Pointnet + + network, the local area set of the three-dimensional point cloud data is constructed, so that the field query time of the Pointnet + + network can be greatly shortened, and the method has high practicability and popularization value.
2. The design carries out local area set construction on the three-dimensional point cloud data through the octree, the octree algorithm is simple to realize, the set operation of three-dimensional targets, such as intersection, combination, complement, difference and the like, can be rapidly carried out, and the search of the nearest area or point can be rapidly carried out. Therefore, a local area set is constructed through the octree, the feature extraction of the local area on a packet layer in the Pointernet + + network is completed, the classification accuracy of the Pointernet + + network can be improved, the time for extracting the three-dimensional point cloud data features by the Pointernet + + network is greatly shortened, and the three-dimensional data is efficiently and accurately processed.
3. Noise and foreign points in the three-dimensional point cloud data can be effectively removed by preprocessing the three-dimensional point cloud data, the three-dimensional point cloud data is simplified on the basis of keeping geometric characteristics, and the efficiency and the accuracy of processing the three-dimensional data are further improved.
4. According to the design, Dropout processing is carried out on the output of the full connection layer, so that the phenomenon of overfitting in the training process of the Pointnet + + network can be avoided, and the network running time is further prolonged. By preventing overfitting through Dropout, convergence time during network training can be reduced, and efficiency during processing of three-dimensional data is further improved.
5. The ratio of Dropout is set to be 0.3-0.8, and the Dropout ratio is limited in the interval, so that Dropout can randomly generate more network structures, and the occurrence of an overfitting phenomenon can be well reduced.
6. The data processing module in the three-dimensional data processing system builds a local area set of the three-dimensional point cloud data and extracts the characteristics of the three-dimensional point cloud data in the local area set through a Pointnet + + network so as to establish a classification network and/or a segmentation network. The density of three-dimensional point cloud data acquired by the Pointernet + + network at different positions is different, and multi-scale features can be combined in a self-adaptive manner, so that the problem of uneven sampling of the three-dimensional point cloud data can be solved by adopting the characteristics of the Pointernet + + network for extracting the three-dimensional point cloud data. Meanwhile, the distance measurement between a point and a middle point in a three-dimensional space is also considered in the extraction of the Pointnet + + network, and the local features can be learned by using the increase of context scale by using the distance of the measurement space, so that the network structure is more effective and more robust. Before the characteristics are extracted through the Pointnet + + network, the neighborhood query time of the Pointnet + + network can be greatly shortened through the constructed local area set, and the method has high practicability and popularization value.
7. According to the method, the octree is used for carrying out local area set construction on the three-dimensional point cloud data, the algorithm of the octree is simple to realize, the set operation of a three-dimensional target can be carried out rapidly, such as intersection, combination, complement, difference and the like, and the most adjacent area or point can be searched rapidly, so that the local area set is constructed by constructing the octree, the feature extraction of the local area on a packet layer in a Pointernet + + network is completed, the classification accuracy of the Pointernet + + network can be improved, the time for extracting the three-dimensional point cloud data features by the Pointernet + + network is greatly shortened, the three-dimensional data can be processed efficiently and accurately, and the method has high practicability and popularization value.
The three-dimensional data processing method and system disclosed by the embodiment of the invention are described in detail, a specific example is applied in the text to explain the principle and the implementation of the invention, and the description of the embodiment is only used for helping to understand the method and the core idea of the invention; meanwhile, for the persons skilled in the art, according to the idea of the present invention, there may be variations in the specific embodiments and the application scope, and in summary, the content of the present description should not be construed as a limitation to the present invention, and any modification, equivalent replacement, and improvement made within the principle of the present invention should be included in the protection scope of the present invention.
Claims (10)
1. A three-dimensional data processing method used in image recognition and retrieval, characterized in that: the method comprises the following steps:
acquiring three-dimensional point cloud data of an image;
carrying out local region set construction on the three-dimensional point cloud data;
extracting the characteristics of the three-dimensional point cloud data in the local area set through a Pointnet + + network;
and establishing a classification network and/or a segmentation network according to the extracted features, and analyzing the three-dimensional data of the image according to the classification network and/or the segmentation network.
2. The three-dimensional data processing method according to claim 1, characterized in that: and carrying out local region set construction on the three-dimensional point cloud data through an octree.
3. The three-dimensional data processing method according to claim 2, characterized in that: the specific steps of constructing the local area set of the three-dimensional point cloud data through the octree are as follows:
setting a maximum recursion depth;
determining the maximum size of a scene in the three-dimensional data, and establishing a first cube according to the size;
sequentially placing the unit elements in the first cube into a second cube without sub-nodes;
the second cube is subdivided continuously until a set maximum recursion depth is reached.
4. The three-dimensional data processing method according to claim 3, characterized in that: the specific steps for continuously subdividing the second cube are as follows:
the second cube is subdivided into eight third cubes and the unit cell elements in the second cube are assigned to the eight third cubes.
5. The three-dimensional data processing method according to claim 1, characterized in that: after the three-dimensional point cloud data of the image is acquired, the method further comprises the following steps:
preprocessing three-dimensional point cloud data;
the preprocessing comprises blocking, sampling, translation and normalization processing.
6. The three-dimensional data processing method according to claim 1, characterized in that: after extracting the three-dimensional point cloud data in the local area set, obtaining N scores from the characteristics of the three-dimensional point cloud data in the local area set through a full connection layer to establish a classification network; wherein, N scores correspond to N categories, and N is a natural number greater than zero.
7. The three-dimensional data processing method according to claim 6, characterized in that: before obtaining the N scores, Dropout processing is performed on the output of the fully connected layer.
8. The three-dimensional data processing method according to claim 7, characterized in that: the ratio of Dropout is 0.3 to 0.8.
9. A three-dimensional data processing system for use in image recognition and retrieval, comprising: the three-dimensional data processing system comprises a data processing module and a data analysis module, and the data processing module is in communication connection with the data analysis module;
the data processing module acquires three-dimensional point cloud data of an image, constructs a local area set of the three-dimensional point cloud data, and extracts the characteristics of the three-dimensional point cloud data in the local area set through a Pointernet + + network so as to establish a classification network and/or a segmentation network;
the data analysis module analyzes the three-dimensional data of the image through a classification network and/or a segmentation network.
10. The three-dimensional data processing system of claim 9, wherein: the data processing module constructs a local area set of the three-dimensional point cloud data through an octree, and specifically comprises the following steps:
setting a maximum recursion depth;
determining the maximum size of a scene in the three-dimensional data, and establishing a first cube according to the size;
sequentially placing the unit elements in the first cube into a second cube without sub-nodes;
the second cube is subdivided continuously until a set maximum recursion depth is reached.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110599023.9A CN113256793A (en) | 2021-05-31 | 2021-05-31 | Three-dimensional data processing method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110599023.9A CN113256793A (en) | 2021-05-31 | 2021-05-31 | Three-dimensional data processing method and system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113256793A true CN113256793A (en) | 2021-08-13 |
Family
ID=77183811
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110599023.9A Pending CN113256793A (en) | 2021-05-31 | 2021-05-31 | Three-dimensional data processing method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113256793A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113724270A (en) * | 2021-08-25 | 2021-11-30 | 华南农业大学 | Intelligent segmentation method and system for point cloud on surface of livestock |
| CN114004934A (en) * | 2021-11-02 | 2022-02-01 | 国网浙江省电力有限公司湖州供电公司 | Power transmission line point cloud classification method based on grouping batch normalization |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110246112A (en) * | 2019-01-21 | 2019-09-17 | 厦门大学 | Three-dimensional point cloud quality evaluating method in the room laser scanning SLAM based on deep learning |
| CN111444927A (en) * | 2020-03-26 | 2020-07-24 | 广州市炜城智能科技有限公司 | Method for identifying animal and plant pests by using laser technology in multidimensional manner |
| WO2020199834A1 (en) * | 2019-04-03 | 2020-10-08 | 腾讯科技（深圳）有限公司 | Object detection method and apparatus, and network device and storage medium |
| CN111798475A (en) * | 2020-05-29 | 2020-10-20 | 浙江工业大学 | Indoor environment 3D semantic map construction method based on point cloud deep learning |
| CN111862101A (en) * | 2020-07-15 | 2020-10-30 | 西安交通大学 | 3D point cloud semantic segmentation method under aerial view coding visual angle |
| CN112396703A (en) * | 2020-11-18 | 2021-02-23 | 北京工商大学 | Single-image three-dimensional point cloud model reconstruction method |
| CN112819135A (en) * | 2020-12-21 | 2021-05-18 | 中国矿业大学 | Sorting method for guiding mechanical arm to grab materials in different poses based on ConvPoint model |
-
2021
- 2021-05-31 CN CN202110599023.9A patent/CN113256793A/en active Pending
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110246112A (en) * | 2019-01-21 | 2019-09-17 | 厦门大学 | Three-dimensional point cloud quality evaluating method in the room laser scanning SLAM based on deep learning |
| WO2020199834A1 (en) * | 2019-04-03 | 2020-10-08 | 腾讯科技（深圳）有限公司 | Object detection method and apparatus, and network device and storage medium |
| CN111444927A (en) * | 2020-03-26 | 2020-07-24 | 广州市炜城智能科技有限公司 | Method for identifying animal and plant pests by using laser technology in multidimensional manner |
| CN111798475A (en) * | 2020-05-29 | 2020-10-20 | 浙江工业大学 | Indoor environment 3D semantic map construction method based on point cloud deep learning |
| CN111862101A (en) * | 2020-07-15 | 2020-10-30 | 西安交通大学 | 3D point cloud semantic segmentation method under aerial view coding visual angle |
| CN112396703A (en) * | 2020-11-18 | 2021-02-23 | 北京工商大学 | Single-image three-dimensional point cloud model reconstruction method |
| CN112819135A (en) * | 2020-12-21 | 2021-05-18 | 中国矿业大学 | Sorting method for guiding mechanical arm to grab materials in different poses based on ConvPoint model |
Non-Patent Citations (1)
| Title |
|---|
| CHARLES R.QI: "PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space", 《ARXIV》, pages 106 - 107 * |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113724270A (en) * | 2021-08-25 | 2021-11-30 | 华南农业大学 | Intelligent segmentation method and system for point cloud on surface of livestock |
| CN113724270B (en) * | 2021-08-25 | 2023-08-08 | 华南农业大学 | Intelligent segmentation method and system for livestock surface point cloud |
| CN114004934A (en) * | 2021-11-02 | 2022-02-01 | 国网浙江省电力有限公司湖州供电公司 | Power transmission line point cloud classification method based on grouping batch normalization |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Ye et al. | 3d recurrent neural networks with context fusion for point cloud semantic segmentation | |
| CN110660062B (en) | Point cloud instance segmentation method and system based on PointNet | |
| CN110827398B (en) | Automatic semantic segmentation method for indoor three-dimensional point cloud based on deep neural network | |
| CN110210431B (en) | Point cloud semantic labeling and optimization-based point cloud classification method | |
| CN114926699B (en) | Indoor three-dimensional point cloud semantic classification method, device, medium and terminal | |
| CN112802204B (en) | Target semantic navigation method and system for three-dimensional space scene prior in unknown environment | |
| CN111753698A (en) | Multi-mode three-dimensional point cloud segmentation system and method | |
| CN114638954B (en) | Training method of point cloud segmentation model, point cloud data segmentation method and related device | |
| CN106780639B (en) | Hash coding method based on significance characteristic sparse embedding and extreme learning machine | |
| CN113256793A (en) | Three-dimensional data processing method and system | |
| CN111125397B (en) | Cloth image retrieval method based on convolutional neural network | |
| CN113487523B (en) | Method and device for optimizing graph contour, computer equipment and storage medium | |
| CN113988147B (en) | Multi-label classification method and device for remote sensing image scene based on graph network, and multi-label retrieval method and device | |
| CN111524140B (en) | Medical image semantic segmentation method based on CNN and random forest method | |
| Phalak et al. | Scan2plan: Efficient floorplan generation from 3d scans of indoor scenes | |
| CN113139967A (en) | Point cloud instance segmentation method, related system and storage medium | |
| CN113570573A (en) | Pulmonary nodule false positive eliminating method, system and equipment based on mixed attention mechanism | |
| CN109300149A (en) | The three-dimensional image registration method optimized based on gravity treatment point strategy and artificial bee colony | |
| CN112257810A (en) | Submarine biological target detection method based on improved FasterR-CNN | |
| CN115115847B (en) | Three-dimensional sparse reconstruction method and device and electronic device | |
| CN114638953B (en) | Point cloud data segmentation method and device and computer readable storage medium | |
| CN116977265A (en) | Training method and device for defect detection model, computer equipment and storage medium | |
| CN115544811A (en) | Data management method applied to digital twin model | |
| CN111986223B (en) | Method for extracting trees in outdoor point cloud scene based on energy function | |
| CN115937507A (en) | Point cloud semantic segmentation method based on point void direction convolution |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WD01 | Invention patent application deemed withdrawn after publication | ||
| WD01 | Invention patent application deemed withdrawn after publication |
Application publication date: 20210813 |