CN117315146B - Reconstruction method and storage method of three-dimensional model based on trans-scale multi-source data - Google Patents
Reconstruction method and storage method of three-dimensional model based on trans-scale multi-source data Download PDFInfo
- Publication number
- CN117315146B CN117315146B CN202311236479.4A CN202311236479A CN117315146B CN 117315146 B CN117315146 B CN 117315146B CN 202311236479 A CN202311236479 A CN 202311236479A CN 117315146 B CN117315146 B CN 117315146B
- Authority
- CN
- China
- Prior art keywords
- data
- point cloud
- optimization
- image
- modeling
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 72
- 238000003860 storage Methods 0.000 title claims abstract description 32
- 238000005457 optimization Methods 0.000 claims abstract description 79
- 230000004927 fusion Effects 0.000 claims abstract description 28
- 230000000750 progressive effect Effects 0.000 claims abstract description 12
- 238000005259 measurement Methods 0.000 claims abstract description 5
- 238000004364 calculation method Methods 0.000 claims description 15
- 230000011218 segmentation Effects 0.000 claims description 10
- 238000002372 labelling Methods 0.000 claims description 6
- 238000000605 extraction Methods 0.000 claims description 5
- 238000013507 mapping Methods 0.000 claims description 5
- 238000012545 processing Methods 0.000 claims description 5
- 238000005070 sampling Methods 0.000 claims description 5
- 239000007787 solid Substances 0.000 claims description 4
- 238000013500 data storage Methods 0.000 claims description 3
- 238000013528 artificial neural network Methods 0.000 description 5
- 239000013598 vector Substances 0.000 description 4
- 230000007547 defect Effects 0.000 description 3
- 238000010586 diagram Methods 0.000 description 3
- 230000011514 reflex Effects 0.000 description 3
- XLYOFNOQVPJJNP-UHFFFAOYSA-N water Substances O XLYOFNOQVPJJNP-UHFFFAOYSA-N 0.000 description 3
- 238000007726 management method Methods 0.000 description 2
- 238000004519 manufacturing process Methods 0.000 description 2
- 238000004458 analytical method Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 238000013480 data collection Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000003384 imaging method Methods 0.000 description 1
- 238000002360 preparation method Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T15/00—3D [Three Dimensional] image rendering
- G06T15/005—General purpose rendering architectures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2200/00—Indexing scheme for image data processing or generation, in general
- G06T2200/08—Indexing scheme for image data processing or generation, in general involving all processing steps from image acquisition to 3D model generation
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02T—CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION
- Y02T10/00—Road transport of goods or passengers
- Y02T10/10—Internal combustion engine [ICE] based vehicles
- Y02T10/40—Engine management systems
Abstract
The invention discloses a three-dimensional model reconstruction method, namely a storage method, based on trans-scale multi-source data, which comprises the following steps: a cross-scale measurement control network is arranged on a modeling object to acquire coordinate information of each scale related point in the modeling object, and different image acquisition devices are adopted to acquire image data of different scale scenes of the modeling object; according to the coordinate information in the step 1, adopting a progressive space three-solution method to perform space three-solution on the obtained image data of different scale scenes, and then performing fusion optimization on the space three-solution point cloud data to obtain fusion optimization point cloud data of a modeling object; and (3) modeling the modeling object by adopting a semantic optimization modeling method according to the fusion optimization point cloud data obtained in the step (2) to obtain a semantic annotation three-dimensional model. The invention solves the problem of three failures in cross-scale and indoor and outdoor modeling, and the three-dimensional model constructed by the semantic optimization modeling method has higher quality.
Description
Technical Field
The invention belongs to the technical field of three-dimensional model reconstruction, and particularly relates to a three-dimensional model reconstruction method and a storage method based on trans-scale multi-source data.
Background
Through three-dimensional modeling technologies such as photogrammetry and laser scanning, real-scene three-dimensional reconstruction of important scenes such as urban scenes, cultural heritage, archaeological exploration parties and the like can be realized. The high-precision geometric information and high-reality texture information of a scene can be recorded in an omnibearing manner by the three-dimensional reconstruction of the real scene, and great importance is placed on various industries. With the popularity of live-action three-dimensional modeling, many industry applications place more demands on it. In practical applications, many scenes need to pay attention to modeling objects in kilometers, hundred meters, meters and millimeters at the same time, and even more scenes need to pay attention to modeling objects outdoors and indoors at the same time. Live-action three-dimensional modeling generally requires that all modeled objects have real geographic coordinates or real dimensions, otherwise the value and usability of the model are greatly reduced. In such a context, implementing three-dimensional modeling faces a number of challenges, including mainly: (1) Different data acquisition modes are applicable to different scenes, and how to effectively fuse various data sources; (2) The control point layout is carried out under the condition that no RTK signal exists indoors; (3) The space three solution is difficult for the joint modeling of the cross-scale and the indoor and outdoor; (4) The indoor model and the outdoor model occupy the same coordinates and model storage conflicts caused by the same tile.
The existing live-action three-dimensional modeling uses different data acquisition modes for different areas, and one set of data completes one-time modeling. For example, kilometer level scenes use unmanned aerial vehicles to collect data, indoor scenes use single-lens reflex cameras, and millimeter level employ hand-held laser scanners to collect data. The operation mode leads to quite scattered modeling results related to trans-scale or indoor and outdoor scenes, uneven model quality in different areas, incapability of uniformly browsing and measuring, and incapability of generating uniform 4D products.
In this case, even if the models of the respective areas are converted into a uniform coordinate system, there are many problems of edge confusion and seams between the models, which have serious influence on the value, usability and aesthetic degree of the models. In particular, if one item realizes outdoor and indoor modeling at the same time, after the model is converted to a unified coordinate system, coordinate conflict and tile conflict can occur to the outdoor model and the indoor model, and in particular, the tile conflict can cause errors in model storage, browsing and retrieval.
Disclosure of Invention
The invention aims to provide a three-dimensional model reconstruction method based on cross-scale multi-source data, aiming at the defects of the prior art, and the method solves the problem that the cross-scale multi-source data is difficult to fuse in the three-dimensional model reconstruction process.
In order to solve the technical problems, the invention adopts the following technical scheme:
a reconstruction method of a three-dimensional model based on trans-scale multi-source data comprises the following steps:
step 1, laying a cross-scale measurement control network on a modeling object to obtain coordinate information of each scale related point in the modeling object, and acquiring image data of different scale scenes of the modeling object by adopting different image acquisition equipment;
step 2, according to the coordinate information in the step 1, adopting a progressive null three-resolving method to respectively perform null three-resolving on the obtained image data of different scale scenes, and then performing fusion optimization on the null three-resolved point cloud data to obtain fusion optimized point cloud data of a modeling object;
and step 3, modeling the modeling object by adopting a semantic optimization modeling method according to the fusion optimization point cloud data obtained in the step 2 to obtain a semantic annotation three-dimensional model.
Further, in step 1, unmanned aerial vehicles are adopted to carry multiple lenses to collect inclined images for outdoor scenes of modeling objects, and handheld single-lens reflex is adopted to collect multi-angle images for indoor scenes.
In step 1, an unmanned aerial vehicle is used for carrying an airborne laser radar to collect point cloud data of an outdoor scene, and a standing laser scanner and a handheld laser scanner are used for collecting laser radar point cloud of an indoor scene.
Further, the conditions satisfied by the image data collected in the step 1 are:
when different image acquisition devices are adopted for acquisition, intersection sets are defined for acquisition ranges of two different image acquisition image devices in adjacent areas to be acquired together, data in the intersection sets are used as buffer data, and the area of the buffer data is not less than 10% of the acquisition area of one device;
collecting targets with different scales of outdoor scenes at different altitudes, and if the altitude difference exceeds 150 meters, collecting data of the minimum-scale scene in the outdoor scenes by using the middle altitude as buffer data, wherein the area of the buffer data completely covers the minimum-scale scene;
the method comprises the steps of collecting targets with different scales of indoor and outdoor scenes, collecting partial indoor scenes by using an outdoor collected image collecting device as buffer data, or collecting the outdoor scenes by using an indoor collected image collecting device as buffer data, wherein the collection area of the buffer data exceeds 10% of the indoor area.
Further, in the step 2, the method for obtaining the image data with different scales by adopting the progressive space three-solution method is as follows:
1) Data packet: marking and grouping the obtained image data of the scenes with different scales according to the input path, the data type and the metadata information;
2) Partial space three solutions: performing space three-dimensional calculation on each data set to obtain a local calculation pose and a local point cloud;
3) Joint space three solution: according to the local resolving pose and the local point cloud, resolving all image data from a global angle again by means of a point cloud registration method to obtain image global optimization point cloud data;
4) And (3) air triple fusion optimization: for a scene with both image and laser scanning point clouds, registering and aligning image global optimization point cloud data and laser scanning point clouds, then removing noise and outlier points by using a RANSAC algorithm, and finally uniformly sampling so as to optimize and fuse the image global optimization point clouds and the laser scanning point clouds to obtain fused optimization point cloud data; and for a scene where laser point cloud data are not collected, carrying out RANSAC on the image global optimization point cloud data to remove noise and outliers, and then carrying out space uniform sampling to obtain fusion optimization point cloud data.
Further, the method for combining the space three solutions is as follows:
registering and aligning a plurality of local point clouds obtained by the partial space three-dimensional calculation;
reversely optimizing the three solutions of the local space again by means of a collineation equation through the object space coordinates represented by the registered point clouds to obtain local solution pose, further obtaining all the global optimization pose of the image, and then solving all the global optimization pose of the image to obtain global optimization point clouds; the basic form of the collinearity equation is:
wherein:
-x, y is the image plane coordinates of the image point;
-x 0 ，y 0 f is an internal azimuth element of the image;
-X S ，Y S ，Z S the object space coordinates of the shooting station point;
-X A ，Y A ，Z A an object space coordinate which is an object point;
a i ，b i ，c i (i=1, 2, 3) is 9 directional cosine of the 3 external azimuth elements of the image.
Further, the semantic optimization modeling method in the step 3 comprises the following steps:
1) Presetting a scene semantic category;
2) Performing point cloud semantic segmentation on the fusion optimization point cloud data by adopting a PointNet++ algorithm to obtain corresponding semantic categories;
3) Setting different category constraints on objects of different semantic categories, setting corresponding optimization modeling algorithms on the objects, and modeling by adopting corresponding modeling optimization algorithms according to the global point cloud data obtained in the step 2 under the corresponding category constraints to obtain three-dimensional models of all semantic categories;
4) And carrying out semantic edge connection optimization on the three-dimensional models of each semantic category to obtain an overall three-dimensional model, and carrying out model texture mapping on the overall three-dimensional model after edge connection to obtain a final semantic labeling three-dimensional model.
Further, the semantic segmentation method by adopting the PointNet++ algorithm comprises the following steps:
1) Clustering the fusion optimization point clouds to obtain a plurality of point cloud subsets;
2) Extracting the characteristics of each point cloud subset from each point cloud subset through a shared multi-layer perceptron;
3) Repeating the steps for a plurality of times in a layering manner to obtain high-level point cloud characteristics containing rich context information in local neighborhood;
4) According to the learned high-level point cloud characteristics, using a fully connected network to score the category of the point cloud of each cluster, wherein the category with the highest score is the final category;
5) And giving the classification labels to all points originally contained in the category, and realizing semantic segmentation of the point cloud, wherein each point contains a semantic label.
Further, the implementation method of semantic edge optimization comprises the following steps:
1) Edge vertex extraction: detecting edges of the three-dimensional models through triangle edges which are only used once in the three-dimensional models, and further detecting edge vertexes;
2) Structured edge joint error identification: starting from the top points of the first common edges of two adjacent three-dimensional models, carrying out ID labeling to enable the edges to obtain structural information, and comparing and identifying edge connecting errors through the structural information;
3) Edge connection error optimization: and automatically selecting and eliminating redundant edge vertexes or adding missing edge vertexes by judging the principle of the minimum number of related triangles, and processing the edges of two adjacent three-dimensional models to be completely consistent, namely realizing the edge connection error optimization.
Another object of the present invention is to provide a data storage method according to the above method for reconstructing a three-dimensional model based on trans-scale multi-source data, comprising the steps of:
uniformly dividing the solid three-dimensional model into a plurality of grids, and constructing a tile storage unit for each grid;
respectively constructing a hierarchical tile storage unit for triangular networks with different semantics contained in each grid, wherein each hierarchical tile storage unit stores all model data contained in the triangular networks with the same semantics in the grid;
the tile storage unit packages and stores all data contained in all hierarchical tile storage units contained in the tile storage unit.
Compared with the prior art, the invention has the beneficial effects that:
(1) The invention solves the problem of three failures of air space commonly existing in cross-scale and indoor and outdoor modeling through the optimization of a data acquisition mode; the invention provides a progressive space three-resolving method, which improves the serious defects of high failure rate, poor adjustment, slow adjustment and the like of the conventional space three-resolving; and fusion optimization of the point cloud and laser scanning of image space three-resolution is realized, and the dilemma that the existing algorithm can only select two or one is avoided.
(2) According to the invention, fusion point cloud semantic segmentation is provided in a model production link, so that a constructed three-dimensional model is continuous and complete from the whole view, and in fact, models with different semantic labels are formed by different triangular nets, so that the constructed three-dimensional model has better precision and better quality;
(3) The invention provides a concept of hierarchical tiles, which not only can keep the advantage of being convenient for unified management of model files when tiles are divided according to geographic positions, but also can conveniently store semantic information of a three-dimensional model; meanwhile, the problems of display errors, difficult browsing operation and the like caused by superposition of the indoor and the outdoor models are solved.
Drawings
FIG. 1 is a flow chart of a method for reconstructing a three-dimensional model based on cross-scale multi-source data according to an embodiment of the present invention;
FIG. 2 is a difference diagram of progressive space three-to-conventional space three solution according to an embodiment of the present invention;
FIG. 3 is a graph of collinear equations for an embodiment of the present invention;
FIG. 4 is a schematic flow chart of joint space three solution according to an embodiment of the present invention;
FIG. 5 is a flow chart comparing a modeling method according to an embodiment of the present invention with a conventional modeling method;
FIG. 6 is a schematic diagram of a semantic optimization modeling method according to an embodiment of the present invention;
FIG. 7 is a schematic diagram of a semantic edge optimization method according to an embodiment of the present invention.
Detailed Description
The technical solutions of the embodiments of the present invention will be clearly and completely described in the following in conjunction with the embodiments of the present invention, and it is obvious that the described embodiments are only some embodiments of the present invention, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
It should be noted that, without conflict, the embodiments of the present invention and features of the embodiments may be combined with each other.
The invention will be further illustrated, but is not limited, by the following examples.
As shown in fig. 1, the invention discloses a method for reconstructing a three-dimensional model based on trans-scale multi-source data, which is characterized by comprising the following steps:
step 1, arranging a cross-scale measurement control network on a modeling object to model coordinate information of related points of different scales in the object, and acquiring image data of scenes of different scales of the modeling object by adopting different image acquisition devices;
the data is mainly collected, and the data collection mainly comprises the following steps: and (3) jointly arranging a cross-scale control network inside and outside the building object to acquire multi-scale related point coordinate information and cross-scale multi-source data. In this embodiment, control networks are arranged inside and outside the modeling object, corresponding control points are arranged according to scale levels (such as kilometers, meters, millimeters, etc.), and coordinates of the control points are obtained through measurement. The indoor control point coordinate information is measured by using the total station by using the indoor and outdoor vision areas.
The method for acquiring the cross-scale multi-source data is flexibly adjusted according to the reality of a modeling project, and in the embodiment, the acquisition of the multi-source data comprises the following steps: 1) The outdoor scene uses unmanned aerial vehicle to carry multiple lenses to collect inclined images; 2) The indoor scene uses a handheld single lens reflex to collect multi-angle images; 3) For the projects with high precision requirements, the unmanned aerial vehicle can be further selected to be used for carrying the airborne laser radar to acquire the point cloud data of the outdoor scene, and if the conditions are met, the orthographic images can be further selected to be acquired simultaneously; 4) For the projects with high precision requirements, the indoor laser radar point cloud can be collected by using a standing laser scanner and a handheld laser scanner.
In order to ensure the success rate of cross-scale indoor and outdoor automatic fusion three-dimensional reconstruction of multi-source data and solve the problem of three failures of cross-scale and indoor and outdoor modeling, the embodiment provides a buffer data acquisition concept. The buffer data acquisition refers to providing redundant buffer data power-assisted three-settlement when different devices are adopted, targets with different scales are aimed, or indoor and outdoor scenes are involved. The method aims at the multi-source data cross-scale indoor and outdoor automatic fusion three-dimensional reconstruction scene, and specifically comprises the following implementation modes: 1) When different equipment is adopted for collection, the collection ranges of two different equipment in adjacent areas should be defined for collection together, data in the intersection are taken as buffer data, and the area of the buffer data is not less than 10% of the collection area of one of the equipment; 2) Aiming at targets with different scales of outdoor scenes, different voyages are generally adopted, if the voyage difference exceeds 150 meters, data of small-scale scenes are acquired by using the middle voyage as buffer data, and the area of the buffer data is required to completely cover the small-scale scenes; 3) When the indoor and outdoor scenes are involved, the outdoor scene is required to be properly collected by using the outdoor collected equipment as buffer data, or the outdoor scene is required to be properly collected by using the indoor collected equipment as buffer data, and the collection area of the buffer data should exceed 10% of the indoor area.
Step 2, according to the coordinate information obtained in the step 1, adopting a progressive null three-resolving method to respectively perform null three-resolving on the obtained image data of different scale scenes, and then fusing the null three-resolved point cloud data to obtain fused optimized point cloud data of a modeling object;
in the application scene of multi-source data cross-scale indoor and outdoor automatic fusion three-dimensional reconstruction, the conventional air three-solution method obviously shows serious defects of high failure rate, difficult adjustment, slow adjustment and the like. In an actual project, three-dimensional reconstruction software of the main stream at home and abroad has a scene of final adjustment failure and blank three-resolution failure of more than 72 hours of adjustment. In order to solve the problem, the embodiment of the invention improves the conventional space three resolving method and proposes progressive space three resolving, specifically, as shown in fig. 2, the progressive space three resolving method is as follows:
1) Data packet: all image data input by a user are received, the image data are marked and grouped according to the input path, the data type and the metadata information to obtain a plurality of groups of data groups, and images with different paths, different data types or different metadata are classified into different data groups; in the preparation stage of the third space, the embodiment provides a concept of data grouping, and the data of the area network adjustment which is difficult to complete as a whole is divided into a plurality of groups to be subjected to the third space in an easy-to-operate and easy-to-automatic mode, so that the success rate and the average difference efficiency of the third space can be improved;
2) Partial space three solutions: and carrying out partial space three-dimensional calculation on each data set to obtain a partial calculation pose and a partial point cloud, wherein the steps mainly comprise feature point extraction, feature point matching, area network adjustment, dense matching and encryption point coordinate calculation. The feature point extraction is to extract points with obvious color or texture changes in the image as feature points through a SIFT algorithm. After the feature points are extracted, feature vectors of the feature points are calculated, the feature points with consistent feature vectors between two adjacent images are associated and stored as homonymous point pairs, and then a correlation coefficient method is adopted for feature point matching. The homonymous point pairs mean that they point to the same point on the feature and should therefore have the same object coordinates.
Because each image in the original data does not contain pose or has larger error, a large number of homonymous point pairs are obtained after matching, and under the constraint that the homonymous point pairs have the same coordinates as the object point, the external orientation elements of the image are adjusted according to the collineation equation. And (3) solving the external azimuth elements of all the images under the constraint of all the homonymy point pairs, and enabling the total error to be minimum, thus finishing the regional network adjustment. The external azimuth element of the image comprises a position coordinate (X, Y, Z) and a gesture (Omega, phi, kappa) of the imaging moment of the camera, which are abbreviated as image pose.
Dense matching is to search as many homonymy point pairs as possible pixel by pixel in an image through a dense matching algorithm in photogrammetry. In this embodiment, the SGM algorithm is used for dense matching. The encryption point coordinate calculation is to calculate the object coordinates of all the same-name point pairs on two adjacent images where the same-name point pairs are positioned by using a front intersection principle (a basic principle of mapping) according to the object coordinate consistency constraint of the same-name point pairs by using the image pose and a collinearity equation. And constructing a three-dimensional point by the coordinates, so that all the point pairs with the same name form a point cloud.
Wherein, see fig. 3, the basic form of the collinearity equation is:
wherein:
-x, y is the image plane coordinates of the image point;
-x 0 ，y 0 f is an internal azimuth element of the image;
-X S ，Y S ，Z S the object space coordinates of the shooting station point;
-X A ，Y A ，Z A an object space coordinate which is an object point;
a i ，b i ，c i (i=1, 2, 3) is 9 directional cosine of the 3 external azimuth elements of the image.
3) Joint space three solution: the data acquisition stage ensures about 10% of buffer data and is the basis for ensuring the joint space three calculation of progressive space three. The data acquisition stage ensures about 10% of buffered data, and the point cloud of the buffered data has consistency in the data of two different groups and can be spliced and fused in a characteristic matching mode, so that the smooth implementation of the combined space three-solution of progressive space three can be ensured. And carrying out registration alignment according to a plurality of local point clouds obtained by the partial space three-solution, reversely optimizing the local solution pose obtained by the partial space three-solution again by means of a collineation equation through the object space point coordinates represented by the registered point clouds, further obtaining global optimization poses of all images, and carrying out dense matching and encryption coordinate point solution on the global optimization poses of all images to obtain the global optimization point clouds. As shown in fig. 4, after the local solution, a local point cloud 1 formed by the points A, B, C and a local point cloud 2 formed by the points E, F, G are obtained, wherein the local point cloud 1 corresponds to the image P, the local point cloud 2 corresponds to the image Q, the local point cloud 1 and the local point cloud 2 are registered and aligned, and the local solution pose of the image P and the image Q does not satisfy the contribution equation because the registration and alignment result in slight coordinate transformation of the point cloud 1 and the point cloud 2, so that the object point coordinates represented by the point cloud need to be aligned, the local solution pose of the image P and the image Q is optimized again according to the point cloud coordinates and the collineation equation after registration, and then the global optimization pose of the images P and Q is obtained, and then the global optimization pose of all images is densely matched and the encryption coordinate point solution is obtained. The method mainly utilizes the characteristics of simple, efficient and high-precision point cloud registration process, and helps to avoid the problems of high failure rate, poor precision and low efficiency of carrying out regional network adjustment on all data in the whole in conventional space three-solution.
4) Performing air-to-three fusion optimization; the space-three fusion optimization can obtain more accurate image pose on the basis of obtaining global optimization pose by combining space-three calculation, so that the image overlapping relationship can be better judged, and further, the dense matching efficiency and the matching point number are improved; meanwhile, the more accurate image pose can improve the precision of the coordinate calculation of the encryption point, and the global optimization point cloud is obtained.
If the project collects not only image data but also laser point cloud data, the laser point cloud data and the global optimization point cloud can be further fused and optimized, namely, firstly, registration alignment of the laser point cloud data and the global optimization point cloud is carried out, then noise and outliers are removed by using a RANSAC algorithm, and finally, spatial uniform sampling is carried out, so that fused and optimized point cloud can be obtained. If the project does not collect laser point cloud data, the registration and alignment steps of the project and the project can be omitted, the global optimization point cloud is directly subjected to RANSAC to remove noise and outliers, and then the space is uniformly sampled, so that the fusion optimization point cloud can be obtained.
And 3, modeling the modeling object by adopting a semantic optimization modeling method according to the fusion optimization point cloud data obtained in the step 2 to obtain a solid three-dimensional model.
Common photogrammetry three-dimensional reconstruction software only carries out geometric model construction and model texture mapping on the empty three-calculated point cloud in the model production link to obtain a three-dimensional model, so that the constructed three-dimensional model has low quality. In order to execute a specific optimization scheme aiming at modeling objects of different categories and strengthen analysis and application values of a model, an embodiment of the present invention proposes a method for modeling semantic optimization, see fig. 5, mainly comprising the following steps:
1) Presetting scene semantic categories, such as trees, buildings, water bodies and the like;
2) Performing semantic segmentation on the global optimization point cloud by adopting a PointNet++ algorithm to obtain different semantic categories; the semantic segmentation method by adopting the PointNet++ algorithm comprises the following steps:
a. clustering the fusion optimization point clouds to generate a plurality of point cloud subsets, wherein a plurality of center points are firstly extracted according to a kernel density estimation algorithm, and each point is clustered to the center point closest to the center point according to a distance relation, so that clustering is completed;
b. extracting the characteristics of each point cloud subset from each point cloud subset through a shared multi-layer perceptron, wherein the multi-layer perceptron is a feedforward artificial neural network model, and a group of characteristic vectors with preset dimensionality are obtained after calculation of a multi-layer neural network, wherein the characteristic vectors are the characteristics of the point cloud subset;
c. repeating the steps for a plurality of times in a layering manner, namely, using a plurality of point sets output by each clustering algorithm as abstract point clouds, and then carrying out next processing so as to obtain high-level point cloud characteristics containing rich context information in local neighborhood;
d. according to the learned high-level point cloud characteristics, a full-connection network is used for grading the class of each clustered point cloud, the full-connection neural network is a basic feedforward neural network, after calculation of a plurality of layers of neural networks, each output node of the last layer can obtain a probability grade, and the class with the highest grading can be designated as the class of the point cloud;
e. and giving the classification labels to all points originally contained in the category, and realizing semantic segmentation of the point cloud, wherein each point contains a semantic label.
3) As shown in fig. 6, different class constraints are set for objects with different semantic classes, corresponding optimization modeling algorithms are set for the objects with different semantic classes, and under the corresponding class constraints, global point cloud data is obtained according to the step 2, and modeling is performed by adopting the corresponding modeling optimization algorithms to obtain three-dimensional models of the semantic classes; for example, aiming at a modeling object which is very difficult to match in the water body micro-dynamics, the embodiment increases plane constraint, and performs modeling by an optimization modeling algorithm for automatically homogenizing the point cloud and the triangular network, so that the modeling quality of the water surface is better and the effect is better; aiming at modeling objects of the tree with extremely complex blade structures and large storage space occupied by the model, the embodiment increases contour constraint, and performs modeling by an optimization modeling algorithm of automatic thinning of point cloud and triangular net, so that the data volume of the model can be optimized while the basic appearance of the tree is maintained;
5) Because the three-dimensional models of different semantic categories are independently constructed, overlapping errors or gap errors are necessarily generated between the three-dimensional models, and the embodiment adopts semantic edge connection optimization to eliminate the problems, see fig. 7, and mainly comprises the following steps:
edge vertex extraction: edges of each three-dimensional model can be detected through triangle sides which are used once only, and then edge vertexes are detected;
structured edge joint error identification: starting from the first common edge vertex of two adjacent three-dimensional models, carrying out ID labeling to enable the edges to obtain structural information, and finding redundant edge vertices (or missing edge vertices of the other model) by comparing the structural information so as to further identify edge connecting errors between the two adjacent three-dimensional models; in the figure, ID labeling is carried out on edge vertexes of a model 1 and a model 2 from the first common edge vertex, wherein the edge vertexes in the model 1 are sequentially labeled as 1,2,3, 4, 5, 6 and 7, the edge vertexes in the model 2 are sequentially labeled as 1,2,3, 5 and 7, and compared with the prior art, the No. 4 and the No. 6 in the model 2 are missing edge vertexes, so that overlapping errors and crack errors of the two models are respectively caused;
edge connection error optimization: the method comprises the steps of automatically selecting and eliminating redundant edge vertexes or adding missing edge vertexes by judging the principle of 'the number of involved triangles is minimum', and processing edges of two adjacent three-dimensional models to be completely consistent, so that edge joint error optimization can be realized; in fig. 7, in view of missing edge vertices No. 4 and No. 6 in model 2, in order to process two model edges to be identical, redundant edge vertices 6 of model 1 are eliminated, missing edge vertices 4 of model 2 are added, and thus the two models are perfectly bordered. After the processing, the three-dimensional models of the semantic categories can be integrated into a complete global three-dimensional model;
6) And mapping the texture acquired by the image acquisition device onto the global three-dimensional model to obtain a final semantic annotation three-dimensional model. The semantic annotation three-dimensional model obtained by the embodiment is observed from the whole of the live-action three-dimensional model, the model is continuous and complete, and geometric models of different semantic categories are actually constructed separately, so that the model essentially consists of disconnected triangular meshes.
Another object of the present invention is to provide a data storage method according to the above method for reconstructing a three-dimensional model based on trans-scale multi-source data, comprising the steps of:
uniformly dividing the solid three-dimensional model into a plurality of grids, and constructing a tile storage unit for each grid;
respectively constructing a hierarchical tile storage unit for triangular networks with different semantics in each grid, wherein each hierarchical tile storage unit stores all model data included in the triangular networks with the same semantics in the grid;
and the tile storage unit packages and stores all data covered by all the hierarchical tile storage units contained in the tile storage unit.
According to the embodiment, the semantic annotation three-dimensional model is uniformly divided into a plurality of grids with the same size, and a tile storage unit is built for each grid, so that the problem that memory is exploded when the model is built and loaded is avoided. Since the semantic annotation model is a natural way to separate different types of objects into nets, the triangle mesh of the model is divided into natural curves. In order to avoid conflict and error in storage, the embodiment provides a concept of hierarchical tile, namely, triangular networks with different semantics in each grid respectively construct a hierarchical tile storage unit, each hierarchical tile storage unit stores all model data included in the triangular network with the same semantics in the grid, and the tile storage unit packages and stores all data covered by all hierarchical tile storage units included in the tile storage unit. Thus, in hierarchical tiles, one tile is still a file, but the file is similar to a compressed package file, and model files with different semantics are packaged and stored in sequence according to a specified sequence. The method has the advantages that the method can not only keep the advantage of being convenient for unified management of model files when tiles are divided according to geographic positions, but also conveniently store semantic information of the three-dimensional model. For the three-dimensional reconstruction work of indoor and outdoor automatic fusion, the grading type tile has the most obvious advantages that the triangular network can be distinguished to belong to the outdoor or indoor according to semantic information when the model is browsed, so that the indoor model is shielded when the outdoor model is displayed, the outdoor model is shielded when the indoor model is displayed, and display errors and browsing operation difficulties caused by superposition of the indoor model and the outdoor model are avoided.
The foregoing is merely illustrative of the preferred embodiments of the present invention and is not intended to limit the embodiments and scope of the present invention, and it should be appreciated by those skilled in the art that equivalent substitutions and obvious variations may be made using the teachings of the present invention, which are intended to be included within the scope of the present invention.
Claims (9)
1. The reconstruction method of the three-dimensional model based on the trans-scale multi-source data is characterized by comprising the following steps of:
step 1, laying a cross-scale measurement control network on a modeling object to obtain coordinate information of each scale related point in the modeling object, and acquiring image data of different scale scenes of the modeling object by adopting different image acquisition equipment;
step 2, according to the coordinate information in the step 1, adopting a progressive null three-resolving method to respectively perform null three-resolving on the obtained image data of different scale scenes, and then performing fusion optimization on the null three-resolved point cloud data to obtain fusion optimized point cloud data of a modeling object;
step 3, modeling a modeling object by adopting a semantic optimization modeling method according to the fusion optimization point cloud data obtained in the step 2 to obtain a semantic annotation three-dimensional model;
the method for obtaining the image data with different scales by adopting the progressive space three-solution method in the step 2 is as follows:
1) Data packet: marking and grouping the obtained image data of the scenes with different scales according to the input path, the data type and the metadata information;
2) Partial space three solutions: performing space three-dimensional calculation on each data set to obtain a local calculation pose and a local point cloud;
3) Joint space three solution: according to the local resolving pose and the local point cloud, resolving all image data from a global angle again by means of a point cloud registration method to obtain image global optimization point cloud data;
4) And (3) air triple fusion optimization: for a scene with both image and laser scanning point clouds, registering and aligning image global optimization point cloud data and the laser scanning point clouds, then removing noise by using a RANSAC algorithm, and finally uniformly sampling, so as to perform optimization fusion on the image global optimization point clouds and the laser scanning point clouds to obtain fusion optimization point clouds; and for a scene where laser point cloud data are not collected, carrying out RANSAC on the image global optimization point cloud data to remove noise and outliers, and then carrying out space uniform sampling to obtain fusion optimization point cloud data.
2. The method for reconstructing a three-dimensional model based on trans-scale multi-source data according to claim 1, wherein in step 1, unmanned aerial vehicle is adopted to carry multi-lens to collect inclined images for an outdoor scene of a modeling object, and a handheld single-lens camera is adopted to collect multi-angle images for an indoor scene.
3. The method for reconstructing the three-dimensional model based on the cross-scale multi-source data according to claim 1, wherein in the step 1, unmanned aerial vehicle carried on the vehicle is adopted to collect point cloud data of an outdoor scene, and a standing laser scanner and a handheld laser scanner are used to collect laser radar point cloud of an indoor scene.
4. The method for reconstructing a three-dimensional model based on trans-scale multi-source data according to claim 1, wherein the conditions satisfied by the acquired image data in step 1 are:
when different image acquisition devices are adopted for acquisition, intersection sets are defined for acquisition ranges of two different image acquisition image devices in adjacent areas to be acquired together, data in the intersection sets are used as buffer data, and the area of the buffer data is not less than 10% of the acquisition area of one device;
collecting targets with different scales of outdoor scenes at different altitudes, and if the altitude difference exceeds 150 meters, collecting data of the minimum-scale scene in the outdoor scenes by using the middle altitude as buffer data, wherein the area of the buffer data completely covers the minimum-scale scene;
the method comprises the steps of collecting targets with different scales of indoor and outdoor scenes, collecting partial indoor scenes by using an outdoor collected image collecting device as buffer data, or collecting the outdoor scenes by using an indoor collected image collecting device as buffer data, wherein the collection area of the buffer data exceeds 10% of the indoor area.
5. The method for reconstructing a three-dimensional model based on trans-scale multi-source data according to claim 1, wherein the method for combining the space three solutions is as follows:
registering and aligning a plurality of local point clouds obtained by the partial space three-dimensional calculation;
reversely optimizing the local solution pose obtained by the local space three-solution again by means of a collineation equation through object space point coordinates represented by the registered point clouds, so that all images obtain global optimization pose, and then solving all the image global optimization pose to obtain global optimization point clouds; the basic form of the collinearity equation is:
wherein:
image plane coordinates for the image point;
is the internal azimuth element of the image;
the object space coordinates of the shooting station point;
an object space coordinate which is an object point;
9 direction cosine composed of 3 external azimuth angle elements of the image, < +.>。
6. The method for reconstructing a three-dimensional model based on trans-scale multi-source data according to claim 1, wherein the method for semantic optimization modeling in step 3 comprises the steps of:
1) Presetting a scene semantic category;
2) Performing point cloud semantic segmentation on the global optimization point cloud data by adopting a PointNet++ algorithm to obtain corresponding semantic categories;
3) Setting different category constraints on objects of different semantic categories, setting corresponding optimization modeling algorithms on the objects, and modeling by adopting corresponding modeling optimization algorithms according to the global point cloud data obtained in the step 2 under the corresponding category constraints to obtain three-dimensional models of all semantic categories;
4) And carrying out semantic edge connection optimization on the three-dimensional models of each semantic category to obtain an overall three-dimensional model, and carrying out model texture mapping on the overall three-dimensional model after edge connection to obtain a final semantic labeling three-dimensional model.
7. The method for reconstructing a three-dimensional model based on trans-scale multi-source data according to claim 6, wherein the semantic segmentation method using the PointNet++ algorithm is as follows:
1) Generating a plurality of point cloud subsets by using a clustering mode;
2) Extracting the characteristics of each point cloud subset from each point cloud subset through a shared multi-layer perceptron;
3) Repeating the steps for a plurality of times in a layering manner to obtain high-level point cloud characteristics containing rich context information in local neighborhood;
4) According to the learned high-level point cloud characteristics, using a fully connected network to score the category of the point cloud of each cluster, wherein the category with the highest score is the final category;
5) And giving the classification labels to all points originally contained in the category, and realizing semantic segmentation of the point cloud, wherein each point contains a semantic label.
8. The method for reconstructing a three-dimensional model based on trans-scale multi-source data according to claim 1, wherein the implementation method for semantic edge optimization is as follows:
1) Edge vertex extraction: detecting edges of the three-dimensional models through triangle edges which are only used once in the three-dimensional models, and further detecting edge vertexes;
2) Structured edge joint error identification: starting from the top point of the first common edge adjacent to the two three-dimensional models, carrying out ID labeling to enable the edge to obtain structural information, and comparing and identifying the edge connecting error through the structural information;
3) Edge connection error optimization: and automatically selecting and eliminating redundant edge vertexes or adding missing edge vertexes by judging the principle of the minimum number of related triangles, and processing the edges of two adjacent three-dimensional models to be completely consistent, namely realizing the edge connection error optimization.
9. A data storage method of a method of reconstructing a three-dimensional model based on trans-scale multi-source data according to any one of claims 1-8, comprising the steps of:
uniformly dividing the solid three-dimensional model into a plurality of grids, and constructing a tile storage unit for each grid;
respectively constructing a hierarchical tile storage unit for triangular networks with different semantics contained in each grid, wherein each hierarchical tile storage unit stores all model data contained in the triangular networks with the same semantics in the grid;
the tile storage unit packages and stores all data contained in all hierarchical tile storage units contained in the tile storage unit.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311236479.4A CN117315146B (en) | 2023-09-22 | 2023-09-22 | Reconstruction method and storage method of three-dimensional model based on trans-scale multi-source data |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311236479.4A CN117315146B (en) | 2023-09-22 | 2023-09-22 | Reconstruction method and storage method of three-dimensional model based on trans-scale multi-source data |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN117315146A CN117315146A (en) | 2023-12-29 |
| CN117315146B true CN117315146B (en) | 2024-04-05 |
Family
ID=89287735
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311236479.4A Active CN117315146B (en) | 2023-09-22 | 2023-09-22 | Reconstruction method and storage method of three-dimensional model based on trans-scale multi-source data |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117315146B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117523206B (en) * | 2024-01-04 | 2024-03-29 | 南京航空航天大学 | Automatic assembly method based on cross-source point cloud and multi-mode information |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109827548A (en) * | 2019-02-28 | 2019-05-31 | 华南机械制造有限公司 | The processing method of aerial survey of unmanned aerial vehicle data |
| CN111815776A (en) * | 2020-02-04 | 2020-10-23 | 山东水利技师学院 | Three-dimensional building fine geometric reconstruction method integrating airborne and vehicle-mounted three-dimensional laser point clouds and streetscape images |
| CN113920262A (en) * | 2021-10-15 | 2022-01-11 | 中国矿业大学(北京) | Mining area FVC calculation method and system for enhancing edge sampling and improving Unet model |
| WO2022252274A1 (en) * | 2021-05-31 | 2022-12-08 | 北京理工大学 | Point cloud segmentation and virtual environment generation method and apparatus based on pointnet network |
| WO2023045455A1 (en) * | 2021-09-21 | 2023-03-30 | 西北工业大学 | Non-cooperative target three-dimensional reconstruction method based on branch reconstruction registration |
| CN115937288A (en) * | 2022-10-12 | 2023-04-07 | 国网四川省电力公司电力科学研究院 | Three-dimensional scene model construction method for transformer substation |
-
2023
- 2023-09-22 CN CN202311236479.4A patent/CN117315146B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109827548A (en) * | 2019-02-28 | 2019-05-31 | 华南机械制造有限公司 | The processing method of aerial survey of unmanned aerial vehicle data |
| CN111815776A (en) * | 2020-02-04 | 2020-10-23 | 山东水利技师学院 | Three-dimensional building fine geometric reconstruction method integrating airborne and vehicle-mounted three-dimensional laser point clouds and streetscape images |
| WO2022252274A1 (en) * | 2021-05-31 | 2022-12-08 | 北京理工大学 | Point cloud segmentation and virtual environment generation method and apparatus based on pointnet network |
| WO2023045455A1 (en) * | 2021-09-21 | 2023-03-30 | 西北工业大学 | Non-cooperative target three-dimensional reconstruction method based on branch reconstruction registration |
| CN113920262A (en) * | 2021-10-15 | 2022-01-11 | 中国矿业大学(北京) | Mining area FVC calculation method and system for enhancing edge sampling and improving Unet model |
| CN115937288A (en) * | 2022-10-12 | 2023-04-07 | 国网四川省电力公司电力科学研究院 | Three-dimensional scene model construction method for transformer substation |
Non-Patent Citations (3)
| Title |
|---|
| 基于倾斜摄影测量的三维建模关键技术研究与应用;赵政权;《贵州大学学报( 自然科学版)》;20200731;第35-39页 * |
| 基于点云数据的三目标识别和模型分割方法;牛辰庚;《图学学报》;20190430;第274-281页 * |
| 赵政权.基于倾斜摄影测量的三维建模关键技术研究与应用.《贵州大学学报( 自然科学版)》.2020,第35-39页. * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN117315146A (en) | 2023-12-29 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110570428B (en) | Method and system for dividing building roof sheet from large-scale image dense matching point cloud | |
| CN107133325B (en) | Internet photo geographic space positioning method based on street view map | |
| JP6080642B2 (en) | 3D point cloud analysis method | |
| Moons et al. | Automatic modelling and 3D reconstruction of urban house roofs from high resolution aerial imagery | |
| CN109883401B (en) | Method and system for measuring visual field of city mountain watching | |
| Zhu et al. | Leveraging photogrammetric mesh models for aerial-ground feature point matching toward integrated 3D reconstruction | |
| JP6080641B2 (en) | 3D point cloud analysis method | |
| Verykokou et al. | Oblique aerial images: a review focusing on georeferencing procedures | |
| CN113096250A (en) | Three-dimensional building model library system construction method based on unmanned aerial vehicle aerial image sequence | |
| CN114998536A (en) | Model generation method and device based on novel basic mapping and storage medium | |
| CN117315146B (en) | Reconstruction method and storage method of three-dimensional model based on trans-scale multi-source data | |
| JP6080640B2 (en) | 3D point cloud analysis method | |
| CN115358486B (en) | Port freight volume prediction method, system and application based on stereoscopic satellite images | |
| CN110223380A (en) | Fusion is taken photo by plane and the scene modeling method of ground multi-view image, system, device | |
| Özdemir et al. | A multi-purpose benchmark for photogrammetric urban 3D reconstruction in a controlled environment | |
| JP2023530449A (en) | Systems and methods for air and ground alignment | |
| Frommholz et al. | Extracting semantically annotated 3D building models with textures from oblique aerial imagery | |
| Cui et al. | Tracks selection for robust, efficient and scalable large-scale structure from motion | |
| Sun et al. | Geographic, geometrical and semantic reconstruction of urban scene from high resolution oblique aerial images. | |
| CN113240755B (en) | City scene composition method and system based on street view image and vehicle-mounted laser fusion | |
| CN113781639B (en) | Quick construction method for digital model of large-scene road infrastructure | |
| Atik et al. | An automatic image matching algorithm based on thin plate splines | |
| CN113867410A (en) | Unmanned aerial vehicle aerial photography data acquisition mode identification method and system | |
| Liang et al. | Efficient match pair selection for matching large-scale oblique UAV images using spatial priors | |
| Munoz-Silva et al. | A Survey on Point Cloud Generation for 3D Scene Reconstruction |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |