CN115345990A - Oblique photography three-dimensional reconstruction method and device for weak texture scene - Google Patents
Oblique photography three-dimensional reconstruction method and device for weak texture scene Download PDFInfo
- Publication number
- CN115345990A CN115345990A CN202210852833.5A CN202210852833A CN115345990A CN 115345990 A CN115345990 A CN 115345990A CN 202210852833 A CN202210852833 A CN 202210852833A CN 115345990 A CN115345990 A CN 115345990A
- Authority
- CN
- China
- Prior art keywords
- dimensional model
- weak texture
- generate
- integral
- range
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 48
- 238000013135 deep learning Methods 0.000 claims abstract description 31
- 238000012545 processing Methods 0.000 claims abstract description 12
- 238000013507 mapping Methods 0.000 claims description 13
- 238000013527 convolutional neural network Methods 0.000 claims description 10
- 238000000605 extraction Methods 0.000 claims description 9
- 238000004458 analytical method Methods 0.000 claims description 7
- 238000005215 recombination Methods 0.000 claims description 7
- 230000006798 recombination Effects 0.000 claims description 7
- 238000004590 computer program Methods 0.000 claims description 5
- 238000012805 post-processing Methods 0.000 abstract 1
- 230000000694 effects Effects 0.000 description 12
- 230000008569 process Effects 0.000 description 8
- 230000006870 function Effects 0.000 description 6
- 238000005516 engineering process Methods 0.000 description 4
- 238000004364 calculation method Methods 0.000 description 3
- 238000010586 diagram Methods 0.000 description 3
- 239000011159 matrix material Substances 0.000 description 3
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000011160 research Methods 0.000 description 2
- 238000012549 training Methods 0.000 description 2
- XLYOFNOQVPJJNP-UHFFFAOYSA-N water Substances O XLYOFNOQVPJJNP-UHFFFAOYSA-N 0.000 description 2
- 230000003190 augmentative effect Effects 0.000 description 1
- 238000004422 calculation algorithm Methods 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 239000003814 drug Substances 0.000 description 1
- 238000007499 fusion processing Methods 0.000 description 1
- 230000001788 irregular Effects 0.000 description 1
- 238000005259 measurement Methods 0.000 description 1
- 230000005055 memory storage Effects 0.000 description 1
- 238000013508 migration Methods 0.000 description 1
- 230000005012 migration Effects 0.000 description 1
- 238000010295 mobile communication Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 229920001296 polysiloxane Polymers 0.000 description 1
- 238000007781 pre-processing Methods 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
- 238000002310 reflectometry Methods 0.000 description 1
- 238000005070 sampling Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F17/00—Digital computing or data processing equipment or methods, specially adapted for specific functions
- G06F17/10—Complex mathematical operations
- G06F17/16—Matrix or vector computation, e.g. matrix-matrix or matrix-vector multiplication, matrix factorization
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T15/00—3D [Three Dimensional] image rendering
- G06T15/04—Texture mapping
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T3/4023—Scaling of whole images or parts thereof, e.g. expanding or contracting based on decimating pixels or lines of pixels; based on inserting pixels or lines of pixels
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T3/4046—Scaling of whole images or parts thereof, e.g. expanding or contracting using neural networks
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Data Mining & Analysis (AREA)
- Mathematical Optimization (AREA)
- Pure & Applied Mathematics (AREA)
- Computer Graphics (AREA)
- Mathematical Analysis (AREA)
- Evolutionary Computation (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- Computational Mathematics (AREA)
- Artificial Intelligence (AREA)
- Life Sciences & Earth Sciences (AREA)
- Health & Medical Sciences (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Databases & Information Systems (AREA)
- General Health & Medical Sciences (AREA)
- Molecular Biology (AREA)
- Algebra (AREA)
- Geometry (AREA)
- Image Processing (AREA)
Abstract
The application discloses a method and a device for oblique photography three-dimensional reconstruction of a weak texture scene. The method comprises the following steps: generating integral dense point cloud according to the obtained oblique photography image, processing the integral dense point cloud to obtain an integral three-dimensional model, generating a digital orthophoto map according to the integral three-dimensional model, identifying and extracting a weak texture region range of the digital orthophoto map by using a trained deep learning network model, interpolating the point cloud in the weak texture region range, performing interpolation post-processing on the weak texture region to generate a reconstructed three-dimensional model of the weak texture region, recombining the reconstructed three-dimensional model of the weak texture region with the integral three-dimensional model, and generating a final three-dimensional model. By the implementation of the method and the device, the three-dimensional reconstruction and display of the weak texture scene or the similar scene with higher accuracy can be realized.
Description
Technical Field
The application relates to the technical field of three-dimensional reconstruction, in particular to a method and a device for oblique photography three-dimensional reconstruction of a weak texture scene.
Background
The three-dimensional reconstruction is a research hotspot in the fields of computer vision and computer graphics, and plays a strong auxiliary role in the research of the fields of cultural relic reconstruction, AR (augmented reality) tourism, automatic driving, smart home, clinical medicine and the like. The three-dimensional reconstruction is a process of reconstructing three-dimensional information of a real object by acquiring a single-view or multi-view oblique photographic image of the real world real object and position and posture information of the image by using a low-altitude aerial photography oblique photography technology of an unmanned aerial vehicle and the like.
Currently, the main processes of three-dimensional reconstruction include: acquiring a real object oblique photography image, preprocessing a graph, extracting and matching features, calculating and fusing a depth map, pasting a texture map and visualizing a scene.
However, for weak texture scenes or similar scenes such as water areas or deserts with single color, repeated texture, many similar regions and few dotted line textures, the current three-dimensional reconstruction method may cause the problems of inaccurate, leaky and incomplete reconstruction of the three-dimensional reconstruction result due to difficult feature extraction and serious depth map calculation deviation, and even the situation of reconstruction failure occurs.
Disclosure of Invention
The embodiment of the application provides the oblique photography three-dimensional reconstruction method and device for the weak texture scene, solves the problems of difficult feature extraction, serious depth map matching deviation and the like in the three-dimensional reconstruction process of the weak texture scene in the prior art, and realizes the high-precision three-dimensional reconstruction and display for the weak texture scene or the similar scene.
In a first aspect, an embodiment of the present application provides a method for oblique photography three-dimensional reconstruction of a weak texture scene, where the method includes:
generating an integral three-dimensional model according to the obtained oblique photography image; generating a digital orthophoto map according to the integral three-dimensional model; recognizing and extracting the range of a weak texture region of the digital orthophoto map by using the trained deep learning network model; uniformly interpolating the range of the weak texture region to generate dense point cloud of the weak texture region; gridding and mapping the weak texture region to generate a reconstructed three-dimensional model of the weak texture region; cutting data in the range of the weak texture area in the integral three-dimensional model to generate a cut three-dimensional model; and recombining the reconstructed three-dimensional model of the weak texture region and the data of the cut three-dimensional model to generate a final three-dimensional model.
With reference to the first aspect, in one possible implementation manner, generating an overall three-dimensional model from an acquired oblique photography image includes: acquiring aerial image data according to oblique photography, and recording pose information of a photo in the image; acquiring the pose information and converting the pose information into a ground-fixed coordinate system; extracting the characteristics of the oblique photography image, and performing characteristic matching and space-three encryption to generate an integral sparse point cloud; calculating and fusing a depth map of the integral sparse point cloud and the image to generate integral dense point cloud; and carrying out meshing and texture mapping processing on the overall dense point cloud to generate the overall three-dimensional model.
With reference to the first aspect, in a possible implementation manner, the identifying and extracting a range of a weak texture region of the digital orthophoto map by using the trained deep learning network model includes: constructing the deep learning network model according to a deep convolutional neural network DeeplabV3 +; and selecting a plurality of weak texture scene samples to train the deep learning network model until the deep learning network model can accurately identify and extract the range of the weak texture region in the digital orthophoto map.
With reference to the first aspect, in a possible implementation manner, the clipping data in the range of the weak texture region in the entire three-dimensional model to generate a clipped three-dimensional model includes: performing buffer area analysis on the boundary of the weak texture area range of the integral three-dimensional model to generate a new boundary; and cutting data in the range of the weak texture region in the integral three-dimensional model according to the new boundary to generate a cut three-dimensional model which does not contain the weak texture region.
In a second aspect, an embodiment of the present application provides an oblique photography three-dimensional reconstruction apparatus for a weak texture scene, the apparatus includes: the first generation module is used for generating an integral three-dimensional model according to the acquired oblique photography image; the second generation module is used for generating the digital orthophoto map according to the integral three-dimensional model; the weak texture range extraction module is used for identifying and extracting the range of the weak texture region according to the digital orthophoto map by using the deep learning network model; the weak texture range interpolation module is used for uniformly interpolating the range of the weak texture region to generate dense point cloud of the weak texture region; the reconstruction three-dimensional model module is used for generating dense point cloud after interpolating the weak texture region, and performing gridding and mapping processing to generate a reconstruction three-dimensional model of the weak texture region; the third generation module is used for cutting data in the range of the weak texture area in the integral three-dimensional model to generate a cut three-dimensional model; and the three-dimensional model recombination module is used for recombining the reconstructed three-dimensional model of the weak texture region and the data of the cut three-dimensional model to generate a final three-dimensional model.
With reference to the second aspect, in a possible implementation manner, the first generating module is specifically configured to: acquiring aerial image data according to oblique photography, and recording pose information of a photo in the image;
acquiring the pose information and converting the pose information into a ground-fixed coordinate system; extracting the characteristics of the oblique photography image, and performing characteristic matching and space-three encryption to generate an integral sparse point cloud; calculating and fusing a depth map of the integral sparse point cloud and the image to generate integral dense point cloud; and carrying out meshing and texture mapping processing on the overall dense point cloud to generate an overall three-dimensional model.
With reference to the second aspect, in a possible implementation manner, the weak texture range extraction module is specifically configured to: constructing the deep learning network model according to a deep convolutional neural network DeeplabV3 +; and selecting a plurality of weak texture scene samples to train the deep learning network model until the deep learning network model can accurately identify and extract the range of the weak texture region in the digital orthophoto map.
With reference to the second aspect, in a possible implementation manner, the weak texture range extraction module is specifically configured to: performing buffer area analysis on the boundary of the weak texture region range of the integral three-dimensional model to generate a new boundary; and cutting data in the range of the weak texture region in the integral three-dimensional model according to the new boundary to generate a cut three-dimensional model not containing the weak texture region.
In a third aspect, an embodiment of the present application provides an apparatus, where the apparatus includes: a processor; a memory for storing processor-executable instructions; the processor, when executing the executable instructions, performs the method as described in the first aspect or any one of the possible implementations of the first aspect.
In a fourth aspect, embodiments of the present application provide a non-transitory computer-readable storage medium including instructions or computer programs for storing a computer program, which when executed, causes a method according to the first aspect or any one of the possible implementations of the first aspect to be implemented.
One or more technical solutions provided in the embodiments of the present application have at least the following technical effects or advantages:
the oblique photography three-dimensional reconstruction method for the weak texture scene is adopted, the deep learning network model is trained, the range of the weak texture region in the digital orthophoto map can be accurately identified and extracted, and the process is simple and quick. The method has the advantages that the uniform interpolation of the weak texture region range in the digital orthographic projection image is extracted, dense point cloud is generated, the dense reconstruction effect of a weak texture scene or a similar scene can be improved, the problems of leaks and incompleteness in the three-dimensional reconstruction of the weak texture scene or the similar scene are solved, and the condition that the reconstruction of the weak texture scene or the similar scene fails is avoided. By implementing the embodiment of the application, the three-dimensional restoration and display of the weak texture scene or the similar scene with higher accuracy are realized.
Drawings
In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings needed to be used in the description of the embodiments of the present application or the prior art will be briefly introduced below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and other drawings can be obtained by those skilled in the art without creative efforts.
Fig. 1 is a schematic flowchart of a oblique photography three-dimensional reconstruction method for a weak texture scene according to an embodiment of the present application;
FIG. 2 is a schematic flow chart illustrating the generation of an overall three-dimensional model from an acquired oblique photographic image according to an embodiment of the present disclosure;
fig. 3 is a schematic flowchart of training a deep learning network model according to an embodiment of the present disclosure;
FIG. 4 is a schematic flow chart of generating a three-dimensional model for clipping according to an embodiment of the present disclosure;
fig. 5 is a schematic block diagram of an oblique photography three-dimensional reconstruction apparatus for a weak texture scene according to an embodiment of the present application;
FIG. 6 is a set of oblique photographic images provided in accordance with an embodiment of the present application;
FIG. 7 is a digital orthophotomap generated from the oblique photography image of FIG. 6 according to an embodiment of the present application;
FIG. 8 is a graphical representation of the overall three-dimensional model effect generated from the acquired oblique photographic image of FIG. 6;
fig. 9 is a diagram illustrating a final three-dimensional model effect generated by the three-dimensional reconstruction method according to the oblique photographic image in fig. 6 according to the embodiment of the present application.
Detailed Description
The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. It is to be understood that the embodiments described are only a few embodiments of the present invention, and not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
The embodiment of the application provides an oblique photography three-dimensional reconstruction method and device for a weak texture scene, which are used for solving the problems that a weak texture region has a leak and the reconstruction is incomplete in the three-dimensional reconstruction process and improving the accuracy and the integrity of a three-dimensional reconstruction model.
Some of the techniques referred to in the embodiments of the present application are described below, including the use of custom terms for the embodiments of the present application to facilitate understanding, which should be considered exemplary only. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the present application. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.
Weak texture scene: the weak texture scene mainly refers to desert, forest, ice and snow, shadow, water surface and other areas in traditional aerial survey, and the scene generally shows that the scene texture is similar, the reflectivity is low, the pixel values of adjacent pixel points are close, and the feature matching is difficult.
Oblique photography: the oblique photography technology is a high and new technology developed in recent years in the international surveying and mapping field, which overturns the limitation that the original orthoimage can only be shot from a vertical angle.
Three-dimensional reconstruction: the English term is 3D Reconstruction, and in computer vision, three-dimensional Reconstruction refers to a process of reconstructing three-dimensional information from single-view or multi-view images. Since the information of a single view is incomplete, the three-dimensional reconstruction requires the use of empirical knowledge. Multi-view three-dimensional reconstruction (human-like binocular positioning) is relatively easy, and the method is a process of calculating 3D information from a group of 2D images of a space-time sequence, namely, the relation between an image coordinate system of a camera and a world coordinate system is calculated. Three-dimensional information is then reconstructed using the information in the plurality of two-dimensional images.
Depth map: each pixel value of the image represents the distance between a point in the scene and the XY plane of the camera, and the unit is: mm.
Point cloud: the English term is Point Cloud, which represents the spatial coordinates of each sampling Point on the surface of an object, and the obtained Point is a set of points.
Digital orthophotographs: the english term Digital Ortho Map (DOM) is an image based on aerial or remote sensing images (monochrome/color) with both Map geometric accuracy and image features. And the digital orthophoto map is digital, can enlarge on the computer partially, have good interpretation performance and measurement performance and managerial ability.
Cutting a three-dimensional model: in the embodiment of the application, the boundary of the weak texture region range of the integral three-dimensional model is subjected to buffer area analysis to generate a new boundary; and cutting data in the range of the weak texture region in the whole three-dimensional model according to the new boundary to generate a cut three-dimensional model without the weak texture region.
Fig. 1 is a flowchart illustrating a method for oblique photography three-dimensional reconstruction of a weak texture scene according to an embodiment of the present application, including steps 101 to 107. Fig. 1 is only one execution sequence shown in the embodiment of the present application, and does not represent the only execution sequence of the oblique photography three-dimensional reconstruction method for a weak texture scene, and the steps shown in fig. 1 may be executed in parallel or reversed in case that the final result can be achieved.
Step 101: and generating an integral three-dimensional model according to the obtained oblique photography image. It should be noted that the "oblique photography image" and "image" mentioned in the embodiment of the present application refer to multiple images with multiple viewing angles obtained by the unmanned aerial vehicle mounted with the five-lens camera for low-altitude aerial photography, and are not intended to refer to a specific image at a specific viewing angle.
In step 101 of the embodiment of the present application, an entire three-dimensional model is generated from the obtained oblique photography image, and the entire three-dimensional model shown in fig. 8 is generated from the oblique photography image shown in fig. 6.
The specific implementation of step 101 and the technical effects thereof can be referred to the embodiment shown in fig. 2, which includes steps 201 to 205, and are described in detail as follows.
Step 201: and acquiring aerial image data according to oblique photography, and recording the pose information of the photo in the image. An example listing is shown for oblique photography of the image shown in fig. 6.
Step 202: and acquiring pose information and converting the pose information into a ground-fixed coordinate system. And reading the point coordinates and the attitude angle of the photo POS (longitude and latitude high coordinates). EXIF information is an abbreviation of exchangeable image files, which are set specifically for photographs of digital cameras, for recording attribute information and photographing data of digital photographs. The XMP information can record some metadata information of the movie, including the pose of the movie, and the pose angle of the camera is recorded in the general image XMP. Reading EXIF and XMP information through software to obtain position information of corresponding oblique photography image, namely longitude and latitude information, wherein the coordinate system of the attitude angle is relative to the body coordinate system of the aircraft, and converting the attitude angle (psi, theta, phi) under the attitude Z-Y-X corner system into the ground-fixed coordinate system by using the attitude angle information, firstly converting the body coordinate system into a navigation coordinate system, and converting a matrix into a matrix
The specific conversion method is as follows:
navigation coordinate system coordinates:
wherein r is b Is a coordinate of a coordinate system of the body, r n Are navigational coordinate system coordinates.
Rotation matrix from earth-fixed coordinate system to navigation coordinate system
The following:
the body coordinate system is converted into a ground-fixed coordinate system:
wherein r is e Is the coordinates of a ground fixed coordinate system.
The attitude angle of the camera can be converted into the ground-fixed coordinate system through the transformation.
Step 203: extracting the characteristics of the oblique photography image, and generating an integral sparse point cloud by characteristic matching and space-three encryption. And extracting the characteristic points of the oblique photography image, performing characteristic matching and space-three encryption, and automatically performing characteristic point matching according to a characteristic extraction result. According to the position and pose information of the POS (longitude and latitude high coordinate) read in step 201, an aerial triangulation is assisted, a bundle adjustment is performed to obtain a position and a posture of the camera and a three-dimensional coordinate of a connection point, and an achievement is stored in a designated achievement coordinate system, which is generally a projection coordinate system. The step is used for generating the whole sparse point cloud of the image and calculating the position and posture information of the image, and the specific algorithm is not described too much here.
Step 204: and carrying out depth map calculation and fusion processing on the whole sparse point cloud and the image to generate the whole dense point cloud. And obtaining the overall sparse point cloud of the oblique photography image generated in the step 203, and calculating and fusing through the depth map to generate the overall dense point cloud. Dense corresponding relation is built in the camera pose images obtained through space-three encryption calculation, the depth maps of each oblique photography image are estimated, the depth maps are fused to obtain unified point cloud representation, and dense three-dimensional point cloud reconstruction results are generated.
Step 205: and carrying out meshing and texture mapping processing on the integral dense point cloud to generate an integral three-dimensional model. And (3) constructing an irregular triangulation network according to the dense point cloud obtained in the step (204), dividing the whole area by a triangulation network model, constructing a three-dimensional mesh, enabling any point in the area to fall on the vertex, the edge or the triangle of the corresponding triangular surface, and after finishing mesh generation, performing texture mapping to generate an integral three-dimensional model, as shown in an example figure 8.
Step 102: a digital orthophotomap is generated from the overall three-dimensional model. Take the digital orthophoto map shown in fig. 7 as an example.
Step 103: and identifying and extracting the range of the weak texture region of the digital orthophoto map by using the trained deep learning network model.
The specific implementation manner and technical effects of step 103 can refer to the embodiment shown in fig. 3, including steps 301 to 302, which are described in detail below.
Step 301: and constructing a deep learning network model according to the deep convolutional neural network DeeplabV3 +. And constructing a deep learning network model according to the deep convolutional neural network DeeplabV3+, and selecting a plurality of weak texture scene samples to train the deep learning network model.
Step 301 in the embodiment of the present application adopts a deep convolutional neural network deplabv 3+, which can achieve the effect of accurately identifying a weak texture region. Of course, the embodiment of the present application is not limited to the deep convolutional neural network deplabv 3+, and other deep convolutional neural network models may also be adopted, such as: deeplabV2, deeplabV3. Or, the updated deep convolutional neural network model generated with the development of the technology in the computer vision field still belongs to the technical scope of the present invention.
Step 302: and selecting a plurality of weak texture scene samples to train the deep learning network model until the deep learning network model can accurately identify and extract the range of the weak texture region of the digital orthophoto map. And recognizing and extracting the range of the weak texture region of the digital orthophoto map according to the deep learning network model, so that the deep learning network model can accurately recognize and extract the weak texture region of the digital orthophoto map of all sample images. It is noted that references to "a plurality" in this application are intended to be illustrative rather than limiting, and those skilled in the art will appreciate that references to "two or more" are intended to be exemplary and not limiting unless the context clearly indicates otherwise.
Step 104: and (4) uniformly interpolating the range of the weak texture area to generate dense point cloud of the weak texture area.
And (5) according to the range identified by the weak texture area in the step 103, matching the dense point clouds in the step 204, and uniformly interpolating the whole weak texture range to generate dense point clouds.
Step 105: and carrying out meshing and mapping treatment on the dense point cloud of the weak texture region to generate a reconstructed three-dimensional model of the weak texture region.
Step 106: and cutting data in the range of the weak texture area in the whole three-dimensional model to generate a cut three-dimensional model. The term "cropping a three-dimensional model" is understood herein to mean the result of performing steps 401 and 402.
The specific implementation of step 106 and the technical effects thereof can be referred to the embodiment shown in fig. 4, which includes steps 401 to 402, as described in detail below.
Step 401: and (4) carrying out buffer area analysis on the boundary of the weak texture region range of the integral three-dimensional model to generate a new boundary.
In order to avoid a gap between the cut three-dimensional model and the reconstructed three-dimensional model of the weak texture region, the two three-dimensional models need to be overlapped, and the buffer area analysis is performed on the boundary of the weak texture region identified in the step 103 to generate a new boundary.
Step 402: and cutting data in the range of the weak texture region in the whole three-dimensional model according to the new boundary to generate a cut three-dimensional model without the weak texture region.
Step 107: and recombining the reconstructed three-dimensional model of the weak texture region and the cut three-dimensional model data to generate a final three-dimensional model. Fig. 9 illustrates an example of a final three-dimensional model effect diagram according to an embodiment of the present application.
The restructuring method mentioned in the embodiment of the present application in "restructuring the three-dimensional model of the weak texture region and cutting the three-dimensional model data" should be understood by those skilled in the art that the desired effect of the embodiment of the present application can be achieved by using the restructuring method in the present application.
According to the embodiment of the application, the range of the weak texture region in the digital orthophoto map can be accurately identified and extracted by training the deep learning network model. The method comprises the steps of extracting the uniform interpolation of the weak texture area range in the digital orthographic projection image, generating dense point cloud, and improving the reconstruction effect of a weak texture scene or a similar scene, so that the problems of vulnerability and incompleteness in the three-dimensional reconstruction of the weak texture scene or the similar scene are solved, and the condition that the reconstruction of the weak texture scene or the similar scene fails is avoided. The reconstructed three-dimensional model and the cut three-dimensional model of the weak texture region are recombined through data recombination, the recombination effect can be improved, and the problem that the recombination has gaps is solved. By implementing the embodiment of the application, the three-dimensional restoration and display of the weak texture scene or the similar scene with higher accuracy are realized.
Although the present application provides method steps as described in an embodiment or flowchart, additional or fewer steps may be included based on conventional or non-inventive efforts. The sequence of steps recited in this embodiment is only one of many steps in execution sequence, and does not represent a unique order of execution. When the device or the client product in practice executes, it can execute sequentially or in parallel according to the method shown in the embodiment or the figures (for example, in the context of parallel processors or multi-thread processing).
As shown in fig. 5, an embodiment of the present application further provides an oblique photography three-dimensional reconstruction apparatus 500 for a weak texture scene. The device includes: the three-dimensional reconstruction method comprises a first generation module 501, a second generation module 502, a weak texture range extraction module 503, a weak texture range interpolation module 504, a three-dimensional reconstruction model module 505, a third generation module 506 and a three-dimensional model recombination module 507.
The first generation module 501 is used to generate an overall three-dimensional model from the acquired oblique photography images. In this embodiment of the present application, the first generating module 501 is specifically configured to: acquiring aerial image data according to oblique photography, and recording pose information of a photo in the image; acquiring pose information and converting the pose information into a ground-fixed coordinate system; extracting the characteristics of the oblique photography image, and performing characteristic matching and space-three encryption to generate an integral sparse point cloud; calculating and fusing a depth map of the whole sparse point cloud and the image to generate a whole dense point cloud; and carrying out meshing and texture mapping processing on the integral dense point cloud to generate an integral three-dimensional model.
The second generation module 502 is used to generate a digital orthophoto map from the overall three-dimensional model.
The weak texture range extraction module 503 is configured to identify and extract a range of the weak texture region from the digital orthophoto map using a deep learning network model. In this embodiment of the present application, the weak texture range extracting module 503 is specifically configured to: constructing a deep learning network model according to the deep convolutional neural network DeeplabV3 +; and selecting a plurality of weak texture scene samples to train the deep learning network model until the deep learning network model can accurately identify and extract the range of the weak texture region in the digital orthophoto map.
The weak texture range interpolation module 504 is configured to uniformly interpolate a weak texture region range to generate dense point clouds of the weak texture region.
The reconstructed three-dimensional model module 505 is configured to perform gridding and mapping processing after interpolating the weak texture region, and generate a reconstructed three-dimensional model of the weak texture region.
The third generating module 506 is configured to crop data in the range of the weak texture region in the entire three-dimensional model, and generate a cropped three-dimensional model. In this embodiment of the present application, the weak texture range interpolation module 504 is specifically configured to: performing buffer area analysis on the boundary of the weak texture area range of the whole three-dimensional model to generate a new boundary; and cutting data in the range of the weak texture region in the whole three-dimensional model according to the new boundary to generate a cut three-dimensional model without the weak texture region.
The three-dimensional model recombination module 507 is used for reconstructing the three-dimensional model of the weak texture region and recombining the data of the cut three-dimensional model to generate a final three-dimensional model.
Some of the modules in the apparatus described herein may be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, classes, etc. that perform particular tasks or implement particular abstract data types. The application may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media including memory storage devices.
The apparatuses or modules illustrated in the embodiments of the application may be specifically implemented by a computer chip or an entity, or implemented by a product with certain functions. For convenience of description, the above devices are described as being divided into various modules by functions, and are described separately. The functionality of the various modules may be implemented in the same one or more pieces of software and/or hardware in implementing embodiments of the application. Of course, a module that implements a certain function may be implemented by a plurality of sub-modules or sub-units in combination.
The methods, apparatus or modules described herein may be implemented in a computer readable program code means for a controller in any suitable manner, for example, the controller may take the form of, for example, a microprocessor or processor and a computer readable medium storing computer readable program code (e.g., software or firmware) executable by the (micro) processor, logic gates, switches, application Specific Integrated Circuits (ASICs), programmable logic controllers and embedded microcontrollers, examples of which include, but are not limited to, the following microcontrollers: ARC 625D, atmel AT91SAM, microchip PIC18F26K20, and Silicone Labs C8051F320, the memory controller may also be implemented as part of the control logic for the memory. Those skilled in the art will also appreciate that, in addition to implementing the controller as pure computer readable program code, the same functionality can be implemented by logically programming method steps such that the controller is in the form of logic gates, switches, application specific integrated circuits, programmable logic controllers, embedded microcontrollers and the like. Such a controller may therefore be considered as a hardware component, and the means included therein for performing the various functions may also be considered as a structure within the hardware component. Or even means for performing the functions may be regarded as being both a software module for performing the method and a structure within a hardware component.
An embodiment of the present application further provides an apparatus, including: a processor; a memory for storing processor-executable instructions; when the processor executes the executable instructions, the method according to the embodiment of the application is realized.
Embodiments of the present application further provide a non-transitory computer-readable storage medium, on which a computer program or instructions are stored, which, when executed, cause the method according to the first aspect or any one of the possible implementation manners of the first aspect to be implemented.
In addition, functional modules in the embodiments of the present invention may be integrated into one processing module, or each of the modules may exist alone, or two or more modules may be integrated into one module.
The storage medium includes, but is not limited to, a Random Access Memory (RAM), a Read-Only Memory (ROM), a Cache, a Hard Disk Drive (HDD), or a Memory Card (Memory Card). The memory may be used to store computer program instructions.
From the above description of the embodiments, it is clear to those skilled in the art that the present application can be implemented by software plus necessary hardware. Based on such understanding, the technical solutions of the present application may be embodied in the form of software products or in the implementation process of data migration, which essentially or partially contributes to the prior art. The computer software product may be stored in a storage medium such as ROM/RAM, magnetic disk, optical disk, etc., and includes instructions for causing a computer device (which may be a personal computer, mobile terminal, server, or network device, etc.) to perform the methods described in the various embodiments or portions of the embodiments of the present application.
The embodiments in the present specification are described in a progressive manner, and the same or similar parts among the embodiments may be referred to each other, and each embodiment focuses on the differences from the other embodiments. All or portions of the present application are operational with numerous general purpose or special purpose computing system environments or configurations. For example: personal computers, server computers, hand-held or portable devices, tablet-type devices, mobile communication terminals, multiprocessor systems, microprocessor-based systems, programmable electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments that include any of the above systems or devices, and the like.
The above embodiments are only used to illustrate the technical solutions of the present application, and are not intended to limit the present application; although the present application has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some or all of the technical features may be equivalently replaced; such modifications or substitutions do not depart from the spirit and scope of the present disclosure.
Claims (7)
1. A method for oblique photography three-dimensional reconstruction of a weak texture scene is characterized by comprising the following steps:
generating an integral three-dimensional model according to the obtained oblique photography image;
generating a digital orthophoto map according to the integral three-dimensional model;
recognizing and extracting the range of a weak texture region of the digital orthophoto map by using the trained deep learning network model;
uniformly interpolating the range of the weak texture region to generate dense point cloud of the weak texture region;
carrying out meshing and mapping treatment on the dense point cloud of the weak texture area to generate a reconstructed three-dimensional model of the weak texture area;
cutting data in the range of the weak texture area in the integral three-dimensional model to generate a cut three-dimensional model;
and recombining the reconstructed three-dimensional model of the weak texture region and the data of the cut three-dimensional model to generate a final three-dimensional model.
2. The method of claim 1, wherein generating the global three-dimensional model from the acquired oblique photographic images comprises:
acquiring aerial image data according to oblique photography, and recording pose information of a photo in the image;
acquiring the pose information and converting the pose information into a ground-fixed coordinate system;
extracting the characteristics of the oblique photography image, and performing characteristic matching and space-three encryption to generate an integral sparse point cloud;
calculating and fusing a depth map of the overall sparse point cloud and the image to generate an overall dense point cloud;
and carrying out meshing and texture mapping processing on the integral dense point cloud to generate the integral three-dimensional model.
3. The method of claim 1, wherein the identifying and extracting the range of the weak texture region of the digital orthophotomap using the trained deep learning network model comprises:
constructing the deep learning network model according to a deep convolutional neural network DeeplabV3 +;
and selecting a plurality of weak texture scene samples to train the deep learning network model until the deep learning network model can accurately identify and extract the range of the weak texture region in the digital orthophoto map.
4. The method of claim 1, wherein said cropping data within said range of weak texture regions in said overall three-dimensional model to generate a cropped three-dimensional model comprises:
performing buffer area analysis on the boundary of the weak texture area range of the integral three-dimensional model to generate a new boundary;
and cutting data in the range of the weak texture region in the integral three-dimensional model according to the new boundary to generate a cut three-dimensional model which does not contain the weak texture region.
5. A tilted photography three-dimensional reconstruction device for weak texture scenes, comprising:
the first generation module is used for generating an integral three-dimensional model according to the acquired oblique photography image;
the second generation module is used for generating the digital orthophoto map according to the integral three-dimensional model;
the weak texture range extraction module is used for identifying and extracting the range of the weak texture region according to the digital orthographic image by using the deep learning network model;
the weak texture range interpolation module is used for uniformly interpolating the range of the weak texture region to generate dense point cloud of the weak texture region;
the reconstruction three-dimensional model module is used for generating dense point cloud after interpolating the weak texture area, and performing gridding and mapping processing to generate a reconstruction three-dimensional model of the weak texture area;
the third generation module is used for cutting the data in the weak texture area range in the integral three-dimensional model to generate a cut three-dimensional model;
and the three-dimensional model recombination module is used for recombining the reconstructed three-dimensional model of the weak texture region and the data of the cut three-dimensional model to generate a final three-dimensional model.
6. An apparatus, comprising:
a processor;
a memory for storing processor-executable instructions;
the processor, when executing the executable instructions, implements the method of any of claims 1 to 4.
7. A non-transitory computer-readable storage medium comprising instructions for storing a computer program or instructions that, when executed, cause the method of any one of claims 1 to 4 to be implemented.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210852833.5A CN115345990A (en) | 2022-07-19 | 2022-07-19 | Oblique photography three-dimensional reconstruction method and device for weak texture scene |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210852833.5A CN115345990A (en) | 2022-07-19 | 2022-07-19 | Oblique photography three-dimensional reconstruction method and device for weak texture scene |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115345990A true CN115345990A (en) | 2022-11-15 |
Family
ID=83949807
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210852833.5A Pending CN115345990A (en) | 2022-07-19 | 2022-07-19 | Oblique photography three-dimensional reconstruction method and device for weak texture scene |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115345990A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116109781A (en) * | 2023-04-12 | 2023-05-12 | 深圳市其域创新科技有限公司 | Three-dimensional reconstruction method and system |
-
2022
- 2022-07-19 CN CN202210852833.5A patent/CN115345990A/en active Pending
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116109781A (en) * | 2023-04-12 | 2023-05-12 | 深圳市其域创新科技有限公司 | Three-dimensional reconstruction method and system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11887247B2 (en) | Visual localization | |
| CN110135455B (en) | Image matching method, device and computer readable storage medium | |
| CN112085844B (en) | Unmanned aerial vehicle image rapid three-dimensional reconstruction method for field unknown environment | |
| CN109949399B (en) | Scene three-dimensional reconstruction method based on unmanned aerial vehicle aerial image | |
| Shan et al. | Accurate geo-registration by ground-to-aerial image matching | |
| EP3279803B1 (en) | Picture display method and device | |
| CN107862744B (en) | Three-dimensional modeling method for aerial image and related product | |
| JP4685313B2 (en) | Method for processing passive volumetric image of any aspect | |
| US8437501B1 (en) | Using image and laser constraints to obtain consistent and improved pose estimates in vehicle pose databases | |
| AU2011362799B2 (en) | 3D streets | |
| KR102200299B1 (en) | A system implementing management solution of road facility based on 3D-VR multi-sensor system and a method thereof | |
| US8463024B1 (en) | Combining narrow-baseline and wide-baseline stereo for three-dimensional modeling | |
| WO2023280038A1 (en) | Method for constructing three-dimensional real-scene model, and related apparatus | |
| CN111582022B (en) | Fusion method and system of mobile video and geographic scene and electronic equipment | |
| Zhang et al. | Personal photograph enhancement using internet photo collections | |
| Wang et al. | Accurate georegistration of point clouds using geographic data | |
| d'Angelo et al. | Dense multi-view stereo from satellite imagery | |
| Kim et al. | Interactive 3D building modeling method using panoramic image sequences and digital map | |
| CN113673400A (en) | Real scene three-dimensional semantic reconstruction method and device based on deep learning and storage medium | |
| Wang et al. | Terrainfusion: Real-time digital surface model reconstruction based on monocular slam | |
| CN116977596A (en) | Three-dimensional modeling system and method based on multi-view images | |
| CN115345990A (en) | Oblique photography three-dimensional reconstruction method and device for weak texture scene | |
| CN113129422A (en) | Three-dimensional model construction method and device, storage medium and computer equipment | |
| Qin et al. | A coarse elevation map-based registration method for super-resolution of three-line scanner images | |
| CN114387532A (en) | Boundary identification method and device, terminal, electronic equipment and unmanned equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |