CN117557617B

CN117557617B - Multi-view dense matching method, system and equipment based on plane priori optimization

Info

Publication number: CN117557617B
Application number: CN202410044496.6A
Authority: CN
Inventors: 范雪妍
Original assignee: Shandong Normal University
Current assignee: Shandong Normal University
Priority date: 2024-01-12
Filing date: 2024-01-12
Publication date: 2024-04-09
Anticipated expiration: 2044-01-12
Also published as: CN117557617A

Abstract

The invention aims to provide a multi-view dense matching method, system and equipment based on plane prior optimization, and relates to the technical field of dense matching. The method comprises the following steps: initializing depth information of all pixel points in a current reference image, and dividing the current reference image into a line characteristic region and a non-line characteristic region; determining a visible image set according to the visibility of the neighborhood pixel set of each pixel in the current reference image in the neighbor image set of the current reference image; constructing a new multi-view matching cost function by using the probability map model through the plane prior information, the visible image set and the current line features; updating the depth information of all pixel points in the current reference image according to the multi-view matching cost function to obtain a dense matching result of the current reference image; and fusing the dense matching results of the multiple images to obtain a multi-view dense matching result of the scene to be modeled. The invention can determine the depth information of the weak texture region by constructing a new multi-view matching cost function.

Description

Multi-view dense matching method, system and equipment based on plane priori optimization

Technical Field

The invention relates to the technical field of dense matching, in particular to a multi-view dense matching method, system and equipment based on plane prior optimization.

Background

Dense matching is a three-dimensional reconstruction technique used in computer vision. The technique aims at estimating the three-dimensional geometry of its corresponding scene from two or more two-dimensional images. Dense matching can provide more accurate depth information and generate a three-dimensional model with more detail and realism than sparse reconstruction. The core idea of dense matching is to overlap images of different viewpoints together by matching at the pixel level, thereby calculating depth information of each pixel. Such matching can be achieved based on a variety of methods, the most common implementation being with dense matching methods based on photometric consistency. The core idea of the method is to align the images of different visual angles, keep them uniform in luminosity, and match pixel levels on the basis of the images, so as to calculate the depth information of each pixel. The method improves the matching precision and robustness mainly by utilizing the luminosity information among different images. However, in the weak texture region in the image, the luminosity consistency is not reliable, the calculated depth information is inaccurate in the weak texture region, and the dense point cloud obtained after the subsequent depth fusion has the divergence phenomenon.

Disclosure of Invention

The invention aims to provide a multi-view dense matching method, a system and equipment based on plane prior optimization, which can determine depth information of a weak texture region.

In order to achieve the above object, the present invention provides the following solutions: a multi-view dense matching method based on plane prior optimization comprises the following steps: acquiring sparse point clouds and a plurality of images of a scene to be modeled; the shooting angles of different images are different; determining any image as a current reference image; determining a neighbor image set of the current reference image from the plurality of images according to the sparse point cloud of the scene to be modeled; initializing depth information of all pixel points in a current reference image by utilizing triangulation processing of sparse point cloud to obtain an initialized depth map of the current reference image; the depth information includes a depth value and a normal value; extracting a line feature set in a current reference image initialization depth map; dividing the current reference image into a line characteristic region and a non-line characteristic region according to the line characteristic set; constructing plane prior information according to the line characteristic region and the non-line characteristic region; determining a neighborhood pixel set of each pixel in the current reference image; determining a visible image set according to the visibility of the neighborhood pixel set of each pixel in the current reference image in the neighbor image set of the current reference image; constructing a new multi-view matching cost function by using the probability map model through the plane prior information, the visible image set and the current line features; updating the depth information of all pixel points in the current reference image according to the multi-view matching cost function to obtain a dense matching result of the current reference image; updating the current reference image and returning to the step of determining a neighbor image set of the current reference image from the plurality of images according to the sparse point cloud of the scene to be modeled until all the images are traversed, so as to obtain a dense matching result of the plurality of images; and fusing the dense matching results of the multiple images to obtain a multi-view dense matching result of the scene to be modeled.

A multi-view dense matching system based on planar prior optimization, comprising: the scene data acquisition module is used for acquiring sparse point clouds and a plurality of images of a scene to be modeled; the shooting angles of different images are different; the current reference image determining module is used for determining any image as a current reference image; the neighbor image set determining module is used for determining a neighbor image set of the current reference image from the plurality of images according to the sparse point cloud of the scene to be modeled; the initialization depth module is used for initializing depth information of all pixel points in the current reference image by utilizing the triangulation processing of the sparse point cloud to obtain an initialization depth map of the current reference image; the depth information includes a depth value and a normal value; the line feature set extraction module is used for extracting a line feature set in the initialization depth map of the current reference image; the line characteristic region dividing module is used for dividing the current reference image into a line characteristic region and a non-line characteristic region according to the line characteristic set; the plane prior information construction module is used for constructing plane prior information according to the line characteristic region and the non-line characteristic region; the neighborhood pixel set determining module is used for determining a neighborhood pixel set of each pixel in the current reference image; and the visible image set determining module is used for determining a visible image set according to the visibility of the neighborhood pixel set of each pixel in the current reference image in the neighbor image set of the current reference image.

The multi-view matching cost function determining module is used for constructing a new multi-view matching cost function by utilizing the probability map model and utilizing the plane prior information, the visible image set and the current line features; the sub-dense matching result determining module is used for updating the depth information of all pixel points in the current reference image according to the multi-view matching cost function to obtain a dense matching result of the current reference image; the total dense matching result determining module is used for updating the current reference image and returning to the neighbor image set determining module until all images are traversed to obtain dense matching results of a plurality of images; and the multi-view dense matching module is used for fusing dense matching results of the plurality of images to obtain a multi-view dense matching result of the scene to be modeled.

An electronic device comprising a memory for storing a computer program and a processor that runs the computer program to cause the electronic device to perform the multi-view dense matching method based on planar prior optimization.

According to the specific embodiment provided by the invention, the invention discloses the following technical effects: the invention aims to provide a multi-view dense matching method, a system and equipment based on plane prior optimization, and aims to solve the problem of inaccurate depth information estimation of a weak texture region, a multi-view dense matching method based on plane prior auxiliary optimization of scene line feature constraint is designed based on an ACMP algorithm, two different stages of depth information optimization methods are provided, and firstly, sparse point cloud initialization depth information of sparse reconstruction is utilized, so that rapid propagation of depth information with higher precision is realized; secondly, a high-quality plane prior model is constructed by utilizing line characteristic constraint of the scene, the depth information of the weak texture region is optimized, and accurate calculation of the depth information of the weak texture region is realized.

Drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions of the prior art, the drawings that are needed in the embodiments will be briefly described below, it being obvious that the drawings in the following description are only some embodiments of the present invention, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.

Fig. 1 is a flowchart of a multi-view dense matching method based on planar prior optimization in embodiment 1 of the present invention.

Fig. 2 is a schematic diagram of a minimum bounding box area formed by sparse points in a reference image in embodiment 1 of the present invention.

Fig. 3 is a schematic diagram of an included angle formed by a center of a camera and a sparse point in embodiment 1 of the present invention.

Fig. 4 is a projection distance of a unit sphere centered on a sparse point on an image in embodiment 1 of the present invention.

Fig. 5 is a schematic diagram of an iteration result of random initialization of depth information in embodiment 1 of the present invention.

Fig. 6 is a schematic diagram of a result of random initialization secondary iteration of depth information in embodiment 1 of the present invention.

Fig. 7 is a schematic diagram of three iteration results of random initialization of depth information in embodiment 1 of the present invention.

Fig. 8 is a schematic diagram of a result of one iteration of rule initialization of depth information in embodiment 1 of the present invention.

Fig. 9 is a schematic diagram of a result of a second iteration of rule initialization of depth information in embodiment 1 of the present invention.

Fig. 10 is a schematic diagram of the result of three iterations of rule initialization of depth information in embodiment 1 of the present invention.

Fig. 11 is a schematic diagram illustrating the judgment of the included angle of the line segment in embodiment 1 of the present invention.

Fig. 12 is a schematic diagram illustrating the judgment of the vertical distance of the line segment in embodiment 1 of the present invention.

Fig. 13 is a schematic diagram illustrating the determination of the end point distance of a line segment according to embodiment 1 of the present invention.

Fig. 14 is a schematic diagram illustrating line segment registration determination in embodiment 1 of the present invention.

Fig. 15 is a schematic diagram of segment endpoint clustering determination in embodiment 1 of the present invention.

Fig. 16 is a schematic diagram of an original image in embodiment 1 of the present invention.

Fig. 17 is a schematic view of a mask in embodiment 1 of the present invention.

Fig. 18 is a schematic diagram of a parallel propagation strategy of "red and black checkerboard" in embodiment 1 of the present invention.

Fig. 19 is a schematic diagram of a propagation path of a parallel propagation strategy of "red and black checkerboard" in embodiment 1 of the present invention.

Fig. 20 is a schematic diagram of a neighborhood pixel in embodiment 1 of the present invention.

Fig. 21 is a schematic diagram of a result of extracting plane prior information of a region one in the Dortmund dataset by the ACMP method in embodiment 1 of the present invention.

Fig. 22 is a schematic diagram of a planar prior information extraction result of region one in the Dortmund dataset according to embodiment 1 of the present invention.

Fig. 23 is a schematic diagram of a result of extracting planar prior information of the region two in the Dortmund dataset by the ACMP method in embodiment 1 of the present invention.

Fig. 24 is a schematic diagram of the extraction result of planar prior information of the region two in the Dortmund dataset according to embodiment 1 of the present invention.

Fig. 25 is a schematic diagram of the ACMP method of embodiment 1 of the present invention for extracting planar prior information of region three in the Garden dataset.

FIG. 26 is a graph showing the result of planar prior information extraction for region three in the Garden dataset according to example 1 of the present invention.

Fig. 27 is a schematic diagram of the ACMP method of embodiment 1 of the present invention for extracting planar prior information of region four in the Garden dataset.

FIG. 28 is a graph showing the result of planar prior information extraction for region four in the Garden dataset according to example 1 of the present invention.

Fig. 29 is a schematic diagram showing the result of extracting planar prior information of the region five in the Central-Urban dataset by the ACMP method in embodiment 1 of the present invention.

Fig. 30 is a schematic diagram of the extraction result of planar prior information of the region five in the Central-Urban dataset according to embodiment 1 of the present invention.

Fig. 31 is a schematic diagram of the ACMP method of embodiment 1 of the present invention for extracting planar prior information of region six in the Central-Urban dataset.

Fig. 32 is a schematic diagram of the extraction result of planar prior information of the region six in the Central-Urban dataset according to embodiment 1 of the present invention.

FIG. 33 is a diagram showing the result of calculating the depth information of the region one in the Dortmuld dataset according to the ACMP method of embodiment 1 of the present invention.

Fig. 34 is a schematic diagram of a depth information calculation result of the region one in the Dortmund dataset according to embodiment 1 of the present invention.

Fig. 35 is a schematic diagram of a calculation result of depth information of a region two in the Dortmund dataset according to the ACMP method in embodiment 1 of the present invention.

Fig. 36 is a schematic diagram of a depth information calculation result of the region two in the Dortmund dataset according to embodiment 1 of the present invention.

Fig. 37 is a schematic diagram showing the calculation result of the ACMP method of embodiment 1 of the present invention on the depth information of the region three in the Garden dataset.

Fig. 38 is a schematic diagram of the depth information calculation result of the region three in the Garden dataset according to embodiment 1 of the present invention.

Fig. 39 is a schematic diagram showing the calculation result of the ACMP method of the embodiment 1 of the present invention on the depth information of the region four in the Garden dataset.

Fig. 40 is a schematic diagram of a depth information calculation result of the region four in the Garden dataset according to embodiment 1 of the present invention.

Fig. 41 is a schematic diagram showing the calculation result of the ACMP method of the embodiment 1 of the present invention for the depth information of the region five in the Central-Urban dataset.

Fig. 42 is a schematic diagram of a depth information calculation result of a region five in a Central-Urban dataset according to embodiment 1 of the present invention.

Fig. 43 is a schematic diagram showing the result of calculating depth information of a region six in a Central-Urban dataset by the ACMP method in embodiment 1 of the present invention.

Fig. 44 is a schematic diagram of a depth information calculation result of a region six in a Central-Urban dataset according to embodiment 1 of the present invention.

FIG. 45 is an elevation view of the ACMP method of example 1 for region one of the Dortmuld dataset.

FIG. 46 is an elevation view of region one of the Dortmuld dataset of example 1 of the present invention.

FIG. 47 is an elevation view of the ACMP method of example 1 for region two in the Dortmuld dataset.

FIG. 48 is an elevation view of region two in the Dortmuld dataset of example 1 of the present invention.

FIG. 49 is an elevation view of the ACMP method of example 1 of the present invention versus region three within the Garden dataset.

FIG. 50 is an elevation view of region three within the Garden dataset of example 1 of the present invention.

FIG. 51 is an elevation view of the ACMP method of example 1 of the present invention versus region four within the Garden dataset.

FIG. 52 is an elevation view of region four in the Garden dataset of example 1 of the present invention.

FIG. 53 is an elevation view of the ACMP method of example 1 of the present invention for region five within the Central-Urman dataset.

FIG. 54 is an elevation view of region five within the Central-Urman dataset of example 1 of the present invention.

FIG. 55 is an elevation view of the ACMP method of example 1 of the present invention for region six within the Central-Urman dataset.

FIG. 56 is an elevation view of region six in the Central-Urman dataset of example 1 of the present invention.

Detailed Description

The following description of the embodiments of the present invention will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present invention, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.

In order that the above-recited objects, features and advantages of the present invention will become more readily apparent, a more particular description of the invention will be rendered by reference to the appended drawings and appended detailed description.

Example 1: as shown in fig. 1, the present embodiment provides a multi-view dense matching method based on planar prior optimization, including: step 101, acquiring sparse point cloud and multiple images of a scene to be modeled. The shooting angles of different images are different.

Step 102, determining any image as the current reference image.

Step 103, determining a neighbor image set of the current reference image from the plurality of images according to the sparse point cloud of the scene to be modeled.

And 104, initializing the depth information of all pixel points in the current reference image by utilizing the triangulation processing of the sparse point cloud to obtain the initialized depth map of the current reference image. The depth information includes a depth value and a normal value.

Step 105, extracting a line feature set in the initialization depth map of the current reference image.

And 106, dividing the current reference image into a line characteristic region and a non-line characteristic region according to the line characteristic set.

And 107, constructing plane prior information according to the line characteristic region and the non-line characteristic region.

Step 108, determining a neighborhood pixel set of each pixel in the current reference image.

Step 109, determining a visible image set according to the visibility of the neighborhood pixel set of each pixel in the current reference image in the neighborhood image set of the current reference image.

And 1010, constructing a new multi-view matching cost function by using the prior plane information, the visible image set and the current line features by using the probability map model.

And 1011, updating the depth information of all pixel points in the current reference image according to the multi-view matching cost function to obtain a dense matching result of the current reference image.

Step 1012, updating the current reference image and returning to step 103 until all the images are traversed, so as to obtain dense matching results of the plurality of images.

And step 1013, fusing dense matching results of the multiple images to obtain a multi-view dense matching result of the scene to be modeled.

Step 103, including:

step 103-1, determining a visible sparse point set of the current reference image from a sparse point cloud of a scene to be modeled.

Step 103-2, determining any image to be determined as the current image to be determined. The pending images are a plurality of images other than the current reference image.

And 103-3, determining a visible sparse point set of the current undetermined image from the sparse point cloud of the scene to be modeled.

Step 103-4, determining an intersection of the visible sparse point set of the current reference image and the visible sparse point set of the current pending image as a common visible sparse point set of the current reference image and the current pending image.

And 103-5, determining a correlation score of the current reference image and the current image to be determined according to the common visible sparse point set.

Step 103-6, updating the current image to be determined, and returning to step 103-1 until all the images to be determined are traversed, and obtaining the correlation scores of the current reference image and all the images to be determined.

And 103-7, arranging the undetermined images in descending order according to the relevance scores.

Step 103-8, determining the preset number of undetermined images as neighbor image sets of the current reference image.

Step 104, including: step 104-1, projecting the visible sparse point set of the current reference image onto the current reference image to obtain a projection point set of the current reference image.

And 104-2, triangulating the projection point set of the current reference image by using a Delaunay algorithm to generate a two-dimensional grid.

And 104-3, constructing a three-dimensional grid according to the sparse depth information of the projection point set of the current reference image.

And 104-4, determining the image pose of the current reference image by using sparse reconstruction.

Step 104-5, determining any pixel point in the current reference image as the current pixel point.

And 104-6, calculating the current projection light corresponding to the current pixel point according to the image pose.

And 104-7, projecting the current projection light to the three-dimensional grid, and determining a triangular patch intersecting the current projection light in the three-dimensional grid as a current triangular patch.

And 104-8, determining a plane equation of the current triangular patch according to the 3 vertex coordinates of the current triangular patch.

And 104-9, determining the depth value and the normal value of the current pixel point according to the coefficient of the plane equation of the current triangular patch.

Step 104-10, updating the current pixel point and returning to step 104-6 until all the pixel points in the current reference image are traversed, and obtaining the initialization depth map of the current reference image.

Step 105, comprising: and 105-1, extracting a plurality of line features in the initialization depth map of the current reference image by using an LSD straight line detection method to obtain an initial line feature set. The line features are line segments.

Step 105-2 of deleting line features in the initial set of line features having a length less than the length threshold.

Step 105-3, connecting collinear line features in the initial line feature set.

Step 105-4 of determining any line feature in the initial set of line features as the current line feature.

And 105-5, determining a plurality of line features outside the current line feature in the initial line feature set as line features to be matched.

And 105-6, determining any line feature to be matched as the current line feature to be matched.

And 105-7, determining the included angle between the current line characteristic and the current line characteristic to be matched as the current included angle.

And 105-8, updating the characteristics of the current line to be matched when the current included angle is greater than or equal to the included angle threshold value, and returning to the step 105-7.

And 105-9, when the current included angle is smaller than the included angle, determining that the current line characteristic and the current line characteristic to be matched are the current approximate parallel line characteristic pair.

And 105-10, constructing a linear equation of the current line feature to be matched according to the two end point coordinates of the current line feature to be matched.

And 105-11, determining the distance from one end point of the current line characteristic to be matched as a first distance according to a linear equation of the current line characteristic to be matched.

And 105-12, determining the distance from the other end point of the current line characteristic to be matched as a second distance according to the linear equation of the current line characteristic to be matched.

Step 105-13 of determining the mean of the first distance and the second distance as the vertical distance of the current pair of approximately parallel line features.

And 105-14, when the vertical distance is smaller than the vertical distance threshold value, fitting the current approximate parallel line characteristic pair by using a least square method to obtain a fitting line characteristic.

Step 105-15 of replacing the current pair of approximately parallel line features in the initial set of line features with the fitted line feature.

And step 105-16, taking the fitting line characteristic as the current line characteristic, and returning to step 105-6 until all the line characteristics to be matched are traversed.

Step 105-17, updating the current line feature, and returning to step 105-5 until the initial line feature set is traversed, and determining the to-be-aligned feature set.

And step 105-18, determining any line characteristic in the to-be-wired characteristic set as the current line characteristic.

Step 105-19 of determining any endpoint of the current line feature as the current endpoint.

And 105-20, determining the end points of all the line features except the current line feature in the to-be-determined line feature set as the current end point set.

And step 105-21, determining a plurality of endpoints in the current endpoint set, the distances between the endpoints and the current endpoint being smaller than a distance threshold, as endpoints to be combined.

And 105-22, determining the current endpoint and the multiple endpoints to be combined as a point set to be combined.

And 105-23, determining the line characteristic of the point set to be combined as the line characteristic set to be combined.

And 105-24, determining the coordinate mean value of the point set to be merged as the coordinate of the merging point.

And 105-25, connecting all points in the point set to be combined with the combining point to obtain the combining line characteristic.

And 105-26, replacing the to-be-merged line feature set in the to-be-merged line feature set by adopting the merged line feature, and returning to 105-18 until the to-be-merged line feature set is traversed to obtain the line feature set in the current reference image initialization depth map.

Step 107, including: and 107-1, performing primary matching by using an ACMH method, and determining that a matching point with the confidence coefficient smaller than a first confidence coefficient threshold value in the current reference image is a pending structural point.

Step 107-2, determining all undetermined structural points in the line feature area as selected structural points.

And step 107-3, determining candidate structural points with the confidence degree smaller than a second confidence degree threshold value in the non-line characteristic region as selected structural points. The second confidence threshold is less than the first confidence threshold.

And 107-4, constructing a network by adopting a Delaunay algorithm based on the plurality of selected structure points to generate a plurality of triangle primitives.

And 107-5, determining any triangle primitive as the current triangle primitive.

And 107-6, determining current plane prior information according to the 3 vertexes of the current triangle primitive.

And 107-7, constructing a plane equation of the current triangle primitive.

And step 107-8, determining the current plane prior information as the plane prior information of all pixels in the current triangle primitive.

And step 107-9, updating the current triangle primitive, and returning to step 107-6 until all triangle primitives are traversed, and determining the plane prior information of all pixels in the current reference image.

Step 1010, comprising: step 1010-1, determining any pixel as the current pixel.

Step 1010-2 of determining depth information of the current pixel and depth information of each neighboring pixel in the neighboring pixel set of the current pixel as candidate hypothesis sets.

Step 1010-3, determining the photometric consistency of the current pixel with the visible image when selecting each candidate hypothesis in the candidate hypothesis set to construct a matching cost matrix.

And 1010-4, determining the final matching cost of each candidate hypothesis according to the matching cost matrix.

Step 1010-5, updating the current pixel and returning to step 1010-4 to obtain the final matching cost of selecting different candidate hypotheses for each pixel.

And 1010-6, constructing a plane optimization probability map model according to the final matching cost of different candidate hypotheses selected by each pixel and the plane prior information of each pixel.

Specifically, the multi-view dense matching method based on planar prior optimization in this embodiment may generally include: (1) calculation of image combination: and taking each image and the sparse point cloud as data sources, sequentially taking the images as reference images, calculating scores based on factors such as common visible sparse points, projection areas, included angles and the like between the reference images and the rest images, and selecting a group of neighbor image sets for the images. (2) initialization of depth information: and using the sparse point cloud and the image pose meter as data input, and finishing the depth information initialization task of each reference image through triangularization of the sparse point cloud. (3) scene line feature region extraction: the line feature area of the scene is a line type buffer area which is built by taking the line feature extracted from the image as the center and taking a certain pixel size as the radius. In an image, the weak texture regions typically have strong planar characteristics, and these regions extract fewer line features and are therefore typically non-line feature regions. In contrast, strongly textured regions typically have significant structural features, and these regions extract more line features and are therefore typically line feature regions. Firstly, extracting straight line segments through steps of scale scaling, gradient calculation, region growth and the like; and then the line segment is closed through the steps of fine segment filtering, the connection of the line segments positioned on the same straight line, the connection of the line segments which are not positioned on the same straight line but have similar end point distances, and the like. (4) depth value optimization strategy of plane prior optimization: the method mainly comprises the steps of extracting plane prior information, selecting images at pixel level and constructing a plane optimization model. Firstly, constructing plane prior information by using a matching point with good confidence as a structural point, then selecting a visible image set by using the visibility of a neighborhood pixel of each pixel in a neighbor image, and secondly, combining the plane prior information, the luminosity consistency and scene line characteristics extracted in the previous step by using a probability map model to construct a new multi-view matching cost function, so as to finish optimization of depth values and finally achieve the aim of improving the depth value precision of a weak texture region.

1. And (5) dense matching pretreatment.

1.1 And (5) calculating image combination.

When multi-view dense matching is performed, each image is sequentially used as a reference image, the image combination is calculated by selecting a group of neighbor image sets for each reference image,is convenient for use in subsequent steps such as multi-view dense matching, depth fusion and the like. According to the global view selection algorithm, a group of image sets with good neighborhood is selected for each image in the scene. These images are ideal neighbor images in terms of scene content, appearance and scale. The core idea is to calculate a correlation score based on common visible sparse points p between a reference image and the rest images, sort the scores from high to low, and select N from the image set according to the sorting of the scores _best The sheet image is used as a neighbor image of the reference image. In the present embodiment take. Correlation score->The calculation formula is shown as formula (1):

。

in the formula (1), the components are as follows,is a reference image I _r Sparse set of points seen, ->Is image I _n A set of sparse points is seen.Is a reference image I _r And image I _n The area weight factor formed by the common visible sparse points p is calculated as formula (2).

(2)。

Wherein,the minimum bounding box area formed in the reference image for the commonly visible sparse points is shown in fig. 2; / >Is a reference image I _r Is provided. In FIG. 2, the solid line box represents the reference image I _r The dashed box represents image I _n Circles marked with orange characters represent visible sparse points of the reference image; circles marked with "purple" words represent images I _n Sparse points are visible; circles marked with "blue" words represent reference image and image I _n Sparse points are commonly visible.

In the formula (1), the components are as follows,is a reference image I _r And image I _n The angle weight factor formed on the common visible sparse point p is calculated as formula (3).

(3)。

Wherein,representing reference image I _r And image I _n The corresponding camera center forms an included angle with the sparse point p, as shown in fig. 3. In fig. 3, circles marked with "orange" words represent visible sparse points of the reference image; circles marked with "purple" words represent images I _n Sparse points are visible; circles marked with "blue" words represent reference image and image I _n Sparse points are commonly visible. In practical use +.>Should be taken within a certain range, if +.>Smaller, reference image I _r And image I _n The imaging geometry condition formed is poor, which causes inaccurate estimation of depth values; if->Larger, reference image I _r And image I _n The visual angle difference between the two is larger, and inaccurate matching cost calculation is easy to cause. Therefore, the present embodiment selects the minimum angle +. >Maximum angle->。

In the formula (1), the components are as follows,is a reference image I _r And image I _n The scale weight factor formed on the common visible sparse point p is calculated as formula (4).

(4)。

Wherein,，representing a unit sphere centered on a sparse point p in an image I _i The projection distance on the lens is shown in fig. 4. When the unit sphere with the sparse point p as the center is positioned in the reference image I _r And image I _n When the scale difference of the projection distances on the image I is small, the sparse point p is positioned on the image I _r And I _n The more accurate the computation of the matching cost.

1.2 Initialization of depth information.

Dense matching algorithms based on photometric consistency do not work well in terms of using sparse point clouds. The depth information is for example randomly initialized only from the depth range of the sparse point cloud. Although dense matching algorithms based on photometric consistency do not require very good initial values, this increases the number of iterations of the algorithm, reducing robustness, especially when the field depth range is large. To overcome these problems, the present embodiment initializes depth information using sparse point clouds obtained by a sparse reconstruction step. This may generate more accurate initial depth information for each image. The process includes regularization of the sparse point cloud and rule initialization depth information.

Sparse point cloud regularization refers to utilizing image I _i Visible sparse point cloudThe general shape of the scene is acquired. The process is divided into three steps: first, sparse point cloud->Projection onto image I _i Obtaining corresponding projection point set +.>The method comprises the steps of carrying out a first treatment on the surface of the Then triangulating the projection points by using a Delaunay algorithm to generate a two-dimensional grid; finally, combining the depth information of the projection points to convert the two-dimensional grid into a three-dimensional grid M _i . Specifically, for each projection point, its depth information may be estimated by a sparse reconstruction algorithm. After the two-dimensional grid is generated by using the Delaunay algorithm, the connection relation between projection points is known. Combining the depth information of the projection points with the connection relation between the projection points, the two-dimensional grid can be converted into a three-dimensional grid, so that the approximate shape of the scene is obtained.

Rule initialization depth information is based on image I _i Calculated three-dimensional grid M _i For image I _i Is initialized. The process is divided into three steps: firstly, calculating an image I according to the image pose obtained by sparse reconstruction _i Middle pixelCorresponding projection ray->. Projection light +.>Projected onto a three-dimensional grid M _i In (1) obtaining and projecting light->Intersecting triangular patches- >The method comprises the steps of carrying out a first treatment on the surface of the Then calculating the mathematical expression of the plane of the triangular patch based on the vertex coordinates of the triangular patchThe method comprises the steps of carrying out a first treatment on the surface of the Finally, pixel +.> Initial depth information, i.e. depth valuesAnd normal value->。

（5）。

Wherein,representing projection light +.>And triangular face piece->Is defined by the intersection coordinates of the two points. For image I _i Repeating the above calculation to obtain image I _i Is included in the depth information.

For a strong texture region with obvious characteristics in an image, an initial value close to a true value can be obtained through random initialization and rule initialization. However, the rule initialization uses the sparse point cloud information with better scene structure retention, so that compared with random initialization, the rule initialization needs fewer iteration times and is more robust. For weak texture regions with insignificant features in the image, rule initialization can obtain initial values closer to true values than random initialization. The iterative results of the random initialization and rule initialization of depth information are shown in fig. 5-10.

2. Multi-view dense matching based on scene line feature constraints.

The depth map of each oblique image obtained by multi-view dense matching is an important data source based on three-dimensional reconstruction of the oblique images. However, there are two general problems in the oblique image processing process: (1) The photometric consistency of the weak texture regions in the oblique images is no longer reliable, resulting in inaccurate matching results between images. (2) The ACMP algorithm does not consider the structural features of the scene when constructing the planar prior model, resulting in inaccurate planar prior information obtained. Aiming at the problems in the two aspects, the embodiment is based on an ACMP algorithm, and designs a multi-view dense matching method based on plane prior auxiliary optimization of scene line feature constraint. The key idea is that the line characteristics of the images can generally describe the structural characteristics of the scene, so that the images are divided into line characteristic areas and non-line characteristic areas based on the line characteristics of each image. And selecting matching points with different confidence intervals as structural points for different areas, and constructing a Delaunay triangular network to generate high-quality plane prior information. And constructing a plane optimization probability map model by combining the plane prior information and the luminosity consistency between images, so that the depth information of the weak texture region tends to the depth information of the plane structure, and the problem of inaccurate estimation of the depth information of the weak texture region is solved. The main flow comprises scene line characteristic region extraction and multi-view dense matching of plane prior auxiliary optimization.

1. Scene line feature region extraction.

The line feature area of the scene is a line type buffer area which is built by taking the line feature extracted from the image as the center and taking a certain pixel size as the radius. In an image, the weak texture regions typically have strong planar characteristics, and these regions extract fewer line features and are therefore typically non-line feature regions. In contrast, strongly textured regions typically have significant structural features, and these regions extract more line features and are therefore typically line feature regions. Selecting a small number of matching points with high confidence as structural points in a non-linear characteristic region in the image, and ensuring that a weak texture region generates larger triangle primitives, so that larger plane structure constraint is adopted in the weak texture region, and depth information at a plane structure is kept flat; on the contrary, a large number of matching points with credibility are selected from the line characteristic area in the image to serve as structural points, and the strong texture area is guaranteed to generate finely divided triangle primitives, so that small plane structure constraint is adopted in the strong texture area, and depth information mutation at the edge structure is reserved.

Currently, the common line feature detection methods include: canny edge detection algorithm, hough transformation detection method, EDLines straight line detection method and LSD straight line detection method.

(1) Canny edge detection algorithm: the basic idea of the Canny edge detection algorithm is to detect edges in an image through multiple steps using gradient information in the image. The Canny algorithm can accurately detect edges in the image, fine extraction is carried out on the edges, noise in the image is removed through Gaussian filtering, and accuracy of edge detection can be effectively improved. However, the Canny algorithm needs to carry out convolution and operation on the image for a plurality of times, so that the calculated amount is large, and the processing speed is low. The threshold in the algorithm needs to be adjusted according to a specific image, and how to select the threshold is not explicitly specified, and needs to be tried and adjusted according to actual situations. In general, the Canny edge detection algorithm is a high-precision image edge detection algorithm, but in practical application, the problems of large calculation amount, difficult parameter setting and the like need to be considered.

(2) Hough transform detection method: the Hough transformation is to transform an image coordinate space into a parameter space, and transform a curve in an original image space into a point in the parameter space through a curve expression form by utilizing the duality of points and lines. This translates the problem of curve detection in the original image into the problem of finding peaks in the parameter space. The Hough transformation detection method has good detection effect on irregular shapes, and can detect objects with any shapes, including irregular shapes such as straight lines, circles and polygons. However, the method has high computational complexity, and each pixel point needs to be calculated, so that the computational complexity is high when processing a large image, and the method usually has computational errors.

(3) EDLines straight line detection method: the basic idea of the EDLines straight line detection method is that all initial straight line segments are screened out by traversing on a pixel chain generated in advance, and are fitted by a fitting method, so that the initial straight line segments are more complete straight line segments. According to the method, the straight line segment is extracted through connecting the edge points, so that a relatively accurate straight line segment detection result can be obtained, and the method has relatively high calculation speed. However, the method is sensitive to noise and partial shielding, and the accuracy of the straight line segment detection result is easily affected. The straight line segment detection result of the method is related to parameter selection, and a certain parameter adjustment is needed to obtain a good detection effect.

(4) LSD straight line detection method: the LSD straight line detection algorithm is a straight line detection algorithm based on image gradients, can detect straight lines at sub-pixel level in linear time, improves calculation speed by combining pixel points in similar gradient directions, and has self-adaptive parameter adjustment and controllable error rate. Thus, the LSD line detection algorithm is considered a milestone in modern line detection algorithms. However, this algorithm has a limit on the length of the straight line, and a long straight line is easily divided into a plurality of short straight lines.

Among the four line feature detection algorithms, the Canny edge detection algorithm and the Hough transform detection method have higher computational complexity and longer operation time, so that the two methods are not suitable for line feature extraction in the method of the embodiment. In contrast, the LSD straight line detection method and the EDLines straight line detection method have higher detection speeds, but the LSD straight line detection method can detect a straight line at a sub-pixel level by using adaptive parameters. Therefore, the present embodiment extracts line features in an image based on the LSD linear detection method, and performs binary division on the image based on the line features, thereby dividing the image into line feature regions and non-line feature regions.

The LSD straight line detection method mainly comprises four key steps, namely: scaling, gradient calculation, region growing, and straight line extraction.

(1) Scaling: the image is scaled by gaussian downsampling, thereby slowing down or solving the problems of aliasing and quantization artifacts occurring in the image.

(2) Gradient calculation: gradient calculation typically uses pixels to the right and below each pixel to calculate the magnitude of the gradient corresponding to that pixelAnd direction->The specific calculation formulas are shown as formula (6) and formula (7). This approach aims to use as few other pixels as possible, thereby enhancing its robustness to noisy images.

（6）。

（7）。

Wherein,and->The calculation formula of (2) is shown as formula (8).Representing the magnitude of the gradient in the x-direction at pixel (x, y);Representing the magnitude of the gradient in the y-direction at pixel (x, y).

（8）。

Wherein,for pixels +.>Image gray value at the position.

(3) Region growth: through greedy algorithm, adjacent pixel points with consistent gradient directions are connected to form a connected domain, and a rectangular frame is generated on the basis of the connected domain, wherein the connected domain is also called a linear supporting region. After the rectangular frame is generated, judging whether the connected domain needs to be disconnected according to rules according to the size of the rectangular degree, so as to form a plurality of connected domains with larger rectangular degree.

(4) And (3) straight line extraction: rectangular improvement and screening are carried out on all the generated connected domains, and the conditions are kept to be satisfiedAnd the connected domain of the condition is the final straight line detection result.The calculation formula of (2) is shown as formula (9).

(9)。

Wherein,is the image size of the image, +.>The number k of the pixel points in the linear supporting area _lsd The number of pixel points perpendicular to the direction of the linear support area, +.>Improving accuracy for rectangle->Representation->Is a number of different values of (a).

Since the LSD linear detection method is a local detection algorithm, the extracted line features are scattered and discontinuous. Therefore, in order to better embody the structural features of the scene, the embodiment performs segment connection on the line features extracted by the LSD straight line detection method. The main steps include the following three steps.

(1) In order to better represent the structure of the scene and reduce the computational complexity, the line segment set extracted by the LSD linear detection method is filtered to remove the length less than the length threshold valueIs a line segment of (c). These shorter line segments do not effectively represent the structure of the scene, but rather increase the computational effort of the subsequent steps. Therefore, the segments should be filtered and removed before they are connected.

(2) And connecting the line segments on the same straight line after the line segment set is filtered. Traversing all the line segments in the line segment set in turn, and selecting any one line segmentOther line segment in line segment set->And (3) performing matching verification, judging whether the two line segments can be connected according to the conditions of included angles, distances and the like between the line segments, wherein the specific judging process is as follows:

(1) judging the included angle of the line segments: according to line segments、Slope of +.>、Calculating the included angle between the two line segments>The calculation formula is shown as formula (10).

（10）。

If it isLess than the angle threshold->It is determined that the two line segments are approximately parallel as shown in fig. 11.

(2) Judging the vertical relative distance: through line segmentsCalculating line segment +.>Straight line equation of (2). Calculate line segment +.>Is>、To line segment->Distance of->、The calculation formula is shown as formula (11).

(11)。

Taking out、Is taken as the mean value of line segment->、Vertical distance between>. If the vertical distance->Less than distance threshold->It is illustrated that the two line segments are located on the same horizontal line, as shown in fig. 12.

(3) Endpoint distance judgment: calculation pointTo the point->、Distance between、Point->To the point->、Distance between->、. From->、、、Finding out the maximum and minimum, and recording as +.>、The calculation formula is shown as formula (12).

(12)。

If it isLess than distance threshold->Two line segments are illustrated as being adjacent as shown in fig. 13. />

If it isGreater than distance threshold->But there is a superposition of the two line segmentsIt is also possible to illustrate that two line segments are adjacent, as shown in fig. 14.

And if the two line segments meet the three conditions, fitting and optimizing the two line segments by using a least square method to synthesize an optimal straight line segment. The least squares fit optimization is performed as follows.

By straight lineIs converted into +.>Form of (1), wherein->，. According to the least squares principle, there is a function as in equation (13).

(13)。

Wherein,representing the number of endpoints involved in the fit, in this example +.>. When this function takes the minimum, two parameters A, B of the fitted line are found. According to the extremum theorem of the function, there is a system of equations as in equation (14).

(14)。

And solving the equation set to obtain the value of the parameter A, B, and obtaining the expression of the best fit straight line. According to line segments 、The two end points which are farthest apart are calculated to fit the end points of the line segment. After calculation of the fitted line segment, the line segment +.>、And add the fitted line segment. Then, the fitted line segment is taken as new +.>And matching and verifying with other line segments. This process is repeated until all segments in the set of segments are used as +.>And performing matching verification to obtain a new line segment set.

(3) After a new segment set is obtained, segments that are not in the same straight line but have similar end points are connected, as shown in fig. 15. Traversing all the line segments in the new line segment set in turn, and calculating any one line segmentIs +.>Is less than the distance threshold +.>Line segments of (1) constitute->Collecting the corresponding line segment set and adding line segment +.>Adding the collection line segment set. And traversing all the line segments in the collection line segment set in sequence, and calculating the intersection point of the straight line where the line segment is located and the straight lines where other line segments in the line segment set are located. The calculated average value of all intersection point coordinatesAs a new endpoint for collecting line segments in a line segment set.

After connecting the line features extracted by the LSD straight line detection method by the above steps, a buffer (Buffer zone of feature line, BZFL) is constructed based on the empirical value (5 pixels). Finally, the image is subjected to binary division according to the formula (15).

(15)。/>

Wherein,representing pixel coordinates, pixel values when the pixel coordinates are located in the non-line feature regionThe method comprises the steps of carrying out a first treatment on the surface of the When the pixel coordinates are located in the line feature region, the pixel value +.>. The resulting mask patterns are shown in FIGS. 16-17, front and back.

Multi-view dense matching of plane prior optimization.

The dense matching method based on luminosity consistency mainly relies on luminosity consistency among images to evaluate matching accuracy. Because the strong texture areas in the images have obvious scene structure characteristics, the luminosity consistency of the areas is reliable, and accurate depth information can be obtained. However, since the weak texture region generally has a planar characteristic, photometric consistency is no longer reliable in the weak texture region, and there is a problem in that the obtained depth information is inaccurate. The core idea of the multi-view dense matching method based on plane priori optimization is to construct a probability map model based on plane priori information and combining luminosity consistency, so that the depth information of a weak texture region tends to the depth information of a plane structure, and the problem of inaccurate estimation of the depth information of the weak texture region is optimized. The main flow comprises the steps of plane priori information extraction, pixel-level image selection and plane optimization model construction.

The extraction of the plane prior information is to construct the plane prior information by using the selected matching points with better confidence as the structural points. In this embodiment, an ACMH method is adopted to perform initial matching, an initial depth map of each image is obtained, and matching points with different confidence intervals are selected according to a binarization result of the image. The confidence coefficient value is greater than or equal to 0, taking the second confidence coefficient threshold value as 0.05 and the first confidence coefficient threshold value as 0.1 as an example, the specific steps are as follows: firstly, selecting a matching point with the confidence coefficient smaller than 0.1 as a candidate structural point; and then judging the region where the candidate structural point is located according to the pixel coordinates of the candidate structural point. If the structural points are non-linear characteristic areas, structural points with confidence intervals of 0-0.05 are selected, so that the quality of the structural points is improved, and the number of the structural points is reduced. If the structural points are line characteristic areas, selecting structural points with confidence intervals of 0-0.1, and improving the number of the structural points on the premise of keeping the accuracy of depth information; and finally, constructing a network by adopting a Delaunay algorithm and generating a triangle primitive. For each triangle primitive, plane prior information is calculated using its corresponding three vertices, and plane equations in object space. Pixels within the same triangle primitive share the same plane prior information.

The image selection at the pixel level is to select a visible image set by utilizing the visibility of the neighbor pixels of each pixel in the neighbor image. The present embodiment refers to the "red-black checkerboard" parallel propagation strategy to divide the reference image pixels into "red-black" pixels, as shown in fig. 18. In fig. 18, circles labeled with "red" words represent "red" pixels; circles filled with black represent "black" pixels; in the parallel propagation process, the parallel propagation of the depth information of the red pixel and the black pixel is sequentially performed, namely, the depth information of the neighborhood red pixel is utilized to propagate to the black pixel, and vice versa, and the propagation path is as shown in fig. 19. In fig. 19, circles labeled with "red" words represent "red" pixels; dividing the neighborhood pixels into 4 levels according to the distance of the propagation path, as shown in fig. 20, circles marked with "red" words represent "red" pixels; circles labeled with "yellow" words represent yellow pixels; each level represents a distance from near to far from deep to shallow. Two pixels with the smallest matching cost are selected from each hierarchy as pixels with reliable depth information, and 8 pixels are selected as neighborhood pixels for view selection, as shown by yellow pixels in fig. 20. Finally, the visibility of the neighbor images is calculated according to a distance weighting method, and a specific calculation formula is shown as a formula (16).

(16)。/>

Wherein,visibility of the neighbor image j, +.>For neighborhood pixels->Distance reciprocal weight of ∈ ->Representing neighborhood pixels +.>The NCC matching cost in the previous iteration is taken according to the experience value>. From equation (16), it can be derived that the closer the neighboring pixel is to the current pixel, and the higher the matching cost with respect to the neighboring image j, the higher the visibility of the neighboring image j. The visibility of all neighbor images of the reference image is ordered from high to low, taking the former +.>The neighbor images are taken as the visible image set of the current pixel.

The construction of the plane optimization model is to combine the plane prior information and the luminosity consistency by using a probability map model to construct a new multi-view matching cost. Because the new multi-view matching cost simultaneously considers the luminosity consistency and the plane prior information, the method is not only suitable for restoring the depth information of the strong texture region, but also can effectively restore the depth information of the weak texture region. Firstly, taking the depth information of the current pixel to be updated and the depth information of 8 neighborhood pixels selected in the previous step as a candidate hypothesis set of the current pixel to be updated.

。

Wherein the method comprises the steps ofDepth information d representing the corresponding planar structure of the pixel to be updated _i The parallax value of the ith pixel (the pixel to be updated currently); n is n _i Is the normal value of the ith pixel point. Relative to->Each candidate hypothesis will have +.>The NCC matches the cost, i.e. the current pixel q to be updated is taking the candidate hypothesis +.>And the brightness of the visible image j is consistent, and the calculation formula is shown as formula (17).

(17)。

Wherein,representing +.A. constructed centered on the current pixel to be updated q>Rectangular window of->Representing pixel +.>Pixel values in the reference image, +.>Representing pixel +.>And the pixel value of the corresponding pixel after homography transformation on the visible image. These candidate hypotheses correspond +.>The matching costs form a matrix as in equation (18).

(18)。

The matching cost for each row of the matrix may measure candidate hypothesesWith respect to the reliability of the visible image, therefore, equation (19) is used to take the weighted average of the top m best NCC matching costs as the pixel to be updated currently is +_in relation to the candidate hypothesis>Is used to match the final cost of matching.

(19)。

Wherein,the cost is matched for the NCC.

Secondly, in order to adopt plane prior information to assist in optimizing dense matching, the embodiment constructs a plane optimization probability map model, and a calculation formula of the plane optimization probability map model is shown as a formula (20).

(20)。

Wherein d _p The parallax value of the p-th pixel point; n is n _p The normal value of the p-th pixel point; neighbor propagation weight factorPlane a priori measurement constant +.>Step size of parallax ∈>Step size of normal difference ∈>The value is shown in the formula (21).

(21)。

Wherein d _max Is the maximum value of the normal; d, d _min Is the minimum of the normal.

For equation (20), the first term to the right of the equation is the matching cost, corresponding to the current pixel q to be updated in the candidate hypothesisIs a function of the photometric consistency of the (c). The second term is a regular term of plane prior, if the depth information of the current pixel q to be updated is consistent with the depth information of the plane structure, rewarding is given, otherwise punishment is given, so that the depth information of the weak texture region tends to the depth information of the plane structure, and the depth estimation of the weak texture region is optimized. Finally take->The best candidate hypothesis is the depth information for the pixel currently to be updated.

3. Experiment and analysis.

3.1 Experimental data and environment.

The experiment of the embodiment verifies the effectiveness of the method of the embodiment from the aspects of plane information extraction results, depth information calculation results, plane area flatness and the like. Experimental data included: (1) The published data set Dortmund provided by ispss is captured by a five-lens oblique imaging system, the number of images is 584, and the resolution is 8176×6132. (2) The field collected oblique photographic data set mainly comprises two different scenes of Garden and Central-Urman, the number of images is 162 and 566 respectively, the resolution is 5472×3648 and 4864×3648 respectively, and the main types of ground objects covered include: building, vegetation, road, surface of water etc. has the general meaning to experimental verification.

The experimental operation environment is a workstation, windows 10 64-bit operating system, intel Core (TM) i9-10900X CPU (main frequency is 3.70 GHz), 128GB memory.

3.2 And (5) extracting and analyzing the result of the plane prior information.

The plane prior information refers to a triangle primitive generated by selecting a matching point with better partial confidence as a structural point and adopting a Delaunay algorithm to construct a network. The existing ACMP method does not consider scene structure characteristics when selecting matching points, the method is improved on the basis of the existing ACMP method, a scene is divided into a line characteristic area and a non-line characteristic area, a small number of matching points with high confidence coefficient are selected in the non-line characteristic area, a large number of matching points with reliable confidence coefficient are selected in the line characteristic area, therefore, large plane structure constraint in a weak texture area is guaranteed, small plane structure constraint in a strong texture area is guaranteed, and depth estimation results of different areas are optimized. Fig. 21-32 show the ACMP method and the extraction results of the method of this example in the Dortmund dataset, the Garden dataset, and the Central-Urban dataset.

In a strong texture region in an image, triangle primitives formed by the ACMP method and the method in the embodiment are relatively smaller, and the structural characteristics of a scene can be well maintained by both methods; in the weak texture region of the image, the triangle primitive formed by the ACMP method and the method of the present embodiment is relatively large, but in the partial weak texture region, the triangle primitive formed by the ACMP method is too finely divided. In the method, the line characteristic constraint of the scene is utilized to construct plane prior information, a small number of matching points with high confidence coefficient are selected in the non-line characteristic region to serve as structural points, so that the formed triangle primitive is relatively large, and the accuracy of depth estimation of the weak texture region can be better guaranteed.

3.3 And analyzing the depth information calculation result.

For the Dortmund dataset, the Garden dataset, and the Central-Urban dataset, the ACMP method and the method of the present embodiment calculate depth information in regions one through six as shown in fig. 33-44.

As can be seen from fig. 33 to fig. 44, the ACMP method and the method according to the embodiment can calculate more accurate depth information in the strong texture region. In most weak texture regions, the calculated depth information is also smoother. Mainly because the ACMP method and the method of the present embodiment add plane prior information on the basis of photometric consistency, the depth estimation of the weak texture region can be optimized. However, in a part of the weak texture region, the depth information calculated by the ACMP method may have a part of outliers. Mainly because the planar prior information generated by the ACMP method in these areas is too finely divided, the abnormal depth information of the weak texture areas can only be partially relieved. The method adds sparse point cloud initialization depth information and construction of plane prior information based on scene line feature constraint on the basis of an ACMP method, so that the problem of inaccurate estimation of the depth information of a weak texture region can be better solved.

3.4 And (5) analyzing flatness of the plane area.

In the multi-view dense matching process, the ACMP method does not consider scene structural characteristics, and only adopts matching points with good matching cost to construct plane prior information, so that the plane prior information in a weak texture area is more, and the generated dense point cloud is not smooth enough in the plane area. According to the method, on the basis of the ACMP method, the depth information is initialized by using the sparse point cloud, the plane prior information is constructed by using the scene line feature constraint, the number of the plane prior information of the weak texture region is optimized, and the flatness of the generated dense point cloud in the plane region can be improved. In order to verify the analysis results, in the experiment of this section, the method of this embodiment adopts the depth fusion method in the ACMP method to generate a dense point cloud, and evaluates the flatness of the ACMP method and the method of this embodiment in the planar area with local plane precision. Where local plane accuracy refers to the root mean square error value of the distance of the dense point cloud to the local fitting plane. Fig. 45-56 show elevation views of dense point clouds of planar regions (region one through region six). The local plane accuracy statistics of these planar regions are shown in table 1. The local plane accuracy is in cm in table 1.

Table 1 comparison table of flatness of planar areas

	ACMP method	Method of the present embodiment	Improvement (%)
				Zone one	0.328	0.227	30.79%
Zone two	0.247	0.184	25.51%
				Region three	0.195	0.149	23.58%
Zone four	0.24	0.174	27.5%
				Zone five	0.308	0.216	29.87%
Region six	0.214	0.156	27.1%

As can be seen from fig. 45-56 and table 1, in the first to sixth regions, the flatness of the dense point cloud generated by the fusion of the method of this embodiment is higher than that of the ACMP method, for example: 30.79% improvement in zone one and 23.58% improvement in zone three, with an overall average improvement of 27.39%. The method is mainly because sparse point cloud initial depth information is introduced, and plane prior information is constructed by utilizing scene line feature constraint. In the same iteration times, the method of the embodiment can converge more quickly, the abnormal depth information generated in the weak texture region is less, and the flatness of the weak texture region can be effectively improved.

The three-dimensional modeling technology based on the inclined image is used as an emerging technology in photogrammetry and computer vision, and is widely applied to the application of urban live-action three-dimensional model reconstruction, novel basic mapping and the like due to the characteristics of high reconstruction efficiency, vivid effect and the like. Dense reconstruction is used as a key step for constructing a high-quality model in a three-dimensional modeling technology based on oblique images, and the problem of low matching precision of a weak texture region exists. Aiming at the problem, the embodiment performs research from planar prior optimization multi-view dense matching, and the main content and conclusion are as follows.

Firstly, preprocessing of multi-view dense matching is introduced, and main contents comprise calculation of image combination and initialization of depth information; then, outline explanation is carried out on a multi-view dense matching method and a theoretical basis of plane priori auxiliary optimization based on scene line feature constraint, and main contents comprise scene line feature region extraction and multi-view dense matching of plane priori optimization; finally, three data sets are adopted for experiments and qualitative and quantitative analysis is carried out.

Experiments prove that the rapid propagation of accurate depth information can be accelerated and the structural characteristics of a scene can be well maintained by initializing the depth information through the sparse point cloud; through plane prior auxiliary optimization multi-view dense matching based on scene line feature constraint, on the premise of keeping the structural features of a strong texture region, abnormal depth information of the weak texture region can be effectively removed, flatness of the weak texture region is improved, and experiments show that overall average is improved by 27.39%. Therefore, the method of the embodiment can effectively improve the dense matching precision of the weak texture region.

Example 2: in order to perform the method corresponding to the above embodiment 1 to achieve the corresponding functions and technical effects, a multi-view dense matching system based on planar prior optimization is provided below, which includes: the scene data acquisition module is used for acquiring sparse point clouds and multiple images of a scene to be modeled. The shooting angles of different images are different.

And the current reference image determining module is used for determining any image as the current reference image.

The neighbor image set determining module is used for determining a neighbor image set of the current reference image from the plurality of images according to the sparse point cloud of the scene to be modeled.

The initialization depth module is used for initializing the depth information of all pixel points in the current reference image by utilizing the triangulation processing of the sparse point cloud to obtain the initialization depth map of the current reference image. The depth information includes a depth value and a normal value.

And the line feature set extraction module is used for extracting the line feature set in the initialization depth map of the current reference image.

The line characteristic region dividing module is used for dividing the current reference image into a line characteristic region and a non-line characteristic region according to the line characteristic set.

And the plane prior information construction module is used for constructing plane prior information according to the line characteristic region and the non-line characteristic region.

And the neighborhood pixel set determining module is used for determining a neighborhood pixel set of each pixel in the current reference image.

And the visible image set determining module is used for determining a visible image set according to the visibility of the neighborhood pixel set of each pixel in the current reference image in the neighbor image set of the current reference image.

And the multi-view matching cost function determining module is used for constructing a new multi-view matching cost function by utilizing the probability map model and using the plane prior information, the visible image set and the current line features.

And the sub-dense matching result determining module is used for updating the depth information of all pixel points in the current reference image according to the multi-view matching cost function to obtain the dense matching result of the current reference image.

And the total dense matching result determining module is used for updating the current reference image and returning to the neighbor image set determining module until all the images are traversed, so as to obtain dense matching results of a plurality of images.

And the multi-view dense matching module is used for fusing dense matching results of the plurality of images to obtain a multi-view dense matching result of the scene to be modeled.

Example 3: the present embodiment provides an electronic device, including a memory and a processor, where the memory is configured to store a computer program, and the processor is configured to execute the computer program to cause the electronic device to perform a multi-view dense matching method based on planar prior optimization as described in embodiment 1. Wherein the memory is a readable storage medium.

In the present specification, each embodiment is described in a progressive manner, and each embodiment is mainly described in a different point from other embodiments, and identical and similar parts between the embodiments are all enough to refer to each other. For the system disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant points refer to the description of the method section.

The present embodiment has been described with specific examples to illustrate the principles and embodiments of the present invention, and the description of the above embodiments is only for aiding in understanding the method and core idea of the present invention; also, it is within the scope of the present invention to be modified by those of ordinary skill in the art in light of the present teachings. In view of the foregoing, this description should not be construed as limiting the invention.

Claims

1. The multi-view dense matching method based on plane prior optimization is characterized by comprising the following steps of:

acquiring sparse point clouds and a plurality of images of a scene to be modeled; the shooting angles of different images are different;

determining any image as a current reference image;

determining a neighbor image set of the current reference image from the plurality of images according to the sparse point cloud of the scene to be modeled;

initializing depth information of all pixel points in a current reference image by utilizing triangulation processing of sparse point cloud to obtain an initialized depth map of the current reference image; the depth information includes a depth value and a normal value;

extracting a line feature set in a current reference image initialization depth map;

dividing the current reference image into a line characteristic region and a non-line characteristic region according to the line characteristic set;

Constructing plane prior information according to the line characteristic region and the non-line characteristic region;

determining a neighborhood pixel set of each pixel in the current reference image;

determining a visible image set according to the visibility of the neighborhood pixel set of each pixel in the current reference image in the neighbor image set of the current reference image;

constructing a new multi-view matching cost function by using the probability map model through the plane prior information, the visible image set and the current line features;

updating the depth information of all pixel points in the current reference image according to the multi-view matching cost function to obtain a dense matching result of the current reference image;

updating the current reference image and returning to the step of determining a neighbor image set of the current reference image from the plurality of images according to the sparse point cloud of the scene to be modeled until all the images are traversed, so as to obtain a dense matching result of the plurality of images;

and fusing the dense matching results of the multiple images to obtain a multi-view dense matching result of the scene to be modeled.

2. The multi-view dense matching method based on planar prior optimization of claim 1, wherein determining a neighbor image set of a current reference image from a plurality of images according to a sparse point cloud of a scene to be modeled comprises:

Determining a visible sparse point set of the current reference image from a sparse point cloud of a scene to be modeled;

determining any image to be fixed as a current image to be fixed; the undetermined images are a plurality of images except the current reference image;

determining a visible sparse point set of the current undetermined image from a sparse point cloud of a scene to be modeled;

determining the intersection of the visible sparse point set of the current reference image and the visible sparse point set of the current undetermined image as a common visible sparse point set of the current reference image and the current undetermined image;

determining a correlation score of the current reference image and the current image to be determined according to the common visible sparse point set;

updating the current image to be determined, and returning to the step of determining a visible sparse point set of the current image to be determined from the sparse point cloud of the scene to be modeled until all the images to be determined are traversed, so as to obtain the relevance scores of the current reference image and all the images to be determined;

descending order of the images to be determined according to the relevance scores;

and determining the pre-set number of undetermined images as a neighbor image set of the current reference image.

3. The multi-view dense matching method based on plane prior optimization according to claim 2, wherein initializing depth information of all pixel points in a current reference image by using triangulation processing of sparse point cloud to obtain an initialized depth map of the current reference image comprises:

Projecting the visible sparse point set of the current reference image onto the current reference image to obtain a projection point set of the current reference image;

triangulating the projection point set of the current reference image by using a Delaunay algorithm to generate a two-dimensional grid;

constructing a three-dimensional grid according to sparse depth information of a projection point set of the current reference image;

determining the image pose of the current reference image by using sparse reconstruction;

determining any pixel point in the current reference image as a current pixel point;

calculating the current projection light corresponding to the current pixel point according to the image pose;

projecting the current projection light to the three-dimensional grid, and determining a triangular patch intersecting the current projection light in the three-dimensional grid as a current triangular patch;

determining a plane equation of the current triangular patch according to the 3 vertex coordinates of the current triangular patch;

determining the depth value and the normal value of the current pixel point according to the coefficient of the plane equation of the current triangular patch;

updating the current pixel point and returning to the step of calculating the current projection light corresponding to the current pixel point according to the pose of the image until all the pixel points in the current reference image are traversed, and obtaining the initialization depth map of the current reference image.

4. The multi-view dense matching method based on planar prior optimization of claim 1, wherein extracting a line feature set in a current reference image initialization depth map comprises:

extracting a plurality of line features in the initialization depth map of the current reference image by using an LSD linear detection method to obtain an initial line feature set; the line features are line segments;

deleting line features with lengths smaller than a length threshold value in the initial line feature set;

connecting collinear line features in the initial line feature set;

determining any line feature in the initial line feature set as a current line feature;

determining a plurality of line features except the current line feature in the initial line feature set as line features to be matched;

determining any line feature to be matched as the current line feature to be matched;

determining an included angle between the current line characteristic and the current line characteristic to be matched as a current included angle;

when the current included angle is larger than or equal to the included angle threshold value, updating the current line feature to be matched, and returning to the step of determining that the included angle between the current line feature and the current line feature to be matched is the current included angle;

when the current included angle is smaller than the included angle, determining that the current line characteristic and the current to-be-matched line characteristic are current approximate parallel line characteristic pairs;

Constructing a linear equation of the current line feature to be matched according to two end point coordinates of the current line feature to be matched;

determining the distance from one end point of the current line characteristic to be matched as a first distance according to a linear equation of the current line characteristic to be matched;

determining the distance from the other end point of the current line characteristic to be matched as a second distance according to a linear equation of the current line characteristic to be matched;

determining the average value of the first distance and the second distance as the vertical distance of the current approximate parallel line feature pair;

when the vertical distance is smaller than the vertical distance threshold, fitting the current approximate parallel line characteristic pair by using a least square method to obtain a fitting line characteristic;

replacing the current approximate parallel line feature pairs in the initial line feature set by fitting line features;

taking the fitting line characteristic as the current line characteristic, and returning to the step of determining any line characteristic to be matched as the current line characteristic to be matched until all line characteristics to be matched are traversed;

updating the current line characteristics, returning to the step of determining that a plurality of line characteristics except the current line characteristics in the initial line characteristic set are to-be-matched line characteristics until the initial line characteristic set is traversed, and determining the to-be-fixed line characteristic set.

5. The multi-view dense matching method based on planar prior optimization of claim 4, further comprising, after determining the set of features to be defined:

determining any line characteristic in the to-be-determined line characteristic set as a current line characteristic;

determining any endpoint of the current line feature as a current endpoint;

determining the end points of all line features except the current line feature in the to-be-determined line feature set as a current end point set;

determining a plurality of endpoints, of which the distances from the current endpoint in the current endpoint set are smaller than a distance threshold, as endpoints to be combined;

determining the current endpoint and a plurality of endpoints to be combined as a point set to be combined;

determining the line characteristics of the point set to be combined as the line characteristic set to be combined;

determining the coordinate mean value of the point set to be combined as the coordinate of the combining point;

all points in the point set to be combined are connected with the combining point to obtain combining line characteristics;

and replacing the to-be-merged line feature set in the to-be-determined line feature set by adopting the merged line feature, and returning to the step of determining any line feature in the to-be-determined line feature set as the current line feature until the to-be-determined line feature set is traversed, so as to obtain the line feature set in the initialization depth map of the current reference image.

6. The multi-view dense matching method based on planar prior optimization of claim 1, wherein constructing planar prior information from the line feature region and the non-line feature region comprises:

performing primary matching by using an ACMH method, and determining that a matching point with the confidence coefficient smaller than a first confidence coefficient threshold value in the current reference image is a pending structural point;

determining all undetermined structural points in the line characteristic area as selected structural points;

determining candidate structural points with the confidence coefficient smaller than a second confidence coefficient threshold value in the non-line characteristic region as selected structural points; the second confidence threshold is less than the first confidence threshold;

constructing a network by adopting a Delaunay algorithm based on a plurality of selected structure points to generate a plurality of triangle primitives;

determining any triangle primitive as the current triangle primitive;

determining current plane prior information according to 3 vertexes of the current triangle primitive;

constructing a plane equation of the current triangle primitive;

determining the prior information of the current plane as the prior information of the planes of all pixels in the current triangle primitive;

updating the current triangle primitive, and returning to the step of determining the prior information of the current plane according to the 3 vertexes of the current triangle primitive until all the triangle primitives are traversed, and determining the prior information of the planes of all the pixels in the current reference image.

7. The multi-view dense matching method based on plane prior optimization according to claim 4, wherein constructing a new multi-view matching cost function by using a probability map model from plane prior information, a visible image set and a plurality of current line features comprises:

determining any pixel as a current pixel;

determining the depth information of the current pixel and the depth information of each neighborhood pixel in the neighborhood pixel set of the current pixel as candidate hypothesis sets;

determining the luminosity consistency of the current pixel and the visible image when each candidate hypothesis in the candidate hypothesis set is selected to construct a matching cost matrix;

determining the final matching cost of each candidate hypothesis according to the matching cost matrix;

updating the current pixel and returning to the step of determining the depth information of the current pixel and the depth information of each neighborhood pixel in the neighborhood pixel set of the current pixel as a candidate hypothesis set to obtain the final matching cost of selecting different candidate hypotheses for each pixel;

and selecting final matching cost of different candidate hypotheses according to each pixel and plane prior information of each pixel, and constructing a plane optimization probability map model.

8. A multi-view dense matching system based on planar prior optimization, comprising:

The scene data acquisition module is used for acquiring sparse point clouds and a plurality of images of a scene to be modeled; the shooting angles of different images are different;

the current reference image determining module is used for determining any image as a current reference image;

the neighbor image set determining module is used for determining a neighbor image set of the current reference image from the plurality of images according to the sparse point cloud of the scene to be modeled;

the initialization depth module is used for initializing depth information of all pixel points in the current reference image by utilizing the triangulation processing of the sparse point cloud to obtain an initialization depth map of the current reference image; the depth information includes a depth value and a normal value;

the line feature set extraction module is used for extracting a line feature set in the initialization depth map of the current reference image;

the line characteristic region dividing module is used for dividing the current reference image into a line characteristic region and a non-line characteristic region according to the line characteristic set;

the plane prior information construction module is used for constructing plane prior information according to the line characteristic region and the non-line characteristic region;

the neighborhood pixel set determining module is used for determining a neighborhood pixel set of each pixel in the current reference image;

The visible image set determining module is used for determining a visible image set according to the visibility of the neighborhood pixel set of each pixel in the current reference image in the neighbor image set of the current reference image;

the multi-view matching cost function determining module is used for constructing a new multi-view matching cost function by utilizing the probability map model and utilizing the plane prior information, the visible image set and the current line features;

the sub-dense matching result determining module is used for updating the depth information of all pixel points in the current reference image according to the multi-view matching cost function to obtain a dense matching result of the current reference image;

the total dense matching result determining module is used for updating the current reference image and returning to the neighbor image set determining module until all images are traversed to obtain dense matching results of a plurality of images;

9. An electronic device comprising a memory for storing a computer program and a processor that runs the computer program to cause the electronic device to perform a multi-view dense matching method based on planar prior optimization as claimed in any one of claims 1 to 7.

10. The electronic device of claim 9, wherein the memory is a readable storage medium.