CN117557617B - Multi-view dense matching method, system and equipment based on plane priori optimization - Google Patents

Abstract

The invention aims to provide a multi-view dense matching method, system and equipment based on plane prior optimization, and relates to the technical field of dense matching. The method comprises the following steps: initializing depth information of all pixel points in a current reference image, and dividing the current reference image into a line characteristic region and a non-line characteristic region; determining a visible image set according to the visibility of the neighborhood pixel set of each pixel in the current reference image in the neighbor image set of the current reference image; constructing a new multi-view matching cost function by using the probability map model through the plane prior information, the visible image set and the current line features; updating the depth information of all pixel points in the current reference image according to the multi-view matching cost function to obtain a dense matching result of the current reference image; and fusing the dense matching results of the multiple images to obtain a multi-view dense matching result of the scene to be modeled. The invention can determine the depth information of the weak texture region by constructing a new multi-view matching cost function.

Description

A multi-view dense matching method, system and device based on plane prior optimization

Technical Field

The present invention relates to the field of dense matching technology, and in particular to a multi-view dense matching method, system and device based on plane prior optimization.

Background Art

Dense matching is a 3D reconstruction technique used in computer vision. The technique aims to estimate the 3D geometric structure of the corresponding scene from two or more 2D images. Compared with sparse reconstruction, dense matching can provide more accurate depth information and generate more detailed and realistic 3D models. The core idea of dense matching is to overlap images from different viewpoints through pixel-level matching to calculate the depth information of each pixel. This matching can be implemented based on a variety of methods. The most common implementation method is to use the dense matching method based on photometric consistency. The core idea of this method is to align images from different perspectives to make them consistent in photometrics, and then perform pixel-level matching on this basis to calculate the depth information of each pixel. This method mainly improves the accuracy and robustness of matching by utilizing the photometric information between different images. However, in the weak texture area of the image, the photometric consistency is no longer reliable, the calculated depth information is inaccurate in the weak texture area, and the dense point cloud obtained after subsequent depth fusion has a divergent phenomenon.

Summary of the invention

The purpose of the present invention is to provide a multi-view dense matching method, system and device based on plane prior optimization, which can determine the depth information of weak texture areas.

To achieve the above-mentioned purpose, the present invention provides the following scheme: a multi-view dense matching method based on plane prior optimization, comprising: obtaining a sparse point cloud and multiple images of a scene to be modeled; different images have different shooting angles; determining any image as a current reference image; determining a neighbor image set of the current reference image from multiple images according to the sparse point cloud of the scene to be modeled; initializing the depth information of all pixels in the current reference image by triangulation processing of the sparse point cloud to obtain an initialized depth map of the current reference image; the depth information includes depth values and normal values; extracting a line feature set in the initialized depth map of the current reference image; dividing the current reference image into a line feature region and a non-line feature region according to the line feature set; constructing a plane prior according to the line feature region and the non-line feature region. Prior information; determine the neighborhood pixel set of each pixel in the current reference image; determine the visible image set according to the visibility of the neighborhood pixel set of each pixel in the current reference image in the neighbor image set of the current reference image; use the probabilistic graph model to construct a new multi-view matching cost function with the plane prior information, the visible image set and multiple current line features; update the depth information of all pixels in the current reference image according to the multi-view matching cost function to obtain the dense matching result of the current reference image; update the current reference image and return to the step "determine the neighbor image set of the current reference image from multiple images according to the sparse point cloud of the scene to be modeled" until all images are traversed to obtain the dense matching results of multiple images; fuse the dense matching results of multiple images to obtain the multi-view dense matching result of the scene to be modeled.

A multi-view dense matching system based on plane prior optimization includes: a scene data acquisition module, used to acquire a sparse point cloud and multiple images of a scene to be modeled; different images have different shooting angles; a current reference image determination module, used to determine any image as the current reference image; a neighbor image set determination module, used to determine the neighbor image set of the current reference image from multiple images according to the sparse point cloud of the scene to be modeled; an initialization depth module, used to initialize the depth information of all pixels in the current reference image by triangulation processing of the sparse point cloud, and obtain an initialization depth map of the current reference image; the depth information includes depth values and method Line value; a line feature set extraction module, used to extract the line feature set in the initialization depth map of the current reference image; a line feature area division module, used to divide the current reference image into a line feature area and a non-line feature area according to the line feature set; a plane prior information construction module, used to construct plane prior information according to the line feature area and the non-line feature area; a neighborhood pixel set determination module, used to determine the neighborhood pixel set of each pixel in the current reference image; a visible image set determination module, used to determine the visible image set according to the visibility of the neighborhood pixel set of each pixel in the current reference image in the neighbor image set of the current reference image.

A multi-view matching cost function determination module is used to use a probabilistic graph model to construct a new multi-view matching cost function using plane prior information, visible image sets and multiple current line features; a sub-dense matching result determination module is used to update the depth information of all pixels in the current reference image according to the multi-view matching cost function to obtain the dense matching result of the current reference image; a total dense matching result determination module is used to update the current reference image and return to execute the neighbor image set determination module until all images are traversed to obtain the dense matching results of multiple images; a multi-view dense matching module is used to fuse the dense matching results of multiple images to obtain the multi-view dense matching results of the scene to be modeled.

An electronic device comprises a memory and a processor, wherein the memory is used to store a computer program, and the processor runs the computer program to enable the electronic device to execute a multi-view dense matching method based on plane prior optimization.

According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects: The purpose of the present invention is to provide a multi-view dense matching method, system and device based on plane prior optimization. In view of the problem of inaccurate depth information estimation in weak texture areas, a multi-view dense matching method based on plane prior assisted optimization with scene line feature constraints is designed based on the ACMP algorithm, and two depth information optimization methods at different stages are proposed. One is to initialize depth information using sparse point clouds reconstructed from sparse points to achieve rapid propagation of high-precision depth information; the other is to use the line feature constraints of the scene to construct a high-quality plane prior model, optimize the depth information in weak texture areas, and achieve accurate calculation of depth information in weak texture areas.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

FIG1 is a flow chart of a multi-view dense matching method based on plane prior optimization in Embodiment 1 of the present invention.

FIG. 2 is a schematic diagram of the minimum bounding box area formed by sparse points in a reference image in Embodiment 1 of the present invention.

FIG. 3 is a schematic diagram of the angle formed by the camera center and the sparse points in Example 1 of the present invention.

FIG. 4 is a diagram showing the projection distance of a unit sphere centered on a sparse point on an image in Embodiment 1 of the present invention.

FIG. 5 is a schematic diagram of a result of one iteration of random initialization of depth information in Embodiment 1 of the present invention.

FIG. 6 is a schematic diagram of a result of a second iteration of random initialization of depth information in Embodiment 1 of the present invention.

FIG. 7 is a schematic diagram of the results of three iterations of random initialization of depth information in Embodiment 1 of the present invention.

FIG. 8 is a schematic diagram of the result of one iteration of the depth information rule initialization in Embodiment 1 of the present invention.

FIG. 9 is a schematic diagram of secondary iteration results of depth information rule initialization in Embodiment 1 of the present invention.

FIG. 10 is a schematic diagram of the results of three iterations of initializing the depth information rule in Example 1 of the present invention.

FIG. 11 is a schematic diagram of determining the angle of a line segment in Embodiment 1 of the present invention.

FIG. 12 is a schematic diagram of determining the vertical distance of a line segment in Embodiment 1 of the present invention.

FIG. 13 is a schematic diagram of determining the distance between endpoints of a line segment in Embodiment 1 of the present invention.

FIG. 14 is a schematic diagram of line segment overlap determination in Embodiment 1 of the present invention.

FIG. 15 is a schematic diagram of segment endpoint clustering determination in Embodiment 1 of the present invention.

FIG. 16 is a schematic diagram of an original image in Embodiment 1 of the present invention.

FIG. 17 is a schematic diagram of a mask image in Embodiment 1 of the present invention.

FIG18 is a schematic diagram of the “red and black chessboard” parallel propagation strategy in Example 1 of the present invention.

FIG19 is a schematic diagram of the propagation path of the “red and black chessboard” parallel propagation strategy in Example 1 of the present invention.

FIG20 is a schematic diagram of neighborhood pixels in Example 1 of the present invention.

FIG. 21 is a schematic diagram of the results of extracting plane prior information of region 1 in the Dortmund dataset using the ACMP method in Example 1 of the present invention.

FIG. 22 is a schematic diagram of the results of extracting plane prior information of region 1 in the Dortmund dataset according to Example 1 of the present invention.

FIG23 is a schematic diagram of the results of extracting plane prior information of region 2 in the Dortmund dataset using the ACMP method in Example 1 of the present invention.

FIG. 24 is a schematic diagram of the results of plane prior information extraction for region 2 in the Dortmund dataset according to Example 1 of the present invention.

FIG. 25 is a schematic diagram of the results of extracting plane prior information of region three in the Garden data set using the ACMP method in Example 1 of the present invention.

FIG26 is a schematic diagram of the results of extracting plane prior information of area three in the Garden data set according to Example 1 of the present invention.

FIG27 is a schematic diagram of the results of extracting plane prior information of area 4 in the Garden data set using the ACMP method in Example 1 of the present invention.

FIG28 is a schematic diagram of the results of extracting plane prior information of area 4 in the Garden data set according to Example 1 of the present invention.

FIG29 is a schematic diagram of the results of extracting plane prior information of area five in the Central-Urban dataset using the ACMP method in Example 1 of the present invention.

FIG30 is a schematic diagram of the results of plane prior information extraction for region five in the Central-Urban dataset according to Example 1 of the present invention.

FIG31 is a schematic diagram of the results of extracting plane prior information of area six in the Central-Urban dataset using the ACMP method in Example 1 of the present invention.

FIG32 is a schematic diagram of the results of extracting plane prior information of area six in the Central-Urban dataset according to Example 1 of the present invention.

FIG33 is a schematic diagram of the calculation results of the depth information of region 1 in the Dortmund dataset using the ACMP method in Example 1 of the present invention.

FIG34 is a schematic diagram of the depth information calculation results of region 1 in the Dortmund dataset according to Example 1 of the present invention.

FIG35 is a schematic diagram showing the calculation results of the depth information of region 2 in the Dortmund dataset using the ACMP method in Example 1 of the present invention.

FIG36 is a schematic diagram of the depth information calculation results of region 2 in the Dortmund dataset according to Example 1 of the present invention.

FIG37 is a schematic diagram of the depth information calculation results of area three in the Garden data set using the ACMP method in Example 1 of the present invention.

FIG38 is a schematic diagram of the depth information calculation results of area three in the Garden data set according to Example 1 of the present invention.

FIG39 is a schematic diagram of the depth information calculation results of area 4 in the Garden data set using the ACMP method in Example 1 of the present invention.

FIG40 is a schematic diagram of the depth information calculation results of area 4 in the Garden data set according to Example 1 of the present invention.

FIG41 is a schematic diagram of the depth information calculation results of area five in the Central-Urban dataset using the ACMP method in Example 1 of the present invention.

FIG42 is a schematic diagram of the depth information calculation results of area five in the Central-Urban dataset according to Example 1 of the present invention.

FIG43 is a schematic diagram of the depth information calculation results of area six in the Central-Urban dataset using the ACMP method in Example 1 of the present invention.

FIG44 is a schematic diagram of the depth information calculation results of area six in the Central-Urban dataset according to Example 1 of the present invention.

Figure 45 is a vertical view of area 1 in the Dortmund dataset using the ACMP method in Example 1 of the present invention.

FIG. 46 is an elevation view of area 1 in the Dortmund dataset according to Example 1 of the present invention.

FIG47 is a vertical elevation diagram of region 2 in the Dortmund dataset using the ACMP method in Example 1 of the present invention.

FIG. 48 is an elevation view of area 2 in the Dortmund dataset according to Example 1 of the present invention.

FIG49 is a vertical view of region three in the Garden data set using the ACMP method in Example 1 of the present invention.

FIG50 is a vertical view of area three in the Garden data set according to Example 1 of the present invention.

Figure 51 is a vertical view of area 4 in the Garden data set using the ACMP method in Example 1 of the present invention.

FIG52 is a vertical elevation view of area 4 in the Garden data set according to Example 1 of the present invention.

Figure 53 is a vertical view of area 5 in the Central-Urban dataset using the ACMP method in Example 1 of the present invention.

FIG54 is a elevation view of area five in the Central-Urban dataset according to Example 1 of the present invention.

Figure 55 is a vertical view of area six in the Central-Urban dataset using the ACMP method in Example 1 of the present invention.

FIG56 is a elevation view of area six in the Central-Urban dataset according to Example 1 of the present invention.

DETAILED DESCRIPTION

The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

The purpose of the present invention is to provide a multi-view dense matching method, system and device based on plane prior optimization, which can determine the depth information of weak texture areas.

In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

Embodiment 1: As shown in FIG1 , this embodiment provides a multi-view dense matching method based on plane prior optimization, including: Step 101: obtaining a sparse point cloud and multiple images of a scene to be modeled. Different images are shot at different angles.

Step 102: Determine any image as a current reference image.

Step 103: According to the sparse point cloud of the scene to be modeled, a neighbor image set of the current reference image is determined from multiple images.

Step 104: Initialize the depth information of all pixels in the current reference image by triangulation of the sparse point cloud to obtain the initialization depth map of the current reference image. The depth information includes depth values and normal values.

Step 105: Extract the line feature set in the current reference image initialization depth map.

Step 106: Divide the current reference image into a line feature region and a non-line feature region according to the line feature set.

Step 107: construct plane prior information based on the line feature area and the non-line feature area.

Step 108: Determine the neighborhood pixel set of each pixel in the current reference image.

Step 109: Determine a visible image set according to the visibility of a neighborhood pixel set of each pixel in the current reference image in a neighbor image set of the current reference image.

Step 1010: Use the probabilistic graphical model to construct a new multi-view matching cost function by combining the plane prior information, the visible image set and multiple current line features.

Step 1011: Update the depth information of all pixels in the current reference image according to the multi-view matching cost function to obtain a dense matching result of the current reference image.

Step 1012: Update the current reference image and return to step 103 until all images are traversed to obtain dense matching results of multiple images.

Step 1013: Fuse the dense matching results of multiple images to obtain a multi-view dense matching result of the scene to be modeled.

Step 103 includes:

Step 103-1: Determine a set of visible sparse points of the current reference image from the sparse point cloud of the scene to be modeled.

Step 103-2: Determine any pending image as the current pending image. The pending images are multiple images other than the current reference image.

Step 103-3: Determine the visible sparse point set of the current image to be determined from the sparse point cloud of the scene to be modeled.

Step 103-4: Determine that the intersection of the visible sparse point set of the current reference image and the visible sparse point set of the current pending image is the common visible sparse point set of the current reference image and the current pending image.

Step 103-5: Determine the correlation score between the current reference image and the current pending image based on the common visible sparse point set.

Step 103-6: Update the current pending image and return to step 103-1 until all pending images are traversed to obtain the correlation scores between the current reference image and all pending images.

Step 103-7: Arrange the pending images in descending order according to the correlation scores.

Step 103-8: Determine a preset number of pending images as a neighbor image set of the current reference image.

Step 104 includes: Step 104-1: projecting the visible sparse point set of the current reference image onto the current reference image to obtain the projection point set of the current reference image.

Step 104-2: Use the Delaunay algorithm to triangulate the projection point set of the current reference image to generate a two-dimensional grid.

Step 104-3: Construct a three-dimensional grid based on the sparse depth information of the projection point set of the current reference image.

Step 104-4: Determine the image pose of the current reference image using sparse reconstruction.

Step 104-5: Determine any pixel point in the current reference image as the current pixel point.

Step 104-6: Calculate the current projection light corresponding to the current pixel point according to the image posture.

Step 104-7: Project the current projection light onto the three-dimensional mesh, and determine that the triangle facet in the three-dimensional mesh that intersects with the current projection light is the current triangle facet.

Step 104-8: Determine the plane equation of the current triangle according to the coordinates of the three vertices of the current triangle.

Step 104-9: Determine the depth value and normal value of the current pixel point according to the coefficients of the current triangle plane equation.

Step 104-10: Update the current pixel and return to step 104-6 until all pixels in the current reference image are traversed to obtain the initialization depth map of the current reference image.

Step 105 includes: Step 105-1: extract multiple line features in the initialization depth map of the current reference image using the LSD line detection method to obtain an initial line feature set. The line feature is a line segment.

Step 105-2: Delete the line features in the initial line feature set whose length is less than the length threshold.

Step 105-3: Connect the collinear line features in the initial line feature set.

Step 105-4: Determine any line feature in the initial line feature set as the current line feature.

Step 105-5: Determine multiple line features other than the current line feature in the initial line feature set as line features to be matched.

Step 105-6: Determine any line feature to be matched as the current line feature to be matched.

Step 105-7: Determine the angle between the current line feature and the current line feature to be matched as the current angle.

Step 105-8: When the current angle is greater than or equal to the angle threshold, update the current line feature to be matched and return to step 105-7.

Step 105-9: When the current angle is smaller than the angle, determine that the current line feature and the current line feature to be matched are a current approximately parallel line feature pair.

Step 105-10: construct the straight line equation of the current line feature to be matched according to the coordinates of the two endpoints of the current line feature to be matched.

Step 105-11: According to the straight line equation of the current line feature to be matched, determine the distance from one endpoint of the current line feature to the current line feature to be matched as the first distance.

Step 105-12: According to the straight line equation of the current line feature to be matched, determine the distance from the other endpoint of the current line feature to the current line feature to be matched as the second distance.

Step 105-13: Determine the mean of the first distance and the second distance as the perpendicular distance of the current pair of approximate parallel line features.

Step 105-14: When the vertical distance is less than the vertical distance threshold, the current approximate parallel line feature pair is fitted using the least squares method to obtain a fitted line feature.

Step 105-15: Use the fitted line features to replace the current approximate parallel line feature pairs in the initial line feature set.

Step 105-16: Use the fitted line feature as the current line feature, and return to step 105-6 until all line features to be matched are traversed.

Step 105-17: Update the current line feature and return to step 105-5 until the initial line feature set is traversed and the pending line feature set is determined.

Step 105-18: Determine any line feature in the pending line feature set as the current line feature.

Step 105-19: Determine any endpoint of the current line feature as the current endpoint.

Step 105-20: Determine the endpoints of all line features other than the current line feature in the pending line feature set as the current endpoint set.

Step 105-21: Determine multiple endpoints in the current endpoint set whose distances from the current endpoint are less than a distance threshold as endpoints to be merged.

Step 105-22: Determine that the current endpoint and multiple endpoints to be merged are a point set to be merged.

Step 105-23: Determine the line features where the point set to be merged is located as the line feature set to be merged.

Step 105-24: Determine the coordinate mean of the point set to be merged as the coordinate of the merged point.

Step 105-25: Connect all points in the point set to be merged with the merge point to obtain a merge line feature.

Step 105-26: Use the merged line features to replace the line feature set to be merged in the line feature set to be determined, and return to step 105-18 until the line feature set to be determined is traversed to obtain the line feature set in the depth map initialized by the current reference image.

Step 107 includes: Step 107-1: Performing initial matching using the ACMH method to determine matching points in the current reference image whose confidence level is less than a first confidence level threshold as pending structural points.

Step 107-2: Determine all pending structural points in the line feature region as selected structural points.

Step 107-3: Determine the candidate structure points in the non-linear feature region whose confidence is less than a second confidence threshold as selected structure points. The second confidence threshold is less than the first confidence threshold.

Step 107-4: Based on the multiple selected structural points, a Delaunay algorithm is used to construct a mesh to generate multiple triangular primitives.

Step 107-5: Determine any triangle primitive as the current triangle primitive.

Step 107-6: Determine the current plane prior information based on the three vertices of the current triangle primitive.

Step 107-7: Construct the plane equation of the current triangle primitive.

Step 107-8: Determine that the current plane prior information is the plane prior information of all pixels in the current triangle primitive.

Step 107-9: Update the current triangle primitive and return to step 107-6 until all triangle primitives are traversed to determine the plane prior information of all pixels in the current reference image.

Step 1010 includes: Step 1010-1: Determine any pixel as the current pixel.

Step 1010-2: Determine the depth information of the current pixel and the depth information of each neighboring pixel in the neighborhood pixel set of the current pixel as a candidate hypothesis set.

Step 1010-3: Determine the photometric consistency of the current pixel with the visible image when selecting each candidate hypothesis in the candidate hypothesis set to construct a matching cost matrix.

Step 1010-4: Determine the final matching cost of each candidate hypothesis according to the matching cost matrix.

Step 1010-5: Update the current pixel and return to step 1010-4 to obtain the final matching cost of selecting different candidate hypotheses for each pixel.

Step 1010-6: construct a plane optimization probability graph model based on the final matching costs of different candidate hypotheses selected for each pixel and the plane prior information of each pixel.

Specifically, the core steps of the multi-view dense matching method based on plane prior optimization in this embodiment may include: (1) Calculation of image combination: using each image and sparse point cloud as data sources, and taking them as reference images in turn, the scores are calculated based on factors such as the commonly visible sparse points, projection area, and angle between the reference image and the remaining images, and a set of neighbor image sets is selected for it. (2) Initialization of depth information: using the sparse point cloud and image pose as data input, the depth information initialization task of each reference image is completed through triangulation of the sparse point cloud. (3) Scene line feature region extraction: The line feature region of the scene is a linear buffer region established with the line feature extracted from the image as the center and a certain pixel size as the radius. In the image, weak texture regions usually have strong plane characteristics, and fewer line features are extracted from these regions, so they are usually non-line feature regions. On the contrary, strong texture regions usually have obvious structural features, and more line features are extracted from these regions, so they are usually line feature regions. To complete the scene line feature region extraction, first extract the straight line segment through the steps of scale scaling, gradient calculation, and region growing; then complete the line segment closure through the steps of fine segment filtering, connecting the line segments on the same straight line, and connecting the line segments that are not on the same straight line but have similar endpoint distances. (4) Depth value optimization strategy of plane prior optimization: mainly includes plane prior information extraction, pixel-level image selection, and plane optimization model construction. First, use the selected matching points with good confidence as structural points to construct plane prior information, and then use the visibility of each pixel's neighborhood pixels in the neighboring images to select the visible image set. Secondly, use the probabilistic graph model to combine the plane prior information, photometric consistency, and the scene line features extracted in the previous step to construct a new multi-view matching cost function to optimize the depth value, and finally achieve the purpose of improving the depth value accuracy of the weak texture area.

1. Dense matching preprocessing.

1.1 Calculation of image combination.

When performing multi-view dense matching, each image will be used as a reference image in turn, and the calculation of the image combination is to select a set of neighboring images for each reference image, so as to facilitate use in subsequent steps such as multi-view dense matching and deep fusion. This embodiment selects a set of images with good neighborhoods for each image in the scene based on the global view selection algorithm. These images are ideal neighbor images in terms of scene content, appearance and scale. The core idea is to calculate the correlation score based on the commonly visible sparse points p between the reference image and the remaining images. The scores are sorted from high to low, and the N _best images are selected from the image set as neighbor images of the reference image according to the score sorting. In this embodiment, . Relevance score The calculation formula is as follows:

.

In formula (1), is the sparse point set seen in the reference image I _r , is the sparse point set seen in image I _n . is the area weight factor composed of the sparse points p that are commonly visible in the reference image I _r and the image I _n , and the calculation formula is as shown in formula (2).

(2).

in, It is the minimum bounding box area formed by the commonly visible sparse points in the reference image, as shown in Figure 2; is the image area of the reference image I _r . In Figure 2, the solid line frame represents the reference image I _r , the dotted line frame represents the image I _n , the circles marked with “orange” represent the sparse points visible in the reference image; the circles marked with “purple” represent the sparse points visible in the image I _n ; the circles marked with “blue” represent the sparse points visible in both the reference image and the image I _n .

In formula (1), is the angle weight factor formed by the reference image I _r and the image I _n at the commonly visible sparse point p, and the calculation formula is as shown in formula (3).

(3).

in, represents the angle between the camera center and the sparse point p corresponding to the reference image I _r and the image I _n , as shown in Figure 3. In Figure 3, the circle marked with "orange" indicates the sparse point visible in the reference image; the circle marked with "purple" indicates the sparse point visible in the image I _n ; the circle marked with "blue" indicates the sparse point visible in both the reference image and the image I _n . Should be within a certain range. If the image geometry formed by the reference image I _r and the image I _n is poor, the depth value estimation is inaccurate. The larger the angle of view, the greater the difference between the reference image I _r and the image I _n , which may easily lead to inaccurate calculation of the matching cost. , the maximum angle .

In formula (1), is the scale weight factor of the reference image _Ir and the _image In at the commonly visible sparse point p, and the calculation formula is as shown in formula (4).

(4).

in, , It represents the projection distance of the unit sphere centered on the sparse point p on the image I _i , as shown in Figure 4. When the scale difference of the projection distance of the unit sphere centered on the sparse point p on the reference image I _r and the image I _n is small, the resolution of the sparse point p in the images I _r and I _n is similar, and the calculation of the matching cost is more accurate.

1.2 Initialization of depth information.

Dense matching algorithms based on photometric consistency do not work well with sparse point clouds. For example, the depth information is randomly initialized based only on the depth range of the sparse point cloud. Although dense matching algorithms based on photometric consistency do not require good initial values, this will increase the number of iterations of the algorithm and reduce robustness, especially when the scene depth range is large. To overcome these problems, the present embodiment uses the sparse point cloud obtained in the sparse reconstruction step to initialize the depth information. This can generate more accurate initial depth information for each image. The process includes regularization of the sparse point cloud and regular initialization of the depth information.

Sparse point cloud regularization refers to the use of sparse point clouds visible in image I _i Get the approximate shape of the scene. The process is divided into three steps: first, the sparse point cloud Project it onto image I _i and get the corresponding projection point set ; Then the projection points are triangulated using the Delaunay algorithm to generate a two-dimensional mesh; Finally, the two-dimensional mesh is converted into a three-dimensional mesh _Mi by combining the depth information of the projection points. Specifically, for each projection point, its depth information can be estimated by a sparse reconstruction algorithm. After the two-dimensional mesh is generated using the Delaunay algorithm, the connection relationship between the projection points is known. Combining the depth information of the projection points with the connection relationship between the projection points can convert the two-dimensional mesh into a three-dimensional mesh, thereby obtaining the approximate shape of the scene.

The rule of initializing depth information is to use the three-dimensional grid _Mi calculated from image _Ii to initialize the depth information of image _Ii . The process is divided into three steps: First, the pixel position in image _Ii is calculated based on the image pose obtained by sparse reconstruction. The corresponding projection light . Project the light Projected into the three-dimensional grid _Mi , obtain the same Intersecting triangles ; Then calculate the mathematical expression of the plane where the triangle is located based on the vertex coordinates of the triangle ; Finally, the pixel is calculated according to formula (5) The initial depth information, that is, the depth value and normal value .

(5).

in, Represents projection light With triangles The above calculation is repeated for all pixels of image I _i , and finally the initial depth information of image I _i is obtained.

For strong texture areas with significant features in the image, both random initialization and rule initialization can obtain initial values close to the true value. However, rule initialization uses sparse point cloud information with good scene structure preservation, so it requires fewer iterations and is more robust than random initialization. For weak texture areas with insignificant features in the image, rule initialization can obtain initial values closer to the true value than random initialization. The iterative results of random initialization and rule initialization of depth information are shown in Figures 5 to 10.

2. Multi-view dense matching based on scene line feature constraints.

The depth map of each oblique image obtained by multi-view dense matching is an important data source for three-dimensional reconstruction based on oblique images. However, there are usually two problems in the process of oblique image processing: (1) The photometric consistency of weak texture areas in oblique images is no longer reliable, resulting in inaccurate matching results between images. (2) The ACMP algorithm does not consider the structural features of the scene when constructing the plane prior model, resulting in inaccurate plane prior information. In response to the above two problems, this embodiment is based on the ACMP algorithm and designs a multi-view dense matching method based on the plane prior assisted optimization of the scene line feature constraint. The core idea is that since the line features of the image can roughly describe the structural features of the scene, the image is divided into a line feature area and a non-line feature area based on the line features of each image. Matching points with different confidence intervals are selected as structural points for different areas, and a Delaunay triangulation is constructed to generate high-quality plane prior information. A plane optimization probability graph model is constructed by combining the plane prior information and the photometric consistency between images, so that the depth information of the weak texture area tends to the depth information of the plane structure, thereby optimizing the problem of inaccurate depth information estimation in the weak texture area. The main process includes scene line feature region extraction and multi-view dense matching assisted by plane prior optimization.

1 Scene line feature region extraction.

The line feature area of the scene is a linear buffer area established with the line feature extracted from the image as the center and a certain pixel size as the radius. In the image, weak texture areas usually have strong planar characteristics, and fewer line features are extracted from these areas, so they are usually non-line feature areas. On the contrary, strong texture areas usually have obvious structural features, and more line features are extracted from these areas, so they are usually line feature areas. A small number of high-confidence matching points are selected as structural points in the non-line feature area of the image to ensure that the weak texture area generates larger triangular primitives, so that a larger plane structure constraint is used in the weak texture area to retain the flatness of the depth information at the plane structure; on the contrary, a large number of matching points with credible confidence are selected as structural points in the line feature area of the image to ensure that the strong texture area generates relatively fine triangular primitives, so that a smaller plane structure constraint is used in the strong texture area to retain the depth information mutation at the edge structure.

Currently, the commonly used line feature detection methods are: Canny edge detection algorithm, Hough transform detection method, EDLines straight line detection method, and LSD straight line detection method.

(1) Canny edge detection algorithm: The basic idea of the Canny edge detection algorithm is to use the gradient information in the image to detect the edges in the image through multiple steps. The Canny algorithm can accurately detect the edges in the image and extract the edges finely. It also removes the noise in the image through Gaussian filtering, which can effectively improve the accuracy of edge detection. However, the Canny algorithm requires multiple convolutions and operations on the image, which requires a large amount of calculation and a slow processing speed. The threshold in the algorithm needs to be adjusted according to the specific image. There is no clear regulation on how to choose the threshold, and it needs to be tried and adjusted according to the actual situation. In general, the Canny edge detection algorithm is a high-precision image edge detection algorithm, but in practical applications, it is necessary to consider its large amount of calculation and difficult parameter setting.

(2) Hough transform detection method: Hough transform transforms the image coordinate space into parameter space, and uses the duality of points and lines to transform the curve in the original image space into a point in the parameter space through curve expression. In this way, the curve detection problem in the original image is transformed into the problem of finding the peak in the parameter space. The Hough transform detection method has a good effect on the detection of irregular shapes and can detect objects of any shape, including irregular shapes such as straight lines, circles, and polygons. However, this method has high computational complexity and needs to be calculated for each pixel. Therefore, the computational complexity is high when processing large images, and this method usually has computational errors.

(3) EDLines line detection method: The basic idea of EDLines line detection method is to first traverse the pixel chain generated in advance to screen out all the initial straight line segments, and then use the fitting method to fit them to make them more complete straight line segments. This method extracts straight line segments by connecting edge points, and can obtain more accurate straight line segment detection results. This method has a faster calculation speed. However, this method is sensitive to noise and partial occlusion, which can easily affect the accuracy of straight line segment detection results. In addition, the straight line segment detection results of this method are related to parameter selection, and certain parameter adjustments are required to obtain better detection results.

(4) LSD line detection method: The LSD line detection algorithm is a line detection algorithm based on image gradients. It can detect sub-pixel lines in linear time and improve the calculation speed by merging pixels with similar gradient directions. It also has adaptive parameter adjustment and controllable error rate. Therefore, the LSD line detection algorithm is regarded as a milestone in modern line detection algorithms. However, the algorithm has certain limitations on the length of lines, and long lines are easily split into multiple short lines.

Among the above four line feature detection algorithms, the Canny edge detection algorithm and the Hough transform detection method have high computational complexity and require long running time, so these two methods are not suitable for line feature extraction in the method of this embodiment. In contrast, both the LSD line detection method and the EDLines line detection method have faster detection speeds, but the LSD line detection method can use adaptive parameters to detect sub-pixel level lines. To this end, this embodiment extracts line features in the image based on the LSD line detection method, and uses this to perform binary division on the image, dividing the image into a line feature area and a non-line feature area.

The LSD line detection method mainly includes four key steps: scale scaling, gradient calculation, region growing and line extraction.

(1) Scaling: The image is scaled by Gaussian downsampling to reduce or resolve the aliasing and quantization artifacts that appear in the image.

(2) Gradient calculation: Gradient calculation usually uses the pixels to the right and below each pixel to calculate the gradient size corresponding to the pixel. and direction , the specific calculation formula is shown in equation (6) and equation (7). This method aims to use other pixels as little as possible, thereby enhancing its robustness to noisy images.

(6).

(7).

in, and The calculation formula of is shown in formula (8). Represents the gradient magnitude in the x direction at the pixel (x, y); Represents the gradient magnitude in the y direction at the pixel (x, y).

(8).

in, Pixel The gray value of the image at .

(3) Region growing: Through a greedy algorithm, adjacent pixels with consistent gradient directions are connected to form a connected domain, and a rectangular frame is generated on this basis. The connected domain is also called the straight line support region. After the rectangular frame is generated, it is determined whether the connected domain needs to be disconnected according to the rules based on the size of the rectangularity to form multiple connected domains with larger rectangularity.

(4) Line extraction: Rectangle improvement and screening of all generated connected domains are performed, and those that meet the requirements are retained. The connected domain of the condition is the final straight line detection result. The calculation formula of is shown in formula (9).

(9).

in, is the image size, is the number of pixels in the straight line support area, k _lsd is the number of pixels perpendicular to the straight line support area, Improved precision for rectangles, express The number of different values of .

Since the LSD line detection method is a local detection algorithm, the extracted line features are relatively scattered and discontinuous. Therefore, in order to better reflect the structural characteristics of the scene, this embodiment connects the line features extracted by the LSD line detection method with line segments. The main steps include the following three steps.

(1) In order to better represent the structure of the scene and reduce the computational complexity, the line segment set extracted by the LSD line detection method is filtered to remove the line segments whose length is less than the length threshold. These shorter line segments cannot effectively represent the structure of the scene, but will increase the amount of calculation in subsequent steps. Therefore, they should be filtered and removed before the line segments are connected.

(2) After filtering the line segment set, connect the line segments on the same straight line. Traverse all the line segments in the line segment set in turn and connect any line segment With other segments in the segment set Perform matching verification and determine whether two line segments can be connected based on the angle, distance and other conditions between the line segments. The specific judgment process is as follows:

① Determination of line segment angle: According to the line segment , The slope , , calculate the angle between two line segments , the calculation formula is as shown in formula (10).

(10).

if Less than angle threshold , then the two line segments are judged to be approximately parallel, as shown in Figure 11.

②Vertical relative distance judgment: through line segments Calculate the line segment using the endpoint coordinates Equation of the line . Calculate the line segments separately The two endpoints , To Segment Distance , , the calculation formula is as shown in formula (11).

(11).

Pick , The mean of , The vertical distance between If the vertical distance Less than distance threshold , it means that the two line segments are on the same horizontal line, as shown in Figure 12.

③Endpoint distance judgment: calculation point To point , The distance between , ,point To point , The distance between , .from , , , Find the maximum and minimum values in , , the calculation formula is as shown in formula (12).

(12).

if Less than distance threshold , it means that the two line segments are adjacent, as shown in Figure 13.

if Greater than distance threshold , but if there is an overlapping part between the two line segments, it can also be said that the two line segments are adjacent, as shown in Figure 14.

If both line segments meet the above three conditions, the least squares method is used to fit and optimize the two line segments to synthesize the optimal straight line segment. The steps of least squares method fitting optimization are as follows.

The straight line The representation is converted into of the form, in which , According to the least squares principle, there exists a function such as formula (13).

(13).

in, Indicates the number of endpoints involved in the fitting. In this embodiment, When this function takes the minimum value, the two parameters A and B of the fitting line are obtained. According to the extreme value theorem of the function, there exists a set of equations such as equation (14).

(14).

Solve the equations to get the values of parameters A and B, and obtain the expression of the best fitting straight line. , The two endpoints that are farthest apart are used to calculate the endpoints of the fitted line segment. After the fitted line segment is calculated, the line segment is removed from the line segment set. , , and add the fitting line segment. Then, the fitting line segment is used as the new Verify by matching with other segments. Repeat this process until all segments in the segment set have been used as Perform matching verification to obtain a new set of line segments.

(3) After obtaining a new line segment set, connect the line segments that are not on the same straight line but have similar endpoint distances, as shown in Figure 15. Traverse all the line segments in the new line segment set in turn and calculate any line segment With other segments in the segment set The endpoint distance is less than the distance threshold. The line segments constitute The corresponding collection segment set and the segment Add to the collection line segment set. Traverse all line segments in the collection line segment set in turn, and calculate the intersection of the line segment with the lines of other line segments in the line segment set. The average of all calculated intersection coordinates is used as the new endpoint of the line segment in the collection line segment set.

After connecting the line features extracted by the LSD line detection method through the above steps, a buffer zone of feature line (BZFL) is constructed according to the empirical value (5 pixels). Finally, the image is divided into two values according to formula (15).

(15).

in, Represents the pixel coordinates. When the pixel coordinates are located in the non-line feature area, the pixel value ; When the pixel coordinates are in the line feature area, the pixel value The generated mask images before and after are shown in Figures 16 and 17.

Multi-view dense matching with planar prior optimization.

The dense matching method based on photometric consistency mainly relies on the photometric consistency between images to evaluate the accuracy of matching. Since the strong texture areas in the image have obvious scene structure characteristics, the photometric consistency of these areas is relatively reliable and accurate depth information can be obtained. However, since the weak texture areas usually have planar characteristics, the photometric consistency in the weak texture areas is no longer reliable, and the depth information obtained is inaccurate. The core idea of the multi-view dense matching method based on plane prior optimization is to construct a probabilistic graph model based on plane prior information and combined with photometric consistency, so that the depth information of the weak texture area tends to the depth information of the plane structure, thereby optimizing the inaccurate depth information estimation problem of the weak texture area. The main process includes plane prior information extraction, pixel-level image selection and construction of a plane optimization model.

The extraction of plane prior information refers to the use of selected matching points with better confidence as structure points to construct plane prior information. This embodiment uses the ACMH method for initial matching to obtain the initial depth map of each image, and selects matching points of different confidence intervals according to the binarization result of the image. The value of the confidence is greater than or equal to 0. Taking the second confidence threshold as 0.05 and the first confidence threshold as 0.1 as an example, the specific steps are: first, select matching points with a confidence less than 0.1 as candidate structure points; then determine the area where the candidate structure points are located according to the pixel coordinates of the candidate structure points. If it is a non-line feature area, select structure points with a confidence interval of [0~0.05], thereby improving the quality of the structure points and reducing the number of structure points. If it is a line feature area, select structure points with a confidence interval of [0~0.1], thereby increasing the number of structure points while maintaining the accuracy of the depth information; finally, use the Delaunay algorithm to construct a network and generate triangular primitives. For each triangular primitive, use its corresponding three vertices to calculate the plane prior information and the plane equation in the object space. Pixels within the same triangle primitive share the same plane prior information.

Image selection at the pixel level is to select the visible image set by using the visibility of the neighboring pixels of each pixel in the neighboring image. This embodiment draws on the "red and black chessboard" parallel propagation strategy to divide the reference image pixels into "red and black" pixels, as shown in Figure 18. In Figure 18, the circle marked with the word "red" represents the "red" pixel; the circle filled with black represents the "black" pixel; in the parallel propagation process, the depth information of the "red" pixel and the "black" pixel is propagated in parallel in sequence, that is, the depth information of the neighboring "red" pixel is propagated to the "black" pixel, and vice versa. The propagation path is shown in Figure 19. In Figure 19, the circle marked with the word "red" represents the "red" pixel; according to the distance of the propagation path, the neighborhood pixels are divided into 4 levels, as shown in Figure 20. In Figure 20, the circle marked with the word "red" represents the "red" pixel; the circle marked with the word "yellow" represents the yellow pixel; each level from deep to shallow represents the distance from near to far. From each level, two pixels with the smallest matching cost are selected as pixels with reliable depth information, and a total of 8 pixels are selected as neighborhood pixels for view selection, as shown in the yellow pixels in Figure 20. Finally, the visibility of neighbor images is calculated using the distance weighted method, and the specific calculation formula is shown in Equation (16).

(16).

in, represents the visibility of neighbor image j, Neighborhood pixels The inverse distance weight of Represents the neighborhood pixels In the previous iteration, the NCC matching cost is taken according to the empirical value. From equation (16), it can be concluded that the closer the distance between the neighbor pixel and the current pixel is, and the higher the matching cost relative to the neighbor image j, the higher the visibility of the neighbor image j. Sort the visibility of all neighbor images of the reference image from high to low, and take the top neighbor images as the visible image set of the current pixel.

Constructing a plane optimization model means using a probabilistic graph model to combine plane prior information and photometric consistency to construct a new multi-view matching cost. Since this new multi-view matching cost takes into account both photometric consistency and plane prior information, it is not only suitable for recovering depth information in strong texture areas, but also effectively recovering depth information in weak texture areas. First, the depth information of the current pixel to be updated and the depth information of the 8 neighboring pixels selected in the previous step are used as the candidate hypothesis set for the current pixel to be updated.

.

in Indicates the depth information of the plane structure corresponding to the current pixel to be updated, d _i is the disparity value of the i-th pixel (the current pixel to be updated); _ni is the normal value of the i-th pixel. visible images, each candidate hypothesis will have NCC matching cost, that is, the current pixel q to be updated is taken in the candidate hypothesis The photometric consistency with the visible image j is calculated as shown in formula (17).

(17).

in, It means that the pixel q to be updated is the center of the A rectangular window, Represents pixels The pixel value in the reference image, Represents pixels The pixel values of the corresponding pixels after homography transformation on the visible image. These candidate hypotheses correspond to The matching costs form a matrix as shown in formula (18).

(18).

The matching cost of each row of the matrix can measure the candidate hypothesis Relative to the reliability of the visible image, we use formula (19) to take the weighted average of the top m best NCC matching costs as the current pixel to be updated relative to the candidate hypothesis The final matching cost.

(19).

in, Match the cost for NCC.

Secondly, in order to use plane prior information to assist in optimizing dense matching, this embodiment constructs a plane optimization probability graph model, and its calculation formula is as shown in formula (20).

(20).

Among them, _dp is the disparity value of the p-th pixel; _np is the normal value of the p-th pixel; the neighbor propagation weight factor , plane priori weight constant , the parallax step size , the step size of the normal difference , the value is shown in formula (21).

(twenty one).

Among them, d _max is the maximum value of the normal line; d _min is the minimum value of the normal line.

For equation (20), the first term on the right side of the equation is the matching cost, corresponding to the current pixel q to be updated in the candidate hypothesis The second term is the regular term of the plane prior. If the depth information of the current pixel q to be updated is consistent with the depth information of the plane structure, a reward is given, otherwise a penalty is given, so that the depth information of the weak texture area tends to the depth information of the plane structure, and the depth estimation of the weak texture area is optimized. Finally, The best candidate hypothesis is used as the depth information of the current pixel to be updated.

3 Experiment and analysis

3.1 Experimental data and environment.

The experiment of this embodiment will verify the effectiveness of the method of this embodiment from the aspects of plane information extraction results, depth information calculation results and plane area flatness. The experimental data include: (1) the public dataset Dortmund provided by ISPRS, which is captured by a five-lens oblique imaging system, with 584 images and a resolution of 8176×6132. (2) the oblique photography dataset collected on site, mainly composed of two different scenes, Garden and Central-Urban, with 162 and 566 images respectively, and resolutions of 5472×3648 and 4864×3648 respectively. The main types of objects covered include: buildings, vegetation, roads, water surfaces, etc., which are of universal significance for experimental verification.

The experimental running environment is a workstation with Windows 10 64-bit operating system, Intel Core(TM) i9-10900X CPU (main frequency is 3.70GHz), and 128GB memory.

3.2 Analysis of plane prior information extraction results.

Plane prior information refers to the triangular primitives generated by selecting some matching points with good confidence as structural points and using the Delaunay algorithm to construct the network. The existing ACMP method does not consider the scene structure characteristics when selecting matching points. The method of this embodiment is improved on the basis of the existing ACMP method, and the scene is divided into a line feature area and a non-line feature area. A small number of high-confidence matching points are selected in the non-line feature area, and a large number of matching points with credible confidence are selected in the line feature area, thereby ensuring that there are larger plane structure constraints in weak texture areas and smaller plane structure constraints in strong texture areas, and optimizing the depth estimation results in different areas. Figures 21-32 show the extraction results of the ACMP method and the method of this embodiment on the Dortmund dataset, Garden dataset, and Central-Urban dataset.

In the strong texture area of the image, the triangular primitives formed by the ACMP method and the method of this embodiment are relatively small, and both methods can better maintain the structural features of the scene; in the weak texture area of the image, the triangular primitives formed by the ACMP method and the method of this embodiment are generally relatively large, but in some weak texture areas, the triangular primitives formed by the ACMP method are too fragmented. The method of this embodiment uses the line feature constraints of the scene to construct plane prior information, and selects a small number of high-confidence matching points as structural points in the non-line feature area, so the triangular primitives formed are relatively large, which can better ensure the accuracy of depth estimation in weak texture areas.

3.3 Analysis of depth information calculation results.

For the Dortmund dataset, the Garden dataset, and the Central-Urban dataset, the depth information calculated by the ACMP method and the method of this embodiment in regions 1 to 6 is shown in FIGS. 33 to 44 .

As can be seen from Figures 33 to 44, the ACMP method and the method of this embodiment can calculate relatively accurate depth information in strong texture areas. In most weak texture areas, the calculated depth information is also relatively smooth. This is mainly because the ACMP method and the method of this embodiment add plane prior information on the basis of photometric consistency, so that the depth estimation of weak texture areas can be optimized. However, in some weak texture areas, the depth information calculated by the ACMP method will have some outliers. This is mainly because the plane prior information generated by the ACMP method in these areas is too fragmented, so it can only partially alleviate the abnormal depth information in weak texture areas. The method of this embodiment adds sparse point cloud initialization depth information and the construction of plane prior information based on scene line feature constraints on the basis of the ACMP method, so it can better improve the problem of inaccurate depth information estimation in weak texture areas.

3.4 Flatness analysis of plane area.

In the multi-view dense matching process, the ACMP method does not consider the scene structure features, and only uses matching points with better matching costs to construct plane prior information, resulting in more plane prior information in weak texture areas, and the generated dense point cloud is not flat enough in the plane area. On the basis of the ACMP method, the method of this embodiment uses sparse point clouds to initialize depth information, and uses scene line feature constraints to construct plane prior information, optimizes the number of plane prior information in weak texture areas, and can improve the flatness of the generated dense point cloud in the plane area. To verify the above analysis results, in this section of the experiment, the method of this embodiment uses the deep fusion method in the ACMP method to generate dense point clouds, and uses local plane accuracy to evaluate the flatness of the ACMP method and the method of this embodiment in the plane area. The local plane accuracy refers to the root mean square error value of the distance from the dense point cloud to the local fitting plane. Figures 45-56 show the elevation views of the dense point clouds in the plane area (area one to area six). The statistical results of the local plane accuracy of these plane areas are shown in Table 1. The unit of local plane accuracy in Table 1 is cm.

Table 1. Flatness comparison of plane areas

ACMP Method The method of this embodiment improve(%) Area 1 0.328 0.227 30.79% Area 2 0.247 0.184 25.51% Area 3 0.195 0.149 23.58% Area 4 0.24 0.174 27.5% Area 5 0.308 0.216 29.87% Area 6 0.214 0.156 27.1%

It can be concluded from Figures 45-56 and Table 1 that in areas 1 to 6, the flatness of the dense point cloud generated by the fusion of the method of this embodiment is higher than that of the ACMP method, for example: an increase of 30.79% in area 1, an increase of 23.58% in area 3, and an overall average increase of 27.39%. This is mainly because the method of this embodiment introduces the initial depth information of the sparse point cloud and uses the scene line feature constraints to construct the plane prior information. In the same number of iterations, the method of this embodiment can converge faster, generate less abnormal depth information in the weak texture area, and can effectively improve the flatness of the weak texture area.

As an emerging technology in photogrammetry and computer vision, the 3D modeling technology based on oblique images is widely used in applications such as urban real-scene 3D model reconstruction and new basic surveying and mapping due to its high reconstruction efficiency and realistic effects. Dense reconstruction is a key step in building high-quality models in 3D modeling technology based on oblique images, but there is a problem of low matching accuracy in weak texture areas. To address this problem, this embodiment studies multi-view dense matching from the perspective of plane prior optimization. The main contents and conclusions are as follows.

Firstly, the preprocessing of multi-view dense matching is introduced, which mainly includes the calculation of image combination and the initialization of depth information. Then, the multi-view dense matching method and theoretical basis based on plane prior-assisted optimization with scene line feature constraints are briefly described, which mainly includes scene line feature region extraction and multi-view dense matching with plane prior optimization. Finally, three sets of data sets are used for experiments and qualitative and quantitative analysis.

Experiments have shown that initializing depth information through sparse point clouds can speed up the rapid propagation of accurate depth information and better maintain the structural features of the scene; multi-view dense matching based on plane prior-assisted optimization based on scene line feature constraints can effectively remove abnormal depth information in weak texture areas while maintaining the structural features of strong texture areas, and improve the flatness of weak texture areas. Experiments show that the overall average improvement is 27.39%. Therefore, the method of this embodiment can effectively improve the dense matching accuracy of weak texture areas.

Embodiment 2: In order to execute the method corresponding to the above embodiment 1 to achieve the corresponding functions and technical effects, a multi-view dense matching system based on plane prior optimization is provided below, including: a scene data acquisition module, which is used to acquire a sparse point cloud and multiple images of the scene to be modeled. Different images are shot at different angles.

The current reference image determination module is used to determine any image as the current reference image.

The neighbor image set determination module is used to determine the neighbor image set of the current reference image from multiple images according to the sparse point cloud of the scene to be modeled.

The initialization depth module is used to initialize the depth information of all pixels in the current reference image by triangulation of the sparse point cloud, and obtain the initialization depth map of the current reference image. The depth information includes depth value and normal value.

The line feature set extraction module is used to extract the line feature set in the depth map initialized by the current reference image.

The line feature region division module is used to divide the current reference image into a line feature region and a non-line feature region according to a line feature set.

The plane prior information construction module is used to construct plane prior information according to the line feature area and the non-line feature area.

The neighborhood pixel set determination module is used to determine the neighborhood pixel set of each pixel in the current reference image.

The visible image set determination module is used to determine the visible image set according to the visibility of the neighborhood pixel set of each pixel in the current reference image in the neighbor image set of the current reference image.

The multi-view matching cost function determination module is used to construct a new multi-view matching cost function by using a probabilistic graph model to combine plane prior information, a visible image set and multiple current line features.

The sub-dense matching result determination module is used to update the depth information of all pixels in the current reference image according to the multi-view matching cost function to obtain the dense matching result of the current reference image.

The total dense matching result determination module is used to update the current reference image and return to execute the neighbor image set determination module until all images are traversed to obtain the dense matching results of multiple images.

The multi-view dense matching module is used to fuse the dense matching results of multiple images to obtain the multi-view dense matching results of the scene to be modeled.

Embodiment 3: This embodiment provides an electronic device, including a memory and a processor, wherein the memory is used to store a computer program, and the processor runs the computer program to enable the electronic device to perform a multi-view dense matching method based on plane prior optimization described in Embodiment 1. The memory is a readable storage medium.

In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the system disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

In this embodiment, specific examples are used to illustrate the principle and implementation of the present invention. The above description of the embodiment is only used to help understand the method and core idea of the present invention; at the same time, for those skilled in the art, according to the idea of the present invention, there will be changes in the specific implementation and application scope. In summary, the content of this specification should not be understood as limiting the present invention.

Claims (10)

1. The multi-view dense matching method based on plane prior optimization is characterized by comprising the following steps of:

acquiring sparse point clouds and a plurality of images of a scene to be modeled; the shooting angles of different images are different;

determining any image as a current reference image;

determining a neighbor image set of the current reference image from the plurality of images according to the sparse point cloud of the scene to be modeled;

initializing depth information of all pixel points in a current reference image by utilizing triangulation processing of sparse point cloud to obtain an initialized depth map of the current reference image; the depth information includes a depth value and a normal value;

extracting a line feature set in a current reference image initialization depth map;

dividing the current reference image into a line characteristic region and a non-line characteristic region according to the line characteristic set;

Constructing plane prior information according to the line characteristic region and the non-line characteristic region;

determining a neighborhood pixel set of each pixel in the current reference image;

determining a visible image set according to the visibility of the neighborhood pixel set of each pixel in the current reference image in the neighbor image set of the current reference image;

constructing a new multi-view matching cost function by using the probability map model through the plane prior information, the visible image set and the current line features;

updating the depth information of all pixel points in the current reference image according to the multi-view matching cost function to obtain a dense matching result of the current reference image;

updating the current reference image and returning to the step of determining a neighbor image set of the current reference image from the plurality of images according to the sparse point cloud of the scene to be modeled until all the images are traversed, so as to obtain a dense matching result of the plurality of images;

and fusing the dense matching results of the multiple images to obtain a multi-view dense matching result of the scene to be modeled.

2. The multi-view dense matching method based on planar prior optimization of claim 1, wherein determining a neighbor image set of a current reference image from a plurality of images according to a sparse point cloud of a scene to be modeled comprises:

Determining a visible sparse point set of the current reference image from a sparse point cloud of a scene to be modeled;

determining any image to be fixed as a current image to be fixed; the undetermined images are a plurality of images except the current reference image;

determining a visible sparse point set of the current undetermined image from a sparse point cloud of a scene to be modeled;

determining the intersection of the visible sparse point set of the current reference image and the visible sparse point set of the current undetermined image as a common visible sparse point set of the current reference image and the current undetermined image;

determining a correlation score of the current reference image and the current image to be determined according to the common visible sparse point set;

updating the current image to be determined, and returning to the step of determining a visible sparse point set of the current image to be determined from the sparse point cloud of the scene to be modeled until all the images to be determined are traversed, so as to obtain the relevance scores of the current reference image and all the images to be determined;

descending order of the images to be determined according to the relevance scores;

and determining the pre-set number of undetermined images as a neighbor image set of the current reference image.

3. The multi-view dense matching method based on plane prior optimization according to claim 2, wherein initializing depth information of all pixel points in a current reference image by using triangulation processing of sparse point cloud to obtain an initialized depth map of the current reference image comprises:

Projecting the visible sparse point set of the current reference image onto the current reference image to obtain a projection point set of the current reference image;

triangulating the projection point set of the current reference image by using a Delaunay algorithm to generate a two-dimensional grid;

constructing a three-dimensional grid according to sparse depth information of a projection point set of the current reference image;

determining the image pose of the current reference image by using sparse reconstruction;

determining any pixel point in the current reference image as a current pixel point;

calculating the current projection light corresponding to the current pixel point according to the image pose;

projecting the current projection light to the three-dimensional grid, and determining a triangular patch intersecting the current projection light in the three-dimensional grid as a current triangular patch;

determining a plane equation of the current triangular patch according to the 3 vertex coordinates of the current triangular patch;

determining the depth value and the normal value of the current pixel point according to the coefficient of the plane equation of the current triangular patch;

updating the current pixel point and returning to the step of calculating the current projection light corresponding to the current pixel point according to the pose of the image until all the pixel points in the current reference image are traversed, and obtaining the initialization depth map of the current reference image.

4. The multi-view dense matching method based on planar prior optimization of claim 1, wherein extracting a line feature set in a current reference image initialization depth map comprises:

extracting a plurality of line features in the initialization depth map of the current reference image by using an LSD linear detection method to obtain an initial line feature set; the line features are line segments;

deleting line features with lengths smaller than a length threshold value in the initial line feature set;

connecting collinear line features in the initial line feature set;

determining any line feature in the initial line feature set as a current line feature;

determining a plurality of line features except the current line feature in the initial line feature set as line features to be matched;

determining any line feature to be matched as the current line feature to be matched;

determining an included angle between the current line characteristic and the current line characteristic to be matched as a current included angle;

when the current included angle is larger than or equal to the included angle threshold value, updating the current line feature to be matched, and returning to the step of determining that the included angle between the current line feature and the current line feature to be matched is the current included angle;

when the current included angle is smaller than the included angle, determining that the current line characteristic and the current to-be-matched line characteristic are current approximate parallel line characteristic pairs;

Constructing a linear equation of the current line feature to be matched according to two end point coordinates of the current line feature to be matched;

determining the distance from one end point of the current line characteristic to be matched as a first distance according to a linear equation of the current line characteristic to be matched;

determining the distance from the other end point of the current line characteristic to be matched as a second distance according to a linear equation of the current line characteristic to be matched;

determining the average value of the first distance and the second distance as the vertical distance of the current approximate parallel line feature pair;

when the vertical distance is smaller than the vertical distance threshold, fitting the current approximate parallel line characteristic pair by using a least square method to obtain a fitting line characteristic;

replacing the current approximate parallel line feature pairs in the initial line feature set by fitting line features;

taking the fitting line characteristic as the current line characteristic, and returning to the step of determining any line characteristic to be matched as the current line characteristic to be matched until all line characteristics to be matched are traversed;

updating the current line characteristics, returning to the step of determining that a plurality of line characteristics except the current line characteristics in the initial line characteristic set are to-be-matched line characteristics until the initial line characteristic set is traversed, and determining the to-be-fixed line characteristic set.

5. The multi-view dense matching method based on planar prior optimization of claim 4, further comprising, after determining the set of features to be defined:

determining any line characteristic in the to-be-determined line characteristic set as a current line characteristic;

determining any endpoint of the current line feature as a current endpoint;

determining the end points of all line features except the current line feature in the to-be-determined line feature set as a current end point set;

determining a plurality of endpoints, of which the distances from the current endpoint in the current endpoint set are smaller than a distance threshold, as endpoints to be combined;

determining the current endpoint and a plurality of endpoints to be combined as a point set to be combined;

determining the line characteristics of the point set to be combined as the line characteristic set to be combined;

determining the coordinate mean value of the point set to be combined as the coordinate of the combining point;

all points in the point set to be combined are connected with the combining point to obtain combining line characteristics;

and replacing the to-be-merged line feature set in the to-be-determined line feature set by adopting the merged line feature, and returning to the step of determining any line feature in the to-be-determined line feature set as the current line feature until the to-be-determined line feature set is traversed, so as to obtain the line feature set in the initialization depth map of the current reference image.

6. The multi-view dense matching method based on planar prior optimization of claim 1, wherein constructing planar prior information from the line feature region and the non-line feature region comprises:

performing primary matching by using an ACMH method, and determining that a matching point with the confidence coefficient smaller than a first confidence coefficient threshold value in the current reference image is a pending structural point;

determining all undetermined structural points in the line characteristic area as selected structural points;

determining candidate structural points with the confidence coefficient smaller than a second confidence coefficient threshold value in the non-line characteristic region as selected structural points; the second confidence threshold is less than the first confidence threshold;

constructing a network by adopting a Delaunay algorithm based on a plurality of selected structure points to generate a plurality of triangle primitives;

determining any triangle primitive as the current triangle primitive;

determining current plane prior information according to 3 vertexes of the current triangle primitive;

constructing a plane equation of the current triangle primitive;

determining the prior information of the current plane as the prior information of the planes of all pixels in the current triangle primitive;

updating the current triangle primitive, and returning to the step of determining the prior information of the current plane according to the 3 vertexes of the current triangle primitive until all the triangle primitives are traversed, and determining the prior information of the planes of all the pixels in the current reference image.

7. The multi-view dense matching method based on plane prior optimization according to claim 4, wherein constructing a new multi-view matching cost function by using a probability map model from plane prior information, a visible image set and a plurality of current line features comprises:

determining any pixel as a current pixel;

determining the depth information of the current pixel and the depth information of each neighborhood pixel in the neighborhood pixel set of the current pixel as candidate hypothesis sets;

determining the luminosity consistency of the current pixel and the visible image when each candidate hypothesis in the candidate hypothesis set is selected to construct a matching cost matrix;

determining the final matching cost of each candidate hypothesis according to the matching cost matrix;

updating the current pixel and returning to the step of determining the depth information of the current pixel and the depth information of each neighborhood pixel in the neighborhood pixel set of the current pixel as a candidate hypothesis set to obtain the final matching cost of selecting different candidate hypotheses for each pixel;

and selecting final matching cost of different candidate hypotheses according to each pixel and plane prior information of each pixel, and constructing a plane optimization probability map model.

8. A multi-view dense matching system based on planar prior optimization, comprising:

The scene data acquisition module is used for acquiring sparse point clouds and a plurality of images of a scene to be modeled; the shooting angles of different images are different;

the current reference image determining module is used for determining any image as a current reference image;

the neighbor image set determining module is used for determining a neighbor image set of the current reference image from the plurality of images according to the sparse point cloud of the scene to be modeled;

the initialization depth module is used for initializing depth information of all pixel points in the current reference image by utilizing the triangulation processing of the sparse point cloud to obtain an initialization depth map of the current reference image; the depth information includes a depth value and a normal value;

the line feature set extraction module is used for extracting a line feature set in the initialization depth map of the current reference image;

the line characteristic region dividing module is used for dividing the current reference image into a line characteristic region and a non-line characteristic region according to the line characteristic set;

the plane prior information construction module is used for constructing plane prior information according to the line characteristic region and the non-line characteristic region;

the neighborhood pixel set determining module is used for determining a neighborhood pixel set of each pixel in the current reference image;

The visible image set determining module is used for determining a visible image set according to the visibility of the neighborhood pixel set of each pixel in the current reference image in the neighbor image set of the current reference image;

the multi-view matching cost function determining module is used for constructing a new multi-view matching cost function by utilizing the probability map model and utilizing the plane prior information, the visible image set and the current line features;

the sub-dense matching result determining module is used for updating the depth information of all pixel points in the current reference image according to the multi-view matching cost function to obtain a dense matching result of the current reference image;

the total dense matching result determining module is used for updating the current reference image and returning to the neighbor image set determining module until all images are traversed to obtain dense matching results of a plurality of images;

and the multi-view dense matching module is used for fusing dense matching results of the plurality of images to obtain a multi-view dense matching result of the scene to be modeled.

9. An electronic device comprising a memory for storing a computer program and a processor that runs the computer program to cause the electronic device to perform a multi-view dense matching method based on planar prior optimization as claimed in any one of claims 1 to 7.

10. The electronic device of claim 9, wherein the memory is a readable storage medium.