CN115719320B

CN115719320B - Tilt correction dense matching method based on remote sensing image

Info

Publication number: CN115719320B
Application number: CN202310029421.6A
Authority: CN
Inventors: 杨阿华; 张强; 常鑫; 赵斐; 高鹏; 汪世辉; 王雅楠; 于潇; 王栋; 张大伟; 闫孝鲁
Original assignee: 63921 Troops of PLA
Current assignee: 63921 Troops of PLA
Priority date: 2023-01-09
Filing date: 2023-01-09
Publication date: 2023-04-14
Anticipated expiration: 2043-01-09
Also published as: CN115719320A

Abstract

The invention relates to the technical field of image processing, in particular to a tilt correction dense matching method based on remote sensing images, which comprises the steps of firstly generating horizontal epipolar line images by sparse homonymy points, further estimating initial parallaxes of left and right horizontal epipolar line image pairs, thereby improving the parallax estimation precision, reducing the search range of matching points, reducing the error matching probability, and reducing the calculated amount of gray level calculation by adopting increment calculation gray level mean value when determining the initial dense homonymy points through one-dimensional search; when searching for the matching point, the searching step length is set to be 2, so that the matching times are reduced, the matching efficiency is improved, and the initial dense homonymous points are finely adjusted through least square image matching, so that the aim of improving the matching precision is fulfilled.

Description

Tilt correction dense matching method based on remote sensing image

Technical Field

The invention relates to the technical field of image processing, in particular to a tilt correction dense matching method based on remote sensing images.

Background

Currently, there are two main ways of reconstructing three-dimensional objects and scenes, namely, active and passive. The method has high reconstruction precision and high reconstruction efficiency on the three-dimensional structure of the scene, but because the method depends on expensive measuring equipment and has a complicated data acquisition process, the method needs to acquire image information while acquiring three-dimensional information when acquiring a virtual three-dimensional model with photo reality, and involves the problem of registration between point cloud and images when data is post-processed, and the acquired data amount and the workload of a calculation process are huge; the passive mode adopts an image sensor to obtain image information of the surface of an object, and recovers three-dimensional structure information of the surface of the object from a two-dimensional image with a certain parallax according to the principle of binocular stereo vision. Active three-dimensional scanning has become an important means for acquiring three-dimensional information in many application fields, but passive three-dimensional modeling based on images is still the most economical, flexible, feasible and widely used method and plays an important role in various fields.

The key point of reconstructing the three-dimensional object, the scene and the earth surface based on the image is dense matching of the homonymous points, and the accurate and dense homonymous points are the basis for acquiring high-precision three-dimensional information. Dense matching is widely applied to fine reconstruction of objects, reconstruction of realistic scenes and generation of digital surface models and real digital orthoimages, and if accurate, dense and efficient matching of homonymous points can be achieved, the problem of low reconstruction efficiency caused by complex and time-consuming precision and data processing in three-dimensional reconstruction based on images is solved.

In the prior art, image matching methods are divided into a feature-based method and an image correlation-based method, the feature-based method is high in precision and accuracy, but only aims at characteristic significant areas such as corners, lines, edges and the like in an image, so that only a sparse matching point set can be obtained, single-point calculation cost is much higher than that of the image correlation method, and the method is generally used for providing seed points or constraint conditions for dense matching; the image correlation-based method has large calculation amount and low efficiency, has self-similarity in a single texture region, and is easy to generate mismatching, so that the use of the method is limited, and therefore, the traditional image matching method is still a main implementation way of three-dimensional reconstruction based on images.

Chinese patent publication No. CN113034556a discloses a frequency domain related semi-dense remote sensing image matching method, aiming at obtaining matching information between two images; twisting one of the two images according to the initial matching information to obtain an image to be matched; dividing a reference image and an image to be matched into a plurality of areas; taking four small regions as a group of region image blocks, setting the centers of the region image blocks as seed points, and counting the gradient direction histograms of the region image blocks to obtain gradient amplitudes and angles corresponding to four angle ranges; setting four priority values of different levels; obtaining a new matching result combining the gradient information of the four adjacent domains of the seed points; adding the matching results of all the seed points into the sparse matching point set to obtain a semi-dense remote sensing image matching point set; and carrying out reference point mismatching correction on the dense matching points to obtain a matching result. According to the technical scheme, when the number of the image blocks in the initial matching area is large, the data processing and calculation workload is large, so that the three-dimensional reconstruction difficulty is increased, and when the number of the image blocks in the initial matching area is small, the matching precision is difficult to guarantee.

Disclosure of Invention

Therefore, the invention provides a tilt correction dense matching method based on a remote sensing image, which is used for solving the problem that the three-dimensional reconstruction precision is difficult to improve due to complex image matching data processing in the process of performing three-dimensional reconstruction by adopting image matching in the prior art.

In order to achieve the above object, the present invention provides a tilt correction dense matching method based on remote sensing images, comprising the following steps:

step S1, respectively extracting feature points from a left oblique image shot by a left camera and a right oblique image shot by a right camera by adopting an SIFT feature descriptor to perform feature matching to obtain sparse homonymy points;

s2, carrying out relative orientation on the stereopair based on the sparse homonymy points and the camera intrinsic parameters to obtain the relative pose relationship of the left camera and the right camera;

s3, constructing a horizontal epipolar line coordinate system based on the relative pose relationship, and correcting the left oblique image and the right oblique image into a left horizontal epipolar line image and a right horizontal epipolar line image respectively through the mapping relationship between the horizontal epipolar line coordinate system and the left camera coordinate system as well as the mapping relationship between the horizontal epipolar line coordinate system and the right camera coordinate system;

s4, determining a plurality of dense points to be matched at set matching intervals in the left horizontal epipolar line image acquired in the S3, and performing one-dimensional gray-scale correlation matching on any dense point to be matched in the right horizontal epipolar line image to acquire initial dense homonymous points;

and S5, performing least square image matching on the initial dense homonymous points to obtain sub-pixel-level dense homonymous points.

Further, in step S2, the relative pose relationship is obtained by the sparse homonymy point and the camera intrinsic parameter by a relative orientation direct solution and determined by bundle adjustment optimization, and the relative pose relationship is determined by a pose transformation matrix

Describe, is taken up or taken off>

Setting the position of any point P in the left camera coordinate system as ^>

Position of the same-name point in the right camera coordinate system >>

，

Wherein, the first and the second end of the pipe are connected with each other,

is a rotation matrix of the right camera coordinate system relative to the left camera coordinate system, is based on the value of the reference value>

Is a translation vector.

Further, in step S3, the horizontal epipolar line coordinate system is a right-handed three-dimensional rectangular coordinate system, the origin of the horizontal epipolar line coordinate system is set as the origin of the left camera coordinate system, the X-axis of the horizontal epipolar line coordinate system is a connection line between the left camera projection center and the right camera projection center,

the coordinates of the X-axis vector of the horizontal epipolar line coordinate system in the left camera coordinate system are as follows:

；

the coordinates of the Y-axis vector of the horizontal epipolar line coordinate system in the left camera coordinate system are as follows:

；

the coordinate of the Z-axis vector of the horizontal epipolar line coordinate system in the left camera coordinate system is as follows:

；

the rotation transformation matrix from the vector under the left camera coordinate system to the direction of the horizontal epipolar line coordinate system is as follows:

the rotation transformation matrix from the vector under the coordinate system of the right camera to the direction of the horizontal epipolar line coordinate system is as follows:

wherein the content of the first and second substances,

is the average of the Z-axis direction vector of the left camera coordinate system and the Z-axis direction vector of the right camera coordinate system,

，/>

is the Z-axis vector of the left camera coordinate system,

is a Z-axis vector of the right camera coordinate system, is->

Is said rotation matrix>

The third column of elements.

Further, in step S3, by

And &>

Solving corresponding coordinates of each pixel point in the original image in the horizontal epipolar line coordinate system to form mapping of each pixel point position of the original image and the corresponding point position of the horizontal epipolar line image, and filling color values of each pixel point of the original image into the corresponding point position of the horizontal epipolar line image to generate a corrected horizontal epipolar line image, wherein the original image comprises a left inclined image under the left camera coordinate system and a right inclined image under the right camera coordinate system, the left inclined image is corrected to generate a left horizontal epipolar line image, and the right inclined image is corrected to generate a right horizontal epipolar line image.

Further, in step S4, the step of obtaining the initial dense homonym includes:

step S41, respectively calculating the initial parallaxes of the left horizontal epipolar line image and the right horizontal epipolar line image in the X axis and the Y axis of the horizontal epipolar line coordinate system;

step S42, determining a rough matching point of any dense point to be matched in the left horizontal epipolar line image in the right horizontal epipolar line image according to the initial parallax;

step S43, determining a one-dimensional search range in the right horizontal epipolar line image according to the position coordinate of the rough matching point as a search center and a set search radius, so as to determine a plurality of matching candidate points;

step S44, respectively calculating a normalized cross correlation coefficient between the gray mean value of the source image block with the dense points to be matched as the center and the gray mean value of the target image block with any one of the matching candidate points as the center;

and S45, determining the initial dense homonymous points of the dense points to be matched according to the normalized cross-correlation coefficient.

Further, in step S41, the step of calculating the initial parallax of the left horizontal epipolar line image and the right horizontal epipolar line image is:

step S411, respectively calculating X parallax and Y parallax of each sparse homonymous point in the left horizontal epipolar line image and the right horizontal epipolar line image under the horizontal epipolar line coordinate system;

step S412, selecting three adjacent sparse homonymous point triangulation networks to form a plurality of parallax calculation areas, and calculating the X parallax d of any point in the parallax calculation areas in the single parallax calculation area in the X direction and the Y direction by adopting linear interpolation _x And Y parallax d _y ；

Step S413, when the dense point to be matched is located in a single parallax calculation region, adopting an X parallax d corresponding to the single parallax calculation region _x And Y parallax d _y 。

Further, in step S45, determining whether the normalized cross correlation coefficient meets the standard to determine whether there is an initial dense homonymous point in a single dense point to be matched, setting an initial matching correlation coefficient standard t and an initial matching correlation coefficient difference standard e, where t ≧ 0.7,0 < e ≦ 0.01, recording the maximum value of the normalized cross correlation coefficient between the gray level mean of the source image block centered on the dense point to be matched and the gray level mean of the target image block centered on any one of the matching candidate points as Enccmax1, recording the second largest value as Enccmax2, setting Δ enccc = Enccmax1-Enccmax2,

when Enccmax1 is larger than or equal to t and delta Encc is larger than e, judging that the normalized cross-correlation coefficient accords with an initial matching standard, and setting a matching candidate point corresponding to Enccmax1 as an initial dense homonymous point of the point to be matched;

and when Enccmax1 is less than t or delta Encc is less than or equal to e, judging that the normalized cross-correlation coefficient does not accord with the initial matching standard, eliminating the points to be matched, which correspond to Enccmax1, and not having the initial dense homonymous points.

Further, in step S43, a search step is set to 2 pixels to determine the matching candidate points within the one-dimensional search range.

Further, in step S44, the gray level mean value is calculated in an incremental manner, where the incremental calculation formula is:

wherein, the target image block taking the mth matching candidate point as the center is recorded as the mth target image block,

is the mean value of the gray scale of the mth target image block>

Is the mean value of the gray scale of the (m + 1) th target image block>

Is the mean value of the gray levels of the pixels in the leftmost two columns in the mth target image block, and is greater than or equal to>

And the gray level average value of the pixels in the rightmost two columns in the (m + 1) th target image block is obtained.

Further, in step S5, performing position adjustment on each initial dense homonymous point by using least square image matching to obtain two times dense homonymous points, determining a position coordinate of a sub-pixel level dense homonymous point according to a maximum value Encc2 in a normalized cross-correlation coefficient between a gray-scale mean value of an image block corresponding to each two times dense homonymous point and a gray-scale mean value of a source image block with the corresponding dense homonymous point as a center, and setting a two times matching correlation coefficient standard tm, wherein tm is greater than or equal to 0.9,

when Encc2 is larger than or equal to tm, judging that the two-times normalized cross-correlation coefficient meets a sub-pixel level matching standard, and setting a two-times dense homonymy point corresponding to Encc2 as a sub-pixel level dense homonymy point of the dense point to be matched;

when Encc2 < tm, the two-times normalized cross-correlation coefficient is determined not to meet the sub-pixel level matching criterion.

Further, in step S5, after the sub-pixel level dense homonymy point is obtained, the sub-pixel level dense homonymy point and the rough matching point are subjected to actual position deviation checking calculation in the X direction in the horizontal epipolar line image to obtain an X direction deviation

And according to>

It is determined whether an adjustment is made to the search radius r,

when the temperature is higher than the set temperature

＞/>

Then, the search radius is judged to be smaller than the actual position deviation, r is adjusted, and the setting is carried out

；

When in use

≤/>

And then, judging that the search radius accords with the actual position deviation without adjusting r.

Further, in step S5, when all the sub-pixel level dense synonym points in any scanning line are obtained, calculating the actual position deviation of each sub-pixel level dense synonym point and the corresponding rough matching point in the X direction in the horizontal epipolar line image to obtain an X direction deviation mean value

And according to >>

Determining an adjustment to the search radius r,

when the temperature is higher than the set temperature

When the radius is less than or equal to r, the search radius is judged to be large, r is adjusted, and the judgment is set>

；

When the temperature is higher than the set temperature

If r is greater than r, the search radius is determined to be small and r is adjusted, set->

。

Furthermore, the source image block and the target image block are both rectangular and have the same size, the dense point to be matched is located in the center of the source image block, the matching candidate point is located in the center of the target image block, and the size of the image block is taken

Wherein the length unit is a pixel.

Compared with the prior art, the tilt correction dense matching method based on the remote sensing image has the advantages that the horizontal epipolar line image is generated by the sparse homonymy points, the initial parallax of the left and right horizontal epipolar line image pairs is estimated, accordingly, the parallax estimation precision is improved, the search range of the matching points is reduced, the gray level mean value is calculated in an incremental mode when the initial dense homonymy points are determined through one-dimensional search, and the calculation amount of gray level calculation is reduced; when the matching point is searched, the searching step length is set to be 2, so that the matching times are reduced, and the initial dense homonymous points are finely adjusted through least square image matching so as to achieve the purposes of improving the matching efficiency and reducing mismatching.

Furthermore, when sparse homonymy points are obtained, the accurate relative orientation elements of the stereo image can be obtained by adopting a relative orientation direct solution method and determining the relative orientation elements after adjustment optimization by a beam method, so that the accuracy of the relative pose relationship of two cameras forming the stereo model is determined, and the accuracy degree of the subsequent left and right horizontal epipolar line images is ensured.

Furthermore, the original images shot by the left camera and the right camera are respectively converted into the left horizontal epipolar line image and the right horizontal epipolar line image under the horizontal epipolar line coordinate system by constructing the horizontal epipolar line coordinate system, the parallax of the three-dimensional images shot by the left camera and the right camera is analyzed in the same coordinate system, the convenience degree and the accuracy of parallax analysis of the left image and the right image are effectively improved, the generated left horizontal epipolar line image and the right horizontal epipolar line image can be used as the basis for dense matching, and the efficiency of dense homonymy point matching is improved.

Further, the invention carries out dense homonymy point matching on the left horizontal epipolar line image and the right horizontal epipolar line image in the horizontal epipolar line coordinate system, because the parallax calculation in the X direction and the Y direction is carried out on the horizontal epipolar line coordinate system, when the dense homonymy point matching is carried out on a single point to be matched, the matching precision of the corresponding matched rough matching point reaches certain precision, and because the X axis of the constructed horizontal epipolar line coordinate system is constructed by the original points of the left camera coordinate system and the right camera coordinate system, the parallax in the Y axis direction of the horizontal epipolar line coordinate system has fixity theoretically.

Furthermore, the similarity degree of the two image blocks is measured by adopting the normalized cross-correlation coefficient, and the normalized cross-correlation coefficient has better robustness on linear brightness change between the image blocks, so that the linear brightness change existing in the image block characteristic represented by the gray mean value can be overcome, the characteristic correlation degree of matching correlation conforming to the matching point when the dense homonymous point is matched is further improved, and the degree of closeness of the dense homonymous point obtained by matching with the actual image by the method is improved.

Furthermore, the invention determines whether the single point to be matched has initial dense homonymy points by judging whether the normalized cross-correlation coefficient accords with the standard, and sets an initial matching correlation coefficient standard t and an initial matching correlation coefficient difference standard e, so as to calculate the normalized cross-correlation coefficient of the single point to be matched and a plurality of matching candidate points to obtain the initial dense homonymy points, ensures that the initial dense homonymy points have certain matching precision by judging the size relation of the normalized cross-correlation coefficient and the initial matching correlation coefficient standard t, ensures the correctness of the initial matching, avoids the increase of invalid calculation amount caused by carrying out least square matching on the points with poor matching precision, further improves the calculation efficiency of homonymy point matching, and simultaneously takes the initial matching correlation coefficient difference standard e as the matching significance standard of the initial dense homonymy points, further ensures that the judged initial dense homonymy points have better matching with the points to be matched compared with other matching candidate matching points, and improves the matching credibility.

Furthermore, the searching step length is set to be 2 pixels during one-dimensional searching, on one hand, the number of the matching candidate points is reduced to half of the number of the matching candidate points during pixel-by-pixel matching through expanding the step length, so that the matching calculation amount is reduced by half of the original calculation amount, and the searching calculation efficiency is improved; on the other hand, since the final dense homonymy point is obtained by performing least square matching after the initial dense homonymy point is obtained subsequently, the matching performance of pixels adjacent to the initial dense homonymy point as matching candidate points through the covering calculation of the least square matching is ensured by setting the search step length to be 2 pixels, so that the matching of the dense homonymy point by the method disclosed by the invention can cover each pixel in the search range, the optimal matching point of the matched dense homonymy point and the non-to-be-matched point caused by the sparse search point is avoided, and the balance of the search efficiency and the accuracy degree is further ensured.

Furthermore, the gray level mean value of the target image block is calculated in an incremental calculation mode, and the matching candidate points have certain density, so that the adjacent target image blocks are overlapped in a certain area, and when the gray level mean value of the previous target image block is obtained, the gray level mean value is quickly calculated by only calculating the gray level value of the non-overlapped area in the adjacent target image block, so that the problem of large calculation amount caused by repeatedly calculating the gray level value of the same pixel block is solved, and the calculation efficiency of the method is further improved.

Further, after the sub-pixel level dense homonymy points are obtained, the sub-pixel level dense homonymy points and the rough matching points are subjected to actual position deviation checking calculation in the X direction of the horizontal epipolar line image to obtain X direction deviation

And according to>

Determining whether to adjust the search radius r, and when all the sub-pixel level dense synonym points in any scanning line are obtained, calculating the actual position deviation of each sub-pixel level dense synonym point and the corresponding rough matching point in the X direction in the horizontal epipolar image to obtain an X direction deviation mean value ∑>

And according to >>

And determining an adjustment mode of the search radius r, and adjusting the search radius r to ensure that the subsequent one-dimensional search range is more accurate, further reducing invalid calculation amount and improving the calculation efficiency of the method.

Drawings

FIG. 1 is a diagram of the steps of the tilt correction dense matching method based on remote sensing images of the present invention;

FIG. 2 is a diagram of the steps of the present invention to obtain the initial dense homonyms;

FIG. 3 is a diagram illustrating an initial parallax calculation step according to the present invention;

FIG. 4 is a schematic diagram of a triangle difference according to an embodiment of the present invention.

Detailed Description

In order that the objects and advantages of the invention will be more clearly understood, the invention is further described below with reference to examples; it should be understood that the specific embodiments described herein are merely illustrative of the invention and do not delimit the invention.

Preferred embodiments of the present invention are described below with reference to the accompanying drawings. It should be understood by those skilled in the art that these embodiments are only for explaining the technical principle of the present invention, and do not limit the scope of the present invention.

It should be noted that in the description of the present invention, the terms of direction or positional relationship indicated by the terms "upper", "lower", "left", "right", "inner", "outer", etc. are based on the directions or positional relationships shown in the drawings, which are only for convenience of description, and do not indicate or imply that the device or element must have a specific orientation, be constructed in a specific orientation, and be operated, and thus, should not be construed as limiting the present invention.

Furthermore, it should be noted that, in the description of the present invention, unless otherwise explicitly specified or limited, the terms "mounted," "connected," and "connected" are to be construed broadly, and may be, for example, fixedly connected, detachably connected, or integrally connected; can be mechanically or electrically connected; they may be connected directly or indirectly through intervening media, or they may be interconnected between two elements. The specific meanings of the above terms in the present invention can be understood by those skilled in the art according to specific situations.

Referring to fig. 1, it is a step diagram of the tilt correction dense matching method based on remote sensing images of the present invention, and the present invention provides a tilt correction dense matching method based on remote sensing images, including:

s2, carrying out relative orientation on the stereopair based on the sparse homonymous points and the camera intrinsic parameters to obtain the relative pose relationship of the left camera and the right camera;

s3, constructing a horizontal epipolar line coordinate system based on the relative pose relationship, and correcting the left oblique image and the right oblique image into a left horizontal epipolar line image and a right horizontal epipolar line image respectively through the mapping relationship between the horizontal epipolar line coordinate system and the left camera coordinate system and the mapping relationship between the horizontal epipolar line coordinate system and the right camera coordinate system;

According to the tilt correction dense matching method based on the remote sensing image, the horizontal epipolar line image is generated by sparse homonymy points, the initial parallax of the left horizontal epipolar line image pair and the right horizontal epipolar line image pair is estimated, the parallax estimation precision is improved, the search range of the matching points is reduced, the gray average value is calculated by adopting increment when the initial dense homonymy points are determined through one-dimensional search, and the calculated amount of gray calculation is reduced; when the matching point is searched, the searching step length is set to be 2, so that the matching times are reduced, and the initial dense homonymy points are finely adjusted through least square image matching so as to achieve the purposes of improving the matching efficiency and reducing mismatching.

Referring to fig. 1, in step S2, the relative pose relationship is obtained by the sparse homonymy point and the camera intrinsic parameter by using a relative orientation direct solution and determined by adjustment optimization by a beam method, and the relative pose relationship is determinedRelationship adoption pose transformation matrix

Describe, is taken up or taken off>

Setting the position of any point P in the left camera coordinate system as ^>

Position of point of same name in the right camera coordinate system->

，

Wherein the content of the first and second substances,

Is a translation vector.

According to the method, when sparse homonymy points are obtained, the sparse homonymy points are obtained by a relative orientation direct solution method and are determined after adjustment optimization by a beam method, and accurate relative orientation elements of the stereo image can be obtained, so that the relative pose relationship of two cameras forming the stereo model is determined to have accuracy, and the accuracy degree of subsequent left and right horizontal epipolar line images is ensured.

Referring to fig. 1, in step S3, the horizontal epipolar coordinate system is a right-handed three-dimensional rectangular coordinate system, the origin of the horizontal epipolar coordinate system is set as the origin of the left-camera coordinate system, the X-axis of the horizontal epipolar coordinate system is the connection line between the left-camera projection center and the right-camera projection center,

the coordinate of the X-axis vector of the horizontal epipolar line coordinate system under the left camera coordinate systemComprises the following steps:

；/>

；

；

the rotation transformation matrix of the vector under the left camera coordinate system to the direction of the horizontal epipolar coordinate system is as follows:

the rotation transformation matrix of the vector under the right camera coordinate system to the direction of the horizontal epipolar coordinate system is as follows:

wherein the content of the first and second substances,

，/>

is a Z-axis vector of the left camera coordinate system, is greater than or equal to>

Is a Z-axis vector of the right camera coordinate system, is->

Is the rotation matrix->

The third column of elements.

According to the invention, the horizontal epipolar line coordinate system is constructed to respectively convert the original images shot by the left camera and the right camera into the left horizontal epipolar line image and the right horizontal epipolar line image under the horizontal epipolar line coordinate system, the parallax of the three-dimensional images shot by the left camera and the right camera is analyzed in the same coordinate system, the convenience and accuracy of parallax analysis of the left image and the right image are effectively improved, the generated left horizontal epipolar line image and the right horizontal epipolar line image can be used as the basis for carrying out dense matching, and the efficiency of dense homonymy point matching is improved.

Continuing to refer to FIG. 1, in step S3, the process proceeds by

And &>

Solving corresponding coordinates of each pixel point in the original image in the horizontal epipolar line coordinate system to form mapping of each pixel point position of the original image and the corresponding point position of the horizontal epipolar line image, and filling the color value of each pixel point of the original image in the corresponding point position of the horizontal epipolar line image to generate a corrected horizontal epipolar line image, wherein the original image comprises a left inclined image under the left camera coordinate system and a right inclined image under the right camera coordinate system, the left inclined image is corrected to generate a left horizontal epipolar line image, and the right inclined image is corrected to generate a right horizontal epipolar line image.

Please refer to fig. 2, which is a diagram illustrating the steps of obtaining the initial dense homonym according to the present invention, in step S4, the step of obtaining the initial dense homonym includes:

step S44, respectively calculating a normalized cross correlation coefficient between the gray average value of the source image block with the dense point to be matched as the center and the gray average value of the target image block with any one matching candidate point as the center;

Referring to fig. 3, which is a diagram illustrating an initial parallax calculation step according to the present invention, in step S41, the initial parallax calculation steps of the left horizontal epipolar line image and the right horizontal epipolar line image are as follows:

Step S413, when the dense point to be matched is located in a single parallax calculation area, adopting an X parallax d corresponding to the single parallax calculation area _x And Y parallax d _y 。

Specifically, the initial parallax of the left horizontal epipolar line image and the right horizontal epipolar line image is calculated, and any dense point to be matched in the left horizontal epipolar line image is determined according to the initial parallax

The rough matching point in the right horizontal epipolar line image is recorded as

Will >>

Set as the search center, r is the ^ h or greater of the search radius on the right horizontal epipolar line image>

Scanning line->

Performing one-dimensional search in the pixel range to determine a plurality of matching candidate points, and respectively calculating the dense points to be matched>

Normalized cross-correlation coefficient between the mean value of the gray level of the centered source image block and the mean value of the gray level of the target image block centered on each of the matching candidate points to determine initial dense synonym points, wherein>

Is->

Coordinates in a left horizontal epipolar line image>

Is->

Coordinates in the right horizontal epipolar image.

The invention carries out dense homonymy point matching on the left horizontal epipolar line image and the right horizontal epipolar line image in the horizontal epipolar line coordinate system, because the parallax calculation in the X direction and the Y direction is carried out on the horizontal epipolar line coordinate system, when the dense homonymy point matching is carried out on a single point to be matched, the matching precision of the corresponding matched rough matching point reaches certain precision, and because the X axis of the constructed horizontal epipolar line coordinate system is constructed by the original points of the left camera coordinate system and the right camera coordinate system, the parallax in the Y axis direction of the horizontal epipolar line coordinate system has fixity theoretically.

Continuing to refer to fig. 2, in step S45, determining whether the normalized cross-correlation coefficient meets the criterion to determine whether there is an initial dense homonymous point at a single point to be matched, setting an initial matching correlation coefficient criterion t and an initial matching correlation coefficient difference criterion e, where t ≧ 0.7,0 < e ≦ 0.01, recording the maximum value of the calculated normalized cross-correlation coefficients as Enccmax1, the second largest value as Enccmax2, setting Δ Encc = Enccmax1-Enccmax2,

The method measures the similarity degree of the two image blocks by adopting the normalized cross-correlation coefficient, and the normalized cross-correlation coefficient has better robustness to the linear brightness change between the image blocks, so that the linear brightness change existing in the image block characteristic represented by the gray mean value can be overcome, the matching correlation degree of the method for matching the dense homonymous points is further improved to accord with the characteristic correlation degree of the matching points, and the proximity degree of the dense homonymous points matched by the method of the invention and the homonymous points of the actual image is improved.

The method determines whether the single point to be matched has initial dense homonymy points or not by judging whether the normalized cross-correlation coefficient accords with the standard or not and setting an initial matching correlation coefficient standard t and an initial matching correlation coefficient difference standard e so as to calculate the normalized cross-correlation coefficient of the single point to be matched and a plurality of matching candidate points to obtain the initial dense homonymy points, ensures that the initial dense homonymy points have certain matching precision by judging the size relationship between the normalized cross-correlation coefficient and the initial matching correlation coefficient standard t so as to ensure the correctness of the initial matching, avoids the increase of invalid calculation amount caused by the least square matching of the points with poor matching precision, further improves the calculation efficiency of the homonymy point matching, and simultaneously takes the initial matching correlation coefficient difference standard e as the matching significance standard of the initial homonymy points, further ensures that the judged initial dense homonymy points have better matching with the points to be matched compared with other matching candidate matching points, and improves the matching credibility.

Referring to fig. 2, in step S43, the first horizontal epipolar line image

Of the scanning line

When one-dimensional search is carried out in the pixel range, the search step length is set to be 2 pixels,

the coordinates of the matching candidate points are obtained by calculation

Wherein i =0,1, … …, r.

According to the method, the search step length is set to be 2 pixels during one-dimensional search, on one hand, the number of the matching candidate points is reduced to half of the number of the matching candidate points during pixel-by-pixel matching by expanding the step length, so that the matching calculation amount is reduced by half of the original calculation amount, and the search calculation efficiency is improved; on the other hand, since the final dense homonymy point is obtained by performing least square matching after the initial dense homonymy point is obtained subsequently, the matching performance of pixels adjacent to the initial dense homonymy point as matching candidate points through the covering calculation of the least square matching is ensured by setting the search step length to be 2 pixels, so that the matching of the dense homonymy point by the method disclosed by the invention can cover each pixel in the search range, the optimal matching point of the matched dense homonymy point and the non-to-be-matched point caused by the sparse search point is avoided, and the balance of the search efficiency and the accuracy degree is further ensured.

As shown in fig. 1, in the step S4, the gray level mean value is calculated in an incremental manner, where the incremental formula is:

is the mean value of the gray scale of the mth target image block>

Is the mean value of the gray level of the (m + 1) th target image block>

Is the mean value of the gray scales of the pixels of the leftmost two columns in the mth target image block, and is combined with the gray scales of the pixels of the leftmost two columns in the mth target image block>

According to the invention, the gray level mean value of the target image block is calculated in an incremental calculation mode, and the matching candidate points have certain density, so that the adjacent target image blocks are overlapped in a certain area, and therefore, when the gray level mean value of the previous target image block is obtained, the gray level mean value is quickly calculated by only calculating the gray level value of the non-overlapped area in the adjacent target image block, so that the problem of large calculation amount caused by repeatedly calculating the gray level value of the same pixel block is reduced, and the calculation efficiency of the method is further improved.

As shown in fig. 1, in the step S5, performing position adjustment on each initial dense homonymous point by using least square image matching to obtain two times dense homonymous points, determining a position coordinate of a sub-pixel level dense homonymous point according to a maximum value Encc2 in a normalized cross-correlation coefficient between a gray level of an image block corresponding to each two times dense homonymous point and a gray level of a source image block centered on the corresponding dense point to be matched, and setting a two times matching correlation coefficient standard tm, where tm is greater than or equal to 0.9,

Specifically, in step S5, after the sub-pixel level dense homonymy point is obtained, the sub-pixel level dense homonymy point and the rough matching point are subjected to actual position deviation checking in the X direction in the horizontal epipolar line image to obtain an X direction deviation

And according to>

Determining whether to adjust the preset search radius r,

when the temperature is higher than the set temperature

＞/>

Then, the preset search half is judgedThe diameter is smaller than the actual position deviation, and r is adjusted and set

；

When the temperature is higher than the set temperature

≤/>

And then, judging that the preset search radius accords with the actual position deviation without adjusting r.

And according to>

It is determined whether an adjustment is made to the search radius r,

when in use

＞/>

；

When in use

≤/>

When the search radius is determinedAnd the actual position deviation is met, and r does not need to be adjusted.

Specifically, in step S5, when all the sub-pixel level dense synonym points in any scanning line are obtained, calculating the actual position deviation of each sub-pixel level dense synonym point and the corresponding rough matching point in the X direction in the horizontal epipolar line image to obtain an X direction deviation mean value

And according to >>

Determining an adjustment to the search radius r,

when in use

；

When in use

If r is greater than r, the search radius is determined to be small, r is adjusted, and the value is set>

。

Wherein the length unit is a pixel.

Specifically, when the first step is obtained

When all the sub-pixel level dense homonymous points in the scanning line are scanned, calculating actual position deviation of each sub-pixel level dense homonymous point and the corresponding rough matching point in the X direction of the horizontal epipolar line image to obtain deviation in the X direction, and averaging the deviation to obtain ^ er>

。

Example (b):

the embodiment provides a specific calculation process of a tilt correction dense matching method based on a remote sensing image, and for a specific stereoscopic image E, a left tilt image obtained by shooting the stereoscopic image E by a left camera and a right tilt image obtained by shooting the stereoscopic image E by a right camera are adopted for an original image, and the method comprises the following steps:

step one, feature matching;

firstly, extracting feature points of a left oblique image and a right oblique image by adopting an SIFT feature descriptor, and performing feature matching; and solving the affine transformation relation of the two images according to the feature matching points by adopting an RANSAC algorithm, eliminating a small number of mismatching points, finally obtaining a certain number of reliable homonymous image points distributed in the whole overlapping area, and forming a three-dimensional model by the left image and the right image.

Step two, relative orientation;

based on the homonymy points and camera intrinsic parameters of the left oblique image and the right oblique image, the precise relative orientation elements of the stereo image E can be obtained by a relative orientation direct solution method and by combining with bundle adjustment optimization, namely the relative pose relationship of the left camera and the right camera forming the stereo image E is determined, namely the rotation matrix relation of a right camera coordinate system relative to a left camera coordinate system

And the translation vector pick>

。

Step three, generating a horizontal epipolar line image of the stereoscopic image E, comprising:

step 31, constructing a horizontal epipolar coordinate system;

and taking the origin of the left camera coordinate system as the origin of the horizontal epipolar line coordinate system, and defining the horizontal epipolar line coordinate system by determining the coordinates of the vector corresponding to the triaxial of the horizontal epipolar line coordinate system X, Y, Z in the left camera coordinate system. Setting a connecting line vector of the projection centers of the left camera and the right camera as an X axis of a horizontal epipolar coordinate system, wherein the coordinate of the X axis vector of the horizontal epipolar coordinate system under a left camera coordinate system is a translation vector t obtained by relative orientation, and the coordinate of the X axis vector of the horizontal epipolar coordinate system under the left camera coordinate system is as follows:

；

；

；

to this end, the coordinates of the three axis vectors of the horizontal epipolar coordinate system in the left camera coordinate system have been determined, and the process described above constructs a right-hand three-dimensional rectangular coordinate system.

After the horizontal epipolar line coordinate system is constructed, virtual left and right horizontal epipolar line cameras are constructed simultaneously, wherein three axes X, Y, Z of the left and right horizontal epipolar line camera coordinate systems are parallel to three axes of the horizontal epipolar line coordinate system, and the original points of the left and right horizontal epipolar line camera coordinate systems are respectively set as the original points of the left and right oblique camera coordinate systems, namely the projection centers of the left and right oblique cameras.

After the horizontal epipolar coordinate system is constructed, the rotation transformation matrix from the vector under the left camera coordinate system to the horizontal epipolar coordinate system direction is as follows:

wherein the content of the first and second substances,

，/>

Is a Z-axis vector of a right camera coordinate system>

Is said rotation matrix>

The third column of elements.

Step 32, correcting the image;

taking the left oblique image corresponding to the left camera as an example, the process of correcting the oblique image into the horizontal image is as follows,

for any pixel point in left inclined image

Let its corresponding point in the left horizontal epipolar line image be

And is and/>

corresponding normalized homogeneous coordinate is->

The method comprises the following steps:

（1）

and &>

The conversion relationship between the two is as follows: />

（2）

Wherein the content of the first and second substances,

is the inverse of the orientation matrix in the camera, is>

Is the focal length of the left camera, and>

the coordinates of the image principal point of the left camera; />

Is an internal orientation matrix of a horizontal epipolar camera, based on a predetermined number of pixels>

Is the focal length of a horizontal epipolar camera, set->

To ensure horizontal epipolar line shadowLike having a similar resolution to the original image,

is the image principal point coordinates of the horizontal epipolar line camera.

Combining the formula (1) and the formula (2), any pixel in the left oblique image can be mapped to the left horizontal epipolar line image, and vice versa;

for the right oblique image corresponding to the right camera, the image obtained by the method in the formula (2)

Is replaced by>

The mapping relationship between the right oblique image and the right horizontal epipolar line image can be obtained.

In the specific implementation, firstly, the following steps are carried out

、/>

Setting the coordinate of four vertex pixels of the rectangle of the imaging range of the inclined image as 0, calculating the coordinate of the four vertex pixels of the rectangle of the imaging range of the inclined image in the horizontal epipolar line image by adopting the formula (1) and the formula (2), obtaining the minimum rectangle bounding box of the imaging range of the horizontal epipolar line image, and judging whether the minimum rectangle bounding box is greater than or equal to the minimum rectangle bounding box>

Set to the x coordinate of the lower left corner of the minimum rectangular bounding box, <' > based on>

Setting the minimum rectangular bounding box to have the y coordinate of the lower left corner (namely the minimum x and y coordinates of the bounding box) of the minimum rectangular bounding box, so that the lower left corner of the minimum rectangular bounding box is superposed with the lower left corner of the horizontal epipolar image, setting the width and height of the minimum rectangular bounding box to be the corresponding width and height of the horizontal epipolar image, scanning each pixel point of the horizontal epipolar image, solving the corresponding point coordinate of the pixel point on the original image, and further, setting the width and height of the minimum rectangular bounding box to be the corresponding width and height of the horizontal epipolar imageThe color value of each point on the original image obtained by the gray resampling is filled in the corresponding point of the horizontal epipolar line image, thereby finally generating the corrected horizontal image.

Fourthly, carrying out relevant matching on gray scales, including;

step 41, disparity estimation;

in order to estimate the initial parallax of a horizontal image pair, firstly adopting an equation (2) to convert the original image coordinates of all sparse homonymous points into horizontal epipolar line image coordinates; and then, interpolating the x-disparity and the y-disparity of each pixel of the horizontal image in the triangle according to the horizontal image disparity of three vertexes of each triangle in the x-direction and the y-direction by using a sparse point triangulation network.

Please refer to fig. 4, which is a schematic diagram of a triangle difference according to an embodiment of the present invention, wherein

、/>

、

Three vertexes of the triangle are corresponding to x parallaxes which are respectively d _x1 、d _x2 、d _x3 ，/>

Is a point inside the triangle and is,

、/>

is the y-th pixel row and two sides P of the triangle ₁ P ₂ 、P ₁ P ₃ The intersection point of (a). First, a point p is found by linear interpolation in the y direction ₁₂ 、p ₁₃ X parallax of d _x12 、d _x13 As shown in formula (3): />

(3)

By converting the parallax in the equation (3) into the x coordinate of the corresponding point, the point can be obtained

、/>

X coordinate x of ₁₂ 、x ₁₃ . Then taken by a point>

、/>

Linear interpolation in the x direction, i.e. a value->

X parallax d of points _x As shown in formula (4):

(4)

by the same method, can obtain

Y parallax d of points _y 。

It should be noted that the y-parallax at any position in the horizontal epipolar image pair is theoretically fixed, and the y-parallax is theoretically corrected by the principal point row coordinates

The method is used for determining the parallax error of the horizontal epipolar line image, and actually processing the parallax error and the distortion uncertainty of the camera distortion coefficient, so that the parallax error of the generated horizontal epipolar line image in the y direction is not fixed but fluctuates in a plurality of pixel ranges near a theoretical value, the y parallax is interpolated according to three vertexes of a triangle, the estimation precision of the parallax error is greatly improved, and further the parallax error is estimated in the three-dimensional imageThe matching accuracy is improved.

Step 24, correlation matching;

the similarity degree of the two image blocks is measured by adopting a normalized cross-correlation coefficient, the normalized cross-correlation coefficient is robust to linear brightness change between the image blocks, and the calculation formula is as follows:

（5）

wherein:

is the mean of the gray levels of the two related blocks, N is the number of pixels in the block;

it should be noted that, for a color image, the gray level mean value of each color channel should be calculated separately, and all the color channels should be connected in parallel to calculate the correlation coefficient, and the corresponding calculation formula is as follows:

（6）

wherein c is the number of color channels;

setting the matching interval of dense points to be matched as mxn, namely matching one dense homonymous point every m rows and n columns, wherein the smaller the interval is, the higher the density is, the higher the reconstruction fineness is, and the values of m and n can be set according to the three-dimensional scene complexity of an actual scene, and are not described again;

for any point to be matched on the left horizontal epipolar line image

Estimating the rough matching point in the right horizontal epipolar line image as ^ based on the parallax at the point>

And is based on->

Is taken as the center and is on the right horizontal image ^ h>

Scanning line->

Performing one-dimensional search matching in a pixel range, wherein r is a search radius;

it should be noted that although y parallax is not fixed, y parallax fluctuation is small in a small range of the search area, so that by fixing y parallax, the search range can be reduced, and the amount of calculation data can be reduced; because the estimation precision of the rough matching point is higher, r is not necessarily large, on one hand, the search range is reduced, the matching efficiency is improved, on the other hand, the probability of mismatching is reduced, and the matching accuracy is improved.

For any rough point to be matched, calculating a normalized correlation coefficient between a target image block taking the rough point to be matched as a center and a source image block taking a corresponding dense point to be matched as a center according to the formula (6); taking the rough point to be matched corresponding to the maximum value of the normalized correlation coefficient calculation value from the normalized correlation coefficient calculation values of all the rough points to be matched, and if the correlation coefficient of the point is greater than t and the difference between the maximum value of the normalized correlation coefficient calculation value and the second largest value is greater than e, considering the point as an initial dense homonymous point; otherwise, rejecting the point; t is more than or equal to 0.7 and is used for ensuring the correctness of initial matching, and e is more than 0 and less than or equal to 0.01 so as to ensure that the homonymy points have certain significance and improve the reliability.

Specifically, the target image block size is generally 11 pixels × 11 pixels.

Specifically, when searching for the initial dense homonym point, the following 2 strategies are adopted to improve the matching efficiency:

the method comprises the steps that firstly, when one-dimensional search is carried out in the search range of any scanning line of the right horizontal epipolar line image, the search step length is set to be 2 pixels, and the coordinates of the matching candidate points are obtained through calculation;

that is, the coordinates of the matching candidate points are not traversed to each pixel in the search range, but the correlation coefficient is obtained every other pixel, so that the search amount is halved, and the purpose of improving the matching efficiency is achieved.

The rationale for this is: because the searched initial dense homonymous points are subjected to fine adjustment by adopting least square matching, and the correction range of the least square matching can reach 2 pixels, even if the actual best matching point is in the interval pixels which are not scanned, the searched initial dense homonymous points are adjacent pixels of the actual best matching point, the actual best position can be corrected by the least square matching, and point-by-point matching is not needed.

And strategy two, when searching for initial dense homonymous points in the search range, incremental calculation is adopted when calculating the gray average value in the correlation coefficient:

calculating to obtain the source image mean value at the initial matching candidate point of the search range

And the mean value of the target image->

(ii) a Subsequently, in a search match performed successively along the search range, a value is selected which is greater than or equal to>

Remains unchanged, is paired with>

An update is made by the sum of the gray levels (N × ^ er) from the previous window>

) Subtracting the gray scale of the front 2 columns of the window and adding the gray scale of the rear 2 columns of the rear window to obtain the gray scale sum of the rear window so as to obtain the sum of the gray scales of the rear window and the greater or lesser of the gray scale sum>

；

The incremental calculation formula is:

is the mean value of the gray scale of the mth target image block>

Is the mean value of the gray scale of the (m + 1) th target image block>

And the gray level average value of the pixels in the rightmost two columns in the (m + 1) th target image block is obtained. />

The use of the incremental calculation process can reduce the amount of gradation calculation.

Strategy three, adding sequence consistency constraint to the matching points along the scanning line, namely adding two points on the left horizontal epipolar line image

、/>

The corresponding matching points on the right horizontal epipolar line image are ^ and ^ respectively>

、/>

Is set>

Is located->

To the right side of

Must be located->

To the right, such processing further enhances the reliability of the matching.

Step five, least square matching;

and C, performing least square image matching on the initial dense homonymous points acquired in the step four, so that the matching precision of the homonymous points reaches a sub-pixel level.

Because point-by-point searching and matching can only reach the matching precision of 1 pixel, the searched initial matching point is finely adjusted by adopting least square matching, so that the homonymy point reaches the matching precision of a sub-pixel level, and the visual effect of a reconstructed scene is improved.

So far, the technical solutions of the present invention have been described in connection with the preferred embodiments shown in the drawings, but it is easily understood by those skilled in the art that the scope of the present invention is obviously not limited to these specific embodiments. Equivalent changes or substitutions of related technical features can be made by those skilled in the art without departing from the principle of the invention, and the technical scheme after the changes or substitutions can be within the protection scope of the invention.

The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention; various modifications and alterations to this invention will become apparent to those skilled in the art. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A tilt correction dense matching method based on remote sensing images is characterized by comprising the following steps:

step S4, determining a plurality of dense points to be matched at set matching intervals in the left horizontal epipolar line image obtained in the step S3, and performing one-dimensional gray-scale correlation matching on any dense point to be matched in the right horizontal epipolar line image to obtain initial dense homonymy points;

s5, performing least square image matching on the initial dense homonymy points to obtain sub-pixel level dense homonymy points;

in step S4, the step of obtaining the initial dense homonym point includes:

s45, determining initial dense homonymous points of the dense points to be matched according to the normalized cross-correlation coefficient;

in the step S43, setting a search step size to 2 pixels to determine the matching candidate points within the one-dimensional search range;

in step S44, the gray level mean value is calculated in an incremental manner, where the incremental calculation formula is:

is the mean value of the gray scale of the mth target image block>

Is the mean value of the gray scale of the (m + 1) th target image block>

2. The tilt correction dense matching method based on remote sensing images as claimed in claim 1, wherein in step S2, the relative pose relationship is obtained by the sparse homonymy point and camera intrinsic parameters by using a relative orientation direct solution and determined after adjustment optimization by a beam method, and the relative pose relationship is determined by using a pose transformation matrix

Description of the invention，/>

Setting the position of any point P in the left camera coordinate system as ^>

Position of point of same name in the right camera coordinate system->

，/>

Wherein the content of the first and second substances,

is phase/based on the coordinate system of the right camera>

For a rotation matrix of the left camera coordinate system, ->

Is a translation vector.

3. The tilt-correction dense matching method based on remote-sensing images according to claim 2, wherein in step S3, the horizontal epipolar coordinate system is a right-hand three-dimensional rectangular coordinate system, the origin of the horizontal epipolar coordinate system is set as the origin of the left camera coordinate system, the X-axis of the horizontal epipolar coordinate system is the connecting line of the left camera projection center and the right camera projection center,

；

；

；

wherein the content of the first and second substances,

，/>

is a Z-axis vector of the left camera coordinate system>

Is a Z-axis vector of the right camera coordinate system, is->

Is the rotation matrix->

The third column of elements.

4. The method of claim 3, wherein the tilt correction is based on a dense matching of the remote sensing imageCharacterized in that, in the step S3, the step B is performed by

And &>

Solving the corresponding coordinates of each pixel point in the original image in the horizontal epipolar line coordinate system to form the mapping of the position of each pixel point in the original image and the position of the corresponding point of the horizontal epipolar line image, and filling the color value of each pixel point in the original image into the position of the corresponding point of the horizontal epipolar line image to generate a corrected horizontal epipolar line image, wherein,

the original image comprises a left inclined image under the left camera coordinate system and a right inclined image under the right camera coordinate system, the left inclined image generates a left horizontal epipolar line image after being corrected, and the right inclined image generates a right horizontal epipolar line image after being corrected.

5. The method for densely matching the tilt correction based on the remote sensing image according to claim 4, wherein in the step S41, the calculation of the initial parallax of the left horizontal epipolar line image and the right horizontal epipolar line image comprises:

6. The tilt-corrected dense matching method based on remote sensing images according to claim 5, wherein in step S45, an initial matching correlation coefficient standard t and an initial matching correlation coefficient difference standard e are set by determining whether the normalized cross-correlation coefficient meets the standard to determine whether an initial dense homonymous point exists at a single dense point to be matched, wherein t ≧ 0.7,0 < e ≦ 0.01, the maximum value of the normalized cross-correlation coefficients between the gray-scale mean of the source image block centered at the dense point to be matched and the gray-scale mean of the target image block centered at any one of the matching candidate points is designated as Enccmax1, the next largest value is designated as Enccmax2, and Δ Encc = Enccmax1-Enccmax2 are set,

7. The tilt-correction dense matching method based on remote sensing images according to claim 6, wherein in the step S5, the position of each initial dense homonymous point is adjusted by using least square image matching to obtain two times dense homonymous points, the position coordinates of the sub-pixel level dense homonymous points are determined according to a maximum value Encc2 in the normalized cross-correlation coefficient between the gray-scale mean value of each image block corresponding to the two times dense homonymous points and the gray-scale mean value of the source image block with the corresponding dense point to be matched as the center, and a two times matching correlation coefficient standard tm is set, wherein tm is greater than or equal to 0.9,