CN115719320B - Tilt correction dense matching method based on remote sensing image - Google Patents

Tilt correction dense matching method based on remote sensing image Download PDF

Info

Publication number
CN115719320B
CN115719320B CN202310029421.6A CN202310029421A CN115719320B CN 115719320 B CN115719320 B CN 115719320B CN 202310029421 A CN202310029421 A CN 202310029421A CN 115719320 B CN115719320 B CN 115719320B
Authority
CN
China
Prior art keywords
matching
dense
point
coordinate system
image
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202310029421.6A
Other languages
Chinese (zh)
Other versions
CN115719320A (en
Inventor
杨阿华
张强
常鑫
赵斐
高鹏
汪世辉
王雅楠
于潇
王栋
张大伟
闫孝鲁
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
63921 Troops of PLA
Original Assignee
63921 Troops of PLA
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 63921 Troops of PLA filed Critical 63921 Troops of PLA
Priority to CN202310029421.6A priority Critical patent/CN115719320B/en
Publication of CN115719320A publication Critical patent/CN115719320A/en
Application granted granted Critical
Publication of CN115719320B publication Critical patent/CN115719320B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • YGENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02TCLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION
    • Y02T10/00Road transport of goods or passengers
    • Y02T10/10Internal combustion engine [ICE] based vehicles
    • Y02T10/40Engine management systems

Landscapes

  • Image Processing (AREA)

Abstract

The invention relates to the technical field of image processing, in particular to a tilt correction dense matching method based on remote sensing images, which comprises the steps of firstly generating horizontal epipolar line images by sparse homonymy points, further estimating initial parallaxes of left and right horizontal epipolar line image pairs, thereby improving the parallax estimation precision, reducing the search range of matching points, reducing the error matching probability, and reducing the calculated amount of gray level calculation by adopting increment calculation gray level mean value when determining the initial dense homonymy points through one-dimensional search; when searching for the matching point, the searching step length is set to be 2, so that the matching times are reduced, the matching efficiency is improved, and the initial dense homonymous points are finely adjusted through least square image matching, so that the aim of improving the matching precision is fulfilled.

Description

Tilt correction dense matching method based on remote sensing image
Technical Field
The invention relates to the technical field of image processing, in particular to a tilt correction dense matching method based on remote sensing images.
Background
Currently, there are two main ways of reconstructing three-dimensional objects and scenes, namely, active and passive. The method has high reconstruction precision and high reconstruction efficiency on the three-dimensional structure of the scene, but because the method depends on expensive measuring equipment and has a complicated data acquisition process, the method needs to acquire image information while acquiring three-dimensional information when acquiring a virtual three-dimensional model with photo reality, and involves the problem of registration between point cloud and images when data is post-processed, and the acquired data amount and the workload of a calculation process are huge; the passive mode adopts an image sensor to obtain image information of the surface of an object, and recovers three-dimensional structure information of the surface of the object from a two-dimensional image with a certain parallax according to the principle of binocular stereo vision. Active three-dimensional scanning has become an important means for acquiring three-dimensional information in many application fields, but passive three-dimensional modeling based on images is still the most economical, flexible, feasible and widely used method and plays an important role in various fields.
The key point of reconstructing the three-dimensional object, the scene and the earth surface based on the image is dense matching of the homonymous points, and the accurate and dense homonymous points are the basis for acquiring high-precision three-dimensional information. Dense matching is widely applied to fine reconstruction of objects, reconstruction of realistic scenes and generation of digital surface models and real digital orthoimages, and if accurate, dense and efficient matching of homonymous points can be achieved, the problem of low reconstruction efficiency caused by complex and time-consuming precision and data processing in three-dimensional reconstruction based on images is solved.
In the prior art, image matching methods are divided into a feature-based method and an image correlation-based method, the feature-based method is high in precision and accuracy, but only aims at characteristic significant areas such as corners, lines, edges and the like in an image, so that only a sparse matching point set can be obtained, single-point calculation cost is much higher than that of the image correlation method, and the method is generally used for providing seed points or constraint conditions for dense matching; the image correlation-based method has large calculation amount and low efficiency, has self-similarity in a single texture region, and is easy to generate mismatching, so that the use of the method is limited, and therefore, the traditional image matching method is still a main implementation way of three-dimensional reconstruction based on images.
Chinese patent publication No. CN113034556a discloses a frequency domain related semi-dense remote sensing image matching method, aiming at obtaining matching information between two images; twisting one of the two images according to the initial matching information to obtain an image to be matched; dividing a reference image and an image to be matched into a plurality of areas; taking four small regions as a group of region image blocks, setting the centers of the region image blocks as seed points, and counting the gradient direction histograms of the region image blocks to obtain gradient amplitudes and angles corresponding to four angle ranges; setting four priority values of different levels; obtaining a new matching result combining the gradient information of the four adjacent domains of the seed points; adding the matching results of all the seed points into the sparse matching point set to obtain a semi-dense remote sensing image matching point set; and carrying out reference point mismatching correction on the dense matching points to obtain a matching result. According to the technical scheme, when the number of the image blocks in the initial matching area is large, the data processing and calculation workload is large, so that the three-dimensional reconstruction difficulty is increased, and when the number of the image blocks in the initial matching area is small, the matching precision is difficult to guarantee.
Disclosure of Invention
Therefore, the invention provides a tilt correction dense matching method based on a remote sensing image, which is used for solving the problem that the three-dimensional reconstruction precision is difficult to improve due to complex image matching data processing in the process of performing three-dimensional reconstruction by adopting image matching in the prior art.
In order to achieve the above object, the present invention provides a tilt correction dense matching method based on remote sensing images, comprising the following steps:
step S1, respectively extracting feature points from a left oblique image shot by a left camera and a right oblique image shot by a right camera by adopting an SIFT feature descriptor to perform feature matching to obtain sparse homonymy points;
s2, carrying out relative orientation on the stereopair based on the sparse homonymy points and the camera intrinsic parameters to obtain the relative pose relationship of the left camera and the right camera;
s3, constructing a horizontal epipolar line coordinate system based on the relative pose relationship, and correcting the left oblique image and the right oblique image into a left horizontal epipolar line image and a right horizontal epipolar line image respectively through the mapping relationship between the horizontal epipolar line coordinate system and the left camera coordinate system as well as the mapping relationship between the horizontal epipolar line coordinate system and the right camera coordinate system;
s4, determining a plurality of dense points to be matched at set matching intervals in the left horizontal epipolar line image acquired in the S3, and performing one-dimensional gray-scale correlation matching on any dense point to be matched in the right horizontal epipolar line image to acquire initial dense homonymous points;
and S5, performing least square image matching on the initial dense homonymous points to obtain sub-pixel-level dense homonymous points.
Further, in step S2, the relative pose relationship is obtained by the sparse homonymy point and the camera intrinsic parameter by a relative orientation direct solution and determined by bundle adjustment optimization, and the relative pose relationship is determined by a pose transformation matrix
Figure DEST_PATH_IMAGE001
Describe, is taken up or taken off>
Figure DEST_PATH_IMAGE002
Setting the position of any point P in the left camera coordinate system as ^>
Figure DEST_PATH_IMAGE003
Position of the same-name point in the right camera coordinate system >>
Figure DEST_PATH_IMAGE004
Figure DEST_PATH_IMAGE005
Wherein,
Figure DEST_PATH_IMAGE006
is a rotation matrix of the right camera coordinate system relative to the left camera coordinate system, is based on the value of the reference value>
Figure DEST_PATH_IMAGE007
Is a translation vector.
Further, in step S3, the horizontal epipolar line coordinate system is a right-handed three-dimensional rectangular coordinate system, the origin of the horizontal epipolar line coordinate system is set as the origin of the left camera coordinate system, the X-axis of the horizontal epipolar line coordinate system is a connection line between the left camera projection center and the right camera projection center,
the coordinates of the X-axis vector of the horizontal epipolar line coordinate system in the left camera coordinate system are as follows:
Figure DEST_PATH_IMAGE008
the coordinates of the Y-axis vector of the horizontal epipolar line coordinate system in the left camera coordinate system are as follows:
Figure DEST_PATH_IMAGE009
the coordinate of the Z-axis vector of the horizontal epipolar line coordinate system in the left camera coordinate system is as follows:
Figure DEST_PATH_IMAGE010
the rotation transformation matrix from the vector under the left camera coordinate system to the direction of the horizontal epipolar line coordinate system is as follows:
Figure DEST_PATH_IMAGE011
the rotation transformation matrix from the vector under the coordinate system of the right camera to the direction of the horizontal epipolar line coordinate system is as follows:
Figure DEST_PATH_IMAGE012
wherein,
Figure DEST_PATH_IMAGE013
is the average of the Z-axis direction vector of the left camera coordinate system and the Z-axis direction vector of the right camera coordinate system,
Figure DEST_PATH_IMAGE014
Figure DEST_PATH_IMAGE015
is the Z-axis vector of the left camera coordinate system,
Figure DEST_PATH_IMAGE016
is a Z-axis vector of the right camera coordinate system, is->
Figure DEST_PATH_IMAGE017
Is said rotation matrix>
Figure 421624DEST_PATH_IMAGE006
The third column of elements.
Further, in step S3, by
Figure DEST_PATH_IMAGE018
And &>
Figure DEST_PATH_IMAGE019
Solving corresponding coordinates of each pixel point in the original image in the horizontal epipolar line coordinate system to form mapping of each pixel point position of the original image and the corresponding point position of the horizontal epipolar line image, and filling color values of each pixel point of the original image into the corresponding point position of the horizontal epipolar line image to generate a corrected horizontal epipolar line image, wherein the original image comprises a left inclined image under the left camera coordinate system and a right inclined image under the right camera coordinate system, the left inclined image is corrected to generate a left horizontal epipolar line image, and the right inclined image is corrected to generate a right horizontal epipolar line image.
Further, in step S4, the step of obtaining the initial dense homonym includes:
step S41, respectively calculating the initial parallaxes of the left horizontal epipolar line image and the right horizontal epipolar line image in the X axis and the Y axis of the horizontal epipolar line coordinate system;
step S42, determining a rough matching point of any dense point to be matched in the left horizontal epipolar line image in the right horizontal epipolar line image according to the initial parallax;
step S43, determining a one-dimensional search range in the right horizontal epipolar line image according to the position coordinate of the rough matching point as a search center and a set search radius, so as to determine a plurality of matching candidate points;
step S44, respectively calculating a normalized cross correlation coefficient between the gray mean value of the source image block with the dense points to be matched as the center and the gray mean value of the target image block with any one of the matching candidate points as the center;
and S45, determining the initial dense homonymous points of the dense points to be matched according to the normalized cross-correlation coefficient.
Further, in step S41, the step of calculating the initial parallax of the left horizontal epipolar line image and the right horizontal epipolar line image is:
step S411, respectively calculating X parallax and Y parallax of each sparse homonymous point in the left horizontal epipolar line image and the right horizontal epipolar line image under the horizontal epipolar line coordinate system;
step S412, selecting three adjacent sparse homonymous point triangulation networks to form a plurality of parallax calculation areas, and calculating the X parallax d of any point in the parallax calculation areas in the single parallax calculation area in the X direction and the Y direction by adopting linear interpolation x And Y parallax d y
Step S413, when the dense point to be matched is located in a single parallax calculation region, adopting an X parallax d corresponding to the single parallax calculation region x And Y parallax d y
Further, in step S45, determining whether the normalized cross correlation coefficient meets the standard to determine whether there is an initial dense homonymous point in a single dense point to be matched, setting an initial matching correlation coefficient standard t and an initial matching correlation coefficient difference standard e, where t ≧ 0.7,0 < e ≦ 0.01, recording the maximum value of the normalized cross correlation coefficient between the gray level mean of the source image block centered on the dense point to be matched and the gray level mean of the target image block centered on any one of the matching candidate points as Enccmax1, recording the second largest value as Enccmax2, setting Δ enccc = Enccmax1-Enccmax2,
when Enccmax1 is larger than or equal to t and delta Encc is larger than e, judging that the normalized cross-correlation coefficient accords with an initial matching standard, and setting a matching candidate point corresponding to Enccmax1 as an initial dense homonymous point of the point to be matched;
and when Enccmax1 is less than t or delta Encc is less than or equal to e, judging that the normalized cross-correlation coefficient does not accord with the initial matching standard, eliminating the points to be matched, which correspond to Enccmax1, and not having the initial dense homonymous points.
Further, in step S43, a search step is set to 2 pixels to determine the matching candidate points within the one-dimensional search range.
Further, in step S44, the gray level mean value is calculated in an incremental manner, where the incremental calculation formula is:
Figure DEST_PATH_IMAGE020
wherein, the target image block taking the mth matching candidate point as the center is recorded as the mth target image block,
Figure DEST_PATH_IMAGE021
is the mean value of the gray scale of the mth target image block>
Figure DEST_PATH_IMAGE022
Is the mean value of the gray scale of the (m + 1) th target image block>
Figure DEST_PATH_IMAGE023
Is the mean value of the gray levels of the pixels in the leftmost two columns in the mth target image block, and is greater than or equal to>
Figure DEST_PATH_IMAGE024
And the gray level average value of the pixels in the rightmost two columns in the (m + 1) th target image block is obtained.
Further, in step S5, performing position adjustment on each initial dense homonymous point by using least square image matching to obtain two times dense homonymous points, determining a position coordinate of a sub-pixel level dense homonymous point according to a maximum value Encc2 in a normalized cross-correlation coefficient between a gray-scale mean value of an image block corresponding to each two times dense homonymous point and a gray-scale mean value of a source image block with the corresponding dense homonymous point as a center, and setting a two times matching correlation coefficient standard tm, wherein tm is greater than or equal to 0.9,
when Encc2 is larger than or equal to tm, judging that the two-times normalized cross-correlation coefficient meets a sub-pixel level matching standard, and setting a two-times dense homonymy point corresponding to Encc2 as a sub-pixel level dense homonymy point of the dense point to be matched;
when Encc2 < tm, the two-times normalized cross-correlation coefficient is determined not to meet the sub-pixel level matching criterion.
Further, in step S5, after the sub-pixel level dense homonymy point is obtained, the sub-pixel level dense homonymy point and the rough matching point are subjected to actual position deviation checking calculation in the X direction in the horizontal epipolar line image to obtain an X direction deviation
Figure DEST_PATH_IMAGE025
And according to>
Figure 507829DEST_PATH_IMAGE025
It is determined whether an adjustment is made to the search radius r,
when the temperature is higher than the set temperature
Figure 521921DEST_PATH_IMAGE025
Figure DEST_PATH_IMAGE026
Then, the search radius is judged to be smaller than the actual position deviation, r is adjusted, and the setting is carried out
Figure DEST_PATH_IMAGE027
When in use
Figure 289763DEST_PATH_IMAGE025
Figure 731109DEST_PATH_IMAGE026
And then, judging that the search radius accords with the actual position deviation without adjusting r.
Further, in step S5, when all the sub-pixel level dense synonym points in any scanning line are obtained, calculating the actual position deviation of each sub-pixel level dense synonym point and the corresponding rough matching point in the X direction in the horizontal epipolar line image to obtain an X direction deviation mean value
Figure DEST_PATH_IMAGE028
And according to >>
Figure 369856DEST_PATH_IMAGE028
Determining an adjustment to the search radius r,
when the temperature is higher than the set temperature
Figure 187639DEST_PATH_IMAGE028
When the radius is less than or equal to r, the search radius is judged to be large, r is adjusted, and the judgment is set>
Figure DEST_PATH_IMAGE029
When the temperature is higher than the set temperature
Figure 701666DEST_PATH_IMAGE028
If r is greater than r, the search radius is determined to be small and r is adjusted, set->
Figure DEST_PATH_IMAGE030
Furthermore, the source image block and the target image block are both rectangular and have the same size, the dense point to be matched is located in the center of the source image block, the matching candidate point is located in the center of the target image block, and the size of the image block is taken
Figure DEST_PATH_IMAGE031
Wherein the length unit is a pixel.
Compared with the prior art, the tilt correction dense matching method based on the remote sensing image has the advantages that the horizontal epipolar line image is generated by the sparse homonymy points, the initial parallax of the left and right horizontal epipolar line image pairs is estimated, accordingly, the parallax estimation precision is improved, the search range of the matching points is reduced, the gray level mean value is calculated in an incremental mode when the initial dense homonymy points are determined through one-dimensional search, and the calculation amount of gray level calculation is reduced; when the matching point is searched, the searching step length is set to be 2, so that the matching times are reduced, and the initial dense homonymous points are finely adjusted through least square image matching so as to achieve the purposes of improving the matching efficiency and reducing mismatching.
Furthermore, when sparse homonymy points are obtained, the accurate relative orientation elements of the stereo image can be obtained by adopting a relative orientation direct solution method and determining the relative orientation elements after adjustment optimization by a beam method, so that the accuracy of the relative pose relationship of two cameras forming the stereo model is determined, and the accuracy degree of the subsequent left and right horizontal epipolar line images is ensured.
Furthermore, the original images shot by the left camera and the right camera are respectively converted into the left horizontal epipolar line image and the right horizontal epipolar line image under the horizontal epipolar line coordinate system by constructing the horizontal epipolar line coordinate system, the parallax of the three-dimensional images shot by the left camera and the right camera is analyzed in the same coordinate system, the convenience degree and the accuracy of parallax analysis of the left image and the right image are effectively improved, the generated left horizontal epipolar line image and the right horizontal epipolar line image can be used as the basis for dense matching, and the efficiency of dense homonymy point matching is improved.
Further, the invention carries out dense homonymy point matching on the left horizontal epipolar line image and the right horizontal epipolar line image in the horizontal epipolar line coordinate system, because the parallax calculation in the X direction and the Y direction is carried out on the horizontal epipolar line coordinate system, when the dense homonymy point matching is carried out on a single point to be matched, the matching precision of the corresponding matched rough matching point reaches certain precision, and because the X axis of the constructed horizontal epipolar line coordinate system is constructed by the original points of the left camera coordinate system and the right camera coordinate system, the parallax in the Y axis direction of the horizontal epipolar line coordinate system has fixity theoretically.
Furthermore, the similarity degree of the two image blocks is measured by adopting the normalized cross-correlation coefficient, and the normalized cross-correlation coefficient has better robustness on linear brightness change between the image blocks, so that the linear brightness change existing in the image block characteristic represented by the gray mean value can be overcome, the characteristic correlation degree of matching correlation conforming to the matching point when the dense homonymous point is matched is further improved, and the degree of closeness of the dense homonymous point obtained by matching with the actual image by the method is improved.
Furthermore, the invention determines whether the single point to be matched has initial dense homonymy points by judging whether the normalized cross-correlation coefficient accords with the standard, and sets an initial matching correlation coefficient standard t and an initial matching correlation coefficient difference standard e, so as to calculate the normalized cross-correlation coefficient of the single point to be matched and a plurality of matching candidate points to obtain the initial dense homonymy points, ensures that the initial dense homonymy points have certain matching precision by judging the size relation of the normalized cross-correlation coefficient and the initial matching correlation coefficient standard t, ensures the correctness of the initial matching, avoids the increase of invalid calculation amount caused by carrying out least square matching on the points with poor matching precision, further improves the calculation efficiency of homonymy point matching, and simultaneously takes the initial matching correlation coefficient difference standard e as the matching significance standard of the initial dense homonymy points, further ensures that the judged initial dense homonymy points have better matching with the points to be matched compared with other matching candidate matching points, and improves the matching credibility.
Furthermore, the searching step length is set to be 2 pixels during one-dimensional searching, on one hand, the number of the matching candidate points is reduced to half of the number of the matching candidate points during pixel-by-pixel matching through expanding the step length, so that the matching calculation amount is reduced by half of the original calculation amount, and the searching calculation efficiency is improved; on the other hand, since the final dense homonymy point is obtained by performing least square matching after the initial dense homonymy point is obtained subsequently, the matching performance of pixels adjacent to the initial dense homonymy point as matching candidate points through the covering calculation of the least square matching is ensured by setting the search step length to be 2 pixels, so that the matching of the dense homonymy point by the method disclosed by the invention can cover each pixel in the search range, the optimal matching point of the matched dense homonymy point and the non-to-be-matched point caused by the sparse search point is avoided, and the balance of the search efficiency and the accuracy degree is further ensured.
Furthermore, the gray level mean value of the target image block is calculated in an incremental calculation mode, and the matching candidate points have certain density, so that the adjacent target image blocks are overlapped in a certain area, and when the gray level mean value of the previous target image block is obtained, the gray level mean value is quickly calculated by only calculating the gray level value of the non-overlapped area in the adjacent target image block, so that the problem of large calculation amount caused by repeatedly calculating the gray level value of the same pixel block is solved, and the calculation efficiency of the method is further improved.
Further, after the sub-pixel level dense homonymy points are obtained, the sub-pixel level dense homonymy points and the rough matching points are subjected to actual position deviation checking calculation in the X direction of the horizontal epipolar line image to obtain X direction deviation
Figure 281290DEST_PATH_IMAGE025
And according to>
Figure 287292DEST_PATH_IMAGE025
Determining whether to adjust the search radius r, and when all the sub-pixel level dense synonym points in any scanning line are obtained, calculating the actual position deviation of each sub-pixel level dense synonym point and the corresponding rough matching point in the X direction in the horizontal epipolar image to obtain an X direction deviation mean value ∑>
Figure 174345DEST_PATH_IMAGE028
And according to >>
Figure 949403DEST_PATH_IMAGE028
And determining an adjustment mode of the search radius r, and adjusting the search radius r to ensure that the subsequent one-dimensional search range is more accurate, further reducing invalid calculation amount and improving the calculation efficiency of the method.
Drawings
FIG. 1 is a diagram of the steps of the tilt correction dense matching method based on remote sensing images of the present invention;
FIG. 2 is a diagram of the steps of the present invention to obtain the initial dense homonyms;
FIG. 3 is a diagram illustrating an initial parallax calculation step according to the present invention;
FIG. 4 is a schematic diagram of a triangle difference according to an embodiment of the present invention.
Detailed Description
In order that the objects and advantages of the invention will be more clearly understood, the invention is further described below with reference to examples; it should be understood that the specific embodiments described herein are merely illustrative of the invention and do not delimit the invention.
Preferred embodiments of the present invention are described below with reference to the accompanying drawings. It should be understood by those skilled in the art that these embodiments are only for explaining the technical principle of the present invention, and do not limit the scope of the present invention.
It should be noted that in the description of the present invention, the terms of direction or positional relationship indicated by the terms "upper", "lower", "left", "right", "inner", "outer", etc. are based on the directions or positional relationships shown in the drawings, which are only for convenience of description, and do not indicate or imply that the device or element must have a specific orientation, be constructed in a specific orientation, and be operated, and thus, should not be construed as limiting the present invention.
Furthermore, it should be noted that, in the description of the present invention, unless otherwise explicitly specified or limited, the terms "mounted," "connected," and "connected" are to be construed broadly, and may be, for example, fixedly connected, detachably connected, or integrally connected; can be mechanically or electrically connected; they may be connected directly or indirectly through intervening media, or they may be interconnected between two elements. The specific meanings of the above terms in the present invention can be understood by those skilled in the art according to specific situations.
Referring to fig. 1, it is a step diagram of the tilt correction dense matching method based on remote sensing images of the present invention, and the present invention provides a tilt correction dense matching method based on remote sensing images, including:
step S1, respectively extracting feature points from a left oblique image shot by a left camera and a right oblique image shot by a right camera by adopting an SIFT feature descriptor to perform feature matching to obtain sparse homonymy points;
s2, carrying out relative orientation on the stereopair based on the sparse homonymous points and the camera intrinsic parameters to obtain the relative pose relationship of the left camera and the right camera;
s3, constructing a horizontal epipolar line coordinate system based on the relative pose relationship, and correcting the left oblique image and the right oblique image into a left horizontal epipolar line image and a right horizontal epipolar line image respectively through the mapping relationship between the horizontal epipolar line coordinate system and the left camera coordinate system and the mapping relationship between the horizontal epipolar line coordinate system and the right camera coordinate system;
s4, determining a plurality of dense points to be matched at set matching intervals in the left horizontal epipolar line image acquired in the S3, and performing one-dimensional gray-scale correlation matching on any dense point to be matched in the right horizontal epipolar line image to acquire initial dense homonymous points;
and S5, performing least square image matching on the initial dense homonymous points to obtain sub-pixel-level dense homonymous points.
According to the tilt correction dense matching method based on the remote sensing image, the horizontal epipolar line image is generated by sparse homonymy points, the initial parallax of the left horizontal epipolar line image pair and the right horizontal epipolar line image pair is estimated, the parallax estimation precision is improved, the search range of the matching points is reduced, the gray average value is calculated by adopting increment when the initial dense homonymy points are determined through one-dimensional search, and the calculated amount of gray calculation is reduced; when the matching point is searched, the searching step length is set to be 2, so that the matching times are reduced, and the initial dense homonymy points are finely adjusted through least square image matching so as to achieve the purposes of improving the matching efficiency and reducing mismatching.
Referring to fig. 1, in step S2, the relative pose relationship is obtained by the sparse homonymy point and the camera intrinsic parameter by using a relative orientation direct solution and determined by adjustment optimization by a beam method, and the relative pose relationship is determinedRelationship adoption pose transformation matrix
Figure 247398DEST_PATH_IMAGE001
Describe, is taken up or taken off>
Figure 6276DEST_PATH_IMAGE002
Setting the position of any point P in the left camera coordinate system as ^>
Figure 380293DEST_PATH_IMAGE003
Position of point of same name in the right camera coordinate system->
Figure 275437DEST_PATH_IMAGE004
Figure 229487DEST_PATH_IMAGE005
Wherein,
Figure 475660DEST_PATH_IMAGE006
is a rotation matrix of the right camera coordinate system relative to the left camera coordinate system, is based on the value of the reference value>
Figure 688205DEST_PATH_IMAGE007
Is a translation vector.
According to the method, when sparse homonymy points are obtained, the sparse homonymy points are obtained by a relative orientation direct solution method and are determined after adjustment optimization by a beam method, and accurate relative orientation elements of the stereo image can be obtained, so that the relative pose relationship of two cameras forming the stereo model is determined to have accuracy, and the accuracy degree of subsequent left and right horizontal epipolar line images is ensured.
Referring to fig. 1, in step S3, the horizontal epipolar coordinate system is a right-handed three-dimensional rectangular coordinate system, the origin of the horizontal epipolar coordinate system is set as the origin of the left-camera coordinate system, the X-axis of the horizontal epipolar coordinate system is the connection line between the left-camera projection center and the right-camera projection center,
the coordinate of the X-axis vector of the horizontal epipolar line coordinate system under the left camera coordinate systemComprises the following steps:
Figure 437855DEST_PATH_IMAGE008
;/>
the coordinates of the Y-axis vector of the horizontal epipolar line coordinate system in the left camera coordinate system are as follows:
Figure 828385DEST_PATH_IMAGE009
the coordinate of the Z-axis vector of the horizontal epipolar line coordinate system in the left camera coordinate system is as follows:
Figure 30696DEST_PATH_IMAGE010
the rotation transformation matrix of the vector under the left camera coordinate system to the direction of the horizontal epipolar coordinate system is as follows:
Figure 63243DEST_PATH_IMAGE011
the rotation transformation matrix of the vector under the right camera coordinate system to the direction of the horizontal epipolar coordinate system is as follows:
Figure 916667DEST_PATH_IMAGE012
wherein,
Figure 478099DEST_PATH_IMAGE013
is the average of the Z-axis direction vector of the left camera coordinate system and the Z-axis direction vector of the right camera coordinate system,
Figure 433285DEST_PATH_IMAGE014
Figure 738364DEST_PATH_IMAGE015
is a Z-axis vector of the left camera coordinate system, is greater than or equal to>
Figure 458014DEST_PATH_IMAGE016
Is a Z-axis vector of the right camera coordinate system, is->
Figure 924767DEST_PATH_IMAGE017
Is the rotation matrix->
Figure 101671DEST_PATH_IMAGE006
The third column of elements.
According to the invention, the horizontal epipolar line coordinate system is constructed to respectively convert the original images shot by the left camera and the right camera into the left horizontal epipolar line image and the right horizontal epipolar line image under the horizontal epipolar line coordinate system, the parallax of the three-dimensional images shot by the left camera and the right camera is analyzed in the same coordinate system, the convenience and accuracy of parallax analysis of the left image and the right image are effectively improved, the generated left horizontal epipolar line image and the right horizontal epipolar line image can be used as the basis for carrying out dense matching, and the efficiency of dense homonymy point matching is improved.
Continuing to refer to FIG. 1, in step S3, the process proceeds by
Figure 210441DEST_PATH_IMAGE018
And &>
Figure 38458DEST_PATH_IMAGE019
Solving corresponding coordinates of each pixel point in the original image in the horizontal epipolar line coordinate system to form mapping of each pixel point position of the original image and the corresponding point position of the horizontal epipolar line image, and filling the color value of each pixel point of the original image in the corresponding point position of the horizontal epipolar line image to generate a corrected horizontal epipolar line image, wherein the original image comprises a left inclined image under the left camera coordinate system and a right inclined image under the right camera coordinate system, the left inclined image is corrected to generate a left horizontal epipolar line image, and the right inclined image is corrected to generate a right horizontal epipolar line image.
Please refer to fig. 2, which is a diagram illustrating the steps of obtaining the initial dense homonym according to the present invention, in step S4, the step of obtaining the initial dense homonym includes:
step S41, respectively calculating the initial parallaxes of the left horizontal epipolar line image and the right horizontal epipolar line image in the X axis and the Y axis of the horizontal epipolar line coordinate system;
step S42, determining a rough matching point of any dense point to be matched in the left horizontal epipolar line image in the right horizontal epipolar line image according to the initial parallax;
step S43, determining a one-dimensional search range in the right horizontal epipolar line image according to the position coordinate of the rough matching point as a search center and a set search radius, so as to determine a plurality of matching candidate points;
step S44, respectively calculating a normalized cross correlation coefficient between the gray average value of the source image block with the dense point to be matched as the center and the gray average value of the target image block with any one matching candidate point as the center;
and S45, determining the initial dense homonymous points of the dense points to be matched according to the normalized cross-correlation coefficient.
Referring to fig. 3, which is a diagram illustrating an initial parallax calculation step according to the present invention, in step S41, the initial parallax calculation steps of the left horizontal epipolar line image and the right horizontal epipolar line image are as follows:
step S411, respectively calculating X parallax and Y parallax of each sparse homonymous point in the left horizontal epipolar line image and the right horizontal epipolar line image under the horizontal epipolar line coordinate system;
step S412, selecting three adjacent sparse homonymous point triangulation networks to form a plurality of parallax calculation areas, and calculating the X parallax d of any point in the parallax calculation areas in the single parallax calculation area in the X direction and the Y direction by adopting linear interpolation x And Y parallax d y
Step S413, when the dense point to be matched is located in a single parallax calculation area, adopting an X parallax d corresponding to the single parallax calculation area x And Y parallax d y
Specifically, the initial parallax of the left horizontal epipolar line image and the right horizontal epipolar line image is calculated, and any dense point to be matched in the left horizontal epipolar line image is determined according to the initial parallax
Figure DEST_PATH_IMAGE032
The rough matching point in the right horizontal epipolar line image is recorded as
Figure DEST_PATH_IMAGE033
Will >>
Figure 394221DEST_PATH_IMAGE033
Set as the search center, r is the ^ h or greater of the search radius on the right horizontal epipolar line image>
Figure DEST_PATH_IMAGE034
Scanning line->
Figure DEST_PATH_IMAGE035
Performing one-dimensional search in the pixel range to determine a plurality of matching candidate points, and respectively calculating the dense points to be matched>
Figure 760218DEST_PATH_IMAGE032
Normalized cross-correlation coefficient between the mean value of the gray level of the centered source image block and the mean value of the gray level of the target image block centered on each of the matching candidate points to determine initial dense synonym points, wherein>
Figure DEST_PATH_IMAGE036
Is->
Figure DEST_PATH_IMAGE037
Coordinates in a left horizontal epipolar line image>
Figure DEST_PATH_IMAGE038
Is->
Figure DEST_PATH_IMAGE039
Coordinates in the right horizontal epipolar image.
The invention carries out dense homonymy point matching on the left horizontal epipolar line image and the right horizontal epipolar line image in the horizontal epipolar line coordinate system, because the parallax calculation in the X direction and the Y direction is carried out on the horizontal epipolar line coordinate system, when the dense homonymy point matching is carried out on a single point to be matched, the matching precision of the corresponding matched rough matching point reaches certain precision, and because the X axis of the constructed horizontal epipolar line coordinate system is constructed by the original points of the left camera coordinate system and the right camera coordinate system, the parallax in the Y axis direction of the horizontal epipolar line coordinate system has fixity theoretically.
Continuing to refer to fig. 2, in step S45, determining whether the normalized cross-correlation coefficient meets the criterion to determine whether there is an initial dense homonymous point at a single point to be matched, setting an initial matching correlation coefficient criterion t and an initial matching correlation coefficient difference criterion e, where t ≧ 0.7,0 < e ≦ 0.01, recording the maximum value of the calculated normalized cross-correlation coefficients as Enccmax1, the second largest value as Enccmax2, setting Δ Encc = Enccmax1-Enccmax2,
when Enccmax1 is larger than or equal to t and delta Encc is larger than e, judging that the normalized cross-correlation coefficient accords with an initial matching standard, and setting a matching candidate point corresponding to Enccmax1 as an initial dense homonymous point of the point to be matched;
and when Enccmax1 is less than t or delta Encc is less than or equal to e, judging that the normalized cross-correlation coefficient does not accord with the initial matching standard, eliminating the points to be matched, which correspond to Enccmax1, and not having the initial dense homonymous points.
The method measures the similarity degree of the two image blocks by adopting the normalized cross-correlation coefficient, and the normalized cross-correlation coefficient has better robustness to the linear brightness change between the image blocks, so that the linear brightness change existing in the image block characteristic represented by the gray mean value can be overcome, the matching correlation degree of the method for matching the dense homonymous points is further improved to accord with the characteristic correlation degree of the matching points, and the proximity degree of the dense homonymous points matched by the method of the invention and the homonymous points of the actual image is improved.
The method determines whether the single point to be matched has initial dense homonymy points or not by judging whether the normalized cross-correlation coefficient accords with the standard or not and setting an initial matching correlation coefficient standard t and an initial matching correlation coefficient difference standard e so as to calculate the normalized cross-correlation coefficient of the single point to be matched and a plurality of matching candidate points to obtain the initial dense homonymy points, ensures that the initial dense homonymy points have certain matching precision by judging the size relationship between the normalized cross-correlation coefficient and the initial matching correlation coefficient standard t so as to ensure the correctness of the initial matching, avoids the increase of invalid calculation amount caused by the least square matching of the points with poor matching precision, further improves the calculation efficiency of the homonymy point matching, and simultaneously takes the initial matching correlation coefficient difference standard e as the matching significance standard of the initial homonymy points, further ensures that the judged initial dense homonymy points have better matching with the points to be matched compared with other matching candidate matching points, and improves the matching credibility.
Referring to fig. 2, in step S43, the first horizontal epipolar line image
Figure 41988DEST_PATH_IMAGE034
Of the scanning line
Figure 475244DEST_PATH_IMAGE035
When one-dimensional search is carried out in the pixel range, the search step length is set to be 2 pixels,
the coordinates of the matching candidate points are obtained by calculation
Figure DEST_PATH_IMAGE040
Wherein i =0,1, … …, r.
According to the method, the search step length is set to be 2 pixels during one-dimensional search, on one hand, the number of the matching candidate points is reduced to half of the number of the matching candidate points during pixel-by-pixel matching by expanding the step length, so that the matching calculation amount is reduced by half of the original calculation amount, and the search calculation efficiency is improved; on the other hand, since the final dense homonymy point is obtained by performing least square matching after the initial dense homonymy point is obtained subsequently, the matching performance of pixels adjacent to the initial dense homonymy point as matching candidate points through the covering calculation of the least square matching is ensured by setting the search step length to be 2 pixels, so that the matching of the dense homonymy point by the method disclosed by the invention can cover each pixel in the search range, the optimal matching point of the matched dense homonymy point and the non-to-be-matched point caused by the sparse search point is avoided, and the balance of the search efficiency and the accuracy degree is further ensured.
As shown in fig. 1, in the step S4, the gray level mean value is calculated in an incremental manner, where the incremental formula is:
Figure 657701DEST_PATH_IMAGE020
wherein, the target image block taking the mth matching candidate point as the center is recorded as the mth target image block,
Figure 543617DEST_PATH_IMAGE021
is the mean value of the gray scale of the mth target image block>
Figure 728611DEST_PATH_IMAGE022
Is the mean value of the gray level of the (m + 1) th target image block>
Figure 265640DEST_PATH_IMAGE023
Is the mean value of the gray scales of the pixels of the leftmost two columns in the mth target image block, and is combined with the gray scales of the pixels of the leftmost two columns in the mth target image block>
Figure 776256DEST_PATH_IMAGE024
And the gray level average value of the pixels in the rightmost two columns in the (m + 1) th target image block is obtained.
According to the invention, the gray level mean value of the target image block is calculated in an incremental calculation mode, and the matching candidate points have certain density, so that the adjacent target image blocks are overlapped in a certain area, and therefore, when the gray level mean value of the previous target image block is obtained, the gray level mean value is quickly calculated by only calculating the gray level value of the non-overlapped area in the adjacent target image block, so that the problem of large calculation amount caused by repeatedly calculating the gray level value of the same pixel block is reduced, and the calculation efficiency of the method is further improved.
As shown in fig. 1, in the step S5, performing position adjustment on each initial dense homonymous point by using least square image matching to obtain two times dense homonymous points, determining a position coordinate of a sub-pixel level dense homonymous point according to a maximum value Encc2 in a normalized cross-correlation coefficient between a gray level of an image block corresponding to each two times dense homonymous point and a gray level of a source image block centered on the corresponding dense point to be matched, and setting a two times matching correlation coefficient standard tm, where tm is greater than or equal to 0.9,
when Encc2 is larger than or equal to tm, judging that the two-times normalized cross-correlation coefficient meets a sub-pixel level matching standard, and setting a two-times dense homonymy point corresponding to Encc2 as a sub-pixel level dense homonymy point of the dense point to be matched;
when Encc2 < tm, the two-times normalized cross-correlation coefficient is determined not to meet the sub-pixel level matching criterion.
Specifically, in step S5, after the sub-pixel level dense homonymy point is obtained, the sub-pixel level dense homonymy point and the rough matching point are subjected to actual position deviation checking in the X direction in the horizontal epipolar line image to obtain an X direction deviation
Figure 415048DEST_PATH_IMAGE025
And according to>
Figure 403733DEST_PATH_IMAGE025
Determining whether to adjust the preset search radius r,
when the temperature is higher than the set temperature
Figure 546001DEST_PATH_IMAGE025
Figure 488504DEST_PATH_IMAGE026
Then, the preset search half is judgedThe diameter is smaller than the actual position deviation, and r is adjusted and set
Figure 614592DEST_PATH_IMAGE027
When the temperature is higher than the set temperature
Figure 406968DEST_PATH_IMAGE025
Figure 403742DEST_PATH_IMAGE026
And then, judging that the preset search radius accords with the actual position deviation without adjusting r.
Specifically, in step S5, after the sub-pixel level dense homonymy point is obtained, the sub-pixel level dense homonymy point and the rough matching point are subjected to actual position deviation checking in the X direction in the horizontal epipolar line image to obtain an X direction deviation
Figure 521740DEST_PATH_IMAGE025
And according to>
Figure 118812DEST_PATH_IMAGE025
It is determined whether an adjustment is made to the search radius r,
when in use
Figure 714879DEST_PATH_IMAGE025
Figure 831739DEST_PATH_IMAGE026
Then, the search radius is judged to be smaller than the actual position deviation, r is adjusted, and the setting is carried out
Figure 855059DEST_PATH_IMAGE027
When in use
Figure 690160DEST_PATH_IMAGE025
Figure 73606DEST_PATH_IMAGE026
When the search radius is determinedAnd the actual position deviation is met, and r does not need to be adjusted.
Specifically, in step S5, when all the sub-pixel level dense synonym points in any scanning line are obtained, calculating the actual position deviation of each sub-pixel level dense synonym point and the corresponding rough matching point in the X direction in the horizontal epipolar line image to obtain an X direction deviation mean value
Figure 310552DEST_PATH_IMAGE028
And according to >>
Figure 239194DEST_PATH_IMAGE028
Determining an adjustment to the search radius r,
when in use
Figure 573309DEST_PATH_IMAGE028
When the radius is less than or equal to r, the search radius is judged to be large, r is adjusted, and the judgment is set>
Figure 494867DEST_PATH_IMAGE029
When in use
Figure 586319DEST_PATH_IMAGE028
If r is greater than r, the search radius is determined to be small, r is adjusted, and the value is set>
Figure 685862DEST_PATH_IMAGE030
Furthermore, the source image block and the target image block are both rectangular and have the same size, the dense point to be matched is located in the center of the source image block, the matching candidate point is located in the center of the target image block, and the size of the image block is taken
Figure 761135DEST_PATH_IMAGE031
Wherein the length unit is a pixel.
Specifically, when the first step is obtained
Figure 486383DEST_PATH_IMAGE034
When all the sub-pixel level dense homonymous points in the scanning line are scanned, calculating actual position deviation of each sub-pixel level dense homonymous point and the corresponding rough matching point in the X direction of the horizontal epipolar line image to obtain deviation in the X direction, and averaging the deviation to obtain ^ er>
Figure 432342DEST_PATH_IMAGE028
Example (b):
the embodiment provides a specific calculation process of a tilt correction dense matching method based on a remote sensing image, and for a specific stereoscopic image E, a left tilt image obtained by shooting the stereoscopic image E by a left camera and a right tilt image obtained by shooting the stereoscopic image E by a right camera are adopted for an original image, and the method comprises the following steps:
step one, feature matching;
firstly, extracting feature points of a left oblique image and a right oblique image by adopting an SIFT feature descriptor, and performing feature matching; and solving the affine transformation relation of the two images according to the feature matching points by adopting an RANSAC algorithm, eliminating a small number of mismatching points, finally obtaining a certain number of reliable homonymous image points distributed in the whole overlapping area, and forming a three-dimensional model by the left image and the right image.
Step two, relative orientation;
based on the homonymy points and camera intrinsic parameters of the left oblique image and the right oblique image, the precise relative orientation elements of the stereo image E can be obtained by a relative orientation direct solution method and by combining with bundle adjustment optimization, namely the relative pose relationship of the left camera and the right camera forming the stereo image E is determined, namely the rotation matrix relation of a right camera coordinate system relative to a left camera coordinate system
Figure 968366DEST_PATH_IMAGE006
And the translation vector pick>
Figure 999776DEST_PATH_IMAGE007
Step three, generating a horizontal epipolar line image of the stereoscopic image E, comprising:
step 31, constructing a horizontal epipolar coordinate system;
and taking the origin of the left camera coordinate system as the origin of the horizontal epipolar line coordinate system, and defining the horizontal epipolar line coordinate system by determining the coordinates of the vector corresponding to the triaxial of the horizontal epipolar line coordinate system X, Y, Z in the left camera coordinate system. Setting a connecting line vector of the projection centers of the left camera and the right camera as an X axis of a horizontal epipolar coordinate system, wherein the coordinate of the X axis vector of the horizontal epipolar coordinate system under a left camera coordinate system is a translation vector t obtained by relative orientation, and the coordinate of the X axis vector of the horizontal epipolar coordinate system under the left camera coordinate system is as follows:
Figure 263136DEST_PATH_IMAGE008
the coordinates of the Y-axis vector of the horizontal epipolar line coordinate system in the left camera coordinate system are as follows:
Figure 594760DEST_PATH_IMAGE009
the coordinate of the Z-axis vector of the horizontal epipolar line coordinate system in the left camera coordinate system is as follows:
Figure 36106DEST_PATH_IMAGE010
to this end, the coordinates of the three axis vectors of the horizontal epipolar coordinate system in the left camera coordinate system have been determined, and the process described above constructs a right-hand three-dimensional rectangular coordinate system.
After the horizontal epipolar line coordinate system is constructed, virtual left and right horizontal epipolar line cameras are constructed simultaneously, wherein three axes X, Y, Z of the left and right horizontal epipolar line camera coordinate systems are parallel to three axes of the horizontal epipolar line coordinate system, and the original points of the left and right horizontal epipolar line camera coordinate systems are respectively set as the original points of the left and right oblique camera coordinate systems, namely the projection centers of the left and right oblique cameras.
After the horizontal epipolar coordinate system is constructed, the rotation transformation matrix from the vector under the left camera coordinate system to the horizontal epipolar coordinate system direction is as follows:
Figure 554812DEST_PATH_IMAGE011
the rotation transformation matrix of the vector under the right camera coordinate system to the direction of the horizontal epipolar coordinate system is as follows:
Figure 384314DEST_PATH_IMAGE012
wherein,
Figure 819712DEST_PATH_IMAGE013
is the average of the Z-axis direction vector of the left camera coordinate system and the Z-axis direction vector of the right camera coordinate system,
Figure 697538DEST_PATH_IMAGE014
Figure 703540DEST_PATH_IMAGE015
is a Z-axis vector of the left camera coordinate system, is greater than or equal to>
Figure 325014DEST_PATH_IMAGE016
Is a Z-axis vector of a right camera coordinate system>
Figure 100072DEST_PATH_IMAGE017
Is said rotation matrix>
Figure 398067DEST_PATH_IMAGE006
The third column of elements.
Step 32, correcting the image;
taking the left oblique image corresponding to the left camera as an example, the process of correcting the oblique image into the horizontal image is as follows,
for any pixel point in left inclined image
Figure DEST_PATH_IMAGE041
Let its corresponding point in the left horizontal epipolar line image be
Figure DEST_PATH_IMAGE042
And is and
Figure 343895DEST_PATH_IMAGE042
corresponding normalized homogeneous coordinate is->
Figure DEST_PATH_IMAGE043
The method comprises the following steps:
Figure DEST_PATH_IMAGE044
(1)
Figure DEST_PATH_IMAGE045
and &>
Figure DEST_PATH_IMAGE046
The conversion relationship between the two is as follows: />
Figure DEST_PATH_IMAGE047
(2)
Wherein,
Figure DEST_PATH_IMAGE048
is the inverse of the orientation matrix in the camera, is>
Figure DEST_PATH_IMAGE049
Is the focal length of the left camera, and>
Figure DEST_PATH_IMAGE050
the coordinates of the image principal point of the left camera;
Figure DEST_PATH_IMAGE051
Is an internal orientation matrix of a horizontal epipolar camera, based on a predetermined number of pixels>
Figure DEST_PATH_IMAGE052
Is the focal length of a horizontal epipolar camera, set->
Figure DEST_PATH_IMAGE053
To ensure horizontal epipolar line shadowLike having a similar resolution to the original image,
Figure DEST_PATH_IMAGE054
is the image principal point coordinates of the horizontal epipolar line camera.
Combining the formula (1) and the formula (2), any pixel in the left oblique image can be mapped to the left horizontal epipolar line image, and vice versa;
for the right oblique image corresponding to the right camera, the image obtained by the method in the formula (2)
Figure 181445DEST_PATH_IMAGE018
Is replaced by>
Figure 76588DEST_PATH_IMAGE019
The mapping relationship between the right oblique image and the right horizontal epipolar line image can be obtained.
In the specific implementation, firstly, the following steps are carried out
Figure DEST_PATH_IMAGE055
Figure DEST_PATH_IMAGE056
Setting the coordinate of four vertex pixels of the rectangle of the imaging range of the inclined image as 0, calculating the coordinate of the four vertex pixels of the rectangle of the imaging range of the inclined image in the horizontal epipolar line image by adopting the formula (1) and the formula (2), obtaining the minimum rectangle bounding box of the imaging range of the horizontal epipolar line image, and judging whether the minimum rectangle bounding box is greater than or equal to the minimum rectangle bounding box>
Figure DEST_PATH_IMAGE057
Set to the x coordinate of the lower left corner of the minimum rectangular bounding box, <' > based on>
Figure DEST_PATH_IMAGE058
Setting the minimum rectangular bounding box to have the y coordinate of the lower left corner (namely the minimum x and y coordinates of the bounding box) of the minimum rectangular bounding box, so that the lower left corner of the minimum rectangular bounding box is superposed with the lower left corner of the horizontal epipolar image, setting the width and height of the minimum rectangular bounding box to be the corresponding width and height of the horizontal epipolar image, scanning each pixel point of the horizontal epipolar image, solving the corresponding point coordinate of the pixel point on the original image, and further, setting the width and height of the minimum rectangular bounding box to be the corresponding width and height of the horizontal epipolar imageThe color value of each point on the original image obtained by the gray resampling is filled in the corresponding point of the horizontal epipolar line image, thereby finally generating the corrected horizontal image.
Fourthly, carrying out relevant matching on gray scales, including;
step 41, disparity estimation;
in order to estimate the initial parallax of a horizontal image pair, firstly adopting an equation (2) to convert the original image coordinates of all sparse homonymous points into horizontal epipolar line image coordinates; and then, interpolating the x-disparity and the y-disparity of each pixel of the horizontal image in the triangle according to the horizontal image disparity of three vertexes of each triangle in the x-direction and the y-direction by using a sparse point triangulation network.
Please refer to fig. 4, which is a schematic diagram of a triangle difference according to an embodiment of the present invention, wherein
Figure DEST_PATH_IMAGE059
Figure DEST_PATH_IMAGE060
Figure DEST_PATH_IMAGE061
Three vertexes of the triangle are corresponding to x parallaxes which are respectively d x1 、d x2 、d x3
Figure DEST_PATH_IMAGE062
Is a point inside the triangle and is,
Figure DEST_PATH_IMAGE063
Figure DEST_PATH_IMAGE064
is the y-th pixel row and two sides P of the triangle 1 P 2 、P 1 P 3 The intersection point of (a). First, a point p is found by linear interpolation in the y direction 12 、p 13 X parallax of d x12 、d x13 As shown in formula (3): />
Figure DEST_PATH_IMAGE065
(3)
By converting the parallax in the equation (3) into the x coordinate of the corresponding point, the point can be obtained
Figure DEST_PATH_IMAGE066
Figure DEST_PATH_IMAGE067
X coordinate x of 12 、x 13 . Then taken by a point>
Figure 567656DEST_PATH_IMAGE066
Figure 797518DEST_PATH_IMAGE067
Linear interpolation in the x direction, i.e. a value->
Figure DEST_PATH_IMAGE068
X parallax d of points x As shown in formula (4):
Figure DEST_PATH_IMAGE069
(4)
by the same method, can obtain
Figure 693885DEST_PATH_IMAGE068
Y parallax d of points y
It should be noted that the y-parallax at any position in the horizontal epipolar image pair is theoretically fixed, and the y-parallax is theoretically corrected by the principal point row coordinates
Figure 692802DEST_PATH_IMAGE056
The method is used for determining the parallax error of the horizontal epipolar line image, and actually processing the parallax error and the distortion uncertainty of the camera distortion coefficient, so that the parallax error of the generated horizontal epipolar line image in the y direction is not fixed but fluctuates in a plurality of pixel ranges near a theoretical value, the y parallax is interpolated according to three vertexes of a triangle, the estimation precision of the parallax error is greatly improved, and further the parallax error is estimated in the three-dimensional imageThe matching accuracy is improved.
Step 24, correlation matching;
the similarity degree of the two image blocks is measured by adopting a normalized cross-correlation coefficient, the normalized cross-correlation coefficient is robust to linear brightness change between the image blocks, and the calculation formula is as follows:
Figure DEST_PATH_IMAGE070
(5)
wherein:
Figure DEST_PATH_IMAGE071
is the mean of the gray levels of the two related blocks, N is the number of pixels in the block;
it should be noted that, for a color image, the gray level mean value of each color channel should be calculated separately, and all the color channels should be connected in parallel to calculate the correlation coefficient, and the corresponding calculation formula is as follows:
Figure DEST_PATH_IMAGE072
(6)
wherein c is the number of color channels;
setting the matching interval of dense points to be matched as mxn, namely matching one dense homonymous point every m rows and n columns, wherein the smaller the interval is, the higher the density is, the higher the reconstruction fineness is, and the values of m and n can be set according to the three-dimensional scene complexity of an actual scene, and are not described again;
for any point to be matched on the left horizontal epipolar line image
Figure DEST_PATH_IMAGE073
Estimating the rough matching point in the right horizontal epipolar line image as ^ based on the parallax at the point>
Figure DEST_PATH_IMAGE074
And is based on->
Figure DEST_PATH_IMAGE075
Is taken as the center and is on the right horizontal image ^ h>
Figure DEST_PATH_IMAGE076
Scanning line->
Figure DEST_PATH_IMAGE077
Performing one-dimensional search matching in a pixel range, wherein r is a search radius;
it should be noted that although y parallax is not fixed, y parallax fluctuation is small in a small range of the search area, so that by fixing y parallax, the search range can be reduced, and the amount of calculation data can be reduced; because the estimation precision of the rough matching point is higher, r is not necessarily large, on one hand, the search range is reduced, the matching efficiency is improved, on the other hand, the probability of mismatching is reduced, and the matching accuracy is improved.
For any rough point to be matched, calculating a normalized correlation coefficient between a target image block taking the rough point to be matched as a center and a source image block taking a corresponding dense point to be matched as a center according to the formula (6); taking the rough point to be matched corresponding to the maximum value of the normalized correlation coefficient calculation value from the normalized correlation coefficient calculation values of all the rough points to be matched, and if the correlation coefficient of the point is greater than t and the difference between the maximum value of the normalized correlation coefficient calculation value and the second largest value is greater than e, considering the point as an initial dense homonymous point; otherwise, rejecting the point; t is more than or equal to 0.7 and is used for ensuring the correctness of initial matching, and e is more than 0 and less than or equal to 0.01 so as to ensure that the homonymy points have certain significance and improve the reliability.
Specifically, the target image block size is generally 11 pixels × 11 pixels.
Specifically, when searching for the initial dense homonym point, the following 2 strategies are adopted to improve the matching efficiency:
the method comprises the steps that firstly, when one-dimensional search is carried out in the search range of any scanning line of the right horizontal epipolar line image, the search step length is set to be 2 pixels, and the coordinates of the matching candidate points are obtained through calculation;
that is, the coordinates of the matching candidate points are not traversed to each pixel in the search range, but the correlation coefficient is obtained every other pixel, so that the search amount is halved, and the purpose of improving the matching efficiency is achieved.
The rationale for this is: because the searched initial dense homonymous points are subjected to fine adjustment by adopting least square matching, and the correction range of the least square matching can reach 2 pixels, even if the actual best matching point is in the interval pixels which are not scanned, the searched initial dense homonymous points are adjacent pixels of the actual best matching point, the actual best position can be corrected by the least square matching, and point-by-point matching is not needed.
And strategy two, when searching for initial dense homonymous points in the search range, incremental calculation is adopted when calculating the gray average value in the correlation coefficient:
calculating to obtain the source image mean value at the initial matching candidate point of the search range
Figure DEST_PATH_IMAGE078
And the mean value of the target image->
Figure DEST_PATH_IMAGE079
(ii) a Subsequently, in a search match performed successively along the search range, a value is selected which is greater than or equal to>
Figure 325077DEST_PATH_IMAGE078
Remains unchanged, is paired with>
Figure 527389DEST_PATH_IMAGE079
An update is made by the sum of the gray levels (N × ^ er) from the previous window>
Figure 543624DEST_PATH_IMAGE079
) Subtracting the gray scale of the front 2 columns of the window and adding the gray scale of the rear 2 columns of the rear window to obtain the gray scale sum of the rear window so as to obtain the sum of the gray scales of the rear window and the greater or lesser of the gray scale sum>
Figure 147781DEST_PATH_IMAGE079
The incremental calculation formula is:
Figure 709212DEST_PATH_IMAGE020
wherein, the target image block taking the mth matching candidate point as the center is recorded as the mth target image block,
Figure 398819DEST_PATH_IMAGE021
is the mean value of the gray scale of the mth target image block>
Figure 703899DEST_PATH_IMAGE022
Is the mean value of the gray scale of the (m + 1) th target image block>
Figure 677409DEST_PATH_IMAGE023
Is the mean value of the gray levels of the pixels in the leftmost two columns in the mth target image block, and is greater than or equal to>
Figure 409741DEST_PATH_IMAGE024
And the gray level average value of the pixels in the rightmost two columns in the (m + 1) th target image block is obtained. />
The use of the incremental calculation process can reduce the amount of gradation calculation.
Strategy three, adding sequence consistency constraint to the matching points along the scanning line, namely adding two points on the left horizontal epipolar line image
Figure DEST_PATH_IMAGE080
Figure DEST_PATH_IMAGE081
The corresponding matching points on the right horizontal epipolar line image are ^ and ^ respectively>
Figure DEST_PATH_IMAGE082
Figure DEST_PATH_IMAGE083
Is set>
Figure 752691DEST_PATH_IMAGE081
Is located->
Figure 595882DEST_PATH_IMAGE080
To the right side of
Figure 174631DEST_PATH_IMAGE083
Must be located->
Figure 592712DEST_PATH_IMAGE082
To the right, such processing further enhances the reliability of the matching.
Step five, least square matching;
and C, performing least square image matching on the initial dense homonymous points acquired in the step four, so that the matching precision of the homonymous points reaches a sub-pixel level.
Because point-by-point searching and matching can only reach the matching precision of 1 pixel, the searched initial matching point is finely adjusted by adopting least square matching, so that the homonymy point reaches the matching precision of a sub-pixel level, and the visual effect of a reconstructed scene is improved.
So far, the technical solutions of the present invention have been described in connection with the preferred embodiments shown in the drawings, but it is easily understood by those skilled in the art that the scope of the present invention is obviously not limited to these specific embodiments. Equivalent changes or substitutions of related technical features can be made by those skilled in the art without departing from the principle of the invention, and the technical scheme after the changes or substitutions can be within the protection scope of the invention.
The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention; various modifications and alterations to this invention will become apparent to those skilled in the art. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims (7)

1. A tilt correction dense matching method based on remote sensing images is characterized by comprising the following steps:
step S1, respectively extracting feature points from a left oblique image shot by a left camera and a right oblique image shot by a right camera by adopting an SIFT feature descriptor to perform feature matching to obtain sparse homonymy points;
s2, carrying out relative orientation on the stereopair based on the sparse homonymous points and the camera intrinsic parameters to obtain the relative pose relationship of the left camera and the right camera;
s3, constructing a horizontal epipolar line coordinate system based on the relative pose relationship, and correcting the left oblique image and the right oblique image into a left horizontal epipolar line image and a right horizontal epipolar line image respectively through the mapping relationship between the horizontal epipolar line coordinate system and the left camera coordinate system as well as the mapping relationship between the horizontal epipolar line coordinate system and the right camera coordinate system;
step S4, determining a plurality of dense points to be matched at set matching intervals in the left horizontal epipolar line image obtained in the step S3, and performing one-dimensional gray-scale correlation matching on any dense point to be matched in the right horizontal epipolar line image to obtain initial dense homonymy points;
s5, performing least square image matching on the initial dense homonymy points to obtain sub-pixel level dense homonymy points;
in step S4, the step of obtaining the initial dense homonym point includes:
step S41, respectively calculating the initial parallaxes of the left horizontal epipolar line image and the right horizontal epipolar line image in the X axis and the Y axis of the horizontal epipolar line coordinate system;
step S42, determining a rough matching point of any dense point to be matched in the left horizontal epipolar line image in the right horizontal epipolar line image according to the initial parallax;
step S43, determining a one-dimensional search range in the right horizontal epipolar line image according to the position coordinate of the rough matching point as a search center and a set search radius, so as to determine a plurality of matching candidate points;
step S44, respectively calculating a normalized cross correlation coefficient between the gray average value of the source image block with the dense point to be matched as the center and the gray average value of the target image block with any one matching candidate point as the center;
s45, determining initial dense homonymous points of the dense points to be matched according to the normalized cross-correlation coefficient;
in the step S43, setting a search step size to 2 pixels to determine the matching candidate points within the one-dimensional search range;
in step S44, the gray level mean value is calculated in an incremental manner, where the incremental calculation formula is:
Figure QLYQS_1
wherein, the target image block taking the mth matching candidate point as the center is recorded as the mth target image block,
Figure QLYQS_2
is the mean value of the gray scale of the mth target image block>
Figure QLYQS_3
Is the mean value of the gray scale of the (m + 1) th target image block>
Figure QLYQS_4
Is the mean value of the gray scales of the pixels of the leftmost two columns in the mth target image block, and is combined with the gray scales of the pixels of the leftmost two columns in the mth target image block>
Figure QLYQS_5
And the gray level average value of the pixels in the rightmost two columns in the (m + 1) th target image block is obtained.
2. The tilt correction dense matching method based on remote sensing images as claimed in claim 1, wherein in step S2, the relative pose relationship is obtained by the sparse homonymy point and camera intrinsic parameters by using a relative orientation direct solution and determined after adjustment optimization by a beam method, and the relative pose relationship is determined by using a pose transformation matrix
Figure QLYQS_6
Description of the invention,
Figure QLYQS_7
Setting the position of any point P in the left camera coordinate system as ^>
Figure QLYQS_8
Position of point of same name in the right camera coordinate system->
Figure QLYQS_9
Figure QLYQS_10
Wherein,
Figure QLYQS_11
is phase/based on the coordinate system of the right camera>
Figure QLYQS_12
For a rotation matrix of the left camera coordinate system, ->
Figure QLYQS_13
Is a translation vector.
3. The tilt-correction dense matching method based on remote-sensing images according to claim 2, wherein in step S3, the horizontal epipolar coordinate system is a right-hand three-dimensional rectangular coordinate system, the origin of the horizontal epipolar coordinate system is set as the origin of the left camera coordinate system, the X-axis of the horizontal epipolar coordinate system is the connecting line of the left camera projection center and the right camera projection center,
the coordinates of the X-axis vector of the horizontal epipolar line coordinate system in the left camera coordinate system are as follows:
Figure QLYQS_14
the coordinates of the Y-axis vector of the horizontal epipolar line coordinate system in the left camera coordinate system are as follows:
Figure QLYQS_15
the coordinate of the Z-axis vector of the horizontal epipolar line coordinate system in the left camera coordinate system is as follows:
Figure QLYQS_16
the rotation transformation matrix from the vector under the left camera coordinate system to the direction of the horizontal epipolar line coordinate system is as follows:
Figure QLYQS_17
the rotation transformation matrix of the vector under the right camera coordinate system to the direction of the horizontal epipolar coordinate system is as follows:
Figure QLYQS_18
wherein,
Figure QLYQS_19
is the average of the Z-axis direction vector of the left camera coordinate system and the Z-axis direction vector of the right camera coordinate system,
Figure QLYQS_20
Figure QLYQS_21
is a Z-axis vector of the left camera coordinate system>
Figure QLYQS_22
Is a Z-axis vector of the right camera coordinate system, is->
Figure QLYQS_23
Is the rotation matrix->
Figure QLYQS_24
The third column of elements.
4. The method of claim 3, wherein the tilt correction is based on a dense matching of the remote sensing imageCharacterized in that, in the step S3, the step B is performed by
Figure QLYQS_25
And &>
Figure QLYQS_26
Solving the corresponding coordinates of each pixel point in the original image in the horizontal epipolar line coordinate system to form the mapping of the position of each pixel point in the original image and the position of the corresponding point of the horizontal epipolar line image, and filling the color value of each pixel point in the original image into the position of the corresponding point of the horizontal epipolar line image to generate a corrected horizontal epipolar line image, wherein,
the original image comprises a left inclined image under the left camera coordinate system and a right inclined image under the right camera coordinate system, the left inclined image generates a left horizontal epipolar line image after being corrected, and the right inclined image generates a right horizontal epipolar line image after being corrected.
5. The method for densely matching the tilt correction based on the remote sensing image according to claim 4, wherein in the step S41, the calculation of the initial parallax of the left horizontal epipolar line image and the right horizontal epipolar line image comprises:
step S411, respectively calculating X parallax and Y parallax of each sparse homonymous point in the left horizontal epipolar line image and the right horizontal epipolar line image under the horizontal epipolar line coordinate system;
step S412, selecting three adjacent sparse homonymous point triangulation networks to form a plurality of parallax calculation areas, and calculating the X parallax d of any point in the parallax calculation areas in the single parallax calculation area in the X direction and the Y direction by adopting linear interpolation x And Y parallax d y
Step S413, when the dense point to be matched is located in a single parallax calculation region, adopting an X parallax d corresponding to the single parallax calculation region x And Y parallax d y
6. The tilt-corrected dense matching method based on remote sensing images according to claim 5, wherein in step S45, an initial matching correlation coefficient standard t and an initial matching correlation coefficient difference standard e are set by determining whether the normalized cross-correlation coefficient meets the standard to determine whether an initial dense homonymous point exists at a single dense point to be matched, wherein t ≧ 0.7,0 < e ≦ 0.01, the maximum value of the normalized cross-correlation coefficients between the gray-scale mean of the source image block centered at the dense point to be matched and the gray-scale mean of the target image block centered at any one of the matching candidate points is designated as Enccmax1, the next largest value is designated as Enccmax2, and Δ Encc = Enccmax1-Enccmax2 are set,
when Enccmax1 is larger than or equal to t and delta Encc is larger than e, judging that the normalized cross-correlation coefficient accords with an initial matching standard, and setting a matching candidate point corresponding to Enccmax1 as an initial dense homonymous point of the point to be matched;
and when Enccmax1 is less than t or delta Encc is less than or equal to e, judging that the normalized cross-correlation coefficient does not accord with the initial matching standard, eliminating the points to be matched, which correspond to Enccmax1, and not having the initial dense homonymous points.
7. The tilt-correction dense matching method based on remote sensing images according to claim 6, wherein in the step S5, the position of each initial dense homonymous point is adjusted by using least square image matching to obtain two times dense homonymous points, the position coordinates of the sub-pixel level dense homonymous points are determined according to a maximum value Encc2 in the normalized cross-correlation coefficient between the gray-scale mean value of each image block corresponding to the two times dense homonymous points and the gray-scale mean value of the source image block with the corresponding dense point to be matched as the center, and a two times matching correlation coefficient standard tm is set, wherein tm is greater than or equal to 0.9,
when Encc2 is larger than or equal to tm, judging that the two-times normalized cross-correlation coefficient meets a sub-pixel level matching standard, and setting a two-times dense homonymy point corresponding to Encc2 as a sub-pixel level dense homonymy point of the dense point to be matched;
when Encc2 < tm, the two-times normalized cross-correlation coefficient is determined not to meet the sub-pixel level matching criterion.
CN202310029421.6A 2023-01-09 2023-01-09 Tilt correction dense matching method based on remote sensing image Active CN115719320B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202310029421.6A CN115719320B (en) 2023-01-09 2023-01-09 Tilt correction dense matching method based on remote sensing image

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202310029421.6A CN115719320B (en) 2023-01-09 2023-01-09 Tilt correction dense matching method based on remote sensing image

Publications (2)

Publication Number Publication Date
CN115719320A CN115719320A (en) 2023-02-28
CN115719320B true CN115719320B (en) 2023-04-14

Family

ID=85257909

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202310029421.6A Active CN115719320B (en) 2023-01-09 2023-01-09 Tilt correction dense matching method based on remote sensing image

Country Status (1)

Country Link
CN (1) CN115719320B (en)

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116597184B (en) * 2023-07-11 2023-09-22 中国人民解放军63921部队 Least square image matching method

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108592884A (en) * 2018-04-24 2018-09-28 武汉大学 A kind of general linear array satellite core line image generating method
CN113034556A (en) * 2021-03-19 2021-06-25 南京天巡遥感技术研究院有限公司 Frequency domain related semi-dense remote sensing image matching method

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111080709B (en) * 2019-11-22 2023-05-05 大连理工大学 Multispectral stereo camera self-calibration algorithm based on track feature registration

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108592884A (en) * 2018-04-24 2018-09-28 武汉大学 A kind of general linear array satellite core line image generating method
CN113034556A (en) * 2021-03-19 2021-06-25 南京天巡遥感技术研究院有限公司 Frequency domain related semi-dense remote sensing image matching method

Non-Patent Citations (7)

* Cited by examiner, † Cited by third party
Title
廖小超 ; 胡坚 ; 李传荣 ; 吴克 ; 薛博 ; 李群智 ; .基于迭代式动态规划的影像密集匹配.遥感技术与应用.2015,(第01期),全文. *
张漫 等.无人机核线影像的稀疏匹配与稠密匹配.测绘通报.2017,(第05期),全文. *
杨晟 ; 李学军 ; 王珏 ; 杨阿华 ; .连续尺度复合分析核线重排列影像准稠密匹配.计算机技术与发展.2013,(第04期),全文. *
杨阿华 ; 李学军 ; 谢剑薇 ; 李东岳 ; .由航空影像自动重建大范围3维地形.中国图象图形学报.2014,(第10期),全文. *
杨阿华.基于直接解算与迭代优化的相对定向方法.计算机应用.2014,全文. *
王奋飞 ; 任鑫 ; 刘建军 ; 施俊 ; 李春来 ; .嫦娥二号影像密集匹配及DEM制作方法.武汉大学学报(信息科学版).2017,(第07期),全文. *
赵爽 ; 李学军 ; 刘涛 ; 谢剑薇 ; .基于大比例尺航空影像共面约束条件的相机自检校方法.测绘学报.2017,(第01期),全文. *

Also Published As

Publication number Publication date
CN115719320A (en) 2023-02-28

Similar Documents

Publication Publication Date Title
CN110288642B (en) Three-dimensional object rapid reconstruction method based on camera array
CN111126148B (en) DSM (digital communication system) generation method based on video satellite images
JP4382797B2 (en) Corresponding point search method and three-dimensional position measurement method
TWI555379B (en) An image calibrating, composing and depth rebuilding method of a panoramic fish-eye camera and a system thereof
CN103345736B (en) A kind of virtual viewpoint rendering method
CN113034568B (en) Machine vision depth estimation method, device and system
CN107767440A (en) Historical relic sequential images subtle three-dimensional method for reconstructing based on triangulation network interpolation and constraint
CN103456038A (en) Method for rebuilding three-dimensional scene of downhole environment
WO2017156905A1 (en) Display method and system for converting two-dimensional image into multi-viewpoint image
CN103198524A (en) Three-dimensional reconstruction method for large-scale outdoor scene
CN106846416A (en) Unit beam splitting bi-eye passiveness stereo vision Accurate Reconstruction and subdivision approximating method
CN104537707A (en) Image space type stereo vision on-line movement real-time measurement system
CN108399631B (en) Scale invariance oblique image multi-view dense matching method
CN113112588A (en) Underground pipe well three-dimensional visualization method based on RGB-D depth camera reconstruction
CN111028281A (en) Depth information calculation method and device based on light field binocular system
CN117456114B (en) Multi-view-based three-dimensional image reconstruction method and system
CN115719320B (en) Tilt correction dense matching method based on remote sensing image
CN114627491A (en) Single three-dimensional attitude estimation method based on polar line convergence
JP7416573B2 (en) Stereoscopic image generation device and its program
CN114935316B (en) Standard depth image generation method based on optical tracking and monocular vision
CN113808070B (en) Binocular digital speckle image related parallax measurement method
CN112258635B (en) Three-dimensional reconstruction method and device based on improved binocular matching SAD algorithm
CN110068308B (en) Distance measurement method and distance measurement system based on multi-view camera
Zhang et al. Non-rigid registration of mural images and laser scanning data based on the optimization of the edges of interest
JP2001167249A (en) Method and device for synthesizing image and recording medium stored with image synthesizing program

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant