WO2020206903A1

WO2020206903A1 - Image matching method and device, and computer readable storage medium

Info

Publication number: WO2020206903A1
Application number: PCT/CN2019/102187
Authority: WO
Inventors: 王义文; 王健宗
Original assignee: 平安科技（深圳）有限公司
Priority date: 2019-04-08
Filing date: 2019-08-23
Publication date: 2020-10-15
Also published as: CN110135455A; CN110135455B

Abstract

An image matching method and device (1), and a computer readable storage medium. The method comprises: generating an image imaging picture according to a scene image captured by an aerial photography instrument, performing primary image matching on the image imaging picture by using a scale invariant feature transformation method, and generating a primary image matching set (S1); on the basis of the primary image matching set, generating epipolar line images, calculating the degree of overlap between the epipolar line images, completing secondary image matching, and generating a secondary image matching set (S2); and on the basis of the secondary image matching set, establishing dense matching of all pixel points between images, generating a third-time image matching set, and executing three-dimensional reconstruction to obtain a reconstructed scene image (S3). According to the novel image matching solution applied to a dense three-dimensional scene under aerial photography, the image matching efficiency can be improved.

Description

Image matching method, device and computer readable storage medium

This application is based on the Paris Convention declaration that it enjoys the priority of the Chinese patent application filed on April 8, 2019 with the application number CN201910274078.5 and titled "Image matching method, device and computer readable storage medium". The whole content is incorporated in this application by reference.

Technical field

This application relates to the field of computer technology, and in particular to an image matching method, device and computer-readable storage medium.

Background technique

Image matching refers to the process of identifying points with the same name between two or more images through a certain matching algorithm. It is an important preliminary step in image fusion, target recognition, target change detection, computer vision and other problems. It has a wide range of applications in many fields such as remote sensing, digital photogrammetry, computer vision, cartography and military applications. At present, the most effective and effective method for image matching is to judge the different points of the image by visual inspection; secondly, according to the principle of the essence of the image, which is the principle of pixels, compare the gray values of all pixels in the target area; or find the target image based on the principle of template matching The same or the most similar position to the sub-image in the search image, etc.

The above methods are all feasible in some fields, but when applied to the matching of dense stereo scenes under aerial photography, the effect will be unsatisfactory. Because the number of images under aerial photography is not only huge, but the objects in each picture are also very dense, human eyes are overwhelmed and impractical; while pixel matching is used to make difference between multiple image pixels, it is subject to noise, quantization errors, and tiny Illumination changes, minimal translation and other effects will produce large pixel differences, which will affect the matching effect; while the template matching method is applied to dense images, it needs to generate a large number of matching modules to search in the picture, and the timeliness is general , At the same time, affected by image noise, the probability of mismatch is also very high. In general, because the dense scenes under aerial photography are complex and changeable, the ability to perform 3D reconstruction can more effectively help users to analyze and research, but the above methods lack this function.

Summary of the invention

The present application provides an image matching method, device, and computer-readable storage medium, the main purpose of which is to provide a new image matching method applied to dense stereo scenes under aerial photography to improve image matching efficiency.

In order to achieve the above objective, an image matching method provided by this application includes:

According to the scene image taken by the aerial camera, the image imaging map is generated, and the first image matching is performed on the image imaging map using the scale-invariant feature transformation method to generate the first image matching set;

Based on the first image matching set, generating epipolar images and calculating the degree of overlap between the epipolar images, completing the second image matching, and generating a second image matching set;

Based on the second image matching set, a dense matching of all pixels between images is established, a third image matching set is generated, and three-dimensional reconstruction is performed to obtain a reconstructed scene image.

In addition, in order to achieve the above-mentioned object, the present application also provides an image matching device. The device includes a memory and a processor. The memory stores an image matching program that can run on the processor. The image matching program When executed by the processor, the steps of the image matching method described above are realized.

In addition, in order to achieve the above object, the present application also provides a computer-readable storage medium with an image matching program stored on the computer-readable storage medium, and the image matching program can be executed by one or more processors to achieve The steps of the image matching method described above.

The image matching method, device, and computer-readable storage medium proposed in this application generate an image imaging map based on the scene image shot by an aerial camera, and use the scale-invariant feature transformation method to perform the initial image matching on the image imaging map to generate the initial image matching set, Based on the first image matching set, generate epipolar images and calculate the degree of overlap between the epipolar images, complete the second image matching, and generate a second image matching set based on the second image matching set Establish dense matching of all pixels between images, generate a third image matching set, and perform 3D reconstruction to obtain a reconstructed scene image. This application improves the efficiency of image matching, and can perform three-dimensional reconstruction of images of dense scenes under aerial photography, so as to more effectively help users to conduct analysis and research.

Description of the drawings

FIG. 1 is a schematic flowchart of an image matching method provided by an embodiment of this application;

2 is a schematic diagram of the internal structure of an image matching device provided by an embodiment of the application;

Fig. 3 is a schematic diagram of modules of an image matching program in an image matching device provided by an embodiment of the application.

The realization, functional characteristics, and advantages of the purpose of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings.

detailed description

In order to make the purpose, technical solutions, and advantages of this application clearer, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the application, and not used to limit the application. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art without creative work shall fall within the protection scope of this application.

The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of this application and the above-mentioned drawings are used to distinguish similar objects, without having to use To describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances so that the embodiments described herein can be implemented in an order other than the content illustrated or described herein. In addition, the descriptions of "first", "second", etc. are only for descriptive purposes, and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of indicated technical features. Therefore, the features defined with "first" and "second" may explicitly or implicitly include at least one of the features.

Further, the terms "including" and "having" and any variations of them are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to clearly listed Instead, it may include other steps or units that are not clearly listed or are inherent to these processes, methods, products, or equipment.

In addition, the technical solutions between the various embodiments can be combined with each other, but it must be based on what can be achieved by a person of ordinary skill in the art. When the combination of technical solutions is contradictory or cannot be achieved, it should be considered that such a combination of technical solutions does not exist. , Not within the scope of protection required by this application.

This application provides an image matching method. Referring to FIG. 1, it is a schematic flowchart of an image matching method provided by an embodiment of this application. The method can be executed by a device, and the device can be implemented by software and/or hardware.

In this embodiment, the image matching method includes:

S10. Generate an image imaging map according to the scene image taken by the aerial camera, and perform an initial image matching on the image imaging map by using a scale-invariant feature transformation method to generate an initial image matching set.

The scene images taken by aerial equipment, such as drones, helicopters and other flight control systems, have a large number of images and a wide viewing angle, especially buildings, which are dense and dense. Therefore, this application first restores the overlapping image sets to their respective positions, and reconstructs the imaging map of the object.

In the preferred embodiment of the present application, based on the low-precision position, attitude information, and approximate height of the survey area recorded by the aerial equipment when shooting scene images, the model formula for object imaging under aerial photography is used to recover overlapping scene images. Position, generate image imaging map.

In a preferred embodiment of the present application, the model formula of the object imaging is as follows:

sm=KR[I-C]M,

Among them, s is the scale factor; m is the coordinate of the image point, and M is the coordinate of the object point (the object point and the image point are the object position and the image position in the optical imaging respectively); K is the parameter matrix in the aerial photography tool, It is composed of focal length and principal point coordinates; R is a rotation matrix, which can be converted to approximate values according to the yaw, pitch, and roll recorded by the aerial tool’s system; C is the projection center The position vector can be approximated directly from the longitude, latitude, and altitude recorded by the GPS of the aerial photography tool; I is the third-order unit matrix.

Using the above-mentioned object imaging model formula, the imaging map of n images can be obtained.

The generation of n pieces of video imaging FIG, referred to as an image set, image set into the free edge set corresponding to FIG _{_{E: G n = (V n}} , E n), wherein, referred to as a vertex set V _n, En _is called an edge set (a graph is a widely used data structure. The nodes in the graph are called vertices. The relationship between two vertices can be represented by a pair, called an edge. If the graph represents an edge Even pairs are ordered, then the graph is called a directed graph, if the pairs representing edges are disordered, then it is called an undirected graph). No set of edges E in the drawing, represents the number nE E _n-side is, for each table represents one image, on behalf of the E nE _n-th image for subsequent image matching process performed between only one image pair nE . If the relationship between images is not considered and the exhaustive traversal strategy is used for image matching, the total number of matches is

Usually n*(n-1)/2 will be much larger than nE. Therefore, the method of constructing the image relationship undirected graph limits the scope of image matching, can avoid blind image matching, reduce the total image matching calculation complexity from O(n ² ) to O(n), and improve the matching calculation efficiency ; At the same time, it can effectively eliminate the interference of unrelated image pairs, fundamentally avoid mismatches caused by non-overlapping images, and improve the accuracy of matching and the robustness of reconstruction.

In the undirected graph edge set E, the scale-invariant feature transform (SIFT) algorithm is used for image matching. In image matching, if there are few matching points in the two images I _i and I _j , which are smaller than the threshold N ₁ , it means that the overlap is small or the correlation is weak, and (I _i , I _j ) is removed from the set E. If the number of matching points in the two images I _i and I _j is greater than the threshold N ₁ , then the imaging image pair is retained to generate

There are n ₁ E image pairs in total, and the first image matching set E _{1 is} generated.

S20: Based on the first image matching set, generate an epipolar image and calculate the degree of overlap between the epipolar images, complete the second image matching, and generate a second image matching set.

The above S10 only filters out images with no repetition or low repetition. For images with certain repetition, this application continues to use the epipolar image method to perform matching filtering.

The epipolar image is a method of changing the search range from a two-dimensional plane image to a one-dimensional straight line during the matching process. Specifically, in dense three-dimensional aerial photography, the plane formed by the shooting baseline and any ground point is called the nuclear surface, and the intersection line between the nuclear surface and the image surface is called the nuclear line. In a stereo pair, the image points with the same name must be on the epipolar line of the same name, and the image points on the epipolar line of the same name have a one-to-one correspondence. Therefore, if an epipolar pair with the same name can be determined on a stereo image pair, then using the above-mentioned properties of the epipolar pair with the same name, the search and matching of the two-dimensional image can be transformed into the search and matching along the epipolar line. The epipolar image eliminates the upper and lower parallax between the stereo images, narrows the search range, reduces the amount of matching calculations, and improves the matching accuracy, so it is of great significance for dense stereo image matching.

A preferred embodiment of the present application discloses a method for making and matching epipolar images for generating epipolar images and calculating the degree of overlap between the epipolar images. The method includes: (a) using the SIFT algorithm to compare the

After image pair point feature extraction, uniformly distributed high-precision points with the same name are obtained, the basic matrix estimation based on the RANSAC strategy is used to obtain the basic matrix of n ₁ E image pairs; (b) using the basic matrix to determine each group of points with the same name Corresponding core line with the same name; (c) According to the principle that the core line must intersect at the core point, the least square method is used to determine

The core point coordinates of the image pair are used to generate a quick mapping of the epipolar lines between the images according to the core point coordinates, and the epipolar line is resampled by bilinear interpolation along the epipolar line direction to complete the epipolar image production and matching regeneration

There are a total of n ₂ E image pairs to generate the second image matching set.

The epipolar image production method based on the basic matrix can avoid the problems of iterative calculation and initial value assignment when calculating the relative relationship, and it can also have good accuracy when the aerial photography uses a large angle of view. The following are the specific steps of the above step (c):

(1) Determine the coordinates of the nuclear point:

Based on the basic matrix, a rotation matrix and a projection matrix are constructed, and the rotation matrix is divided into x, y, and z axes, each of which is

among them,

Is expressed as follows:

And get the projection matrix

among them,

Is the camera parameter of the camera left,

Are the camera parameters of camera right, t ^left and t ^right are the components of the camera parameters of camera left and camera right in the x, y, and z axes respectively;

According to the principle that the core line must intersect at the core point, the least square method is used to determine the core point coordinates (x _p , y _p ) of the image from the calculation result of the projection matrix.

(2) Perform nuclear mapping:

The purpose of mapping is to directly correct the epipolar image on the central projection image. The specific steps of the mapping are: derive the central projection according to the collinear condition equation, calculate the angle relationship between two adjacent epipolar lines when the epipolar lines are sampled, and complete the determination of each epipolar line on the central projection image; use The basic matrix generated above can determine each image pair

The corresponding core lines are based on

The nuclear point coordinates of each nuclear image in the image determine the epipolar line of the point, and complete the epipolar line correspondence between the same image pair to obtain

The epipolar equation is:

Among them, (x _p , y _p ) are the coordinates of the core point calculated above, (x _base , y _base ) are the reference coordinates of the center projection image, and the same goes for

The epipolar equation.

(3) Generate the second image matching set:

Based on the epipolar equation, each image pair is established

After the epipolar line mapping, according to the resampling rule of the bilinear interpolation method, the epipolar line image is generated and the overlap degree is calculated. The image pairs whose overlap degree of the epipolar line image is less than the threshold N ₂ are discarded to generate

There are n ₂ E image pairs in total, and the second image matching set E _{2 is obtained} .

Although the second image matching set E ₂ generated above solves most of the similarity problems of stereo matching and meets the standard of image matching, it is still necessary for dense stereo environments such as battlefield monitoring and disaster search and rescue. The two-dimensional image restores the geometric structure of the three-dimensional object surface, so further processing is required, and the following S30 is executed.

S30. Based on the second image matching set, establish dense matching of all pixels between the images to realize three-dimensional reconstruction.

The preferred embodiment of the present application adopts a dense matching algorithm, and on the basis of the second image matching set E ₂ , the corner detection algorithm of the least equivalent segmentation absorbing core is used to extract respectively

The corner points of the corner points form a set of matching points of the corner points; combined with the epipolar geometric constraints, dynamic programming algorithms and other methods to establish

Dense matching of all pixels between images. Specific steps are as follows:

The second image matching set of corner points in an image in E ₂ (a) detection.

The corner point is the point where the local curvature changes the most on the contour of the image. It contains important information in the image, so it is of great significance for the detail matching of the image in the image set E ₂ . The preferred embodiment of the present application adopts the smallest uni-value segment assimilating nucleus (SUSAN) method to detect image corners: Gaussian smoothing is performed on the input image; each pixel in the image is traversed, and Sobel is used first The operator (it is a discrete first-order difference operator, used to calculate the approximate value of a gradient of the image brightness function) to determine whether the pixel is an edge point; if it is an edge point, the loss function according to the global energy equation is the smallest To determine whether the path loss L _r (p, d) of the corner point is minimized, the determination principle is as follows:

Where C(p,d) is the local loss of the path,

Is the loss of the previous step of the path, L _r ,min(pr) is the minimum loss of the previous step of the path, from which it can be determined whether the point is a corner point, and redundant corner points are removed; further judge if the two corners are detected If the points are adjacent, the corner points with larger L _r (p,d) are removed. After the above steps, the corner points of the image pair in the second image matching set E ₂ can be detected.

(2) Yes

The corner points of the image pair are automatically matched to obtain a matching point set.

The automatic matching of corner points can effectively distinguish the difference between image pairs based on the similarities and differences of the corner points, which is an effective means to achieve precise matching. The corner point automatic matching can be divided into these steps: ①For

Each point in the set of image corners is in

Find matching corner points in the corresponding search area in the image, similarly, for

Each point in the set of image corners is searched in the same way

Find their corresponding matching points in the image, and call the intersection of these two matching point sets the initial matching point set K ₁ ; ②In the initial matching point set K ₁ , the pair is

The corner point where the diagonal points of the image are concentrated with each other, find the matching point in the corresponding search area, calculate the similarity between this point and each candidate matching point in the search area, and select the candidate matching point with the greatest similarity as its match point. In a preferred embodiment of the present application, the similarity calculation method adopts the gradient size similarity method: if the gradient size of a pixel is g, and the gradient of another pixel that matches it approximately obeys the normal distribution, then the two The similarity l _g of each pixel is:

Among them, d(x) is the density function and k is the density coefficient. After calculating the similarity, the matching point set K _{2 is obtained} .

(3) According to the matching point set K ₂ , establish

Dense matching of all pixels between images.

In a preferred embodiment of the present application, the dense matching includes: ① Obtain the matching point set K ₂ according to the limit geometric constraint relationship

The polar line correspondence of the image is obtained, and the limit correspondence set K _{3 is obtained} . The so-called limit geometric constraint relationship means that if l and l'are the two corresponding epipolar lines in the left and right images, the corresponding point of the point p on the epipolar line l in the left image in the right image must be on the epipolar line l' on. ②From the generated limit correspondence set K ₃ , the polar lines in K ₃ are segmented according to gray levels. Each polar line is divided into several gray segments, and the gray values of pixels on each segment are similar. . The formula for gray scale segmentation is as follows:

The physical meaning of the above formula is to divide the continuous points of gray value in a certain range into one section. Among them, I (x _t , y _t ) is the gray value of the pixel (x _t , y _t ); w is the number of pixels on a gray segment, that is, the length of the gray segment; T is For a certain threshold, the smaller the value, the fewer pixels are divided into a certain gray-scale segment, and the more gray-scale segments. According to experimental research, the matching effect is the best when T is set to 3. The gray-level segment set of is K ₄ ; ③Using dynamic programming algorithm (an optimization method to find the best matching path) to establish the correspondence between gray-level segments, and using linear interpolation to save the corresponding gray-level The corresponding relationship between the pixels is established between the segments, so that the dense matching of all the pixels between the images is realized, and the third image matching set E _{3 is obtained} .

After dense matching, all pixels in the third image matching set E ₃ have a corresponding relationship, so the depth of field of the scene can be calculated, and the preset 3Dmax software can be used to reconstruct the scene to restore the three-dimensional geometry of the scene space Information to get the reconstructed image.

The application also provides an image matching device. Referring to FIG. 2, it is a schematic diagram of the internal structure of an image matching device provided by an embodiment of this application.

In this embodiment, the image matching device 1 may be a PC (Personal Computer, personal computer), or a terminal device such as a smart phone, a tablet computer, or a portable computer. The image matching device 1 at least includes a memory 11, a processor 12, a communication bus 13, and a network interface 14.

The memory 11 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), magnetic memory, magnetic disk, optical disk, and the like. The memory 11 may be an internal storage unit of the image matching device 1 in some embodiments, such as a hard disk of the image matching device 1. In other embodiments, the memory 11 may also be an external storage device of the image matching device 1, such as a plug-in hard disk, a smart media card (SMC), and a secure digital (Secure Digital, SD card, Flash Card, etc. Further, the memory 11 may also include both an internal storage unit of the image matching device 1 and an external storage device. The memory 11 can be used not only to store application software and various data installed in the image matching device 1, such as the code of the image matching program 01, etc., but also to temporarily store data that has been output or will be output.

The processor 12 may be a central processing unit (CPU), controller, microcontroller, microprocessor or other data processing chip in some embodiments, and is used to run the program code or processing stored in the memory 11 Data, such as execution of image matching program 01, etc.

The communication bus 13 is used to realize the connection and communication between these components.

The network interface 14 may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface), and is usually used to establish a communication connection between the device 1 and other electronic devices.

Optionally, the device 1 may also include a user interface. The user interface may include a display (Display) and an input unit such as a keyboard (Keyboard). The optional user interface may also include a standard wired interface and a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an OLED (Organic Light-Emitting Diode, organic light emitting diode) touch device, etc. Among them, the display can also be called a display screen or a display unit as appropriate, and is used to display the information processed in the image matching device 1 and to display a visualized user interface.

FIG. 2 only shows the image matching device 1 with components 11-14 and the image matching program 01. Those skilled in the art can understand that the structure shown in FIG. 1 does not constitute a limitation on the image matching device 1, and may include Fewer or more components than shown, or some combination of components, or different component arrangement.

In the embodiment of the device 1 shown in FIG. 2, the image matching program 01 is stored in the memory 11; when the processor 12 executes the image matching program 01 stored in the memory 11, the following steps are implemented:

The first step is to generate an image imaging map according to the scene image taken by the aerial camera, and use the scale-invariant feature transformation method to perform the initial image matching on the image imaging map to generate the initial image matching set.

The scene images captured by aerial equipment, such as drones, helicopters and other flight control systems, have a large number of images and a wide viewing angle, especially buildings, which are dense in number. Therefore, this application first restores the overlapping image sets to their respective positions, and reconstructs the imaging map of the object.

sm=KR[I-C]M,

The second step is to generate an epipolar image based on the initial image matching set and calculate the degree of overlap between the epipolar images to complete the second image matching and generate a second image matching set.

The above-mentioned first step is only to filter out images with no repetition or low repetition. For images with a certain degree of repetition, this application continues to use the epipolar image method to perform matching filtering.

(1) Determine the coordinates of the nuclear point:

among them,

Is expressed as follows:

And get the projection matrix

among them,

Is the camera parameter of the camera left,

(2) Perform nuclear mapping:

The corresponding core lines are based on

The epipolar equation is:

The epipolar equation.

(3) Generate the second image matching set:

Based on the epipolar equation, each image pair is established

Although the second image matching set E ₂ generated above solves most of the similarity problems of stereo matching and meets the standard of image matching, it is still necessary for dense stereo environments such as battlefield monitoring and disaster search and rescue. The two-dimensional image restores the geometric structure of the three-dimensional object surface, so further processing is required, and the following third step is performed.

In the third step, based on the second image matching set, a dense matching of all pixels between images is established to realize 3D reconstruction.

Dense matching of all pixels between images. Specific steps are as follows:

Where C(p,d) is the local loss of the path,

Is the loss of the previous step of the path, L _r , min(pr) is the minimum loss of the previous step of the path, from which it can be determined whether the point is a corner point, and redundant corner points are removed; further judge if the two corners are detected If the points are adjacent, the corner points with larger L _r (p,d) are removed. After the above steps, the corner points of the image pair in the second image matching set E ₂ can be detected.

(2) Yes

The corner points of the image pair are automatically matched to obtain a set of matching points.

Each point in the set of image corners is in

Each point in the set of image corners is searched in the same way

Among them, d(x) is the density function, and k is the density coefficient. After calculating the similarity, the matching point set K _{2 is obtained} .

(3) According to the matching point set K ₂ , establish

Dense matching of all pixels between images.

Optionally, in other embodiments, the image matching program may also be divided into one or more modules, and the one or more modules are stored in the memory 11 and are executed by one or more processors (in this embodiment, processing The module 12) is executed to complete the application. The module referred to in the application refers to a series of computer program instruction segments capable of completing specific functions, and is used to describe the execution process of the image matching program in the image matching device.

For example, referring to FIG. 3, it is a schematic diagram of the program modules of the image matching program in an embodiment of the image matching device of the present application. In this embodiment, the image matching program can be divided into a primary matching module 10, a secondary matching module 20, The three-time matching module 30 and the reconstruction module 40 are exemplary:

The first matching module 10 is used to generate an image imaging map according to the scene image taken by the aerial camera, and use the scale-invariant feature transformation method to perform the first image matching on the image imaging map to generate a first image matching set.

The secondary matching module 20 is configured to generate epipolar images based on the primary image matching set and calculate the degree of overlap between the epipolar images, complete the second image matching, and generate a second image matching set.

The third-order matching module 30 is configured to: based on the second-order image matching set, establish dense matching of all pixels between the images, and generate a third-order image matching set.

The reconstruction module 40 is configured to perform three-dimensional reconstruction according to the image matching set to obtain a reconstructed scene image.

The functions or operation steps implemented by the program modules such as the primary matching module 10, the secondary matching module 20, the tertiary matching module 30, and the reconstruction module 40 when executed are substantially the same as those in the above embodiment, and will not be repeated here.

In addition, an embodiment of the present application also proposes a computer-readable storage medium having an image matching program stored on the computer-readable storage medium, and the image matching program can be executed by one or more processors to implement the following operations:

The specific implementations of the computer-readable storage medium of the present application are basically the same as the foregoing embodiments of the image matching device and method, and will not be repeated here.

It should be noted that the serial numbers of the above embodiments of the present application are only for description, and do not represent the advantages and disadvantages of the embodiments. And the terms "include", "include" or any other variants thereof in this article are intended to cover non-exclusive inclusion, so that a process, device, article or method including a series of elements not only includes those elements, but also includes The other elements listed may also include elements inherent to the process, device, article, or method. If there are no more restrictions, the element defined by the sentence "including a..." does not exclude the existence of other identical elements in the process, device, article or method that includes the element.

Through the description of the above embodiments, those skilled in the art can clearly understand that the method of the above embodiments can be implemented by means of software plus the necessary general hardware platform. Of course, it can also be implemented by hardware, but in many cases the former is better.的实施方式。 Based on this understanding, the technical solution of this application essentially or the part that contributes to the existing technology can be embodied in the form of a software product, and the computer software product is stored in a storage medium (such as ROM/RAM) as described above. , Magnetic disk, optical disk), including several instructions to make a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) execute the method described in each embodiment of the present application.

The above are only preferred embodiments of this application, and do not limit the scope of this application. Any equivalent structure or equivalent process transformation made using the content of the description and drawings of this application, or directly or indirectly used in other related technical fields , The same reason is included in the scope of patent protection of this application.

Claims

An image matching method, characterized in that the method includes:

According to the scene image taken by the aerial camera, the image imaging map is generated, and the first image matching is performed on the image imaging map using the scale-invariant feature transformation method to generate the first image matching set;

Based on the first image matching set, generating epipolar images and calculating the degree of overlap between the epipolar images, completing the second image matching, and generating a second image matching set;

Based on the second image matching set, a dense matching of all pixels between images is established, a third image matching set is generated, and three-dimensional reconstruction is performed to obtain a reconstructed scene image.
5. The image matching method of claim 1, wherein generating an image imaging map according to a scene image shot by an aerial camera comprises:

According to the parameters recorded when the aerial photography instrument took the scene image, including the low-precision position, attitude information and the approximate height of the survey area, the model formula for object imaging under aerial photography is used to recover the overlapping scene images taken by the aerial photography instrument. Position, generate n image imaging maps, where the model formula of the object imaging is as follows:

sm=KR[I-C]M,

Among them, s is the scale factor, m is the image point coordinates, M is the object point coordinates, K is the parameter matrix in the aerial photography tool, R is the rotation matrix, C is the projection center position vector, and I is the third-order unit matrix.
3. The image matching method of claim 2, wherein the first image matching is performed on the image imaging map by using the scale-invariant feature transformation method to generate the first image matching set, comprising:

Converting the image set composed of the n image imaging images into a corresponding undirected image edge set E;

In the undirected graph edge set E, the scale-invariant feature transformation algorithm is used for image matching, and in the image matching, for the image (I i , I j ) ∈ E, if the two images I i and I j match If the number of points is less than the threshold N 1 , then (I i , I j ) will be removed from the set of undirected graph edges. If the number of matching points in the two images I i and I j is greater than the threshold N 1 , the imaging image pair is retained ,generate
Image pairs, generating the first image matching set.
The image matching method of claim 3, wherein based on the first image matching set, an epipolar image is generated and the degree of overlap between the epipolar images is calculated, and the second image matching is completed to generate a second Matching sets of images, including:

(a) Use the scale invariant feature transformation algorithm to analyze the
Perform point feature extraction on the image pair, after obtaining uniformly distributed high-precision points with the same name, use the RANSAC-based strategy to estimate the basic matrix to obtain the basic matrix;

(b) Using the basic matrix to determine the core line with the same name corresponding to each group of points with the same name;

(c) According to the principle that the core line must intersect at the core point, the least square method is used to determine
The core point coordinates of the image pair are used to generate a quick mapping of the epipolar lines between the images according to the core point coordinates, and the epipolar line is resampled by bilinear interpolation along the epipolar line direction to complete the epipolar image production and matching regeneration
Image pairs, generating the second image matching set.
The image matching method of claim 4, wherein the (c) comprises: building a rotation matrix and a projection matrix based on the basic matrix, and dividing the rotation matrix into x, y, and z axes, each for
among them,
Is expressed as follows:

And get the projection matrix

among them,
Is the camera parameter of the camera left,
Are the camera parameters of camera right, t left and t right are the components of the camera parameters of camera left and camera right in the x, y, and z axes respectively;

According to the principle that the core line must intersect at the core point, the least square method is used to determine the core point coordinates (x p , y p ) of the image from the calculation result of the projection matrix;

The central projection is derived according to the collinear condition equation. When the epipolar lines are sampled, the angle relationship between two adjacent epipolar lines is calculated to complete the determination of each epipolar line on the central projection image;

Using the basic matrix generated above, each image pair can be determined
The corresponding core lines are based on
The nuclear point coordinates of each nuclear image in the image determine the epipolar line of the point, and complete the epipolar line correspondence between the same image pair to obtain
The epipolar equation is:

Among them, (x p , y p ) are the coordinates of the core point calculated above, (x base , y base ) are the reference coordinates of the center projection image, and the same goes for
The epipolar equation;

Based on the epipolar equation, each image pair is established
After the epipolar line mapping, according to the resampling rule of the bilinear interpolation method, the epipolar line image is generated and the overlap degree is calculated. The image pairs whose overlap degree of the epipolar line image is less than the threshold N 2 are discarded to generate
Image pair to obtain the second image matching set.
8. The image matching method of claim 5, wherein the establishing a dense match of all pixels between images based on the second image matching set comprises:

On the basis of the second image matching set, the corner detection algorithm of the least equivalent segmentation absorbing core is used to extract the
The corner points of the corner points form a set of matching points of the corner points, combined with epipolar geometric constraints and dynamic programming algorithms to establish
Dense matching of all pixels between images.
5. The image matching method of claim 5, wherein the establishing a dense matching of all pixels between images based on the second image matching set to generate a third image matching set comprises:

Perform Gaussian smoothing on the input image, traverse each pixel in the image, and use the Sobel operator to determine whether the pixel is an edge point. If it is an edge point, further determine the path loss L r according to the principle of minimizing the loss function of the global energy equation (p,d) Whether to minimize, the judgment principle is as follows:

Among them, C(p,d) is the local loss of the path,

Is the loss of the previous step of the path, L r ,min(pr) is the minimum loss of the previous step of the path, which can be used to determine whether the point is a corner point. If the detected two corner points are adjacent, remove L r ( p, d) Larger corner points to detect the corner points of the image pair in the second image matching set;

for
Each point in the set of image corners is in
Find the matching corner point in the corresponding search area in the image, and call the intersection of the two matching point sets obtained as the initial matching point set K 1. In the initial matching point set K 1 , the pair is
The corner point where the diagonal points of the image are concentrated with each other, find the matching point in the corresponding search area, calculate the similarity between this point and each candidate matching point in the search area, and select the candidate matching point with the greatest similarity as its match Point, get the matching point set K 2 of the corner point;

Utilize the matching point set K 2 of the corner points, and get
Polar correspondence of images, generating a relationship between the limit set K 3, according to the gradation of the electrode line in the segmented K 3, each of the electrode lines are divided into several segments gradation, the gradation segment set is generated K 4 , use the dynamic programming algorithm to establish the correspondence between the gray-scale segments, and use the linear interpolation method to establish the corresponding relationship between the pixels between the corresponding gray-scale segments, so as to realize all the pixels between the images To obtain the third image matching set.
The image matching method according to any one of claims 1 to 7, wherein the performing three-dimensional reconstruction to obtain the reconstructed scene image comprises:

After dense matching, the corresponding relationship between all pixels in the third image matching set is used to calculate the depth of field of the scene, and the preset 3Dmax software is used to reconstruct the scene to restore the three-dimensional geometric information of the scene space. The constructed scene image.
An image matching device, characterized in that the device includes a memory and a processor, the memory stores an image matching program running on the processor, and the image matching program is executed when the processor is executed The following steps:

According to the scene image taken by the aerial camera, the image imaging map is generated, and the first image matching is performed on the image imaging map using the scale-invariant feature transformation method to generate the first image matching set;

Based on the first image matching set, generating epipolar images and calculating the degree of overlap between the epipolar images, completing the second image matching, and generating a second image matching set;

Based on the second image matching set, a dense matching of all pixels between images is established, a third image matching set is generated, and three-dimensional reconstruction is performed to obtain a reconstructed scene image.
9. The image matching device of claim 9, wherein generating an image imaging map according to a scene image captured by an aerial camera comprises:

According to the parameters recorded when the aerial photography instrument took the scene image, including the low-precision position, attitude information and the approximate height of the survey area, the model formula for object imaging under aerial photography is used to recover the overlapping scene images taken by the aerial photography instrument. Position, generate n image imaging maps, where the model formula of the object imaging is as follows:

sm=KR[I-C]M,

Among them, s is the scale factor, m is the image point coordinates, M is the object point coordinates, K is the parameter matrix in the aerial photography tool, R is the rotation matrix, C is the projection center position vector, and I is the third-order unit matrix.
10. The image matching device of claim 10, wherein the first image matching is performed on the image imaging map using the scale-invariant feature transformation method to generate the first image matching set, comprising:

Converting the image set composed of the n image imaging images into a corresponding undirected image edge set E;

In the undirected graph edge set E, the scale-invariant feature transformation algorithm is used for image matching, and in the image matching, for the image (I i , I j ) ∈ E, if the two images I i and I j match If the number of points is less than the threshold N 1 , then (I i , I j ) will be removed from the set of undirected graph edges. If the number of matching points in the two images I i and I j is greater than the threshold N 1 , the imaging image pair is retained ,generate
Image pairs, generating the first image matching set.
The image matching device of claim 11, wherein based on the first image matching set, an epipolar image is generated and the degree of overlap between the epipolar images is calculated, and the second image matching is completed to generate a second Matching sets of images, including:

(a) Use the scale invariant feature transformation algorithm to analyze the
Perform point feature extraction on the image pair, after obtaining uniformly distributed high-precision points with the same name, use the RANSAC-based strategy to estimate the basic matrix to obtain the basic matrix;

(b) Using the basic matrix to determine the core line with the same name corresponding to each group of points with the same name;

(c) According to the principle that the core line must intersect at the core point, the least square method is used to determine
The core point coordinates of the image pair are used to generate a quick mapping of the epipolar lines between the images according to the core point coordinates, and the epipolar line is resampled by bilinear interpolation along the epipolar line direction to complete the epipolar image production and matching regeneration
Image pairs, generating the second image matching set.
The image matching device of claim 12, wherein the (c) comprises: building a rotation matrix and a projection matrix based on the basic matrix, and dividing the rotation matrix into x, y, and z axes, each for
among them,
Is expressed as follows:

And get the projection matrix

among them,
Is the camera parameter of the camera left,
Are the camera parameters of camera right, t left and t right are the components of the camera parameters of camera left and camera right in the x, y, and z axes respectively;

According to the principle that the core line must intersect at the core point, the least square method is used to determine the core point coordinates (x p , y p ) of the image from the calculation result of the projection matrix;

The central projection is derived according to the collinear condition equation. When the epipolar lines are sampled, the angle relationship between two adjacent epipolar lines is calculated to complete the determination of each epipolar line on the central projection image;

Using the basic matrix generated above, each image pair can be determined
The corresponding core lines are based on
The nuclear point coordinates of each nuclear image in the image determine the epipolar line of the point, and complete the epipolar line correspondence between the same image pair to obtain
The epipolar equation is:

Among them, (x p , y p ) are the coordinates of the core point calculated above, (x base , y base ) are the reference coordinates of the center projection image, and the same goes for
The epipolar equation;

Based on the epipolar equation, each image pair is established
After the epipolar line mapping, according to the resampling rule of the bilinear interpolation method, the epipolar line image is generated and the overlap degree is calculated. The image pairs whose overlap degree of the epipolar line image is less than the threshold N 2 are discarded to generate
Image pair to obtain the second image matching set.
The image matching device of claim 13, wherein the establishing a dense match of all pixels between the images based on the second image matching set comprises:

On the basis of the second image matching set, the corner detection algorithm of the least equivalent segmentation absorbing core is used to extract the
The corner points of the corner points form a set of matching points of the corner points, combined with epipolar geometric constraints and dynamic programming algorithms to establish
Dense matching of all pixels between images.
The image matching device according to any one of claims 9 to 14, wherein the performing three-dimensional reconstruction to obtain the reconstructed scene image comprises:

After dense matching, the corresponding relationship between all pixels in the third image matching set is used to calculate the depth of field of the scene, and the preset 3Dmax software is used to reconstruct the scene to restore the three-dimensional geometric information of the scene space. The constructed scene image.
A computer-readable storage medium, characterized in that an image matching program is stored on the computer-readable storage medium, and when the image matching program is executed by a processor, the following steps are implemented:

According to the scene image taken by the aerial camera, the image imaging map is generated, and the first image matching is performed on the image imaging map using the scale-invariant feature transformation method to generate the first image matching set;

Based on the first image matching set, generating epipolar images and calculating the degree of overlap between the epipolar images, completing the second image matching, and generating a second image matching set;

Based on the second image matching set, a dense matching of all pixels between images is established, a third image matching set is generated, and three-dimensional reconstruction is performed to obtain a reconstructed scene image.
15. The computer-readable storage medium of claim 16, wherein generating an image imaging map according to a scene image shot by an aerial camera comprises:

According to the parameters recorded when the aerial photography instrument took the scene image, including the low-precision position, attitude information and the approximate height of the survey area, the model formula for object imaging under aerial photography is used to recover the overlapping scene images taken by the aerial photography instrument. Position, generate n image imaging maps, where the model formula of the object imaging is as follows:

sm=KR[I-C]M,

Among them, s is the scale factor, m is the image point coordinates, M is the object point coordinates, K is the parameter matrix in the aerial photography tool, R is the rotation matrix, C is the projection center position vector, and I is the third-order unit matrix.
17. The computer-readable storage medium according to claim 17, wherein said using a scale-invariant feature transformation method to perform an initial image matching on an image imaging map to generate an initial image matching set comprises:

Converting the image set composed of the n image imaging images into a corresponding undirected image edge set E;

In the undirected graph edge set E, the scale-invariant feature transformation algorithm is used for image matching, and in the image matching, for the image (I i , I j ) ∈ E, if the two images I i and I j match If the number of points is less than the threshold N 1 , then (I i , I j ) will be removed from the set of undirected graph edges. If the number of matching points in the two images I i and I j is greater than the threshold N 1 , the imaging image pair is retained ,generate
Image pairs, generating the first image matching set.
The computer-readable storage medium of claim 18, wherein based on the first image matching set, an epipolar image is generated and the degree of overlap between the epipolar images is calculated to complete the second image matching to generate The second image matching set, including:

(a) Use the scale invariant feature transformation algorithm to analyze the
Perform point feature extraction on the image pair, after obtaining uniformly distributed high-precision points with the same name, use the RANSAC-based strategy to estimate the basic matrix to obtain the basic matrix;

(b) Using the basic matrix to determine the core line with the same name corresponding to each group of points with the same name;

(c) According to the principle that the core line must intersect at the core point, the least square method is used to determine
The core point coordinates of the image pair are used to generate a quick mapping of the epipolar lines between the images according to the core point coordinates, and the epipolar line is resampled by bilinear interpolation along the epipolar line direction to complete the epipolar image production and matching regeneration
Image pairs, generating the second image matching set.
15. The computer-readable storage medium of claim 15, wherein the performing three-dimensional reconstruction to obtain the reconstructed scene image comprises:

After dense matching, the corresponding relationship between all pixels in the third image matching set is used to calculate the depth of field of the scene, and the preset 3Dmax software is used to reconstruct the scene to restore the three-dimensional geometric information of the scene space. The constructed scene image.