CN114494372B - Remote sensing image registration method based on unsupervised deep learning - Google Patents

Abstract

The invention discloses a remote sensing image registration method based on unsupervised deep learning, which converts image registration into regression optimization, and can integrate a feature extraction network, image similarity measure and feature descriptors of various forms and parameters. According to the invention, the depth characteristics of the images to be registered are extracted on a plurality of scales by using a model network, geometric transformation parameters are obtained through parameter regression, the images are geometrically corrected by using the parameters, and the multi-scale gradual registration of the images from coarse to fine is realized. According to the invention, registration truth values are not needed to be used as training samples, the loss functions on multiple scales are jointly trained by constructing the loss functions based on the similarity measure and the feature descriptors between the images, the parameters of each model network are updated by back propagation, the geometric transformation parameters are optimized, and the high-precision and high-robustness multi-source remote sensing image registration is realized.

Description

A remote sensing image registration method based on unsupervised deep learning

Technical Field

The present invention belongs to the field of remote sensing technology, and in particular relates to the design of a remote sensing image registration method based on unsupervised deep learning.

Background Art

With the rapid development of aerospace and remote sensing technology, the means of acquiring remote sensing images are increasing and the types are becoming more diverse. Due to the differences in equipment technology and imaging mechanisms of various sensors, remote sensing images from a single data source are difficult to fully reflect the characteristics of ground objects. In order to make full use of multi-source remote sensing data obtained by different types of sensors and realize integration and information complementarity, multi-source remote sensing images need to be registered.

Multi-source remote sensing image registration refers to the process of aligning and superimposing information on multi-sensor remote sensing images of the same area acquired at different times, different perspectives or different sensor conditions, so that the same-name points on the aligned images have the same geographic coordinates. In the prior art, the methods for multi-source remote sensing image registration include traditional methods that do not require the use of deep learning technology and methods based on deep learning. Traditional methods are based on features or regional templates, which rely on manually designed features. For remote sensing image registration of different sensors and different modalities, these manual features usually need to be redesigned. Methods based on deep learning extract deep features from multi-source remote sensing images and have better versatility than manual features. At present, methods based on supervised deep learning require a large number of samples with true value labels as training data, but at present, there is no large amount of data in the remote sensing field for training, and cost factors limit the practical application of such methods.

Summary of the invention

The purpose of the present invention is to solve the problem that it is difficult to obtain a large number of training samples in the existing remote sensing image registration method based on supervised deep learning, and propose a remote sensing image registration method based on unsupervised deep learning, which can achieve accurate registration between remote sensing images without training samples.

The technical solution of the present invention is: a remote sensing image registration method based on unsupervised deep learning, comprising the following steps:

S1. Establish a multi-source remote sensing image registration dataset including two sets of image data, where each two images of the two sets of image data correspond to each other one by one, one set of image data is used as a reference image dataset, and the other set of image data is used as an image dataset to be corrected.

S2. Select a reference image f from the reference image data set, select an image m to be corrected corresponding to the reference image f from the image data set to be corrected, and use the reference image f and the image m to be corrected as end-to-end inputs on a training sample.

S3. Calculate the transformation parameters μ ₁ , μ ₂ , and μ ₃ of the image on the model network of each scale at three scales respectively, correct the image m to be corrected step by step, generate corrected images m ₁ , m ₂ , and m ₃ , back-propagate the loss function of the model network of each scale, and use the corrected image m ₃ and the transformation parameter μ ₃ as the end-to-end output on a training sample.

S4. Initialize the model network parameters of the three scales respectively.

S5. Jointly train the model networks of the three scales in an end-to-end manner to optimize the joint loss function at the three scales.

S6. Use a deep learning optimizer to find the direction in which the joint loss function value decreases fastest, perform back propagation on the model network in the direction, iteratively update the model network parameters, and when the joint loss function drops to a preset threshold and converges, save the network model parameters at this time, and output the registered reference image f and the corrected image m ₃ .

Further, step S3 includes the following sub-steps:

S3-1. Input the reference image f and the image to be corrected m into the model network of the first scale to obtain the transformation parameter μ ₁ of the first scale.

S3-2. Use the transformation parameter μ ₁ to perform geometric correction on the image to be corrected m to generate a corrected image m ₁ .

S3-3. Calculate the loss function of the model network of the first scale.

S3-4. Input the reference image f and the corrected image m ₁ into the model network of the second scale to obtain the residual Δμ ₁ of the transformation parameter, and combine it with the transformation parameter μ ₁ to obtain the transformation parameter μ ₂ of the second scale.

S3-5. Use the transformation parameter μ ₂ to perform geometric correction on the correction image m ₁ to generate a correction image m ₂ .

S3-6. Calculate the loss function of the model network of the second scale.

S3-7. Input the reference image f and the corrected image _m2 into the model network of the third scale to obtain the residual _Δμ2 of the transformation parameter, and combine it with the transformation parameter _μ2 to obtain the transformation parameter _μ3 of the third scale.

S3-8. Use the transformation parameter μ ₃ to perform geometric correction on the correction image m ₂ to generate a correction image m ₃ .

S3-9. Calculate the loss function of the model network of the third scale.

S3-10. Take the corrected image m ₃ and the transformation parameter μ ₃ as the end-to-end output on a training sample.

Furthermore, step S3-1 includes the following sub-steps:

S3-1-1. Downsample the reference image f and the image to be corrected m to 1/4 of their original sizes respectively, and superimpose the two images generated after downsampling in the channel direction to generate a superimposed image.

S3-1-2. Input the superimposed image into the feature extraction part of the model network of the first scale to generate deep features.

S3-1-3. Pass the deep features through the parameter regression part of the model network of the first scale to obtain the transformation parameter μ ₁ of the first scale.

Further, step S3-2 includes the following sub-steps:

S3-2-1. The geometric transformation matrix T _μ1 is formed by the transformation parameters μ ₁ .

S3-2-2. Perform geometric transformation on the image to be corrected m using the geometric transformation matrix T _μ1 to generate a corrected image m ₁ .

Further, step S3-4 includes the following sub-steps:

S3-4-1. Downsample the reference image f and the correction image _m1 to 1/2 of their original sizes respectively, and superimpose the two images generated after downsampling in the channel direction to generate a superimposed image.

S3-4-2. Input the superimposed image into the feature extraction part of the model network of the second scale to generate deep features.

S3-4-3. Pass the deep features through the parameter regression part of the model network of the second scale to obtain the residual Δμ ₁ of the transformation parameter.

S3-4-4. Combine the residual Δμ ₁ with the transformation parameter μ ₁ to obtain the transformation parameter μ ₂ of the second scale.

Further, step S3-5 includes the following sub-steps:

S3-5-1. The geometric transformation matrix T _μ2 is formed by the transformation parameters μ ₂ .

S3-5-2. Perform geometric transformation on the correction image m ₁ through the geometric transformation matrix T _μ2 to generate a correction image m ₂ .

Further, step S3-7 includes the following sub-steps:

S3-7-1. Overlay the reference image f and the correction image _m2 in the channel direction to generate an overlay image.

S3-7-2. Input the superimposed image into the feature extraction part of the model network of the third scale to generate deep features.

S3-7-3. Pass the deep features through the parameter regression part of the model network of the third scale to obtain the residual Δμ ₂ of the transformation parameter.

S3-7-4. Combine the residual Δμ ₂ and the transformation parameter μ ₂ to obtain the transformation parameter μ ₃ of the third scale.

Further, step S3-8 includes the following sub-steps:

S3-8-1. The geometric transformation matrix T _μ3 is formed by the transformation parameters μ ₃ .

S3-8-2. Perform geometric transformation on the corrected image m ₂ using the geometric transformation matrix T _μ3 to generate a corrected image m ₃ .

Furthermore, the loss function Loss _sim (f, m, μ ₁ ) of the model network of the first scale in step S3-3 is:

The loss function Loss _sim (f, m ₁ , μ ₂ ) of the model network of the second scale in step S3-6 is:

The loss function Loss _sim (f, m ₂ , μ ₃ ) of the model network of the third scale in step S3-9 is:

The joint loss function Loss in step S5 is:

Loss＝λ ₁ ×Loss _sim (f, m, μ ₁ )+λ ₂ ×Loss _sim (f, m ₁ , μ ₂ )+λ ₃ ×Loss _sim (f, m ₂ , μ ₃ )

Where Sim(·) represents the similarity measure, λ ₁ , λ ₂ , λ ₃ are the weight factors of the loss function of each scale model network.

Further, step S4 includes the following sub-steps:

S4-1. Train the model network of the first scale by minimizing the loss function Loss _sim (f, m, μ ₁ ).

S4-2. Fix the parameters of the model network of the first scale and train the model network of the second scale to minimize the loss function Loss _sim (f, m ₁ , μ ₂ ).

S4-3. Fix the parameters of the model network of the first scale and the model network of the second scale, and train the model network of the third scale to minimize the loss function Loss _sim (f, m ₂ , μ ₃ ).

The beneficial effects of the present invention are:

(1) The present invention transforms image registration into a regression optimization problem, which can integrate feature extraction networks, image similarity measures and feature descriptors of various forms and parameters, and realizes multi-scale image precise registration with fully unsupervised learning and end-to-end mapping.

(2) The present invention uses a model network to extract the depth features of the image to be registered at multiple scales, obtains geometric transformation parameters through parameter regression, and uses the parameters to perform geometric correction on the image, thereby achieving multi-scale step-by-step registration of the image "from coarse to fine".

(3) The present invention does not require the true value of the registration as a training sample. By constructing a loss function based on the similarity measurement between images and the feature descriptor, the loss function at multiple scales is jointly trained to update the parameters of each model network by back propagation, optimize the geometric transformation parameters, and achieve high-precision and high-robust multi-source remote sensing image registration.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is a flow chart of a remote sensing image registration method based on unsupervised deep learning provided by an embodiment of the present invention.

FIG. 2 is a schematic diagram showing a reference image, an image to be corrected, and a corrected image provided by an embodiment of the present invention.

FIG3 is a schematic diagram showing the overall framework of the remote sensing image registration method provided by an embodiment of the present invention.

FIG4 is a schematic diagram showing the structure of a model network 1 provided in an embodiment of the present invention.

FIG5 is a schematic diagram showing a method for calculating a similarity measure of multi-source remote sensing images provided by an embodiment of the present invention.

DETAILED DESCRIPTION

Now, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings. It should be understood that the embodiments shown and described in the accompanying drawings are only exemplary and are intended to explain the principles and spirit of the present invention, rather than to limit the scope of the present invention.

The embodiment of the present invention provides a remote sensing image registration method based on unsupervised deep learning, as shown in FIG1 , comprising the following steps S1 to S6:

S1. Establish a multi-source remote sensing image registration dataset including two sets of image data, where each two images of the two sets of image data correspond to each other one by one, one set of image data is used as a reference image dataset, and the other set of image data is used as an image dataset to be corrected.

In the embodiment of the present invention, the image to be corrected in the image data set to be corrected should be an image that has a certain overlap (greater than or equal to 70% in the embodiment of the present invention) with the ground object information contained in the reference image and has geometric distortion.

In one embodiment of the present invention, the registration of optical images and synthetic aperture radar (SAR) images is taken as an example to further illustrate step S1. As shown in FIG2, the embodiment of the present invention uses image a with a fixed resolution as a reference image, and image b that overlaps with a portion of image a and has geometric distortion as an image to be corrected. After registration and correction by the registration method provided by the present invention, an image c that is aligned pixel by pixel with the overlapping area of image a is obtained. The multi-source remote sensing image data set contains multiple pairs of regional images similar to the above-mentioned image a and image b. It should be understood that other embodiments of the present invention include but are not limited to the registration of multi-source optical images, the registration of optical images and infrared images, the registration of optical images and laser radar (Light Detection and Ranging, LiDAR) intensity and elevation images, and the registration of optical images and raster maps. The registration method provided by the present invention should be within the protection effect of the present invention.

S2. Select a reference image f from the reference image data set, select an image m to be corrected corresponding to the reference image f from the image data set to be corrected, and use the reference image f and the image m to be corrected as end-to-end inputs on a training sample.

S3. Calculate the transformation parameters μ ₁ , μ ₂ , and μ ₃ of the image on the model network of each scale at three scales respectively, correct the image m to be corrected step by step, generate corrected images m ₁ , m ₂ , and m ₃ , back-propagate the loss function of the model network of each scale, and use the corrected image m ₃ and the transformation parameter μ ₃ as the end-to-end output on a training sample.

The embodiment of the present invention adopts a "coarse-to-fine" multi-scale matching strategy, and jointly trains the model networks at three scales in an end-to-end framework to predict the transformation parameters and their residuals, thereby achieving accurate image registration. The end-to-end framework refers to the input of the reference image f and the image to be corrected m in the embodiment of the present invention, and the output of the corrected image m ₃ and the transformation parameter μ ₃ , which constitute an end-to-end mapping relationship.

As shown in FIG3 , step S3 includes the following sub-steps S3-1 to S3-10:

S3-1. Input the reference image f and the image to be corrected m into a model network (referred to as "model network 1" in the embodiment of the present invention) of the first scale (referred to as "scale 1" in the embodiment of the present invention) to obtain the transformation parameter μ ₁ of the first scale.

Step S3-1 includes the following sub-steps S3-1-1 to S3-1-3:

S3-1-1. Downsample the reference image f and the image to be corrected m to 1/4 of their original sizes respectively, and superimpose the two images generated after downsampling in the channel direction to generate a superimposed image.

In the embodiment of the present invention, the size of the reference image f is fixed. If the size of the image to be corrected m is inconsistent with the size of the reference image f, zero padding or cropping is usually used to adjust the size of the image to be corrected m to be consistent with the reference image f.

S3-1-2. Input the superimposed image into the feature extraction part of the model network of the first scale to generate deep features.

As shown in FIG4 , in one embodiment of the present invention, the feature extraction part of the model network 1 is composed of k groups of interconnected convolution blocks and downsampling layers, each convolution block includes a convolution layer, a local response normalization layer and a linear unit activation function layer, and each downsampling layer reduces the image resolution to 1/2 of the original. Experiments show that by reasonably selecting the value of k, the size of the feature map generated by the last convolution block is between [4, 7], and the number of convolution kernel channels of the convolution layer is set to 1/4 of the size of the image to be corrected, which is conducive to the subsequent steps to generate more accurate transformation parameters μ ₁ . In an embodiment of the present invention, if the size of the reference image f and the image to be corrected m is 512×512, the image size after 1/4 downsampling is 128×128, the number of convolution kernel channels of each convolution block is set to 32, and the value of k is set to 5, and the size of the feature map generated by the superimposed image through 5 groups of convolution blocks and downsampling layers is 4.

In another embodiment of the present invention, the feature extraction part of the model network 1 includes but is not limited to a U-net, a fully convolutional neural network (FCN), etc.

S3-1-3. Pass the deep features through the parameter regression part of the model network of the first scale to obtain the transformation parameter μ ₁ of the first scale.

As shown in Figure 4, in one embodiment of the present invention, the parameter regression part of the model network 1 is composed of t parallel connected fully connected layers. The value of t can be set based on the comprehensive calculation speed and the range of image scale transformation, and the present invention does not limit this. Experiments have shown that if the scaling factor is in [0.5, 2], setting 4 parallel fully connected layers has a better effect. The parallel fully connected layers are similar to the pyramid strategy used in traditional image registration, the difference being that the initial values of the output space transformation parameters are different in scale. Compared with using a single fully connected layer output parameter, the calculation of multiple parallel fully connected layers will greatly accelerate the convergence of the loss function.

It should be understood that the present invention does not impose any formal or parametric limitations on the implementation of the feature extraction part and the parameter regression part of the model network 1. Any idea of using an input method of superimposed images and extracting deep features in the channel direction and outputting geometric transformation parameters through convolutional neural networks (CNN) of various forms and parameters is within the protection scope of the present invention.

S3-2. Use the transformation parameter μ ₁ to perform geometric correction on the image to be corrected m to generate a corrected image m ₁ .

Step S3-2 includes the following sub-steps S3-2-1 to S3-2-2:

S3-2-1. The geometric transformation matrix T _μ1 is formed by the transformation parameters μ ₁ .

In one embodiment of the present invention, as shown in FIG4 , six geometric transformation parameters a ₁ , a ₂ , a ₃ , a ₄ , a ₅ , a ₆ are output in step S3 - 1 - 3 , namely, a two-dimensional affine matrix T _μ1 is formed:

在y方向上的错切角为ω，则二维仿射矩阵T_μ1中6个参数由上述操作的任意排列组合得到：The six parameters in the affine transformation matrix represent the translation, rotation, scaling and shearing operations on the image pixel coordinates. Assume that the geometric transformation of the image includes: the translation amount in the x direction is D _x , the translation amount in the y direction is D _x ; the scaling factor in the x direction is S _x , the scaling factor in the y direction is _Sy ; the clockwise rotation angle θ; the shearing angle in the x direction is

The misalignment angle in the y direction is ω, and the six parameters in the two-dimensional affine matrix T _μ1 are obtained by any permutation and combination of the above operations:

In one embodiment of the present invention, the parameter regression part of the model network 1 outputs a greater or lesser number of geometric transformation parameters to form other geometric transformation matrices besides the affine transformation, such as perspective transformation, rigid transformation, etc., which is not limited by the present invention.

S3-2-2. Perform geometric transformation on the image to be corrected m through the geometric transformation matrix T _μ1 to generate a corrected image m ₁ :

m ₁ ＝T _μ1 (m)

Specifically, for each pixel with coordinates (x, y) on the corrected image m, set its gray value to σ, calculate its coordinates (X, Y) on the corrected image after spatial transformation, and generate the corrected image m ₁ according to a certain resampling and interpolation method. In the embodiment of affine transformation, there is:

S3-3. Calculate the loss function Loss _sim (f, m, μ ₁ ) of the model network of the first scale:

表示T_μ1的几何逆变换，其定义为：in,

represents the geometric inverse transformation of T _μ1 , which is defined as:

Sim(·) represents a similarity measure, that is, Sim(A, B) represents a similarity measure between images A and B. Commonly used similarity measure calculation methods include Sum of Squared Difference (SSD), Normalized Cross Correlation (NCC), and Phase Correlation, etc.:

和

分别是影像A和影像B的灰度均值。The size of image A and image B is w×w,

and

are the grayscale means of image A and image B respectively.

Calculating traditional similarity measures (such as SSD or NCC) is time-consuming. Based on the fact that the correlation or convolution of two images in the spatial domain is equal to their product in the frequency domain, phase correlation with faster calculation speed is used. The specific steps are as follows:

Assume that image A and image B have a displacement relationship (x ₀ , y ₀ ) in the spatial domain, that is, B(x, y) = A(xx ₀ , yy ₀ ), which are respectively expressed as _FA (u, v) and _FB (u, v) by Fourier transform. The two have the following relationship in the frequency domain:

F _B (u, v) = F _A (u, v) exp (-i (ux ₀ + vy ₀ ))

The normalized cross-power spectrum of the two is expressed as:

The superscript * indicates complex conjugation.

In one embodiment of the present invention, image A and image B are multi-source optical remote sensing images acquired by the same type of sensor in the same area, and grayscale values are used as input for calculating similarity measures between image A and image B.

In another embodiment of the present invention, image A and image B are remote sensing images acquired in the same area by sensors of different types (such as optical, infrared, SAR, etc.). Grayscale values are not directly used as input for calculating the similarity measure between image A and image B. Instead, local feature descriptors of image A and image B are calculated pixel by pixel, such as Chanel Feature of Orientated Gradient (CFOG), Histogram of Orientated Gradient (HOG), Local Self-similarity Descriptor (LSS), and Histogram of Orientated Phase Congruency (HOPC). As shown in FIG5 , the SSD, NCC or phase correlation between the feature descriptors of the two images is used as the similarity measure.

Steps S3-1 to S3-3 generate transformation parameters and correct images at scale 1, and calculate the loss function. These steps introduce the specific implementation methods of the relevant operations in detail. Subsequent steps (steps S3-4 to S3-9) will repeat similar operations at other scales. The only difference between them and the relevant operations at scale 1 is the parameters. The process will be briefly outlined without repeating the detailed description of the principles.

S3-4, input the reference image f and the corrected image _m1 into the model network of the second scale (referred to as "scale 2" in the embodiment of the present invention) (referred to as "model network 2" in the embodiment of the present invention), obtain the residual _Δμ1 of the transformation parameter, and combine it with the transformation parameter _μ1 to obtain the transformation parameter _μ2 of the second scale.

Step S3-4 includes the following sub-steps S3-4-1 to S3-4-4:

S3-4-1. Downsample the reference image f and the correction image _m1 to 1/2 of their original sizes respectively, and superimpose the two images generated after downsampling in the channel direction to generate a superimposed image.

S3-4-2. Input the superimposed image into the feature extraction part of the model network of the second scale to generate deep features.

In the embodiment of the present invention, the network structure of the model network 2 is similar to the network structure of the model network 1, and only the parameter settings are different. Now, the specific implementation of feature extraction in step S3-4-2 is further described in conjunction with a specific embodiment. If the size of the reference image f and the correction image _m1 is 512×512, the image size after 1/2 downsampling is 256×256, the number of convolution kernel channels of each convolution block is set to 64, and the value of k is set to 6, the feature map size generated by the superimposed image through 6 groups of convolution blocks and downsampling layers is 4.

S3-4-3. Pass the deep features through the parameter regression part of the model network of the second scale to obtain the residual Δμ ₁ of the transformation parameter.

S3-4-4. Combine the residual Δμ ₁ with the transformation parameter μ ₁ to obtain the transformation parameter μ ₂ of the second scale:

μ ₂ =μ ₁ *Δμ ₁

Where * represents matrix multiplication.

S3-5. Use the transformation parameter μ ₂ to perform geometric correction on the correction image m ₁ to generate a correction image m ₂ .

Step S3-5 includes the following sub-steps S3-5-1 to S3-5-2:

S3-5-1. The geometric transformation matrix T _μ2 is formed by the transformation parameters μ ₂ .

S3-5-2. Perform geometric transformation on the correction image m ₁ through the geometric transformation matrix T _μ2 to generate the correction image m ₂ :

m ₂ ＝T _{μ 2} (m ₁ )

S3-6. Calculate the loss function Loss _sim (f, m ₁ , μ ₂ ) of the model network of the second scale:

S3-7, input the reference image f and the corrected image _m2 into the model network of the third scale (referred to as "scale 3" in the embodiment of the present invention) (referred to as "model network 3" in the embodiment of the present invention), obtain the residual _Δμ2 of the transformation parameter, and combine it with the transformation parameter _μ2 to obtain the transformation parameter _μ3 of the third scale.

Step S3-7 includes the following sub-steps S3-7-1 to S3-7-4:

S3-7-1. Overlay the reference image f and the correction image _m2 in the channel direction to generate an overlay image.

S3-7-2. Input the superimposed image into the feature extraction part of the model network of the third scale to generate deep features.

In the embodiment of the present invention, the network structure of the model network 3 is similar to the network structure of the model network 1 and the model network 2, and only the parameter settings are different. Now, the specific implementation of feature extraction in step S3-7-2 is further described in conjunction with a specific embodiment. If the size of the image f and the image m is 512×512, the number of convolution kernel channels of each convolution block is set to 128, and the value of k is set to 7. The feature map size generated by the superimposed image through 7 groups of convolution blocks and downsampling layers is 4.

S3-7-3. Pass the deep features through the parameter regression part of the model network of the third scale to obtain the residual Δμ ₂ of the transformation parameter.

S3-7-4. Combine the residual Δμ ₂ with the transformation parameter μ ₂ to obtain the transformation parameter μ ₃ of the third scale:

μ ₃ =μ ₂ *Δμ ₂

Where * represents matrix multiplication.

S3-8. Use the transformation parameter μ ₃ to perform geometric correction on the correction image m ₂ to generate a correction image m ₃ .

Step S3-8 includes the following sub-steps S3-8-1 to S3-8-2:

S3-8-1. The geometric transformation matrix T _μ3 is formed by the transformation parameters μ ₃ .

S3-8-2. Perform geometric transformation on the correction image m ₂ through the geometric transformation matrix T _μ3 to generate the correction image m ₃ :

m ₃ =T _μ3 (m ₂ )

S3-9. Calculate the loss function Loss _sim (f, m ₂ , μ ₃ ) of the model network of the third scale:

S3-10. Take the corrected image m ₃ and the transformation parameter μ ₃ as the end-to-end output on a training sample.

S4. Initialize the model network parameters of the three scales respectively.

Step S4 includes the following sub-steps S4-1 to S4-3:

S4-1. Train the model network of the first scale by minimizing the loss function Loss _sim (f, m, μ ₁ ).

S4-2. Fix the parameters of the model network of the first scale and train the model network of the second scale to minimize the loss function Loss _sim (f, m ₁ , μ ₂ ).

S4-3. Fix the parameters of the model network of the first scale and the model network of the second scale, and train the model network of the third scale to minimize the loss function Loss _sim (f, m ₂ , μ ₃ ).

S5. Jointly train the model networks of the three scales in an end-to-end manner to optimize the joint loss function at the three scales.

In the embodiment of the present invention, before jointly training the model networks of three scales, it is necessary to unfix the parameters of all the model networks.

In the embodiment of the present invention, the joint loss function Loss is:

Loss＝λ ₁ ×Loss _sim (f, m, μ ₁ )+λ ₂ ×Loss _sim (f, m ₁ , μ ₂ )+λ ₃ ×Loss _sim (f, m ₂ , μ ₃ )

Wherein λ ₁ , λ ₂ , and λ ₃ are weight factors of the loss function of each scale model network. In the embodiment of the present invention, the values of λ ₁ , λ ₂ , and λ ₃ are 0.05, 0.05, and 0.9, respectively.

S6. Use a deep learning optimizer to find the direction in which the joint loss function value decreases fastest, perform backpropagation on the model network in the direction, iteratively update the model network parameters, and when the joint loss function drops to a preset threshold and converges, all model networks of end-to-end mapping have the overall optimal parameters, and the reference image f and the corrected image m ₃ have the best similarity. Save the network model parameters at this time, and output the aligned reference image f and corrected image m ₃ .

Therefore, the present invention realizes the precise registration of multi-scale remote sensing images with completely unsupervised learning and end-to-end mapping.

Those skilled in the art will appreciate that the embodiments described herein are intended to help readers understand the principles of the present invention, and should be understood that the protection scope of the present invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific variations and combinations that do not deviate from the essence of the present invention based on the technical revelations disclosed by the present invention, and these variations and combinations are still within the protection scope of the present invention.

Claims (2)

1. A remote sensing image registration method based on unsupervised deep learning, characterized in that it comprises the following steps: S1. Establish a multi-source remote sensing image registration dataset including two sets of image data, wherein each two images of the two sets of image data correspond to each other one by one, wherein one set of image data is used as a reference image dataset, and the other set of image data is used as an image dataset to be corrected; S2, select a reference image f from the reference image data set, select an image m to be corrected corresponding to the reference image f from the image data set to be corrected, and use the reference image f and the image m to be corrected as end-to-end input on a training sample; S3, respectively calculating the transformation parameters μ ₁ , μ ₂ , μ ₃ of the image on the model network of each scale at three scales, gradually correcting the image m to be corrected, generating corrected images m ₁ , m ₂ , m ₃ , back-propagating the loss function of the model network of each scale, and using the corrected image m ₃ and the transformation parameter μ ₃ as the end-to-end output on a training sample; S4, initialize the model network parameters of three scales respectively; S5. Jointly train the model networks of the three scales in an end-to-end manner to optimize the joint loss function of the three scales. S6. Use a deep learning optimizer to find the direction in which the joint loss function value decreases fastest, perform back propagation on the model network in the direction, iteratively update the model network parameters, and when the joint loss function drops to a preset threshold and converges, save the network model parameters at this time, and output the registered reference image f and the corrected image m ₃ ; The step S3 comprises the following sub-steps: S3-1, input the reference image f and the image to be corrected m into the model network of the first scale to obtain the transformation parameter μ ₁ of the first scale; S3-2, using the transformation parameter μ ₁ to perform geometric correction on the image to be corrected m, to generate a corrected image m ₁ ; S3-3. Calculate the loss function of the model network of the first scale; S3-4, input the reference image f and the correction image m ₁ into the model network of the second scale to obtain the residual Δμ ₁ of the transformation parameter, and combine it with the transformation parameter μ ₁ to obtain the transformation parameter μ ₂ of the second scale; S3-5, using the transformation parameter μ ₂ to perform geometric correction on the correction image m ₁ to generate a correction image m ₂ ; S3-6. Calculate the loss function of the model network of the second scale; S3-7, input the reference image f and the correction image m ₂ into the model network of the third scale, obtain the residual Δμ ₂ of the transformation parameter, and combine it with the transformation parameter μ ₂ to obtain the transformation parameter μ ₃ of the third scale; S3-8, using the transformation parameter μ ₃ to perform geometric correction on the correction image m ₂ to generate a correction image m ₃ ; S3-9, calculate the loss function of the model network of the third scale; S3-10, using the corrected image m ₃ and the transformation parameter μ ₃ as the end-to-end output on a training sample; The step S3-1 includes the following sub-steps: S3-1-1, down-sampling the reference image f and the image to be corrected m to 1/4 of the original size respectively, and superimposing the two images generated after downsampling in the channel direction to generate a superimposed image; S3-1-2, inputting the superimposed image into the feature extraction part of the model network of the first scale to generate deep features; S3-1-3, passing the deep features through the parameter regression part of the model network of the first scale to obtain the transformation parameter μ ₁ of the first scale; The step S3-2 includes the following sub-steps: S3-2-1, forming a geometric transformation matrix T _μ1 from transformation parameters μ ₁ ; S3-2-2, geometrically transform the image to be corrected m by using the geometric transformation matrix T _μ1 to generate a corrected image m ₁ ; The step S3-4 includes the following sub-steps: S3-4-1, down-sampling the reference image f and the correction image _m1 to 1/2 of the original size respectively, and superimposing the two images generated after downsampling in the channel direction to generate a superimposed image; S3-4-2, inputting the superimposed image into the feature extraction part of the model network of the second scale to generate deep features; S3-4-3, passing the deep features through the parameter regression part of the model network of the second scale to obtain the residual Δμ ₁ of the transformation parameter; S3-4-4, combining the residual Δμ ₁ with the transformation parameter μ ₁ to obtain the transformation parameter μ ₂ of the second scale; The step S3-5 comprises the following sub-steps: S3-5-1, forming a geometric transformation matrix T _μ2 from transformation parameters μ ₂ ; S3-5-2, geometrically transform the correction image m ₁ by using the geometric transformation matrix T _μ2 to generate a correction image m ₂ ; The step S3-7 includes the following sub-steps: S3-7-1, superimposing the reference image f and the correction image _m2 in the channel direction to generate a superimposed image; S3-7-2, inputting the superimposed image into the feature extraction part of the model network of the third scale to generate deep features; S3-7-3, passing the deep features through the parameter regression part of the model network of the third scale to obtain the residual Δμ ₂ of the transformation parameter; S3-7-4, combining the residual Δμ ₂ with the transformation parameter μ ₂ to obtain the transformation parameter μ ₃ of the third scale; The step S3-8 comprises the following sub-steps: S3-8-1, forming a geometric transformation matrix T _μ3 from transformation parameters μ ₃ ; S3-8-2, geometrically transform the correction image m ₂ by using the geometric transformation matrix T _μ3 to generate a correction image m ₃ ; The loss function Loss _sim (f, m, μ ₁ ) of the model network of the first scale in step S3-3 is:

The loss function Loss _sim (f, m ₁ , μ ₂ ) of the model network of the second scale in step S3-6 is:

The loss function Loss _sim (f, m ₂ , μ ₃ ) of the model network of the third scale in step S3-9 is:

The joint loss function Loss in step S5 is: Loss＝λ ₁ ×Loss _sim (f,m,μ ₁ )+λ ₂ ×Loss _sim (f,m ₁ ,μ ₂ )+λ ₃ ×Loss _sim (f,m ₂ ,μ ₃ ) Where Sim(·) represents the similarity measure, and λ ₁ ,λ ₂ ,λ ₃ are the weight factors of the loss function of each scale model network. 2. The remote sensing image registration method according to claim 1, characterized in that step S4 comprises the following sub-steps: S4-1, training the model network of the first scale by minimizing the loss function Loss _sim (f,m,μ ₁ ); S4-2, fixing the parameters of the model network of the first scale, and training the model network of the second scale to minimize the loss function Loss _sim (f,m ₁ ,μ ₂ ); S4-3. Fix the parameters of the model network of the first scale and the model network of the second scale, and train the model network of the third scale to minimize the loss function loss _sim (f,m ₂ ,μ ₃ ).

Images