CN114219995A - Spatial target heterogeneous image matching method based on image completion - Google Patents

Spatial target heterogeneous image matching method based on image completion Download PDF

Info

Publication number
CN114219995A
CN114219995A CN202111544857.6A CN202111544857A CN114219995A CN 114219995 A CN114219995 A CN 114219995A CN 202111544857 A CN202111544857 A CN 202111544857A CN 114219995 A CN114219995 A CN 114219995A
Authority
CN
China
Prior art keywords
image
isar
target
matching
completion
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202111544857.6A
Other languages
Chinese (zh)
Inventor
张云
袁浩轩
冀振元
李宏博
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Harbin Institute of Technology
Original Assignee
Harbin Institute of Technology
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Harbin Institute of Technology filed Critical Harbin Institute of Technology
Priority to CN202111544857.6A priority Critical patent/CN114219995A/en
Publication of CN114219995A publication Critical patent/CN114219995A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/22—Matching criteria, e.g. proximity measures

Landscapes

  • Engineering & Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • Theoretical Computer Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Artificial Intelligence (AREA)
  • Evolutionary Biology (AREA)
  • Evolutionary Computation (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Image Processing (AREA)

Abstract

A spatial target heterogeneous image matching method based on image completion belongs to the technical field of ISAR image and optical image processing. The method aims to solve the problem that the matching precision is low in the existing image matching method under the condition that ISAR pixel points are missing. Firstly, simulating to obtain an ISAR image sequence under the condition of target spinning based on a three-dimensional space target model and a range-Doppler algorithm; obtaining an optical sample of the space target in the same posture as the ISAR image by using a projection method; then, filling missing pixels of the ISAR image corresponding to the space target by a HaLRTC method based on the image sequence and the optimal weight; and finally, matching the space target ISAR image with the optical image based on a pyramid resampling and optimization iteration method. The invention is mainly used for matching the ISAR image and the optical image.

Description

Spatial target heterogeneous image matching method based on image completion
Technical Field
The invention relates to a space target heterogeneous image matching method, and belongs to the technical field of ISAR image and optical image processing.
Background
The heterogeneous visual image matching technology is a key technology widely applied to visual navigation, pattern recognition, guidance and terrain measurement of airplanes, remote sensing satellites, missiles and the like, and sensors with different imaging mechanisms are adopted in the systems. By heterogeneous image matching is meant a technique for matching images from different imaging sensors. The images are images of the same scene or object and target formed by different imaging sensors under different conditions such as different lighting environments and different imaging mechanisms, and mainly comprise image types such as visible light images, infrared images and radar images (ISAR images). Due to the difference of structures, imaging principles and the like of different types of sensors, the gray scale and the contrast of corresponding areas on a heterogeneous image have larger difference. Therefore, heterogeneous image matching is a very difficult task.
For an ISAR image of a space target, due to the influence of space electromagnetic interference, ISAR echo data often face the problem of missing, pixel points of the presented ISAR image are missing, and therefore, the completion of the image pixel points is firstly carried out before the matching of heterogeneous images. At present, algorithms for directly matching heterogeneous images are introduced in many domestic and foreign documents, and due to the fact that ISAR pixel points are lost, even the lost points may exist in key matching positions, for example, satellite images may exist on solar wings, matching accuracy is low when the images are matched, and the technical problem to be solved by the invention is solved.
Disclosure of Invention
The invention aims to solve the problem of low matching precision of the existing image matching method under the condition of ISAR pixel point missing.
The image completion-based space target heterogeneous image matching method comprises the following steps of:
step 1, simulating and obtaining an ISAR image sequence under the condition of target spinning based on a three-dimensional space target model and a range-Doppler algorithm; obtaining an optical sample of the space target in the same posture as the ISAR image by using a projection method;
step 2, complementing the missing pixels of the ISAR image corresponding to the space target by a HaLRTC method based on the image sequence and the optimal weight;
and 3, matching the space target ISAR image with the optical image based on the pyramid resampling and optimization iteration method.
Further, the process of completing the missing pixels of the spatial target ISAR image by the HaLRTC method based on the image sequence and the optimal weight in step 2 includes the following steps:
2.1, constructing an ISAR image sequence containing the space target, taking the ISAR image containing the space target as a tensor X, and dividing the tensor into three input channel tensors Xi,i＝1,2,3，XiThe size is nxnxnxn;
2.2, constructing an ISAR image missing pixel sequence by using a missing model, wherein each input channel tensor has a missing model SiI is 1,2,3, and the ISAR image missing pixel sequence after the processing of the missing model is expressed as tensor
Figure BDA0003415557450000021
The size is nxnxnxn;
2.3, image completion is carried out by using a HaLRTC method based on optimization weight:
2.3.1, aiming at K ISAR image sequences, constructing ISAR image missing pixel sequences
Figure BDA0003415557450000022
2.3.2, calculating loss function and updating weight:
Figure BDA0003415557450000023
Figure BDA0003415557450000024
Figure BDA0003415557450000025
where, k represents the iteration to the k-th round,
Figure BDA0003415557450000026
is the weight corresponding to the ith channel of the jth image sequence in the (k + 1) th round,
Figure BDA0003415557450000027
representing a loss function; lr is the update rate;
recording the current weight at each iteration
Figure BDA0003415557450000028
The two largest and smallest weight values in each channel are dropped and the optimized weight is found using the following equation:
Figure BDA0003415557450000029
when showing loss function
Figure BDA00034155574500000210
Less than a threshold value LgateStopping iteration when the iteration is stopped, or stopping iteration when the maximum iteration times are reached, substituting the obtained optimized weight into a HaLRTC method to perform image completion, and obtaining the tensor data of each channel after completion
Figure BDA00034155574500000211
And combining the data of each channel in sequence to obtain the supplemented ISAR image data.
Further, the process of matching the spatial target ISAR image and the optical image by the pyramid resampling and optimization iteration-based method described in step 3 includes the following steps:
3.1, filtering the supplemented ISAR image to obtain a filtered ISAR image W;
then converting the optical image and the ISAR image into a gray scale image;
3.2 setting a pre-parameter wcCarrying out affine transformation such as translation, rotation and scaling on the optical image; the optical image after affine transformation is
Figure BDA00034155574500000212
The affine transformation matrix is S', the pre-parameter wcIs a parameter set at the beginning of the affine transformation matrix;
3.3 Filter image W corresponding to ISAR image converted into Gray-level image and affine-transformed optical image
Figure BDA0003415557450000031
Respectively constructing Laplacian pyramid residual images;
3.4, optimizing the matching result by using a gradient descent optimizer:
Figure BDA0003415557450000032
wherein, woldRepresenting a pre-parameter wcParameter before update, wnewRepresenting a pre-parameter wcThe updated parameters are used to determine the parameters of the system,
Figure BDA0003415557450000033
is the gradient of the loss value to the parameter, which determines the direction of updating the parameter, η is the learning rate; loss value loss (-) for filtered image W and optical image
Figure BDA0003415557450000034
The mutual information MI (A, B), A, B of the corresponding Laplacian pyramid residual images are the filtered image W and the optical image respectively
Figure BDA0003415557450000035
A corresponding Laplacian pyramid residual image;
when the mutual information MI (A, B) is smaller than a mutual information threshold gamma, continuing iteration until the mutual information threshold gamma is reached;
3.5, non-rigid body matching is carried out by utilizing a differential homoembryo method.
Further, the filtered image W and the optical image
Figure BDA0003415557450000036
The mutual information MI (A, B) of the corresponding Laplacian pyramid residual image is as follows
MI(A,B)＝H(A)+H(B)-H(AB) (8)
Wherein
Figure BDA0003415557450000037
Figure BDA0003415557450000038
Figure BDA0003415557450000039
Figure BDA00034155574500000310
Figure BDA00034155574500000311
Figure BDA00034155574500000312
Wherein, A and B are respectively a filter image W and an optical image
Figure BDA00034155574500000313
A corresponding Laplacian pyramid residual image; a and b are pixel points corresponding to the images respectively; h (a, b) is a joint gray histogram between images.
Further, the process of constructing the laplacian pyramid residual image described in step 3.3 includes the following steps:
firstly, constructing a Gaussian pyramid:
Figure BDA00034155574500000314
wherein G isl-1(. h) represents a layer l-1 pyramid image; l represents the layer of the image, q, e are two variables; w (q, e) is a Gaussian template;
and then, carrying out reverse operation, and expanding a certain level of image in the Gaussian pyramid into the size of the previous level of image through interpolation operation, wherein the size is expressed as follows:
Figure BDA0003415557450000041
subtracting the original image of the previous layer from the interpolated result of the image of the previous layer to obtain the image of the previous layer, namely a residual image Gl+1(i, j) is the expression of the l +1 layer, and the Laplacian pyramid residual image is obtained by subtracting the image of the previous layer.
Further, before filtering the supplemented ISAR image and before converting the optical image into a grayscale image, the ISAR image needs to be filtered and the optical image needs to be scaled according to the markers in the image, so that the spatial proportion difference of the heterogeneous images does not exceed 10%.
Furthermore, the process of filtering the supplemented ISAR image is realized by adopting enhanced Lee filtering,
Figure BDA0003415557450000042
wherein I denotes an image pixel within the sliding window,
Figure BDA0003415557450000043
is the mean of the pixels within the sliding window, W is the estimate of the image, i.e. the filtered image; cITo representLocal standard deviation coefficient of image, Cmin＝Cu,
Figure BDA0003415557450000044
L represents the equivalent view, u represents the noise, CuIs the noise local standard deviation coefficient; p is the weighting coefficient of the Lee filter.
Further, the process of constructing an ISAR image sequence containing a spatial target according to step 2.1 includes the following steps:
selecting continuous N ISAR image sequences, intercepting a square area with the size of nxn at the center of an image to ensure that the area can completely contain a space target, and forming an input data tensor X after interception, wherein the size of the input data tensor X is nxnxnxnxnx3XN, and 3 is three color channels representing the ISAR images.
Further, the process of obtaining the ISAR image sequence under the target spinning condition based on the three-dimensional space target model and the range-Doppler algorithm simulation comprises the following steps:
1.1, opening a 3D model file of a space object, and storing a file in a parasolid format and an mdl file which are convenient to import for FEKO software;
1.2, opening an x _ t file in FEKO software, setting a solving algorithm, a target material, grid quality and an electromagnetic wave carrier frequency parameter, and storing the three-dimensional grid of the target as a nas frame file;
1.3, after the frame file is saved as a text file, processing character strings in the text file, exporting grid point coordinates of scattering point clusters to MATLAB, uniformly selecting N point coordinates from an output result, and completely displaying a target on an ISAR image so as to obtain the distribution of discrete scattering points of the model;
1.4, ISAR imaging is carried out on the scattering points with known distribution by using a range-Doppler algorithm, and an ISAR image sequence under the condition of target spinning is simulated.
Further, the construction process of the deletion model comprises the following steps:
assuming that the ISAR image X pixel size is m × m × 3, some elements thereof are missing, and the observed index set is (i, j, k) ∈ Ω, where i, j, kIndexes of ISAR image pixels on three channels are respectively, omega is an observation set, and missing pixels are not in the observation set; let the deficiency model S satisfy Sij1, (i, j, k) e Ω, otherwise
Figure BDA0003415557450000052
The ISAR image after missing pixels can be represented as
Figure BDA0003415557450000051
Has the advantages that:
the method is different from the traditional heterogeneous image matching method, utilizes a two-step strategy, firstly focuses on the problem that the ISAR image has pixel point missing in the imaging process due to a part of low matching accuracy of the ISAR image and the optical image, and completes image pixel points aiming at the problem, wherein an innovative HaLRTC (high acquisition low rank extender complete) method based on optimization weight is used; and secondly, heterogenous matching of the images after completion is carried out by using a pyramid resampling and optimization iteration method, and the accuracy of the optimization iteration method is greatly improved compared with that of the traditional method for extracting feature points and carrying out registration.
The method comprises three parts, namely matching of a spatial target ISAR with an optical sample, a HaLRTC method full image based on a sequence and an optimized weight, and a heterogeneous image based on pyramid resampling and optimized iteration; the space target ISAR and the optical sample part are responsible for simulating and generating a large number of ISAR images of the space target in a spinning state and optical image sequences in the same posture; the HaLRTC method completion image part based on the sequence and the optimized weight obtains the distribution of pixels by inputting an ISAR image sequence sample of a certain channel, then obtains the optimal weight by using an iterative method, and substitutes the optimal weight into the original algorithm to realize image completion; and the final matching part firstly performs rotation, translation and scale scaling on the optical image to ensure the mismatching of the image, then matches the supplemented ISAR image with the corresponding optical image by using a pyramid resampling and optimization iteration based method, and finally optimizes the result by using a differential homoembryo non-rigid body matching method.
The heterogeneous image matching method adopting the two-step walking strategy can match ISAR images with pixel point missing in actual conditions with optical images, and the trained HaLRTC method for optimizing the weight can obtain the optimal weight which enables the algorithm to be fast converged and has good completion effect, so that the precision of subsequent matching results is improved, the method meets the actual requirements, and the implementation is convenient.
According to the invention, scattering point information required by ISAR imaging is obtained through the three-dimensional model of the space target and electromagnetic parameter simulation, the ISAR imaging result is approximately consistent with the actually measured ISAR image, and ISAR simulation image sequences and optical image sequences of five types of targets are obtained, so that the requirement of the method on samples is ensured, and the smooth application of the algorithm is ensured.
The invention innovatively provides a HaLRTC method based on optimized weight to complete ISAR images in pixel, and a pyramid resampling and optimized iteration method to match heterogeneous images, inputs ISAR space target image sequences and optical image sequences with missing pixel points, and outputs an affine transformation matrix from the optical images to the ISAR images, thereby having certain method superiority.
The method utilizes the ISAR imaging technology and the digital image processing technology to better meet the requirements of different-source image matching in different scenes, and the method keeps certain stability on the visual angle change, affine transformation and noise of the input image.
Drawings
FIG. 1 is an overall flow diagram of a spatial target heterogeneous image matching method based on image completion;
FIG. 2 is a schematic flow chart of image completion;
FIG. 3 is a flow chart of image matching based on a pyramid resampling and optimization iteration method;
FIG. 4 is a flow chart of a simulated ISAR image and optical image sequence acquisition;
fig. 5(a) is an ISAR original, fig. 5(b) is an image of a missing pixel, and fig. 5(c) is a supplemented image;
fig. 6(a) is a registered optical image of the spatial target and ISAR image, and fig. 6(b) is a registered optical image of the spatial target and ISAR.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
It should be noted that the embodiments and features of the embodiments may be combined with each other without conflict. The invention is further described with reference to the following drawings and specific examples, which are not intended to be limiting.
The first embodiment is as follows: the present embodiment is described with reference to figures 1 to 4,
the method for matching the space target heterogeneous images based on image completion comprises the following steps:
step 1, simulating based on a three-dimensional space target model and a range-Doppler algorithm to obtain an ISAR spin space target sequence sample; obtaining an optical sample of the space target in the same posture as the ISAR image by using a projection method;
step 2, complementing the missing pixels of the space target ISAR image by a HaLRTC method based on the image sequence and the optimal weight;
step 3, matching the ISAR image and the optical image of the space target based on a pyramid resampling and optimization iteration method;
the specific process of the step 1 comprises the following steps:
1.1, opening a 3D model file of a space target in SOLIDWORKS software, and storing a file in a parasolid format and an mdl file which are convenient to import for FEKO software;
the 3D model in this embodiment is obtained by inverting the target by 3Dmax, but an off-the-shelf model downloaded over the internet may be used,
1.2, opening an x _ t file in FEKO software, setting parameters such as a solving algorithm, a target material, grid quality, electromagnetic wave carrier frequency and the like, and storing the three-dimensional grid of the target as a nas frame file;
1.3, storing the frame file as a text file, processing a character string in the text file, exporting coordinates of a scattering point cluster grid point to MATLAB, uniformly selecting N point coordinates from an output result, and multiplying the N point coordinates by a coefficient to enable a complete space target to appear in an ISAR image, thereby obtaining the distribution of discrete scattering points of the model;
the specific values of the coefficients depend on the electromagnetic wave frequencies used for the simulation.
1.4, ISAR imaging is carried out on scattering points with known distribution by using a range-Doppler algorithm, and an ISAR image sequence under the condition of target spinning, namely an ISAR spinning space target sequence, is simulated; and obtaining an optical image O with the same posture as the ISAR image by using a projection method based on the mdl model in the step 1.1.
Before step 2, a deletion model is required to be constructed, and the process of constructing the deletion model comprises the following steps:
assuming that the size of an X pixel of an ISAR image is m × m × 3, some elements of the image are missing, and an observed index set is (i, j, k) epsilon Ω, where i, j, k are indexes of the image pixel on three channels, Ω is an observed set, and the other in the set is a missing pixel; let the deficiency model S satisfy Sij1, (i, j, k) e Ω, otherwise
Figure BDA0003415557450000072
The ISAR image after missing pixels can be represented as
Figure BDA0003415557450000071
sijThe missing model is an element of the missing model, the missing model is equivalent to a template and consists of 1 and 0, and each channel data of the image is multiplied by different templates respectively, so that the missing pixel result is obtained.
Step 2, the process of complementing the missing pixels of the spatial target ISAR image by the HaLRTC method based on the image sequence and the optimal weight comprises the following steps:
2.1, because the principle of using a HaLRTC method to perform ISAR image pixel completion lies in that the correlation of sample pixels among similar ISAR images is utilized, an ISAR image sequence is established first, and the ISAR image sequence is constructed:
selecting N continuous ISAR image sequences, wherein the size of N is smaller than the maximum value NmaxEnsuring that the target rotation angle and the attitude change in the sequence are within a certain range; intercepting a square area with the size of nxn at the center of the image to ensure that the area can completely contain a space target, forming an input data tensor X after interception, wherein the size of the input data tensor X is nxnxnxnxnx3XN, 3 is three color channels representing an ISAR image, and dividing the input data tensor into three input channel tensors Xi,i＝1,2,3，XiThe size is nxnxnxn;
in fact, the image sequence simulated in step 1 is an overall data set, in this embodiment, 200 targets are provided for each type of image sequence simulated in step 1, and 10 targets may be provided for each type of sequence obtained in step 2.1, so that in order to perform image completion by using the HaLRTC method, target images exist in a continuously changing sequence manner, and essentially, pixel relationships between the images are used for completion, which can be understood as the relationship between the total rainingset and the trained batch.
2.2, constructing an ISAR image missing pixel sequence:
simulating data missing condition in the actual ISAR imaging process, constructing ISAR image missing pixel sequence by using a missing model, and providing a missing model S for each input channel tensoriI is 1,2,3, the ISAR image missing pixel sequence after the processing of the missing model can be expressed as tensor
Figure BDA0003415557450000088
The size is nxnxnxn;
2.3, utilizing a HaLRTC method based on optimized weight to perform image completion:
because the convergence rate of the HaLRTC method for image completion depends on the weight rho selected by the HaLRTC method, the invention provides the HaLRTC method based on the optimized weight.
2.3.1, firstly selecting K ISAR image sequences, and constructing ISAR image missing pixel sequences according to the steps 2.1-2.2
Figure BDA0003415557450000081
2.3.2, setting a hyper-parameter in HaLRTC;
the hyper-parameters include:
for the HaLRTC method based on the optimized weight, the sequence number of ISAR images is selected to be 10, the training sample number K is selected to be 20, the maximum iteration number maximum is 1000, and the initial weight rho is selectedinit0.01, the update rate lr is 0.0001;
for the gradient descent optimizer, the number of spatial samples is 500, the number of histogram bars is 50, the growth factor is 1.05, the initial search radius is 0.000625, the maximum iteration number is 200, and the pyramid is set to be 3 layers;
for the differential homoembryo non-rigid body matching method, the pyramid iteration number of each layer is set as 100, the smoothness is set as 1.5, and the pyramid is set as 7 layers;
the update and loss functions of the weights are set using the following equations:
Figure BDA0003415557450000082
Figure BDA0003415557450000083
Figure BDA0003415557450000084
where, k represents the iteration to the k-th round,
Figure BDA0003415557450000085
the weight corresponding to the ith channel of the jth image sequence in the (k + 1) th round is obtained, and L represents a loss function; lr is the update rate.
Recording the current weight at each iteration
Figure BDA0003415557450000086
The two largest and smallest weight values in each channel are dropped and the optimized weight is found using the following equation:
Figure BDA0003415557450000087
according to the loss function in equation (1), when it is less than the threshold value LgateStopping iteration when the iteration is stopped, or stopping iteration when the maximum iteration times are reached, substituting the obtained optimized weight into a HaLRTC method to perform image completion, and obtaining the tensor data of each channel after completion
Figure BDA0003415557450000091
Sequentially combining the data of each channel to obtain supplemented ISAR image data;
in the embodiment, the pixel sizes of the simulated space target ISAR image and the simulated optical image are 256 × 256 × 3; the size of a sequence tensor obtained by constructing an ISAR image sequence is 256 multiplied by 0256 multiplied by 13 multiplied by 2200; the size of the intercepted input data tensor is 128 multiplied by 3 multiplied by 200; for the optimal weight-based HaLRTC method, each input channel tensor is 128 multiplied by 10, 20 multiplied by 3 groups of input channel tensors are shared, the tensor size processed by the missing model is 128 multiplied by 10, 20 ISAR missing pixel sequences are selected for each channel to train, and the optimal weight of each channel is obtained through 1000 iterations by the method in the step 2.3.2
Figure BDA0003415557450000092
Substituting it into HaLRTC method and setting threshold LgateThe size of the image is 1000, iteration is stopped when the maximum iteration number or threshold is reached, the size of each channel of each image is 128 multiplied by 128, and the three channels are combined to obtain a supplemented image with the size of 128 multiplied by 3.
The process of matching the ISAR image and the optical image of the space target by the pyramid resampling and optimization iteration-based method in the step 3 comprises the steps of preprocessing the image, affine transformation of the optical image, resampling of the pyramid, adjusting variation parameters by an optimizer and calculating mutual information;
3.1, preprocessing of images:
because the method is not suitable for the registration of the heterogeneous images with overlarge space proportion difference, the images are preprocessed firstly, and the images are zoomed according to the markers in the images, so that the space proportion difference of the heterogeneous images is not more than 10 percent; due to the problem of an imaging mechanism, a large amount of speckle noise exists in the ISAR image, and has negative influence on the registration of the heterogeneous images, the supplemented ISAR image is filtered by adopting a method for enhancing Lee filtering, the speckle noise of the ISAR image is removed, and the Lee filtering in the embodiment selects a 3 x 3 sliding window; converting the optical image and the ISAR image into a gray scale image;
the enhanced Lee filtering is performed using the following equation:
Figure BDA0003415557450000093
wherein I denotes an image pixel within the sliding window,
Figure BDA0003415557450000094
is the mean of the pixels within the sliding window, W is the estimate of the image, i.e. the filtered image; cIRepresenting the local standard deviation coefficient, C, of the imagemin＝Cu,
Figure BDA0003415557450000095
L represents the equivalent view, u represents the noise, CuIs the noise local standard deviation coefficient; p is the weighting coefficient of the Lee filter.
Specifically, the algorithm divides the image area into three types, and the processing modes of each type are different:
1) when C is presentI＜Cu(u represents noise) represents a uniform region, and the average value of pixels in a sliding window is used as the average valueAs a center value (window center value);
2) when C is presentu＜CI＜CmaxWhen a weak texture area is represented, a traditional Lee filter is adopted to process the weak texture area;
3) when C is presentI＞CmaxAnd representing a non-uniform area, wherein speckle noise is not fully developed, other processing is not carried out, and the original value is directly reserved.
3.2, setting a pre-parameter w at first due to the difference of the remote sensing images on the scalecPerforming affine transformation such as translation, rotation and scaling on the optical image, and assuming that an affine transformation matrix is S', the expression of which is as follows:
Figure BDA0003415557450000101
a pre-parameter wcThe parameters set at the beginning of the affine transformation matrix include a rotation angle theta, a scaling ratio s, and a translation amount txAnd ty，tx、tyRespectively are translation amounts in two directions in two-dimensional translation;
the optical image is O, the affine-transformed optical image is
Figure BDA0003415557450000102
3.3 Filter image W corresponding to ISAR image converted into Gray-level image and affine-transformed optical image
Figure BDA0003415557450000106
The Laplacian pyramid residual images are respectively constructed, and due to the fact that the difference between the pixels of the heterogeneous images is large, the matching capacity of the images after Gaussian pyramid down-sampling is not strong, the Laplacian pyramid residual images are adopted, and a Gaussian pyramid is constructed by using a formula (4):
Figure BDA0003415557450000103
wherein G isl-1(. h) represents a layer l-1 pyramid image; l represents the layer of the image, q, e are two variables; w (q, e) is a Gaussian template, and the formula represents that the Gaussian template is multiplied by the source image and then added, and then interval sampling is carried out;
then, by using the inverse operation of the above equation, a certain level of image in the gaussian pyramid is expanded to the size of the previous level of image by interpolation operation, which is generally interpolation of 0 point, and is expressed as follows:
Figure BDA0003415557450000104
and performing interpolation and subtraction for many times to obtain a Laplacian pyramid, namely a Laplacian pyramid residual image. The Laplacian pyramid residual image is actually obtained by subtracting the original image of the previous layer from the result of the interpolation of the image of the previous layer, namely the residual image, Gl+1(i, j) is not the final result of the residual image, and is an expression of the l +1 layer, which subtracts the image of the previous layer to obtain the residual image.
3.4, optimizing the matching result by using a gradient descent optimizer, wherein the algorithm can be expressed as:
Figure BDA0003415557450000105
wherein, woldRepresenting a pre-parameter wcParameter before update, wnewRepresenting a pre-parameter wcThe updated parameters are used to determine the parameters of the system,
Figure BDA0003415557450000111
is the gradient of the loss value to the parameter, which determines the direction of updating the parameter, the loss value loss () being the mutual information MI (a, B); η is the learning rate, which scales the gradient advance length, determining the speed of parameter update.
To describe this process more clearly, the above process can be broken down into three steps, g representing the gradient, v being the learning rate η multiplied by the inverse gradient-g, the update amount of the parameter vector:
Figure BDA0003415557450000112
and adjusting transformation parameters by conventional step gradient descent optimization so that the optimization follows the gradient of the image similarity measurement in the direction of the extreme value. It uses a step of constant length along the gradient between calculations until the gradient changes direction. And then the step size is gradually reduced according to a set rule until the algorithm stops iteration. The above loss value, i.e. the image similarity metric, is calculated by using the mutual information of the images, and is defined as:
MI(A,B)＝H(A)+H(B)-H(AB) (8)
wherein
Figure BDA0003415557450000113
Figure BDA0003415557450000114
Figure BDA0003415557450000115
Wherein, A and B are images input by algorithm, namely a filtering image W and an optical image respectively
Figure BDA0003415557450000119
A corresponding Laplacian pyramid residual image; a and b are pixel points corresponding to the images respectively;
the probability distribution is calculated by the joint gray histogram h (a, b) between the images, i.e.:
Figure BDA0003415557450000116
Figure BDA0003415557450000117
Figure BDA0003415557450000118
for the present invention, a mutual information threshold Γ is first set that is considered to be registered, and when the mutual information is less than the threshold, the algorithm continues to iterate until the threshold is reached.
The "pre-parameter" in step 3.2 is an initial value, the mutual information calculated by using the initial value is definitely very small, then the iteration optimization is carried out by using the step (6) until a very large mutual information, namely a threshold value, is reached, the algorithm considers that the mutual information is matched when the threshold value is reached, and the iteration is completed. Actually, the HaLRTC is responsible for completing pixels of an ISAR image, so that the subsequent matching effect is better, the matching is an optimization process, because the optical image and the ISAR image have rotation translation and scale transformation, an affine transformation matrix can be used for description, four parameters are in the matrix, and the parameters are optimized accurately, so that a good matching effect can be obtained.
And 3.5, finally, introducing a differential homoembryo method to carry out non-rigid body matching, so that the local registration result is more accurate, setting super parameters such as smoothness and pyramid layer number to obtain a final more accurate registration result, and realizing the registration of the space target ISAR and the heterogeneous image data of the optical image.
In the embodiment, for the heterogeneous image matching method, the pixel sizes of the input ISAR image and the optical image are both 128 × 128 × 3, the ISAR image is preprocessed by the enhanced Lee filtering and converted into a gray image together with the optical image, and the pixel size is 128 × 128; then, carrying out affine change on the optics, firstly setting an initial transformation matrix S', wherein the pixel size after transformation is 128 multiplied by 128; and (3) starting iteration, respectively constructing three layers of Laplacian residual pyramids of the ISAR image and the optical image, calculating mutual information by using a formula (8) to obtain a loss function, and optimizing a matching result by using a formula (6), namely adjusting parameters of an affine transformation matrix to ensure that the optimization follows the gradient of the image similarity measurement in the direction of an extreme value. It uses a step of constant length along the gradient between calculations until the gradient changes direction. Then, the step length is gradually reduced according to a set rule until the algorithm reaches the maximum iteration times;
and finally, optimizing the matching result by utilizing a differential homoembryo non-rigid body matching method and the hyperparameter set in the claim 3 to obtain a final matching transformation matrix, and realizing the matching between the space target heterogeneous images.
Examples
As shown in fig. 1, the present embodiment mainly includes the following steps:
firstly, simulating to obtain an ISAR spin space target sequence sample based on a three-dimensional space target model and a range-Doppler algorithm; obtaining an optical sample of the space target in the same posture as the ISAR image by using a projection method;
constructing an ISAR space target image sequence of the missing pixel points through a missing model, and completing the missing pixels of the space target ISAR image by using a HaLRTC method based on the image sequence and the optimized weight; in the process, firstly, a part of training sequence is utilized to obtain the optimal weight, and then the optimal weight is utilized to complement the missing pixels of the target image sequence;
finally, matching the complemented space target ISAR image with the corresponding optical image based on a pyramid resampling and optimization iteration method;
in this embodiment, the ISAR spin space target sample is an image sequence generated by simulation, and is used as a fixed image of a reference after completion, and an affine transformation matrix of an optical image relative to the reference image is finally obtained at a matching stage;
as shown in fig. 4, the obtaining process of the ISAR spin space sample includes: electromagnetic parameter setting and ISAR imaging. Setting the frequency of grid generation according to the size of the model by electromagnetic parameter setting, and taking the corner points of grid lines on the target model as the space coordinates of three-dimensional scattering points; the ISAR imaging part utilizes the three-dimensional information of scattering points of the target to carry out ISAR imaging simulation according to the set radar parameters and target motion parameters to obtain an ISAR spinning space target image; and then simulating to obtain images of 5 types of targets to form a simulated image sequence.
The pixel sizes of the simulated space target ISAR image and the simulated optical image are 256 multiplied by 3; the size of a sequence tensor obtained by constructing an ISAR image sequence is 256 multiplied by 3 multiplied by 200; the size of the intercepted input data tensor is 128 multiplied by 3 multiplied by 200; for the HaLRTC method based on the optimized weight, each input channel tensor is 128 multiplied by 10, 20 multiplied by 3 groups of input channel tensors are shared, the tensor size processed by the missing model is 128 multiplied by 10, and each channel selects 20 ISAR missing pixel sequences to train; for the heterogeneous image matching method, the pixel sizes of the input ISAR image and the optical image are both 128 multiplied by 3, and the pixel size of the preprocessed image is 128 multiplied by 128; the pixel size after affine transformation is 128 × 128; the size of the affine transformation matrix obtained finally is 3 multiplied by 3;
the hyper-parameters include: the carrier frequency of the grid electromagnetic wave set in the FEKO is 1GHz, the number of scattering points selected in the imaging process is 500, the initial angle between a reference coordinate system and a local coordinate system is set to-130 degrees, and the frequency of a linear frequency modulation signal is set to fc5.52GHz, bandwidth B300 MHz, pulse signal time width Tp＝25.6×10-6s, distance sampling frequency FM＝2.5e6Hz. the rotation angular velocity of the space target is set to 0.1rad/s, including 5 kinds of space targets; the pixel sizes of the simulated space target ISAR image and the simulated optical image are 256 multiplied by 256, each target has 200 sequences, and n is 128 when the ISAR image sequence is constructed;
for the HaLRTC method based on the optimized weight, the sequence number of ISAR images is selected to be 10, the training sample number K is selected to be 20, the maximum iteration number maximum is 1000, and the initial weight rho is selectedinit0.01, the update rate lr is 0.0001;
for the gradient descent optimizer, the number of spatial samples is 500, the number of histogram bars is 50, the growth factor is 1.05, the initial search radius is 0.000625, the maximum iteration number is 200, and the pyramid is set to be 3 layers;
for the differential homoembryo non-rigid body matching method, the pyramid iteration number of each layer is set as 100, the smoothness is set as 1.5, and the pyramid is set as 7 layers;
constructing an ISAR space target image sequence of missing pixel points through a missing model, and in the process of completing the missing pixels of the space target ISAR image by using a HaLRTC method based on the image sequence and optimized weight, firstly using a part of training sequence to obtain optimal weight, and then completing the missing pixels of the target image sequence by using the optimal weight; finally, matching the complemented space target ISAR image with the corresponding optical image based on a pyramid resampling and optimization iteration method; and after obtaining the affine transformation matrix corresponding to the optical image, ending the method operation. As shown in fig. 2, the operation flow for obtaining the optimal weight is as follows:
firstly, constructing an ISAR image sequence of K missing pixel points, taking 20 from K in the method, combining the ISAR image sequence into a channel input tensor with the size of 128 multiplied by 10, and then setting initial hyper-parameters including the maximum iteration number maximum of 1000 and the initial weight rhoinit0.01, the update rate lr is 0.0001, etc.; completing by a HaLRTC method, calculating loss during each iteration, updating the weight by a calculation result, and finally obtaining the optimal weight when the maximum iteration times are reached; finally, the optimal weight is used for completing the target ISAR image sequence with the missing pixel points, and the speed and the precision are improved when the optimal weight is used for completing the completion;
as shown in the architecture diagram of heterogeneous image matching in fig. 3, the process for heterogeneous image matching includes the following steps:
the pixel size of the input ISAR image and the pixel size of the optical image are both 128 multiplied by 3, the ISAR image is preprocessed through enhanced Lee filtering and is converted into a gray scale image together with the optical image, and the pixel size is 128 multiplied by 128; then, carrying out affine change on the optics, firstly setting an initial transformation matrix S, wherein the pixel size after transformation is 128 multiplied by 128; and starting iteration, firstly constructing three layers of Laplacian pyramid residual images of the ISAR image and the optical image, then calculating mutual information to obtain a loss function, finally optimizing a matching result, and adjusting parameters of an affine transformation matrix to ensure that the optimization follows the gradient of the image similarity measurement in the direction of an extreme value. It uses a step of constant length along the gradient between calculations until the gradient changes direction. And then the step length is gradually reduced according to a set rule until the algorithm reaches the maximum iteration times.
And finally, optimizing the matching result by utilizing a differential homoembryo non-rigid body matching method and through the set hyper-parameters to obtain a final matching transformation matrix, and realizing the matching between the space target heterogeneous images.
The effect of the present embodiment is shown in fig. 5(a) to 5(c), and fig. 6(a) to 6 (b).
Although the invention herein has been described with reference to particular embodiments, it is to be understood that these embodiments are merely illustrative of the principles and applications of the present invention. It is therefore to be understood that numerous modifications may be made to the illustrative embodiments and that other arrangements may be devised without departing from the spirit and scope of the present invention as defined by the appended claims. It should be understood that features described in different dependent claims and herein may be combined in ways different from those described in the original claims. It is also to be understood that features described in connection with individual embodiments may be used in other described embodiments.

Claims (10)

1. The image completion-based space target heterogeneous image matching method is characterized by comprising the following steps of:
step 1, simulating and obtaining an ISAR image sequence under the condition of target spinning based on a three-dimensional space target model and a range-Doppler algorithm; obtaining an optical sample of the space target in the same posture as the ISAR image by using a projection method;
step 2, complementing the missing pixels of the ISAR image corresponding to the space target by a HaLRTC method based on the image sequence and the optimal weight;
and 3, matching the space target ISAR image with the optical image based on the pyramid resampling and optimization iteration method.
2. The image completion-based spatial target heterogeneous image matching method according to claim 1, wherein the step 2 of performing completion on the missing pixels of the spatial target ISAR image by the HaLRTC method based on the image sequence and the optimal weight comprises the following steps:
2.1, constructing an ISAR image sequence containing the space target, taking the ISAR image containing the space target as a tensor X, and dividing the tensor into three input channel tensors Xi,i＝1,2,3，XiThe size is nxnxnxn;
2.2, constructing an ISAR image missing pixel sequence by using a missing model, wherein each input channel tensor has a missing model SiI is 1,2,3, and the ISAR image missing pixel sequence after the processing of the missing model is expressed as tensor
Figure FDA0003415557440000011
The size is nxnxnxn;
2.3, image completion is carried out by using a HaLRTC method based on optimization weight:
2.3.1, aiming at K ISAR image sequences, constructing ISAR image missing pixel sequences
Figure FDA0003415557440000012
2.3.2, calculating loss function and updating weight:
Figure FDA0003415557440000013
Figure FDA0003415557440000014
Figure FDA0003415557440000015
where, k represents the iteration to the k-th round,
Figure FDA0003415557440000016
is the weight corresponding to the ith channel of the jth image sequence in the (k + 1) th round,
Figure FDA0003415557440000017
representing a loss function; lr is the update rate;
recording the current weight at each iteration
Figure FDA0003415557440000018
The two largest and smallest weight values in each channel are dropped and the optimized weight is found using the following equation:
Figure FDA0003415557440000021
when showing loss function
Figure FDA0003415557440000022
Less than a threshold value LgateStopping iteration when the iteration is stopped, or stopping iteration when the maximum iteration times are reached, substituting the obtained optimized weight into a HaLRTC method to perform image completion, and obtaining the tensor data of each channel after completion
Figure FDA0003415557440000023
And combining the data of each channel in sequence to obtain the supplemented ISAR image data.
3. The image completion-based spatial target heterogeneous image matching method according to claim 2, wherein the pyramid resampling and optimization iteration-based method in step 3 is used for matching the spatial target ISAR image and the optical image, and comprises the following steps:
3.1, filtering the supplemented ISAR image to obtain a filtered ISAR image W;
then converting the optical image and the ISAR image into a gray scale image;
3.2 setting a pre-parameter wcCarrying out affine transformation such as translation, rotation and scaling on the optical image; the optical image after affine transformation is
Figure FDA0003415557440000024
The affine transformation matrix is S', the pre-parameter wcIs a parameter set at the beginning of the affine transformation matrix;
3.3 Filter image W corresponding to ISAR image converted into Gray-level image and affine-transformed optical image
Figure FDA0003415557440000029
Respectively constructing Laplacian pyramid residual images;
3.4, optimizing the matching result by using a gradient descent optimizer:
wnew＝wold-η·▽loss(wold) (6)
wherein, woldRepresenting a pre-parameter wcParameter before update, wnewRepresenting a pre-parameter wcUpdated parameter, # loss (w)old) Is the gradient of the loss value to the parameter, which determines the direction of updating the parameter, η is the learning rate; loss value loss (-) for filtered image W and optical image
Figure FDA0003415557440000025
The mutual information MI (A, B), A, B of the corresponding Laplacian pyramid residual images are the filtered image W and the optical image respectively
Figure FDA0003415557440000026
A corresponding Laplacian pyramid residual image;
when the mutual information MI (A, B) is smaller than a mutual information threshold gamma, continuing iteration until the mutual information threshold gamma is reached;
3.5, non-rigid body matching is carried out by utilizing a differential homoembryo method.
4. The image completion-based spatial target heterogeneous image matching method according to claim 3, wherein the filtered image W and the optical image
Figure FDA0003415557440000027
The mutual information MI (A, B) of the corresponding Laplacian pyramid residual image is as follows
MI(A,B)＝H(A)+H(B)-H(AB) (8)
Wherein
Figure FDA0003415557440000028
Figure FDA0003415557440000031
Figure FDA0003415557440000032
Figure FDA0003415557440000033
Figure FDA0003415557440000034
Figure FDA0003415557440000035
Wherein, A and B are respectively a filter image W and an optical image
Figure FDA0003415557440000036
A corresponding Laplacian pyramid residual image; a and b are pixel points corresponding to the images respectively; h (a, b) is a joint gray histogram between images.
5. The image completion-based spatial target heterogeneous image matching method according to claim 4, wherein the process of constructing the Laplacian pyramid residual image in step 3.3 comprises the following steps:
firstly, constructing a Gaussian pyramid:
Figure FDA0003415557440000037
wherein G isl-1(. h) represents a layer l-1 pyramid image; l represents the layer of the image, q, e are two variables; w (q, e) is a Gaussian template;
and then, carrying out reverse operation, and expanding a certain level of image in the Gaussian pyramid into the size of the previous level of image through interpolation operation, wherein the size is expressed as follows:
Figure FDA0003415557440000038
subtracting the original image of the previous layer from the interpolated result of the image of the previous layer to obtain the image of the previous layer, namely a residual image Gl+1(i, j) is the expression of the l +1 layer, and the Laplacian pyramid residual image is obtained by subtracting the image of the previous layer.
6. The image completion-based spatial target heterogeneous image matching method according to claim 5, wherein before filtering the completed ISAR image and before converting the optical image into the gray scale image, the ISAR image is filtered according to the markers in the image and the optical image is scaled so that the spatial scale difference of the heterogeneous images does not exceed 10%.
7. The image completion-based spatial target heterogeneous image matching method according to claim 6, wherein the filtering process of the completed ISAR image is implemented by using enhanced Lee filtering,
Figure FDA0003415557440000041
wherein I denotes an image pixel within the sliding window,
Figure FDA0003415557440000042
is the mean of the pixels within the sliding window, W is the estimate of the image, i.e. the filtered image; cIRepresenting the local standard deviation coefficient, C, of the imagemin＝Cu,
Figure FDA0003415557440000043
L represents the equivalent view, u represents the noise, CuIs the noise local standard deviation coefficient; p is the weighting coefficient of the Lee filter.
8. The method for matching the heterogeneous images of the spatial target based on the image completion of claim 7, wherein the step 2.1 of constructing the ISAR image sequence containing the spatial target comprises the following steps:
selecting continuous N ISAR image sequences, intercepting a square area with the size of nxn at the center of an image to ensure that the area can completely contain a space target, and forming an input data tensor X after interception, wherein the size of the input data tensor X is nxnxnxnxnx3XN, and 3 is three color channels representing the ISAR images.
9. The method for matching the spatial target heterogeneous images based on image completion according to claim 8, wherein the process of obtaining the ISAR image sequence under the target spinning condition based on the three-dimensional spatial target model and the range-Doppler algorithm simulation comprises the following steps:
1.1, opening a 3D model file of a space object, and storing a file in a parasolid format and an mdl file which are convenient to import for FEKO software;
1.2, opening an x _ t file in FEKO software, setting a solving algorithm, a target material, grid quality and an electromagnetic wave carrier frequency parameter, and storing the three-dimensional grid of the target as a nas frame file;
1.3, after the frame file is saved as a text file, processing character strings in the text file, exporting grid point coordinates of scattering point clusters to MATLAB, uniformly selecting N point coordinates from an output result, and completely displaying a target on an ISAR image so as to obtain the distribution of discrete scattering points of the model;
1.4, ISAR imaging is carried out on the scattering points with known distribution by using a range-Doppler algorithm, and an ISAR image sequence under the condition of target spinning is simulated.
10. The image completion-based spatial target heterogeneous image matching method according to one of claims 1 to 9, wherein the construction process of the missing model comprises the following steps:
assuming that the size of an X pixel of the ISAR image is m × m × 3, some elements of the ISAR image are missing, and the observed index set is (i, j, k) epsilon Ω, where i, j, k are indexes of the pixels of the ISAR image on three channels, and Ω is the observed set, and the pixels not in the set are missing pixels; let the deficiency model S satisfy Sij1, (i, j, k) e Ω, otherwise
Figure FDA0003415557440000044
The ISAR image after missing pixels can be represented as
Figure FDA0003415557440000045
CN202111544857.6A 2021-12-16 2021-12-16 Spatial target heterogeneous image matching method based on image completion Pending CN114219995A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202111544857.6A CN114219995A (en) 2021-12-16 2021-12-16 Spatial target heterogeneous image matching method based on image completion

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202111544857.6A CN114219995A (en) 2021-12-16 2021-12-16 Spatial target heterogeneous image matching method based on image completion

Publications (1)

Publication Number Publication Date
CN114219995A true CN114219995A (en) 2022-03-22

Family

ID=80703106

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202111544857.6A Pending CN114219995A (en) 2021-12-16 2021-12-16 Spatial target heterogeneous image matching method based on image completion

Country Status (1)

Country Link
CN (1) CN114219995A (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117274110A (en) * 2023-11-14 2023-12-22 吉林大学 Image optimization method based on generation type artificial intelligence and graph neural network

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117274110A (en) * 2023-11-14 2023-12-22 吉林大学 Image optimization method based on generation type artificial intelligence and graph neural network
CN117274110B (en) * 2023-11-14 2024-02-13 吉林大学 Image optimization method based on generation type artificial intelligence and graph neural network

Similar Documents

Publication Publication Date Title
CN110675418B (en) Target track optimization method based on DS evidence theory
CN112686935B (en) Airborne sounding radar and multispectral satellite image registration method based on feature fusion
CN110689562A (en) Trajectory loop detection optimization method based on generation of countermeasure network
CN104869387B (en) Method for acquiring binocular image maximum parallax based on optical flow method
CN109255781B (en) Object-oriented multispectral high-resolution remote sensing image change detection method
CN114419147A (en) Rescue robot intelligent remote human-computer interaction control method and system
US11238307B1 (en) System for performing change detection within a 3D geospatial model based upon semantic change detection using deep learning and related methods
CN110728706B (en) SAR image fine registration method based on deep learning
US11636649B2 (en) Geospatial modeling system providing 3D geospatial model update based upon predictively registered image and related methods
CN116543117B (en) High-precision large-scene three-dimensional modeling method for unmanned aerial vehicle images
CN114429555A (en) Image density matching method, system, equipment and storage medium from coarse to fine
CN111998862A (en) Dense binocular SLAM method based on BNN
CN116310219A (en) Three-dimensional foot shape generation method based on conditional diffusion model
CN114549669A (en) Color three-dimensional point cloud obtaining method based on image fusion technology
CN116958420A (en) High-precision modeling method for three-dimensional face of digital human teacher
CN114677479A (en) Natural landscape multi-view three-dimensional reconstruction method based on deep learning
CN114332156B (en) Real-time three-dimensional motion completion method based on graph convolution neural network
CN117372460A (en) Moon rover autonomous positioning method based on self-adaptive threshold edge registration visual odometer
CN116129292A (en) Infrared vehicle target detection method and system based on few sample augmentation
CN114219995A (en) Spatial target heterogeneous image matching method based on image completion
CN106097277B (en) A kind of rope substance point-tracking method that view-based access control model measures
CN118154770A (en) Single tree image three-dimensional reconstruction method and device based on nerve radiation field
Herau et al. 3DGS-Calib: 3D Gaussian Splatting for Multimodal SpatioTemporal Calibration
CN112115786A (en) Monocular vision odometer method based on attention U-net
CN111160300A (en) Deep learning hyperspectral image saliency detection algorithm combined with global prior

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination