CN113538534B - Image registration method based on depth reinforcement learning nano imaging - Google Patents

Image registration method based on depth reinforcement learning nano imaging Download PDF

Info

Publication number
CN113538534B
CN113538534B CN202110700487.4A CN202110700487A CN113538534B CN 113538534 B CN113538534 B CN 113538534B CN 202110700487 A CN202110700487 A CN 202110700487A CN 113538534 B CN113538534 B CN 113538534B
Authority
CN
China
Prior art keywords
image
registered
network
action
reinforcement learning
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202110700487.4A
Other languages
Chinese (zh)
Other versions
CN113538534A (en
Inventor
蒋林华
杜云龙
张冠华
曾新华
庞成鑫
宋梁
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Fudan University
Original Assignee
Fudan University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Fudan University filed Critical Fudan University
Priority to CN202110700487.4A priority Critical patent/CN113538534B/en
Publication of CN113538534A publication Critical patent/CN113538534A/en
Application granted granted Critical
Publication of CN113538534B publication Critical patent/CN113538534B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/30—Determination of transform parameters for the alignment of images, i.e. image registration
    • G06T7/33—Determination of transform parameters for the alignment of images, i.e. image registration using feature-based methods
    • G06T7/337—Determination of transform parameters for the alignment of images, i.e. image registration using feature-based methods involving reference images or patches
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/047—Probabilistic or stochastic networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/30—Determination of transform parameters for the alignment of images, i.e. image registration
    • G06T7/33—Determination of transform parameters for the alignment of images, i.e. image registration using feature-based methods
    • G06T7/344—Determination of transform parameters for the alignment of images, i.e. image registration using feature-based methods involving models
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/30—Determination of transform parameters for the alignment of images, i.e. image registration
    • G06T7/35—Determination of transform parameters for the alignment of images, i.e. image registration using statistical methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/30—Determination of transform parameters for the alignment of images, i.e. image registration
    • G06T7/38—Registration of image sequences
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10116—X-ray image
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Biomedical Technology (AREA)
  • Artificial Intelligence (AREA)
  • Computational Linguistics (AREA)
  • Data Mining & Analysis (AREA)
  • Evolutionary Computation (AREA)
  • General Health & Medical Sciences (AREA)
  • Molecular Biology (AREA)
  • Computing Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • Biophysics (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Health & Medical Sciences (AREA)
  • Probability & Statistics with Applications (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Evolutionary Biology (AREA)
  • Image Analysis (AREA)

Abstract

The invention belongs to the technical field of nano imaging, and particularly relates to an image registration method of nano imaging based on depth reinforcement learning. Constructing a deep reinforcement learning network model, and carrying out image registration by using the network; the network model comprises two branches; one branch comprises a full link layer, and the input is an action sequence; one branch comprises two convolution layers and a pooling layer, and the input is a selected reference picture and a picture to be registered; the output is an action probability distribution representing a policy function; an image registration part, which designs 8 action sequences to finely adjust the image to be registered; the method specifically comprises the following steps: resampling an image to be registered; and inputting the image to be registered, the reference image and the resample image into the constructed network model, and outputting the probability distribution of the strategy action. The invention has the advantages of high speed, high precision, good robustness and strong adaptability; the image registration is performed fully automatically, and the trouble of manual marking is eliminated.

Description

Image registration method based on depth reinforcement learning nano imaging
Technical Field
The invention belongs to the technical field of nano imaging, and particularly relates to an image registration method of nano imaging.
Background
An image registration method for extracting nano imaging by using a depth reinforcement learning method. The image registration is a method for (locally) optimally mapping one or more pictures onto a target picture based on some evaluation criteria by using a certain algorithm, and belongs to the field of software methods.
A deep reinforcement learning method relates to a neural network structure and a reinforcement learning control strategy in deep learning, and belongs to the field of software methods.
Due to the jitter of the stepping motor and the influence of other factors such as long-time irradiation of the X-ray, each picture formed after the X-ray transmission is slightly shifted at the center of the circle. In processing nano-imaged images, registration of the images is required to eliminate misalignment due to stepper motor jitter or other causes.
The basic classifications of image registration are: (1) gray scale based image registration; (2) feature-based image registration. A specific image registration algorithm is an algorithm based on a mixture or variation of these two points. In general, image registration techniques include four aspects: transformation models, feature spaces, similarity measures, search spaces, and search strategies.
According to these four characteristics, the image registration step can be generally divided into the following five steps:
(1) selecting a proper transformation model according to the actual application occasion;
(2) selecting a suitable feature space, either gray-level based or feature-based
(3) Determining the range of possible parameter change according to the parameter configuration of the transformation model and the selected characteristics, and selecting an optimal search strategy;
(4) searching in a search space according to an optimization criterion by applying similarity measurement, and searching for a maximum correlation point so as to solve unknown parameters in a transformation model;
(5) and corresponding the images to be registered to the reference images according to the transformation model, and realizing the matching between the images.
Wherein, how to select the proper features for matching is the key of registration. Taking feature-based image registration as an example, the process is as follows:
(1) feature detection
An important task of the image registration process. Depending on the complexity of the problem, manual or automatic detection is often distinguished, but automatic feature detection is often preferred. Closed borders, edges, contours, line intersections, corner points, and their representative points such as center of gravity or line ends (collectively referred to as control points) may be used as features. These features, which consist of special objects, must be easy to detect, i.e. the features will be physically interpretable and identifiable. The reference image must share a sufficient set of common features with the floating image without being affected by any unknown occlusion or accidental change. The algorithm used for detection should be robust enough to be able to detect the same features in all projections of the scene without being affected by any particular image deformation or degradation.
(2) Matching the characteristics;
this step is essentially established on the correspondence between the image to be registered and the features detected in the reference image. In addition to the spatial relationship between features, different feature descriptors (feature descriptors) and similarity measures are also employed to determine the accuracy of the registration. The feature descriptors must be reasonably configured so that they remain unchanged at any degradation, while at the same time they need to be noise free and able to properly distinguish between different features.
(3) Evaluation of image transformation models
To register the floating image with the reference image, the parameters of the mapping function need to be estimated. These parameters are calculated using the corresponding features obtained from the previous step. The choice of mapping function depends on the image acquisition process and a priori knowledge of the expected image deformation. The flexibility of the model must be ensured without any a priori information.
(4) Image transformation
The floating images are registered using an image transformation of the map.
In the traditional method, a large amount of time is spent on image feature detection and image feature matching, the running speed is slow, the effect is not good, and the influence of the shooting effect of the sample is large. Although the manual marking method has high precision, the sample preparation is difficult, the workload is large, the efficiency is low, the cost is high, and the applicability is poor.
Disclosure of Invention
The invention aims to provide a full-automatic nano imaging image registration method which is high in precision, good in robustness and strong in applicability.
The image registration method of the nano imaging provided by the invention is based on a depth reinforcement learning technology, and comprises the following specific steps:
(I) construction of deep reinforcement learning network model (STN)
The structure of the deep reinforcement learning network model constructed by the invention is shown in figure 1; the network model is divided into a left branch and a right branch, and finally the left branch and the right branch are combined into a network.
Branch network on the left: the method comprises a full connection layer 1, and the input is an action sequence;
the branch network on the right: the method comprises the following steps: the method comprises the following steps that a convolutional layer 4, a pooling layer 1 and a convolutional layer 5 are input to be selected reference pictures and pictures to be registered, and the reference pictures and the pictures are divided into two small branches after passing through the convolutional layer and the pooling layer;
wherein, the small branch network on the left comprises three layers of convolution: a convolutional layer 6, a convolutional layer 7, a convolutional layer 8; the right small branch network includes a Spatial Transform Network (STN) and two layers of convolution: convolutional layer 9 and convolutional layer 10.
And finally, combining the characteristics of the two branch networks, and then performing two full-connection layers: full connectivity layer 2 and full connectivity layer 3, a softmax layer, are used to represent the action probability distribution of the policy function.
In the figure, fc denotes the fully-connected layer, conv denotes the convolutional layer, ReLU denotes the form of the activation function used, drop denotes the prevention of overfitting by means of dropout, and pool adopts maximum pooling, and it is noted that a Batch Normalization layer (BN) may be added before the activation function to accelerate the convergence of the network.
The model parameters are preferably as follows:
Figure GDA0003501508780000031
(II) image registration
For image registration, the change of the graph can be subdivided into the following actions: enlargement (recording) of imagesIs a1) Reduction of image (denoted as a)2) Counterclockwise rotation of the image (denoted as a)3Clockwise selection of images (denoted as a)4Upward translation of the image (denoted as a)5) Leftward translation of the image (denoted as a)6) Rightward translation of the image (denoted as a)7) Translation of the image downwards (denoted as a)8)。
In general, the magnification increment and decrement for enlargement and reduction are set to 0.05, the counterclockwise and clockwise rotations are set to 1 degree, and the translation distance of the image is set to 1 mm. The above 8 motion sequences are defined to fine-tune the image to be registered.
Through the above 8 motion sequences, fine tuning is performed on the image to be registered. The specific process is as follows:
(1) an action is selected from the above sequence of actions to move the image to be registered and a reward value is returned according to the probability distribution. And resampling the image to be registered according to the iteration times or judging whether the network state value reaches a threshold value.
(2) Inputting the image to be registered and the reference image in the single mode and the above action sequence into the network model in the step (1), and outputting the probability distribution of the strategy action.
(3) According to the probability of epsilon, selecting an image with the maximum probability distribution from the action sequence, moving the image to be registered according to the action, and returning a reward value; the bonus value here is measured by the similarity of two pictures, which can be stopped if the similarity reaches a certain threshold. If the threshold value is not reached, the new image to be registered and the reference image generated after the movement are put into the network again for operation.
In the invention, a Space Transformation Network (STN) is added in the network model to cope with the rotation, the scaling and the translation of the partial region of the picture when the registration is carried out, so as to make up the insensitivity of the convolutional neural network to the partial feature.
Compared with the prior art, the invention has the advantages that:
1. the speed is high, and the precision is ensured;
2. the robustness is good;
3. the image registration is performed fully automatically, the trouble of manual marking is avoided, and the adaptability is good.
Drawings
FIG. 1 is a structural diagram of a deep reinforcement learning network model constructed by the present invention.
Detailed Description
The original sample starts to rotate and project from 0 degrees according to a central axis, one picture is obtained after the original sample rotates 1 degree, 360 degrees are obtained after the original sample rotates one circle, the 360 pictures are used as a data set, the initial picture of 0 degree is selected as a reference picture, the picture at the time is in an initial state and does not rotate, heat radiation caused by shaking of a stepping motor and long-time X-ray irradiation does not exist, the rest pictures of 1 to 359 degrees are remained, each frame of picture is closely related to the previous frame of picture, and each picture is obtained by rotating and transforming the sample. Thus, it can be quite normally assumed that there is a simpler transformation between two adjacent frame pictures.
The 1 degree image is taken as the image to be registered, and the two images are stacked and taken as the input of the network. The method comprises the following steps of maintaining an action experience pool, wherein eight last action sequences are stored in the action experience pool each time; in the initial state, these are eight null sequences. After the network outputs the Q value of the action, the action sequence is selected according to the probability distribution of the Q value, and the action is stored in the action sequence experience pool in a first-in first-out mode.
Parameters in the network are initialized, the learning rate is set to be 1e-5, and the round is stopped when the similarity between the picture to be registered and the reference picture reaches a threshold value. And resampling the picture to be registered to generate a registered picture, taking the generated picture as a new reference picture, and taking the picture of the next frame as the picture to be registered. Scaling and cutting the two graphs to the same size, stacking and putting the two graphs into a network input layer; and circulating the previous flow. This process continues until all pictures are registered.
The output is an 8-dimensional information vector, the value of the action sequence is appointed, one of the pictures is updated according to the appointed value of the action sequence according to the rotation angle sequence, the similarity is calculated by comparing the floating picture with the reference picture, the value of the similarity is used as a reward value, and the network parameters are updated by back propagation by using a loss function of the reward value.

Claims (3)

1. An image registration method based on depth reinforcement learning nano imaging is characterized by comprising the following specific steps:
(I) constructing a deep reinforcement learning network model
The deep reinforcement learning network model is divided into a left branch and a right branch; wherein:
left branch network: the method comprises a full connection layer 1, and the input is an action sequence;
the right branch network: the method comprises the following steps: the method comprises the following steps that a convolutional layer 4, a pooling layer 1 and a convolutional layer 5 are input to be selected reference pictures and pictures to be registered, and the reference pictures and the pictures are divided into two small branches after passing through the convolutional layer and the pooling layer; wherein:
and the left small branch network comprises three layers of convolution: a convolutional layer 6, a convolutional layer 7, a convolutional layer 8; the small branch network on the right includes a space transformation network and two layers of convolution: a convolutional layer 9 and a convolutional layer 10;
finally, the characteristics of the left branch network and the right branch network are merged, and the merged network passes through two full connection layers: a full connection layer 2 and a full connection layer 3, namely a softmax layer, which is used for representing the action probability distribution of the strategy function;
(II) image registration
For the variation of the graph, the following actions are subdivided: the enlargement of the image is denoted as a1The reduction of the image is denoted as a2And the counterclockwise rotation of the image is denoted as a3Clockwise selection of an image is denoted as a4The upward shift of the image is denoted as a5Left translation of the image is denoted as a6The rightward shift of the image is denoted as a7The downward translation of the image is denoted as a8(ii) a Fine adjustment is carried out on the image to be registered through the above 8 motion sequences; the specific process is as follows:
(1) according to the probability distribution, selecting an action to move the image to be registered from the action sequence, and returning a reward value; resampling the image to be registered according to the iteration times or judging whether the network state value reaches a threshold value;
(2) inputting the image to be registered, the reference image and the action sequence in the single mode into the depth reinforcement learning network model constructed in the step (I), and outputting the probability distribution of the strategy action;
(3) according to the probability of epsilon, selecting an image with the maximum probability distribution from the action sequence, moving the image to be registered according to the action, and returning a reward value; the reward value uses the similarity of the two images as a measure, and stops if the similarity reaches a certain threshold; if the threshold value is not reached, the new image to be registered and the reference image generated after the movement are put into the network again for operation.
2. The image registration method based on deep reinforcement learning nano imaging of claim 1, wherein in the deep reinforcement learning network model constructed in the step (one), specific parameters of the model are as follows:
Figure FDA0003501508770000011
Figure FDA0003501508770000021
3. the depth-enhanced learning-based nanoimaging image registration method according to claim 2, wherein in the 8 actions in step (two), the magnification increment and decrement for zooming in and out are set to 0.05, the counterclockwise and clockwise rotations are set to 1 degree, and the translation distance of the image is set to 1 mm.
CN202110700487.4A 2021-06-23 2021-06-23 Image registration method based on depth reinforcement learning nano imaging Active CN113538534B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202110700487.4A CN113538534B (en) 2021-06-23 2021-06-23 Image registration method based on depth reinforcement learning nano imaging

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202110700487.4A CN113538534B (en) 2021-06-23 2021-06-23 Image registration method based on depth reinforcement learning nano imaging

Publications (2)

Publication Number Publication Date
CN113538534A CN113538534A (en) 2021-10-22
CN113538534B true CN113538534B (en) 2022-05-20

Family

ID=78096575

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202110700487.4A Active CN113538534B (en) 2021-06-23 2021-06-23 Image registration method based on depth reinforcement learning nano imaging

Country Status (1)

Country Link
CN (1) CN113538534B (en)

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110009669A (en) * 2019-03-22 2019-07-12 电子科技大学 A kind of 3D/2D medical image registration method based on deeply study
CN110211165A (en) * 2019-06-10 2019-09-06 成都信息工程大学 A kind of image multi-mode registration method based on the study of asynchronous deeply
CN111028277A (en) * 2019-12-10 2020-04-17 中国电子科技集团公司第五十四研究所 SAR and optical remote sensing image registration method based on pseudo-twin convolutional neural network

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110009669A (en) * 2019-03-22 2019-07-12 电子科技大学 A kind of 3D/2D medical image registration method based on deeply study
CN110211165A (en) * 2019-06-10 2019-09-06 成都信息工程大学 A kind of image multi-mode registration method based on the study of asynchronous deeply
CN111028277A (en) * 2019-12-10 2020-04-17 中国电子科技集团公司第五十四研究所 SAR and optical remote sensing image registration method based on pseudo-twin convolutional neural network

Non-Patent Citations (3)

* Cited by examiner, † Cited by third party
Title
Image-and-Spatial Transformer Networks for Structure-Guided Image Registration;Matthew C.H. Lee，et al.;《arXiv》;20190722;全文 *
Recurrent spatial transformer network for high-accuracy image registration in moving PCB defect detection;Weibo Huang，et al.;《The Journal of Engineering》;20200706;全文 *
基于深度强化学习的多模态医学图像配准;姚明青，胡靖;《计算机辅助设计与图形学学报》;20200831;第32卷(第8期);全文 *

Also Published As

Publication number Publication date
CN113538534A (en) 2021-10-22

Similar Documents

Publication Publication Date Title
Schneider et al. RegNet: Multimodal sensor registration using deep neural networks
WO2022100379A1 (en) Object attitude estimation method and system based on image and three-dimensional model, and medium
CN107980150A (en) Three dimensions is modeled
EP3012781A1 (en) Method and apparatus for extracting feature correspondences from multiple images
CN113361645B (en) Target detection model construction method and system based on meta learning and knowledge memory
CN111105439A (en) Synchronous positioning and mapping method using residual attention mechanism network
CN109916921A (en) Circuit board defect processing method, device and equipment
CN110826411B (en) Vehicle target rapid identification method based on unmanned aerial vehicle image
US11880997B2 (en) Method and apparatus with pose estimation
CN112364881B (en) Advanced sampling consistency image matching method
CN112184809A (en) Relative pose estimation method, device, electronic device and medium
EP4392935A1 (en) Robustifying nerf model novel view synthesis to sparse data
CN116258877A (en) Land utilization scene similarity change detection method, device, medium and equipment
CN114663880A (en) Three-dimensional target detection method based on multi-level cross-modal self-attention mechanism
CN113538534B (en) Image registration method based on depth reinforcement learning nano imaging
KR20240115727A (en) Apparatus and method for performing visual localization
US20240161362A1 (en) Target-augmented material maps
CN115331021A (en) Dynamic feature extraction and description method based on multilayer feature self-difference fusion
CN113066165B (en) Three-dimensional reconstruction method and device for multi-stage unsupervised learning and electronic equipment
CN115018884A (en) Visible light infrared visual tracking method based on multi-strategy fusion tree
CN115018787A (en) Anomaly detection method and system based on gradient enhancement
Wang et al. Dyeing creation: a textile pattern discovery and fabric image generation method
Orts-Escolano et al. Processing point cloud sequences with Growing Neural Gas
CN111862158A (en) Staged target tracking method and device, terminal and readable storage medium
CN111899284A (en) Plane target tracking method based on parameterized ESM network

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant