CN115082530A - Three-dimensional image registration method and system based on unsupervised learning method - Google Patents
Three-dimensional image registration method and system based on unsupervised learning method Download PDFInfo
- Publication number
- CN115082530A CN115082530A CN202110274132.3A CN202110274132A CN115082530A CN 115082530 A CN115082530 A CN 115082530A CN 202110274132 A CN202110274132 A CN 202110274132A CN 115082530 A CN115082530 A CN 115082530A
- Authority
- CN
- China
- Prior art keywords
- image
- network
- registration
- loss
- dimensional
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/30—Determination of transform parameters for the alignment of images, i.e. image registration
- G06T7/33—Determination of transform parameters for the alignment of images, i.e. image registration using feature-based methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10004—Still image; Photographic image
- G06T2207/10012—Stereo images
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- General Health & Medical Sciences (AREA)
- General Engineering & Computer Science (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Evolutionary Computation (AREA)
- Artificial Intelligence (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- Biomedical Technology (AREA)
- Life Sciences & Earth Sciences (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Health & Medical Sciences (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Image Analysis (AREA)
- Image Processing (AREA)
Abstract
The invention discloses a three-dimensional image registration method and a three-dimensional image registration system based on an unsupervised learning method, wherein the method comprises the following steps: acquiring a three-dimensional medical image; carrying out down-sampling on the image to obtain a down-sampled image; inputting the down-sampled fixed image and the image to be registered into a cascade registration sub-network, and aligning the displacement deviation parts in the two images by using the cascade registration sub-network; inputting the images output by the cascade registration sub-network into a full connection layer, and integrating image characteristics; bringing the image output by the full connection layer into an LOSS function, adding and summing the output results of the functions, measuring the optimization degree of the algorithm network by using a summation value, stopping training and obtaining an image registration network structure after training if the summation value reaches a preset threshold value, and adjusting parameters to perform iterative training on the algorithm network if the summation value does not reach the preset threshold value; and inputting the three-dimensional image to be registered into the image registration network structure to obtain a registered three-dimensional image.
Description
Technical Field
The invention relates to the technical field of image processing, in particular to a three-dimensional image registration method and a three-dimensional image registration system based on an unsupervised learning method.
Background
Three-dimensional medical image registration is clinically significant. Image registration is the process of mapping images to the same coordinate system by finding spatial correspondence between the images. It has wide application in medical image processing, such as aligning images taken of a subject at different times; as another example, an image of a subject is matched to some predefined coordinate system. Image registration, which is widely used today, is based on supervised learning, e.g. FlowNet 2.0, voxelmorphh, but these methods require a large number of accurate manual annotations.
FlowNet 2.0: the method uses two independent registration lines, and then the result is fused into the final estimation, the operation flow of the method is complex, and some external interference is easily introduced, so that the result is poor. Voxelmorphh: the method is based on a supervised learning method, so a large amount of manual labeling work is needed during data set production, and labeling errors are easy to generate in the labeling process to influence the algorithm effect.
Since the ground truth of the three-dimensional medical image is very difficult to obtain, the quality of the labels directly affects the result of supervision, and much effort is required for supervision in the tasks of traditional classification, segmentation and the like. However, optical flow is a dense and fuzzy quantity, and is almost impossible to label manually, and automatically generated datasets are inappropriate to deviate from realistic requirements. Therefore, the supervision method is hardly applicable.
In summary, a technical solution for improving the image registration effect by overcoming the above problems is continuously provided.
Disclosure of Invention
In order to overcome the defects of the prior art, the invention provides a three-dimensional image registration method and a three-dimensional image registration system based on an unsupervised learning method. Since unmarked medical images are ubiquitous, the development of image registration technology can be better pushed through the unsupervised framework presented by the invention.
In a first aspect of the embodiments of the present invention, a three-dimensional image registration method based on an unsupervised learning method is provided, where the method includes:
image acquisition: acquiring a three-dimensional medical image, wherein the three-dimensional medical image comprises a fixed image and an image to be registered;
training an algorithm network: carrying out down-sampling on the image to obtain a down-sampled image;
inputting the down-sampled fixed image and the image to be registered into a cascade registration sub-network, and aligning the displacement deviation parts in the two images by using the cascade registration sub-network;
inputting the image output by the cascade registration sub-network into a full connection layer, and integrating image characteristics;
bringing the image output by the full connection layer into an LOSS function, adding and summing the output results of the functions, measuring the optimization degree of the algorithm network by using a summation value, stopping training and obtaining an image registration network structure after training if the summation value reaches a preset threshold value, and adjusting parameters to perform iterative training on the algorithm network if the summation value does not reach the preset threshold value;
image registration: and inputting the three-dimensional image to be registered into the image registration network structure to obtain a registered three-dimensional image.
Further, the image acquisition comprises:
and acquiring two three-dimensional medical images with the first size, wherein one of the three-dimensional medical images is a fixed image, and the other one is an image to be registered.
Further, the algorithm network is an unsupervised end-to-end registration network, and consists of a cascade registration sub-network and convolutional layers in a plurality of CNNs; wherein after each cascaded registration sub-network, the moving image is warped; when iterative training of the algorithm network is carried out, the heterogeneity between the fixed image and each warped image is combined with the regularization loss value of the sub-network predicted flow to guide.
Further, down-sampling the two images to obtain a down-sampled image, including:
inputting the two three-dimensional medical images with the first size into a first convolution layer of an algorithm network for down-sampling to obtain an image with a second size;
inputting the image with the second size into a second convolution layer of the algorithm network for down-sampling to obtain an image with a third size; wherein the second convolutional layer adopts an Affinine sub-network.
Further, the number of cascaded registration subnetworks is 4.
Further, the LOSS of correlation coefficient, LOSS of orthogonality, LOSS of determinism and LOSS of total variation are included in the LOSS function; wherein the content of the first and second substances,
in the loss of correlation coefficient, the covariance between two images is defined as:
wherein, Cov [ I ] 1 ,I 2 ]Is the covariance of the image, I 1 、I 2 The image data of the two images are respectively, x is a horizontal axis coordinate in omega, y is a vertical axis coordinate in omega, and omega is a cube for defining an input image;
the correlation coefficient is defined as:
wherein, CorrCoef [ I ] 1 ,I 2 ]Is a correlation coefficient;
the loss of correlation coefficient is defined as:
L corr (I 1 ,I 2 )＝1-CorrCoef[I 1 ,I 2 ]
wherein L is corr (I 1 ,I 2 ) Is a correlation systemLoss of numbers;
in loss of orthogonality, a loss of non-orthogonality to I + a is introduced, where a represents a transformation matrix produced by the affinity registration network; let λ be the singular value of I + a, the loss of orthogonality is:
wherein L is ortho For loss of orthogonality, λ i Are singular values; the larger the deviation of the I + A from the orthogonal matrix is, the larger the orthogonality loss is, and if the I + A is the orthogonal matrix, the corresponding value is zero;
in deterministic loss, assuming that the images are taken with the same singularity, let det (I + a) >0, the determinant loss is set as:
L det ＝(-1+det(I+A)) 2 ；
wherein L is det For determinant losses, det (I + a) is the determinant value of I + a;
in total variation loss, normalization will be performed for dense flow fields using a loss function:
wherein L is TV For total variation loss, x is the abscissa in Ω, c i Is the probability that voxel i belongs to the foreground.
Further, if the summation value does not reach the preset threshold, adjusting parameters to perform iterative training on the algorithm network, further comprising:
in the iterative training process, parameters of the algorithm network are updated by a gradient descent method for the LOSS function reverse derivation, and the operation is carried out again until the summation value corresponding to the LOSS function reaches a preset threshold value.
In a second aspect of the embodiments of the present invention, a three-dimensional image registration system based on an unsupervised learning method is provided, the system including:
the image acquisition module is used for acquiring a three-dimensional medical image, wherein the three-dimensional medical image comprises a fixed image and an image to be registered;
the algorithm network training module is used for carrying out down-sampling on the image to obtain a down-sampled image; inputting the down-sampled fixed image and the image to be registered into a cascade registration sub-network, and aligning the displacement deviation parts in the two images by using the cascade registration sub-network; inputting the image output by the cascade registration sub-network into a full connection layer, and integrating image characteristics; bringing the image output by the full connection layer into an LOSS function, adding and summing the output results of the functions, measuring the optimization degree of the algorithm network by using a summation value, stopping training and obtaining an image registration network structure after training if the summation value reaches a preset threshold value, and adjusting parameters to perform iterative training on the algorithm network if the summation value does not reach the preset threshold value;
and the image registration module is used for inputting the three-dimensional image to be registered into the image registration network structure to obtain a registered three-dimensional image.
In a third aspect of embodiments of the present invention, a computer device is presented, comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing a three-dimensional image registration method based on an unsupervised learning method when executing the computer program.
In a fourth aspect of embodiments of the present invention, a computer-readable storage medium is presented, which stores a computer program that, when executed by a processor, implements a three-dimensional image registration method based on an unsupervised learning method.
The three-dimensional image registration method and system based on the unsupervised learning method, provided by the invention, utilize a cascade registration subnetwork to align the displacement deviation parts in two images, improve the performance of registering most displacement images, utilize a full-link layer to integrate image characteristics, bring images output by the full-link layer into an LOSS function, add and sum the output results of the functions, utilize a summation value to measure the optimization degree of an algorithm network, iteratively train the algorithm network according to the optimization degree, improve the registration performance by inverting LOSS in the training process, finally utilize the trained image registration network to register three-dimensional images needing to be registered, do not need to carry out a large amount of standard work in the registration process, and obviously improve the registration efficiency.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed to be used in the description of the embodiments are briefly introduced below, and it is obvious that the drawings in the following description are some embodiments of the present application, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.
Fig. 1 is a schematic flow chart of a three-dimensional image registration method based on an unsupervised learning method according to an embodiment of the present invention.
Fig. 2 is a schematic diagram of a three-dimensional image registration network training process based on unsupervised learning according to an embodiment of the present invention.
Fig. 3 is a schematic diagram of a prediction process of a three-dimensional image registration network based on unsupervised learning according to an embodiment of the present invention.
Fig. 4 is a schematic structural diagram of a three-dimensional image registration system based on an unsupervised learning method according to an embodiment of the present invention.
Fig. 5 is a schematic structural diagram of a computer device according to an embodiment of the present invention.
Detailed Description
The principles and spirit of the present invention will be described with reference to a number of exemplary embodiments. It is understood that these embodiments are given solely for the purpose of enabling those skilled in the art to better understand and to practice the invention, and are not intended to limit the scope of the invention in any way. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.
As will be appreciated by one skilled in the art, embodiments of the present invention may be embodied as a system, apparatus, device, method, or computer program product. Accordingly, the present disclosure may be embodied in the form of: entirely hardware, entirely software (including firmware, resident software, micro-code, etc.), or a combination of hardware and software.
According to the embodiment of the invention, a three-dimensional image registration method and a three-dimensional image registration system based on an unsupervised learning method are provided.
The principles and spirit of the present invention are explained in detail below with reference to several representative embodiments of the invention.
Fig. 1 is a schematic flow chart of a three-dimensional image registration method based on an unsupervised learning method according to an embodiment of the present invention. As shown in fig. 1, the method includes:
step S1, image acquisition: acquiring a three-dimensional medical image, wherein the three-dimensional medical image comprises a fixed image and an image to be registered;
step S2, training algorithm network:
step S21, down-sampling the image to obtain a down-sampled image;
step S22, inputting the down-sampled fixed image and the image to be registered into a cascade registration sub-network, and aligning the displacement deviation parts in the two images by using the cascade registration sub-network;
step S23, inputting the image output by the cascade registration sub-network into the full connection layer, and integrating the image characteristics;
step S24, bringing the image output by the full connection layer into an LOSS function, adding and summing the output results of the function, measuring the optimization degree of the algorithm network by using a summation value, stopping training and obtaining an image registration network structure after training if the summation value reaches a preset threshold value, and adjusting parameters to perform iterative training on the algorithm network if the summation value does not reach the preset threshold value;
step S3, image registration: and inputting the three-dimensional image to be registered into the image registration network structure to obtain a registered three-dimensional image.
For a clearer explanation of the above three-dimensional image registration method based on the unsupervised learning method, the following description is made in detail with reference to each step.
Step S1, image acquisition:
and acquiring two three-dimensional medical images with the first size, wherein one of the three-dimensional medical images is a fixed image, and the other one is an image to be registered. Wherein the first size may be 128 x 128.
The input of the image registration problem is composed of two images I 1 、I 2 Composition, function omega → R of the two images c . Wherein Ω is R n C represents the number of channels. Since the present invention is mainly used for three-dimensional medical image registration, the present invention is limited to the case where Ω is a cube and c is 1 (grayscale image). In particular, this means that
And each image is a function omega → R, the aim of image registration is to find a displacement field (or flow field) f 1,2 :Ω→R 3 Thus, there are:
I 1 (x)≈I 2 (x+f(x))；
where the exact meaning of "≈" depends on the specific application. Field f {1,2} Is called slave I 1 To I 2 Of which is represented by I 1 Each voxel in (1) is in 2 Of (c) is used. Will warp (I) 2 And f) image I warped according to f 2 I.e. warping (I) 2 ，f)(x)＝I 2 (x + f (x)). The above objective can be rewritten as finding f-max I 1 And warpage (I) 2 And f) similarity between them.
Step S2, training algorithm network:
fig. 2 is a schematic diagram of a three-dimensional image registration network training process based on unsupervised learning according to an embodiment of the present invention. As shown in fig. 2, the algorithm network is an unsupervised end-to-end registration network, and is composed of cascaded registration sub-networks and convolutional layers in multiple CNNs; wherein after each cascaded registration sub-network, the moving image is warped; when iterative training of the algorithm network is carried out, the heterogeneity between the fixed image and each warped image is combined with the regularization loss value of the sub-network predicted flow to guide.
The warping operation, i.e. the sampler in the STN, can be differentiated due to the three-line interpolation. The warping operation propagates the gradient back to the input image and input flow field, which is important for training the cascade network. These subnetworks are trained to work cooperatively to continuously and gradually align moving images.
For better performance, an initial rigid transformation is usually applied as a global alignment before predicting the dense flow field. The preprocessing stage is integrated into a top-level sub-network, and the integrated affine registration sub-network not only has negligible running time, but also has better performance than the traditional affine stage.
In step S21, the down-sampling of the two images to obtain a down-sampled image includes:
inputting the two three-dimensional medical images with the first size into a first convolution layer of an algorithm network for down-sampling to obtain an image with a second size; wherein the second size may be 43 × 43.
Inputting the image with the second size into a second convolution layer of the algorithm network for down-sampling to obtain an image with a third size; wherein the second convolutional layer adopts an Affinine sub-network. The third size may be 23 × 23.
In step S22, the cascaded registration subnetworks are 4. Namely, the fixed image after down sampling and the image to be registered are input into 4 cascade registration sub-networks, and the parts of displacement deviation in the two images are aligned by utilizing the cascade registration sub-networks
In step S23, the features extracted in the previous step can be integrated and the class probability to which the input image belongs can be calculated using the full-link layer.
In step S24, the algorithm network brings the output of the full connection layer into the following 4 LOSS functions, and the sum of the added values of the function output results is used as an index to measure how well the whole algorithm network is optimized. If the output result does not reach the preset threshold, the algorithm network can carry out self iteration, the LOSS function is reversely differentiated in the iteration process, the parameters of the network are updated by using a gradient descent method, and then the steps are operated again until the function calculation result reaches the preset threshold.
In order to train the model in an unsupervised manner, the invention measures the (non-) similarity between moving images distorted by the spatial transformer and stationary images. There is a wealth of research in similarity metrics applicable to medical image registration. Furthermore, regularization losses are introduced to prevent flow field unreal or overfitting. In the present embodiment, Ω denotes a cube (or grid) defining an input image.
The loss function used in the experiment is described below:
the LOSS function comprises correlation coefficient LOSS, orthogonality LOSS, decisive LOSS and total variation LOSS; wherein, the first and the second end of the pipe are connected with each other,
1. loss of correlation coefficient:
in the loss of correlation coefficient, the covariance between two images is defined as:
wherein, Cov [ I ] 1 ,I 2 ]Is the covariance of the image, I 1 、I 2 Image data of two images are respectively, x is a horizontal axis coordinate in omega, y is a vertical axis coordinate in omega, and omega is a cube defining an input image;
the correlation coefficient is defined as:
wherein, CorrCoef [ I ] 1 ,I 2 ]Is a correlation coefficient;
the image is treated as a random variable whose sample space is a point with voxel values. The correlation coefficient is in the range of-1, which measures the degree of linear correlation of the two images and reaches ± 1 if and only if the two are linear functions of each other. Applying a non-derived linear function to any one of the images does not change their correlation coefficient, and therefore the measurement method is more robust than the L2 loss. For real-world images, the correlation coefficient should be non-negative (unless one of the images is negative).
The loss of correlation coefficient is defined as:
L corr (I 1 ,I 2 )＝1-CorrCoef[I 1 ,I 2 ]
wherein L is corr (I 1 ,I 2 ) Is the loss of correlation coefficient;
2. loss of orthogonality:
the present invention is directed to medical image registration, where typically only a small amount of scaling and rotation of the input images is required for affinity alignment. It is therefore desirable to penalize the network for producing transforms that are too non-rigid. In the loss of orthogonality, a loss of non-orthogonality to I + a is introduced, where a represents a transformation matrix produced by the affinity registration network; let λ be the singular value of I + a, the loss of orthogonality is:
wherein L is ortho For loss of orthogonality, λ i Are singular values; the larger the deviation of I + A from the orthogonal matrix is, the larger the orthogonality loss is, and if I + A is the orthogonal matrix, the corresponding value is zero;
3. the decisive loss:
in deterministic loss, assuming that the images are taken with the same singularity, let det (I + a) >0, the determinant loss is set as:
L det ＝(-1+det(I+A)) 2 ；
wherein L is det For determinant losses, det (I + a) is the determinant value of I + a;
4. total variation loss (plateau term):
in total variation loss, the normalization process will be performed using a loss function for dense flow fields to prevent discontinuities:
wherein L is TV For total variation loss, x is the abscissa in Ω, c i Is the probability that voxel i belongs to the foreground. e1,2,3 constitutes the natural basis for R3, which is also called the L2 stationary term.
Referring again to fig. 2, each subnetwork is responsible for aligning the fixed image and the current moving image. After each subnetwork, the moving image is warped with the predicted traffic, and the warped image is fed into the next cascaded subnetwork. The composition of the flow field yields the final estimate; in the training process, all layers are differentiable, and the algorithm network is trained through the back propagation of the gradient.
Step S3, image registration:
fig. 3 is a schematic diagram of a prediction process of a three-dimensional image registration network based on unsupervised learning according to an embodiment of the present invention. As shown in fig. 3, the three-dimensional image to be registered is input to the image registration network structure, so as to obtain a registered three-dimensional image.
It should be noted that although the operations of the method of the present invention have been described in the above embodiments and the accompanying drawings in a particular order, this does not require or imply that these operations must be performed in this particular order or that all of the illustrated operations must be performed to achieve desirable results. Additionally or alternatively, certain steps may be omitted, multiple steps combined into one step execution, and/or one step broken down into multiple step executions.
For a clearer explanation of the above three-dimensional image registration method based on the unsupervised learning method, a specific embodiment is described below, however, it should be noted that the embodiment is only for better explaining the present invention, and should not be construed as an undue limitation to the present invention.
Taking the registration of three-dimensional medical images as an example:
step S101, two three-dimensional medical images with a size of 128 × 128 are input to an algorithm network, one is a fixed image and the other is an image to be registered, and the two images are down-sampled to 43 × 43 at a first layer of the algorithm network.
Step S102, when the two images are input into an affinity subnetwork at the second layer of the algorithm network, the images are down-sampled to 23 × 23.
In step S103, the down-sampled images are input into 4 cascaded registration sub-networks, in which the parts with large displacement deviation in the two images are aligned step by step.
In step S104, after the image is output from the cascade registration sub-network, the result is input to the last layer (fully connected layer) of the algorithm network.
Step S105, the algorithm network brings the output of the full connection layer into the previous 4 LOSS functions, the sum of the numerical values obtained by adding the function output results is used as an index for measuring the optimization quality of the whole algorithm network, if the output result does not reach a preset threshold value, the algorithm network can carry out self-iteration, the LOSS functions are reversely derived in the iteration process, parameters of the network are updated by using a gradient descent method, and then the steps are operated again until the function calculation result reaches the preset threshold value.
And S106, inputting a three-dimensional image to be registered, and performing image registration by adopting a trained network structure, wherein the network output is a registered three-dimensional picture.
Comparing the image registration of the invention with the prior art, the image registration method based on the deep learning in the prior art is based on the supervised learning, the method needs a large amount of labeling work, but the labeling work of the three-dimensional image is very difficult, therefore, the invention provides the three-dimensional registration method based on the unsupervised learning, and experiments show that the method of the invention also achieves the most advanced performance, and compared with the traditional medical image registration method, the speed of 880 times (or 3.3 times without GPU acceleration) is improved.
Having described the method of an exemplary embodiment of the present invention, a three-dimensional image registration system based on an unsupervised learning method of an exemplary embodiment of the present invention will be described next with reference to fig. 4.
The implementation of the three-dimensional image registration system based on the unsupervised learning method can be referred to the implementation of the above method, and repeated details are not repeated. The term "module" or "unit" used hereinafter may be a combination of software and/or hardware that implements a predetermined function. Although the means described in the embodiments below are preferably implemented in software, an implementation in hardware, or a combination of software and hardware is also possible and contemplated.
Based on the same inventive concept, the invention also provides a three-dimensional image registration system based on the unsupervised learning method, as shown in fig. 4, the system comprises:
an image acquisition module 410, configured to acquire a three-dimensional medical image, where the three-dimensional medical image includes a fixed image and an image to be registered;
the algorithm network training module 420 is configured to perform downsampling on the image to obtain a downsampled image; inputting the down-sampled fixed image and the image to be registered into a cascade registration sub-network, and aligning the displacement deviation parts in the two images by using the cascade registration sub-network; inputting the image output by the cascade registration sub-network into a full connection layer, and integrating image characteristics; bringing the image output by the full connection layer into an LOSS function, adding and summing the output results of the functions, measuring the optimization degree of the algorithm network by using a summation value, stopping training and obtaining an image registration network structure after training if the summation value reaches a preset threshold value, and adjusting parameters to perform iterative training on the algorithm network if the summation value does not reach the preset threshold value;
and an image registration module 430, configured to input the three-dimensional image to be registered to the image registration network structure, so as to obtain a registered three-dimensional image.
In this embodiment, the image capturing module 410 is specifically configured to:
and acquiring two three-dimensional medical images with the first size, wherein one of the three-dimensional medical images is a fixed image, and the other one is an image to be registered.
In this embodiment, the algorithm network is an unsupervised end-to-end registration network, and is composed of cascaded registration subnetworks and convolutional layers in multiple CNNs; wherein after each cascaded registration sub-network, the moving image is warped; when iterative training of the algorithm network is carried out, the heterogeneity between the fixed image and each warped image is combined with the regularization loss value of the sub-network predicted flow to guide.
In this embodiment, the algorithm network training module 420 is specifically configured to:
inputting the two three-dimensional medical images with the first size into a first convolution layer of an algorithm network for down-sampling to obtain an image with a second size;
inputting the image with the second size into a second convolution layer of the algorithm network for down-sampling to obtain an image with a third size; wherein the second convolutional layer adopts an Affinine sub-network.
In this embodiment, the number of cascaded registration subnetworks is 4.
In the present embodiment, the LOSS function includes a correlation coefficient LOSS, an orthogonality LOSS, a deterministic LOSS and a total variation LOSS; wherein the content of the first and second substances,
in the loss of correlation coefficient, the covariance between two images is defined as:
wherein, Cov [ I ] 1 ,I 2 ]Is the covariance of the image, I 1 、I 2 Image data of two images are respectively, x is a horizontal axis coordinate in omega, y is a vertical axis coordinate in omega, and omega is a cube defining an input image;
the correlation coefficient is defined as:
wherein, CorrCoef [ I ] 1 ,I 2 ]Is a correlation coefficient;
the loss of correlation coefficient is defined as:
L corr (I 1 ,I 2 )＝1-CorrCoef[I 1 ,I 2 ]
wherein L is corr (I 1 ,I 2 ) Is the loss of correlation coefficient;
in loss of orthogonality, a loss of non-orthogonality to I + a is introduced, where a represents a transformation matrix produced by the affinity registration network; let λ be the singular value of I + a, the loss of orthogonality is:
wherein L is ortho For loss of orthogonality, λ i Are singular values; the larger the deviation of I + A from the orthogonal matrix is, the larger the orthogonality loss is, and if I + A is the orthogonal matrix, the corresponding value is zero;
in deterministic loss, assuming that the images are taken with the same singularity, let det (I + a) >0, the determinant loss is set as:
L det ＝(-1+det(I+A)) 2 ；
wherein L is det Det (I + a) is the determinant value of I + a for determinant losses;
in total variation loss, normalization will be performed for dense flow fields using a loss function:
wherein L is TV For total variation loss, x is the abscissa in Ω, c i Is the probability that voxel i belongs to the foreground.
In the iterative training process of the algorithm network training module 420, if the summation value does not reach the preset threshold, adjusting the parameter to perform iterative training on the algorithm network, further comprising:
in the iterative training process, parameters of the algorithm network are updated by a gradient descent method for the LOSS function reverse derivation, and the operation is carried out again until the summation value corresponding to the LOSS function reaches a preset threshold value.
It should be noted that although several modules of the three-dimensional image registration system based on the unsupervised learning method are mentioned in the above detailed description, such partitioning is merely exemplary and not mandatory. Indeed, the features and functionality of two or more of the modules described above may be embodied in one module according to embodiments of the invention. Conversely, the features and functions of one module described above may be further divided into embodiments by a plurality of modules.
Based on the aforementioned inventive concept, as shown in fig. 5, the present invention further proposes a computer device 500, which comprises a memory 510, a processor 520 and a computer program 530 stored on the memory 510 and executable on the processor 520, wherein the processor 520 executes the computer program 530 to implement the aforementioned three-dimensional image registration method based on the unsupervised learning method.
Based on the foregoing inventive concept, the present invention proposes a computer-readable storage medium storing a computer program which, when executed by a processor, implements the foregoing three-dimensional image registration method based on an unsupervised learning method.
The three-dimensional image registration method and the three-dimensional image registration system based on the unsupervised learning method, provided by the invention, utilize a cascade registration subnetwork to align the displacement deviation parts in two images, improve the performance of registering most displacement images, utilize a full-connection layer to integrate image characteristics, bring images output by the full-connection layer into an LOSS function, add and sum the output results of the functions, utilize a sum value to measure the optimization degree of an algorithm network, iteratively train the algorithm network according to the optimization degree, improve the registration performance by inverting LOSS in the training process, finally utilize the trained image registration network to register three-dimensional images needing registration, do not need to carry out a large amount of standard work in the registration process, and obviously improve the registration efficiency.
As will be appreciated by one skilled in the art, embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present invention has been described with reference to flowchart illustrations and/or block diagrams of methods and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
Finally, it should be noted that: although the present invention has been described in detail with reference to the foregoing embodiments, it should be understood by those skilled in the art that the following descriptions are only illustrative and not restrictive, and that the scope of the present invention is not limited to the above embodiments: those skilled in the art can still make modifications or changes to the embodiments described in the foregoing embodiments, or make equivalent substitutions for some features, within the scope of the disclosure; such modifications, changes or substitutions do not depart from the spirit and scope of the embodiments of the present invention, and they should be construed as being included therein. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.
Claims (10)
1. A three-dimensional image registration method based on an unsupervised learning method is characterized by comprising the following steps:
image acquisition: acquiring a three-dimensional medical image, wherein the three-dimensional medical image comprises a fixed image and an image to be registered;
training an algorithm network: carrying out down-sampling on the image to obtain a down-sampled image;
inputting the down-sampled fixed image and the image to be registered into a cascade registration sub-network, and aligning the displacement deviation parts in the two images by using the cascade registration sub-network;
inputting the image output by the cascade registration sub-network into a full connection layer, and integrating image characteristics;
bringing the images output by the full connection layer into an LOSS function, adding and summing the output results of the functions, measuring the optimization degree of the algorithm network by using a summation value, stopping training and obtaining a trained image registration network structure if the summation value reaches a preset threshold value, and adjusting parameters to carry out iterative training on the algorithm network if the summation value does not reach the preset threshold value;
image registration: and inputting the three-dimensional image to be registered into the image registration network structure to obtain a registered three-dimensional image.
2. The unsupervised learning method-based three-dimensional image registration method according to claim 1, wherein the image acquisition comprises:
and acquiring two three-dimensional medical images with the first size, wherein one of the three-dimensional medical images is a fixed image, and the other one is an image to be registered.
3. The unsupervised learning method-based three-dimensional image registration method of claim 2, wherein the algorithm network is an unsupervised end-to-end registration network consisting of cascaded registration sub-networks and convolutional layers in a plurality of CNNs; wherein after each cascaded registration sub-network, the moving image is warped; when iterative training of the algorithm network is carried out, the heterogeneity between the fixed image and each warped image is combined with the regularization loss value of the sub-network predicted flow to guide.
4. The three-dimensional image registration method based on the unsupervised learning method as claimed in claim 3, wherein down-sampling two images to obtain a down-sampled image comprises:
inputting the two three-dimensional medical images with the first size into a first convolution layer of an algorithm network for down-sampling to obtain an image with a second size;
inputting the image with the second size into a second convolution layer of an algorithm network for down-sampling to obtain an image with a third size; wherein the second convolutional layer adopts an Affinine sub-network.
5. The unsupervised learning method-based three-dimensional image registration method according to claim 4, wherein the number of cascaded registration subnetworks is 4.
6. The unsupervised learning method-based three-dimensional image registration method according to claim 5, wherein the LOSS function includes correlation coefficient LOSS, orthogonality LOSS, decisive LOSS and total variation LOSS; wherein the content of the first and second substances,
in the loss of correlation coefficient, the covariance between two images is defined as:
wherein, Cov [ I ] 1 ,I 2 ]Is the covariance of the image, I 1 、I 2 Image data of two images are respectively, x is a horizontal axis coordinate in omega, y is a vertical axis coordinate in omega, and omega is a cube defining an input image;
the correlation coefficient is defined as:
wherein, CorrCoef [ I ] 1 ,I 2 ]Is a correlation coefficient;
the loss of correlation coefficient is defined as:
L corr (I 1 ,I 2 )＝1-CorrCoef[I 1 ,I 2 ]
wherein L is corr (I 1 ,I 2 ) Is the loss of correlation coefficient;
in the loss of orthogonality, a loss of non-orthogonality to I + a is introduced, where a represents a transformation matrix produced by the affinity registration network; let λ be the singular value of I + a, the loss of orthogonality is:
wherein L is ortho For loss of orthogonality, λ i Are singular values; the larger the deviation of I + A from the orthogonal matrix is, the larger the orthogonality loss is, and if I + A is the orthogonal matrix, the corresponding value is zero;
in deterministic loss, assuming that the images are taken with the same singularity, let det (I + a) >0, the determinant loss is set as:
L det ＝(-1+det(I+A)) 2 ；
wherein L is det For determinant losses, det (I + a) is the determinant value of I + a;
in total variation loss, normalization will be performed for dense flow fields using a loss function:
wherein L is TV For total variation loss, x is the abscissa in Ω, c i Is the probability that voxel i belongs to the foreground.
7. The unsupervised learning method-based three-dimensional image registration method according to claim 6, wherein if the summation value does not reach a preset threshold, the parameter is adjusted to perform iterative training on the algorithm network, and further comprising:
in the iterative training process, parameters of the algorithm network are updated by a gradient descent method for the LOSS function reverse derivation, and the operation is carried out again until the summation value corresponding to the LOSS function reaches a preset threshold value.
8. A three-dimensional image registration system based on an unsupervised learning method is characterized by comprising the following steps:
the image acquisition module is used for acquiring a three-dimensional medical image, wherein the three-dimensional medical image comprises a fixed image and an image to be registered;
the algorithm network training module is used for carrying out down-sampling on the image to obtain a down-sampled image; inputting the down-sampled fixed image and the image to be registered into a cascade registration sub-network, and aligning the displacement deviation parts in the two images by using the cascade registration sub-network; inputting the image output by the cascade registration sub-network into a full connection layer, and integrating image characteristics; bringing the image output by the full connection layer into an LOSS function, adding and summing the output results of the functions, measuring the optimization degree of the algorithm network by using a summation value, stopping training and obtaining an image registration network structure after training if the summation value reaches a preset threshold value, and adjusting parameters to perform iterative training on the algorithm network if the summation value does not reach the preset threshold value;
and the image registration module is used for inputting the three-dimensional image to be registered into the image registration network structure to obtain a registered three-dimensional image.
9. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the method of any one of claims 1 to 7 when executing the computer program.
10. A computer-readable storage medium, characterized in that the computer-readable storage medium stores a computer program which, when executed by a processor, implements the method of any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110274132.3A CN115082530A (en) | 2021-03-15 | 2021-03-15 | Three-dimensional image registration method and system based on unsupervised learning method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110274132.3A CN115082530A (en) | 2021-03-15 | 2021-03-15 | Three-dimensional image registration method and system based on unsupervised learning method |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115082530A true CN115082530A (en) | 2022-09-20 |
Family
ID=83241665
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110274132.3A Pending CN115082530A (en) | 2021-03-15 | 2021-03-15 | Three-dimensional image registration method and system based on unsupervised learning method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115082530A (en) |
-
2021
- 2021-03-15 CN CN202110274132.3A patent/CN115082530A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Gu et al. | Self-guided network for fast image denoising | |
| CN108846445B (en) | Image processing method | |
| US11403735B2 (en) | Deep-learning based structure reconstruction method and apparatus | |
| US10755139B2 (en) | Random sample consensus for groups of data | |
| WO2021253686A1 (en) | Feature point tracking training and tracking methods, apparatus, electronic device, and storage medium | |
| CN111968138B (en) | Medical image segmentation method based on 3D dynamic edge insensitivity loss function | |
| CN112819853B (en) | Visual odometer method based on semantic priori | |
| CN116150888A (en) | Digital twin-based aeroengine multi-field coupling simulation method and system | |
| US8548225B2 (en) | Point selection in bundle adjustment | |
| Sun et al. | Enhancing gland segmentation in colon histology images using an instance-aware diffusion model | |
| Kuang et al. | Correction of spurious phase sign in single closed-fringe demodulation using transformer based Swin-ResUnet | |
| CN115082530A (en) | Three-dimensional image registration method and system based on unsupervised learning method | |
| US20230132630A1 (en) | Apparatus and method with neural network training based on knowledge distillation | |
| CN116228542A (en) | Image super-resolution reconstruction method based on trans-scale non-local attention mechanism | |
| CN116385520A (en) | Wear surface topography luminosity three-dimensional reconstruction method and system integrating full light source images | |
| CN113450394B (en) | Different-size image registration method based on Siamese network | |
| Lin et al. | Click-Pixel Cognition Fusion Network With Balanced Cut for Interactive Image Segmentation | |
| Wang et al. | Depth estimation from a single-shot fringe pattern based on DD-Inceptionv2-UNet | |
| Khoroshiltseva et al. | Relaxation labeling meets gans: solving jigsaw puzzles with missing borders | |
| CN112991406A (en) | Method for constructing brain atlas based on differential geometry technology | |
| CN117271988B (en) | Tensor wheel-based high-dimensional signal recovery method and device | |
| Jiang et al. | Segmentation, tracking, and sub-cellular feature extraction in 3D time-lapse images | |
| Wang et al. | Dyeing creation: a textile pattern discovery and fabric image generation method | |
| CN116129206B (en) | Processing method and device for image decoupling characterization learning and electronic equipment | |
| Li et al. | Efficient completion for corrupted low-rank images via alternating direction method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |