CN110599528A - Unsupervised three-dimensional medical image registration method and system based on neural network - Google Patents

Unsupervised three-dimensional medical image registration method and system based on neural network Download PDF

Info

Publication number
CN110599528A
CN110599528A CN201910828807.7A CN201910828807A CN110599528A CN 110599528 A CN110599528 A CN 110599528A CN 201910828807 A CN201910828807 A CN 201910828807A CN 110599528 A CN110599528 A CN 110599528A
Authority
CN
China
Prior art keywords
image
layer
feature map
slice
sampling
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN201910828807.7A
Other languages
Chinese (zh)
Other versions
CN110599528B (en
Inventor
赵秀阳
马英君
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
University of Jinan
Original Assignee
University of Jinan
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by University of Jinan filed Critical University of Jinan
Priority to CN201910828807.7A priority Critical patent/CN110599528B/en
Publication of CN110599528A publication Critical patent/CN110599528A/en
Application granted granted Critical
Publication of CN110599528B publication Critical patent/CN110599528B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/30—Determination of transform parameters for the alignment of images, i.e. image registration
    • G06T7/33—Determination of transform parameters for the alignment of images, i.e. image registration using feature-based methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Computational Linguistics (AREA)
  • General Health & Medical Sciences (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Artificial Intelligence (AREA)
  • Biomedical Technology (AREA)
  • Biophysics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Data Mining & Analysis (AREA)
  • Evolutionary Computation (AREA)
  • Health & Medical Sciences (AREA)
  • Molecular Biology (AREA)
  • Computing Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Image Processing (AREA)
  • Image Analysis (AREA)

Abstract

The invention provides an unsupervised three-dimensional medical image registration method and system based on a neural network, comprising the following steps of: collecting an image; preprocessing the acquired three-dimensional medical image; training a neural network to obtain a trained neural network model; and inputting the medical image to be registered into the trained neural network model for registration to obtain and output a registered image. Wherein the invention is based on a deformation fieldDeformation fieldDeformation fieldAnd based on the deformed imageDeformed imageDeformed imageDeformed imageComputing a fixed image I using a loss functionFAnd the deformed imageAnd performing back propagation optimization on the neural network until the calculated loss function value is not reduced or the network training reaches a preset training iteration number, and finishing the neural network training to obtain a trained neural network model. The invention is used for realizing medical image registration.

Description

Unsupervised three-dimensional medical image registration method and system based on neural network
Technical Field
The invention relates to the field of medical image registration, in particular to an unsupervised three-dimensional medical image registration method and system based on a neural network, which are mainly used for registering three-dimensional human brain images.
Background
Medical image registration, refers to seeking a spatial transformation or series of spatial transformations for one medical image to bring it into spatial correspondence with a corresponding point on another medical image. This coincidence means that the same anatomical point on the body has the same spatial position on the two matching images. Generally, an Image to be registered is called a floating Image (Moving Image), and a transformation target Image is called a fixed Image or a reference Image (Fi × ed Image).
There are currently many open source software available for medical image segmentation, registration, etc., for example: FreeSprofer can be used to analyze and visualize structural and functional neuroimaging data from slices or time series, which has better performance in skull stripping, B1 bias field correction, grey white matter segmentation and morphological difference measurement; FSL, similar to fresufer, allows for comprehensive analysis of FMRI, MRI and DTI brain imaging data; the ITK software package can be used for the segmentation and registration of multi-dimensional images; the NiftyReg realizes rigid body, affine and nonlinear registration methods of nifti images, and simultaneously supports GPU operation; elasti x is ITK-based open source software, including common algorithms for processing medical image registration; ANTS is used as a current better registration tool, and can realize deformable registration of differential homoembryo. The above registration tools are based on conventional registration methods to fit the deformation of the image. In addition, currently, many deep learning-based registration methods are proposed in succession, such as: DIRNet, BIRNet, voxelmorph and the like utilize a neural network to obtain image transformation parameters, and then utilize a transformation network to transform the floating image to obtain a registration result.
Although the above registration tools and methods all achieve good results in image registration, the following problems still exist:
(1) some methods require manual labeling and supervision information, have high registration specialty requirements, and have relatively low registration rates. The marking of the feature points and the feature areas of the medical image and the acquisition of the image supervision information need to be finished by professional medical imaging doctors, which is very difficult for registration workers without related medical experience, meanwhile, the feature marks obtained by different doctors and the same doctor at different times may have differences, the manual marking process is time-consuming and labor-consuming, and the subjective judgment of the doctors has a great influence on the registration result.
(2) The registration accuracy is relatively low. Some methods proposed in recent years, although making great progress in the registration result, still have to be improved in accuracy.
Therefore, the invention provides an unsupervised three-dimensional medical image registration method and system based on a neural network, which are used for solving the problems.
Disclosure of Invention
In view of the above disadvantages of the prior art, the present invention provides an unsupervised three-dimensional medical image registration method and system based on a neural network, which are used for realizing rapid registration of medical images. But also to improve registration accuracy.
In a first aspect, the present invention provides an unsupervised three-dimensional medical image registration method based on a neural network, comprising the steps of:
l1, image acquisition: acquiring three-dimensional medical images from public data sets OASIS and ADNI, and/or: acquiring a three-dimensional medical image from a DICOM interface of a CT, MRI or ultrasonic imager;
l2, preprocessing the acquired three-dimensional medical image: including image segmentation, cropping, normalization processing, and affine alignment, andselecting any one image from affine aligned images as a fixed image IFThe rest of the image is taken as a floating image IM(ii) a Wherein the size of the cut images is consistent;
l3 based on the fixed image I obtained after preprocessingFAnd a floating image IMTraining a neural network to obtain a trained neural network model;
l4, inputting the medical image to be registered into the trained neural network model for registration to obtain and output a registration image of the medical image to be registered;
wherein in step L3, based on the fixed image I obtained after preprocessingFAnd a floating image IMTraining a neural network to obtain a trained neural network model, comprising:
s1, fixing image I obtained after preprocessingFAnd a floating image IMInputting the neural network as an input layer of the neural network, wherein each set of input data comprises the fixed image IFAnd one said floating image IM；
S2, inputting fixed image I in input layerFAnd a floating image IMPerforming down sampling to output a characteristic diagram;
the down-sampling comprises 3 down-sampling processes, a convolution calculation process with convolution kernel size of 3 multiplied by 3 and a LeakyReLU activation function calculation process after the 3 down-sampling processes; the 3 downsampling processes correspond to 3 downsampling process layers; the 3 down-sampling process layers are sequentially marked as a first down-sampling process layer, a second down-sampling process layer and a third down-sampling process layer according to the execution sequence of the down-sampling process; each downsampling process layer comprises a convolution layer with convolution kernel size of 3 multiplied by 3, a LeakyReLU activation function layer and a maximum pooling layer;
s3, respectively carrying out feature re-weighting on feature graphs output by LeakyReLU activation function layers in a first downsampling process layer, a second downsampling process layer and a third downsampling process layer corresponding to downsampling to obtain three weighted feature graphs, which are sequentially: a first weighted feature map, a second weighted feature map, and a third weighted feature map;
s4, performing 1 × 1 × 1 convolution on the feature map output in the step S2, and outputting a floating image IMTo a fixed picture IFDeformation field of
S5, inputting the characteristic diagram output in the step S2 into an UpSampling layer for UpSampling, wherein the UpSampling layer comprises 3 UpSampling process layers, each UpSampling process layer comprises an UpSampling layer and a convolutional layer with the convolutional kernel size of 3 multiplied by 3, and a LeakyReLU activation function layer is arranged behind the convolutional layer with the convolutional kernel size of 3 multiplied by 3 in each UpSampling process layer; the 3 upsampling process layers correspond to the 3 upsampling processes of the upsampling; the 3 upsampling process layers are sequentially marked as a first upsampling process layer, a second upsampling process layer and a third upsampling process layer according to the sequence of the 3 upsampling processes;
after the feature map output by the UpSampling layer of the first UpSampling process layer is fused with the third weighted feature map, the feature map is used as the input of the convolutional layer with the convolutional kernel size of 3 multiplied by 3 in the first UpSampling process layer; fusing a feature map output by the UpSamplling layer of the second up-sampling process layer with the second weighted feature map, and taking the feature map as the input of the convolution layer with the convolution kernel size of 3 multiplied by 3 in the second up-sampling process layer; fusing a feature map output by an UpSampling layer of a third up-sampling process layer with the first weighted feature map, and taking the feature map as the input of a convolution layer with the convolution kernel size of 3 multiplied by 3 in the third up-sampling process layer;
s6, respectively carrying out 1 multiplied by 1 convolution on the feature maps output by the first up-sampling process layer, the second up-sampling process layer and the third up-sampling process layer, and outputting the floating images I corresponding to the first up-sampling process layer, the second up-sampling process layer and the third up-sampling process layerMTo a fixed picture IFIn turn is a deformation fieldDeformation fieldAnd deformation field
S7, floating the image IMAnd the deformation field of the above outputInputting into a spatial transformation network, rendering floating images IMAnd the deformation field of the above outputInput to the spatial transformation network, floating image IMAnd the deformation field of the above outputInput to the spatial transformation network, floating image IMAnd the deformation field of the above outputInputting the image data into the space transformation network, and correspondingly obtaining a floating image I through the space transformation of the space transformation networkMThe corresponding deformed images are sequentially deformed imagesDeformed imageDeformed imageAnd the deformed image
S8 deformation field based on the outputDeformation fieldDeformation fieldAnd based on the deformed image obtainedDeformed imageDeformed imageDeformed imageComputing a fixed image I using a loss functionFAnd the deformed imagePerforming back propagation optimization on the neural network until the calculated loss function value is not reduced or the network training reaches a preset training iteration number, and finishing the neural network training to obtain the trained neural network model;
the computational expression of the loss function is:
in the formula (i), the first and second groups,represents the calculated loss function value, α and β are both constants, α + β is 1,is a regularization term, λ is a regularization control constant parameter,representing a predetermined image I fixed by saidFDown-sampling the obtained three-dimensional medical image; three-dimensional medical imageSequentially with the deformed imageEquivalent, three-dimensional medical image Are successively lower and are each smaller than the fixed image IFThe resolution of (a) of (b),representing said fixed image IFAnd the deformed imageA measure of the degree of similarity between the two,representing the three-dimensional medical imageAnd the deformed imageA measure of the degree of similarity between the two,representing the three-dimensional medical imageAnd the deformed imageA measure of the degree of similarity between the two,representing the three-dimensional medical imageAnd the deformed imageA measure of similarity between;the same similarity metric function is used.
Further, the step of implementing feature re-weighting in step S3 includes:
step S31, recording each feature graph output in the down sampling and to be subjected to feature re-weighting is a feature graph X, wherein X is equal to R(H×W×D)Slicing the feature map X in the D dimension of the feature map X, and processing each slice X ∈ R obtained by slicing by using a global average pooling strategy(H×W)And performing global average pooling to obtain a slice descriptor z of each slice X in the D dimension of the feature map X, wherein a specific formula of each slice descriptor z is as follows:
wherein (i, j) represents a pixel point on the slice x, and x (i, j) represents a gray value of the slice x at the pixel point (i, j);
step S32, obtaining a weight S of each slice X in the dimension D of the feature map X, where the calculation formula of the weight S of each slice X is as follows:
s＝σ(δ(z))，
wherein σ represents a sigmoid activation function, δ is a ReLU activation function, and z is a slice descriptor of the slice x obtained in step S31;
step S33, correspondingly loading each weight S obtained in step S32 to the corresponding slice, to obtain each slice X corresponding weighted slice in the D dimension of the feature map XWherein the feature re-weighting calculation formula corresponding to each slice x is as follows:
in the formula, Fscale(x, s) represents the multiplication between a slice x and its corresponding weight s;
step S34, based on the feature map X obtained in step S33, the reweighed slice corresponding to each slice X in the D dimension thereofCorrespondingly obtaining the re-weighted feature map corresponding to the feature map X
Furthermore, the similarity measure function adopts a cross-correlation function.
Furthermore, the space transformation network adopts an STN space transformation network.
Further, the preprocessing also comprises data enhancement;
the data enhancement comprises the following steps: respectively performing bending transformation on each obtained floating image to obtain a bent transformed image corresponding to each obtained floating image; the resulting warped images are newly added floating images.
In a second aspect, the present invention provides an unsupervised three-dimensional medical image registration system based on a neural network, comprising:
an image acquisition unit acquiring three-dimensional medical images from the public data sets OASIS and ADNI, and/or: acquiring a three-dimensional medical image from a DICOM interface of a CT, MRI or ultrasonic imager;
the image preprocessing unit is used for preprocessing the acquired three-dimensional medical image, comprises image segmentation, cutting, normalization processing and affine alignment, and selects any one image from the images subjected to affine alignment as a fixed image IFThe rest of the image is taken as a floating image IM(ii) a Wherein the size of the cut images is consistent;
a neural network training unit based on the fixed image I obtained after preprocessingFAnd a floating image IMTraining a neural network to obtain a trained neural network model;
the image registration unit is used for inputting the medical image to be registered into the trained neural network model for registration to obtain and output a registration image of the medical image to be registered;
wherein, the neural network training unit comprises:
an input module for preprocessing the obtained fixed image IFAnd a floating image IMInputting the neural network as an input layer of the neural network, wherein each set of input data comprises the fixed image IFAnd one said floating image IM；
A down-sampling module for fixed image I input in the input layerFAnd a floating image IMPerforming down sampling to output a characteristic diagram; the down-sampling comprises 3 down-sampling processes, a convolution calculation process with convolution kernel size of 3 multiplied by 3 and a LeakyReLU activation function calculation process after the 3 down-sampling processes; 3 downsampling processes correspond to 3 downsampling process layers; the 3 down-sampling process layers are sequentially marked as a first down-sampling process layer, a second down-sampling process layer and a third down-sampling process layer according to the execution sequence of the down-sampling process; each downsampling process layer comprises a convolution layer with convolution kernel size of 3 multiplied by 3, a LeakyReLU activation function layer and a maximum pooling layer;
the reweighting module is used for respectively performing characteristic reweighting on the characteristic graphs output by the LeakyReLU activation function layer in the first downsampling process layer, the second downsampling process layer and the third downsampling process layer corresponding to downsampling to obtain three weighted characteristic graphs, and the three weighted characteristic graphs are sequentially: a first weighted feature map, a second weighted feature map, and a third weighted feature map;
a first deformation field output module for performing 1 × 1 × 1 convolution on the feature map output by the down-sampling module to output a floating image IMTo a fixed picture IFDeformation field of
The up-sampling module inputs the characteristic diagram output by the down-sampling module into an up-sampling layer for up-sampling, the up-sampling layer comprises 3 up-sampling process layers, each up-sampling process layer comprises an UpSamplling (namely up-sampling) layer and a convolution layer with convolution kernel size of 3 multiplied by 3, and a LeakyReLU activation function layer is respectively arranged behind the convolution layer with convolution kernel size of 3 multiplied by 3 in each up-sampling process layer; the 3 upsampling process layers correspond to the 3 upsampling processes of the upsampling; the 3 upsampling process layers are sequentially marked as a first upsampling process layer, a second upsampling process layer and a third upsampling process layer according to the sequence of the 3 upsampling processes; after the feature map output by the UpSampling layer of the first UpSampling process layer is fused with the third weighted feature map, the feature map is used as the input of the convolutional layer with the convolutional kernel size of 3 multiplied by 3 in the first UpSampling process layer; fusing a feature map output by the UpSamplling layer of the second up-sampling process layer with the second weighted feature map, and taking the feature map as the input of the convolution layer with the convolution kernel size of 3 multiplied by 3 in the second up-sampling process layer; fusing a feature map output by an UpSampling layer of a third up-sampling process layer with the first weighted feature map, and taking the feature map as the input of a convolution layer with the convolution kernel size of 3 multiplied by 3 in the third up-sampling process layer;
a second deformation field output module for performing 1 × 1 × 1 convolution on the feature maps output by the first, second and third upsampling process layers respectively to output the first, second and third upsampling process layersFloating image I corresponding to program layerMTo a fixed picture IFIn turn is a deformation fieldDeformation fieldAnd deformation field
A spatial transformation module for transforming the floating image IMAnd the deformation field of the above outputInputting into a spatial transformation network, rendering floating images IMAnd the deformation field of the above outputInput to the spatial transformation network, floating image IMAnd the deformation field of the above outputInput to the spatial transformation network, floating image IMAnd the deformation field of the above outputInputting the image data into the space transformation network, and correspondingly obtaining a floating image I through the space transformation of the space transformation networkMThe corresponding deformed images are sequentially deformed imagesDeformed imageDeformed imageAnd the figures after deformationImage
Neural network optimization module, deformation field based on the above outputDeformation fieldDeformation fieldAnd based on the deformed image obtainedDeformed imageDeformed imageDeformed imageComputing a fixed image I using a loss functionFAnd the deformed imagePerforming back propagation optimization on the neural network until the calculated loss function value is not reduced or the network training reaches a preset training iteration number, and finishing the neural network training to obtain the trained neural network model;
the computational expression of the loss function is:
in the formula (i), the first and second groups,represents the calculated loss function value, α and β are both constants, α + β is 1,is a regularization term, λ is a regularization control constant parameter,representing a predetermined image I fixed by saidFDown-sampling the obtained three-dimensional medical image; three-dimensional medical imageSequentially with the deformed imageEquivalent, three-dimensional medical imageAre successively lower and are each smaller than the fixed image IFThe resolution of (a) of (b),representing said fixed image IFAnd the deformed imageA measure of the degree of similarity between the two,representing the three-dimensional medical imageAnd the deformed imageA measure of the degree of similarity between the two,representing the three-dimensional medical imageAnd the deformed imageA measure of the degree of similarity between the two,representing the three-dimensional medical imageAnd the deformed imageA measure of similarity between;the same similarity metric function is used.
Further, the re-weighting module includes:
a descriptor obtaining module for recording that each feature graph to be subjected to feature re-weighting output in the down-sampling is X, and X belongs to R(H×W×D)Slicing the feature map X in the D dimension of the feature map X, and processing each slice X ∈ R obtained by slicing by using a global average pooling strategy(H×W)And performing global average pooling to obtain a slice descriptor z of each slice X in the D dimension of the feature map X, wherein a specific formula of each slice descriptor z is as follows:
wherein (i, j) represents a pixel point on the slice x, and x (i, j) represents a gray value of the slice x at the pixel point (i, j);
the slice weight calculation module is used for acquiring the weight s of each slice X on the D dimension of the feature map X, wherein the calculation formula of the weight s of each slice X is as follows:
s＝σ(δ(z))，
wherein, σ represents a sigmoid activation function, δ is a ReLU activation function, and z is a slice descriptor of a slice x obtained by the descriptor obtaining module;
a weighting module for correspondingly loading each weight s obtained by the slice weight calculation module on the corresponding slice to obtain each slice on the D dimension of the characteristic diagram X and the weighted sliceWherein the feature re-weighting calculation formula corresponding to each slice x is as follows:
in the formula, Fscale(x, s) represents the multiplication between a slice x and its corresponding weight s;
a weighted image acquisition module for obtaining a weighted slice corresponding to each slice X on D dimension of the feature image X based on the weighted imageCorrespondingly obtaining the re-weighted feature map corresponding to the feature map X
Furthermore, the similarity measure function adopts a cross-correlation function.
Furthermore, the space transformation network adopts an STN space transformation network.
Further, the preprocessing in the image preprocessing unit further includes data enhancement; wherein said data enhancement comprises: respectively performing bending transformation on each obtained floating image to obtain a bent transformed image corresponding to each obtained floating image; the resulting warped images are newly added floating images.
The beneficial effect of the invention is that,
(1) the unsupervised three-dimensional medical image registration method and the unsupervised three-dimensional medical image registration system both use an unsupervised registration mode, do not need any marking information and registration supervision information in the registration process, reduce the requirement of marking data and errors of artificial subjective judgment, contribute to image registration of medical workers without related medical experience to a certain extent, improve the registration rate to a certain extent, save the registration time and save manpower and material resources to a certain extent.
(2) According to the unsupervised three-dimensional medical image registration method and system based on the neural network, the feature graph obtained by the down-sampling path is weighted and fused to the up-sampling path according to the contribution degree of the feature graph through the feature weighted fusion and the loss function with the multi-level loss supervision function, the loss of the model is supervised from different resolution angles, more effective feature reuse and model supervision are achieved, and the registration accuracy is improved to a certain extent.
In addition, the invention has reliable design principle, simple structure and very wide application prospect.
Drawings
In order to more clearly illustrate the embodiments or technical solutions in the prior art of the present invention, the drawings used in the description of the embodiments or prior art will be briefly described below, and it is obvious for those skilled in the art that other drawings can be obtained based on these drawings without creative efforts.
FIG. 1 is a schematic flow diagram of a method of one embodiment of the invention.
FIG. 2 is a diagram of the deformation field obtained in the method of FIG. 1Deformation fieldDeformation fieldDeformation fieldSchematic diagram of the process indication.
FIG. 3 is a schematic block diagram of a system of one embodiment of the present invention.
Detailed Description
In order to make those skilled in the art better understand the technical solution of the present invention, the technical solution in the embodiment of the present invention will be clearly and completely described below with reference to the drawings in the embodiment of the present invention, and it is obvious that the described embodiment is only a part of the embodiment of the present invention, and not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
The following explains key terms appearing in the present invention.
Fig. 1, 2 are schematic flow diagrams of methods of one embodiment of the invention. This embodiment takes registration of 3D human brain images as an example.
As shown in fig. 1, the method 100 includes:
first, image acquisition.
A public data set of human brain images is downloaded from public data sets OASIS and ADNI.
In particular, those skilled in the art can also acquire the required three-dimensional medical image from the DICOM interface of CT, MRI or ultrasound imager.
Step two, pretreatment:
and preprocessing the three-dimensional medical image acquired in the first step.
The human brain data collected in the first step includes redundant parts such as neck, oral cavity, nasal cavity, skull and the like, and the sizes and gray scales of the redundant parts are different. For this purpose, standard preprocessing is performed on these image data. Firstly, the image is divided, the human brain is separated from the original data, and the obtained human brain image is cut to make the size of the human brain image consistent. Then carrying out normalizationThe voxel values are normalized to [0, 1 ]]. Followed by affine registration. Then selecting an image from the affine aligned data as a fixed image IFThe rest is used as a floating image IM。
In addition, in order to enhance the robustness and generalization capability of the neural network model, the preprocessing in the step further includes data enhancement, that is, each floating image obtained in the preprocessing process is subjected to bending transformation, and all images obtained after the bending transformation are newly added floating images. I.e. the new image resulting from the warping transformation also belongs to the floating image resulting from the preprocessing in this step.
In the embodiment, the warping transformation may adopt three kinds of warping transformations with different degrees, so as to realize data enhancement. In the concrete implementation, the number of the bending transformation with different degrees can be increased or decreased according to the actual situation.
All the floating images obtained by preprocessing in the step constitute a training set of the neural network.
And thirdly, training a neural network to obtain a trained neural network model. Fixed image I obtained after preprocessingFAnd training the neural network by the training set to obtain a trained neural network model.
And fourthly, inputting the three-dimensional medical image to be registered into the neural network model for registration, and finally obtaining and outputting a registration image of the three-dimensional medical image to be registered.
When the invention is used, image acquisition is firstly carried out, then the acquired image is preprocessed, and then a fixed image I is obtained based on preprocessingFAnd training the neural network by the training set to obtain a trained neural network model, inputting the three-dimensional medical image to be registered into the trained neural network model for registration, and finally obtaining and outputting a corresponding registration image.
The method has better effect when the number of the related images to be registered (in the embodiment, the human brain images to be registered) is relatively large. Specifically, after the trained neural network model is obtained, the images to be registered are respectively input into the trained neural network model, and then the registered images corresponding to the images to be registered are obtained. Specifically, after a neural network model is trained, a corresponding registration image can be correspondingly output every time an image to be registered is input; after the last registered image is output, the next image to be registered can be continuously input into the trained neural network model until the image registration of all the images to be registered is completed.
In the third step, the training of the neural network to obtain a trained neural network model includes:
s1, fixing image I obtained after preprocessingFAnd a floating image IMInputting the neural network as an input layer of the neural network, wherein each set of input data comprises the fixed image IFAnd one said floating image IM。
Each set of input data IFAnd IMAnd the 3D images are subjected to lossless splicing to form 2 channels and then are sent to a neural network input layer.
S2, inputting fixed image I in input layerFAnd a floating image IMDown-sampling and outputting the input fixed image IFAnd a floating image IMThe characteristic diagram of (1).
The down-sampling comprises 3 down-sampling processes, a convolution calculation process with convolution kernel size of 3 multiplied by 3 and a LeakyReLU activation function calculation process after the 3 down-sampling processes; the 3 downsampling processes correspond to 3 downsampling process layers; the 3 down-sampling process layers are sequentially marked as a first down-sampling process layer, a second down-sampling process layer and a third down-sampling process layer according to the execution sequence of the down-sampling process; each downsampling process layer includes a convolution layer with a convolution kernel size of 3 × 3 × 3, a LeakyReLU activation function layer, and a max pooling layer.
S3, respectively carrying out feature re-weighting on feature graphs output by LeakyReLU activation function layers in a first downsampling process layer, a second downsampling process layer and a third downsampling process layer corresponding to downsampling to obtain three weighted feature graphs, which are sequentially: a first weighted feature map, a second weighted feature map, and a third weighted feature map.
The step of implementing the feature re-weighting in step S3 includes:
step S31, recording each feature graph output in the down sampling and to be subjected to feature re-weighting is a feature graph X, wherein X is equal to R(H×W×D)Slicing the feature map X in the D dimension of the feature map X, and processing each slice X ∈ R obtained by slicing by using a global average pooling strategy(H×W)And performing global average pooling to obtain a slice descriptor z of each slice X in the D dimension of the feature map X, wherein a specific formula of each slice descriptor z is as follows:
wherein (i, j) represents a pixel point on the slice x, and x (i, j) represents a gray value of the slice x at the pixel point (i, j);
step S32, obtaining a weight S of each slice X in the dimension D of the feature map X, where the calculation formula of the weight S of each slice X is as follows:
s＝σ(δ(z))，
wherein σ represents a sigmoid activation function, δ is a ReLU activation function, and z is a slice descriptor of the slice x obtained in step S31;
step S33, correspondingly loading each weight S obtained in step S32 to the corresponding slice, to obtain each slice X corresponding weighted slice in the D dimension of the feature map XWherein the feature re-weighting calculation formula corresponding to each slice x is as follows:
in the formula, Fscale(x, s) represents the multiplication between a slice x and its corresponding weight s;
step S34, weighting the feature map X obtained in step S33 based on the weight corresponding to each slice X in the D dimension thereofIs sliced intoCorrespondingly obtaining the re-weighted feature map corresponding to the feature map X
Such as for the feature map X e R(H×W×D)The feature map X obtained in step S33 has X in D-dimension1、X2、…、XDSection X1、X2、…、XDThe corresponding feature map after the weighting is sequentiallyThere is a re-weighted feature map corresponding to feature map X
S4, S4, performing 1 × 1 × 1 convolution on the characteristic diagram output in the step S2, and outputting a floating image IMTo a fixed picture IFDeformation field of
S5, inputting the characteristic diagram output in the step S2 into an UpSampling layer for UpSampling, wherein the UpSampling layer comprises 3 UpSampling process layers, each UpSampling process layer comprises an UpSampling layer and a convolutional layer with the convolutional kernel size of 3 multiplied by 3, and a LeakyReLU activation function layer is arranged behind the convolutional layer with the convolutional kernel size of 3 multiplied by 3 in each UpSampling process layer; the 3 upsampling process layers correspond to the 3 upsampling processes of the upsampling; and the 3 upsampling process layers are sequentially marked as a first upsampling process layer, a second upsampling process layer and a third upsampling process layer according to the sequence of the 3 upsampling processes.
After the feature map output by the UpSampling layer of the first UpSampling process layer is fused with the third weighted feature map, the feature map is used as the input of the convolutional layer with the convolutional kernel size of 3 multiplied by 3 in the first UpSampling process layer; fusing a feature map output by the UpSamplling layer of the second up-sampling process layer with the second weighted feature map, and taking the feature map as the input of the convolution layer with the convolution kernel size of 3 multiplied by 3 in the second up-sampling process layer; and fusing the feature map output by the UpSamplling layer of the third up-sampling process layer with the first weighted feature map to be used as the input of the convolution layer with the convolution kernel size of 3 multiplied by 3 in the third up-sampling process layer.
S6, respectively carrying out 1 multiplied by 1 convolution on the feature maps output by the first up-sampling process layer, the second up-sampling process layer and the third up-sampling process layer, and outputting the floating images I corresponding to the first up-sampling process layer, the second up-sampling process layer and the third up-sampling process layerMTo a fixed picture IFIn turn is a deformation fieldDeformation fieldAnd deformation field
S7, floating the image IMAnd the deformation field of the above outputInputting into a spatial transformation network, rendering floating images IMAnd the deformation field of the above outputInput to the spatial transformation network, floating image IMAnd the deformation field of the above outputInput to the spatial transformation network, floating image IMAnd the deformation field of the above outputInputting the image data into the space transformation network, and correspondingly obtaining a floating image I through the space transformation of the space transformation networkMThe corresponding deformed images are sequentially deformed imagesDeformed imageDeformed imageAnd the deformed image
S8 deformation field based on the outputDeformation fieldDeformation fieldAnd based on the deformed image obtainedDeformed imageDeformed imageDeformed imageComputing a fixed image I using a loss functionFAnd the deformed imageAnd performing back propagation optimization on the neural network until the calculated loss function value is not reduced or the network training reaches a preset training iteration number, and finishing the neural network training to obtain the trained neural network model.
Wherein the computational expression of the loss function is:
in the formula (i), the first and second groups,represents the calculated loss function value, α and β are both constants, α + β is 1,is a regularization term, λ is a regularization control constant parameter,representing a predetermined image I fixed by saidFDown-sampling the obtained three-dimensional medical image; three-dimensional medical imageSequentially with the deformed imageEquivalent, three-dimensional medical imageAre successively lower and are each smaller than the fixed image IFThe resolution of (a) of (b),representing said fixed image IFAnd the deformed imageA measure of the degree of similarity between the two,representing the three-dimensional medical imageAnd the deformed imageA measure of the degree of similarity between the two,representing the three-dimensional medical imageAnd the deformed imageA measure of the degree of similarity between the two,representing the three-dimensional medical imageAnd the deformed imageA measure of similarity between;the same similarity metric function is used.
Optionally, the similarity metric function described in this embodiment uses a cross-correlation function; the space transformation network adopts STN space transformation network.
Optionally, the method 100 described in this embodiment may be implemented by using a U-Net neural network as a basic neural network structure.
Alternatively, the fusion mode of the feature maps involved in the present invention may be splicing fusion. In this embodiment, U-Net type channel dimension splicing and fusion can be adopted. In addition, when implementing the feature map fusion according to the present invention, a person skilled in the art may select another fusion method according to actual situations to perform fusion, for example, feature map fusion may be performed by a sum fusion (corresponding point addition) method.
In the present embodiment, the maximum pooling layer down-sampling factor is 2 × 2 × 2 at the time of down-sampling, and accordingly, it is possible to set in advanceFrom a fixed picture IFReduced to 1/2 to obtain,From a fixed picture IFReduced to 1/4 to obtain,From a fixed picture IFThe result of the reduction 1/8 is that,resolution of (2) >Resolution of (2) >＞0。
In conclusion, the unsupervised three-dimensional medical image registration method based on the neural network provided by the invention has the advantages that the use of the loss function realizes the unsupervised registration of the three-dimensional medical image, any marking information and registration supervision information are not needed in the registration process, the requirement of marking data and the errors of artificial subjective judgment are reduced, the image registration is facilitated for medical workers without related medical experience to a certain extent, the registration rate is improved to a certain extent, the registration time is saved, and meanwhile, the manpower and material resources are saved to a certain extent.
Referring to fig. 3, an unsupervised three-dimensional medical image registration system 200 based on neural network of the present invention comprises:
an image acquisition unit 201 that acquires three-dimensional medical images from public data sets OASIS and ADNI;
the image preprocessing unit 202 performs preprocessing on the acquired three-dimensional medical image, including image segmentation, clipping, normalization processing and affine alignment, and selects any one image from the images after affine alignment as a fixed image IFThe rest of the image is taken as a floating image IM(ii) a Wherein the size of the cut images is consistent;
a neural network training unit 203 based on the pre-processed fixed image IFAnd a floating image IMTraining a neural network to obtain a trained neural network model;
an image registration unit 204, which inputs the medical image to be registered into the trained neural network model for registration, and obtains and outputs a registration image of the medical image to be registered;
the neural network training unit 203 includes:
an input module for preprocessing the obtained fixed image IFAnd a floating image IMInputting the neural network as an input layer of the neural network, wherein each set of input data comprises the fixed image IFAnd one said floating image IM；
A down-sampling module for fixed image I input in the input layerFAnd a floating image IMPerforming down sampling to output a characteristic diagram; the down-sampling comprises 3 down-sampling processes, a convolution calculation process with convolution kernel size of 3 multiplied by 3 and a LeakyReLU activation function calculation process after the 3 down-sampling processes; 3 downsampling processes correspond to 3 downsampling process layers; the 3 down-sampling process layers are sequentially marked as a first down-sampling process layer, a second down-sampling process layer and a third down-sampling process layer according to the execution sequence of the down-sampling process; each downsampling process layer comprises a convolution layer with convolution kernel size of 3 × 3 × 3, and a convolution kernelA LeakyReLU activation function layer and a max pooling layer;
the reweighting module is used for respectively performing characteristic reweighting on the characteristic graphs output by the LeakyReLU activation function layer in the first downsampling process layer, the second downsampling process layer and the third downsampling process layer corresponding to downsampling to obtain three weighted characteristic graphs, and the three weighted characteristic graphs are sequentially: a first weighted feature map, a second weighted feature map, and a third weighted feature map;
a first deformation field output module for performing 1 × 1 × 1 convolution on the feature map output by the down-sampling module to output a floating image IMTo a fixed picture IFDeformation field of
The up-sampling module is used for inputting the characteristic diagram output by the down-sampling module into an up-sampling layer for up-sampling, the up-sampling layer comprises 3 up-sampling process layers, each up-sampling process layer comprises an UpSamplling layer and a convolution layer with the convolution kernel size of 3 multiplied by 3, and a LeakyReLU activation function layer is respectively arranged behind the convolution layer with the convolution kernel size of 3 multiplied by 3 in each up-sampling process layer; the 3 upsampling process layers correspond to the 3 upsampling processes of the upsampling; the 3 upsampling process layers are sequentially marked as a first upsampling process layer, a second upsampling process layer and a third upsampling process layer according to the sequence of the 3 upsampling processes; after the feature map output by the UpSampling layer of the first UpSampling process layer is fused with the third weighted feature map, the feature map is used as the input of the convolutional layer with the convolutional kernel size of 3 multiplied by 3 in the first UpSampling process layer; fusing a feature map output by the UpSamplling layer of the second up-sampling process layer with the second weighted feature map, and taking the feature map as the input of the convolution layer with the convolution kernel size of 3 multiplied by 3 in the second up-sampling process layer; fusing a feature map output by an UpSampling layer of a third up-sampling process layer with the first weighted feature map, and taking the feature map as the input of a convolution layer with the convolution kernel size of 3 multiplied by 3 in the third up-sampling process layer;
a second deformation field output module for outputting the first and second up-sampling process layersRespectively carrying out 1 multiplied by 1 convolution on the feature map output by the third up-sampling process layer, and outputting floating images I corresponding to the first up-sampling process layer, the second up-sampling process layer and the third up-sampling process layerMTo a fixed picture IFIn turn is a deformation fieldDeformation fieldAnd deformation field
A spatial transformation module for transforming the floating image IMAnd the deformation field of the above outputInputting into a spatial transformation network, rendering floating images IMAnd the deformation field of the above outputInput to the spatial transformation network, floating image IMAnd the deformation field of the above outputInput to the spatial transformation network, floating image IMAnd the deformation field of the above outputInputting the image data into the space transformation network, and correspondingly obtaining a floating image I through the space transformation of the space transformation networkMThe corresponding deformed images are sequentially deformed imagesDeformed imageDeformed imageAnd the deformed image
Neural network optimization module, deformation field based on the above outputDeformation fieldDeformation fieldAnd based on the deformed image obtainedDeformed imageDeformed imageDeformed imageComputing a fixed image I using a loss functionFAnd the deformed imagePerforming back propagation optimization on the neural network until the calculated loss function value is not reduced or the network training reaches a preset training iteration number, and finishing the neural network training to obtain the trained neural network model;
the computational expression of the loss function is:
in the formula (i), the first and second groups,represents the calculated loss function value, α and β are both constants, α + β is 1,is a regularization term, λ is a regularization control constant parameter,representing a predetermined image I fixed by saidFDown-sampling the obtained three-dimensional medical image; three-dimensional medical imageSequentially with the deformed imageEquivalent, three-dimensional medical imageAre successively lower and are each smaller than the fixed image IFThe resolution of (a) of (b),representing said fixed image IFAnd the deformed imageA measure of the degree of similarity between the two,representing the three-dimensional medical imageAnd the deformed imageA measure of the degree of similarity between the two,representing the three-dimensional medical imageAnd the deformed imageA measure of the degree of similarity between the two,representing the three-dimensional medical imageAnd the deformed imageA measure of similarity between;the same similarity metric function is used.
Wherein, the re-weighting module comprises:
a descriptor obtaining module for recording that each feature graph to be subjected to feature re-weighting output in the down-sampling is X, and X belongs to R(H×W×D)Slicing the feature map X in the D dimension of the feature map X, and processing each slice X ∈ R obtained by slicing by using a global average pooling strategy(H×W)And performing global average pooling to obtain a slice descriptor z of each slice X in the D dimension of the feature map X, wherein a specific formula of each slice descriptor z is as follows:
wherein (i, j) represents a pixel point on the slice x, and x (i, j) represents a gray value of the slice x at the pixel point (i, j);
the slice weight calculation module is used for acquiring the weight s of each slice X on the D dimension of the feature map X, wherein the calculation formula of the weight s of each slice X is as follows:
s＝σ(δ(z))，
wherein, σ represents a sigmoid activation function, δ is a ReLU activation function, and z is a slice descriptor of a slice x obtained by the descriptor obtaining module;
a weighting module for correspondingly loading each weight s obtained by the slice weight calculation module on the corresponding slice to obtain each slice on the D dimension of the characteristic diagram X and the weighted sliceWherein the feature re-weighting calculation formula corresponding to each slice x is as follows:
in the formula, Fscale(x, s) represents the multiplication between a slice x and its corresponding weight s;
a weighted image acquisition module for obtaining a weighted slice corresponding to each slice X on D dimension of the feature image X based on the weighted imageCorrespondingly obtaining the re-weighted feature map corresponding to the feature map X
Optionally, the similarity measure function is a cross-correlation function.
Optionally, the spatial transformation network adopts an STN spatial transformation network.
Optionally, the preprocessing described in the image preprocessing unit 202 further includes data enhancement; wherein said data enhancement comprises: respectively performing bending transformation on each obtained floating image to obtain a bent transformed image corresponding to each obtained floating image; the resulting warped images are newly added floating images.
The same and similar parts in the various embodiments in this specification may be referred to each other. In particular, for the system embodiment, since it is substantially similar to the method embodiment, the description is simple, and the relevant points can be referred to the description in the method embodiment.
Although the present invention has been described in detail by referring to the drawings in connection with the preferred embodiments, the present invention is not limited thereto. Various equivalent modifications or substitutions can be made on the embodiments of the present invention by those skilled in the art without departing from the spirit and scope of the present invention, and these modifications or substitutions are within the scope of the present invention/any person skilled in the art can easily conceive of the changes or substitutions within the technical scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims (10)

1. An unsupervised three-dimensional medical image registration method based on a neural network is characterized by comprising the following steps of:
l1, image acquisition: acquiring three-dimensional medical images from public data sets OASIS and ADNI, and/or: acquiring a three-dimensional medical image from a DICOM interface of a CT, MRI or ultrasonic imager;
l2, preprocessing the acquired three-dimensional medical image: the method comprises image segmentation, clipping, normalization processing and affine alignment, and any one image is selected from images after affine alignment to be used as a fixed image IFThe rest of the image is taken as a floating image IM(ii) a Wherein the size of the cut images is consistent;
l3 based on the fixed image I obtained after preprocessingFAnd a floating image IMTraining a neural network to obtain a trained neural network model;
l4, inputting the medical image to be registered into the trained neural network model for registration to obtain and output a registration image of the medical image to be registered;
wherein in step L3, based on the fixed image I obtained after preprocessingFAnd a floating image IMTraining a neural network to obtain a trained neural network model, comprising:
s1, fixing image I obtained after preprocessingFAnd a floating image IMInputting the neural network as an input layer of the neural network, wherein each set of input data comprises the fixed image IFAnd one said floating image IM；
S2, inputting fixed image I in input layerFAnd a floating image IMPerforming down sampling to output a characteristic diagram;
the down-sampling comprises 3 down-sampling processes, a convolution calculation process with convolution kernel size of 3 multiplied by 3 and a LeakyReLU activation function calculation process after the 3 down-sampling processes; the 3 downsampling processes correspond to 3 downsampling process layers; the 3 down-sampling process layers are sequentially marked as a first down-sampling process layer, a second down-sampling process layer and a third down-sampling process layer according to the execution sequence of the down-sampling process; each downsampling process layer comprises a convolution layer with convolution kernel size of 3 multiplied by 3, a LeakyReLU activation function layer and a maximum pooling layer;
s3, respectively carrying out feature re-weighting on feature graphs output by LeakyReLU activation function layers in a first downsampling process layer, a second downsampling process layer and a third downsampling process layer corresponding to downsampling to obtain three weighted feature graphs, which are sequentially: a first weighted feature map, a second weighted feature map, and a third weighted feature map;
s4, performing 1 × 1 × 1 convolution on the feature map output in the step S2, and outputting a floating image IMTo a fixed picture IFDeformation field of
S _5, inputting the feature map output in the step S2 into an UpSampling layer for UpSampling, wherein the UpSampling layer comprises 3 UpSampling process layers, each UpSampling process layer comprises an UpSampling layer and a convolutional layer with the convolutional kernel size of 3 multiplied by 3, and a LeakyReLU activation function layer is arranged behind the convolutional layer with the convolutional kernel size of 3 multiplied by 3 in each UpSampling process layer; the 3 upsampling process layers correspond to the 3 upsampling processes of the upsampling; the 3 upsampling process layers are sequentially marked as a first upsampling process layer, a second upsampling process layer and a third upsampling process layer according to the sequence of the 3 upsampling processes;
after the feature map output by the UpSampling layer of the first UpSampling process layer is fused with the third weighted feature map, the feature map is used as the input of the convolutional layer with the convolutional kernel size of 3 multiplied by 3 in the first UpSampling process layer; fusing a feature map output by the UpSamplling layer of the second up-sampling process layer with the second weighted feature map, and taking the feature map as the input of the convolution layer with the convolution kernel size of 3 multiplied by 3 in the second up-sampling process layer; fusing a feature map output by an UpSampling layer of a third up-sampling process layer with the first weighted feature map, and taking the feature map as the input of a convolution layer with the convolution kernel size of 3 multiplied by 3 in the third up-sampling process layer;
s6, respectively carrying out 1 multiplied by 1 convolution on the feature maps output by the first up-sampling process layer, the second up-sampling process layer and the third up-sampling process layer, and outputting the floating images I corresponding to the first up-sampling process layer, the second up-sampling process layer and the third up-sampling process layerMTo a fixed picture IFIn turn is a deformation fieldDeformation fieldAnd deformation field
S7, floating the image IMAnd the deformation field of the above outputInputting into a spatial transformation network, rendering floating images IMAnd the deformation field of the above outputInput to the spatial transformation network, floating image IMAnd the deformation field of the above outputInput to the spatial transformation network, floating image IMAnd the deformation field of the above outputInputting the image data into the space transformation network, and correspondingly obtaining a floating image I through the space transformation of the space transformation networkMThe corresponding deformed images are sequentially deformed imagesDeformed imageDeformed imageAnd the deformed image
S8 deformation field based on the outputDeformation fieldDeformation fieldAndbased on the deformed image obtainedDeformed imageDeformed imageDeformed imageComputing a fixed image I using a loss functionFAnd the deformed imagePerforming back propagation optimization on the neural network until the calculated loss function value is not reduced or the network training reaches a preset training iteration number, and finishing the neural network training to obtain the trained neural network model;
the computational expression of the loss function is:
in the formula (i), the first and second groups,represents the calculated loss function value, α and β are both constants, α + β is 1,is a regularization term, λ is a regularization control constant parameter,representing a predetermined map defined by said fixed mapLike IFDown-sampling the obtained three-dimensional medical image; three-dimensional medical imageSequentially with the deformed imageEquivalent, three-dimensional medical imageAre successively lower and are each smaller than the fixed image IFThe resolution of (a) of (b),representing said fixed image IFAnd the deformed imageA measure of the degree of similarity between the two,representing the three-dimensional medical imageAnd the deformed imageA measure of the degree of similarity between the two,representing the three-dimensional medical imageAnd the deformed imageA measure of the degree of similarity between the two,representing the three-dimensional medical imageAnd the deformed imageA measure of similarity between;the same similarity metric function is used.
2. The method for unsupervised three-dimensional medical image registration based on neural network as claimed in claim 1, wherein the step of implementing feature re-weighting in step S3 comprises:
step S31, recording each feature graph output in the down sampling and to be subjected to feature re-weighting is a feature graph X, wherein X is equal to R(H ×W×D)Slicing the feature map X in the D dimension of the feature map X, and processing each slice X ∈ R obtained by slicing by using a global average pooling strategy(H×W)And performing global average pooling to obtain a slice descriptor z of each slice X in the D dimension of the feature map X, wherein a specific formula of each slice descriptor z is as follows:
wherein (i, j) represents a pixel point on the slice x, and x (i, j) represents a gray value of the slice x at the pixel point (i, j);
step S32, obtaining a weight S of each slice X in the dimension D of the feature map X, where the calculation formula of the weight S of each slice X is as follows:
s＝σ(δ(z))，
wherein σ represents a sigmoid activation function, δ is a ReLU activation function, and z is a slice descriptor of the slice x obtained in step S31;
step S33, correspondingly loading each weight S obtained in step S32 to the corresponding slice, to obtain each slice X corresponding weighted slice in the D dimension of the feature map XWherein the feature re-weighting calculation formula corresponding to each slice x is as follows:
in the formula, Fscale(x, s) represents the multiplication between a slice x and its corresponding weight s;
step S34, based on the feature map X obtained in step S33, the reweighed slice corresponding to each slice X in the D dimension thereofCorrespondingly obtaining the re-weighted feature map corresponding to the feature map X
3. The unsupervised three-dimensional medical image registration method based on neural network as claimed in claim 1, wherein said similarity measure function is a cross-correlation function.
4. The unsupervised three-dimensional medical image registration method based on neural network as claimed in claim 1, wherein the spatial transformation network employs STN spatial transformation network.
5. The method of claim 1, wherein the preprocessing further comprises data enhancement;
the data enhancement comprises the following steps: respectively performing bending transformation on each obtained floating image to obtain a bent transformed image corresponding to each obtained floating image; the resulting warped images are newly added floating images.
6. An unsupervised three-dimensional medical image registration system based on a neural network, comprising:
an image acquisition unit acquiring three-dimensional medical images from the public data sets OASIS and ADNI, and/or: acquiring a three-dimensional medical image from a DICOM interface of a CT, MRI or ultrasonic imager;
the image preprocessing unit is used for preprocessing the acquired three-dimensional medical image, comprises image segmentation, cutting, normalization processing and affine alignment, and selects any one image from the images subjected to affine alignment as a fixed image IFThe rest of the image is taken as a floating image IM(ii) a Wherein the size of the cut images is consistent;
a neural network training unit based on the fixed image I obtained after preprocessingFAnd a floating image IMTraining a neural network to obtain a trained neural network model;
the image registration unit is used for inputting the medical image to be registered into the trained neural network model for registration to obtain and output a registration image of the medical image to be registered;
wherein, the neural network training unit comprises:
an input module for preprocessing the obtained fixed image IFAnd a floating image IMInputting the neural network as an input layer of the neural network, wherein each set of input data comprises the fixed image IFAnd one said floating image IM；
A down-sampling module for fixed image I input in the input layerFAnd a floating image IMPerforming down sampling to output a characteristic diagram; the down-sampling comprises 3 down-sampling processes, a convolution calculation process with convolution kernel size of 3 x 3 and a LeakyReLU laser after the 3 down-sampling processesA live function calculation process; 3 downsampling processes correspond to 3 downsampling process layers; the 3 down-sampling process layers are sequentially marked as a first down-sampling process layer, a second down-sampling process layer and a third down-sampling process layer according to the execution sequence of the down-sampling process; each downsampling process layer comprises a convolution layer with convolution kernel size of 3 multiplied by 3, a LeakyReLU activation function layer and a maximum pooling layer;
the reweighting module is used for respectively performing characteristic reweighting on the characteristic graphs output by the LeakyReLU activation function layer in the first downsampling process layer, the second downsampling process layer and the third downsampling process layer corresponding to downsampling to obtain three weighted characteristic graphs, and the three weighted characteristic graphs are sequentially: a first weighted feature map, a second weighted feature map, and a third weighted feature map;
a first deformation field output module for performing 1 × 1 × 1 convolution on the feature map output by the down-sampling module to output a floating image IMTo a fixed picture IFDeformation field of
The up-sampling module is used for inputting the characteristic diagram output by the down-sampling module into an up-sampling layer for up-sampling, the up-sampling layer comprises 3 up-sampling process layers, each up-sampling process layer comprises an UpSamplling layer and a convolution layer with the convolution kernel size of 3 multiplied by 3, and a LeakyReLU activation function layer is respectively arranged behind the convolution layer with the convolution kernel size of 3 multiplied by 3 in each up-sampling process layer; the 3 upsampling process layers correspond to the 3 upsampling processes of the upsampling; the 3 upsampling process layers are sequentially marked as a first upsampling process layer, a second upsampling process layer and a third upsampling process layer according to the sequence of the 3 upsampling processes; after the feature map output by the UpSampling layer of the first UpSampling process layer is fused with the third weighted feature map, the feature map is used as the input of the convolutional layer with the convolutional kernel size of 3 multiplied by 3 in the first UpSampling process layer; fusing a feature map output by the UpSamplling layer of the second up-sampling process layer with the second weighted feature map, and taking the feature map as the input of the convolution layer with the convolution kernel size of 3 multiplied by 3 in the second up-sampling process layer; fusing a feature map output by an UpSampling layer of a third up-sampling process layer with the first weighted feature map, and taking the feature map as the input of a convolution layer with the convolution kernel size of 3 multiplied by 3 in the third up-sampling process layer;
a second deformation field output module, which respectively performs 1 × 1 × 1 convolution on the feature maps output by the first upsampling process layer, the second upsampling process layer and the third upsampling process layer, and outputs the floating images I corresponding to the first upsampling process layer, the second upsampling process layer and the third upsampling process layerMTo a fixed picture IFIn turn is a deformation fieldDeformation fieldAnd deformation field
A spatial transformation module for transforming the floating image IMAnd the deformation field of the above outputInputting into a spatial transformation network, rendering floating images IMAnd the deformation field of the above outputInput to the spatial transformation network, floating image IMAnd the deformation field of the above outputInput to the spatial transformation network, floating image IMAnd the deformation field of the above outputInput to the spatial transformation network, respectively via the null of the spatial transformation networkInter-transforming, corresponding to obtain floating image IMThe corresponding deformed images are sequentially deformed imagesDeformed imageDeformed imageAnd the deformed image
Neural network optimization module, deformation field based on the above outputDeformation fieldDeformation fieldAnd based on the deformed image obtainedDeformed imageDeformed imageDeformed imageComputing a fixed image I using a loss functionFAnd the deformed imagePerforming back propagation optimization on the neural network until the calculated loss function value is not reduced or the network training reaches a preset training iteration number, and finishing the neural network training to obtain the trained neural network model;
the computational expression of the loss function is:
in the formula (i), the first and second groups,represents the calculated loss function value, α and β are both constants, α + β is 1,is a regularization term, λ is a regularization control constant parameter,representing a predetermined image I fixed by saidFDown-sampling the obtained three-dimensional medical image; three-dimensional medical imageSequentially with the deformed imageEquivalent, three-dimensional medical imageAre successively lower and are each smaller than the fixed image IFThe resolution of (a) of (b),representing said fixed image IFAnd the deformed imageA measure of the degree of similarity between the two,representing the three-dimensional medical imageAnd the deformed imageA measure of the degree of similarity between the two,representing the three-dimensional medical imageAnd the deformed imageA measure of the degree of similarity between the two,representing the three-dimensional medical imageAnd the deformed imageA measure of similarity between;the same similarity metric function is used.
7. The unsupervised neural network-based three-dimensional medical image registration system of claim 6, wherein the re-weighting module comprises:
a descriptor obtaining module for recording that each feature graph to be subjected to feature re-weighting output in the down-sampling is X, and X belongs to R(H ×W×D) Slicing the feature map X in the D dimension of the feature map X, and processing each slice X ∈ R obtained by slicing by using a global average pooling strategy(H×W)And performing global average pooling to obtain a slice descriptor z of each slice X in the D dimension of the feature map X, wherein a specific formula of each slice descriptor z is as follows:
wherein (i, j) represents a pixel point on the slice x, and x (i, j) represents a gray value of the slice x at the pixel point (i, j);
the slice weight calculation module is used for acquiring the weight s of each slice X on the D dimension of the feature map X, wherein the calculation formula of the weight s of each slice X is as follows:
s＝σ(δ(z))，
wherein, σ represents a sigmoid activation function, δ is a ReLU activation function, and z is a slice descriptor of a slice x obtained by the descriptor obtaining module;
a weighting module for correspondingly loading each weight s obtained by the slice weight calculation module on the corresponding slice to obtain each slice on the D dimension of the characteristic diagram X and the weighted sliceWherein the feature re-weighting calculation formula corresponding to each slice x is as follows:
in the formula, Fscale(x, s) represents slice x and its corresponding weight sA multiplication operation between;
a weighted image acquisition module for obtaining a weighted slice corresponding to each slice X on D dimension of the feature image X based on the weighted imageCorrespondingly obtaining the re-weighted feature map corresponding to the feature map X
8. The unsupervised three-dimensional medical image registration system based on neural network as claimed in claim 6, wherein said similarity measure function is a cross-correlation function.
9. The unsupervised three-dimensional medical image registration system based on neural network as claimed in claim 6, wherein the spatial transformation network employs STN spatial transformation network.
10. The unsupervised three-dimensional medical image registration system based on neural network as claimed in claim 6, wherein said preprocessing in the image preprocessing unit further comprises data enhancement; wherein said data enhancement comprises: respectively performing bending transformation on each obtained floating image to obtain a bent transformed image corresponding to each obtained floating image; the resulting warped images are newly added floating images.
CN201910828807.7A 2019-09-03 2019-09-03 Unsupervised three-dimensional medical image registration method and system based on neural network Active CN110599528B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201910828807.7A CN110599528B (en) 2019-09-03 2019-09-03 Unsupervised three-dimensional medical image registration method and system based on neural network

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201910828807.7A CN110599528B (en) 2019-09-03 2019-09-03 Unsupervised three-dimensional medical image registration method and system based on neural network

Publications (2)

Publication Number Publication Date
CN110599528A true CN110599528A (en) 2019-12-20
CN110599528B CN110599528B (en) 2022-05-27

Family

ID=68857220

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201910828807.7A Active CN110599528B (en) 2019-09-03 2019-09-03 Unsupervised three-dimensional medical image registration method and system based on neural network

Country Status (1)

Country Link
CN (1) CN110599528B (en)

Cited By (19)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111027508A (en) * 2019-12-23 2020-04-17 电子科技大学 Remote sensing image coverage change detection method based on deep neural network
CN111091575A (en) * 2019-12-31 2020-05-01 电子科技大学 Medical image segmentation method based on reinforcement learning method
CN111242877A (en) * 2019-12-31 2020-06-05 北京深睿博联科技有限责任公司 Mammary X-ray image registration method and device
CN111260705A (en) * 2020-01-13 2020-06-09 武汉大学 Prostate MR image multi-task registration method based on deep convolutional neural network
CN111524170A (en) * 2020-04-13 2020-08-11 中南大学 Lung CT image registration method based on unsupervised deep learning
CN111768379A (en) * 2020-06-29 2020-10-13 深圳度影医疗科技有限公司 Standard section detection method of three-dimensional uterine ultrasound image
CN112102373A (en) * 2020-07-29 2020-12-18 浙江工业大学 Carotid artery multi-mode image registration method based on strong constraint affine deformation feature learning
CN112150425A (en) * 2020-09-16 2020-12-29 北京工业大学 Unsupervised intravascular ultrasound image registration method based on neural network
CN112652002A (en) * 2020-12-25 2021-04-13 江苏集萃复合材料装备研究所有限公司 Medical image registration method based on IDC algorithm
CN112907439A (en) * 2021-03-26 2021-06-04 中国科学院深圳先进技术研究院 Supine position and prone position mammary gland image registration method based on deep learning
CN113034453A (en) * 2021-03-16 2021-06-25 深圳先进技术研究院 Mammary gland image registration method based on deep learning
CN113344876A (en) * 2021-06-08 2021-09-03 安徽大学 Deformable registration method between CT and CBCT
CN113409291A (en) * 2021-06-29 2021-09-17 山东大学 Lung 4D-CT medical image registration method and system
CN113450397A (en) * 2021-06-25 2021-09-28 广州柏视医疗科技有限公司 Image deformation registration method based on deep learning
CN113724307A (en) * 2021-09-02 2021-11-30 深圳大学 Image registration method and device based on characteristic self-calibration network and related components
CN113763441A (en) * 2021-08-25 2021-12-07 中国科学院苏州生物医学工程技术研究所 Medical image registration method and system for unsupervised learning
CN114170276A (en) * 2021-10-15 2022-03-11 烟台大学 Magnetic resonance brain image hippocampus registration method
CN114693897A (en) * 2021-04-28 2022-07-01 上海联影智能医疗科技有限公司 Unsupervised inter-layer super-resolution for medical images
WO2022178997A1 (en) * 2021-02-25 2022-09-01 平安科技（深圳）有限公司 Medical image registration method and apparatus, computer device, and storage medium

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20160093050A1 (en) * 2014-09-30 2016-03-31 Samsung Electronics Co., Ltd. Image registration device, image registration method, and ultrasonic diagnosis apparatus having image registration device
US20190012567A1 (en) * 2010-05-03 2019-01-10 Mim Software Inc. Systems and methods for contouring a set of medical images
CN109584283A (en) * 2018-11-29 2019-04-05 合肥中科离子医学技术装备有限公司 A kind of Medical Image Registration Algorithm based on convolutional neural networks
CN109767459A (en) * 2019-01-17 2019-05-17 中南大学 Novel ocular base map method for registering
CN110021037A (en) * 2019-04-17 2019-07-16 南昌航空大学 A kind of image non-rigid registration method and system based on generation confrontation network

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20190012567A1 (en) * 2010-05-03 2019-01-10 Mim Software Inc. Systems and methods for contouring a set of medical images
US20160093050A1 (en) * 2014-09-30 2016-03-31 Samsung Electronics Co., Ltd. Image registration device, image registration method, and ultrasonic diagnosis apparatus having image registration device
CN109584283A (en) * 2018-11-29 2019-04-05 合肥中科离子医学技术装备有限公司 A kind of Medical Image Registration Algorithm based on convolutional neural networks
CN109767459A (en) * 2019-01-17 2019-05-17 中南大学 Novel ocular base map method for registering
CN110021037A (en) * 2019-04-17 2019-07-16 南昌航空大学 A kind of image non-rigid registration method and system based on generation confrontation network

Non-Patent Citations (4)

* Cited by examiner, † Cited by third party
Title
HONGMING LI 等: "Non-rigid image registration using self-supervised fully convolutional networks without training data", 《2018 IEEE 15TH INTERNATIONAL SYMPOSIUM ON BIOMEDICAL IMAGING (ISBI 2018)》 *
XIAOHUAN CAO 等: "Deformable Image Registration Using a Cue-Aware Deep Regression Network", 《IEEE TRANSACTIONS ON BIOMEDICAL ENGINEERING》 *
张文华 等: "拟合精度引导的扩散加权图像配准", 《计算机科学》 *
纪慧中 等: "基于图像特征和光流场的非刚性图像配准", 《光学精密工程》 *

Cited By (28)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111027508B (en) * 2019-12-23 2022-09-06 电子科技大学 Remote sensing image coverage change detection method based on deep neural network
CN111027508A (en) * 2019-12-23 2020-04-17 电子科技大学 Remote sensing image coverage change detection method based on deep neural network
CN111091575A (en) * 2019-12-31 2020-05-01 电子科技大学 Medical image segmentation method based on reinforcement learning method
CN111242877A (en) * 2019-12-31 2020-06-05 北京深睿博联科技有限责任公司 Mammary X-ray image registration method and device
CN111260705A (en) * 2020-01-13 2020-06-09 武汉大学 Prostate MR image multi-task registration method based on deep convolutional neural network
CN111524170B (en) * 2020-04-13 2023-05-26 中南大学 Pulmonary CT image registration method based on unsupervised deep learning
CN111524170A (en) * 2020-04-13 2020-08-11 中南大学 Lung CT image registration method based on unsupervised deep learning
CN111768379A (en) * 2020-06-29 2020-10-13 深圳度影医疗科技有限公司 Standard section detection method of three-dimensional uterine ultrasound image
CN112102373A (en) * 2020-07-29 2020-12-18 浙江工业大学 Carotid artery multi-mode image registration method based on strong constraint affine deformation feature learning
CN112150425B (en) * 2020-09-16 2024-05-24 北京工业大学 Unsupervised intravascular ultrasound image registration method based on neural network
CN112150425A (en) * 2020-09-16 2020-12-29 北京工业大学 Unsupervised intravascular ultrasound image registration method based on neural network
CN112652002B (en) * 2020-12-25 2024-05-03 江苏集萃复合材料装备研究所有限公司 Medical image registration method based on IDC algorithm
CN112652002A (en) * 2020-12-25 2021-04-13 江苏集萃复合材料装备研究所有限公司 Medical image registration method based on IDC algorithm
WO2022178997A1 (en) * 2021-02-25 2022-09-01 平安科技（深圳）有限公司 Medical image registration method and apparatus, computer device, and storage medium
CN113034453A (en) * 2021-03-16 2021-06-25 深圳先进技术研究院 Mammary gland image registration method based on deep learning
WO2022199135A1 (en) * 2021-03-26 2022-09-29 中国科学院深圳先进技术研究院 Supine position and prone position breast image registration method based on deep learning
CN112907439B (en) * 2021-03-26 2023-08-08 中国科学院深圳先进技术研究院 Deep learning-based supine position and prone position breast image registration method
CN112907439A (en) * 2021-03-26 2021-06-04 中国科学院深圳先进技术研究院 Supine position and prone position mammary gland image registration method based on deep learning
CN114693897A (en) * 2021-04-28 2022-07-01 上海联影智能医疗科技有限公司 Unsupervised inter-layer super-resolution for medical images
CN113344876A (en) * 2021-06-08 2021-09-03 安徽大学 Deformable registration method between CT and CBCT
CN113450397B (en) * 2021-06-25 2022-04-01 广州柏视医疗科技有限公司 Image deformation registration method based on deep learning
CN113450397A (en) * 2021-06-25 2021-09-28 广州柏视医疗科技有限公司 Image deformation registration method based on deep learning
CN113409291A (en) * 2021-06-29 2021-09-17 山东大学 Lung 4D-CT medical image registration method and system
CN113763441A (en) * 2021-08-25 2021-12-07 中国科学院苏州生物医学工程技术研究所 Medical image registration method and system for unsupervised learning
CN113763441B (en) * 2021-08-25 2024-01-26 中国科学院苏州生物医学工程技术研究所 Medical image registration method and system without supervision learning
CN113724307A (en) * 2021-09-02 2021-11-30 深圳大学 Image registration method and device based on characteristic self-calibration network and related components
CN113724307B (en) * 2021-09-02 2023-04-28 深圳大学 Image registration method and device based on characteristic self-calibration network and related components
CN114170276A (en) * 2021-10-15 2022-03-11 烟台大学 Magnetic resonance brain image hippocampus registration method

Also Published As

Publication number Publication date
CN110599528B (en) 2022-05-27

Similar Documents

Publication Publication Date Title
CN110599528B (en) Unsupervised three-dimensional medical image registration method and system based on neural network
WO2020156148A1 (en) Method for training smpl parameter prediction model, computer device, and storage medium
CN110321920B (en) Image classification method and device, computer readable storage medium and computer equipment
CN108776969B (en) Breast ultrasound image tumor segmentation method based on full convolution network
CN111798462B (en) Automatic delineation method of nasopharyngeal carcinoma radiotherapy target area based on CT image
Eppenhof et al. Error estimation of deformable image registration of pulmonary CT scans using convolutional neural networks
WO2020007941A1 (en) Automated determination of a canonical pose of a 3d objects and superimposition of 3d objects using deep learning
Li et al. Automated measurement network for accurate segmentation and parameter modification in fetal head ultrasound images
CN110363802B (en) Prostate image registration system and method based on automatic segmentation and pelvis alignment
CN111640120A (en) Pancreas CT automatic segmentation method based on significance dense connection expansion convolution network
JP3712234B2 (en) Region of interest extraction method and image processing server
US20220335600A1 (en) Method, device, and storage medium for lesion segmentation and recist diameter prediction via click-driven attention and dual-path connection
WO2023044605A1 (en) Three-dimensional reconstruction method and apparatus for brain structure in extreme environments, and readable storage medium
Liu et al. The measurement of Cobb angle based on spine X-ray images using multi-scale convolutional neural network
CN115830016B (en) Medical image registration model training method and equipment
CN112465754B (en) 3D medical image segmentation method and device based on layered perception fusion and storage medium
CN111080658A (en) Cervical MRI image segmentation method based on deformable registration and DCNN
US11941738B2 (en) Systems and methods for personalized patient body modeling
CN110992310A (en) Method and device for determining partition where mediastinal lymph node is located
CN113822323A (en) Brain scanning image identification processing method, device, equipment and storage medium
CN112750110A (en) Evaluation system for evaluating lung lesion based on neural network and related products
CN114787816A (en) Data enhancement for machine learning methods
CN112102327A (en) Image processing method and device and computer readable storage medium
CN114677516A (en) Automatic oral mandibular canal segmentation method based on deep neural network
CN115115772A (en) Key structure reconstruction method and device based on three-dimensional image and computer equipment

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant