CN113269815A - Deep learning-based medical image registration method and terminal - Google Patents

Deep learning-based medical image registration method and terminal Download PDF

Info

Publication number
CN113269815A
CN113269815A CN202110528670.0A CN202110528670A CN113269815A CN 113269815 A CN113269815 A CN 113269815A CN 202110528670 A CN202110528670 A CN 202110528670A CN 113269815 A CN113269815 A CN 113269815A
Authority
CN
China
Prior art keywords
image
registered
moving image
fixed
moving
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202110528670.0A
Other languages
Chinese (zh)
Other versions
CN113269815B (en
Inventor
陈利
邓小武
彭应林
孙文钊
高兴旺
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Sun Yat Sen University Cancer Center
Original Assignee
Sun Yat Sen University Cancer Center
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Sun Yat Sen University Cancer Center filed Critical Sun Yat Sen University Cancer Center
Priority to CN202110528670.0A priority Critical patent/CN113269815B/en
Publication of CN113269815A publication Critical patent/CN113269815A/en
Application granted granted Critical
Publication of CN113269815B publication Critical patent/CN113269815B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/30—Determination of transform parameters for the alignment of images, i.e. image registration
    • G06T7/33—Determination of transform parameters for the alignment of images, i.e. image registration using feature-based methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/088—Non-supervised learning, e.g. competitive learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformations in the plane of the image
    • G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
    • G06T3/4038—Image mosaicing, e.g. composing plane images from plane sub-images
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/0002—Inspection of images, e.g. flaw detection
    • G06T7/0012—Biomedical image inspection
    • G06T7/0014—Biomedical image inspection using an image reference approach
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/20—Analysis of motion
    • G06T7/246—Analysis of motion using feature-based methods, e.g. the tracking of corners or segments
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/20—Analysis of motion
    • G06T7/269—Analysis of motion using gradient-based methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/30—Determination of transform parameters for the alignment of images, i.e. image registration
    • G06T7/38—Registration of image sequences
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2200/00—Indexing scheme for image data processing or generation, in general
    • G06T2200/32—Indexing scheme for image data processing or generation, in general involving image mosaicing
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10004—Still image; Photographic image
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • General Health & Medical Sciences (AREA)
  • Health & Medical Sciences (AREA)
  • Multimedia (AREA)
  • Computing Systems (AREA)
  • Biomedical Technology (AREA)
  • Data Mining & Analysis (AREA)
  • Molecular Biology (AREA)
  • Computational Linguistics (AREA)
  • General Engineering & Computer Science (AREA)
  • Biophysics (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Evolutionary Computation (AREA)
  • Medical Informatics (AREA)
  • Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
  • Radiology & Medical Imaging (AREA)
  • Quality & Reliability (AREA)
  • Artificial Intelligence (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Image Analysis (AREA)
  • Image Processing (AREA)

Abstract

The invention discloses a medical image registration method and a medical image registration terminal based on deep learning, wherein a moving image to be registered and a fixed image to be registered are preprocessed and spliced; predicting a flow field between a moving image to be registered and a fixed image to be registered on the basis of the spliced images by using a U-shaped network, and resampling the moving image to be registered through the obtained flow field, so that dependence on a large number of data labels in the registration process is avoided through an unsupervised learning method, and the registration calculation speed is improved; registering the resampled moving image to be registered and the resampled fixed image to be registered, adjusting the registered image by using a loss function, measuring the distance between a predicted value and a sample value in the registering process by using the loss function, and further improving the precision of the registered image; therefore, the invention uses the U-shaped network to predict the flow field, carries out image registration based on the flow field, uses the loss function to adjust the registered image, and realizes the registration of the medical image quickly and accurately.

Description

Deep learning-based medical image registration method and terminal
Technical Field
The invention relates to the technical field of medical images, in particular to a medical image registration method and a medical image registration terminal based on deep learning.
Background
Medical image registration is a key and difficult problem in the field of medical image processing, and the purpose of registration is to convert image coordinates obtained from different devices or at different times into another image, i.e. to map information from one moving image into another fixed image. From the spatial aspect of the image, the moving image is mapped to the fixed image by finding a spatial transformation, such as rigid body transformation, affine transformation, projection transformation and nonlinear transformation, and the spatial positions are matched by corresponding points corresponding to the same position in space in the moving image and the fixed image one by one, so that the aim of image information fusion is fulfilled.
Medical image registration can be divided into single modality registration and multi-modality registration. The main focus of single modality medical image registration is on the change of tissue organs or lesions over time, and the main focus of multi-modality medical image registration is on the anatomy of its morphology and function.
In the aspect of single-mode registration, the difference of images acquired by different devices may be large, and the local gray level correspondence may be inconsistent, so that it is a difficult point to achieve global gray level correspondence consistency, and the difficulty is a main factor affecting the registration accuracy. In addition, the conventional method has the limitation that the method is only suitable for one specific modality, so that the limitation causes the registration speed of the medical image to be slow and is not suitable for a real application scene, and the early deep learning technology is based on supervised learning, needs a large number of data tags and has a large dependence on the data tags.
Disclosure of Invention
The technical problem to be solved by the invention is as follows: the medical image registration method and the medical image registration terminal based on deep learning are provided, and the speed and the precision of image registration can be improved.
In order to solve the technical problems, the invention adopts the technical scheme that:
a medical image registration method based on deep learning comprises the following steps:
acquiring a moving image to be registered and a fixed image to be registered, and preprocessing the moving image to be registered and the fixed image to be registered;
splicing the preprocessed moving image to be registered and the preprocessed fixed image to be registered to obtain a spliced image;
predicting the spliced image by using a U-shaped network to obtain a flow field between the moving image to be registered and the fixed image to be registered, and resampling the moving image to be registered according to the flow field;
and registering the resampled moving image to be registered and the resampled fixed image to be registered to obtain a registered image, adjusting the registered image by using a plurality of loss functions and verifying the adjusted registered image.
In order to solve the technical problem, the invention adopts another technical scheme as follows:
a deep learning based medical image registration terminal comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the following steps when executing the computer program:
acquiring a moving image to be registered and a fixed image to be registered, and preprocessing the moving image to be registered and the fixed image to be registered;
splicing the preprocessed moving image to be registered and the preprocessed fixed image to be registered to obtain a spliced image;
predicting the spliced image by using a U-shaped network to obtain a flow field between the moving image to be registered and the fixed image to be registered, and resampling the moving image to be registered according to the flow field;
and registering the resampled moving image to be registered and the resampled fixed image to be registered to obtain a registered image, adjusting the registered image by using a plurality of loss functions and verifying the adjusted registered image.
The invention has the beneficial effects that: preprocessing and splicing the moving image to be registered and the fixed image to be registered; predicting a flow field between a moving image to be registered and a fixed image to be registered on the basis of the spliced images by using a U-shaped network, and resampling the moving image to be registered through the obtained flow field, so that dependence on a large number of data labels in the registration process is avoided through an unsupervised learning method, and the registration calculation speed is improved; registering the resampled moving image to be registered and the resampled fixed image to be registered, adjusting the registered image by using a loss function, measuring the distance between a predicted value and a sample value in the registration process by using the loss function, and reducing the difference of local gray scale in the image registration by combining the step of verifying the registered image so as to further improve the precision of the registered image; the U-shaped network combines the functions of object type identification and accurate positioning, so the invention uses the U-shaped network to predict the flow field, carries out image registration based on the flow field, and uses the loss function to adjust the registered image, thereby quickly realizing the registration of the medical image on the basis of ensuring the accuracy of the image.
Drawings
Fig. 1 is a flowchart of a deep learning-based medical image registration method according to an embodiment of the present invention;
fig. 2 is a schematic diagram of a deep learning-based medical image registration terminal according to an embodiment of the present invention;
fig. 3 is a flowchart of a Unet network registration of a deep learning-based medical image registration method according to an embodiment of the present invention;
fig. 4 is an exemplary diagram of a network registration of the Unet of the deep learning-based medical image registration method according to the embodiment of the present invention;
fig. 5 is a structure diagram of a net network of a deep learning-based medical image registration method according to an embodiment of the present invention;
fig. 6 is a graph of variation of loss function values of a deep learning-based medical image registration method according to an embodiment of the present invention;
FIG. 7 is a gradient change diagram of a deep learning-based medical image registration method according to an embodiment of the present invention;
fig. 8 is a circularly consistent Unet network structure registration process diagram of a deep learning-based medical image registration method according to an embodiment of the present invention.
Detailed Description
In order to explain technical contents, achieved objects, and effects of the present invention in detail, the following description is made with reference to the accompanying drawings in combination with the embodiments.
Referring to fig. 1, an embodiment of the present invention provides a depth learning-based medical image registration method, including:
acquiring a moving image to be registered and a fixed image to be registered, and preprocessing the moving image to be registered and the fixed image to be registered;
splicing the preprocessed moving image to be registered and the preprocessed fixed image to be registered to obtain a spliced image;
predicting the spliced image by using a U-shaped network to obtain a flow field between the moving image to be registered and the fixed image to be registered, and resampling the moving image to be registered according to the flow field;
and registering the resampled moving image to be registered and the resampled fixed image to be registered to obtain a registered image, adjusting the registered image by using a plurality of loss functions and verifying the adjusted registered image.
From the above description, the beneficial effects of the present invention are: preprocessing and splicing the moving image to be registered and the fixed image to be registered; predicting a flow field between a moving image to be registered and a fixed image to be registered on the basis of the spliced images by using a U-shaped network, and resampling the moving image to be registered through the obtained flow field, so that dependence on a large number of data labels in the registration process is avoided through an unsupervised learning method, and the registration calculation speed is improved; registering the resampled moving image to be registered and the resampled fixed image to be registered, adjusting the registered image by using a loss function, measuring the distance between a predicted value and a sample value in the registering process by using the loss function, and further improving the precision of the registered image by combining the step of verifying the registered image; the U-shaped network combines the functions of object type identification and accurate positioning, so the invention uses the U-shaped network to predict the flow field, carries out image registration based on the flow field, and uses the loss function to adjust the registered image, thereby quickly realizing the registration of the medical image on the basis of ensuring the accuracy of the image.
Further, the preprocessing the moving image to be registered and the fixed image to be registered includes:
standardizing the sizes of the moving image to be registered and the fixed image to be registered;
extracting the shapes of the standardized moving image and the standardized fixed image, and setting a corresponding zero array;
creating a sequence of the moving images to be registered and the fixed images to be registered based on the shapes of the normalized moving images and the normalized fixed images and the zero array;
the splicing of the preprocessed moving image to be registered and the preprocessed fixed image to be registered to obtain a spliced image comprises the following steps:
and splicing according to the sequence of the moving image to be registered and the sequence of the fixed image to be registered to obtain a spliced image.
According to the description, the shapes of the standardized moving images and the standardized fixed images are extracted, the corresponding zero arrays are set, and data generated in the subsequent image processing can be stored in the zero arrays, so that the data input by the network can be calculated and prepared in advance before model training, the recalculation in the model training process in the traditional method is avoided, the model calculation time and the network complexity are increased, the registration efficiency is improved, and the calculation amount of the whole registration model is reduced.
Further, resampling the moving image to be registered according to the flow field comprises:
obtaining a corresponding sampling network according to the flow field;
and resampling the moving image to be registered during image space transformation through the sampling network.
According to the above description, according to the sampling network corresponding to the flow field, the moving image to be registered is resampled during the image space transformation, so that the resampled moving image and the fixed image to be registered are registered subsequently, and the rapid registration is realized.
Further, the splicing the preprocessed moving image to be registered and the preprocessed fixed image to be registered to obtain a spliced image comprises:
obtaining a loss function loss for penalizing the folding operation:
loss＝cc(trueY,predY)+λ||DutrueX→predY||l2+cc(trueX,predX)+λ||DupredY→predX||l2；
where cc denotes cross-correlation, true x denotes an actual fixed image, true y denotes an actual moving image, predX denotes a predicted fixed image, predY denotes a predicted moving image, λ denotes a parameter, Du denotes a valuetrueX→predYDisplacement field representing a predicted warped image, DupredY→predXRepresents the inverse displacement field corresponding to the predicted warped image, | | | | l2Representing a norm form;
and adjusting the spliced image through a loss function for punishing the folding operation and a vectorization loss function.
As can be seen from the above description, in order to make the stitched image more accurate, a loss function for penalizing the folding operation and a vectorization loss function are added after the generation of the stitched image, so that the local and global losses can be used together, and the accuracy of the image is further improved.
Further, the predicting the stitched image by using a U-shaped network to obtain a flow field between the moving image to be registered and the fixed image to be registered, and resampling the moving image to be registered according to the flow field includes:
carrying out deformation construction on the spliced image, and predicting a deformation construction result by using a U-shaped network;
and converting the prediction result into a flow field between the moving image to be registered and the fixed image to be registered, and twisting the moving image to be registered by using the flow field and resampling.
According to the description, the spliced image is subjected to deformation construction, the result of the deformation construction is predicted by using the U-shaped network, and the prediction result is converted into the flow field, so that the moving image to be registered is quickly distorted and resampled, and the image registration speed is improved.
Referring to fig. 2, another embodiment of the present invention provides a deep learning-based medical image registration terminal, including a memory, a processor, and a computer program stored in the memory and executable on the processor, where the processor executes the computer program to implement the following steps:
acquiring a moving image to be registered and a fixed image to be registered, and preprocessing the moving image to be registered and the fixed image to be registered;
splicing the preprocessed moving image to be registered and the preprocessed fixed image to be registered to obtain a spliced image;
predicting the spliced image by using a U-shaped network to obtain a flow field between the moving image to be registered and the fixed image to be registered, and resampling the moving image to be registered according to the flow field;
and registering the resampled moving image to be registered and the resampled fixed image to be registered to obtain a registered image, adjusting the registered image by using a plurality of loss functions and verifying the adjusted registered image.
According to the description, the mobile image to be registered and the fixed image to be registered are preprocessed and spliced; predicting a flow field between a moving image to be registered and a fixed image to be registered on the basis of the spliced images by using a U-shaped network, and resampling the moving image to be registered through the obtained flow field, so that dependence on a large number of data labels in the registration process is avoided through an unsupervised learning method, and the registration calculation speed is improved; registering the resampled moving image to be registered and the resampled fixed image to be registered, adjusting the registered image by using a loss function, measuring the distance between a predicted value and a sample value in the registering process by using the loss function, and further improving the precision of the registered image by combining the step of verifying the registered image; the U-shaped network combines the functions of object type identification and accurate positioning, so the invention uses the U-shaped network to predict the flow field, carries out image registration based on the flow field, and uses the loss function to adjust the registered image, thereby quickly realizing the registration of the medical image on the basis of ensuring the accuracy of the image.
Further, the preprocessing the moving image to be registered and the fixed image to be registered includes:
standardizing the sizes of the moving image to be registered and the fixed image to be registered;
extracting the shapes of the standardized moving image and the standardized fixed image, and setting a corresponding zero array;
creating a sequence of the moving images to be registered and the fixed images to be registered based on the shapes of the normalized moving images and the normalized fixed images and the zero array;
the splicing of the preprocessed moving image to be registered and the preprocessed fixed image to be registered to obtain a spliced image comprises the following steps:
and splicing according to the sequence of the moving image to be registered and the sequence of the fixed image to be registered to obtain a spliced image.
According to the description, the shapes of the standardized moving images and the standardized fixed images are extracted, the corresponding zero arrays are set, and data generated in the subsequent image processing can be stored in the zero arrays, so that the data input by the network can be calculated and prepared in advance before model training, the recalculation in the model training process in the traditional method is avoided, the model calculation time and the network complexity are increased, the registration efficiency is improved, and the calculation amount of the whole registration model is reduced.
Further, resampling the moving image to be registered according to the flow field comprises:
obtaining a corresponding sampling network according to the flow field;
and resampling the moving image to be registered during image space transformation through the sampling network.
According to the above description, according to the sampling network corresponding to the flow field, the moving image to be registered is resampled during the image space transformation, so that the resampled moving image and the fixed image to be registered are registered subsequently, and the rapid registration is realized.
Further, the splicing the preprocessed moving image to be registered and the preprocessed fixed image to be registered to obtain a spliced image comprises:
obtaining a loss function loss for penalizing the folding operation:
loss＝cc(trueY,predY)+λ||DutrueX→predY||l2+cc(trueX,predX)+λ||DupredY→predX||l2；
where cc denotes cross-correlation, true x denotes an actual fixed image, true y denotes an actual moving image, predX denotes a predicted fixed image, predY denotes a predicted moving image, λ denotes a parameter, Du denotes a valuetrueX→predYDisplacement field representing a predicted warped image, DupredY→predXRepresents the inverse displacement field corresponding to the predicted warped image, | | | | l2Representing a norm form;
and adjusting the spliced image through a loss function for punishing the folding operation and a vectorization loss function.
As can be seen from the above description, in order to make the stitched image more accurate, a loss function for penalizing the folding operation and a vectorization loss function are added after the generation of the stitched image, so that the local and global losses can be used together, and the accuracy of the image is further improved.
Further, the predicting the stitched image by using a U-shaped network to obtain a flow field between the moving image to be registered and the fixed image to be registered, and resampling the moving image to be registered according to the flow field includes:
carrying out deformation construction on the spliced image, and predicting a deformation construction result by using a U-shaped network;
and converting the prediction result into a flow field between the moving image to be registered and the fixed image to be registered, and twisting the moving image to be registered by using the flow field and resampling.
According to the description, the spliced image is subjected to deformation construction, the result of the deformation construction is predicted by using the U-shaped network, and the prediction result is converted into the flow field, so that the moving image to be registered is quickly distorted and resampled, and the image registration speed is improved.
The medical image registration method and the terminal based on deep learning are suitable for various medical image registration scenes, can quickly and accurately perform image registration based on unsupervised deep learning, and are explained in the following through specific implementation modes:
example one
Referring to fig. 1, a medical image registration method based on deep learning includes the steps of:
s1, obtaining a moving image to be registered and a fixed image to be registered, and preprocessing the moving image to be registered and the fixed image to be registered;
wherein, the preprocessing the moving image to be registered and the fixed image to be registered comprises the following steps:
standardizing the sizes of the moving image to be registered and the fixed image to be registered;
extracting the shapes of the standardized moving image and the standardized fixed image, and setting a corresponding zero array;
creating a sequence of the moving images to be registered and the fixed images to be registered based on the shapes of the normalized moving images and the normalized fixed images and the zero array;
the splicing of the preprocessed moving image to be registered and the preprocessed fixed image to be registered to obtain a spliced image comprises the following steps:
splicing according to the sequence of the moving image to be registered and the sequence of the fixed image to be registered to obtain a spliced image;
specifically, in this embodiment, NumPy and OpenCV modules are used to process training data, where the basic parameters include: path of the medical picture, row img _ rows and column img _ cols of the medical picture, color channel color _ type of the medical picture;
loading a moving image to be registered and a fixed image to be registered load imgs [ ], and acquiring a corresponding path according to a color channel of a loaded image by using a cv2 module of OpenCV;
adjusting the sizes of the moving image to be registered and the fixed image to be registered, converting the moving image to be registered and the fixed image to be registered into a 32-bit floating point type data type, and dividing the converted data by 255 to realize image standardization;
extracting the sizes, shapes and dimensions of a moving image to be registered and a fixed image to be registered, preparing a corresponding zero array, and creating a moving image sequence moving _ data to be registered and a sequence fixed _ data of the fixed image to be registered for splicing based on the shapes of the moving image and the fixed image and the zero array;
s2, splicing the preprocessed moving image to be registered and the preprocessed fixed image to be registered to obtain a spliced image;
specifically, OpenCV is used for respectively reading corresponding fixed images fixed _ images and moving images moving _ images according to moving _ data and moving _ data, the sizes of the fixed images are cut according to tensors of the moving images and the fixed images, and the fixed images are adjusted in any direction of the fixed images according to all dimensions of the fixed images;
adjusting in any direction of the moving image according to each dimension of the fixed image, and storing the adjusted fixed image and the moving image into a preset image sequence for splicing to obtain a spliced image;
the method for splicing the preprocessed moving image to be registered and the preprocessed fixed image to be registered to obtain a spliced image comprises the following steps:
obtaining a loss function loss for penalizing the folding operation:
loss＝cc(trueY,predY)+λ||DutrueX-predY||l2+cc(trueX,predX)+λ||DupredY-predX||l2；
where cc denotes cross-correlation, true x denotes an actual fixed image, true y denotes an actual moving image, predX denotes a predicted fixed image, predY denotes a predicted moving image, λ denotes a parameter, Du denotes a valuetrueX→predYDisplacement field representing a predicted warped image, DupredY→predXRepresents the inverse displacement field corresponding to the predicted warped image, | | | | l2Representing a norm form;
adjusting the spliced image through a loss function for punishing folding operation and a vectorization loss function;
specifically, the weight can be set according to the requirement when constructing the loss function, tf. However, for multi-modality image registration, no assumption can be made that there is a non-linear relationship between the registered images. In order to balance the result between accuracy and smoothness, a loss function for penalizing folding operation and a vectorization loss function are added in the embodiment, the global and local loss functions are used together and must be implemented in a vectorization manner, and the speed in the vectorization operation process is much faster than that in normal cycle operation;
to calculate the global loss of the fixed and moving images, the corresponding matrix is first calculated, as shown in the formula:
Figure BDA0003066202380000101
in the formula, s represents samples of a fixed image and a moving image, x represents a continuous contribution value of each sample, x-s represents the intensity of subtracting a binary center of each sample, W (x-s) represents distance weight, sigma represents standard deviation, e represents the base number of a natural logarithm, and the formula is a Gaussian function;
in the process of calculating the Gaussian function, the intensity of a binary center is subtracted, and then element operation is carried out, so that the weight of the loss function is calculated, and the weighting effect is realized;
respectively calculating edge distribution p (A) of a fixed image and edge distribution p (B) of a moving image, calculating the average value of the edge distribution of the fixed image and the moving image along the 0 axis, namely axis is 0, and then calculating the joint distribution p (A, B) of the fixed image and the moving image;
wherein, p (a) tf.reduce _ mean (image a,1, keep _ mean), and p (b) tf.reduce _ mean (image b,1, keep _ mean);
reduce _ mean () function represents the mean of the computed tensor along the specified axis; imageA and imageB are tensors of the input fixed image and the input moving image respectively; 1 represents a designated axis, and if not, an average value of all directional axes is calculated; the keepdims parameter indicates whether dimension reduction is carried out, the True is set to indicate maintaining dimension, and the False is set to indicate dimension reduction;
premuteA＝keras.permute_dimensions(imageA,(0,2,1))；
where the function of keras _ dimensions () represents the axis of displacement in the tensor, where the axis parameters are (0,2, 1);
board _ dot (permata, imageB), board _ dot () representing smart matrix multiplication;
for metric operation, inputting a fixed image fixed _ image and a moving image moving _ image to obtain pred _ fixed _ image, and constructing a loss function for penalizing folding operation based on the pred _ fixed _ image;
optimizing each loss function by using an optimizer build _ optimizer;
adjusting the spliced image through a loss function for punishing folding operation and a vectorization loss function;
s3, predicting the spliced image by using a U-shaped network to obtain a flow field between the moving image to be registered and the fixed image to be registered, and resampling the moving image to be registered according to the flow field;
wherein resampling the moving image to be registered according to the flow field comprises:
obtaining a corresponding sampling network according to the flow field;
resampling the moving image to be registered during image space transformation through the sampling network;
specifically, referring to fig. 3 to 5, a warp flow field is predicted from a stitched image through a U-type network, a sampling network is obtained from the warp flow field, where a downsampling encode of the U-type network is [32,32,32,32], and an upsampling decode is [32,32,32, 16], and a spatial conversion network is used to resample a moving image to generate a final registered image;
the space transformation network mainly comprises a positioning network, a grid generator and a sampler; inputting a moving image and a fixed image, acquiring a feature map, generating corresponding transformation parameters on a space through a positioning network, determining a sampling grid according to the transformation parameters, and finally generating the feature map according to the features extracted by the sampling grid;
s4, registering the resampled moving image to be registered and the resampled fixed image to obtain a registered image, adjusting the registered image by using a plurality of loss functions, and verifying the adjusted registered image;
specifically, in the present embodiment, mse, ncc or the like is used as the loss function, for example, loss _ weight is [1, lambda _ para ] where loss _ weight is a hyper parameter and represents a loss function weight, and the default is 0.01, [ mse, loss. Referring to fig. 6, the ordinate in the graph represents the loss value, the abscissa represents the training batch epoch, fig. 6 shows the overall loss variation trend graph epoch _ loss, the sum variance variation trend graph, large _ sumquarreddifference, the weight update variation graph image _ sumsquarreddifference _ weighted corresponding to the sum variance, the variation graph of the value, label _ DiceScoreLoss _ weighted, and the regularization gradient norm variation graph regularization _ Gradientnorm, respectively; batch regularization is used to prevent gradient disappearance or explosion, and the learning convergence speed, namely the model training speed, is increased;
taking out a piece of data from the verification set, verifying the registration image, referring to fig. 7, the ordinate in the graph represents the training batch epoch, the abscissa represents the quantized gradient value, and the corresponding visualization graph can be observed during model training, so as to visualize the loss function corresponding to the verification result.
In particular, the present embodiment is mathematically considered that the transformation used in the registration process should be differential homomorphism, so that the nature of the topology does not change with the length of the transformation. To achieve near ideal properties, the trained network model also follows its reversibility. The design directly solves the reversibility of the network, the cyclic constraint is also beneficial to learning indirect smooth solution, the design realizes the regularization of the network by force, and the space transformer also learns the inverse solution of the solution while learning the solution. This helps the network to eliminate the possibility of some translation inconsistencies. This design also does not add any additional learnable parameters to the original parameters.
Therefore, in this embodiment, a task separation is created in the first stage, i.e. folding less deformation field, warping the source and target into two competing sub-tasks, this generation network is focused on improving the similarity between the warped and target images, with slight limitation on the smoothing condition; in the second stage, the enhancement network is in an active state, has stronger regularization strength and focuses on reducing the stricter smooth constraint of the folding position;
in order to train the two networks together to reach an ideal equilibrium state, namely, the two networks are accurate and smooth in transformation and less in folding, a loss function method special for punishing folding operation is used, and an alternating training algorithm shown in fig. 8 is adopted, so that accurate registration of the medical image can be realized based on a circularly consistent Unet network structure, namely continuity, reversibility and microminiaturity, and smooth regularization and circular consistency are combined.
Example two
The difference between the present embodiment and the first embodiment is that a method for generating and resampling a flow field is further defined:
specifically, the predicting the stitched image by using a U-shaped network to obtain a flow field between the moving image to be registered and the fixed image to be registered, and the resampling the moving image to be registered according to the flow field includes:
carrying out deformation construction on the spliced image, and predicting a deformation construction result by using a U-shaped network;
converting the prediction result into a flow field between the moving image to be registered and the fixed image to be registered, and twisting the moving image to be registered by using the flow field and resampling;
specifically, a deformation structure is performed on the spliced image, in this embodiment, a U-type network uet predicts a result of the deformation structure, and a Unet _ model is Unet _ core (vol _ size, encode, decode), where vol _ size represents an input size, encode represents downsampling coding, decode represents upsampling decoding, and Unet _ core represents the whole downsampling and upsampling processes, reduces the image to obtain a specific required size, needs to process the pixel level of the image, and then interpolates and amplifies the original image;
[src,tgt]＝unet_model.inputs,x_out＝unet_model.output[-1]；
wherein, the parameters src ═ Input (shape [. vol _ size ]) and tgt ═ Input (shape [. vol _ size ]) are both inputs of the U-type network, and x _ out is an output;
and converting the prediction result into a flow field between the moving image to be registered and the fixed image to be registered, and twisting the moving image to be registered and resampling by using a flow twisting source.
EXAMPLE III
Referring to fig. 2, a deep learning based medical image registration terminal includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor executes the computer program to implement the steps of the deep learning based medical image registration method according to one or two embodiments.
In summary, the medical image registration method and the terminal based on deep learning provided by the invention preprocess and splice the moving image to be registered and the fixed image to be registered, cut and standardize the images in the preprocessing process, and provide convenience for splicing the subsequent images; the loss function of the spliced images is calculated, so that the accuracy of the spliced images can be improved; predicting a flow field between a moving image to be registered and a fixed image to be registered on the basis of the spliced images by using a U-shaped network, and resampling the moving image to be registered through the obtained flow field, so that dependence on a large number of data labels in the registration process is avoided through an unsupervised learning method, and the registration calculation speed is improved; registering the resampled moving image to be registered and the resampled fixed image to be registered, adjusting the registered image by using a loss function, measuring the distance between a predicted value and a sample value in the registration process by using the loss function, and reducing the difference of local gray scale in the image registration by combining the step of verifying the registered image so as to further improve the precision of the registered image; the U-shaped network combines the functions of object type identification and accurate positioning, so the invention uses the U-shaped network to predict the flow field, carries out image registration based on the flow field, and uses the loss function to adjust the registered image, thereby quickly realizing the registration of the medical image on the basis of ensuring the accuracy of the image.
The above description is only an embodiment of the present invention, and not intended to limit the scope of the present invention, and all equivalent changes made by using the contents of the present specification and the drawings, or applied directly or indirectly to the related technical fields, are included in the scope of the present invention.

Claims (10)

1. A medical image registration method based on deep learning is characterized by comprising the following steps:
acquiring a moving image to be registered and a fixed image to be registered, and preprocessing the moving image to be registered and the fixed image to be registered;
splicing the preprocessed moving image to be registered and the preprocessed fixed image to be registered to obtain a spliced image;
predicting the spliced image by using a U-shaped network to obtain a flow field between the moving image to be registered and the fixed image to be registered, and resampling the moving image to be registered according to the flow field;
and registering the resampled moving image to be registered and the resampled fixed image to be registered to obtain a registered image, adjusting the registered image by using a plurality of loss functions and verifying the adjusted registered image.
2. The deep learning-based medical image registration method according to claim 1, wherein the preprocessing the moving image to be registered and the fixed image to be registered includes:
standardizing the sizes of the moving image to be registered and the fixed image to be registered;
extracting the shapes of the standardized moving image and the standardized fixed image, and setting a corresponding zero array;
creating a sequence of the moving images to be registered and the fixed images to be registered based on the shapes of the normalized moving images and the normalized fixed images and the zero array;
the splicing of the preprocessed moving image to be registered and the preprocessed fixed image to be registered to obtain a spliced image comprises the following steps:
and splicing according to the sequence of the moving image to be registered and the sequence of the fixed image to be registered to obtain a spliced image.
3. The deep learning-based medical image registration method according to claim 1, wherein resampling the moving image to be registered according to the flow field comprises:
obtaining a corresponding sampling network according to the flow field;
and resampling the moving image to be registered during image space transformation through the sampling network.
4. The deep learning-based medical image registration method according to claim 1, wherein the step of stitching the preprocessed moving image to be registered and the preprocessed fixed image to be registered to obtain a stitched image comprises:
obtaining a loss function loss for penalizing the folding operation:
loss＝cc(trueY,predY)+λ||DutrueX→predY||l2+cc(trueX,predX)+λ||DupredY→predX||l2；
where cc denotes cross-correlation, true x denotes an actual fixed image, true y denotes an actual moving image, predX denotes a predicted fixed image, predY denotes a predicted moving image, λ denotes a parameter, Du denotes a valuetrueX→predYDisplacement field representing a predicted warped image, DupredY→predXRepresents the inverse displacement field corresponding to the predicted warped image, | | | | l2Representing a norm form;
and adjusting the spliced image through a loss function for punishing the folding operation and a vectorization loss function.
5. The deep learning-based medical image registration method according to any one of claims 1 to 4, wherein the predicting the stitched image using a U-type network obtains a flow field between the moving image to be registered and the fixed image to be registered, and the resampling the moving image to be registered according to the flow field comprises:
carrying out deformation construction on the spliced image, and predicting a deformation construction result by using a U-shaped network;
and converting the prediction result into a flow field between the moving image to be registered and the fixed image to be registered, and twisting the moving image to be registered by using the flow field and resampling.
6. A deep learning based medical image registration terminal comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor implements the following steps when executing the computer program:
acquiring a moving image to be registered and a fixed image to be registered, and preprocessing the moving image to be registered and the fixed image to be registered;
splicing the preprocessed moving image to be registered and the preprocessed fixed image to be registered to obtain a spliced image;
predicting the spliced image by using a U-shaped network to obtain a flow field between the moving image to be registered and the fixed image to be registered, and resampling the moving image to be registered according to the flow field;
and registering the resampled moving image to be registered and the resampled fixed image to be registered to obtain a registered image, adjusting the registered image by using a plurality of loss functions and verifying the adjusted registered image.
7. The deep learning-based medical image registration terminal according to claim 6, wherein the preprocessing the moving image to be registered and the fixed image to be registered includes:
standardizing the sizes of the moving image to be registered and the fixed image to be registered;
extracting the shapes of the standardized moving image and the standardized fixed image, and setting a corresponding zero array;
creating a sequence of the moving images to be registered and the fixed images to be registered based on the shapes of the normalized moving images and the normalized fixed images and the zero array;
the splicing of the preprocessed moving image to be registered and the preprocessed fixed image to be registered to obtain a spliced image comprises the following steps:
and splicing according to the sequence of the moving image to be registered and the sequence of the fixed image to be registered to obtain a spliced image.
8. The deep learning-based medical image registration terminal according to claim 6, wherein resampling the moving image to be registered according to the flow field comprises:
obtaining a corresponding sampling network according to the flow field;
and resampling the moving image to be registered during image space transformation through the sampling network.
9. The deep learning-based medical image registration terminal according to claim 6, wherein the step of splicing the preprocessed moving image to be registered and the preprocessed fixed image to be registered to obtain a spliced image comprises:
obtaining a loss function loss for penalizing the folding operation:
loss＝cc(trueY,predY)+λ||DutrueX→predY||l2+cc(trueX,predX)+λ||DupredY→predX||l2；
where cc denotes cross-correlation, true x denotes an actual fixed image, true y denotes an actual moving image, predX denotes a predicted fixed image, predY denotes a predicted moving image, λ denotes a parameter, Du denotes a valuetrueX→predYDisplacement field representing a predicted warped image, DupredY→predXRepresents the inverse displacement field corresponding to the predicted warped image, | | | | l2Representing a norm form;
and adjusting the spliced image through a loss function for punishing the folding operation and a vectorization loss function.
10. The deep learning-based medical image registration terminal according to any one of claims 6 to 9, wherein the predicting the stitched image using a U-type network obtains a flow field between the moving image to be registered and the fixed image to be registered, and the resampling the moving image to be registered according to the flow field comprises:
carrying out deformation construction on the spliced image, and predicting a deformation construction result by using a U-shaped network;
and converting the prediction result into a flow field between the moving image to be registered and the fixed image to be registered, and twisting the moving image to be registered by using the flow field and resampling.
CN202110528670.0A 2021-05-14 2021-05-14 Deep learning-based medical image registration method and terminal Active CN113269815B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202110528670.0A CN113269815B (en) 2021-05-14 2021-05-14 Deep learning-based medical image registration method and terminal

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202110528670.0A CN113269815B (en) 2021-05-14 2021-05-14 Deep learning-based medical image registration method and terminal

Publications (2)

Publication Number Publication Date
CN113269815A true CN113269815A (en) 2021-08-17
CN113269815B CN113269815B (en) 2022-10-25

Family

ID=77230896

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202110528670.0A Active CN113269815B (en) 2021-05-14 2021-05-14 Deep learning-based medical image registration method and terminal

Country Status (1)

Country Link
CN (1) CN113269815B (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115375971A (en) * 2022-08-24 2022-11-22 北京医智影科技有限公司 Multi-modal medical image registration model training method, registration method, system and equipment

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20100259263A1 (en) * 2007-11-14 2010-10-14 Dominic Holland Longitudinal registration of anatomy in magnetic resonance imaging
US20190205766A1 (en) * 2018-01-03 2019-07-04 Siemens Healthcare Gmbh Medical Imaging Diffeomorphic Registration based on Machine Learning
CN111260705A (en) * 2020-01-13 2020-06-09 武汉大学 Prostate MR image multi-task registration method based on deep convolutional neural network
CN111275749A (en) * 2020-01-21 2020-06-12 沈阳先进医疗设备技术孵化中心有限公司 Image registration and neural network training method and device
CN112102373A (en) * 2020-07-29 2020-12-18 浙江工业大学 Carotid artery multi-mode image registration method based on strong constraint affine deformation feature learning
CN112734798A (en) * 2019-11-27 2021-04-30 上海联影智能医疗科技有限公司 Online adaptive system and method for neural network

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20100259263A1 (en) * 2007-11-14 2010-10-14 Dominic Holland Longitudinal registration of anatomy in magnetic resonance imaging
US20190205766A1 (en) * 2018-01-03 2019-07-04 Siemens Healthcare Gmbh Medical Imaging Diffeomorphic Registration based on Machine Learning
CN112734798A (en) * 2019-11-27 2021-04-30 上海联影智能医疗科技有限公司 Online adaptive system and method for neural network
CN111260705A (en) * 2020-01-13 2020-06-09 武汉大学 Prostate MR image multi-task registration method based on deep convolutional neural network
CN111275749A (en) * 2020-01-21 2020-06-12 沈阳先进医疗设备技术孵化中心有限公司 Image registration and neural network training method and device
CN112102373A (en) * 2020-07-29 2020-12-18 浙江工业大学 Carotid artery multi-mode image registration method based on strong constraint affine deformation feature learning

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
张硕: "标准脑图谱与人脑MRI图像的配准研究", 《中国优秀博硕士学位论文全文数据库(硕士)医药卫生科技辑》 *
彭清河 等: "图像配准方式对宫颈癌后装自适应放射治疗图像配准精度的影响", 《南方医科大学学报》 *

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115375971A (en) * 2022-08-24 2022-11-22 北京医智影科技有限公司 Multi-modal medical image registration model training method, registration method, system and equipment

Also Published As

Publication number Publication date
CN113269815B (en) 2022-10-25

Similar Documents

Publication Publication Date Title
CN113077471B (en) Medical image segmentation method based on U-shaped network
CN111950453B (en) Random shape text recognition method based on selective attention mechanism
CN113450396B (en) Three-dimensional/two-dimensional image registration method and device based on bone characteristics
CN112132878B (en) End-to-end brain nuclear magnetic resonance image registration method based on convolutional neural network
CN115457020B (en) 2D medical image registration method fusing residual image information
CN115731178A (en) Cross-modal unsupervised domain self-adaptive medical image segmentation method
CN114399829B (en) Posture migration method based on generative countermeasure network, electronic device and medium
Daihong et al. Facial expression recognition based on attention mechanism
CN113269815B (en) Deep learning-based medical image registration method and terminal
CN117274599A (en) Brain magnetic resonance segmentation method and system based on combined double-task self-encoder
CN112990359B (en) Image data processing method, device, computer and storage medium
CN114283152A (en) Image processing method, image processing model training method, image processing device, image processing equipment and image processing medium
CN116912367B (en) Method and system for generating image based on lightweight dynamic refinement text
CN116912268A (en) Skin lesion image segmentation method, device, equipment and storage medium
CN116823850A (en) Cardiac MRI segmentation method and system based on U-Net and transducer fusion improvement
Ahlberg Real-time facial feature tracking using an active model with fast image warping
CN112784800B (en) Face key point detection method based on neural network and shape constraint
CN113450394B (en) Different-size image registration method based on Siamese network
CN111598904B (en) Image segmentation method, device, equipment and storage medium
Wang et al. A novel multi-scale architecture driven by decoupled semantic attention transfer for person image generation
Yan et al. Optimized single-image super-resolution reconstruction: A multimodal approach based on reversible guidance and cyclical knowledge distillation
CN114219900B (en) Three-dimensional scene reconstruction method, reconstruction system and application based on mixed reality glasses
CN117292144A (en) Sonar image simulation method based on generation countermeasure network
Hussain et al. Efficiency Enhanced Super Resolution Gener ative Adver sar ial Network via Advanced Knowledge Distillation
CN117197456A (en) HE dyeing-oriented pathological image cell nucleus simultaneous segmentation classification method

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant