CN111260705B - Prostate MR image multi-task registration method based on deep convolutional neural network - Google Patents

Prostate MR image multi-task registration method based on deep convolutional neural network Download PDF

Info

Publication number
CN111260705B
CN111260705B CN202010030035.5A CN202010030035A CN111260705B CN 111260705 B CN111260705 B CN 111260705B CN 202010030035 A CN202010030035 A CN 202010030035A CN 111260705 B CN111260705 B CN 111260705B
Authority
CN
China
Prior art keywords
image
registration
vector field
displacement vector
convolution
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202010030035.5A
Other languages
Chinese (zh)
Other versions
CN111260705A (en
Inventor
杜博
廖健东
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Wuhan University WHU
Original Assignee
Wuhan University WHU
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Wuhan University WHU filed Critical Wuhan University WHU
Priority to CN202010030035.5A priority Critical patent/CN111260705B/en
Publication of CN111260705A publication Critical patent/CN111260705A/en
Application granted granted Critical
Publication of CN111260705B publication Critical patent/CN111260705B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/30—Determination of transform parameters for the alignment of images, i.e. image registration
    • G06T7/38—Registration of image sequences
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/25—Fusion techniques
    • G06F18/253—Fusion techniques of extracted features
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/0002—Inspection of images, e.g. flaw detection
    • G06T7/0012—Biomedical image inspection
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/30—Determination of transform parameters for the alignment of images, i.e. image registration
    • G06T7/33—Determination of transform parameters for the alignment of images, i.e. image registration using feature-based methods
    • G06T7/344—Determination of transform parameters for the alignment of images, i.e. image registration using feature-based methods involving models
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10116—X-ray image
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30004—Biomedical image processing
    • G06T2207/30081—Prostate

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Data Mining & Analysis (AREA)
  • General Engineering & Computer Science (AREA)
  • Health & Medical Sciences (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Artificial Intelligence (AREA)
  • Evolutionary Computation (AREA)
  • General Health & Medical Sciences (AREA)
  • Computational Linguistics (AREA)
  • Biomedical Technology (AREA)
  • Molecular Biology (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Biophysics (AREA)
  • Computing Systems (AREA)
  • Evolutionary Biology (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Medical Informatics (AREA)
  • Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
  • Radiology & Medical Imaging (AREA)
  • Quality & Reliability (AREA)
  • Image Analysis (AREA)
  • Image Processing (AREA)
  • Magnetic Resonance Imaging Apparatus (AREA)

Abstract

The invention discloses a prostate MR image multi-task registration method based on a deep convolutional neural network, which comprises a training stage and an inference stage. The training phase comprises preprocessing of the prostate MR image, building of a neural network model and training of neural network parameters. By expanding the one-way registration task, a multi-task combined training model is realized, the label information of the prostate is used as weak supervision information to guide the training of the network, the cyclic consistency and the inverse consistency are used for restraining the network training, and meanwhile, a dual-path deep convolutional neural network is constructed to realize the sharing of the network weight. And meanwhile, the displacement vector field predicted by the network is regularized, so that the displacement vector field is smoother. In the inference stage, the preprocessed moving image and reference image data are used as the input of a network, a predicted displacement vector field can be obtained through the trained network, and the displacement vector field acts on the moving image, so that the result of the registration of the MR image of the prostate is obtained.

Description

Prostate MR image multi-task registration method based on deep convolutional neural network
Technical Field
The invention belongs to the technical field of medical image processing, and particularly relates to a prostate MR image multi-task registration method based on a deep convolutional neural network.
Background
With the development of computer technology and medical image engineering, more and more scientific technologies are applied to modern medicine, particularly medical imaging technology, which permeates clinical application of medicine, and various medical imaging devices reflect the physical condition of human body from different sides and provide direct and objective information for medical diagnosis and treatment. Nuclear Magnetic Resonance Imaging (MRI), computed tomography (X-ray), ultrasound imaging, Computed Tomography (CT), positron emission computed tomography (PET), etc., common to modern medical imaging techniques, are mainly divided into two main categories: the anatomical image and the functional image have high resolution, can accurately acquire the structural information of the visceral organs, but do not have functional function. The functional image has low resolution, and cannot clearly display the contour information of the organs, but can display the metabolism condition of the human body. Although both imaging technologies are continuously advanced and the imaging result is more and more accurate, in practical clinical application, due to the difference of the imaging principles, a certain imaging technology can only reflect certain specific information of a human body, and a doctor is often required to combine a plurality of imaging technologies to diagnose and treat a patient. However, the information of a plurality of medical images often needs years of experience, is not only limited to a plurality of subjective factors, but also increases the workload of doctors.
The best method for solving the problem is a medical image registration technology, and medical image registration is an important research branch of medical image analysis, is a core technology of medical image fusion and reconstruction, and has important significance in clinical application. Medical image registration techniques refer to geometrically aligning two or more images such that the same pixel represents the same anatomical location. Multiple imaging technologies can be organically combined through a medical image registration technology, multiple medical information is integrated on one image, and a doctor is more intuitively and accurately assisted in diagnosis and treatment.
Image registration techniques can be divided into rigid and non-rigid registration. In the last 80 th century, the registration of medical images mainly takes rigid body registration as a main part, and rigid transformation coefficients between images are estimated by learning medical information such as image gray difference and the like. At present, the algorithm based on rigid body registration is mature and widely applied clinically. But the degree of freedom of rigid body registration is small, and only a small part of the registration problem is solved. For practical medical application, rigid body transformation is far from meeting practical requirements, and more degrees of freedom are needed for the problem of medical image registration. Since the 21 st century, the non-rigid registration technology has been the focus of research and has been rapidly developed, and many scholars have developed many non-linear transformation modes, such as free transformation models based on B-splines, elastic transformation models, physical deformation models based on optical flow diffusion, and the like. The classical non-rigid image registration algorithm usually selects a transformation model at first, then defines a similarity index, and finally optimizes transformation parameters in an iterative manner. Although the classical algorithm achieves good performance, due to the nature of iterative optimization, the speed is often slow, cannot meet the clinical real-time requirements, and may fall into local optima. In addition, different similarity measures have different properties, and different similarity measures need to be defined for different organs or images.
In recent years, deep learning, particularly deep convolutional neural networks, have achieved breakthrough research results in terms of computer vision, and have also rapidly developed in terms of medical image analysis, and deep convolutional neural networks have also achieved certain results in terms of medical image registration, such as DIRNet, voxelmorphh, and the like. Although the training time of the model is long, once the training is finished, the model can rapidly register images to better meet the clinical requirement. Although the current deep learning-based algorithm achieves satisfactory results, the existing method has more defects, and the registration result is often closely related to the size of the training data set, and the physical properties of the displacement vector field are ignored.
Disclosure of Invention
Aiming at the defects in the prior art, the invention provides a high-precision and high-efficiency multitask registration method for a prostate MR image based on a deep convolutional neural network.
In order to solve the technical problems, the invention adopts the following technical scheme: a prostate MR image multi-task registration method based on a deep convolutional neural network comprises the following steps:
step 1, carrying out uniform format processing and screening on prostate MR images with different formats;
step 2, cutting and normalizing the image in a pixel range, counting the voxel space distribution of the image, and resampling the image to unify the voxel space;
step 3, designing a double-path deep convolution neural network model to realize the registration of a plurality of images, wherein the specific processing process of the network model is as follows,
respectively extracting characteristic information of different scales of a moving image and a reference image through convolution layers and down-sampling layers of two different paths in an encoding stage, namely dividing the moving image and the reference image into an upper branch and a lower branch in the encoding stage, wherein the upper branch is input into the moving image, the lower branch is input into the reference image, the upper branch and the lower branch in the encoding stage have the same structural parameters and operation and comprise 10 convolution layers and 5 down-sampling layers, and the down-sampling layers are processed after each two convolution layers; in the decoding stage, the features learned in the encoding stage are decoded, firstly, feature maps of an upper branch and a lower branch are superposed, then, the features of the upper direct current and the lower direct current are fused through 10 convolutional layers and 5 upper sampling layers, the size of a displacement vector field output after decoding is the same as that of an original image, and an upper sampling layer is passed after every two convolutional layers; simultaneously, the feature map obtained after each up-sampling is overlapped with the feature maps of the upper branch and the lower branch in the same scale in the coding stage through jump connection; the last layer uses 1 × 1 convolution to reduce the dimension of the data and output the final displacement vector field;
step 4, guiding the training of the dual-path deep convolutional neural network model by using the label information of the prostate as weak supervision information, and regularizing the displacement vector field;
and 5, inputting the moving image and the reference image into the trained network model to obtain a predicted displacement vector field, and resampling the moving image by using the displacement vector field to obtain a corresponding registration result.
Further, in step 3, in the encoding stage, the sizes of convolution kernels are all 3 × 3, wherein the convolution step size of a convolution layer is 1, the convolution step size of a down-sampling layer is 2, and the number of characteristic channels of the convolution layer is doubled after each down-sampling layer is performed, and the number of characteristic channels is respectively 32, 64, 128, 256 and 512; in the decoding stage, the convolution kernels are all 3 × 3 in size, the convolution step size of the convolutional layer is 1, the convolution step size of the upsampling layer is 2, and the number of characteristic channels of the convolutional layer after each 1-time upsampling layer is reduced by half, wherein the number of the characteristic channels is 512, 256, 128, 64 and 32.
Furthermore, in step 4, the multi-task interconnection is realized by using the constraint of cycle consistency and inverse consistency in the training process, which is specifically realized as follows,
multitasking refers to A → B → A'; b → A → B'; a' → A → A "; b ' → B ″, where a → B → a ' is taken as an example, a denotes a moving image, B denotes a reference image, and a ' denotes a result image of registration; the mathematical expressions of the cyclic consistency constraint and the inverse consistency constraint are as follows:
cycle consistency:
|A-A″|2+|B-B″|2
inverse consistency:
|ΦAB-ΦB′B|2+|ΦBA-ΦA′A|2
wherein phiABRepresenting the displacement vector field, phi, predicted from the image A registration to the image B network modelB′BRepresenting the displacement vector field, phi, predicted from the registration of image B' to the network model of image BBARepresenting the displacement vector field, phi, predicted from the image B registration to the image A network modelA′ARepresenting the displacement vector field predicted from the registration of image a' to the image a network model.
Further, the regularization of the displacement vector field in step 4 includes range-constrained regularization, L2Regularization and reverse folding constraints, wherein the mathematical expression for range constraint regularization is as follows:
Figure BDA0002363968380000041
wherein p represents a pixel point, Ω represents all pixels in space, Φ () represents a generated displacement vector field, h () represents a deformed pixel grid, obtained by adding the displacement vector field to the original coordinates,
Figure BDA0002363968380000042
in order to displace the gradient of the vector field,
Figure BDA0002363968380000043
is L2Regularization, fσIs a piecewise function;
Figure BDA0002363968380000044
where s represents the size of the pixel grid, i.e. the size of the image;
the mathematical expression for the unfolding constraint is as follows:
Figure BDA0002363968380000045
wherein Relu is used to penalize the position where the fold is created,
Figure BDA0002363968380000046
when it indicates that there is a fold, the other values indicate no fold, i.e. no penalty.
The invention has the beneficial effects that:
(1) the invention provides a multi-task registration network, which expands a single task into four tasks, increases the data volume to a certain extent, relieves the problem of small available data volume of medical images, and further improves the registration accuracy.
(2) The invention provides a self-adaptive registration algorithm based on anatomical constraint, wherein an anatomical label of an image represents high-level information of the image, the corresponding relation of the image is learned from the high-level information, and the model is prevented from being trained by utilizing the similarity of the image. The Gaussian matrix is used for carrying out spatial smooth probability mapping on the labels, different label information corresponds to different mapping results, the mapping results are used as weak supervision information, overfitting of the network is reduced, training of the registration network is guided, and the problems of large deformation and difficult registration between image pairs are solved.
(3) The invention provides a double-path convolutional neural network model, which is characterized in that the characteristic information of a moving image and a reference image is respectively learned through two paths in an encoding stage, the characteristic information of the images can be better utilized through jump connection and multi-scale fusion in a decoding stage, and meanwhile, weight sharing can be realized through a double-path network, so that the consumption of resources is reduced.
(4) The invention provides cycle consistency constraint and inverse consistency constraint in the multitask registration of the prostate MR image of the deep convolutional neural network, guides the cooperative training among the four tasks, and further improves the registration precision by carrying out range constraint on a predicted displacement vector field.
Drawings
FIG. 1 is a diagram of the multitask registration of the present invention;
FIG. 2 is a diagram of a dual-path convolutional neural network structure according to the present invention.
Detailed Description
For the convenience of those skilled in the art to understand and implement the technical solution of the present invention, the following detailed description of the present invention is provided in conjunction with the accompanying drawings and examples, it is to be understood that the embodiments described herein are only for illustrating and explaining the present invention and are not to be construed as limiting the present invention.
The invention discloses a prostate MR image multi-task registration method based on a deep convolutional neural network, which comprises a training stage and an inference stage. The training phase comprises preprocessing of the prostate MR image, building of a neural network model and training of neural network parameters. The data preprocessing stage comprises data screening, data uniform storage format, pixel normalization and clipping and image resampling. By expanding the one-way registration task, a multi-task combined training model is realized, the label information of the prostate is used as weak supervision information to guide the training of the network, the cyclic consistency and the inverse consistency are used for restraining the network training, and meanwhile, a dual-path deep convolutional neural network is constructed to realize the sharing of the network weight. And meanwhile, the displacement vector field predicted by the network is regularized, so that the displacement vector field is smoother. In the inference stage, the preprocessed moving image and reference image data are used as the input of a network, a predicted displacement vector field can be obtained through the trained network, and the displacement vector field acts on the moving image, so that the result of the registration of the MR image of the prostate is obtained.
The implementation is realized by adopting a Python platform based on a Tensorflow library, and a Python medical image read-write function is taken as an implementation basis. Calling a medical image reading function, inputting a file name of the medical image, reading the medical image into a matrix with the size of X multiplied by Y multiplied by Z, wherein each element in the matrix is a corresponding pixel value of each dimension, Z is the number of slices of the medical image, and X and Y are the length and the width of the medical image respectively. The Python medical image read/write function is well known in the art and will not be described herein.
In an embodiment, the following operations are performed on the medical image based on the matrix X × Y × Z:
(1) and (4) uniformly formatting and screening the prostate MR images with different formats.
In the embodiment, the specific operation of step (1) is as follows: because the formats of the prostate MR image files acquired by different devices are not consistent, the formats of the prostate MR image files are firstly unified into mhd file format. Meanwhile, images without prostate may exist in the acquired images, so that the images are screened by the labels of the anatomy of the prostate, and only the images with the prostate are selected, thereby facilitating later training.
(2) And (4) cutting and normalizing the image in a pixel range, counting the voxel space distribution of the image, and resampling the image to unify the voxel space. .
In the embodiment, the specific operation of step (2) is as follows: the imaging principle of the medical image is different from that of a natural image, the range of the gray value of the acquired data set is large, the gray value is cut, only the pixel value between 0.5% and 99.5% is reserved, the appearance of abnormal values is reduced, and the contrast of the image is increased. Meanwhile, normalization is carried out, and training of the network model is facilitated. Besides, different imaging modes often cause inconsistency of voxels between images, that is, physical distances represented by two pixel points are different, so that the voxel intervals of statistical data are needed, and the voxel intervals are unified by adopting a mode of resampling the images.
(3) And designing a double-path deep convolution neural network model to realize multi-task registration.
In the embodiment, the specific operation of step (3) is as follows: in the field of medical image processing, medical image data often becomes a key for training a network model, in a current model based on a deep convolutional neural network, only a one-way registration (a → B → a ') process from an image moving image (a) to a reference image (B) is considered, a ' is a registration result, and inverse consistency and cyclic consistency of image deformation are ignored, so that the one-way registration is expanded, and as shown in fig. 1, a → B → a ' is respectively realized; b → A → B'; a' → A → A "; b ' → B ″, and taking a → B → a ' as an example, a denotes a moving image, B denotes a reference image, and a ' denotes a result image of registration. In order to realize weight sharing of a task network, a dual-path convolutional neural network is designed, as shown in fig. 2, feature information of different scales of a moving image and a reference image is respectively extracted through convolutional layers and downsampling layers of two different paths at a coding stage, namely, the moving image and the reference image are divided into an upper tributary and a lower tributary at the coding stage, the upper tributary is input into the moving image, the lower tributary is input into the reference image, structural parameters and operations of the upper tributary and the lower tributary at the coding stage are the same, the convolutional layers and the 5 downsampling layers are all included, the size of the convolutional layers is 3 × 3, the convolutional step size of the convolutional layers is 1, and the convolutional step size of the downsampling layers is 2. After every two times of convolution layers, the number of the convolution layer characteristic channels is doubled after every one time of downsampling layers, and the number of the convolution layer characteristic channels is respectively 32, 64, 128, 256 and 512. In the decoding stage, the features learned in the encoding stage are mainly decoded, the feature maps of the upper and lower branches are firstly superposed, then the features of the upper and lower direct currents are fused through 10 convolutional layers and 5 upper mining layers, and the size of a displacement vector field output after decoding is the same as that of an original image. Meanwhile, the feature map obtained after each up-sampling is overlapped with the feature maps of the upper branch and the lower branch in the same scale in the coding stage through jump connection, for example: the feature map scale before the 1 st down-sampling of the upper and lower branches is the same as the feature map scale after 5 up-sampling, so as to improve the shortage of up-sampling information by using the feature map of the bottom layer. Meanwhile, the number of characteristic channels is halved for each convolution layer after 1-time upsampling layer, namely 512, 256, 128, 64 and 32, which correspond to the encoding stage, wherein the convolution kernel size is 3 multiplied by 3, the convolution step size of the convolution layer is 1, and the convolution step size of the upsampling layer is 2. And the final layer uses 1 multiplied by 1 convolution to reduce the dimension of the data and output the final displacement vector field. The displacement vector field is a matrix of two channels of the original image size, which respectively represents the moving distance of the pixel point along the x axis and the moving distance along the y axis,
(4) the label information of the prostate is used as weak supervision information to guide the training of the network, the cycle consistency and inverse consistency constraint are used to realize multi-task interconnection, and the displacement vector field is regularized.
In the embodiment, the specific operation of step (4) is as follows: because the time interval of the collected images is large, the deformation between the images is huge, and the network is difficult to train by directly utilizing the similarity measurement, the anatomical label of the prostate is used as the priori knowledge to guide the training of the network, and the anatomical label is directly used as the weak supervision information, so that the overfitting of the network is caused, and the training of the network is not facilitated. And performing spatial smooth probability mapping on the labels by utilizing a Gaussian matrix, wherein the size of the Gaussian matrix is consistent by the minimum external square of the prostate anatomy, different label information corresponds to different mapping results, and the mapping results are used as weak supervision information to guide the training of the registration network. Meanwhile, potential association among the four tasks is explored, a cycle consistency constraint and an inverse consistency constraint are designed, and the mathematical expression of the constraint is as follows:
cycle consistency:
|A-A″|2+|B-B″|2
inverse consistency:
|ΦAB-ΦB′B|2+|ΦBA-ΦA′A|2
wherein phiABRepresenting the displacement vector field, phi, predicted from the registration of image A to B networkB′B、ΦBA、ΦA′AAnd so on.
Meanwhile, range constraint regularization is proposed, and a mathematical expression is as follows:
Figure BDA0002363968380000071
wherein p represents a pixel point, Ω represents all pixels in the space, Φ () represents a generated displacement vector field, h () represents a deformed pixel grid, obtained by adding the displacement vector field and the original coordinates,
Figure BDA0002363968380000072
in order to displace the gradient of the vector field,
Figure BDA0002363968380000073
is L2Regularization, fσIs a piecewise function.
Figure BDA0002363968380000074
Where s represents the size of the pixel grid, i.e. the size of the image.
In addition thereto also use L2Regularization and reverse folding constraints, the mathematical expression of which is as follows:
Figure BDA0002363968380000081
wherein Relu is used to penalize the position where the fold is created,
Figure BDA0002363968380000082
when it indicates that there is a fold, the other values indicate no fold, i.e. no penalty. The details are well known in the art and will not be described herein. And (3) training the network by using the constraint function, inputting A, B and B, A, and then inputting A ', A and B', B, training for 200 rounds, and setting the learning rate to be 10 e-4.
(5) And inputting the moving image and the reference image into a double-path depth convolution neural network model to obtain a predicted displacement vector field, and resampling the moving image by using the displacement vector field to obtain a corresponding registration result.
In the embodiment, the specific operation of step (5) is as follows: the motion image and the reference image are input into a registration network, the registration network can predict a corresponding displacement vector field, the displacement vector field acts on the motion image, the position of each pixel point after displacement can be obtained, and the point after displacement is not on an integer coordinate possibly, so resampling is needed, and a final registration result is obtained.
In specific implementation, the automatic operation of the process can be realized by adopting a software mode. The apparatus for operating the process should also be within the scope of the present invention.
The advantageous effects of the present invention are verified by comparative experiments as follows.
The data used in this experiment were prostate MR images, including 72 patient datasets, acquired twice each, with time intervals up to one year, with rectal coils being used for the first acquisition of the images for better focus observation, and no rectal coils for the second acquisition to increase patient comfort, and for better training of the network, the datasets were augmented with 2D training mode, image size 288 x 288, with voxel spacing of 0.3516 mm. The most advanced classical image normalization method (SyN), the DIRNet based on the convolutional neural network and the registration algorithm (Reg _ Ant) based on the weak supervision are adopted for comparison respectively, and the method is taken as an example of a specific implementation mode.
Medical image registration evaluation index: dice and HD.
In the field of image registration, due to the lack of gold standard, the image registration is generally performed by using segmented evaluation indexes Dice and a hausdorff distance HD, wherein the higher Dice indicates that the regions of interest are overlapped more, i.e. the effect is better. The Hausdorff distance is mainly used for evaluating the edge matching degree of the region of interest, and the lower the HD is, the better the registration effect is.
Figure BDA0002363968380000083
Wherein, A and B respectively represent the interesting region anatomical labels, | A ≦ B | represents the intersection of A and B.
Figure BDA0002363968380000091
Where a and B are the points of set a and set B, respectively, and d (a, B) represents the distance between a and B.
TABLE 1 comparative test results
The method of the invention SyN DIRNet Reg_Anat
Dice 86.36(±0.0069) 83.19(±0.0083) 80.28(±0.0097) 84.89(±0.0032)
HD 6.3860(±0.6019) 9.0284(±0.5458) 13.3911(±0.3920) 8.6822(±0.3792)
The standard deviation is shown in parentheses, and as can be seen from table 1, the method of the present invention is optimal in both evaluation indexes, which indicates that the method of the present invention has a better image registration effect. Compared with the current optimal classical registration method SyN, the method has a great improvement, and from the aspect of HD, the method has a better image edge processing effect, consumes far less time than the SyN method, and can better meet clinical requirements. Compared with a deep learning-based method, the method has a better effect on restraining the wrong registration, and the generated displacement vector field is more in line with the physical property.
It can be concluded that the method of the present invention has a higher registration accuracy compared to existing medical image registration methods. The method solves the problem that the registration training cannot be carried out by using a deep network due to insufficient training samples, realizes multi-task registration by using a dual-path convolution neural network, can increase a training data set, and further corrects the prediction of a displacement vector field by using the cyclic consistency and the inverse consistency. The invention provides range-constrained regularization, which can effectively inhibit wrong registration, reduce irregular deformation of images and further improve the registration precision.
It should be understood that parts of the specification not set forth in detail are well within the prior art.
It should be understood that the above description of the preferred embodiments is given for clarity and not for any purpose of limitation, and that various changes, substitutions and alterations can be made herein without departing from the spirit and scope of the invention as defined by the appended claims.

Claims (3)

1. A prostate MR image multitask registration method based on a deep convolutional neural network is characterized by comprising the following steps:
step 1, carrying out uniform format processing and screening on prostate MR images with different formats;
step 2, cutting and normalizing the image in a pixel range, counting the voxel space distribution of the image, and resampling the image to unify the voxel space;
step 3, designing a double-path deep convolution neural network model to realize the registration of a plurality of images, wherein the specific processing process of the network model is as follows,
respectively extracting characteristic information of different scales of a moving image and a reference image through convolution layers and down-sampling layers of two different paths in an encoding stage, namely dividing the moving image and the reference image into an upper branch and a lower branch in the encoding stage, wherein the upper branch is input into the moving image, the lower branch is input into the reference image, the upper branch and the lower branch in the encoding stage have the same structural parameters and operation and comprise 10 convolution layers and 5 down-sampling layers, and the down-sampling layers are processed after each two convolution layers; in the decoding stage, the features learned in the encoding stage are decoded, firstly, feature maps of an upper branch and a lower branch are superposed, then, the features of the upper direct current and the lower direct current are fused through 10 convolutional layers and 5 upper sampling layers, the size of a displacement vector field output after decoding is the same as that of an original image, and an upper sampling layer is passed after every two convolutional layers; simultaneously, the feature map obtained after each up-sampling is overlapped with the feature maps of the upper branch and the lower branch in the same scale in the coding stage through jump connection; the last layer uses 1 × 1 convolution to reduce the dimension of the data and output the final displacement vector field;
step 4, guiding the training of the dual-path deep convolutional neural network model by using the label information of the prostate as weak supervision information, and regularizing the displacement vector field;
in the step 4, the multi-task interconnection is realized by using the constraint of cycle consistency and inverse consistency in the training process, which is concretely realized as follows,
multitasking refers to A → B → A'; b → A → B'; a' → A → A "; b ' → B ″, where a → B → a ' is taken as an example, a denotes a moving image, B denotes a reference image, and a ' denotes a result image of registration; the mathematical expressions of the cyclic consistency constraint and the inverse consistency constraint are as follows:
cycle consistency:
|A-A″|2+|B-B″|2
inverse consistency:
|ΦAB-ΦB′B|2+|ΦBA-ΦA′A|2
wherein phiABRepresenting the displacement vector field, phi, predicted from the image A registration to the image B network modelB′BRepresenting the displacement vector field, phi, predicted from the registration of image B' to the network model of image BBARepresenting the displacement vector field, phi, predicted from the image B registration to the image A network modelA′ARepresenting the displacement vector field predicted from the registration of image a' to the image a network model;
and 5, inputting the moving image and the reference image into the trained network model to obtain a predicted displacement vector field, and resampling the moving image by using the displacement vector field to obtain a corresponding registration result.
2. The method for multitask registration of the MR image of the prostate based on the deep convolutional neural network as claimed in claim 1, wherein: in the step 3, in the encoding stage, the sizes of convolution kernels are all 3 × 3, wherein the convolution step size of a convolution layer is 1, the convolution step size of a down-sampling layer is 2, and the number of characteristic channels of the convolution layer is doubled after each down-sampling layer is carried out, namely 32, 64, 128, 256 and 512; in the decoding stage, the convolution kernels are all 3 × 3 in size, the convolution step size of the convolutional layer is 1, the convolution step size of the upsampling layer is 2, and the number of characteristic channels of the convolutional layer after each 1-time upsampling layer is reduced by half, wherein the number of the characteristic channels is 512, 256, 128, 64 and 32.
3. The method for multitask registration of the MR image of the prostate based on the deep convolutional neural network as claimed in claim 1, wherein: the regularization of the displacement vector field in step 4 includes range-constrained regularization, L2Regularization and reverse folding constraints, wherein the mathematical expression for range constraint regularization is as follows:
Figure FDA0003454115450000021
wherein p represents a pixel point, Ω represents all pixels in space, Φ () represents a generated displacement vector field, h () represents a deformed pixel grid, obtained by adding the displacement vector field to the original coordinates,
Figure FDA0003454115450000022
in order to displace the gradient of the vector field,
Figure FDA0003454115450000023
is L2Regularization, fσIs a piecewise function;
Figure FDA0003454115450000024
where s represents the size of the pixel grid, i.e. the size of the image;
the mathematical expression for the unfolding constraint is as follows:
Figure FDA0003454115450000025
wherein Relu is used to penalize the position where the fold is created,
Figure FDA0003454115450000026
when it indicates that there is a fold, the other values indicate no fold, i.e. no penalty.
CN202010030035.5A 2020-01-13 2020-01-13 Prostate MR image multi-task registration method based on deep convolutional neural network Active CN111260705B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202010030035.5A CN111260705B (en) 2020-01-13 2020-01-13 Prostate MR image multi-task registration method based on deep convolutional neural network

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202010030035.5A CN111260705B (en) 2020-01-13 2020-01-13 Prostate MR image multi-task registration method based on deep convolutional neural network

Publications (2)

Publication Number Publication Date
CN111260705A CN111260705A (en) 2020-06-09
CN111260705B true CN111260705B (en) 2022-03-15

Family

ID=70946880

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202010030035.5A Active CN111260705B (en) 2020-01-13 2020-01-13 Prostate MR image multi-task registration method based on deep convolutional neural network

Country Status (1)

Country Link
CN (1) CN111260705B (en)

Families Citing this family (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112132878B (en) * 2020-11-03 2024-04-05 贵州大学 End-to-end brain nuclear magnetic resonance image registration method based on convolutional neural network
CN112634284B (en) * 2020-12-22 2022-03-25 上海体素信息科技有限公司 Weight map loss-based staged neural network CT organ segmentation method and system
CN112991406B (en) * 2021-02-07 2023-05-23 清华大学深圳国际研究生院 Method for constructing brain map based on differential geometry technology
CN113269815B (en) * 2021-05-14 2022-10-25 中山大学肿瘤防治中心 Deep learning-based medical image registration method and terminal
CN113344876B (en) * 2021-06-08 2023-05-12 安徽大学 Deformable registration method between CT and CBCT
CN114332447B (en) * 2022-03-14 2022-08-09 浙江大华技术股份有限公司 License plate correction method, license plate correction device and computer readable storage medium
CN116523983B (en) * 2023-06-26 2023-10-27 华南师范大学 Pancreas CT image registration method integrating multipath characteristics and organ morphology guidance

Citations (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101887525A (en) * 2010-07-09 2010-11-17 北京师范大学 Grading-based positive and inverse reversing three-dimensional dense point set rapid registering method
CN106875380A (en) * 2017-01-12 2017-06-20 西安电子科技大学 A kind of heterogeneous image change detection method based on unsupervised deep neural network
CN107194872A (en) * 2017-05-02 2017-09-22 武汉大学 Remote sensed image super-resolution reconstruction method based on perception of content deep learning network
CN107403446A (en) * 2016-05-18 2017-11-28 西门子保健有限责任公司 Method and system for the image registration using intelligent human agents
CN108596902A (en) * 2018-05-04 2018-09-28 北京大学 The full reference image quality appraisement method of multitask based on gating convolutional neural networks
CN109598727A (en) * 2018-11-28 2019-04-09 北京工业大学 A kind of CT image pulmonary parenchyma three-dimensional semantic segmentation method based on deep neural network
US10426442B1 (en) * 2019-06-14 2019-10-01 Cycle Clarity, LLC Adaptive image processing in assisted reproductive imaging modalities
CN110363797A (en) * 2019-07-15 2019-10-22 东北大学 A kind of PET and CT method for registering images inhibited based on excessive deformation
CN110473196A (en) * 2019-08-14 2019-11-19 中南大学 A kind of abdominal CT images target organ method for registering based on deep learning

Family Cites Families (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2020510463A (en) * 2017-01-27 2020-04-09 アーテリーズ インコーポレイテッド Automated segmentation using full-layer convolutional networks
CN110599528B (en) * 2019-09-03 2022-05-27 济南大学 Unsupervised three-dimensional medical image registration method and system based on neural network

Patent Citations (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101887525A (en) * 2010-07-09 2010-11-17 北京师范大学 Grading-based positive and inverse reversing three-dimensional dense point set rapid registering method
CN107403446A (en) * 2016-05-18 2017-11-28 西门子保健有限责任公司 Method and system for the image registration using intelligent human agents
CN106875380A (en) * 2017-01-12 2017-06-20 西安电子科技大学 A kind of heterogeneous image change detection method based on unsupervised deep neural network
CN107194872A (en) * 2017-05-02 2017-09-22 武汉大学 Remote sensed image super-resolution reconstruction method based on perception of content deep learning network
CN108596902A (en) * 2018-05-04 2018-09-28 北京大学 The full reference image quality appraisement method of multitask based on gating convolutional neural networks
CN109598727A (en) * 2018-11-28 2019-04-09 北京工业大学 A kind of CT image pulmonary parenchyma three-dimensional semantic segmentation method based on deep neural network
US10426442B1 (en) * 2019-06-14 2019-10-01 Cycle Clarity, LLC Adaptive image processing in assisted reproductive imaging modalities
CN110363797A (en) * 2019-07-15 2019-10-22 东北大学 A kind of PET and CT method for registering images inhibited based on excessive deformation
CN110473196A (en) * 2019-08-14 2019-11-19 中南大学 A kind of abdominal CT images target organ method for registering based on deep learning

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
Prostate MR Image Segmentation With Self-Attention Adversarial Training Based on Wasserstein Distance:webofscience,prostate and MR and segmentation and attention;CHENGWEI SU 等;《SPECIAL SECTION ON ADVANCED DATA MINING METHODS FOR SOCIAL COMPUTING》;20191213;第184276-184284页 *
基于深度学习网络PSP-NET的前列腺MR图像的分割;范嵩 等;《现代电子技术》;20190615;第42卷(第12期);第148-151，155页 *

Also Published As

Publication number Publication date
CN111260705A (en) 2020-06-09

Similar Documents

Publication Publication Date Title
CN111260705B (en) Prostate MR image multi-task registration method based on deep convolutional neural network
CN110475505B (en) Automatic segmentation using full convolution network
US20230104173A1 (en) Method and system for determining blood vessel information in an image
EP4030385A1 (en) Devices and process for synthesizing images from a source nature to a target nature
CN117078692B (en) Medical ultrasonic image segmentation method and system based on self-adaptive feature fusion
CN112216371A (en) Multi-path multi-scale parallel coding and decoding network image segmentation method, system and medium
CN117274599A (en) Brain magnetic resonance segmentation method and system based on combined double-task self-encoder
Du et al. Segmentation and visualization of left atrium through a unified deep learning framework
CN115830016A (en) Medical image registration model training method and equipment
Liu et al. 3-D prostate MR and TRUS images detection and segmentation for puncture biopsy
CN112990359B (en) Image data processing method, device, computer and storage medium
CN116797612B (en) Ultrasonic image segmentation method and device based on weak supervision depth activity contour model
Jiang et al. Ori-net: Orientation-guided neural network for automated coronary arteries segmentation
CN113538209A (en) Multi-modal medical image registration method, registration system, computing device and storage medium
CN115205298B (en) Method and device for segmenting blood vessels of liver region
Tao et al. Tooth CT Image Segmentation Method Based on the U-Net Network and Attention Module.
CN116665896A (en) Model building method for predicting breast cancer axillary lymph node metastasis
CN113436155B (en) Deep learning-based ultrasonic brachial plexus image recognition method
CN116091412A (en) Method for segmenting tumor from PET/CT image
CN114529562A (en) Medical image segmentation method based on auxiliary learning task and re-segmentation constraint
CN115294023A (en) Liver tumor automatic segmentation method and device
CN111612762B (en) MRI brain tumor image generation method and system
CN113936006A (en) Segmentation method and device for processing high-noise low-quality medical image
Chen et al. Radiologically based automated segmentation of cardiac MRI using an improved U-Net neural algorithm
Peng et al. 2D brain MRI image synthesis based on lightweight denoising diffusion probabilistic model

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant