CN113643332A - Image registration method, electronic device and readable storage medium - Google Patents
Image registration method, electronic device and readable storage medium Download PDFInfo
- Publication number
- CN113643332A CN113643332A CN202110793797.5A CN202110793797A CN113643332A CN 113643332 A CN113643332 A CN 113643332A CN 202110793797 A CN202110793797 A CN 202110793797A CN 113643332 A CN113643332 A CN 113643332A
- Authority
- CN
- China
- Prior art keywords
- registration
- image
- displacement field
- picture
- network
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 59
- 238000006073 displacement reaction Methods 0.000 claims abstract description 161
- 230000011218 segmentation Effects 0.000 claims abstract description 112
- 238000012549 training Methods 0.000 claims abstract description 42
- 230000009466 transformation Effects 0.000 claims abstract description 35
- 238000003709 image segmentation Methods 0.000 claims abstract description 15
- 238000005070 sampling Methods 0.000 claims description 5
- 230000006870 function Effects 0.000 description 29
- 230000008569 process Effects 0.000 description 8
- 238000004891 communication Methods 0.000 description 6
- 230000000717 retained effect Effects 0.000 description 6
- 230000008859 change Effects 0.000 description 4
- 238000013135 deep learning Methods 0.000 description 4
- 238000010586 diagram Methods 0.000 description 4
- 238000004364 calculation method Methods 0.000 description 3
- 230000000694 effects Effects 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 230000007812 deficiency Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000003745 diagnosis Methods 0.000 description 1
- 238000002059 diagnostic imaging Methods 0.000 description 1
- 230000014759 maintenance of location Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 210000000056 organ Anatomy 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
- 238000001959 radiotherapy Methods 0.000 description 1
- 239000000126 substance Substances 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/30—Determination of transform parameters for the alignment of images, i.e. image registration
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/0002—Inspection of images, e.g. flaw detection
- G06T7/0012—Biomedical image inspection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Health & Medical Sciences (AREA)
- General Health & Medical Sciences (AREA)
- Artificial Intelligence (AREA)
- Data Mining & Analysis (AREA)
- Radiology & Medical Imaging (AREA)
- Quality & Reliability (AREA)
- Life Sciences & Earth Sciences (AREA)
- Medical Informatics (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
- Evolutionary Computation (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Image Analysis (AREA)
Abstract
The invention discloses an image registration method, electronic equipment and a readable storage medium, wherein the image registration method comprises the following steps: acquiring a moving picture and a fixed picture, and respectively carrying out prediction segmentation on the moving picture and the fixed picture through an image segmentation network; removing interference factors in the moving picture and the fixed picture according to a prediction segmentation result; determining a target displacement field based on the moving picture, the fixed picture and the initial registration network after the interference factors are removed; inputting the moving picture and the target displacement field into a space transformation network, and generating a registration image through the space transformation network. The method removes interference factors in the picture, and solves the problem that other parts except the key part cause interference to network model training.
Description
Technical Field
The present invention relates to the field of medical imaging, and in particular, to an image registration method, an electronic device, and a readable storage medium.
Background
Image registration is the process of superimposing two or more images of the same scene taken at different times, different viewpoints, and/or different sensors. In the medical field, the change conditions of focuses and organs can be quantitatively analyzed by registering dynamic images acquired at different moments, so that medical diagnosis, operation plan making and radiotherapy plan making are more accurate and reliable. The medical image registration technology can be divided into two categories, one category is based on the traditional registration method, the traditional registration method is complicated in steps and long in time consumption, iterative search is carried out on each pair of images needing registration, and the registration effect is not ideal; the other method is a registration method based on deep learning, the method based on deep learning only needs to construct a proper network and design a proper loss function, and the image registration speed is far faster than that of the traditional registration method after the model is trained. However, the image registration technology based on the deep learning also has the disadvantage that images of other parts except for the key part are reserved in the registration process, and the images of other parts can interfere with the image registration to a certain extent.
The above is only for the purpose of assisting understanding of the technical aspects of the present invention, and does not represent an admission that the above is prior art.
Disclosure of Invention
The invention mainly aims to provide an image registration method, which aims to remove interference factors in pictures and solve the problem that other part information causes interference on network model training.
To achieve the above object, the present invention provides an image registration method, comprising the steps of:
acquiring a moving picture and a fixed picture, and respectively carrying out prediction segmentation on the moving picture and the fixed picture through an image segmentation network;
removing interference factors in the moving picture and the fixed picture according to a prediction segmentation result;
determining a target displacement field based on the moving picture, the fixed picture and the initial registration network after the interference factors are removed;
inputting the moving picture and the target displacement field into a space transformation network, and generating a registration image through the space transformation network.
Further, the step of removing the interference factors in the moving picture and the fixed picture according to the prediction segmentation result comprises:
and multiplying the moving picture and the fixed picture by the corresponding prediction segmentation results respectively to remove interference factors in the moving picture and the fixed picture.
Further, the step of determining a target displacement field based on the moving picture and the fixed picture after the interference factors are removed and the initial registration network includes:
merging the moving picture and the fixed picture after the interference factors are removed, and performing down-sampling on the merged picture to obtain merged pictures with different resolutions;
obtaining displacement fields with different resolutions according to the combined pictures with different resolutions;
and combining the displacement fields with different resolutions into a target displacement field through a displacement field cascade module.
Further, the step of combining the displacement fields with different resolutions into the target displacement field by the displacement field cascade module includes:
multiplying displacements of an X axis and a Y axis in a first displacement field by corresponding weight coefficients respectively, wherein the first displacement field is the displacement field corresponding to the picture with the lowest resolution;
determining a first resolution corresponding to the multiplied first displacement field, and searching a second displacement field according to the first resolution, wherein a second resolution corresponding to the second displacement field is the same as the first resolution;
combining the first displacement field and the second displacement field;
and taking the combined displacement field as a first displacement field, and circulating the steps until only one displacement field is left, namely the target displacement field.
Further, after the step of determining the target displacement field based on the moving picture and the fixed picture after the interference factors are removed and the initial registration network, the method further includes:
acquiring actual segmentation results respectively corresponding to the moving picture and the fixed picture;
inputting the actual segmentation result corresponding to the moving picture and the target displacement field into a space transformation network to obtain a segmentation result corresponding to the registered image;
determining label similarity according to the actual segmentation result corresponding to the fixed picture and the segmentation result corresponding to the registration image;
and training the initial registration network according to the label similarity, and storing the trained registration network for next image registration.
Further, after the step of inputting the actual segmentation result corresponding to the moving picture and the target displacement field into a spatial transformation network to obtain the segmentation result corresponding to the registered image, the method further includes:
determining contour pictures respectively corresponding to the segmentation result corresponding to the registration image and the actual segmentation result corresponding to the fixed picture through a Laplacian operator;
determining contour constraint items according to the contour pictures respectively corresponding to the segmentation result corresponding to the registration image and the actual segmentation result corresponding to the fixed picture;
and training the initial registration network according to the contour constraint item, and storing the trained registration network for next image registration.
Further, after the step of inputting the actual segmentation result corresponding to the moving picture and the target displacement field into a spatial transformation network to obtain the segmentation result corresponding to the registered image, the method further includes:
determining a global constraint term according to the segmentation result corresponding to the registration image and the actual segmentation result corresponding to the fixed picture;
and training the initial registration network according to the global constraint item, and storing the trained registration network for next image registration.
Further, after the step of determining the target displacement field based on the moving picture and the fixed picture after the interference factors are removed and the initial registration network, the method further includes:
determining a corresponding regular term according to the target displacement field;
training the initial registration network according to the regular want, and storing the trained registration network for next image registration;
after the step of inputting the moving picture and the target displacement field into a space transformation network to obtain a registration image, the method further comprises the following steps:
determining image similarity of the fixation picture and the registration image;
and training the initial registration network according to the image similarity, and storing the trained registration network for next image registration.
To achieve the above object, the present invention further provides an electronic device comprising a memory, a processor and an image registration program stored on the memory and executable on the processor, the image registration program, when executed by the processor, implementing the steps of the image registration method of any one of the above.
In order to achieve the above object, the present invention further provides a readable storage medium having stored thereon an image registration program, which when executed by a processor, implements the steps of the image registration method of any one of the above.
According to the technical scheme, a moving picture and a fixed picture are obtained, and the moving picture and the fixed picture are respectively subjected to prediction segmentation through an image segmentation network; removing interference factors in the moving picture and the fixed picture according to a prediction segmentation result; determining a target displacement field based on the moving picture, the fixed picture and the initial registration network after the interference factors are removed; inputting the moving picture and the target displacement field into a space transformation network, and generating a registration image through the space transformation network. Therefore, the moving picture and the fixed picture can be processed through the image segmentation network to obtain a prediction segmentation result, the picture is divided into a key part and other parts according to the prediction segmentation result, interference factors in the moving picture and the fixed picture are removed according to the prediction segmentation result, and the problem that other parts except the key part cause interference to network model training is solved.
Drawings
FIG. 1 is a schematic diagram of an apparatus in a hardware operating environment according to an embodiment of the present invention;
FIG. 2 is a flowchart illustrating an embodiment of an image registration method according to the present invention;
fig. 3 is a detailed flowchart of step S300 in the image registration method of the present invention;
fig. 4 is a schematic detailed flow chart of step S330 in the image registration method of the present invention.
The implementation, functional features and advantages of the objects of the present invention will be further explained with reference to the accompanying drawings.
Detailed Description
It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
The main technical scheme of the invention is as follows:
acquiring a moving picture and a fixed picture, and respectively carrying out prediction segmentation on the moving picture and the fixed picture through an image segmentation network;
removing interference factors in the moving picture and the fixed picture according to a prediction segmentation result;
determining a target displacement field based on the moving picture, the fixed picture and the initial registration network after the interference factors are removed;
inputting the moving picture and the target displacement field into a space transformation network, and generating a registration image through the space transformation network.
In the related art, the moving image and the fixed image are input in the process of training the network, and at the moment, information of other parts except for the key parts causes certain interference to the training of the network model.
In the technical scheme of the invention, a moving picture and a fixed picture are obtained, and the moving picture and the fixed picture are respectively subjected to prediction segmentation through an image segmentation network; removing interference factors in the moving picture and the fixed picture according to a prediction segmentation result; determining a target displacement field based on the moving picture, the fixed picture and the initial registration network after the interference factors are removed; inputting the moving picture and the target displacement field into a space transformation network, and generating a registration image through the space transformation network. Therefore, the moving picture and the fixed picture can be processed through the image segmentation network to obtain a prediction segmentation result, the picture is divided into a key part and other parts according to the prediction segmentation result, interference factors in the moving picture and the fixed picture are removed according to the prediction segmentation result, and the problem that other parts except the key part cause interference to network model training is solved.
As shown in fig. 1, fig. 1 is a schematic diagram of a hardware operating environment of a terminal according to an embodiment of the present invention.
As shown in fig. 1, the terminal may include: a processor 1001, such as a CPU, a network interface 1004, a user interface 1003, a memory 1005, a communication bus 1002. Wherein a communication bus 1002 is used to enable connective communication between these components. The user interface 1003 may include a Display screen (Display), an input unit such as a Keyboard (Keyboard), and the optional user interface 1003 may also include a standard wired interface, a wireless interface. The network interface 1004 may optionally include a standard wired interface, a wireless interface (e.g., a non-volatile memory), such as a disk memory. The memory 1005 may alternatively be a storage device separate from the processor 1001.
Those skilled in the art will appreciate that the configuration of the terminal shown in fig. 1 is not intended to be limiting and may include more or fewer components than those shown, or some components may be combined, or a different arrangement of components.
As shown in fig. 1, a memory 1005, which is one type of computer storage medium, may include an operating system, a network communication module, a user interface module, and an image registration program therein.
In the terminal shown in fig. 1, the network interface 1004 is mainly used for connecting to a backend server and performing data communication with the backend server; the user interface 1003 is mainly used for connecting a client (user side) and performing data communication with the client; and the processor 1001 may be configured to invoke an image registration program stored in the memory 1005 and perform the following operations:
acquiring a moving picture and a fixed picture, and respectively carrying out prediction segmentation on the moving picture and the fixed picture through an image segmentation network;
removing interference factors in the moving picture and the fixed picture according to a prediction segmentation result;
determining a target displacement field based on the moving picture, the fixed picture and the initial registration network after the interference factors are removed;
inputting the moving picture and the target displacement field into a space transformation network, and generating a registration image through the space transformation network.
Further, the processor 1001 may invoke an image registration program stored in the memory 1005, and also perform the following operations:
and multiplying the moving picture and the fixed picture by the corresponding prediction segmentation results respectively to remove interference factors in the moving picture and the fixed picture.
Further, the processor 1001 may invoke an image registration program stored in the memory 1005, and also perform the following operations:
merging the moving picture and the fixed picture after the interference factors are removed, and performing down-sampling on the merged picture to obtain merged pictures with different resolutions;
obtaining displacement fields with different resolutions according to the combined pictures with different resolutions;
and combining the displacement fields with different resolutions into a target displacement field through a displacement field cascade module.
Further, the processor 1001 may invoke an image registration program stored in the memory 1005, and also perform the following operations:
multiplying displacements of an X axis and a Y axis in a first displacement field by corresponding weight coefficients respectively, wherein the first displacement field is the displacement field corresponding to the picture with the lowest resolution;
determining a first resolution corresponding to the multiplied first displacement field, and searching a second displacement field according to the first resolution, wherein a second resolution corresponding to the second displacement field is the same as the first resolution;
combining the first displacement field and the second displacement field;
and taking the combined displacement field as a first displacement field, and circulating the steps until only one displacement field is left, namely the target displacement field.
Further, the processor 1001 may invoke an image registration program stored in the memory 1005, and also perform the following operations:
acquiring actual segmentation results respectively corresponding to the moving picture and the fixed picture;
inputting the actual segmentation result corresponding to the moving picture and the target displacement field into a space transformation network to obtain a segmentation result corresponding to the registered image;
determining label similarity according to the actual segmentation result corresponding to the fixed picture and the segmentation result corresponding to the registration image;
and training the initial registration network according to the label similarity, and storing the trained registration network for next image registration.
Further, the processor 1001 may invoke an image registration program stored in the memory 1005, and also perform the following operations:
determining contour pictures respectively corresponding to the segmentation result corresponding to the registration image and the actual segmentation result corresponding to the fixed picture through a Laplacian operator;
determining contour constraint items according to the contour pictures respectively corresponding to the segmentation result corresponding to the registration image and the actual segmentation result corresponding to the fixed picture;
and training the initial registration network according to the contour constraint item, and storing the trained registration network for next image registration.
Further, the processor 1001 may invoke an image registration program stored in the memory 1005, and also perform the following operations:
determining a global constraint term according to the segmentation result corresponding to the registration image and the actual segmentation result corresponding to the fixed picture;
and training the initial registration network according to the global constraint item, and storing the trained registration network for next image registration.
Further, the processor 1001 may invoke an image registration program stored in the memory 1005, and also perform the following operations:
determining a corresponding regular term according to the target displacement field;
training the initial registration network according to the regular term, and storing the trained registration network for next image registration;
after the step of inputting the moving picture and the target displacement field into a space transformation network to obtain a registration image, the method further comprises the following steps:
determining image similarity of the fixation picture and the registration image;
and training the initial registration network according to the image similarity, and storing the trained registration network for next image registration.
As shown in fig. 2, in an embodiment of the present invention, the image registration method includes the following steps:
step S100, obtaining a moving picture and a fixed picture, and respectively carrying out prediction segmentation on the moving picture and the fixed picture through an image segmentation network;
in this embodiment, the image segmentation is a technique and process that divides the image into several specific regions with unique properties and proposes an object of interest. The method comprises the steps that a moving picture and a fixed picture are respectively input into an image segmentation network, the image segmentation network determines key parts in the input picture according to the input picture to obtain a prediction segmentation result, and the prediction segmentation result divides the picture into the key parts and other parts. The prediction segmentation result is a picture, wherein the key part, namely the region needing to be reserved, is set to be white, and the rest parts except the key part are set to be black.
Step S200, removing interference factors in the moving picture and the fixed picture according to a prediction segmentation result;
in this embodiment, the original picture may be subjected to picture multiplication with a corresponding prediction segmentation result, where the picture multiplication may be used to obtain a key portion of the image, and for a region that needs to be retained, the value of the mask image is set to 1, which is white, and in a region that needs to be suppressed, which is a region that does not need to be retained, the value of the mask image is set to 0, which is black. The moving picture is multiplied by the corresponding prediction division result, the image of the corresponding position in the moving picture can be determined and reserved according to the reserved area marked in the prediction division result, the image of the area needing to be restrained is determined, the image of the area needing to be restrained is set to be black, and the interference of other areas to model training can be eliminated.
Step S300, determining a target displacement field based on the moving picture, the fixed picture and the initial registration network after the interference factors are removed;
in this embodiment, only images of key parts are retained in the moving picture and the fixed picture after the interference factors are removed, and other images except the images of the key parts are set to be black. Combining the moving picture and the fixed picture after the interference factors are removed, and inputting the combined pictures into an initial registration network, wherein the initial registration network obtains combined pictures with different resolutions through down-sampling operation, and obtains corresponding displacement fields with different resolutions according to the combined pictures with different resolutions. Aiming at the displacement fields with different resolutions, the registration network combines the displacement fields with different resolutions through a displacement field cascade module to form a final displacement field. The downsampling operation is to reduce the picture to obtain a low-resolution picture, and the initial registration network can be trained through a total loss function, wherein the total loss function comprises a loss function, namely, the image similarity LsimRegular term LregTag similarity LmseGlobal constraint LaeAnd a contour constraint term LcontourAnd adding the loss functions to obtain a total loss function, and training the registration network through the total loss function, wherein the smaller the total loss function is, the better the registration effect of the registration network is.
And S400, inputting the moving picture and the target displacement field into a space transformation network, and generating a registration image through the space transformation network.
In this embodiment, the displacement field contains information of the momentary motion of the pixels. And inputting the moving picture and the displacement field into a space transformation network, and correcting the moving image by the moving network according to the final displacement field to obtain a final registration image.
In summary, in this embodiment, a moving picture and a fixed picture are obtained, and the moving picture and the fixed picture are respectively subjected to prediction segmentation through an image segmentation network; removing interference factors in the moving picture and the fixed picture according to a prediction segmentation result; determining a target displacement field based on the moving picture, the fixed picture and the initial registration network after the interference factors are removed; inputting the moving picture and the target displacement field into a space transformation network, and generating a registration image through the space transformation network. Therefore, the moving picture and the fixed picture can be processed through the image segmentation network to obtain a prediction segmentation result, the picture is divided into a key part and other parts according to the prediction segmentation result, interference factors in the moving picture and the fixed picture are removed according to the prediction segmentation result, and the problem that other parts except the key part cause interference to network model training is solved.
In an embodiment of the present invention, the step S200 includes:
and multiplying the moving picture and the fixed picture by the corresponding prediction segmentation results respectively to remove interference factors in the moving picture and the fixed picture.
In this embodiment, the original picture may be subjected to picture multiplication with a corresponding prediction segmentation result, where the picture multiplication may be used to obtain a key portion of the image, and for a region that needs to be retained, the value of the mask image is set to 1, which is white, and in a region that needs to be suppressed, which is a region that does not need to be retained, the value of the mask image is set to 0, which is black. The moving picture and the fixed picture are multiplied by the corresponding prediction segmentation results respectively, the image of the corresponding position in the moving picture can be determined and retained according to the retention area marked in the prediction segmentation results, the image of the area needing to be restrained is determined, the image of the area needing to be restrained is set to be black, and the interference of other areas on model training can be eliminated. Therefore, the original picture is multiplied by the corresponding prediction segmentation result, the key parts in the picture are reserved, and the rest parts in the picture are covered by black, so that the interference factors are removed, and the problem of interference caused by other parts to network model training is solved.
As shown in fig. 3, in an embodiment of the present invention, the step S300 includes:
step S310, merging the moving picture and the fixed picture after the interference factors are removed, and performing down-sampling on the merged picture to obtain merged pictures with different resolutions;
step S320, obtaining displacement fields with different resolutions according to the combined pictures with different resolutions;
and step S330, combining the displacement fields with different resolutions into a target displacement field through a displacement field cascade module.
In this embodiment, because the existing registration network has good performance in image registration with small distortion, but has a deficiency in image registration with large distortion, the embodiment connects a moving picture and a fixed picture after removing interference factors, merges the moving picture and the fixed picture into a merged picture with two channels at a channel, and inputs the merged picture with two channels into the registration network, and the registration network performs downsampling on the merged picture, that is, obtains pictures with different resolutions of the same picture by reducing the picture. The images with different resolutions enable layers with different depths of the registration network to have more image information, and the expression capability of the network is deepened. Corresponding displacement fields can be obtained through pictures with different resolutions, and the resolutions of the displacement fields are different. The registration network adjusts the resolution of the low-resolution displacement field to be the same as that of the high-resolution displacement field by designing a displacement field cascade module, and combines the two displacement fields into one displacement field. And repeating the steps until only one displacement field is left, and outputting the displacement field as a final displacement field. Therefore, the displacement fields with different resolutions are obtained by obtaining the pictures with different resolutions, and the displacement field cascade module is designed, so that the displacement fields with different resolutions can be combined, and the capability of the network model in large-deformation image registration is improved.
As shown in fig. 4, in an embodiment of the present invention, the step S330 includes:
step S331, respectively multiplying displacements of an X axis and a Y axis in a first displacement field by corresponding weight coefficients, wherein the first displacement field is the displacement field corresponding to the picture with the lowest resolution;
step S332, determining a first resolution corresponding to the multiplied first displacement field, and searching a second displacement field according to the first resolution, wherein the second resolution corresponding to the second displacement field is the same as the first resolution;
step S333, merging the first displacement field and the second displacement field;
and step 334, taking the combined displacement field as a first displacement field, and circulating the steps until only one displacement field is left, namely the target displacement field.
In this embodiment, since the displacement fields with different resolutions cannot be merged, it is necessary to make the resolutions of the displacement fields the same by using the weight coefficient map of the X axis and the weight coefficient map of the Y axis in the displacement field cascade module, so that the displacement fields can be merged. The displacement field cascade module respectively learns the X axis and the Y axis of the displacement field, and respectively learns a weight coefficient map on the X axis and the Y axis, namely, the X axis and the Y axis are learned separately, so as to obtain the X axis weight coefficient map and the Y axis weight coefficient map. The method comprises the steps of obtaining a displacement field corresponding to a picture with the lowest resolution as a first displacement field, multiplying displacements of an X axis and a Y axis in the first displacement field by corresponding weight coefficients respectively, wherein the resolutions of the first displacement field before and after multiplication are different, and searching a displacement field with the same resolution as that of the multiplied first displacement field as a second displacement field, namely, the first resolution corresponding to the multiplied first displacement field is the same as that of the second displacement field. Therefore, the displacement field cascade module analyzes the displacement field from the X axis and the Y axis respectively, can obtain the change of the displacement field on the X axis and the Y axis respectively more clearly, and respectively and independently learn the X axis and the Y axis to obtain the X axis weight coefficient diagram and the Y axis weight coefficient diagram respectively, thereby ensuring that the change of the X axis and the Y axis of the displacement field of the resolution conversion according to the displacement field cascade module is more accurate.
In an embodiment of the present invention, after the step S300, the method further includes:
acquiring actual segmentation results respectively corresponding to the moving picture and the fixed picture;
inputting the actual segmentation result corresponding to the moving picture and the target displacement field into a space transformation network to obtain a segmentation result corresponding to the registered image;
determining label similarity according to the actual segmentation result corresponding to the fixed picture and the segmentation result corresponding to the registration image;
and training the initial registration network according to the label similarity, and storing the trained registration network for next image registration.
In this embodiment, the tag similarity LmseIn order to calculate the mean square error between each pixel of the actual segmentation result corresponding to the fixed picture and the segmentation result corresponding to the registered image, the higher the tag similarity, generally speaking, the higher the calculated tag similarity, the lower the similarity between the two images. Acquiring an actual segmentation result corresponding to the moving picture and the fixed picture in a public data set, inputting the actual segmentation result corresponding to the moving picture and the target displacement field into a space transformation network to obtain a segmentation result corresponding to the registration image, and calculating to obtain the tag similarity, wherein a calculation formula of the tag similarity is as follows:
Lmse＝mse(IF,seg,Iwarp,seg)
wherein, IFRepresenting a fixed image, IwarpRepresenting the registered images. According to the label similarity and the other four loss functions, namely image similarity, a regular term, a global constraint term and a contour constraint term, the registration is carried out by the total loss function obtained by addingAnd training the network, and storing the trained registration network for next image registration. Therefore, the segmentation result corresponding to the registration image is obtained, the label similarity is obtained by calculating the actual segmentation result of the fixed picture and the segmentation result corresponding to the registration image, the similarity degree of the two pictures is judged through the label similarity, and the registration network model is trained according to the total loss function obtained by adding the label similarity and the other four loss functions, so that the registration network outputs a more accurate result.
In an embodiment of the present invention, after the step of inputting the actual segmentation result corresponding to the moving picture and the target displacement field into a spatial transformation network to obtain the segmentation result corresponding to the registered image, the method further includes:
determining contour pictures respectively corresponding to the segmentation result corresponding to the registration image and the actual segmentation result corresponding to the fixed picture through a Laplacian operator;
determining contour constraint items according to the contour pictures respectively corresponding to the segmentation result corresponding to the registration image and the actual segmentation result corresponding to the fixed picture;
and training the initial registration network according to the contour constraint item, and storing the trained registration network for next image registration.
In this embodiment, since in the registration of the weakly supervised medical images, the similarity between the transformed mobile tag and the fixed tag is measured by using the cross entropy as a loss function, and the loss function cannot provide a good global constraint condition, the laplacian operator is added in the process of training the registration network in this embodiment. And in the process of training the registration network, acquiring an actual segmentation result corresponding to the moving picture and the fixed picture, and inputting the actual segmentation result corresponding to the moving picture and the target displacement field into a space transformation network to acquire a segmentation result corresponding to the registration network. Processing the actual segmentation result corresponding to the fixed picture and the segmentation result corresponding to the registration image through a Laplacian operator to obtain the profile pictures of the corresponding key parts, and performing cross entropy on the two profilesProcessing the picture to determine a contour constraint item LcontourAnd calculating the formula of the contour constraint term as follows:
Lcontour＝cross_entropy(Laplace(yF,seg),Laplace(ywarp,seg))
and training the initial registration network through the total loss function obtained by adding the contour constraint item and the other four loss functions, namely the image similarity, the regular item, the label similarity and the global constraint item, so that the contour fitting of the two images registered by the registration network is better, and the trained registration network is stored for the next image registration. In this way, the contour constraint item is determined by adding the laplacian operator, then the contour constraint item and the other four loss functions are added to obtain a total loss function, and the registration network is trained according to the total loss function, so that the effect of better fitting the contour of the image subjected to registration is achieved.
In an embodiment of the present invention, after the step of inputting the actual segmentation result corresponding to the moving picture and the target displacement field into a spatial transformation network to obtain the segmentation result corresponding to the registered image, the method further includes:
determining a global constraint term according to the segmentation result corresponding to the registration image and the actual segmentation result corresponding to the fixed picture;
and training the initial registration network according to the global constraint item, and storing the trained registration network for next image registration.
In this embodiment, the segmentation result of the registration image and the actual segmentation result corresponding to the fixed picture are respectively processed by an encoder module, and based on the result processed by the encoder module and the cross entropy, a global constraint term L is finally obtainedaeThe formula for calculating the global constraint term is as follows:
Lae＝cross_entrop(encoder(IF,seg),encoder(Iwarp.seg))
wherein, IFRepresenting a fixed image, IwarpRepresenting the registered images, the global constraint term being a loss function of the training registration networkAnd one, training the registration network according to the total loss function obtained by adding the global constraint term and the other four loss functions, namely image similarity, regular term, label similarity and contour constraint, and storing the trained registration network for next image registration.
In an embodiment of the present invention, after the step S300, the method further includes:
determining a corresponding regular term according to the target displacement field;
training the initial registration network according to the regular term, and storing the trained registration network for next image registration;
after the step S400, the method further includes:
determining image similarity of the fixation picture and the registration image;
and training the initial registration network according to the image similarity, and storing the trained registration network for next image registration.
In this embodiment, a gradient is obtained for a displacement field output by a registration network, and a regular term is determined according to a result of obtaining the gradient, where a calculation formula of the regular term is
Wherein the content of the first and second substances,
for the gradient of the image, DFV is the displacement field used for image transformation, the regularization term being one of the loss functions for training the registration network. And determining the regular item to constrain the network by utilizing the gradient loss of the image so as to prevent overfitting and improve the generalization capability of the registration network, training the registration network according to the regular item, and storing the trained registration network for next image registration. Because the existing image registration technology based on deep learning only uses the strength-based loss function, namely the NCC function, in the process of training the network, the trained network is very sensitive to noise and gray level change of images, and is not beneficial to network training and model generalization. The true bookEmbodiments propose adding a loss function containing image structure information, i.e. an SSIM function, on the basis of using the NCC function. Image similarity L between the fixed picture and the registered imagesimThen, using an NCC function and an SSIM function, training the registration network according to the image similarity, and storing the trained registration network for next image registration, wherein the calculation formula of the image similarity is as follows:
Lsim＝NCC(IF,Iwarp)+SSIM(IF,Iwarp)
wherein, IFRepresenting a fixed image, IwarpRepresenting the registered images, the NCC is calculated as:
wherein p represents the pixel coordinates of the image, the NCC function is a normalized cross-correlation matching algorithm, and the degree of matching is determined by calculating the cross-correlation value of the fixed picture and the registered image. The SSIM refers to structural similarity, namely an index for measuring the similarity of two images, and by introducing the structural similarity, the images can be ensured to retain more structural information while retaining the gray information. The regularization term and the image similarity and the remaining three loss functions, namely the label similarity, the global constraint term and the contour constraint term, are added to obtain a total loss function, which can be used to train the registration network.
To achieve the above object, the present invention further provides an electronic device comprising a memory, a processor and an image registration program stored on the memory and executable on the processor, the image registration program, when executed by the processor, implementing the steps of the image registration method of any one of the above.
In order to achieve the above object, the present invention further provides a readable storage medium having stored thereon an image registration program, which when executed by a processor, implements the steps of the image registration method of any one of the above.
The above-mentioned serial numbers of the embodiments of the present invention are merely for description and do not represent the merits of the embodiments.
Through the above description of the embodiments, those skilled in the art will clearly understand that the method of the above embodiments can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware, but in many cases, the former is a better implementation manner. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium (e.g., ROM/RAM, magnetic disk, optical disk) as described above and includes instructions for enabling a terminal device (e.g., a mobile phone, a computer, a server, or a network device) to execute the method according to the embodiments of the present invention.
The above description is only a preferred embodiment of the present invention, and not intended to limit the scope of the present invention, and all modifications of equivalent structures and equivalent processes, which are made by using the contents of the present specification and the accompanying drawings, or directly or indirectly applied to other related technical fields, are included in the scope of the present invention.
Claims (10)
1. An image registration method, characterized in that it comprises the steps of:
acquiring a moving picture and a fixed picture, and respectively carrying out prediction segmentation on the moving picture and the fixed picture through an image segmentation network;
removing interference factors in the moving picture and the fixed picture according to a prediction segmentation result;
determining a target displacement field based on the moving picture, the fixed picture and the initial registration network after the interference factors are removed;
inputting the moving picture and the target displacement field into a space transformation network, and generating a registration image through the space transformation network.
2. The image registration method according to claim 1, wherein the step of removing the interference factors in the moving picture and the fixed picture according to the prediction segmentation result comprises:
and multiplying the moving picture and the fixed picture by the corresponding prediction segmentation results respectively to remove interference factors in the moving picture and the fixed picture.
3. The image registration method of claim 1, wherein the step of determining a target displacement field based on the moving picture and the fixed picture after removing the interference factor and an initial registration network comprises:
merging the moving picture and the fixed picture after the interference factors are removed, and performing down-sampling on the merged picture to obtain merged pictures with different resolutions;
obtaining displacement fields with different resolutions according to the combined pictures with different resolutions;
and combining the displacement fields with different resolutions into a target displacement field through a displacement field cascade module.
4. The image registration method of claim 3, wherein the step of combining the displacement fields of different resolutions into the target displacement field by a displacement field cascade module comprises:
multiplying displacements of an X axis and a Y axis in a first displacement field by corresponding weight coefficients respectively, wherein the first displacement field is the displacement field corresponding to the picture with the lowest resolution;
determining a first resolution corresponding to the multiplied first displacement field, and searching a second displacement field according to the first resolution, wherein a second resolution corresponding to the second displacement field is the same as the first resolution;
combining the first displacement field and the second displacement field;
and taking the combined displacement field as a first displacement field, and circulating the steps until only one displacement field is left, namely the target displacement field.
5. The image registration method of claim 1, wherein after the step of determining a target displacement field based on the moving picture and the fixed picture after removing the interference factor and an initial registration network, further comprising:
acquiring actual segmentation results respectively corresponding to the moving picture and the fixed picture;
inputting the actual segmentation result corresponding to the moving picture and the target displacement field into a space transformation network to obtain a segmentation result corresponding to the registered image;
determining label similarity according to the actual segmentation result corresponding to the fixed picture and the segmentation result corresponding to the registration image;
and training the initial registration network according to the label similarity, and storing the trained registration network for next image registration.
6. The image registration method according to claim 5, wherein after the step of inputting the actual segmentation result corresponding to the moving picture and the target displacement field into a spatial transformation network to obtain the segmentation result corresponding to the registered image, the method further comprises:
determining contour pictures respectively corresponding to the segmentation result corresponding to the registration image and the actual segmentation result corresponding to the fixed picture through a Laplacian operator;
determining contour constraint items according to the contour pictures respectively corresponding to the segmentation result corresponding to the registration image and the actual segmentation result corresponding to the fixed picture;
and training the initial registration network according to the contour constraint item, and storing the trained registration network for next image registration.
7. The image registration method according to claim 5, wherein after the step of inputting the actual segmentation result corresponding to the moving picture and the target displacement field into a spatial transformation network to obtain the segmentation result corresponding to the registered image, the method further comprises:
determining a global constraint term according to the segmentation result corresponding to the registration image and the actual segmentation result corresponding to the fixed picture;
and training the initial registration network according to the global constraint item, and storing the trained registration network for next image registration.
8. The image registration method of claim 1, wherein after the step of determining a target displacement field based on the moving picture and the fixed picture after removing the interference factor and an initial registration network, further comprising:
determining a corresponding regular term according to the target displacement field;
training the initial registration network according to the regular term, and storing the trained registration network for next image registration;
after the step of inputting the moving picture and the target displacement field into a space transformation network to obtain a registration image, the method further comprises the following steps:
determining image similarity of the fixation picture and the registration image;
and training the initial registration network according to the image similarity, and storing the trained registration network for next image registration.
9. An electronic device, characterized in that the electronic device comprises a memory, a processor and an image registration program stored on the memory and executable on the processor, which image registration program, when executed by the processor, implements the steps of the image registration method according to any one of claims 1 to 8.
10. A readable storage medium, characterized in that the readable storage medium has stored thereon an image registration program which, when executed by a processor, implements the steps of the image registration method according to any one of claims 1 to 8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110793797.5A CN113643332B (en) | 2021-07-13 | 2021-07-13 | Image registration method, electronic device and readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110793797.5A CN113643332B (en) | 2021-07-13 | 2021-07-13 | Image registration method, electronic device and readable storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113643332A true CN113643332A (en) | 2021-11-12 |
| CN113643332B CN113643332B (en) | 2023-12-19 |
Family
ID=78417306
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110793797.5A Active CN113643332B (en) | 2021-07-13 | 2021-07-13 | Image registration method, electronic device and readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113643332B (en) |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20180192881A1 (en) * | 2017-01-06 | 2018-07-12 | Canon Kabushiki Kaisha | Image processing apparatus and image processing method |
| CN111862175A (en) * | 2020-07-13 | 2020-10-30 | 清华大学深圳国际研究生院 | Cross-modal medical image registration method and device based on cyclic canonical training |
| US20210058617A1 (en) * | 2019-08-23 | 2021-02-25 | Qualcomm Incorporated | Geometric partition mode with harmonized motion field storage and motion compensation |
| CN112767463A (en) * | 2021-01-12 | 2021-05-07 | 深圳大学 | Countermeasure registration method and device, computer equipment and storage medium |
| CN112907439A (en) * | 2021-03-26 | 2021-06-04 | 中国科学院深圳先进技术研究院 | Supine position and prone position mammary gland image registration method based on deep learning |
| CN113034453A (en) * | 2021-03-16 | 2021-06-25 | 深圳先进技术研究院 | Mammary gland image registration method based on deep learning |
-
2021
- 2021-07-13 CN CN202110793797.5A patent/CN113643332B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20180192881A1 (en) * | 2017-01-06 | 2018-07-12 | Canon Kabushiki Kaisha | Image processing apparatus and image processing method |
| US20210058617A1 (en) * | 2019-08-23 | 2021-02-25 | Qualcomm Incorporated | Geometric partition mode with harmonized motion field storage and motion compensation |
| CN111862175A (en) * | 2020-07-13 | 2020-10-30 | 清华大学深圳国际研究生院 | Cross-modal medical image registration method and device based on cyclic canonical training |
| CN112767463A (en) * | 2021-01-12 | 2021-05-07 | 深圳大学 | Countermeasure registration method and device, computer equipment and storage medium |
| CN113034453A (en) * | 2021-03-16 | 2021-06-25 | 深圳先进技术研究院 | Mammary gland image registration method based on deep learning |
| CN112907439A (en) * | 2021-03-26 | 2021-06-04 | 中国科学院深圳先进技术研究院 | Supine position and prone position mammary gland image registration method based on deep learning |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113643332B (en) | 2023-12-19 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109508681B (en) | Method and device for generating human body key point detection model | |
| EP3716198A1 (en) | Image reconstruction method and device | |
| WO2018153322A1 (en) | Key point detection method, neural network training method, apparatus and electronic device | |
| CN107818554A (en) | Message processing device and information processing method | |
| CN114972756A (en) | Semantic segmentation method and device for medical image | |
| US20110249001A1 (en) | Matrix generation apparatus, method, and program, and information processing apparatus | |
| KR20180122548A (en) | Method and apparaturs for processing image | |
| CN113066017A (en) | Image enhancement method, model training method and equipment | |
| CN113066018A (en) | Image enhancement method and related device | |
| CN111861888A (en) | Image processing method, image processing device, electronic equipment and storage medium | |
| CN116934591A (en) | Image stitching method, device and equipment for multi-scale feature extraction and storage medium | |
| CN117011137A (en) | Image stitching method, device and equipment based on RGB similarity feature matching | |
| CN115760807B (en) | Retina fundus image registration method and system | |
| CN116704029A (en) | Dense object semantic map construction method and device, storage medium and electronic equipment | |
| CN113643332B (en) | Image registration method, electronic device and readable storage medium | |
| CN116385281A (en) | Remote sensing image denoising method based on real noise model and generated countermeasure network | |
| CN114998630A (en) | Ground-to-air image registration method from coarse to fine | |
| CN114549992A (en) | Cross-resolution building image extraction method and device | |
| CN116883770A (en) | Training method and device of depth estimation model, electronic equipment and storage medium | |
| CN114155406A (en) | Pose estimation method based on region-level feature fusion | |
| CN111435448A (en) | Image salient object detection method, device, equipment and medium | |
| CN115082295B (en) | Image editing method and device based on self-attention mechanism | |
| CN116433674B (en) | Semiconductor silicon wafer detection method, device, computer equipment and medium | |
| CN115272250B (en) | Method, apparatus, computer device and storage medium for determining focus position | |
| CN111539988B (en) | Visual odometer implementation method and device and electronic equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |