CN113822792A - Image registration method, device, equipment and storage medium - Google Patents

Image registration method, device, equipment and storage medium Download PDF

Info

Publication number
CN113822792A
CN113822792A CN202110661355.5A CN202110661355A CN113822792A CN 113822792 A CN113822792 A CN 113822792A CN 202110661355 A CN202110661355 A CN 202110661355A CN 113822792 A CN113822792 A CN 113822792A
Authority
CN
China
Prior art keywords
image
registration
sample
registered
network
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202110661355.5A
Other languages
Chinese (zh)
Inventor
徐哲
卢东焕
魏东
马锴
郑冶枫
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Tencent Technology Shenzhen Co Ltd
Original Assignee
Tencent Technology Shenzhen Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Tencent Technology Shenzhen Co Ltd filed Critical Tencent Technology Shenzhen Co Ltd
Priority to CN202110661355.5A priority Critical patent/CN113822792A/en
Publication of CN113822792A publication Critical patent/CN113822792A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformations in the plane of the image
    • G06T3/14—Transformations for image registration, e.g. adjusting or mapping for alignment of images
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
    • G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/22—Matching criteria, e.g. proximity measures

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Artificial Intelligence (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Evolutionary Biology (AREA)
  • Evolutionary Computation (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • General Engineering & Computer Science (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Image Analysis (AREA)

Abstract

The application discloses an image registration method, an image registration device, image registration equipment and a storage medium, and relates to the field of artificial intelligence. The method comprises the following steps: acquiring a sample image pair, wherein the sample image pair comprises a sample image to be registered and a sample target image; inputting the sample image into a first registration network to obtain a first deformation field between the sample image to be registered and the sample target image, and performing deformation processing on the sample image to be registered through the first deformation field to obtain a first registration image; inputting the sample image into a second registration network to obtain a second deformation field between the sample image to be registered and the sample target image, and performing deformation processing on the sample image to be registered through the second deformation field to obtain a second registration image, wherein a first network parameter of the first registration network and a second network parameter of the second registration network are network parameters at different training times; a first registration network is trained based on the first deformation field, the first registration image, the second registration image, and the sample target image.

Description

Image registration method, device, equipment and storage medium
Technical Field
The embodiment of the application relates to the field of artificial intelligence, in particular to an image registration method, an image registration device, image registration equipment and a storage medium.
Background
Image registration refers to the process of spatially transforming one or more images into spatial alignment with a reference image.
The image registration process is a process of finding the optimal spatial transformation, namely the optimal deformation field, for each pair of images to be registered, the solution space of the deformation field is not unique, so that the spatial regularization term constraint deformation field is added to the solution space of the constraint deformation field in the training stage of the image registration network based on deep learning.
However, in the process of training the image registration network, the deformation fields have large differences at different training times, and in the related art, only the deformation fields in the space are constrained, so that the training effect of the image registration network is poor, and the accuracy of image registration based on the image registration network is further influenced.
Disclosure of Invention
The embodiment of the application provides an image registration method, an image registration device, image registration equipment and a storage medium, which can improve the performance of predicting a deformation field by an image registration network and further improve the accuracy of image registration. The technical scheme is as follows:
in one aspect, an embodiment of the present application provides an image registration method, where the method includes:
obtaining a sample image pair, wherein the sample image pair comprises a sample image to be registered and a sample target image;
inputting the sample image into a first registration network to obtain a first deformation field between the sample image to be registered and the sample target image, and performing deformation processing on the sample image to be registered through the first deformation field to obtain a first registration image;
inputting the sample image into a second registration network to obtain a second deformation field between the sample image to be registered and the sample target image, and performing deformation processing on the sample image to be registered through the second deformation field to obtain a second registration image, wherein a first network parameter of the first registration network and a second network parameter of the second registration network are network parameters under different training times;
training the first registration network based on the first deformation field, the first registration image, the second registration image, and the sample target image.
In another aspect, an embodiment of the present application provides an image registration apparatus, including:
the system comprises a sample image acquisition module, a sample image acquisition module and a sample image matching module, wherein the sample image acquisition module is used for acquiring a sample image pair which comprises a sample image to be registered and a sample target image;
the first registration module is used for inputting the sample image into a first registration network to obtain a first deformation field between the sample image to be registered and the sample target image, and performing deformation processing on the sample image to be registered through the first deformation field to obtain a first registration image;
the second registration module is configured to input the sample image into a second registration network to obtain a second deformation field between the sample image to be registered and the sample target image, and perform deformation processing on the sample image to be registered through the second deformation field to obtain a second registration image, where a first network parameter of the first registration network and a second network parameter of the second registration network are network parameters at different training times;
a training module to train the first registration network based on the first deformation field, the first registration image, the second registration image, and the sample target image.
In another aspect, embodiments of the present application provide a computer device including a processor and a memory, where at least one instruction, at least one program, a set of codes, or a set of instructions is stored in the memory, and the at least one instruction, the at least one program, the set of codes, or the set of instructions is loaded and executed by the processor to implement the image registration method according to the above aspect.
In another aspect, there is provided a computer readable storage medium having stored therein at least one instruction, at least one program, set of codes, or set of instructions, which is loaded and executed by a processor to implement the image registration method as described in the above aspect.
In another aspect, embodiments of the present application provide a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions to cause the computer device to perform the image registration method provided by the above aspect.
The beneficial effects brought by the technical scheme provided by the embodiment of the application at least comprise:
the image registration of the image to be registered and the target image is respectively carried out through the first registration network and the second registration network, deformation fields corresponding to network parameters under different training time are obtained, finally, different registration images are obtained after deformation processing is carried out on the image to be registered based on the different deformation fields, then the first registration network can be trained through the different registration images and the target image, the network parameters obtained by the first registration network under different training time tend to be stable, even if the deformation fields predicted by the first registration network under different training time tend to be consistent, time constraint is added in the training process, the training effect of the first registration network is favorably improved, and the accuracy of the deformation field predicted by the first registration network is further improved.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the description of the embodiments are briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.
Fig. 1 is a schematic diagram illustrating an image registration method provided in an embodiment of the present application;
FIG. 2 illustrates a schematic diagram of an implementation environment provided by an exemplary embodiment of the present application;
FIG. 3 illustrates a flow chart of an image registration method provided by an exemplary embodiment of the present application;
FIG. 4 illustrates a flow chart of an image registration method provided by another exemplary embodiment of the present application;
FIG. 5 shows a flow chart of an image registration method provided by another exemplary embodiment of the present application;
FIG. 6 is an implementation schematic diagram of a first registration network training process shown in an exemplary embodiment;
FIG. 7 shows a flow chart of an image registration method provided by another exemplary embodiment of the present application;
FIG. 8 is a schematic diagram illustrating an implementation of an image registration process using a first registration network in accordance with an illustrative embodiment;
fig. 9 is a block diagram of an image registration apparatus according to an exemplary embodiment of the present application;
fig. 10 shows a schematic structural diagram of a computer device provided in an exemplary embodiment of the present application.
Detailed Description
To make the objects, technical solutions and advantages of the present application more clear, embodiments of the present application will be described in further detail below with reference to the accompanying drawings.
Artificial Intelligence (AI) is a theory, method, technique and application system that uses a digital computer or a machine controlled by a digital computer to simulate, extend and expand human Intelligence, perceive the environment, acquire knowledge and use the knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technique of computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can react in a manner similar to human intelligence. Artificial intelligence is the research of the design principle and the realization method of various intelligent machines, so that the machines have the functions of perception, reasoning and decision making.
The artificial intelligence technology is a comprehensive subject and relates to the field of extensive technology, namely the technology of a hardware level and the technology of a software level. The artificial intelligence infrastructure generally includes technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technologies, operation/interaction systems, mechatronics, and the like. The artificial intelligence software technology mainly comprises a computer vision technology, a voice processing technology, a natural language processing technology, machine learning/deep learning and the like.
Computer Vision technology (CV) is a science for researching how to make a machine "see", and further refers to that a camera and a Computer are used to replace human eyes to perform machine Vision such as identification, tracking and measurement on a target, and further image processing is performed, so that the Computer processing becomes an image more suitable for human eyes to observe or is transmitted to an instrument to detect. As a scientific discipline, computer vision research-related theories and techniques attempt to build artificial intelligence systems that can capture information from images or multidimensional data. The computer vision technology generally includes technologies such as image processing, image recognition, image segmentation, image semantic understanding, image retrieval, video processing, video semantic understanding, video content/behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, synchronous positioning, map construction and the like, and also includes common biometric technologies such as face recognition, fingerprint recognition and the like. The image registration method, namely the application of the computer vision technology in the field of image processing, can increase training time constraint and improve the training effect of a registration network in the process of training the registration network based on a sample image, so as to improve the performance of a predicted deformation field of the registration network.
In the related art, in the training phase of the unsupervised learning-based image registration network, the image registration network is trained through the following loss function:
Figure BDA0003115483170000041
wherein, ImIs a moving image of the image to be moved,
Figure BDA0003115483170000044
is a deformed moving image, IfIs a fixed image of the image to be displayed,
Figure BDA0003115483170000042
the quantized fixed image is spatially different from the morphed moving image. However, in the image registration process, a plurality of solutions of deformation fields exist, namely the solution space of the deformation fields is not unique, so that the loss of spatial regularization is applied
Figure BDA0003115483170000043
The solution space is constrained and λ is the regularization strength.
Therefore, in the related art, only the constraint is applied to the deformation field in space, and in the training process, the deformation fields predicted by the image registration network in different training times (i.e., in the loop iteration process) also have great difference, but the difference of the deformation fields in time is not considered in the related art, so that the second registration network is introduced in the embodiment of the application, the solution space of the deformation field is constrained in time, the training effect of the image registration network is optimized, and the performance of the image registration network is improved.
As shown in fig. 1, a first registration network 102 and a second registration network 103 are provided, a sample image pair 101 is respectively input into the first registration network 102 and the second registration network 103 to obtain a first deformation field 104 and a second deformation field 105, and then a deformation process is performed on an image to be registered 106 based on the first deformation field 104 and the second deformation field 105 to obtain a first registration image 107 and a second registration image 108. Since the network parameters of the first registration network 102 and the second registration network 103 are network parameters at different training times, training the first registration network 102 based on the first registration image 107 and the second registration image 108 can constrain a solution space of the deformation field predicted by the first registration network 102 in time, thereby improving the performance of the first registration network 102, and improving the accuracy of image registration when the trained first registration network 102 is used for image registration.
The image registration method provided by the embodiment of the application can be used for a training process of the image registration network, and the image registration network after training can be used in any image registration scene.
When applied to a medical scene, different devices can be used to acquire images containing accurate anatomical information for the same organ of a patient, such as Computed Tomography (CT), Positron Emission Tomography (PET), and the like; or medical images of different time periods are collected, the collected images are subjected to image registration, information in each image is fused, and therefore medical personnel can observe the change condition of the focus and the organ more accurately from all angles, and medical diagnosis, operation plan making and radiotherapy plan process making are facilitated.
When the method is applied to a target tracking scene, image registration can be performed on target images of the same target acquired at different angles and different times, information in the images is further fused, a target action track is obtained, and target tracking is facilitated.
Of course, besides the above application scenarios, the image registration method provided in the embodiment of the present application may also be applied to other image registration scenarios, and the embodiment of the present application is not limited to a specific application scenario.
FIG. 2 illustrates a schematic diagram of an implementation environment provided by an exemplary embodiment of the present application. Included in the implementation environment are a computer device 210 and a server 220. The computer device 210 and the server 220 perform data communication through a communication network, optionally, the communication network may be a wired network or a wireless network, and the communication network may be at least one of a local area network, a metropolitan area network, and a wide area network.
The computer device 210 is an electronic device with image registration requirements, and the electronic device may be a smart phone, a tablet computer, a personal computer, or the like, and the embodiment is not limited thereto. In fig. 2, a computer used by the medical staff as the computer device 210 is described as an example.
In some embodiments, an application having image registration functionality is installed in the computer device 210. When a target image pair (e.g., medical images scanned at different times, medical images scanned by different devices) needs to be registered, a user inputs the target image pair into an application program, so that the target image pair is uploaded to the server 220, the server 220 performs image registration, and a registration result is fed back.
The server 220 may be an independent physical server, a server cluster or a distributed system formed by a plurality of physical servers, or a cloud server providing basic cloud computing services such as a cloud service, a cloud database, cloud computing, a cloud function, cloud storage, a Network service, cloud communication, a middleware service, a domain name service, a security service, a Content Delivery Network (CDN), a big data and artificial intelligence platform, and the like.
In some embodiments, server 220 is used to provide image registration services for applications installed in computer device 210. Optionally, an image registration network 221 is disposed in the server 220. In a possible implementation manner, after receiving the target image pair 211 sent by the computer device 210, the server 220 performs image registration by using the image registration network 221 to obtain a deformation field 222, performs deformation processing on the image to be registered by using the deformation field 222 to obtain a registration image 223, and returns the registration image 223 to the computer device 210, so that the computer device 210 displays the image registration result.
Of course, in other possible embodiments, the image registration network may also be deployed on the side of the computer device 210, and the computer device 210 locally implements image registration without the aid of the server 220, which is not limited in this embodiment. And the image registration network can be trained on the server side, and can also be trained on the computer equipment side for deployment of the image registration network. For convenience of description, the following embodiments are described as examples in which the image registration method is executed by a computer device.
Referring to fig. 3, a flowchart of an image registration method provided by an exemplary embodiment of the present application is shown. The embodiment is described by taking the method as an example for a computer device, and the method comprises the following steps.
Step 301, a sample image pair is obtained, and the sample image pair includes a sample to-be-registered image and a sample target image.
A sample image pair refers to an image pair acquired under different conditions for the same object. Wherein the different conditions may include at least one of different times, different angles, or different acquisition devices. For example, the sample image pair includes medical images of the lungs acquired during the expiration and inspiration phases of the same patient.
Optionally, any image in the sample image pair may be used as the sample target image. For example, the sample image pair includes an image a and an image B, and when the image a is used as the sample target image, the image B may be spatially transformed so as to be spatially aligned with the image a; when image B is the sample target image, image a may be spatially transformed to be spatially aligned with image a.
Step 302, inputting the sample image into a first registration network to obtain a first deformation field between the sample image to be registered and the sample target image, and performing deformation processing on the sample image to be registered through the first deformation field to obtain a first registration image.
Alternatively, the first registration network may employ a U-Net network that includes an encoder for extracting advanced semantic features and an encoder for restoring the original dimensions and generating a prediction by the user. In the embodiment, a three-dimensional registration prediction network is constructed by adopting a U-Net network, and the three-dimensional image is registered. In addition, the first registration network may also be other structural networks, which is not limited in this embodiment.
And inputting the sample image into a first registration network to obtain a first deformation field, wherein the sample image to be registered is subjected to spatial transformation based on the first deformation field, so that the sample image to be registered and the sample target image are spatially aligned, and the purpose of fusing information in the two images is further achieved.
Optionally, in the process of performing deformation processing on the sample image to be registered, a Spatial Transformer Networks (STNs) may be used to deform the sample image to be registered. The first deformation field and the sample image to be registered are input into the STN, and the STN deforms the sample image to be registered by using the first deformation field to obtain a first registration image.
Step 303, inputting the sample image into a second registration network to obtain a second deformation field between the sample image to be registered and the sample target image, and performing deformation processing on the sample image to be registered through the second deformation field to obtain a second registration image, where a first network parameter of the first registration network and a second network parameter of the second registration network are network parameters at different training times.
In the training process of the first registration network, the deformation fields predicted at different training times have larger differences, and therefore, in order to temporally constrain a solution space of the deformation field, in the embodiment of the present application, the second registration network is set to predict the inter-deformation field by the sample image, that is, the second deformation field is predicted. Since the network parameters of the first registration network and the second registration network are network parameters at different training times, the deformation fields at different training times can be predicted.
After the second deformation field is obtained through prediction, the STN network can be used for carrying out deformation processing on the image to be registered of the sample to obtain a second registration image, and then the first registration image and the second registration image under different training time can be obtained, so that the deformation field can be constrained in time when the first registration network is trained on the basis of the first registration image and the second registration image.
Optionally, the network structure of the first registration network is the same as that of the second registration network, and it is only the adopted network parameters that have temporal differences.
Step 304, training a first registration network based on the first deformation field, the first registration image, the second registration image, and the sample target image.
After the first deformation field, the first registration image and the second registration image are obtained, the first registration network can be trained based on all the characteristics respectively, and then the constraint of obtaining the deformation field in different training time is added in the training process.
Optionally, in the training process, the total loss may be determined based on the characteristics of the first deformation field, the first registration image, the second registration image, and the sample target image, and the first registration network is trained by using a gradient descent or back propagation algorithm, that is, network parameters of the first registration network are adjusted until the training condition is satisfied.
In summary, in the embodiment of the present application, the image registration is performed on the image to be registered and the target image through the first registration network and the second registration network, so as to obtain deformation fields corresponding to network parameters at different training times, and finally, different registration images are obtained after the deformation processing is performed on the image to be registered based on the different deformation fields, so that the first registration network can be trained through the different registration images and the target image, so that the network parameters obtained by the first registration network at the different training times tend to be stable, even if the deformation fields predicted by the first registration network at the different training times tend to be consistent, time constraints are added in the training process, which is beneficial to improving the training effect of the first registration network, and further improves the accuracy of the deformation field predicted by the first registration network.
In the embodiment of the present application, the first registration network is optimized by using timing information in a training process, and in order to constrain differences of deformation fields at different training times, a temporal regularization loss is introduced on the basis of a similarity loss and a spatial regularization loss, so that the first registration network is trained based on the three, and in the training process, constraints are applied to the first registration network in dual dimensions of space and time, which will be described in an exemplary embodiment below.
Referring to fig. 4, a flowchart of an image registration method provided by an exemplary embodiment of the present application is shown. The embodiment is described by taking the method as an example for a computer device, and the method comprises the following steps.
Step 401, a sample image pair is obtained, and the sample image pair includes a sample to-be-registered image and a sample target image.
Step 402, inputting the sample image into a first registration network to obtain a first deformation field between the sample image to be registered and the sample target image, and performing deformation processing on the sample image to be registered through the first deformation field to obtain a first registration image.
Step 403, inputting the sample image into a second registration network to obtain a second deformation field between the sample image to be registered and the sample target image, and performing deformation processing on the sample image to be registered through the second deformation field to obtain a second registration image, where a first network parameter of the first registration network and a second network parameter of the second registration network are network parameters at different training times.
The implementation of steps 401 to 403 can refer to steps 301 to 303, which are not described again in this embodiment.
A similarity loss is determined based on the first registered image and the sample target image, step 404.
As the image registration is to register the image to be registered of the sample and the target image of the sample, in order to improve the accuracy of predicting the deformation field by the first registration network, the similarity loss can be determined according to the difference of the first registration image and the target image of the sample in space, and then the first registration network is trained by utilizing the similarity loss.
Image registration can be divided into single modality image registration, which refers to registering pairs of sample images from the same imaging device, e.g., the pairs of sample images are both from CT, and multi-modality image registration, which refers to registering pairs of sample images from different imaging devices, e.g., one image in a pair of sample images is from CT and one is from PET. In one possible embodiment, to be suitable for both single-Modality image registration and multi-Modality image registration, dissimilarity of a Model Independent Neighbor Descriptor (MIND) is used to quantify the similarity loss.
Wherein MIND is:
Figure BDA0003115483170000091
i is the image, x is the location of the voxel in the image, r is the distance vector, and V (I, x) is the estimate of the local variance，Dp(I, x, x + r) represents the L2 distance between the image block centered at x and the image block centered at x + r.
During training of the first registration network, the objective is to minimize the difference between the MIND features of the first registration image and the MIND features of the sample target image. Thus, in one possible embodiment, the loss function of the similarity loss is:
Figure BDA0003115483170000092
wherein, IwsI.e. the first registered image, IfFor a sample target image, N represents the number of voxels in the image.
Step 405, determining a spatial regularization loss based on the first deformation field, the spatial regularization loss being used to apply a spatial constraint to the deformation field.
In the process of registering the sample image to be registered and the sample target image, when the sample image to be registered is deformed, that is, subjected to spatial transformation, spatial transformation can be performed in various ways, such as affine transformation, projection transformation, bending transformation, and the like, so that the solution space of the deformation field of the sample image to be registered for spatial transformation is not unique, that is, in the process of registering the sample image to be registered and the sample target image by the first registration network, the predicted deformation field has more differences in space, and therefore, the deformation field needs to be spatially constrained.
In one possible implementation, the smoothness of the deformation field output by the first registration network is ensured by setting a spatial regularization loss constraint deformation field and training the first registration network by using the spatial regularization loss. Alternatively, the spatial regularization loss may be measured in terms of the gradient of the first deformation field, with a loss function as follows:
Figure BDA0003115483170000101
wherein,
Figure BDA0003115483170000103
representing the gradient of the first deformation field.
Step 406, determining a temporal regularization loss based on the first and second registered images, the temporal regularization loss for applying a training temporal constraint to the deformation field.
Because the network parameters of the first registration network and the second registration network are network parameters under different training times, the first registration network and the second registration network predict the deformation field to have a difference in time, the difference of the first registration network and the second registration network predict the deformation field can be measured through the similarity of the first registration image and the second registration image in space, namely, the time regularization loss can be determined through the similarity of the first registration image and the second registration image in space, and thus, the time constraint is trained for the deformation field time of the first registration network.
In one possible embodiment, the temporal regularization loss is measured according to the spatial similarity between the first and second registered images, and therefore, the temporal regularization loss can also be determined according to the difference between the MIND features of the first and second registered images, and the loss function is as follows:
Figure BDA0003115483170000102
wherein, IwsI.e. the first registered image, IwtI.e. the second registered image, N is the number of voxels in the image.
Step 407, train the first registration network based on the similarity loss, the spatial regularization loss, and the temporal regularization loss.
After the similarity loss, the spatial regularization loss and the temporal regularization loss are determined, the first registration network is trained based on the similarity loss, the accuracy of predicting the deformation field by the first registration network is improved through the similarity loss, spatial constraint and temporal constraint can be respectively applied to the deformation field through the spatial regularization loss and the temporal regularization loss, and the training effect of the first registration network is optimized.
In the related art, a fixed weight is set for a spatial regularization loss, and the regularization strength is controlled, in the training process, for each training sample, that is, each sample image pair, the training difficulty is different, that is, there is a difference between the number of uncertain deformation fields generated when predicting the deformation fields between the sample image pairs, when the solutions of the predicted deformation fields are more, unnecessary deformation complexity needs to be reduced, that is, a stronger regularization constraint solution space of the deformation fields is needed, and when the solutions of the predicted deformation fields are less, the regularization constraint strength can be reduced. Therefore, if a fixed weight is adopted, that is, a fixed regularization strength is set, the method cannot be adapted to each training sample, and accordingly, the performance of the first registration network obtained by training is poor.
Therefore, in the embodiment of the present application, the loss weight of the spatial regularization loss and the loss weight of the temporal regularization loss are adjusted according to the uncertainty of each sample image to the predicted deformation field, so that the loss weights are suitable for different training samples. In one possible implementation, a first loss weight of the spatial regularization loss and a second loss weight of the temporal regularization loss are determined by a second registration network. Namely, the second registration network can be used for determining the time regularization loss, and can also be used for estimating the uncertainty of the sample image to the predicted deformation field, so as to be used for adaptively adjusting the loss weight. Determining the loss weights by the second registration network may comprise the steps of:
step one, forward prediction is carried out on the sample image pair for n times through a second registration network, and n third deformation fields between the sample image to be registered and the sample target image are obtained, wherein n is an integer larger than or equal to 2.
Because the uncertainty of the predicted deformation field needs to be determined, namely the quantitative trend of the predicted deformation field is judged, the sample image needs to be subjected to forward prediction for n times through the second registration network, so that n deformation fields among the sample image pairs are obtained, the uncertainty of the deformation field can be determined according to the dispersion degree of the n deformation fields, and finally the loss weight of space regularization loss, namely the strength of space regularization, is determined based on the uncertainty of the deformation field.
And secondly, carrying out deformation processing on the sample image to be registered through the n third deformation fields to obtain n third registration images.
Because the time regularization loss is determined according to the spatial similarity between the first registration image and the second registration image, the time regularization strength is determined according to the uncertainty of the obtained registration image after the deformation field is predicted by the second registration network to carry out deformation processing on the image to be registered of the sample. Namely, the n third deformation fields are used for carrying out deformation processing on the sample images to be registered to obtain n third registration images, and the loss weight of time regularization loss is determined according to the uncertainty of the n third registration images.
And step three, determining a first loss weight based on the n third deformation fields.
Optionally, the first loss weight is a loss weight of the spatial regularization loss, and a process of determining the first loss weight according to the n third deformation fields is as follows:
and a, determining the average deformation field and the standard deviation of the deformation fields of the n third deformation fields.
Optionally, the uncertainty of the n third deformation fields is determined according to the degree of dispersion of the n third deformation fields, so that first the standard deviation of the average deformation field of the n third deformation fields from the deformation field is determined.
Wherein the average deformation field of the n third deformation fields is:
Figure BDA0003115483170000121
where c denotes the c-th channel of the deformation field (i.e. displacement in x, y, z direction), i denotes the i-th forward prediction,
Figure BDA0003115483170000122
representing the third deformation field resulting from the i-th forward prediction.
The deformation field standard deviation of the n third deformation fields is as follows:
Figure BDA0003115483170000123
and b, determining the uncertainty of the deformation field based on the standard deviation of the deformation field and the average deformation field.
In one possible embodiment, the uncertainty of the deformation field is determined from the absolute value of the ratio of the standard deviation of the deformation field to the mean deformation field. The manner is as follows:
Figure BDA0003115483170000124
and c, determining a first loss weight based on the uncertainty of the deformation field, wherein the first loss weight and the uncertainty of the deformation field are in positive correlation.
Determining a first loss weight based on the deformation field uncertainty when
Figure BDA0003115483170000125
The larger the distortion field is, the larger the dispersion degree of the predicted distortion field is, that is, the deformation field between the predicted sample image pairs tends to generate more uncertainty prediction, so that a larger first loss weight needs to be set, and the solution space of the distortion field is constrained, that is, the first loss weight and the uncertainty of the deformation field are in a positive correlation.
In one possible embodiment, the first loss weight is determined based on the deformation field uncertainty as follows:
Figure BDA0003115483170000126
wherein λ isφI.e. the first loss weight, II (-) is an indicator function, i.e. when satisfied
Figure BDA0003115483170000128
When it is, it is set to "1", and when it is
Figure BDA0003115483170000129
When the number is "0", v represents the v-th voxel;
Figure BDA0003115483170000127
to represent
Figure BDA00031154831700001210
I.e. the deformation field uncertainty per voxel
Figure BDA0003115483170000136
Sum, k1Is a scaling value for the first loss weight, for giving the weight an empirically reasonable upper limit; tau is1Is a threshold for selecting an uncertain target.
Namely, a threshold value is set for the uncertainty of the deformation field, a first loss weight is determined according to the proportion of the uncertainty of the deformation field, which is greater than the threshold value, in all voxels corresponding to the uncertainty of the deformation field, and when the proportion is greater, namely the uncertainty is greater, the corresponding first loss weight is greater, namely the spatial regularization strength is stronger.
And step four, determining a second loss weight based on the n third registration images.
Optionally, the second loss weight is a loss weight of temporal regularization loss, and the step of determining the second loss weight according to the n third registration images is as follows:
and a, determining an average registered image and a registered image standard deviation of the n third registered images.
Optionally, in the same manner as the above-mentioned method for determining the uncertainty of the deformation field, the uncertainty of the n third registration images is determined according to the dispersion degree of the n third registration images, so that an average registration image and a standard deviation of the registration image of the n third registration images are obtained first.
Wherein the average of the n third registration images is:
Figure BDA0003115483170000131
wherein,
Figure BDA0003115483170000132
namely obtaining a third deformation field through the ith forward prediction, and carrying out deformation processing on the image to be registered of the sample to obtain a third registration image.
The registered image standard deviations for the n third registered images are:
Figure BDA0003115483170000133
and b, determining the uncertainty of the registered image based on the standard deviation of the registered image and the average registered image.
Optionally, the registered image uncertainty is also determined from an absolute value of a ratio of a registered image standard deviation to an average registered image. The manner is as follows:
Figure BDA0003115483170000134
and c, determining a second loss weight based on the uncertainty of the registered image, wherein the second loss weight and the uncertainty of the registered image are in positive correlation.
Determining a first loss weight based on the uncertainty of the registered image when the image is not registered
Figure BDA0003115483170000135
The larger the difference is, the larger the dispersion degree of the registration image obtained based on the predicted deformation field is, and correspondingly, the larger the dispersion degree of the predicted deformation field is, that is, the deformation field between the predicted sample image pairs tends to generate more uncertainty predictions, so that a larger second loss weight needs to be set to constrain a solution space of the deformation field, that is, the second loss weight and the uncertainty of the deformation field are in a positive correlation.
In one possible embodiment, the second loss weight is determined based on the registration image uncertainty as follows:
Figure BDA0003115483170000141
wherein λ iscI.e. the second loss weight, II (-) is an indicator function, i.e. when satisfied
Figure BDA0003115483170000148
When it is, it is set to "1", and when it is
Figure BDA0003115483170000149
When the number is "0", v represents the v-th voxel;
Figure BDA0003115483170000142
to represent
Figure BDA00031154831700001410
I.e. each voxel corresponds to the registered image uncertainty
Figure BDA0003115483170000143
Sum, k2Is a scaling value for the second loss weight, for giving the weight an empirically reasonable upper limit; tau is2Is a threshold for selecting an uncertain target.
Setting a threshold value for the uncertainty of the registered image, determining a second loss weight according to the proportion of the uncertainty of the registered image, which is greater than the threshold value, in the uncertainty of the registered image corresponding to all voxels, wherein when the proportion is greater, namely the uncertainty is greater, the corresponding second loss weight is greater, namely the temporal regularization strength is stronger, and thus the difference of the deformation field of the first registered network training time is constrained.
After determining the first loss weight and the second loss weight, training the first registration network based on the similarity loss, the spatial regularization loss, and the temporal regularization loss comprises the steps of:
step one, based on similarity loss, space regularization loss, first loss weight, time regularization loss and second loss weight, weighting and calculating total loss.
The first loss weight is a loss weight of the spatial regularization loss, the second loss weight is a loss weight of the temporal regularization loss, and a total loss is obtained according to the similarity loss, the spatial regularization loss, the temporal regularization loss and the corresponding weight weighting, wherein a loss function is as follows:
Figure BDA0003115483170000144
wherein,
Figure BDA0003115483170000145
that is to say the loss of the degree of similarity,
Figure BDA0003115483170000146
for spatial regularization loss, λφIs the weight of the first loss, and,
Figure BDA0003115483170000147
for temporal regularization loss, λcIs the second loss weight.
And step two, training a first registration network based on the total loss.
And after the total loss is determined, training the first registration network based on the total loss, and finishing the training of the first registration network when the loss function reaches a convergence condition.
In the embodiment, time regularization loss is introduced, and the first registration network is trained based on similarity loss, time regularization loss and space regularization loss, so that the solution space of the deformation field is constrained in dual dimensions of time and space, the training effect of the first registration network is optimized, and the accuracy of the first registration network in predicting the deformation field is improved.
In addition, in this embodiment, a first loss weight is set for the spatial regularization loss, a second loss weight is set for the temporal regularization loss, and a deformation field uncertainty and a registration image uncertainty are determined based on n forward prediction results of the second registration network, so that the training difficulty of the training sample is determined according to the deformation field uncertainty and the registration image uncertainty, and thus the first loss weight and the second loss weight are adjusted to adapt the weights to the current training sample, so that the training effect of the first registration network is optimized, and the accuracy of predicting the deformation field by the first registration network is improved.
Referring to fig. 5, a flowchart of an image registration method provided by another exemplary embodiment of the present application is shown. The embodiment is described by taking the method as an example for a computer device, and the method comprises the following steps.
Step 501, a sample image pair is obtained, and the sample image pair comprises a sample to-be-registered image and a sample target image.
Step 502, inputting the sample image into a first registration network to obtain a first deformation field between the sample image to be registered and the sample target image, and performing deformation processing on the sample image to be registered through the first deformation field to obtain a first registration image.
The implementation of step 501 and step 502 can refer to step 301 and step 302, and this embodiment is not repeated.
Step 503, add random perturbation to the sample image pair.
The input of the first registration network and the second registration network is the same training sample, namely the same sample image pair, and in the training process, if the deformation fields predicted by the first registration network and the second registration network are consistent under different disturbances, the prediction performance of the first registration network and the second registration network is better, namely the network prediction accuracy and robustness are stronger. Therefore, in order to further improve the prediction performance of the first registration network, when a sample image pair is input to the second registration network, random disturbance is added to the sample image pair, so that the first registration network and the second registration network predict the deformation field between the sample image pair under different disturbances.
Optionally, the added random disturbance may be random gaussian noise, that is, random gaussian noise is added to both the sample image to be registered and the sample target image input to the second registration network.
Step 504, inputting the sample image after the random disturbance is added into a second registration network to obtain a second deformation field between the sample image to be registered and the sample target image.
And inputting the sample image pair added with the random Gaussian noise into a second registration network to obtain a second deformation field, wherein the second deformation field is a deformation field predicted after being disturbed, and the first deformation field is a deformation field predicted without being disturbed.
And 505, performing deformation processing on the sample to-be-registered image without random disturbance through the second deformation field to obtain a second registered image.
In a possible implementation manner, after the second deformation field is obtained, because the deformation processing is performed on the sample image to be registered through the STN network, an interpolation operation needs to be performed in the deformation processing process, and in order to avoid the influence of random disturbance on the interpolation operation, when the second deformation field and the sample image to be registered are input to the STN network, the sample image to be registered without random disturbance is input, that is, the sample image to be registered without random disturbance is subjected to the deformation processing, so that the second registration image is obtained.
Step 506, training a first registration network based on the first deformation field, the first registration image, the second registration image, and the sample target image.
The process of training the first registration network based on the first deformation field, the first registration image, the second registration image, and the sample target image may refer to steps 404 to 407 in the foregoing embodiment, which is not repeated in this embodiment.
It should be noted that, when determining the similarity loss based on the first registered image and the sample target image, the sample target image used is the sample target image to which random disturbance is not added.
Step 507, updating a second network parameter of the second registration network based on the first network parameter of the first registration network.
After each training, the first network parameters of the first registration network are updated therewith, and the second network parameters of the second registration network are network parameters at different training times from the first network parameters, and the second network parameters of the second registration network are updated with the first network parameters of the first registration network.
Optionally, based on the first network parameter, the EMA updates the current second network parameter of the second registration network to obtain an updated second network parameter.
When the second network parameter is updated based on the first network parameter, the second network parameter is updated by using Exponential Moving Average (EMA), in the following manner:
θ′k＝αθ′k-1+(1-α)θk
wherein, theta'kIs the k-th second network parameter, θkThe kth first network parameter is alpha, which is an EMA attenuation parameter and can be 0.99.
Optionally, the first network parameter is the same as the initial parameter of the second network parameter, and after the first network parameter is updated, the second network parameter is updated accordingly.
In this embodiment, when sample image pairs are input to the second registration network, random disturbance is added, so as to train the first registration network, and when the first registration network predicts the deformation field between the same image pair under different disturbances, the prediction results tend to be consistent, thereby improving the accuracy and robustness of predicting the deformation field by the first registration network.
In the above embodiment, when the sample image pairs are input to the second registration network, random disturbance is input to improve the robustness of the first registration network. And the second registration network is also used for carrying out uncertainty estimation, wherein n times of forward prediction are required in the process, in order to further improve the accuracy of network prediction of a deformation field, random disturbance is also added to the sample image pair in the n times of forward prediction process, and then the sample image pair added with the random disturbance is subjected to n times of forward prediction through the second registration network to obtain n third deformation fields between the sample image to be registered and the sample target image, and in order to avoid the influence of noise on difference operation, in the deformation processing process, the sample image to be registered which is not added with the random disturbance is subjected to deformation processing through the n third deformation fields to obtain n third registration images.
In one possible embodiment, as shown in FIG. 6, the first registration network is trained as follows, and the sample to-be-registered image I is inputmWith the sample object image IfTo the first registration netA network 601 and a second registration network 602, wherein a second network parameter of the second registration network 602 is EMA updated based on a first network parameter of the first registration network 601, and a noise ξ is added in a sample image pair input to the second registration network 602.
The deformation field phi is predicted by the first registration network 601sAccording to the deformation field phisDetermining spatial regularization loss
Figure BDA0003115483170000171
And will deform the field phisImage I to be registered with samples without noise addedmInput into STN network 603 to obtain first registration image IwsTo thereby obtain a first registered image IwsWith the sample object image IfDetermining similarity loss
Figure BDA0003115483170000172
And the deformation field phi is predicted by the second registration network 602tWill deform the field phitImage I to be registered with a sample without added noisemInput into STN network 603 to obtain second registration image IwtTo thereby obtain a first registered image IwsWith a second registered image IwtDetermining temporal regularization loss
Figure BDA00031154831700001710
Determining similarity loss
Figure BDA0003115483170000173
Loss of spatial regularization
Figure BDA0003115483170000174
And loss of temporal regularization
Figure BDA0003115483170000175
The loss weight may then also be determined by the second registration network 602. Forward prediction is performed n times by using the second registration network 602 to obtain n third deformation fields, deformation field uncertainty estimation is performed based on the n third deformation fields,obtaining the uncertainty of the deformation field
Figure BDA00031154831700001711
Thereby according to the uncertainty of the deformation field
Figure BDA00031154831700001712
Determining a first loss weight λφ(ii) a And after n third deformation fields are obtained, the n third deformation fields and the sample without noise are subjected to image I registrationmInputting the image data into the STN network 603 to obtain n third registration images, and estimating the uncertainty of the registration images based on the n third registration images to obtain the uncertainty of the registration images
Figure BDA0003115483170000176
Thereby based on registration image uncertainty
Figure BDA0003115483170000177
Determining a second loss weight λcFinally based on similarity loss
Figure BDA0003115483170000178
Loss of spatial regularization
Figure BDA0003115483170000179
First loss weight λφTime regularization loss
Figure BDA0003115483170000181
And a second loss weight λcThe first registration network 601 is trained.
In the above embodiment, a training process of the first registration network is described, and after the training of the first registration network is completed, the first registration network may be used to perform image registration, which will be described in the following with an exemplary embodiment.
Referring to fig. 7, a flowchart of an image registration method provided by another exemplary embodiment of the present application is shown. The embodiment is described by taking the method as an example for a computer device, and the method comprises the following steps.
Step 701, acquiring a target image pair, wherein the target image pair comprises an image to be registered and a reference image.
After the training of the first registration network is completed, the first registration network can be used for image registration, so that the image to be registered and the reference image are spatially aligned, and further image information in the image to be registered and the reference image is fused.
Optionally, when the first registration network is used for image registration, two images can be registered, and any one of the two images can be used as a reference image; image registration may also be performed on multiple images, i.e., the multiple images are spatially transformed to be spatially aligned with a given reference image.
Illustratively, as shown in FIG. 8, the target image pair includes an image I to be registered1And a reference picture I2Which respectively belong to medical images of the lungs acquired at different times for the same patient.
Step 702, inputting the target image pair into the trained first registration network to obtain a target deformation field between the image to be registered and the reference image.
And performing image registration on the image to be registered and the reference image by using the trained first registration network to obtain a target deformation field, and further performing space transformation on the image to be registered based on the target deformation field.
Illustratively, as shown in FIG. 8, a target deformation field φ is obtained by image registration using a first registration network 8011。
And 703, carrying out deformation processing on the image to be registered through the target deformation field to obtain a target registration image.
Optionally, when the image to be registered is subjected to deformation processing through the target deformation field, the STN network may also be used for spatial transformation, and the target deformation field and the image to be registered are input to the STN network to obtain a target registration image, which is spatially aligned with the reference image.
Schematically, as shown in FIG. 8, the target deformation field φ is obtained1Then, the image I is registered with the image I1Are jointly input into the STN802 to obtain a target registration image I3。
When the first registration network obtained by training the scheme provided by the embodiment is adopted to perform image registration, the accuracy of the registration result can be improved. Taking the registered target image as a chest single-peak CT image of the same patient lung in the expiration and inspiration phases as an example, the quantitative analysis of the registration effect of the iterative optimization method SyN and the deep learning-based benchmark test method VM in the related art and the registration effect of the trained first registration network in the embodiment of the present application is shown in table 1:
TABLE 1
Scheme(s) Dice(％) ASD(mm) Percentage of folded voxels (%)
Initial moving image 86.37 2.51 -
SyN 86.67 2.27 0.09％
VM(λ＝1) 90.77 1.90 0.06％
VM(λ＝3) 90.45 1.94 <0.001％
VM(λ＝5) 89.84 2.02 <0.0005％
The scheme of the application 91.40 1.67 <0.0005％
As shown in table 1, SyN, VM (weight λ of spatial regularization loss is 1, 3, and 5), and a deformation field predicted by the present disclosure are applied to an initially moving segmentation mask, and registration accuracy is quantified by using a Dice score and an Average Surface Distance (ASD), where the Dice score and the registration accuracy are in a positive correlation, and the ASD and the registration accuracy are in a negative correlation.
And the registration effect is quantified through the percentage of folded voxels, wherein the percentage of the folded voxels refers to the proportion of the missing voxels in all the voxels of the image when the image is subjected to deformation processing based on the predicted deformation field, and the higher the proportion is, the more the missing image information is, namely, the registration effect is poor.
Therefore, compared with the related technical scheme, the image registration accuracy can be obviously improved.
Fig. 9 is a block diagram of an image registration apparatus according to an exemplary embodiment of the present application, and as shown in fig. 9, the apparatus includes:
a sample image obtaining module 901, configured to obtain a sample image pair, where the sample image pair includes a sample image to be registered and a sample target image;
a first registration module 902, configured to input the sample image pair into a first registration network, to obtain a first deformation field between the sample image to be registered and the sample target image, and perform deformation processing on the sample image to be registered through the first deformation field, to obtain a first registration image;
a second registration module 903, configured to input the sample image pair into a second registration network to obtain a second deformation field between the sample image to be registered and the sample target image, and perform deformation processing on the sample image to be registered through the second deformation field to obtain a second registration image, where a first network parameter of the first registration network and a second network parameter of the second registration network are network parameters in different training times;
a training module 904 for training the first registration network based on the first deformation field, the first registration image, the second registration image, and the sample target image.
Optionally, the training module 904 includes:
a first loss determination unit for determining a similarity loss based on the first registered image and the sample target image;
a second loss determination unit, configured to determine a spatial regularization loss based on the first deformation field, where the spatial regularization loss is used to apply a spatial constraint to the deformation field;
a third loss determination unit, configured to determine a temporal regularization loss based on the first and second registration images, the temporal regularization loss being used to apply a training time constraint to a deformation field;
a training unit to train the first registration network based on the similarity loss, the spatial regularization loss, and the temporal regularization loss.
Optionally, the apparatus further comprises:
a weight determination module to determine a first loss weight of the spatial regularization loss and a second loss weight of the temporal regularization loss through the second registration network.
Optionally, the training unit is further configured to:
weighting a total loss based on the similarity loss, the spatial regularization loss, the first loss weight, the temporal regularization loss, and the second loss weight;
training the first registration network based on the total loss.
Optionally, the weight determining module includes:
the prediction unit is used for carrying out forward prediction on the sample image pair for n times through the second registration network to obtain n third deformation fields between the sample image to be registered and the sample target image, wherein n is an integer greater than or equal to 2;
the first processing unit is used for carrying out deformation processing on the image to be registered of the sample through the n third deformation fields to obtain n third registration images;
a first weight determination unit for determining the first loss weight based on the n third deformation fields;
a second weight determination unit for determining the second loss weight based on the n third registration images.
Optionally, the first weight determining unit is further configured to:
determining an average deformation field and a deformation field standard deviation of the n third deformation fields;
determining a deformation field uncertainty based on the deformation field standard deviation and the average deformation field;
determining the first loss weight based on the deformation field uncertainty, the first loss weight having a positive correlation with the deformation field uncertainty.
Optionally, the second weight determining unit is further configured to:
determining an average registered image and a registered image standard deviation of the n third registered images;
determining a registered image uncertainty based on the registered image standard deviation and the average registered image;
determining the second loss weight based on the registered image uncertainty, the second loss weight having a positive correlation with the registered image uncertainty.
Optionally, the prediction unit is further configured to:
adding random perturbations to the sample image pair;
performing forward prediction on the sample image pair subjected to random disturbance addition for n times through the second registration network to obtain n third deformation fields between the sample image to be registered and the sample target image;
the first processing unit is further configured to:
and carrying out deformation processing on the sample to-be-registered image without random disturbance through the n third deformation fields to obtain n third registered images.
Optionally, the second registration module 902 includes:
a disturbance adding unit for adding random disturbance to the sample image pair;
the registration unit is used for inputting the sample image subjected to random disturbance into a second registration network to obtain the second deformation field between the sample image to be registered and the sample target image;
and the second processing unit is used for carrying out deformation processing on the sample image to be registered without random disturbance through the second deformation field to obtain a second registration image.
Optionally, the apparatus further comprises:
a parameter update module to update the second network parameters of the second registration network based on the first network parameters of the first registration network.
Optionally, the parameter updating module is further configured to:
EMA updating is carried out on the current second network parameter of the second registration network based on the first network parameter, and the updated second network parameter is obtained.
Optionally, the apparatus further comprises:
the target image acquisition module is used for acquiring a target image pair, and the target image pair comprises an image to be registered and a reference image;
the third registration module is used for inputting the target image pair into the trained first registration network to obtain a target deformation field between the image to be registered and the reference image;
and the deformation processing module is used for carrying out deformation processing on the image to be registered through the target deformation field to obtain a target registration image.
In summary, in the embodiment of the present application, the image registration is performed on the image to be registered and the target image through the first registration network and the second registration network, so as to obtain deformation fields corresponding to network parameters at different training times, and finally, different registration images are obtained after the deformation processing is performed on the image to be registered based on the different deformation fields, so that the first registration network can be trained through the different registration images and the target image, so that the network parameters obtained by the first registration network at the different training times tend to be stable, even if the deformation fields predicted by the first registration network at the different training times tend to be consistent, time constraints are added in the training process, which is beneficial to improving the training effect of the first registration network, and further improves the accuracy of the deformation field predicted by the first registration network.
It should be noted that: the device provided in the above embodiment is only illustrated by dividing the functional modules, and in practical applications, the functions may be distributed by different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules, so as to complete all or part of the functions described above. In addition, the apparatus and method embodiments provided by the above embodiments belong to the same concept, and details of the implementation process are referred to as method embodiments, which are not described herein again.
Referring to fig. 10, a schematic structural diagram of a computer device according to an exemplary embodiment of the present application is shown. Specifically, the method comprises the following steps: the computer apparatus 1000 includes a Central Processing Unit (CPU) 1001, a system memory 1004 including a random access memory 1002 and a read only memory 1003, and a system bus 1005 connecting the system memory 1004 and the CPU 1001. The computer device 1000 also includes a basic Input/Output system (I/O system) 1006, which helps to transfer information between devices within the computer, and a mass storage device 1007, which stores an operating system 1013, application programs 1014, and other program modules 1015.
The basic input/output system 1006 includes a display 1008 for displaying information and an input device 1009, such as a mouse, keyboard, etc., for user input of information. Wherein the display 1008 and input device 1009 are connected to the central processing unit 1001 through an input-output controller 1010 connected to the system bus 1005. The basic input/output system 1006 may also include an input/output controller 1010 for receiving and processing input from a number of other devices, such as a keyboard, mouse, or electronic stylus. Similarly, the input-output controller 1010 also provides output to a display screen, a printer, or other type of output device.
The mass storage device 1007 is connected to the central processing unit 1001 through a mass storage controller (not shown) connected to the system bus 1005. The mass storage device 1007 and its associated computer-readable media provide non-volatile storage for the computer device 1000. That is, the mass storage device 1007 may include a computer-readable medium (not shown) such as a hard disk or a drive.
Without loss of generality, the computer-readable media may comprise computer storage media and communication media. Computer storage media includes volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Computer storage media includes Random Access Memory (RAM), Read Only Memory (ROM), flash Memory or other solid state Memory technology, Compact disk Read-Only Memory (CD-ROM), Digital Versatile Disks (DVD), or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices. Of course, those skilled in the art will appreciate that the computer storage media is not limited to the foregoing. The system memory 1004 and mass storage device 1007 described above may be collectively referred to as memory.
The memory stores one or more programs configured to be executed by the one or more central processing units 1001, the one or more programs containing instructions for implementing the methods described above, and the central processing unit 1001 executes the one or more programs to implement the methods provided by the various method embodiments described above.
According to various embodiments of the present application, the computer device 1000 may also operate as a remote computer connected to a network via a network, such as the Internet. That is, the computer device 1000 may be connected to the network 1012 through the network interface unit 1011 connected to the system bus 1005, or the network interface unit 1011 may be used to connect to another type of network or a remote computer system (not shown).
The memory also includes one or more programs, stored in the memory, that include instructions for performing the steps performed by the computer device in the methods provided by the embodiments of the present application.
The present embodiments also provide a computer-readable storage medium, in which at least one instruction, at least one program, a code set, or a set of instructions is stored, and the at least one instruction, the at least one program, the code set, or the set of instructions is loaded and executed by a processor to implement the image registration method according to any one of the above embodiments.
Embodiments of the present application provide a computer program product or computer program comprising computer instructions stored in a computer readable storage medium. The processor of the computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions to cause the computer device to perform the image registration method provided by the above aspect.
Those skilled in the art will appreciate that all or part of the steps in the methods of the above embodiments may be implemented by hardware related to instructions of a program, which may be stored in a computer readable storage medium, which may be a computer readable storage medium contained in a memory of the above embodiments; or it may be a separate computer-readable storage medium not incorporated in the terminal. The computer readable storage medium has stored therein at least one instruction, at least one program, set of codes, or set of instructions that is loaded and executed by a processor to implement the method of image registration according to any of the above method embodiments.
Optionally, the computer-readable storage medium may include: ROM, RAM, Solid State Drives (SSD), or optical disks, etc. The RAM may include a resistive Random Access Memory (ReRAM) and a Dynamic Random Access Memory (DRAM), among others. The above-mentioned serial numbers of the embodiments of the present application are merely for description and do not represent the merits of the embodiments.
It will be understood by those skilled in the art that all or part of the steps for implementing the above embodiments may be implemented by hardware, or may be implemented by a program instructing relevant hardware, where the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a magnetic disk or an optical disk, etc.
The above description is intended to be exemplary only, and not to limit the present application, and any modifications, equivalents, improvements, etc. made within the spirit and scope of the present application are intended to be included therein.

Claims (14)

1. A method of image registration, the method comprising:
obtaining a sample image pair, wherein the sample image pair comprises a sample image to be registered and a sample target image;
inputting the sample image into a first registration network to obtain a first deformation field between the sample image to be registered and the sample target image, and performing deformation processing on the sample image to be registered through the first deformation field to obtain a first registration image;
inputting the sample image into a second registration network to obtain a second deformation field between the sample image to be registered and the sample target image, and performing deformation processing on the sample image to be registered through the second deformation field to obtain a second registration image, wherein a first network parameter of the first registration network and a second network parameter of the second registration network are network parameters under different training times;
training the first registration network based on the first deformation field, the first registration image, the second registration image, and the sample target image.
2. The method of claim 1, wherein training the first registration network based on the first deformation field, the first registration image, the second registration image, and the sample target image comprises:
determining a similarity loss based on the first registered image and the sample target image;
determining a spatial regularization loss based on the first deformation field, the spatial regularization loss being used to impose a spatial constraint on the deformation field;
determining a temporal regularization loss based on the first and second registered images, the temporal regularization loss to apply a training time constraint for a deformation field;
training the first registration network based on the similarity loss, the spatial regularization loss, and the temporal regularization loss.
3. The method of claim 2, further comprising:
determining, by the second registration network, a first loss weight of the spatial regularization loss and a second loss weight of the temporal regularization loss;
the training the first registration network based on the similarity loss, the spatial regularization loss, and the temporal regularization loss comprises:
weighting a total loss based on the similarity loss, the spatial regularization loss, the first loss weight, the temporal regularization loss, and the second loss weight;
training the first registration network based on the total loss.
4. The method of claim 3, wherein the determining, by the second registration network, a first loss weight of the spatial regularization loss and a second loss weight of the temporal regularization loss comprises:
performing forward prediction on the sample image pair for n times through the second registration network to obtain n third deformation fields between the sample image to be registered and the sample target image, wherein n is an integer greater than or equal to 2;
carrying out deformation processing on the sample image to be registered through the n third deformation fields to obtain n third registration images;
determining the first loss weight based on the n third deformation fields;
determining the second loss weight based on the n third registered images.
5. The method of claim 4, wherein determining the first loss weight based on the n third deformation fields comprises:
determining an average deformation field and a deformation field standard deviation of the n third deformation fields;
determining a deformation field uncertainty based on the deformation field standard deviation and the average deformation field;
determining the first loss weight based on the deformation field uncertainty, the first loss weight having a positive correlation with the deformation field uncertainty.
6. The method of claim 4, wherein the determining the second loss weight based on the n third registered images comprises:
determining an average registered image and a registered image standard deviation of the n third registered images;
determining a registered image uncertainty based on the registered image standard deviation and the average registered image;
determining the second loss weight based on the registered image uncertainty, the second loss weight having a positive correlation with the registered image uncertainty.
7. The method of claim 4, wherein the forward predicting the sample image pair n times through the second registration network to obtain n third deformation fields between the sample image to be registered and the sample target image comprises:
adding random perturbations to the sample image pair;
performing forward prediction on the sample image pair subjected to random disturbance addition for n times through the second registration network to obtain n third deformation fields between the sample image to be registered and the sample target image;
the step of performing deformation processing on the sample image to be registered through the n third deformation fields to obtain n third registration images includes:
and carrying out deformation processing on the sample to-be-registered image without random disturbance through the n third deformation fields to obtain n third registered images.
8. The method according to any one of claims 1 to 7, wherein the inputting the sample image pair into a second registration network to obtain a second deformation field between the sample image to be registered and the sample target image, and performing deformation processing on the sample image to be registered through the second deformation field to obtain a second registration image comprises:
adding random perturbations to the sample image pair;
inputting the sample image added with the random disturbance into a second registration network to obtain the second deformation field between the sample image to be registered and the sample target image;
and carrying out deformation processing on the sample to-be-registered image without random disturbance through the second deformation field to obtain a second registered image.
9. The method of any of claims 1 to 7, wherein after training the first registration network based on the first deformation field, the first registration image, the second registration image, and the sample target image, the method further comprises:
updating the second network parameters of the second registration network based on the first network parameters of the first registration network.
10. The method of claim 9, wherein updating the second network parameters of the second registration network based on the first network parameters of the first registration network comprises:
EMA updating is carried out on the current second network parameter of the second registration network based on the first network parameter, and the updated second network parameter is obtained.
11. The method of any of claims 1 to 7, further comprising:
acquiring a target image pair, wherein the target image pair comprises an image to be registered and a reference image;
inputting the target image pair into the trained first registration network to obtain a target deformation field between the image to be registered and the reference image;
and carrying out deformation processing on the image to be registered through the target deformation field to obtain a target registration image.
12. An image registration apparatus, characterized in that the apparatus comprises:
the system comprises a sample image acquisition module, a sample image acquisition module and a sample image matching module, wherein the sample image acquisition module is used for acquiring a sample image pair which comprises a sample image to be registered and a sample target image;
the first registration module is used for inputting the sample image into a first registration network to obtain a first deformation field between the sample image to be registered and the sample target image, and performing deformation processing on the sample image to be registered through the first deformation field to obtain a first registration image;
the second registration module is configured to input the sample image into a second registration network to obtain a second deformation field between the sample image to be registered and the sample target image, and perform deformation processing on the sample image to be registered through the second deformation field to obtain a second registration image, where a first network parameter of the first registration network and a second network parameter of the second registration network are network parameters at different training times;
a training module to train the first registration network based on the first deformation field, the first registration image, the second registration image, and the sample target image.
13. A computer device comprising a processor and a memory having stored therein at least one instruction, at least one program, set of codes, or set of instructions, which is loaded and executed by the processor to implement the image registration method according to any one of claims 1 to 11.
14. A computer readable storage medium having stored therein at least one instruction, at least one program, a set of codes, or a set of instructions, which is loaded and executed by a processor to implement the image registration method according to any one of claims 1 to 11.
CN202110661355.5A 2021-06-15 2021-06-15 Image registration method, device, equipment and storage medium Pending CN113822792A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202110661355.5A CN113822792A (en) 2021-06-15 2021-06-15 Image registration method, device, equipment and storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202110661355.5A CN113822792A (en) 2021-06-15 2021-06-15 Image registration method, device, equipment and storage medium

Publications (1)

Publication Number Publication Date
CN113822792A true CN113822792A (en) 2021-12-21

Family

ID=78923855

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202110661355.5A Pending CN113822792A (en) 2021-06-15 2021-06-15 Image registration method, device, equipment and storage medium

Country Status (1)

Country Link
CN (1) CN113822792A (en)

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114359360A (en) * 2022-03-17 2022-04-15 成都信息工程大学 Two-way consistency constraint medical image registration algorithm based on countermeasure
CN115086121A (en) * 2022-06-15 2022-09-20 Oppo广东移动通信有限公司 Method and device for determining predistortion parameter value, terminal and storage medium
CN115861394A (en) * 2023-02-28 2023-03-28 福建自贸试验区厦门片区Manteia数据科技有限公司 Medical image processing method and device, storage medium and electronic equipment
CN117132515A (en) * 2023-02-20 2023-11-28 荣耀终端有限公司 Image processing method and electronic equipment

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114359360A (en) * 2022-03-17 2022-04-15 成都信息工程大学 Two-way consistency constraint medical image registration algorithm based on countermeasure
CN115086121A (en) * 2022-06-15 2022-09-20 Oppo广东移动通信有限公司 Method and device for determining predistortion parameter value, terminal and storage medium
CN117132515A (en) * 2023-02-20 2023-11-28 荣耀终端有限公司 Image processing method and electronic equipment
CN115861394A (en) * 2023-02-28 2023-03-28 福建自贸试验区厦门片区Manteia数据科技有限公司 Medical image processing method and device, storage medium and electronic equipment

Similar Documents

Publication Publication Date Title
WO2020215984A1 (en) Medical image detection method based on deep learning, and related device
CN110148142B (en) Training method, device and equipment of image segmentation model and storage medium
Eppenhof et al. Error estimation of deformable image registration of pulmonary CT scans using convolutional neural networks
US10664979B2 (en) Method and system for deep motion model learning in medical images
Dangi et al. A distance map regularized CNN for cardiac cine MR image segmentation
Rueckert et al. Model-based and data-driven strategies in medical image computing
Hansen et al. GraphRegNet: Deep graph regularisation networks on sparse keypoints for dense registration of 3D lung CTs
CN113822792A (en) Image registration method, device, equipment and storage medium
CN111369576B (en) Training method of image segmentation model, image segmentation method, device and equipment
CN110599421B (en) Model training method, video fuzzy frame conversion method, device and storage medium
Ngo et al. Lung segmentation in chest radiographs using distance regularized level set and deep-structured learning and inference
Selvan et al. Uncertainty quantification in medical image segmentation with normalizing flows
CN113821668A (en) Data classification identification method, device, equipment and readable storage medium
Krebs et al. Probabilistic motion modeling from medical image sequences: application to cardiac cine-MRI
CN111091010A (en) Similarity determination method, similarity determination device, network training device, network searching device and storage medium
CN110517249A (en) Imaging method, device, equipment and the medium of ultrasonic elastic image
EP3672496A1 (en) Surgical decision support using a decision theoretic model
CN113822845A (en) Method, apparatus, device and medium for hierarchical segmentation of tissue structure in medical image
CN113822289A (en) Training method, device and equipment of image noise reduction model and storage medium
CN113724185A (en) Model processing method and device for image classification and storage medium
Ito et al. An Automated Method for Generating Training Sets for Deep Learning based Image Registration.
Zhang et al. Learning correspondences of cardiac motion from images using biomechanics-informed modeling
CN115120238A (en) Method, device and system for identifying first-onset schizophrenia patient based on federal learning multiple centers
CN113822323A (en) Brain scanning image identification processing method, device, equipment and storage medium
CN112164447B (en) Image processing method, device, equipment and storage medium

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination