WO2022198915A1 - 图像配准方法、装置、电子设备、存储介质及程序 - Google Patents

图像配准方法、装置、电子设备、存储介质及程序 Download PDF

Info

Publication number
WO2022198915A1
WO2022198915A1 PCT/CN2021/114524 CN2021114524W WO2022198915A1 WO 2022198915 A1 WO2022198915 A1 WO 2022198915A1 CN 2021114524 W CN2021114524 W CN 2021114524W WO 2022198915 A1 WO2022198915 A1 WO 2022198915A1
Authority
WO
WIPO (PCT)
Prior art keywords
feature map
image
registration
new
feature
Prior art date
Application number
PCT/CN2021/114524
Other languages
English (en)
French (fr)
Inventor
黄烨翀
叶宇翔
朱雅靖
陈翼男
Original Assignee
上海商汤智能科技有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 上海商汤智能科技有限公司 filed Critical 上海商汤智能科技有限公司
Priority to JP2022544835A priority Critical patent/JP2023522527A/ja
Publication of WO2022198915A1 publication Critical patent/WO2022198915A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/30—Determination of transform parameters for the alignment of images, i.e. image registration
    • G06T7/33—Determination of transform parameters for the alignment of images, i.e. image registration using feature-based methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/50—Image enhancement or restoration by the use of more than one image, e.g. averaging, subtraction
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20212—Image combination
    • G06T2207/20221—Image fusion; Image merging

Definitions

  • the present disclosure relates to the technical field of image processing, and in particular, to an image registration method, apparatus, electronic device, storage medium and program.
  • Image registration is an important part of image processing research, and its purpose is to compare or fuse images acquired under different conditions for the same object. At the same time, image registration has been widely used in computer vision, medical image processing, remote sensing and many other fields.
  • Embodiments of the present disclosure provide an image registration method, apparatus, electronic device, storage medium, and program.
  • An embodiment of the present disclosure provides an image registration method, the method is performed by an electronic device, and the method includes:
  • the resolution of the first feature map before decoding, and the resolution of the new second feature map is different from the resolution of the second feature map before this decoding;
  • the new first feature map and the new second feature map are fused to obtain a velocity field again;
  • a registration parameter for registering the first image and the second image is generated based on the velocity fields obtained by multiple times of the fusion.
  • velocity fields of different scales can be obtained, so that the accuracy of registration parameters can be improved based on the velocity fields of different scales, which is beneficial to improve the registration accuracy.
  • velocity fields of different scales are obtained in multiple stages, it is beneficial to obtain the registration parameters used to positively register the first image to the second image, and it is also beneficial to obtain the registration parameters used to register the second image in a positive direction.
  • the images are inversely registered to the registration parameters of the first image, which in turn can facilitate satisfying the differential homeomorphism.
  • the encoding of the first image to obtain the first feature map, or the encoding of the second image to obtain the second feature map includes: performing feature extraction on the image to obtain multiple channel feature maps; based on the importance of each of the channel feature maps in the multiple channel feature maps, obtain attention weights corresponding to the channel feature maps; and separately use the attention of each of the channel feature maps
  • the force weight performs weighting processing on the feature map corresponding to the channel to obtain the feature map of the image; wherein, in the case where the image is the first image, the feature map obtained through the above steps is the first feature or, when the image is the second image, the feature map obtained through the above steps is the second feature map.
  • weighting the corresponding channel feature maps can help to weaken the feature maps with strong performance in the multiple channel feature maps. Or strengthen the feature maps with weak performance in the feature maps of multiple channels, which can help to obtain similar feature maps after encoding images of different modalities, which can help to meet the registration of multi-modal images, and broaden the application scope.
  • the method before encoding the first image to obtain the first feature map, and encoding the second image to obtain the second feature map, the method further includes: acquiring to-be-registered and taking one of the plurality of images as the first image, and taking the remaining at least one image as the second image respectively.
  • the two images can be registered in the forward and reverse directions with only one registration process, which can help to reduce the number of registrations.
  • multi-modal registration multi-modal image registration can be achieved with only a few registration times.
  • the multiple images are medical images, and the multiple images satisfy any one of the following conditions: the multiple images are scanned by different types of medical equipment; Each image was scanned by the same medical device at different scan times. In this way, it can be beneficial to realize multimodal medical image registration.
  • the use of the obtained velocity field to fuse the new first feature map and the new second feature map to obtain the velocity field again includes: Converting the obtained velocity field to obtain a displacement field; deforming the new first feature map by using the displacement field to obtain a deformed feature map; and fusing the deformed feature map and the new second feature Figure, again to get the velocity field.
  • the displacement field can be obtained through the obtained velocity field
  • the velocity field can be obtained by re-merging the deformation feature map obtained by the deformation of the displacement field and the new second feature map, which is beneficial to the obtained velocity field.
  • the velocity field can be obtained again, which can help to optimize the velocity field through "multi-stage" and improve the accuracy of the velocity field.
  • the fusion of the first feature map and the second feature map to obtain the velocity field includes: splicing the first feature map and the second feature map to obtain splicing feature maps; and performing feature extraction on the splicing feature maps to obtain the velocity field. In this way, the process of acquiring the velocity field can be simplified and the efficiency of acquiring the velocity field can be improved.
  • the method before generating the registration parameters for registering the first image and the second image based on the velocity fields obtained by multiple fusions, the method further includes: Under the condition that the preset condition is satisfied, based on the newly obtained first feature map and second feature map, the decoding of the first feature map and the second feature map is performed again to obtain a new first feature map and The steps of the new second feature map and subsequent steps.
  • the decoding of the first feature map and the second feature map is performed again to obtain a new first feature map and The steps of the new second feature map and subsequent steps.
  • the preset condition includes any one of the following: the number of times of executing the decoding is less than a preset threshold, and the first feature map or the second feature map obtained by the most recent execution of the decoding
  • the resolution of the feature map is smaller than the preset resolution; and/or, the resolution of the new first feature map is greater than the resolution of the first feature map before this decoding, and the resolution of the new second feature map is The resolution is greater than the resolution of the second feature map before this decoding.
  • the method further includes at least the following: One: using the registration parameters to process the first image to obtain a registered image of the first image; using the registration parameters to process the second image to obtain the second image
  • the registration image is obtained; at least one first pixel in the first image is processed by using the registration parameters, and based on the processed at least one first pixel, the at least one second pixel point corresponding to the at least one first pixel point; or processing at least one second pixel point in the second image by using the registration parameter, based on the processed at least one second pixel point point to obtain at least one first pixel point in the first image corresponding to the at least one second pixel point respectively.
  • the registration of at least one pixel in the first image and the second image can be realized, which is beneficial to realize image registration from a local level.
  • the encoding of the first image to obtain the first feature map, and the encoding of the second image to obtain the second feature map includes: using the first encoder of the image registration model
  • the network encodes the first image to obtain a first feature map, and uses the second encoding sub-network of the image registration model to encode the second image to obtain a second feature map; the fusion of the first feature map and the second feature map to obtain a velocity field, including: using the velocity field sub-network of the image registration model to fuse the first feature map and the second feature map to obtain a velocity field;
  • Decoding the first feature map and the second feature map to obtain a new first feature map and a new second feature map, including: using the first decoding sub-network of the image registration model to decode the first feature map.
  • the feature map is decoded to obtain a new first feature map
  • the second feature map is decoded by using the second decoding sub-network of the image registration model to obtain a new second feature map.
  • the image registration model can be used to implement encoding, fusion, decoding, etc., which can help improve the efficiency of image registration.
  • Embodiments of the present disclosure provide an image registration apparatus, including:
  • an image encoding module configured to encode the first image to obtain the first feature map, and to encode the second image to obtain the second feature map;
  • a first fusion module configured to fuse the first feature map and the second feature map to obtain a velocity field
  • the image decoding module is configured to decode the first feature map and the second feature map respectively to obtain a new first feature map and a new second feature map; wherein, the new first feature map The resolution is greater than the resolution of the first feature map before the current decoding, and the resolution of the new second feature map is greater than the resolution of the second feature map before the current decoding;
  • the second fusion module is configured to use the obtained velocity field to fuse the new first feature map and the new second feature map to obtain the velocity field again;
  • the parameter acquisition module is configured to generate registration parameters for registering the first image and the second image based on the velocity fields obtained by the fusion multiple times.
  • the image encoding module includes: a feature extraction sub-module configured to perform feature extraction on an image to obtain multiple channel feature maps; a weight acquisition sub-module configured to The importance of the multiple channel feature maps is obtained, and the attention weight corresponding to the channel feature map is obtained; the feature map weighting sub-module is configured to use the attention weight of each channel feature map to correspond to the channel feature respectively.
  • the image configuration apparatus includes: an image acquisition module configured to acquire a plurality of images to be registered; and take one of the plurality of images as the first image, and respectively The remaining at least one image serves as the second image.
  • the multiple images are medical images, and the multiple images satisfy any one of the following conditions: the multiple images are scanned by different types of medical equipment; Each image was scanned by the same medical device at different scan times.
  • the second fusion module includes: a conversion sub-module configured to convert the obtained velocity field to obtain a displacement field; a deformation sub-module configured to use the displacement field to convert the The new first feature map is deformed to obtain a deformed feature map; the fusion sub-module is configured to fuse the deformed feature map and the new second feature map to obtain the velocity field again.
  • the first fusion module includes: a splicing sub-module configured to splicing the first feature map and the second feature map to obtain a spliced feature map; an extraction sub-module configured to Feature extraction is performed on the spliced feature map to obtain the velocity field.
  • the second fusion module is further configured to, under the condition that a preset condition is satisfied, based on the newly obtained first feature map and the second feature map, re-execute the respective first feature map and the second feature map.
  • the feature map and the second feature map are decoded to obtain a new first feature map and a new second feature map and subsequent steps.
  • the preset condition includes any one of the following: the number of times of executing the decoding is less than a preset threshold, and the first feature map or the second feature map obtained by the most recent execution of the decoding and/or, the resolution of the new first feature map is greater than the resolution of the first feature map before this decoding, and the resolution of the new second feature map It is larger than the resolution of the second feature map before this decoding.
  • the image configuration apparatus includes: an image processing module configured to perform at least one of the following: processing the first image using the registration parameters to obtain a registration of the first image image; processing the second image by using the registration parameters to obtain a registration image of the second image; processing at least one first pixel in the first image by using the registration parameters, Based on the processed at least one first pixel point, at least one second pixel point in the second image corresponding to the at least one first pixel point is obtained; At least one second pixel in the two images is processed, and based on the processed at least one second pixel, at least one first pixel corresponding to the at least one second pixel in the first image is obtained.
  • the image encoding module is further configured to encode the first image by using the first encoding sub-network of the image registration model to obtain a first feature map, and use the first encoding sub-network of the image registration model to encode the first image.
  • the second encoding sub-network encodes the second image to obtain a second feature map;
  • the first fusion module is further configured to use the velocity field sub-network of the image registration model to fuse the first feature map and the second feature map , to obtain the velocity field;
  • the image decoding module is further configured to use the first decoding sub-network of the image registration model to decode the first feature map, obtain a new first feature map, and use the image registration model to decode the first feature map.
  • the second decoding sub-network of the model decodes the second feature map to obtain a new second feature map.
  • An embodiment of the present disclosure further provides an electronic device, including a memory and a processor coupled to each other, the processor is configured to execute program instructions stored in the memory, so as to implement the image registration method described in any of the foregoing embodiments.
  • An embodiment of the present disclosure further provides a computer-readable storage medium, which stores program instructions, and when the program instructions are executed by a processor, implements the image registration method described in any of the foregoing embodiments.
  • An embodiment of the present disclosure further provides a computer program, where the computer program includes computer-readable codes, and when the computer-readable codes are executed in an electronic device, a processor of the electronic device executes any of the foregoing embodiments The described image registration method.
  • a first feature map is obtained by encoding a first image
  • a second image is obtained by encoding a second image.
  • feature map secondly, fuse the first feature map and the second feature map to obtain the velocity field, and decode the first feature map and the second feature map respectively to obtain a new first feature map and a second feature map, and the new
  • the resolution of the first feature map is different from the resolution of the first feature map before this decoding
  • the resolution of the new second feature map is different from the resolution of the second feature map before this decoding
  • the new first feature map and the new second feature map are fused to obtain the velocity field again, and based on the velocity fields obtained by multiple fusions, the first image and the second image are generated for registration.
  • registration parameters In this way, by fusing feature maps of different resolutions in multiple stages, velocity fields of different scales can be obtained, so that the accuracy of registration parameters can be improved based on the velocity fields of different scales, which is beneficial to improve the registration accuracy.
  • velocity fields of different scales are obtained in multiple stages, it is beneficial to obtain the registration parameters used to positively register the first image to the second image, and it is also beneficial to obtain the registration parameters used to register the second image in a positive direction.
  • the images are inversely registered to the registration parameters of the first image, which in turn can facilitate satisfying the differential homeomorphism.
  • FIG. 1 is a schematic flowchart of an embodiment of an image registration method provided by the present disclosure
  • FIG. 2 is a schematic diagram of a framework of an embodiment of an image registration model provided by the present disclosure
  • FIG. 3 is a schematic state diagram of an embodiment of image registration using a velocity field provided by the present disclosure
  • FIG. 4 is a schematic state diagram of another embodiment of image registration using a velocity field provided by the present disclosure.
  • FIG. 5 is a schematic diagram of a system architecture to which an image registration method according to an embodiment of the present disclosure can be applied;
  • FIG. 6 is a schematic flowchart of another embodiment of an image registration method provided by the present disclosure.
  • FIG. 7 is a schematic state diagram of an embodiment of an image registration method provided by the present disclosure.
  • FIG. 8 is a schematic diagram of a framework of an embodiment of a domain attention block provided by the present disclosure.
  • FIG. 9 is a schematic flowchart of an embodiment of a training method for an image registration model provided by the present disclosure.
  • FIG. 10 is a schematic frame diagram of an embodiment of an image registration apparatus 100 provided by the present disclosure.
  • FIG. 11 is a schematic frame diagram of an embodiment of an electronic device 110 provided by the present disclosure.
  • FIG. 12 is a schematic diagram of a framework of an embodiment of a computer-readable storage medium 120 provided by the present disclosure.
  • system and “network” are often used interchangeably herein.
  • the term “and/or” in this article is only an association relationship to describe the associated objects, indicating that there can be three kinds of relationships, for example, A and/or B, it can mean that A exists alone, A and B exist at the same time, and A and B exist independently B these three cases.
  • the character "/" in this document generally indicates that the related objects are an "or” relationship.
  • “multiple” herein means two or more than two. Multiple or multiple in the embodiments of the present disclosure may refer to at least two or at least two, respectively.
  • FIG. 1 is a schematic flowchart of an embodiment of an image registration method provided by the present disclosure. As shown in Figure 1, the following steps may be included:
  • Step S11 Encode the first image to obtain a first feature map, and encode the second image to obtain a second feature map.
  • the first image and the second image are images of the same object under different conditions.
  • the first image and the second image may be obtained by scanning the same object (eg, the abdomen, chest, etc. of the same patient) by different types of medical equipment.
  • the first image and the second image are respectively a computed tomography (CT) image and a magnetic resonance (Magnetic Resonance, MR) image obtained by scanning the abdomen of the patient; or, the first image and the second image may also be It is obtained by scanning the same medical equipment at different scanning times.
  • the scan times may correspond to different contrast durations during a scan.
  • the first image and the second image are any two of a plain scan image, an arterial phase image, a portal venous phase image, and a delayed phase image obtained by performing CT or MR scans of the patient's liver; in addition, the scan time is also Can correspond to different scans.
  • the first image was scanned in January and the second image was scanned in February.
  • the first image and the second image are other types of images other than medical images, it can be deduced by analogy, and the examples will not be exemplified here.
  • feature extraction may be performed on the first image and the second image respectively to obtain multiple channel feature maps of the first image, and multiple channel feature maps of the second image.
  • the first image is a CT image
  • there is at least one channel feature map with strong performance in the CT image which can reflect the grayscale characteristics of the CT image, or there is at least one channel feature map in the CT image with a relatively strong performance.
  • Weak channel feature maps can reflect the texture features of CT images.
  • the second image is an MR image
  • there is at least one channel feature map with strong performance in the MR image which can reflect the texture features of the MR image
  • there is at least one channel feature map with weak performance in the MR image which can reflect the texture features of the MR image.
  • the grayscale features of MR images can be deduced by analogy in other cases, and will not be exemplified one by one here.
  • the attention weight of the corresponding channel feature map can be obtained, and the corresponding channel feature map can be weighted by using the attention weight to obtain A first feature map of the first image, and a second feature map of the second image.
  • a smaller attention weight may be assigned to a channel feature map with stronger performance, or a larger attention weight may be assigned to a channel feature map with weak performance, or, at the same time, a higher attention weight may be assigned to the channel feature map with weaker performance.
  • Strong channel feature maps are assigned smaller attention weights, and weaker channel feature maps are assigned larger attention weights, so that the first and second feature maps are similar.
  • the first image is a CT image
  • the n1 channel feature maps reflect the texture features of the CT image
  • the remaining n2 channel feature maps reflect the grayscale features of the CT image
  • the above n1 channel feature maps can be assigned a relatively high value.
  • a large attention weight assigns a small attention weight to the above n2 channel feature maps, so that the grayscale features of the CT image can be weakened and the texture features of the CT image can be enhanced; and for the case where the second image is an MR image, In the case where m1 channel feature maps reflect the texture features of the MR image, and the remaining m2 channel feature maps reflect the grayscale features of the MR image, a smaller attention weight can be assigned to the m1 channel feature maps above, and Giving a larger attention weight to the above m2 channel feature maps can weaken the texture features of MR images and strengthen the grayscale features of MR images, so that the final coding of CT images and MR images of different modalities can be obtained.
  • the first feature map is similar to the second feature map, which is beneficial to satisfy the registration of multi-modal images and broaden the scope of application.
  • the first image and the second image are other images, it can be deduced by analogy, and the examples will not be exemplified here.
  • an image registration model in order to improve the efficiency of image registration, may be pre-trained, and the image registration model includes a first encoding sub-network and a second encoding sub-network for encoding, so that it can
  • the first encoding sub-network is used to encode the first image to obtain a first feature map
  • the second encoding sub-network is used to encode the second image to obtain a second feature map.
  • the first encoding sub-network may include at least one sequential connection Each feature extraction layer can correspondingly extract feature maps of different resolutions, and the feature map extracted by the last feature extraction layer is used as the first feature map of the first image.
  • the sub-network may also include at least one sequentially connected feature extraction layer, each feature extraction layer can correspondingly extract feature maps of different resolutions, and the feature map extracted by the last feature extraction layer is used as the second feature map.
  • the feature extraction layer may include at least a convolutional layer.
  • a first encoding sub-network may further be provided between adjacent feature extraction layers in the first encoding sub-network Domain attention block, and there is a second domain attention block between adjacent feature extraction layers in the second coding sub-network, both the first domain attention block and the second domain attention block are used to extract the feature extraction layer.
  • the resulting feature maps are domain transformed to make the first feature map and the second feature map similar.
  • linear registration may also be performed on the first image and the second image.
  • Linear registration may include, but is not limited to, rigid body registration and affine registration, which are not limited herein.
  • the steps in the embodiments of the present disclosure are performed using the first image and the second image after linear registration.
  • the accuracy of registration can be improved.
  • when the respective relative positions of the object in the first image and the object in the second image are different eg, when a CT scan is performed on the chest of the same object, the object may be The accuracy of registration can be greatly improved by performing linear registration first.
  • Step S12 Fusing the first feature map and the second feature map to obtain a velocity field.
  • the velocity field may be a physical field composed of velocity vectors at every moment and at every point.
  • each element in the velocity field represents the velocity vector of the center pixel of at least one pixel corresponding to the element in the first image during deformation, and other pixels
  • the velocity vector during deformation can be calculated by interpolation.
  • the velocity vectors of other pixels the velocity vectors of several central pixels closest to the pixel can be obtained, and the velocity vectors corresponding to each central pixel can be obtained.
  • the obtained weight is used to weight the velocity vector of the corresponding central pixel to obtain the velocity vector of the pixel.
  • the weight corresponding to the velocity vector of the center pixel is inversely proportional to the distance from the pixel to the corresponding center pixel, that is, the smaller the distance, the larger the weight, and the larger the distance, the smaller the weight.
  • each element in the velocity field corresponds to the center pixel of the 10*10 area of the first image when deforming
  • the velocity vector of the other pixels can be calculated by the above interpolation; or, if the first image is an image with a resolution of 720*720*720, and the velocity field is a physical field of 72*72*72, then Each element in the velocity field corresponds to the velocity vector of the center pixel in the 10*10*10 area of the first image during deformation, and the velocity vectors of other pixels during deformation can be obtained through the above interpolation calculation.
  • Other situations can be deduced by analogy, and no examples are given here.
  • the first feature map and the second feature map can be spliced to obtain a spliced feature map, and feature extraction is performed on the spliced feature map to obtain a velocity field.
  • the first feature map and the second feature map may be spliced in the channel dimension, so as to obtain a spliced feature map with double the number of channels and the same resolution. For example, if the first feature map and the second feature map are both feature maps with a resolution of W*H and the number of channels is C, then by splicing the first feature map and the second feature map, the number of channels is 2C, and the resolution is 2C. The rate is still the feature map of W*H.
  • the feature extraction of the stitched feature map can reduce the number of channels of the stitched feature map by half.
  • the feature extraction of the stitched feature map can reduce the number of channels of the stitched feature map by half.
  • an image registration model in order to improve the registration efficiency, can be pre-trained, and the image registration model includes a velocity field sub-network, so that the first feature can be fused by using the velocity field sub-network of the image registration model and the second feature map to obtain the velocity field.
  • the velocity field sub-network may include a splicing processing layer and a feature extraction layer that are connected in sequence, wherein the splicing processing layer is used for splicing the first feature map and the second feature map to obtain a splicing feature map, The feature extraction layer is used to perform feature extraction on the spliced feature map to obtain the velocity field.
  • the feature extraction layer may include at least a convolutional layer.
  • Step S13 Decode the first feature map and the second feature map respectively to obtain a new first feature map and a new second feature map.
  • the resolution of the new first feature map is different from the resolution of the first feature map before the current decoding
  • the resolution of the new second feature map is different from the resolution of the second feature before the current decoding.
  • the resolution of the graph For example, the resolution of the new first feature map may be greater than the resolution of the first feature map before the current decoding, and the resolution of the new second feature map may be greater than the resolution of the second feature map before the current decoding .
  • an image registration model in order to improve the registration efficiency, may be pre-trained, and the image registration model includes a first decoding sub-network, and the first decoding sub-network is used to perform a
  • the image registration model further includes a second decoding sub-network, and the second decoding sub-network is used to decode the second feature map, so that the first feature map can be decoded by the first decoding sub-network of the image registration model.
  • the first decoding sub-network may include at least one sequentially connected decoding processing layer.
  • the decoding processing layer may include any one of the following: a deconvolution layer and an upsampling layer, which are not limited here.
  • Step S14 Using the obtained velocity field, fuse the new first feature map and the new second feature map to obtain the velocity field again.
  • the velocity field obtained this time is obtained based on a new first feature map and a new second feature map, and the resolution of the new first feature map is different from that before decoding this time.
  • the resolution of the first feature map and the resolution of the new second feature map are different from the resolution of the second feature map before decoding this time, so the velocity field obtained this time is different from the velocity field obtained the previous time.
  • the resolution of the feature map will increase, resulting in an increase in the size of the velocity field. Obtain the velocity field from small to large scales.
  • the obtained velocity field can be converted to obtain a displacement field, and the new first feature map can be deformed by using the displacement field to obtain a deformed feature map, so that the deformation feature map and the new feature map can be fused.
  • the second feature map of again obtains the velocity field.
  • the displacement field can be obtained through the obtained velocity field, and the velocity field can be obtained by re-merging the deformation feature map obtained by the deformation of the displacement field and the new second feature map, which is beneficial to the obtained velocity field.
  • the velocity field can be obtained again, which can help to optimize the velocity field through "multi-stage" and improve the accuracy of the velocity field.
  • the deformed feature map and the new second feature map may be spliced to obtain a spliced feature map correspondingly, and feature extraction is performed on the spliced feature map to obtain the velocity field again.
  • fusing the deformed feature map and the new second feature map reference may be made to the foregoing description about fusing the first feature map and the second feature map.
  • the obtained velocity fields may be converted separately to obtain displacement fields corresponding to the velocity fields, and then the displacement fields corresponding to the obtained velocity fields may be fused (for example, performing in the channel dimension). stack) to obtain the displacement field used to deform the new first feature map.
  • the velocity field may be iterated for a preset number of times based on a differential manner to obtain a displacement field corresponding to the velocity field. The preset number of times is at least 1 time, for example, 1 time, 2 times, 3 times, or 4 times, etc., which is not limited here.
  • the velocity field can be denoted as VF
  • the displacement field corresponding to the velocity field VF can be denoted as DF
  • formula (1) the relationship between the velocity field and the displacement field can be expressed as formula (1) by ordinary differential equation:
  • t represents time, so the minimum time unit can be recorded as dt, then the displacement VFdt of the velocity field VF corresponding to the minimum time unit can be obtained.
  • the obtained velocity field can be scaled normalized, and then the velocity field obtained after scale normalization can be converted by the following formula (2) to obtain the displacement field:
  • VF 1 and VF 2 represent the velocity field obtained after scale normalization
  • f( ) represents the conversion function that converts the velocity field into the displacement field.
  • the image registration model may further include a deformation layer, which is used to transform the obtained velocity field to obtain a displacement field, and use the displacement field to align the new first feature
  • the graph is deformed to obtain a deformed feature map.
  • the preset condition is met, and if the preset condition is met, based on the newly obtained first feature map and second feature map, the Perform the above step S13 and subsequent steps, and the resolution of the new first feature map is greater than the resolution of the first feature map before this decoding, and the resolution of the new second feature map is greater than the resolution of the second feature before this decoding.
  • the resolution of the graph can help to obtain a velocity field with a scale from small to large in the process of changing the resolution of the feature map from low to high, so as to facilitate the realization of multi-stage registration from "coarse to fine", which can be beneficial to Improve registration accuracy.
  • the velocity field VF 1 can be obtained by fusing the first feature map 01_1 and the second feature map 02_1 with the velocity field sub-network 1 .
  • the decoding processing layer 11 use the decoding processing layer 11 to decode the first feature map 01_1 to obtain a new first feature map 01_2, and use the decoding processing layer 21 to decode the second feature map 02_1 to obtain a new second feature map 02_2, use the deformation layer 1 to convert the velocity field VF 1 to obtain the displacement field f(VF 1 ), and use the displacement field f(VF 1 ) to deform the new first feature map 01_2 to obtain the deformation feature map 01_2', use The velocity field sub-network 2 fuses the deformation feature map 01_2' and the new second feature map 02_2 to obtain the velocity field VF 2 .
  • the newly obtained first feature map is the first feature map 01_2, and the latest obtained first feature map is the first feature map 01_2.
  • the second feature map is the second feature map 02_2.
  • the velocity fields VF 1 , VF 2 and VF 3 can be obtained.
  • f represents the transfer function that transforms the velocity field into a displacement field.
  • the velocity field sub-networks in the image registration model are more (or less) than the image registration model shown in FIG. 2 , the same can be deduced, and no examples are given here.
  • each velocity field sub-network may be the same. Taking FIG. 2 as an example, the velocity field sub-network 1, the velocity field sub-network 2, and the velocity field sub-network 3 may each include a splicing processing layer and a convolutional layer. Floor. In addition, each velocity field sub-network can also be set to have different network structures according to the actual design of the neural network, which is not limited here.
  • the preset conditions include any one of the following: the number of times of decoding is less than a preset threshold, and the resolution of the first feature map or the second feature map obtained by the most recent decoding is less than a preset resolution ; the resolution of the new first feature map is greater than the resolution of the first feature map before the current decoding, and the resolution of the new second feature map is greater than the resolution of the second feature map before the current decoding.
  • the preset threshold when the preset condition includes: when the number of times of performing decoding is less than the preset threshold, the preset threshold may be set to at least 2 times, for example, 2 times, 3 times, or 4 times, etc., here Not limited.
  • the preset condition includes: when the resolution of the first feature map or the second feature map obtained by the latest decoding is smaller than the preset resolution, the preset resolution may be set to the first image or the first image The original resolution of the two images, in addition, the preset resolution may also be smaller than the original resolution, or larger than the original resolution, which is not limited here.
  • the decoding of the first feature map is performed by the first decoding sub-network of the image registration model
  • the decoding of the first feature map is performed by the second decoding sub-network of the image registration model.
  • the preset condition may include any of the following situations: the last decoding processing layer in the first decoding sub-network performs decoding, or the last decoding processing layer in the second decoding sub-network performs decoding Decoding processing layer.
  • step S15 in the embodiments of the present disclosure may be performed to generate a first image for registration based on the velocity fields obtained by multiple fusions and the registration parameters of the second image.
  • Step S15 Generate registration parameters for registering the first image and the second image based on the velocity fields obtained by multiple fusions.
  • the velocity field obtained by multiple fusions may be transformed to obtain a displacement field, so that the displacement field may be used as a registration parameter for registering the first image and the second image.
  • the "velocity field obtained by fusion” may be a velocity field obtained by fusing feature maps, and reference may be made to the foregoing related descriptions.
  • the velocity fields obtained by previous fusions may be converted to obtain a displacement field, and on this basis, the displacement field may be used as a registration for registering the first image and the second image Alternatively, part of the velocity field can be selected from the velocity fields obtained by previous fusions, and the selected velocity field can be converted to obtain the displacement field, so that the displacement field can be used as the first image for registering the second image and the second one.
  • Image registration parameters can be set according to actual application needs.
  • the registration parameters can be obtained based on the velocity fields obtained by previous fusions; while in the case of relatively loose requirements on the accuracy of registration parameters, the registration parameters can be obtained from previous fusions. Some velocity fields are selected from the obtained velocity fields, and registration parameters are obtained based on the selected velocity fields.
  • FIG. 3 is a schematic state diagram of an embodiment of image registration using a velocity field provided by the present disclosure.
  • FIG. 3 is a schematic diagram of a "single-stage" image registration state.
  • the formula Indicates that the original image x, namely 301 (that is, the concentric circle image shown in the left side of Figure 3) is deformed by the displacement field f(V) obtained by the conversion of the velocity field V, and the deformed image (that is, the deformed image shown in the middle of Figure 3) is obtained.
  • the formula It means that the displacement field f(-V) obtained by inverting the velocity field V can deform the deformed image x (that is, the deformed image shown in the middle of Fig. 3), and the original image (that is, the concentric circle image) can still be restored, that is, 303, while the formula It means inverting the displacement field f(V) obtained by converting the velocity field V to obtain a new displacement field -f(V), and using the new displacement field to deform the deformed image x (that is, the deformed image shown in the middle of Fig. 3), The original image (ie the concentric circle image) cannot be obtained, that is, 304 is obtained.
  • both the forward registration parameters for registering the first image to the second image and the registration parameters for registering the second image to the first image can be obtained.
  • FIG. 4 is a schematic state diagram of another embodiment of image registration using a velocity field.
  • FIG. 4 is a schematic diagram of a "multi-stage" image registration state.
  • the image of the concentric circles in the upper left corner of Figure 4 is the original image.
  • the velocity field whose scale changes from small to large
  • four images located on the right side of the same row of the original image in the upper left corner are obtained.
  • the scale Each element in the smaller velocity field corresponds to a larger pixel area of the image, while each element in the larger scale velocity field corresponds to a smaller pixel area of the image.
  • Deformation is performed at the overall level of the original image, that is, the deformation scale is "coarse", and the application of a larger-scale velocity field can deform at the local level of the original image, that is, the deformation scale is more "fine", that is, the first row of Figure 4 "positive”
  • the image in the lower right corner of Figure 4 uses the velocity field whose scale is changed from large to small and reversed, and the image in the lower right corner of Figure 4 can be obtained in the same row and left of the image.
  • the four images on the side that is, the change process shown in the "reverse" in the third row of Figure 4.
  • the images in the forward row and the reverse row in the same column are analyzed for differences, and the images in the second row in Figure 4 can be obtained.
  • the gray values of the images in the middle row of Figure 4 basically remain the same. Change. Therefore, in the "multi-stage" image registration, the image registration based on the velocity field can still satisfy the differential homeomorphism.
  • scale normalization can be performed on the velocity fields obtained by previous fusions, so that formula (3) can be used to generate a displacement field by using the velocity fields obtained by multiple fusions after scale normalization, and Use this displacement field as the forward registration parameter for registering the first image to the second image:
  • DF forward represents the forward registration parameter
  • VF 1 , VF 2 , ..., VF n represent the velocity field obtained by successive fusions after scale normalization, respectively
  • f() represents the velocity field used to convert the velocity field is a function of the displacement field, please refer to the previous description.
  • the velocity fields obtained by previous fusions can also be scaled normalized and inverted, so that formula (4) can be used to obtain the scale normalization and inversion of multiple fusions.
  • the velocity field of generates the displacement field, and uses the displacement field as the back-registration parameter for the user to register the second image to the first image:
  • DF backward represents the reverse registration parameter
  • symbol "—" represents the inversion operation
  • the set of velocity fields obtained through previous fusions can also be used as a registration parameter for registering the first image and the second image, so that the first image can be registered to the second image when needed.
  • the forward registration parameters for registering the first image to the second image can be obtained through the above formula (3), and when the second image needs to be registered to the first image, The inverse registration parameters for registering the second image to the first image can be obtained through the above formula (4).
  • the first image may be processed by using the registration parameters to obtain a registered image of the first image.
  • the first image may be processed by using the forward registration parameters to obtain a registered image of the first image.
  • the forward registration parameters can be used to realize the registration between images scanned by different types of equipment (for example, CT images, MR images), or to realize the correspondence between the same scanning equipment.
  • the registration between images at different scan times eg, images in the unenhanced phase, images in the arterial phase, images in the portal venous phase, and images in the delayed phase corresponding to different angiography durations.
  • the above method can realize the registration of all the pixels in the first image and the second image, which is beneficial to realize the image registration from the overall level.
  • the second image may be processed by using the registration parameters to obtain a registered image of the second image.
  • the second image may be processed by using the reverse registration parameters to obtain a registered image of the second image.
  • the inverse registration parameters can be used to realize the registration between images scanned by different types of equipment (for example, CT images, MR images), or to realize the correspondence between the same scanning equipment.
  • the above method can realize the registration of all the pixels in the first image and the second image, which is beneficial to realize the image registration from the overall level.
  • At least one first pixel in the first image is processed by using the registration parameters, so that the second pixel can be obtained based on the processed at least one first pixel. At least one second pixel point corresponding to at least one first pixel point in the image respectively.
  • at least one first pixel in the first image may be processed by using the forward registration parameters, so that the processed at least one first pixel pixel points, and obtain at least one second pixel point corresponding to at least one first pixel point in the second image respectively.
  • the forward registration parameters can be used to realize the registration between the pixels of the images scanned by different types of equipment (for example, CT images, MR images), or to achieve the same scan.
  • the device corresponds to the registration between pixels of images of different scan times (eg, images of the plain scan phase, images of the arterial phase, images of the portal venous phase, and images of the delayed phase corresponding to different angiography durations).
  • the above manner can realize the registration of at least one pixel in the first image and the second image, which is beneficial to realize image registration from a local level.
  • At least one second pixel in the second image may be processed by using the registration parameters, so that based on the processed at least one second pixel, the At least one first pixel in the first image corresponding to at least one second pixel respectively.
  • at least one second pixel in the second image may be processed by using the reverse registration parameters, so that the processed at least one second pixel pixel points, and at least one first pixel point corresponding to at least one second pixel point in the first image is obtained.
  • the inverse registration parameters can be used to realize the registration between the pixels of the images scanned by different types of equipment (for example, CT images, MR images), or to achieve the same scan.
  • the device corresponds to the registration between pixels of images of different scan times (eg, images of the plain scan phase, images of the arterial phase, images of the portal venous phase, and images of the delayed phase corresponding to different angiography durations).
  • the above manner can realize the registration of at least one pixel in the first image and the second image, which is beneficial to realize image registration from a local level.
  • the first feature map is obtained by encoding the first image
  • the second feature map is obtained by encoding the second image, so as to fuse the first feature map and the second feature map to obtain the velocity field
  • the first feature map and the second feature map are decoded to obtain a new first feature map and a second feature map, and the resolution of the new first feature map is different from the resolution of the first feature map before this decoding
  • the resolution of the new second feature map is different from the resolution of the second feature map before this decoding
  • the obtained velocity field is used to fuse the new first feature map and the new second feature map to re-
  • registration parameters for registering the first image and the second image are generated based on the velocity fields obtained by multiple fusions.
  • velocity fields of different scales can be obtained, so that the accuracy of registration parameters can be improved based on the velocity fields of different scales, which is beneficial to improve the registration accuracy.
  • velocity fields of different scales are obtained in multiple stages, it is beneficial to obtain the registration parameters used to positively register the first image to the second image, and it is also beneficial to obtain the registration parameters used to register the second image in a positive direction.
  • the images are inversely registered to the registration parameters of the first image, which in turn can facilitate satisfying the differential homeomorphism.
  • FIG. 5 shows a schematic diagram of a system architecture to which a display method in an augmented reality scenario according to an embodiment of the present disclosure can be applied; as shown in FIG. 5 , the system architecture includes: an image acquisition terminal 501 , a network 502 and an image registration terminal 503 .
  • the image acquisition terminal 501 and the image registration terminal 503 establish a communication connection through the network 502, the image acquisition terminal 501 reports the first image and the second image to the image registration terminal 503 through the network 502, and the image registration
  • the terminal 503 first encodes the first image to obtain the first feature map, and encodes the second image to obtain the second feature map; secondly, fuses the first feature map and the second feature map to obtain the velocity field, and respectively
  • the first feature map and the second feature map are decoded to obtain a new first feature map and a new second feature map, wherein the resolution of the new first feature map is different from the resolution of the first feature map before this decoding.
  • the image registration terminal 503 uploads the registration parameters to the network 502 , and sends the registration parameters to the image acquisition terminal 501 through the network 502 .
  • the image acquisition terminal 501 may include an image acquisition device, and the image registration terminal 503 may include a vision processing device or a remote server with visual information processing capabilities.
  • Network 502 may employ wired or wireless connections.
  • the image registration terminal 503 is a visual processing device
  • the image acquisition terminal 501 can be connected to the visual processing device through a wired connection, such as data communication through a bus;
  • the image registration terminal 503 is a remote server, the image The acquisition terminal 501 can perform data interaction with a remote server through a wireless network.
  • the image acquisition terminal 501 may be a vision processing device with a video capture module, or a host with a camera.
  • the image registration method of the embodiment of the present disclosure may be executed by the image acquisition terminal 501 , and the above-mentioned system architecture may not include the network 502 and the image registration terminal 503 .
  • FIG. 6 is a schematic flowchart of another embodiment of the image registration method provided by the present disclosure. As shown in Figure 6, the following steps may be included:
  • Step S61 Acquire multiple images to be registered, use one of the multiple images as the first image, and use the remaining at least one image as the second image respectively.
  • the multiple images are medical images, and the multiple images satisfy any one of the following conditions: the multiple images are scanned by different types of medical equipment, or the multiple images are obtained from the same Scanned by medical equipment at different scan times. Reference may be made to the relevant descriptions in the foregoing disclosed embodiments.
  • FIG. 7 is a schematic state diagram of an embodiment of an image registration method provided by the present disclosure.
  • the multiple images include: image A, image B, image C, and image D.
  • Image A is used as the first image
  • image B, image C, and image D are respectively used as the second image.
  • image A is an image of the portal venous phase
  • image B is an image of the unenhanced phase
  • image C is an image of the arterial phase
  • image D is an image of the delayed phase
  • the portal phase image can be used as the first image
  • the unenhanced phase image can be taken as the first image, respectively.
  • An image, an arterial phase image, and a delayed phase image are used as the second image.
  • Other situations can be deduced by analogy, and no examples are given here.
  • Step S62 Perform feature extraction on the image to obtain multiple channel feature maps.
  • the multiple channel feature maps are the channel feature maps corresponding to the first image
  • the multiple channel feature maps are the second image Corresponding channel feature map.
  • feature extraction may be performed on the first image to obtain multiple channel feature maps of the first image
  • feature extraction may be performed on the second image to obtain multiple channel feature maps of the second image.
  • reference may be made to the relevant descriptions in the foregoing disclosed embodiments.
  • Step S63 Based on the importance of each channel feature map in the multiple channel feature maps, the attention weight of the corresponding channel feature map is obtained.
  • a first domain attention block may also be set between adjacent feature extraction layers in the first coding sub-network, and the second There is also a second domain attention block between adjacent feature extraction layers in the coding sub-network, so the attention weights of the multiple channel feature maps of the first image can be obtained through the first domain attention block, and the second image The attention weights of multiple channel feature maps can be obtained through the second domain attention block.
  • the first domain attention block and the second domain attention block may have the same network structure.
  • the first domain attention block and the second domain attention block are collectively referred to as the domain attention block .
  • FIG. 8 is a schematic diagram of a framework of an embodiment of a domain attention block provided by the present disclosure.
  • the domain attention block includes a domain adaptation module and multiple channel attention modules.
  • Each channel attention module obtains the channel attention representation of the overall channel feature map respectively.
  • the attention representation is weighted to obtain the attention weight of each channel feature map.
  • each channel attention module processes the channel feature map of C*H*W, which can be The channel attention representation of C*1 is obtained, so that the first channel attention representation, the second channel attention representation, ..., the kth channel attention representation can be obtained respectively, and the domain adaptation module is for C*H *W channel feature map can be processed to obtain the weight combination of k*1, and then the channel attention representation of C*1 output by each channel attention module can be spliced to obtain the channel attention representation of C*k, and Do a dot product operation by combining the spliced C*k channel attention representation with the weight of k*1 (that is, in Figure 8 operation), the attention weight of C*1 can be obtained, that is, the attention weight of each channel feature map in the C channel feature maps.
  • the domain adaptation module may include a sequentially connected Global Average Pooling (GAP) layer, a Fully Connected (FC) layer, and a softmax.
  • the channel attention block can be SE (Sequeze and Excitation) block.
  • CT images often have more obvious grayscale features (such as the boundaries of bones and soft tissues), while texture features (such as soft tissues) are more obvious.
  • texture features such as soft tissues
  • the internal fine structure is relatively weak.
  • MR images often have weak grayscale features and obvious texture features. Whether it is CT images or MR images, multiple channel feature maps can be extracted through the feature extraction layer.
  • the adaptive channel attention representation (that is, the channel attention representation of the overall channel feature map) is obtained through multiple channel attention modules, and then different weights are given to different channel feature maps according to the channel attention representation through the domain adaptation module, so that The feature difference between images of different modalities can be weakened as much as possible, and the cross-domain adaptability of the image registration model can be improved, which is beneficial to realize the registration of images of different modalities in the same image registration model.
  • Step S64 weighting the corresponding channel feature maps by using the attention weight of each channel feature map, respectively, to obtain the image feature map.
  • the feature map of the image can be obtained by performing weighting processing on the feature map of the corresponding channel by using the attention weight of the feature map of each channel.
  • the feature map of the image when the image is the first image, the feature map of the image is the first feature map, and when the image is the second image, the feature map of the image is the second feature map.
  • the preset condition may include: the number of times of performing the feature extraction in step S62 is less than a preset threshold, and the most recent execution
  • the resolution of the channel feature map obtained by the feature extraction in step S62 is greater than the preset resolution.
  • the image obtained by the weighting process can be used as the input image of the feature processing described in step S62, and step S62 and Next steps.
  • the degree of similarity between the first feature map and the second feature map can be improved by multiple encodings.
  • the preset threshold can be set to at least one time, for example, one time, two times, three times, etc., which is not limited here; the preset resolution can be set according to actual application requirements, for example, It can be set to half or one third of the original resolution of the first image or the second image, which is not limited here.
  • the preset condition may include any one of the following: the feature extraction is performed for the last feature extraction layer of the first coding sub-network, and the feature extraction is performed for the last feature extraction layer of the second coding sub-network. Feature extraction layer.
  • Step S65 Fusing the first feature map and the second feature map to obtain a velocity field.
  • Step S66 Decode the first feature map and the second feature map respectively to obtain a new first feature map and a new second feature map.
  • the resolution of the new first feature map is different from the resolution of the first feature map before the current decoding
  • the resolution of the new second feature map is different from the resolution of the second feature before the current decoding.
  • the resolution of the graph is different from the resolution of the graph.
  • Step S67 Using the obtained velocity field, fuse the new first feature map and the new second feature map to obtain the velocity field again.
  • Step S68 Generate registration parameters for registering the first image and the second image based on the velocity fields obtained by multiple fusions.
  • the first feature map of the first image and the second feature map of the second image can be made similar, so that they can be applied to multi-modal inter-image registration, and at the same time
  • the registration based on the velocity field can satisfy the differential homeomorphism. Therefore, through the steps in the embodiments of the present disclosure, it is not only applicable to the registration between multimodal images, but also can reduce the number of registration times. Please refer to FIG.
  • any One registration must be performed between the two, and only one image registration model is required for these four images, that is, for n images, only one image registration model needs to be trained, and a total of n-1 times of registration is enough; in the case where "multi-modality” is not applicable and “differential homeomorphism” is not satisfied, for image A, image B, image C, and image D, the difference between any two is equal.
  • FIG. 9 is a schematic flowchart of an embodiment of a training method for an image registration model provided by the present disclosure. As shown in Figure 9, the following steps may be included:
  • Step S91 encode the first sample image by using the first encoding sub-network of the image registration model to obtain a first sample feature map, and use the second encoding sub-network of the image registration model to encode the second sample image , to obtain the second sample feature map.
  • Step S92 Using the velocity field sub-network of the image registration model to fuse the first sample feature map and the second sample feature map to obtain a sample velocity field.
  • Step S93 Use the first decoding sub-network of the image registration model to decode the first sample feature map to obtain a new first sample feature map, and use the second decoding sub-network of the image registration model to decode the second sample.
  • the feature map is decoded to obtain a new second sample feature map.
  • the resolution of the new first sample feature map is different from the resolution of the first sample feature map before the current decoding
  • the resolution of the new second sample feature map is different from the current decoding The resolution of the previous second sample feature map.
  • Step S94 Based on the velocity field sub-network of the image registration model, using the obtained sample velocity field, fuse the new first sample feature map and the new second sample feature map to obtain the sample velocity field again.
  • Step S95 Obtain sample registration parameters for registering the first sample image and the second sample image based on the previously obtained sample velocity fields.
  • forward sample registration parameters for registering the first sample image to the second sample image may be obtained based on the sample velocity fields obtained in the past.
  • the reverse sample registration parameters for registering the second sample image to the first sample image may also be obtained based on the sample velocity fields obtained in the past, which is not limited here. .
  • Step S96 Process the first sample image by using the sample registration parameters to obtain a sample registration image of the first sample image.
  • the sample registration parameters may be forward sample registration parameters.
  • the forward sample registration parameters may be used to process the first sample image to obtain the sample registration parameters of the first sample image. standard image.
  • the sample registration parameter is a reverse sample registration parameter
  • the second image may be processed by using the reverse sample registration parameter to obtain a sample registration image of the second sample image, which is not limited herein.
  • Step S97 Adjust the network parameters of the image registration model based on the difference between the second sample image and the sample registration image.
  • a loss value between the second sample image and the sample registration image may be calculated, and network parameters of the image registration model may be adjusted according to the loss value.
  • methods such as Stochastic Gradient Descent (SGD), Batch Gradient Descent (BGD), Mini-Batch Gradient Descent (MBGD), etc. can be used to utilize the loss value pair
  • SGD Stochastic Gradient Descent
  • BGD Batch Gradient Descent
  • MBGD Mini-Batch Gradient Descent
  • the network parameters of the image registration model are adjusted.
  • batch gradient descent refers to using all samples to update parameters in each iteration
  • stochastic gradient descent refers to using one sample to update parameters in each iteration
  • Mini-batch gradient descent refers to using a batch of samples to update parameters at each iteration.
  • a training end condition may also be set, and when the training end condition is satisfied, the training of the image registration model may be ended.
  • the training end conditions may include: the loss value is less than a preset loss threshold; the current training times reaches a preset times threshold (eg, 500 times, 1000 times, etc.), which are not limited herein.
  • the second image can be processed by using the reverse sample registration parameter to obtain the sample registration image of the second sample image, so that the sample registration image of the second sample image can be obtained based on the first sample registration parameter.
  • the difference between the sample registration image of this image and the second sample image adjusts the network parameters of the image registration model.
  • the sample velocity fields of different scales can be obtained, so that the accuracy of the sample registration parameters can be improved based on the sample velocity fields of different scales. It is beneficial to improve the accuracy of the image registration model.
  • the sample velocity fields of different scales are obtained in multiple stages, it is beneficial to obtain the sample registration parameters for positively registering the first sample image to the second sample image, and also to obtain the sample registration parameters.
  • the sample registration parameters for inversely registering the second sample image to the first sample image can be beneficial to satisfy the differential homeomorphism.
  • the embodiment of the present disclosure also provides an image registration device corresponding to the image registration method.
  • the implementation can be found in the implementation of the method.
  • FIG. 10 is a schematic frame diagram of an embodiment of an image registration apparatus 100 provided by the present disclosure.
  • the image registration apparatus 100 includes: an image coding module 1001, a first fusion module 1002, an image decoding module 1003, a second fusion module 1004 and a parameter acquisition module 1005,
  • the image encoding module 1001 is configured to encode the first image to obtain the first feature map, and to encode the second image to obtain the second feature map;
  • the first fusion module 1002 is configured to fuse the first feature map and the second feature map to obtain a velocity field
  • the image decoding module 1003 is configured to decode the first feature map and the second feature map respectively to obtain a new first feature map and a new second feature map; wherein, the resolution of the new first feature map is greater than this time The resolution of the first feature map before decoding, and the resolution of the new second feature map is greater than the resolution of the second feature map before decoding this time;
  • the second fusion module 1004 is configured to use the obtained velocity field to fuse the new first feature map and the new second feature map to obtain the velocity field again;
  • the parameter acquisition module 1005 is configured to generate registration parameters for registering the first image and the second image based on the velocity fields obtained by multiple fusions.
  • the image encoding module 1001 includes: a feature extraction sub-module, configured to perform feature extraction on the image to obtain multiple channel feature maps; The importance of the channel feature map is used to obtain the attention weight of the corresponding channel feature map.
  • the feature map weighting sub-module is configured to use the attention weight of each channel feature map to weight the corresponding channel feature map to obtain the features of the image.
  • the feature map obtained by the above steps is the first feature map; when the image is the second image, the feature map obtained by the above steps is the second feature map.
  • the image registration apparatus 100 includes: an image acquisition module configured to acquire a plurality of images to be registered; and take one of the plurality of images as the first image, and respectively take at least the remaining images One image as the second image.
  • the multiple images are medical images, and the multiple images satisfy the following conditions: the multiple images are scanned by different types of medical equipment; the multiple images are obtained by the same medical equipment in different Scan time scan obtained.
  • the second fusion module 1004 includes: a conversion sub-module configured to convert the obtained velocity field to obtain a displacement field; and a deformation sub-module configured to use the displacement field to transform the new first feature
  • the map is deformed to obtain a deformed feature map, and the sub-module is fused, which is configured to fuse the deformed feature map and a new second feature map, and the velocity field is obtained again.
  • the first fusion module 1002 includes: a splicing sub-module configured to splicing the first feature map and the second feature map to obtain a splicing feature map; an extraction sub-module configured to splicing the splicing feature map Perform feature extraction to get the velocity field.
  • the second fusion module 1004 is further configured to re-execute the first feature map and The second feature map is decoded to obtain a new first feature map and a new second feature map and subsequent steps.
  • the preset conditions include any one of the following: the number of times of decoding is less than a preset threshold, and the resolution of the first feature map or the second feature map obtained by the most recent decoding is less than a preset resolution and/or, the resolution of the new first feature map is greater than the resolution of the first feature map before this decoding, and the resolution of the new second feature map is greater than the resolution of the second feature map before this decoding. Rate.
  • the image registration apparatus 1000 includes: an image processing module configured to perform at least one of the following: processing the first image by using registration parameters to obtain a registered image of the first image; processing the second image with the registration parameters to obtain a registration image of the second image; processing at least one first pixel in the first image by using the registration parameters, and obtaining the second image based on the processed at least one first pixel At least one second pixel point in the image corresponding to at least one first pixel point; or at least one second pixel point in the second image is processed by using registration parameters, and based on the processed at least one second pixel point, obtain At least one first pixel in the first image corresponding to at least one second pixel respectively.
  • an image processing module configured to perform at least one of the following: processing the first image by using registration parameters to obtain a registered image of the first image; processing the second image with the registration parameters to obtain a registration image of the second image; processing at least one first pixel in the first image by using the registration parameters, and obtaining the second image based on the processed at least one
  • the image encoding module 1001 is further configured to encode the first image by using the first encoding sub-network of the image registration model to obtain a first feature map, and use the second encoding sub-network of the image registration model to encode the first image.
  • the encoding sub-network encodes the second image to obtain a second feature map
  • the first fusion module 1002 is further configured to use the velocity field sub-network of the image registration model to fuse the first feature map and the second feature map to obtain a velocity field
  • the decoding module 1003 is further configured to decode the first feature map by using the first decoding sub-network of the image registration model to obtain a new first feature map, and use the second decoding sub-network of the image registration model to decode the second feature map.
  • the image is decoded to obtain a new second feature map.
  • FIG. 11 is a schematic diagram of a framework of an embodiment of an electronic device 110 provided by the present disclosure.
  • the electronic device 110 includes a memory 101 and a processor 102 coupled to each other, and the processor 102 is configured to execute program instructions stored in the memory 101 to implement any of the above image registration methods.
  • the electronic device 110 may include, but is not limited to, a microcomputer and a server.
  • the electronic device 110 may also include a mobile device such as a notebook computer and a tablet computer, which is not limited herein.
  • the processor 102 is configured to control itself and the memory 101 to implement any of the image registration methods described above.
  • the processor 102 may also be referred to as a central processing unit (Central Processing Unit, CPU).
  • the processor 102 may be an integrated circuit chip with signal processing capability.
  • the processor 102 may also be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a Field-Programmable Gate Array (Field-Programmable Gate Array, FPGA) or other Programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.
  • a general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.
  • the processor 102 may be jointly implemented by an integrated circuit chip.
  • FIG. 12 is a schematic diagram of a framework of an embodiment of a computer-readable storage medium 120 provided by the present disclosure.
  • the computer-readable storage medium 120 stores program instructions 121 executable by the processor, and the program instructions 121 are configured to implement any of the image registration methods described above.
  • Embodiments of the present disclosure also provide a computer program, where the computer program includes computer-readable codes, and when the computer-readable codes run in an electronic device, the processor of the electronic device executes image registration according to any of the foregoing embodiments method.
  • the embodiments of the present disclosure further provide another computer program product, the computer program product carries program codes, and the instructions included in the program codes can be configured to execute the image registration method described in the above method embodiments.
  • the above-mentioned computer program product can be realized by means of hardware, software or a combination thereof.
  • the computer program product may be embodied as a computer storage medium, and in other embodiments, the computer program product may be embodied as a software product, such as a software development kit (Software Development Kit, SDK) and the like.
  • the apparatus involved in the embodiments of the present disclosure may be at least one of a system, a method, and a computer program product.
  • the computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of the present disclosure.
  • a computer-readable storage medium may be a tangible device that can hold and store instructions for use by the instruction execution device.
  • the computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing.
  • Examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable Programmable Read Only Memory (Electrical Programmable Read Only Memory, EPROM) or flash memory, Static Random Access Memory (Static Random-Access Memory, SRAM), Portable Compact Disc Read-Only Memory (CD-ROM), Digital Video Discs (DVDs), memory sticks, floppy disks, mechanical coding devices, such as punch cards or raised structures in grooves on which instructions are stored, and any suitable combination of the above.
  • RAM random access memory
  • ROM read-only memory
  • EPROM erasable Programmable Read Only Memory
  • flash memory Static Random Access Memory
  • SRAM Static Random Access Memory
  • CD-ROM Portable Compact Disc Read-Only Memory
  • DVDs Digital Video Discs
  • memory sticks floppy disks
  • mechanical coding devices such as punch cards or raised structures in grooves on which instructions are stored, and any suitable combination of the above.
  • Computer-readable storage media are not to be construed as transient signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (eg, light pulses through fiber optic cables), or through electrical wires transmitted electrical signals.
  • the computer readable program instructions described herein may be downloaded from a computer readable storage medium to various computing/processing devices, or to an external computer or external storage device over a network such as at least one of the Internet, a local area network, a wide area network, and a wireless network .
  • the network may include at least one of copper transmission cables, fiber optic transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and edge servers.
  • a network adapter card or network interface in each computing/processing device receives computer-readable program instructions from a network and forwards the computer-readable program instructions for storage in a computer-readable storage medium in each computing/processing device .
  • the computer program instructions for carrying out the operations of the present disclosure may be assembly instructions, Industry Standard Architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or in one or more source or object code written in any combination of programming languages, including object-oriented programming languages—such as Smalltalk, C++, etc., and conventional procedural programming languages, such as the “C” language or similar programming languages.
  • the computer readable program instructions may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer, or entirely on the remote computer or server implement.
  • the remote computer may be connected to the user's computer through any kind of network, including a Local Area Network (LAN) or Wide Area Network (WAN), or may be connected to an external computer (eg, using Internet service provider to connect via the Internet).
  • LAN Local Area Network
  • WAN Wide Area Network
  • electronic circuits such as programmable logic circuits, FPGAs, or Programmable Logic Arrays (PLAs), that can execute computer-readable Program instructions are read to implement various aspects of the present disclosure.
  • PDAs Programmable Logic Arrays
  • the functions or modules included in the apparatus provided in the embodiments of the present disclosure may be configured to execute the methods described in the above method embodiments, and the implementation process of the implementation may refer to the above method embodiments. describe.
  • the disclosed method and apparatus may be implemented in other manners.
  • the device implementations described above are only illustrative.
  • the division of modules or units is only a logical function division. In actual implementation, there may be other divisions.
  • units or components may be combined or integrated. to another system, or some features can be ignored, or not implemented.
  • the shown or discussed mutual coupling or direct coupling or communication connection may be through some interfaces, indirect coupling or communication connection of devices or units, which may be in electrical, mechanical or other forms.
  • Units described as separate components may or may not be physically separated, and components shown as units may or may not be physical units, that is, may be located in one place, or may be distributed over network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution in this implementation manner.
  • each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, or each unit may exist physically alone, or two or more units may be integrated into one unit.
  • the above-mentioned integrated units may be implemented in the form of hardware, or may be implemented in the form of software functional units.
  • the integrated unit if implemented as a software functional unit and sold or used as a stand-alone product, may be stored in a computer-readable storage medium.
  • the technical solutions of the present disclosure are essentially or contribute to the prior art, or all or part of the technical solutions can be embodied in the form of software product disclosure, and the computer software product is stored in a storage medium , including several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) or a processor (processor) to execute all or part of the steps of the methods of the various embodiments of the present disclosure.
  • the aforementioned storage medium includes: a U disk, a removable hard disk, a ROM, a RAM, a magnetic disk, or an optical disk and other mediums that can store program codes.
  • Embodiments of the present disclosure provide an image registration method, apparatus, electronic device, storage medium, and program, wherein the method is executed by the electronic device, and the method includes: encoding a first image to obtain a first feature map, Encode the second image to obtain a second feature map; fuse the first feature map and the second feature map to obtain a velocity field; decode the first feature map and the second feature map respectively , obtain a new first feature map and a new second feature map, wherein the resolution of the new first feature map is different from the resolution of the first feature map before this decoding, and the new first feature map The resolution of the second feature map is different from the resolution of the second feature map before this decoding; using the obtained velocity field, the new first feature map and the new second feature map are fused , to obtain a velocity field again; and based on the velocity field obtained by the fusion multiple times, a registration parameter for registering the first image and the second image is generated.

Abstract

一种图像配准方法、装置、电子设备、存储介质及程序，其中，所述方法由电子设备执行，所述方法包括：对第一图像进行编码，得到第一特征图，并对第二图像进行编码，得到第二特征图（S11）；融合第一特征图和第二特征图，得到速度场（S12）；分别对第一特征图和第二特征图进行解码，得到新的第一特征图和新的第二特征图（S13）；利用已得到的速度场，对新的第一特征图和新的第二特征图进行融合，再次得到速度场（S14）；以及基于多次融合得到的速度场，生成用于配准第一图像和第二图像的配准参数（S15）。如此，能够提高图像配准精度，可以应用于对医学图像进行配准，以提高医学图像的配准精度。

Description

图像配准方法、装置、电子设备、存储介质及程序
相关申请的交叉引用
本专利申请要求2021年03月26日提交的中国专利申请号为202110325843.9、申请人为上海商汤智能科技有限公司，申请名称为“图像配准方法及相关装置、电子设备、存储介质”的优先权，该申请的全文以引用的方式并入本申请中。
技术领域
本公开涉及图像处理技术领域，特别是涉及一种图像配准方法、装置、电子设备、存储介质及程序。
背景技术
图像配准是图像处理研究领域中的重要一环，其目的在于比较或融合针对同一对象在不同条件下获取的图像。同时图像配准在计算机视觉、医学图像处理、遥感等诸多领域得到了广泛的应用。
发明内容
本公开实施例提供一种图像配准方法、装置、电子设备、存储介质及程序。
本公开实施例提供了一种图像配准方法，所述方法由电子设备执行，所述方法包括：
对第一图像进行编码，得到第一特征图，并对第二图像进行编码，得到第二特征图；
融合所述第一特征图和所述第二特征图，得到速度场；
分别对所述第一特征图和所述第二特征图进行解码，得到新的第一特征图和新的第二特征图；其中，所述新的第一特征图的分辨率不同于本次解码前的第一特征图的分辨率，且所述新的第二特征图的分辨率不同于本次解码前的第二特征图的分辨率；
利用已得到的所述速度场，对所述新的第一特征图和所述新的第二特征图进行融合，再次得到速度场；以及
基于多次所述融合得到的速度场，生成用于配准所述第一图像和所述第二图像的配准参数。
如此，通过在多个阶段分别融合不同分辨率的特征图，能够得到不同尺度的速度场，从而能够基于不同尺度的速度场，提高配准参数的精度，进而有利于提高配准精度。此外，由于在多个阶段分别得到不同尺度的速度场，从而能够有利于得到用于将第一图像正向地配准至第二图像的配准参数，也能够有利于得到用于将第二图像反向地配准至第一图像的配准参数，进而能够有利于满足微分同胚。
在本公开的一些实施例中，所述对第一图像进行编码，得到第一特征图，或者，所述对第二图像进行编码，得到第二特征图，包括：对图像进行特征提取，得到多个通道特征图；基于各个所述通道特征图在所述多个通道特征图中的重要程度，得到对应所述通道特征图的注意力权重；以及分别利用每一所述通道特征图的注意力权重对对应所述通道特征图进行加权处理，得到所述图像的特征图；其中，在所述图像为第一图像的情况下，通过上述步骤得到的所述特征图为所述第一特征图；或者，在所述图像为所述第二图像的情况下，通过上述步骤得到的所述特征图为所述第二特征图。如此，通过各个通道特征图在多个通道特征图中的重要程度而得到的注意力权重，对对应通道特征图进行加权处理，能够有利于弱化多个通道特征图中表现较强的特征图，或者强化多个通道特征图中表现较弱的特征图，从而能够有利于使不同模态的图像经编码后，得到相近的特征图，进而能够有利于满足多模态图像的配准，拓宽适用范围。
在本公开的一些实施例中，在所述对第一图像进行编码，得到第一特征图，并对第二图像进行编码，得到第二特征图之前，所述方法还包括：获取待配准的多个图像；以及将所述多个图像中的一个图像作为所述第一图像，并分别将剩余的至少一个图像作为所述第二图像。如此，在满足“微分同胚”的基础上，能够使两个图像仅需一次配准流程，即可实现正向和反向的配准，故能够有利于减少配准次数。同时在满足“多模态配准”的基础上，能够仅需少量配准次数即可实现多模态图像配准。
在本公开的一些实施例中，所述多个图像均为医学图像，且所述多个图像满足以下任一条件：所述多个图像是由不同种类的医疗设备扫描得到的；所述多个图像是由同一种医疗设备在不同扫描时间扫描得到的。如此，能够有利于实现多模态的医学图像配准。
在本公开的一些实施例中，所述利用已得到的所述速度场，对所述新的第一特征图和所述新的第二特征图进行融合，再次得到速度场，包括：对已得到的所述速度场进行转换，得到位移场；利用所述位移场对所述新的第一特征图进行变形，得到变形特征图；以及融合所述变形特征图和所述新的第 二特征图，再次得到速度场。如此，能够通过已得到的速度场，得到位移场，并利用经位移场变形得到的变形特征图和新的第二特征图的再次融合，得到速度场，从而能够有利于在已得到的速度场的基础上，再次得到速度场，进而能够有利于通过“多阶段”优化速度场，有利于提高速度场的精度。
在本公开的一些实施例中，所述融合所述第一特征图和所述第二特征图，得到速度场，包括：将所述第一特征图和所述第二特征图进行拼接，得到拼接特征图；以及对所述拼接特征图进行特征提取，得到所述速度场。如此，能够有利于简化获取速度场的过程，提高获取速度场的效率。
在本公开的一些实施例中，在所述基于多次融合得到的速度场，生成用于配准所述第一图像和所述第二图像的配准参数之前，所述方法还包括：在满足预设条件的情况下，基于最新得到的第一特征图和第二特征图，重新执行分别对所述第一特征图和所述第二特征图进行解码，得到新的第一特征图和新的第二特征图的步骤以及后续步骤。如此，能够有利于在特征图分辨率由低到高的过程中，得到尺度由小到大的速度场，从而能够有利于实现“由粗到细”多阶段的配准，进而能够有利于提高配准精度。
在本公开的一些实施例中，所述预设条件包括以下任一者：执行所述解码的次数小于预设阈值，最近一次执行所述解码得到的所述第一特征图或所述第二特征图的分辨率小于预设分辨率；和/或，所述新的第一特征图的分辨率大于本次解码前的第一特征图的分辨率，且所述新的第二特征图的分辨率大于本次解码前的第二特征图的分辨率。如此，能够有利于以解码次数或特征图分辨率为参考维度，不断迭代优化速度场。
在本公开的一些实施例中，在所述基于多次融合得到的速度场，生成用于配准所述第一图像和所述第二图像的配准参数之后，所述方法还包括以下至少一者：利用所述配准参数对所述第一图像进行处理，得到所述第一图像的配准图像；利用所述配准参数对所述第二图像进行处理，得到所述第二图像的配准图像；利用所述配准参数对所述第一图像中至少一个第一像素点进行处理，基于处理后的所述至少一个第一像素点，得到所述第二图像中分别与所述至少一个第一像素点对应的至少一个第二像素点；或者利用所述配准参数对所述第二图像中至少一个第二像素点进行处理，基于处理后的所述至少一个第二像素点，得到所述第一图像中分别与所述至少一个第二像素点对应的至少一个第一像素点。如此，能够实现第一图像和第二图像中至少一个像素点的配准，有利于从局部层面实现图像配准。
在本公开的一些实施例中，所述对第一图像进行编码，得到第一特征图，并对第二图像进行编码，得到第二特征图，包括：利用图像配准模型的第一编码子网络对第一图像进行编码，得到第一特征图，并利用所述图像配准模型的第二编码子网络对第二图像进行编码，得到第二特征图；所述融合所述第一特征图和所述第二特征图，得到速度场，包括：利用所述图像配准模型的速度场子网络融合所述第一特征图和所述第二特征图，得到速度场；以及所述分别对所述第一特征图和所述第二特征图进行解码，得到新的第一特征图和新的第二特征图，包括：利用所述图像配准模型的第一解码子网络对所述第一特征图进行解码，得到新的第一特征图，并利用所述图像配准模型的第二解码子网络对所述第二特征图进行解码，得到新的第二特征图。如此，能够利用图像配准模型实现编码、融合和解码等，从而能够有利于提高图像配准的效率。
以下装置、电子设备等的效果描述参见上述图像配准方法的说明。
本公开实施例提供了一种图像配准装置，包括：
图像编码模块，配置为对第一图像进行编码，得到第一特征图，并对第二图像进行编码，得到第二特征图；
第一融合模块，配置为融合所述第一特征图和所述第二特征图，得到速度场；
图像解码模块，配置为分别对所述第一特征图和所述第二特征图进行解码，得到新的第一特征图和新的第二特征图；其中，所述新的第一特征图的分辨率大于本次解码前的第一特征图的分辨率，且所述新的第二特征图的分辨率大于本次解码前的第二特征图的分辨率；
第二融合模块，配置为利用已得到的所述速度场，对所述新的第一特征图和所述新的第二特征图进行融合，再次得到速度场；
参数获取模块，配置为基于多次所述融合得到的速度场，生成用于配准所述第一图像和所述第二图像的配准参数。
在本公开的一些实施例中，图像编码模块包括：特征提取子模块，配置为对图像进行特征提取，得到多个通道特征图；权重获取子模块，配置为基于各个所述通道特征图在所述多个通道特征图中的重要程度，得到对应所述通道特征图的注意力权重；特征图加权子模块，配置为分别利用每一所述通道特征图的注意力权重对对应所述通道特征图进行加权处理，得到所述图像的特征图；其中，在所述图像为所述第一图像的情况下，通过上述步骤得到的所述特征图为所述第一特征图；或者，在所述图 像为所述第二图像的情况下，通过上述步骤得到的所述特征图为所述第二特征图。
在本公开的一些实施例中，图像配置装置包括：图像获取模块，配置为获取待配准的多个图像；以及将所述多个图像中的一个图像作为所述第一图像，并分别将剩余的至少一个图像作为所述第二图像。
在本公开的一些实施例中，所述多个图像均为医学图像，且所述多个图像满足以下任一条件：所述多个图像是由不同种类的医疗设备扫描得到的；所述多个图像是由同一种医疗设备在不同扫描时间扫描得到的。
在本公开的一些实施例中，第二融合模块包括：转换子模块，配置为对已得到的所述速度场进行转换，得到位移场；变形子模块，配置为利用所述位移场对所述新的第一特征图进行变形，得到变形特征图；融合子模块，配置为融合所述变形特征图和所述新的第二特征图，再次得到速度场。
在本公开的一些实施例中，第一融合模块包括：拼接子模块，配置为将所述第一特征图和所述第二特征图进行拼接，得到拼接特征图；提取子模块，配置为对所述拼接特征图进行特征提取，得到所述速度场。
在本公开的一些实施例中，第二融合模块，还配置为在满足预设条件的情况下，基于最新得到的第一特征图和第二特征图，重新执行所述分别对所述第一特征图和所述第二特征图进行解码，得到新的第一特征图和新的第二特征图的步骤以及后续步骤。
在本公开的一些实施例中，预设条件包括以下任一者：执行所述解码的次数小于预设阈值，最近一次执行所述解码得到的所述第一特征图或所述第二特征图的分辨率小于预设分辨率；和/或，所述新的第一特征图的分辨率大于本次解码前的第一特征图的分辨率，且所述新的第二特征图的分辨率大于本次解码前的第二特征图的分辨率。
在本公开的一些实施例中，图像配置装置包括：图像处理模块，配置为执行以下至少一者：利用所述配准参数对所述第一图像进行处理，得到所述第一图像的配准图像；利用所述配准参数对所述第二图像进行处理，得到所述第二图像的配准图像；利用所述配准参数对所述第一图像中至少一个第一像素点进行处理，基于处理后的所述至少一个第一像素点，得到所述第二图像中分别与所述至少一个第一像素点对应的至少一个第二像素点；或者利用所述配准参数对所述第二图像中至少一个第二像素点进行处理，基于处理后的所述至少一个第二像素点，得到所述第一图像中分别与所述至少一个第二像素点对应的至少一个第一像素点。
在本公开的一些实施例中，图像编码模块，还配置为利用图像配准模型的第一编码子网络对第一图像进行编码，得到第一特征图，并利用所述图像配准模型的第二编码子网络对第二图像进行编码，得到第二特征图；第一融合模块，还配置为利用所述图像配准模型的速度场子网络融合所述第一特征图和所述第二特征图，得到速度场；图像解码模块，还配置为利用所述图像配准模型的第一解码子网络对所述第一特征图进行解码，得到新的第一特征图，并利用所述图像配准模型的第二解码子网络对所述第二特征图进行解码，得到新的第二特征图。
本公开实施例还提供了一种电子设备，包括相互耦接的存储器和处理器，处理器配置为执行存储器中存储的程序指令，以实现上述任一实施例所述的图像配准方法。
本公开实施例还提供了一种计算机可读存储介质，其上存储有程序指令，程序指令被处理器执行时实现上述任一实施例所述的图像配准方法。
本公开实施例还提供一种计算机程序，所述计算机程序包括计算机可读代码，在所述计算机可读代码在电子设备中运行的情况下，所述电子设备的处理器执行上述任一实施例所述的图像配准方法。
本公开实施例提供的一种图像配准方法、装置、电子设备、存储介质及程序，首先，通过对第一图像进行编码，得到第一特征图，并对第二图像进行编码，得到第二特征图；其次，融合第一特征图和第二特征图，得到速度场，并分别对第一特征图和第二特征图进行解码，得到新的第一特征图和第二特征图，且新的第一特征图的分辨率不同于本次解码前的第一特征图的分辨率，新的第二特征图的分辨率不同于本次解码前的第二特征图的分辨率；最后，进而利用已得到速度场，对新的第一特征图和新的第二特征图进行融合，再次得到速度场，以及基于多次融合得到的速度场，生成用于配准第一图像和第二图像的配准参数。如此，通过在多个阶段分别融合不同分辨率的特征图，能够得到不同尺度的速度场，从而能够基于不同尺度的速度场，提高配准参数的精度，进而有利于提高配准精度。此外，由于在多个阶段分别得到不同尺度的速度场，从而能够有利于得到用于将第一图像正向地配准至第二图像的配准参数，也能够有利于得到用于将第二图像反向地配准至第一图像的配准参数，进而能够有利于满足微分同胚。
为使本公开实施例的上述目的、特征和优点能更明显易懂，下文特举较佳实施例，并配合所附附图，作详细说明如下。
附图说明
为了更清楚地说明本公开实施例的技术方案，下面将对实施例中所需要使用的附图作简单地介绍，此处的附图被并入说明书中并构成本说明书中的一部分，这些附图示出了符合本公开的实施例，并与说明书一起用于说明本公开实施例的技术方案。应当理解，以下附图仅示出了本公开的某些实施例，因此不应被看作是对范围的限定，对于本领域普通技术人员来讲，在不付出创造性劳动的前提下，还可以根据这些附图获得其他相关的附图。
图1是本公开提供的图像配准方法一实施例的流程示意图；
图2是本公开提供的图像配准模型一实施例的框架示意图；
图3是本公开提供的利用速度场进行图像配准一实施例的状态示意图；
图4是本公开提供的利用速度场进行图像配准另一实施例的状态示意图；
图5为可以应用本公开实施例的图像配准方法的一种系统架构示意图；
图6是本公开提供的图像配准方法另一实施例的流程示意图；
图7是本公开提供的图像配准方法一实施例的状态示意图；
图8是本公开提供的域注意力块一实施例的框架示意图；
图9是本公开提供的图像配准模型的训练方法一实施例的流程示意图；
图10是本公开提供的图像配准装置100一实施例的框架示意图；
图11是本公开提供的电子设备110一实施例的框架示意图；
图12是本公开提供的计算机可读存储介质120一实施例的框架示意图。
具体实施方式
下面结合说明书附图，对本公开实施例的方案进行详细说明。
以下描述中，为了说明而不是为了限定，提出了诸如特定系统结构、接口、技术之类的具体细节，以便透彻理解本公开。
本文中术语“系统”和“网络”在本文中常被可互换使用。本文中术语“和/或”，仅仅是一种描述关联对象的关联关系，表示可以存在三种关系，例如，A和/或B，可以表示：单独存在A，同时存在A和B，单独存在B这三种情况。另外，本文中字符“/”，一般表示前后关联对象是一种“或”的关系。此外，本文中的“多”表示两个或者多于两个。本公开实施例中的多个或者多种可以分别指的是至少两个或者至少两种。
请参阅图1，图1是本公开提供的图像配准方法一实施例的流程示意图。如图1所示，可以包括如下步骤：
步骤S11：对第一图像进行编码，得到第一特征图，并对第二图像进行编码，得到第二特征图。
在本公开的一些实施例中，第一图像和第二图像为同一对象在不同条件下的图像。以第一图像和第二图像均是医学图像为例，第一图像和第二图像可以是由不同种类的医疗设备对同一对象(如，同一患者的腹部、胸部等)扫描得到的。例如，第一图像和第二图像分别是对患者的腹部扫描得到的计算机断层扫描(Computed Tomography，CT)图像、核磁共振(Magnetic Resonance，MR)图像；或者，第一图像和第二图像也可以是由同一种医疗设备在不同扫描时间扫描得到的。在本公开的一些实施例中，扫描时间可以对应于一次扫描过程中的不同造影时长。例如，第一图像和第二图像分别是对患者肝部进行CT或MR扫描得到的平扫期图像、动脉期图像、门脉期图像、延迟期图像中的任意两者；此外，扫描时间也可以对应于不同次的扫描。例如，第一图像是在一月份扫描得到的，而第二图像是在二月份扫描得到的。在第一图像和第二图像为医学图像之外的其他类型图像的情况下，可以以此类推，在此不再一一举例。
在本公开的一些实施例中，可以分别对第一图像和第二图像进行特征提取，得到第一图像的多个通道特征图，并得到第二图像的多个通道特征图。在本公开的一些实施例中，第一图像的多个通道特征图中通常存在至少一个表现较强(或表现较弱)的通道特征图，能够反映第一图像的风格，而与之类似地，第二图像的多个通道特征图中通常也存在至少一个表现较强(或表现较弱)的通道特征图，能够反映第二图像的风格。仍以医学图像为例，例如，第一图像为CT图像，则CT图像中至少存在一个表现较强的通道特征图，能够反映CT图像的灰度特征，或者，CT图像中至少存在一个表现较弱的通道特征图，能够反映CT图像的纹理特征。或者，第二图像为MR图像，则MR图像中至少存 在一个表现较强的通道特征图，能够反映MR图像的纹理特征，或者，MR图像中至少存在一个表现较弱的通道特征图，能够反映MR图像的灰度特征，其他情况可以以此类推，在此不再一一举例。在此情形下，可以根据各个通道特征图在多个通道特征图中的重要程度，得到对应通道特征图的注意力权重，并利用该注意力权重对对应的通道特征图进行加权处理，以得到第一图像的第一特征图，以及第二图像的第二特征图。在本公开的一些实施例中，可以为表现较强的通道特征图赋予较小的注意力权重，或者，为表现较弱的通道特征图赋予较大的注意力权重，或者，同时为表现较强的通道特征图赋予较小的注意力权重，并为表现较弱的通道特征图赋予较大的注意力权重，从而使得第一特征图和第二特征图相近。例如，第一图像为CT图像，且其中n1个通道特征图反映了CT图像的纹理特征，剩余n2个通道特征图反映了CT图像的灰度特征，则可以为上述n1个通道特征图赋予较大的注意力权重，为上述n2个通道特征图赋予较小的注意力权重，从而可以弱化CT图像的灰度特征，并强化CT图像的纹理特征；而对于第二图像为MR图像的情况，在其中m1个通道特征图反映了MR图像的纹理特征，剩余m2个通道特征图反映了MR图像的灰度特征的情况下，可以对上述m1个通道特征图赋予较小的注意力权重，并为上述m2个通道特征图赋予较大的注意力权重，从而可以弱化MR图像的纹理特征，并强化MR图像的灰度特征，进而可以使得不同模态的CT图像和MR图像最终编码得到的第一特征图和第二特征图相近，进而能够有利于满足多模态图像的配准，拓宽适用范围。此外，当第一图像和第二图像为其他图像时，可以以此类推，在此不再一一举例。
在本公开的一些实施例中，为了提高图像配准效率，可以预先训练一图像配准模型，且该图像配准模型包括用于编码的第一编码子网络和第二编码子网络，从而可以利用第一编码子网络对第一图像进行编码，得到第一特征图，并利用第二编码子网络对第二图像进行编码，得到第二特征图。图像配准模型的训练过程可以参阅本公开的图像配准模型的训练方法实施例中的步骤。
在本公开的一些实施例中，请结合参阅图2，图2是本公开提供的图像配准模型一实施例的框架示意图，如图2所示，第一编码子网络可以包括至少一个顺序连接的特征提取层，每个特征提取层能够对应提取到不同分辨率的特征图，并将最后一个特征提取层提取得到的特征图，作为第一图像的第一特征图，类似地，第二编码子网络也可以包括至少一个顺序连接的特征提取层，每个特征提取层能够对应提取到不同分辨率的特征图，并将最后一个特征提取层提取得到的特征图，作为第二特征图。在本公开的一些实施例中，特征提取层至少可以包括卷积层。
在本公开的一些实施例中，为了使第一编码子网络和第二编码子网络能够适用于不同模态的图像，第一编码子网络中相邻特征提取层之间还可以设有第一域注意力块，且第二编码子网络中相邻特征提取层之间还设有第二域注意力块，第一域注意力块和第二域注意力块均用于将特征提取层提取得到的特征图进行域转换，以使第一特征图和第二特征图相近。
在本公开的一些实施例中，在本公开实施例步骤S11之前，还可以先对第一图像和第二图像进行线性配准。线性配准可以包括但不限于：刚体配准、仿射配准，在此不做限定。在此之后，再利用线性配准之后的第一图像和第二图像，执行本公开实施例中步骤。通过上述方式，能够提高配准的准确性。在本公开的一些实施例中，在第一图像的对象和第二图像中的对象各自的相对位置不同的情况下(如，在对同一对象的胸部进行CT扫描时，该对象可能在扫描过程中移动)，通过先执行线性配准，能够大大提高配准的准确性。
步骤S12：融合第一特征图和第二特征图，得到速度场。
本公开实施例中，速度场可以是由每一时刻、每一点上的速度矢量组成的物理场。在本公开的一些实施例中，以线性插值为例，速度场中的每一元素表示第一图像中与该元素对应的至少一个像素点的中心像素点在变形时的速度矢量，其他像素点在变形时的速度矢量可以通过插值计算得到，在计算其他像素点的速度矢量时，可以获取距离该像素点最近的若干个中心像素点的速度矢量，并获取与各个中心像素点的速度矢量对应的权重，从而利用获取到的权重对对应的中心像素点的速度矢量进行加权处理，得到该像素点的速度矢量。
在本公开的一些实施例中，与中心像素点的速度矢量对应的权重和该像素点至对应中心像素点的距离成反比，即距离越小，权重越大，距离越大，权重越小。例如，第一图像是分辨率为480*480的图像，速度场为48*48的物理场，则该速度场中每一元素对应于第一图像的10*10区域的中心像素点在变形时的速度矢量，其他像素点在变形时的速度矢量可以通过上述插值计算得到；或者，第一图像是分辨率为720*720*720的图像，速度场为72*72*72的物理场，则该速度场中每一元素对应于第一图像的10*10*10区域的中心像素点在变形时的速度矢量，其他像素点在变形时的速度矢量可以通过上述插值计算得到。其他情况可以以此类推，在此不再一一举例。
在本公开的一些实施例中，可以将第一特征图和第二特征图进行拼接，得到拼接特征图，并对拼 接特征图进行特征提取，得到速度场。在本公开的一些实施例中，可以将第一特征图和第二特征图在通道维度进行拼接，从而得到通道数翻倍且分辨率不变的拼接特征图。例如，第一特征图和第二特征图均是分辨率为W*H且通道数为C的特征图，则将第一特征图和第二特征图拼接，可以得到通道数为2C，且分辨率仍为W*H的特征图。此外，对拼接特征图进行特征提取，可以使得拼接特征图的通道数减半。上述方式，通过将第一特征图和第二特征图进行拼接，得到拼接特征图，并对拼接特征图进行特征提取，得到速度场，能够有利于简化获取速度场的过程，提高获取速度场的效率。
在本公开的一些实施例中，为了提高配准效率，可以预先训练一图像配准模型，且该图像配准模型包括速度场子网络，从而可以利用图像配准模型的速度场子网络融合第一特征图和第二特征图，得到速度场。在本公开的一些实施例中，速度场子网络可以包括顺序连接的拼接处理层和特征提取层，其中，拼接处理层用于将第一特征图和第二特征图进行拼接，得到拼接特征图，而特征提取层用于对拼接特征图进行特征提取，得到速度场。此外，特征提取层至少可以包括卷积层。
步骤S13：分别对第一特征图和第二特征图进行解码，得到新的第一特征图和新的第二特征图。
本公开实施例中，新的第一特征图的分辨率不同于本次解码前的第一特征图的分辨率，且新的第二特征图的分辨率不同于本次解码前的第二特征图的分辨率。例如，新的第一特征图的分辨率可以大于本次解码前的第一特征图的分辨率，且新的第二特征图的分辨率可以大于本次解码前的第二特征图的分辨率。
在本公开的一些实施例中，为了提高配准效率，可以预先训练一图像配准模型，且该图像配准模型包括第一解码子网络，第一解码子网络用于对第一特征图进行解码，此外，该图像配准模型还包括第二解码子网络，第二解码子网络用于对第二特征图进行解码，从而可以利用图像配准模型的第一解码子网络对第一特征图进行解码，得到新的第一特征图，并利用图像配准模型的第二解码子网络对第二特征图进行解码，得到新的第二特征图。在本公开的一些实施例中，第一解码子网络可以包括至少一个顺序连接的解码处理层。解码处理层可以包括以下任一者：反卷积层、上采样层，在此不做限定。
步骤S14：利用已得到的速度场，对新的第一特征图和新的第二特征图进行融合，再次得到速度场。
在本公开的一些实施例中，本次得到的速度场是基于新的第一特征图和新的第二特征图得到的，而新的第一特征图的分辨率不同于本次解码前的第一特征图的分辨率，新的第二特征图的分辨率不同于本次解码前的第二特征图的分辨率，故本次得到的速度场不同于前一次得到的速度场。在本公开的一些实施例中，每次解码之后，特征图的分辨率均会有所提高，导致速度场的尺寸也有所增大，即在特征图分辨率由低到高的过程中，可以得到尺度由小到大的速度场。
在本公开的一些实施例中，可以对已得到的速度场进行转换，得到位移场，并利用位移场对新的第一特征图进行变形，得到变形特征图，从而可以融合变形特征图和新的第二特征图，再次得到速度场。上述方式，能够通过已得到的速度场，得到位移场，并利用经位移场变形得到的变形特征图和新的第二特征图的再次融合，得到速度场，从而能够有利于在已得到的速度场的基础上，再次得到速度场，进而能够有利于通过“多阶段”优化速度场，有利于提高速度场的精度。
在本公开的一些实施例中，可以将变形特征图和新的第二特征图进行拼接，对应得到一拼接特征图，并对该拼接特征图进行特征提取，从而再次得到速度场。在本公开的一些实施例中，融合变形特征图和新的第二特征图的实施方式可以参阅前述关于融合第一特征图和第二特征图的描述。
在本公开的一些实施例中，可以分别将已得到的速度场进行转换，得到与速度场对应的位移场，再将与已得到的速度场对应的位移场进行融合(如，在通道维度进行堆叠)，得到用于对新的第一特征图进行变形的位移场。在本公开的一些实施例中，可以基于差分方式，将速度场迭代预设次数，得到与速度场对应的位移场。预设次数至少为1次，例如，1次、2次、3次或4次等，在此不做限定。为了便于描述，速度场可以记为VF，与速度场VF对应的位移场可以记为DF，则速度场与位移场之间可以用常微分方程表示为公式(1)：
Figure PCTCN2021114524-appb-000001
上述公式(1)中，t表示时间，故可以记最小时间单位为dt，则可以得到速度场VF对应该最小时间单位的位移VFdt，为了便于描述，记n个最小时间单位下的位移为DF (n)，则
Figure PCTCN2021114524-appb-000002
从而可以根据位移的复合规则，得到
Figure PCTCN2021114524-appb-000003
其中，о表示对后者应用前者变换，进而迭代n次即可得到速度场VF对应的位移场DF。例如，速度场VF对应128(即2的7次方)个最小时间单位，则需迭代7次。其他情况可以以此类推，在此不再一一举 例。
此外，可以将已得到的速度场进行尺度归一化，再通过以下公式(2)，对尺度归一化后已得到的速度场进行转换，得到位移场：
Figure PCTCN2021114524-appb-000004
上述公式(2)中，VF 1、VF 2表示尺度归一化后已得到的速度场，f()表示将速度场转换为位移场的转换函数，可以参阅前述描述。
在本公开的一些实施例中，请结合参阅图2，图像配准模型还可以包括变形层，用于对已得到的速度场进行转换，得到位移场，并利用位移场对新的第一特征图进行变形，得到变形特征图。变形层执行内容可以参阅前述描述。
在本公开的一些实施例中，在再次得到速度场之后，可以检测是否满足预设条件，并在满足预设条件的情况下，可以基于最新得到的第一特征图和第二特征图，重新执行上述步骤S13以及后续步骤，且新的第一特征图的分辨率大于本次解码前的第一特征图的分辨率，新的第二特征图的分辨率大于本次解码前的第二特征图的分辨率。上述方式，能够有利于在特征图分辨率由低到高的过程中，得到尺度由小到大的速度场，从而能够有利于实现“由粗到细”多阶段的配准，进而能够有利于提高配准精度。
请继续结合参阅图2，在第一阶段：利用第一编码子网络对第一图像01进行编码，可以得到第一特征图01_1，利用第二编码子网络对第二图像02进行编码，可以得到第二特征图02_1，利用速度场子网络1融合第一特征图01_1和第二特征图02_1，可以得到速度场VF 1。在第二阶段：利用解码处理层11对第一特征图01_1进行解码，得到新的第一特征图01_2，并利用解码处理层21对第二特征图02_1进行解码，得到新的第二特征图02_2，利用变形层1对速度场VF 1进行转换，得到位移场f(VF 1)，利用位移场f(VF 1)对新的第一特征图01_2进行变形，得到变形特征图01_2’，利用速度场子网络2将变形特征图01_2’和新的第二特征图02_2进行融合，可以得到速度场VF 2，此时最新得到的第一特征图即为第一特征图01_2，而最新得到的第二特征图即为第二特征图02_2。在第三阶段：利用解码处理层12对第一特征图01_2进行解码，得到新的第一特征图01_3，并利用解码处理层22对第二特征图02_2进行解码，得到新的第二特征图02_3，利用变形层2对速度场VF 2进行转换，得到位移场f(VF 2)，利用位移场f(VF 2)对新的第一特征图01_3进行变形，得到变形特征图01_3’，利用速度场子网络3将变形特征图01_3’和新的第二特征图02_3进行融合，得到速度场VF 3。故此，通过上述三个阶段，可以得到速度场VF 1、VF 2和VF 3。在上述各个阶段中，f表示将速度场变换为位移场的转换函数。此外，在图像配准模型中的速度场子网络多于(或少于)图2所示的图像配准模型的情况下，可以以此类推，在此不再一一举例。
在本公开的一些实施例中，各个速度场子网络的网络结构可以相同，以图2为例，速度场子网络1、速度场子网络2和速度场子网络3可以均包括一个拼接处理层和一个卷积层。此外，也可以根据神经网络的实际设计情况，将各个速度场子网络设置为具有不同的网络结构，在此不做限定。
在本公开的一些实施例中，预设条件包括以下任一者：执行解码的次数小于预设阈值，最近一次执行解码得到的第一特征图或第二特征图的分辨率小于预设分辨率；新的第一特征图的分辨率大于本次解码前的第一特征图的分辨率，且新的第二特征图的分辨率大于本次解码前的第二特征图的分辨率。在本公开的一些实施例中，当预设条件包括：执行解码的次数小于预设阈值时，预设阈值可以设置为至少为2次，例如，2次、3次或4次等，在此不做限定。
在本公开的一些实施例中，预设条件包括：最近一次解码得到的第一特征图或第二特征图的分辨率小于预设分辨率时，预设分辨率可以设置为第一图像或第二图像的原始分辨率，此外预设分辨率也可以小于原始分辨率，或者大于原始分辨率，在此不做限定。
在本公开的一些实施例中，请结合参阅图2，在由图像配准模型的第一解码子网络执行对第一特征图的解码，并由图像配准模型的第二解码子网络执行对第二特征图的解码的情况下，预设条件可以包括以下任一情况：执行解码的为第一解码子网络中最后一个解码处理层，或，执行解码的为第二解码子网络的最后一个解码处理层。
在本公开的一些实施例中，在检测不满足预设条件的情况下，可以执行本公开实施例中下述步骤S15，以基于多次融合得到的速度场，生成用于配准第一图像和第二图像的配准参数。
步骤S15：基于多次融合得到的速度场，生成用于配准第一图像和第二图像的配准参数。
在本公开的一些实施例中，可以对多次融合得到的速度场进行转换，得到位移场，从而可以将该 位移场作为用于配准第一图像和第二图像的配准参数。
在本公开的一些实施例中，“融合得到的速度场”可以是通过融合特征图所得到的速度场，可以参阅前述相关描述。在本公开的一些实施例中，可以对历次融合所得到的速度场进行转换，得到位移场，在此基础上，可以将该位移场作为用于配准第一图像和第二图像的配准参数；或者，也可以在历次融合得到的速度场中选择部分速度场，并对所选择的速度场进行转换，得到位移场，从而可以将该位移场作为用于配准第一图像和第二图像的配准参数，可以根据实际应用需要进行设置。例如，在对配准参数的准确性要求较高的情况下，可以基于历次融合得到的速度场，得到配准参数；而在配准参数的准确性要求相对宽松的情况下，可以在历次融合得到的速度场中选择部分速度场，并基于选择的速度场，得到配准参数。
请参阅图3，图3是本公开提供的利用速度场进行图像配准一实施例的状态示意图。在本公开的一些实施例中，图3是“单阶段”的图像配准状态示意图。如图3所示，公式
Figure PCTCN2021114524-appb-000005
表示利用速度场V转换得到的位移场f(V)对原始图像x，即301(即图3左侧所示的同心圆图像)进行变形，得到变形图像(即图3中间所示的变形图像)，即302，而公式
Figure PCTCN2021114524-appb-000006
表示对速度场V取反后转换得到的位移场f(-V)对变形图像x(即图3中间所示的变形图像)进行变形，仍然能够还原得到原始图像(即同心圆图像)，即303，而式子
Figure PCTCN2021114524-appb-000007
表示对速度场V转换得到的位移场f(V)取反得到新的位移场-f(V)，利用新的位移场对变形图像x(即图3中间所示的变形图像)进行变形，不能得到原始图像(即同心圆图像)，即得到304。由此可见，在“单阶段”的图像配准中，直接基于位移场的图像配准，无法满足微分同胚，而基于速度场的图像配准，能够满足微分同胚。故此，基于融合得到的速度场，既能够得到用于将第一图像配准至第二图像的正向配准参数，也能够得到用于将第二图像配准至第一图像的配准参数。
请继续参阅图4，图4是利用速度场进行图像配准另一实施例的状态示意图。在本公开的一些实施例中，图4是“多阶段”的图像配准状态示意图。如图4所示，图4左上角同心圆图像为原始图像，利用尺度由小变大的速度场之后，分别得到位于左上角原始图像同一行右侧的四个图像，如前所述，尺度较小的速度场中每一元素对应于图像的像素区域较大，而尺度较大的速度场中每一元素对应于图像的像素区域较小，故原始图像应用尺度较小的速度场，能够在原始图像整体层面进行变形，即变形尺度较“粗放”，而应用尺度较大的速度场，能够在原始图像局部层面进行变形，即变形尺度较“细致”，即图4第一行“正向”所示的“由粗到细”的变化过程；与之相反，图4右下角图像利用尺度由大变小且取反后的速度场之后，可以得到位于图4右下角图像同一行左侧的四个图像，即图4第三行“反向”所示的变化过程。在此基础上，将同一列中分别位于正向行和反向行的图像进行差异分析，可以得到图4中第二行的图像，显然，图4中间行各个图像的灰度值基本保持不变。故此，在“多阶段”的图像配准中，基于速度场的图像配准，仍然能够满足微分同胚。
在本公开的一些实施例中，可以将历次融合得到的速度场进行尺度归一化，从而可以利用公式(3)，通过尺度归一化后多次融合得到的速度场，生成位移场，并将该位移场作为用于将第一图像配准至第二图像的正向配准参数：
Figure PCTCN2021114524-appb-000008
上述公式(3)中，DF forward表示正向配准参数，VF 1、VF 2、…、VF n分别表示尺度归一化后历次融合得到的速度场，f()表示用于将速度场转换为位移场的函数，可以参阅前述描述。
在本公开的一些实施例中，还可以将历次融合得到的速度场进行尺度归一化，并进行取反，从而可以利用公式(4)，通过尺度归一化取反后的多次融合得到的速度场，生成位移场，并将还位移场作为用户将第二图像配准至第一图像的反向配准参数：
Figure PCTCN2021114524-appb-000009
上式公式(4)中，DF backward表示反向配准参数，符号“—”表示取反操作。
在本公开的一些实施例中，也可以将历次融合得到的速度场的集合，作为用于配准第一图像和第二图像的配准参数，从而在需要将第一图像配准至第二图像的情况下，可以通过上述公式(3)，得到用于将第一图像配准至第二图像的正向配准参数，而在需要将第二图像配准至第一图像的情况下，可以通过上述公式(4)，得到用于将第二图像配准至第一图像的反向配准参数。
在本公开的一些实施例中，在得到配准参数之后，可以利用配准参数对第一图像进行处理，得到第一图像的配准图像。在本公开的一些实施例中，在得到上述正向配准参数之后，可以利用正向配准参数对第一图像进行处理，得到第一图像的配准图像。以第一图像和第二图像均为医学图像为例，通过正向配准参数可以实现不同种类设备扫描得到的图像(比如，CT图像、MR图像)间的配准，或 者实现同一扫描设备对应于不同扫描时间的图像(比如，对应于不同造影时长的平扫期图像、动脉期图像、门脉期图像、延迟期图像)间的配准。上述方式，能够实现第一图像和第二图像中全部像素点的配准，有利于从整体层面实现图像配准。
在本公开的一些实施例中，在得到配准参数之后，可以利用配准参数对第二图像进行处理，得到第二图像的配准图像。本公开的一些实施例，在得到上述反向配准参数之后，可以利用反向配准参数对第二图像进行处理，得到第二图像的配准图像。以第一图像和第二图像均为医学图像为例，通过反向配准参数可以实现不同种类设备扫描得到的图像(比如，CT图像、MR图像)间的配准，或者实现同一扫描设备对应于不同扫描时间的图像(如，对应于不同造影时长的平扫期图像、动脉期图像、门脉期图像、延迟期图像)间的配准。上述方式，能够实现第一图像和第二图像中全部像素点的配准，有利于从整体层面实现图像配准。
在本公开的一些实施例中，在得到配准参数之后，利用配准参数对第一图像中至少一个第一像素点进行处理，从而可以基于处理后的至少一个第一像素点，得到第二图像中分别与至少一个第一像素点对应的至少一个第二像素点。在本公开的一些实施例中，在得到上述正向配准参数之后，可以利用正向配准参数对第一图像中至少一个第一像素点进行处理，从而可以基于处理后的至少一个第一像素点，得到第二图像中分别与至少一个第一像素点对应的至少一个第二像素点。以第一图像和第二图像均为医学图像为例，通过正向配准参数可以实现不同种类设备扫描得到的图像(比如，CT图像、MR图像)像素点间的配准，或者实现同一扫描设备对应于不同扫描时间的图像(比如，对应于不同造影时长的平扫期图像、动脉期图像、门脉期图像、延迟期图像)像素点间的配准。上述方式，能够实现第一图像和第二图像中至少一个像素点的配准，有利于从局部层面实现图像配准。
在本公开的一些实施例中，在得到配准参数之后，还可以利用配准参数对第二图像中至少一个第二像素点进行处理，从而可以基于处理后的至少一个第二像素点，得到第一图像中分别与至少一个第二像素点对应的至少一个第一像素点。在本公开的一些实施例中，在得到上述反向配准参数之后，可以利用反向配准参数对第二图像中至少一个第二像素点进行处理，从而可以基于处理后的至少一个第二像素点，得到第一图像中分别与至少一个第二像素点对应的至少一个第一像素点。以第一图像和第二图像均为医学图像为例，通过反向配准参数可以实现不同种类设备扫描得到的图像(比如，CT图像、MR图像)像素点间的配准，或者实现同一扫描设备对应于不同扫描时间的图像(比如，对应于不同造影时长的平扫期图像、动脉期图像、门脉期图像、延迟期图像)像素点间的配准。上述方式，能够实现第一图像和第二图像中至少一个像素点的配准，有利于从局部层面实现图像配准。
上述方案，通过对第一图像进行编码，得到第一特征图，并对第二图像进行编码，得到第二特征图，从而融合第一特征图和第二特征图，得到速度场，并分别对第一特征图和第二特征图进行解码，得到新的第一特征图和第二特征图，且新的第一特征图的分辨率不同于本次解码前的第一特征图的分辨率，新的第二特征图的分辨率不同于本次解码前的第二特征图的分辨率，进而利用已得到速度场，对新的第一特征图和新的第二特征图进行融合，以再次得到速度场，并基于多次融合得到的速度场，生成用于配准第一图像和第二图像的配准参数。故此，通过在多个阶段分别融合不同分辨率的特征图，能够得到不同尺度的速度场，从而能够基于不同尺度的速度场，提高配准参数的精度，进而有利于提高配准精度。此外，由于在多个阶段分别得到不同尺度的速度场，从而能够有利于得到用于将第一图像正向地配准至第二图像的配准参数，也能够有利于得到用于将第二图像反向地配准至第一图像的配准参数，进而能够有利于满足微分同胚。
图5示出可以应用本公开实施例的增强现实场景下的展示方法的一种系统架构示意图；如图5所示，该系统架构中包括：图像获取终端501、网络502和图像配准终端503。为实现支撑一个示例性应用，图像获取终端501和图像配准终端503通过网络502建立通信连接，图像获取终端501通过网络502向图像配准终端503上报第一图像和第二图像，图像配准终端503首先对第一图像进行编码，得到第一特征图，并对第二图像进行编码，得到第二特征图；其次，融合第一特征图和第二特征图，得到速度场，并分别对第一特征图和第二特征图进行解码，得到新的第一特征图和新的第二特征图，其中，新的第一特征图的分辨率不同于本次解码前的第一特征图的分辨率，且新的第二特征图的分辨率不同于本次解码前的第二特征图的分辨率；最后，利用已得到的速度场，对新的第一特征图和新的第二特征图进行融合，再次得到速度场；以及基于多次融合得到的速度场，生成用于配准第一图像和第二图像的配准参数。同时图像配准终端503将配准参数上传至网络502，并通过网络502发送给图像获取终端501。
作为示例，图像获取终端501可以包括图像采集设备，图像配准终端503可以包括具有视觉信息处理能力的视觉处理设备或远程服务器。网络502可以采用有线或无线连接方式。其中，当图像配准 终端503为视觉处理设备时，图像获取终端501可以通过有线连接的方式与视觉处理设备通信连接，例如通过总线进行数据通信；当图像配准终端503为远程服务器时，图像获取终端501可以通过无线网络与远程服务器进行数据交互。
或者，在一些场景中，图像获取终端501可以是带有视频采集模组的视觉处理设备，可以是带有摄像头的主机。这时，本公开实施例的图像配准方法可以由图像获取终端501执行，上述系统架构可以不包含网络502和图像配准终端503。
请参阅图6，图6是本公开提供的图像配准方法另一实施例的流程示意图。如图6所示，可以包括如下步骤：
步骤S61：获取待配准的多个图像，将多个图像中的一个作为第一图像，并分别将剩余的至少一个图像作为第二图像。
在本公开的一些实施例中，多个图像均为医学图像，且多个图像满足以下任一条件：多个图像是由不同种类的医疗设备扫描得到的，或，多个图像是由同一种医疗设备在不同扫描时间扫描得到的。可以参阅前述公开实施例中的相关描述。
请结合参阅图7，图7是本公开提供的图像配准方法一实施例的状态示意图，如图7所示，多个图像包括：图像A、图像B、图像C、图像D，则可以将图像A作为第一图像，图像B、图像C、图像D分别作为第二图像。例如，图像A为门脉期图像、图像B为平扫期图像、图像C为动脉期图像、图像D为延迟期图像，则可以将门脉期图像作为第一图像，并分别将平扫期图像、动脉期图像、延迟期图像作为第二图像。其他情况可以以此类推，在此不再一一举例。
步骤S62：对图像进行特征提取，得到多个通道特征图。
本公开实施例中，在图像为第一图像的情况下，多个通道特征图为第一图像对应的通道特征图，在图像为第二图像的情况下，多个通道特征图为第二图像对应的通道特征图。在本公开的一些实施例中，可以分别对第一图像进行特征提取，得到第一图像的多个通道特征图，对第二图像进行特征提取，得到第二图像的多个通道特征图。此外，提取通道特征图的实施方式，可以参阅前述公开实施例中的相关描述。
步骤S63：基于各个通道特征图在多个通道特征图中的重要程度，得到对应通道特征图的注意力权重。
在本公开的一些实施例中，请结合参阅图2，如前述公开实施例所述，第一编码子网络中相邻特征提取层之间还可以设有第一域注意力块，且第二编码子网络中相邻特征提取层之间还设有第二域注意力块，则第一图像的多个通道特征图的注意力权重可以通过第一域注意力块得到，而第二图像的多个通道特征图的注意力权重可以通过第二域注意力块得到。第一域注意力块和第二域注意力块可以具有相同的网络结构，为了便于描述，本公开实施例中，将第一域注意力块和第二域注意力块统称为域注意力块。
在本公开的一些实施例中，请结合参阅图8，图8是本公开提供的域注意力块一实施例的框架示意图。如图8所示，域注意力块包括域适配模块和多个通道注意力模块，每个通道注意力模块分别获取对全体通道特征图的通道注意力表示，域适配模块用于对这些注意力表示进行加权处理，得到各个通道特征图的注意力权重。以通道特征图通道数为C，分辨率为H*W为例，在通道注意力模块有k个的情况下，每个通道注意力模块对C*H*W的通道特征图进行处理，可以得到C*1的通道注意力表示，从而可以分别得到第1个通道注意力表示、第2个通道注意力表示、……、第k个通道注意力表示，而域适配模块对C*H*W的通道特征图进行处理，可以得到k*1的权重组合，进而可以将各个通道注意力模块输出的C*1的通道注意力表示进行拼接，得到C*k的通道注意力表示，并将拼接后的C*k的通道注意力表示与k*1的权重组合进行点积运算(即图8中
Figure PCTCN2021114524-appb-000010
运算)，即可得到C*1的注意力权重，即C个通道特征图中每一通道特征图的注意力权重。在本公开的一些实施例中，域适配模块可以包括顺序连接的全局平均池化(Global Average Pooling，GAP)层、全连接(Fully Connected，FC)层和softmax。此外，通道注意力模块可以为SE(Sequeze and Excitation)block。如前述公开实施例所述，诸如CT图像、MR图像等不同模态图像的特征往往不尽相同，如CT图像往往灰度特征(如骨骼和软组织的边界)较为明显，而纹理特征(如软组织内部的精细结构)较为薄弱，MR图像往往灰度特征较为薄弱且纹理特征较为明显，而无论是CT图像，还是MR图像，通过特征提取层都可以提取到多个通道特征图，域注意力块通过多个通道注意力模块得到自适应的通道注意力表示(即全体通道特征图的通道注意力表示)，再通过域适配模块根据通道注意力表示给不同通道特征图予不同的权重，从而能够尽可能地较弱不同模态图像之间特征差异，进而能够提高图像配准模型的跨域适应力，有利于实现不同模态图像在同一图像配准模型内实现配准。
步骤S64：分别利用每一通道特征图的注意力权重对对应通道特征图进行加权处理，得到图像的特征图。
本公开实施例中，利用每一通道特征图的注意力权重对对应通道特征图进行加权处理，可以得到图像的特征图。在本公开的一些实施例中，在图像为第一图像的情况下，图像的特征图为第一特征图，在图像为第二图像的情况下，图像的特征图为第二特征图。
在本公开的一些实施例中，在执行下述步骤S65之前，还可以检测是否满足预设条件，预设条件可以包括：执行步骤S62所述的特征提取的次数小于预设阈值，最近一次执行步骤S62的特征提取得到的通道特征图的分辨率大于预设分辨率，在此情形下，可以将加权处理得到的图像，作为步骤S62所述的特征处理的输入图像，并重新执行步骤S62以及后续步骤。在本公开的一些实施例中，通过多次编码，能够提高第一特征图和第二特征图相近的程度。在本公开的一些实施例中，预设阈值可以设置为至少1次，例如，1次、2次、3次等，在此不做限定；预设分辨率可以根据实际应用需要设置，例如，可以设置为第一图像或第二图像原始分辨率的一半、三分之一等，在此不做限定。
在本公开的一些实施例中，请结合参阅图2，在由图像配准模型的第一编码子网络中的特征提取层执行特征提取操作，并由图像配准模型的第二编码子网络中的特征提取层执行特征提取操作的情况下，预设条件可以包括以下任一者：执行特征提取的为第一编码子网络最后一个特征提取层，执行特征提取的为第二编码子网络最后一个特征提取层。
步骤S65：融合第一特征图和第二特征图，得到速度场。
在本公开的一些实施例中，融合第一特征图和第二特征图得到速度场的实施方式，可以参阅前述公开实施例中的相关步骤。
步骤S66：分别对第一特征图和第二特征图进行解码，得到新的第一特征图和新的第二特征图。
本公开实施例中，新的第一特征图的分辨率不同于本次解码前的第一特征图的分辨率，且新的第二特征图的分辨率不同于本次解码前的第二特征图的分辨率。
步骤S67：利用已得到的速度场，对新的第一特征图和新的第二特征图进行融合，再次得到速度场。
步骤S68：基于多次融合得到的速度场，生成用于配准第一图像和第二图像的配准参数。
在本公开的一些实施例中，基于步骤S62至步骤S64，能够使得第一图像的第一特征图和第二图像的第二特征图相近，从而能够适用于多模态图像间配准，同时基于速度场的配准，能够满足微分同胚。因此，通过本公开实施例中的步骤，不仅能够适用于多模态图像间配准，且能够减少配准次数，请结合参阅图7，对于图像A、图像B、图像C、图像D，任意两者之间均须执行1次配准，而对于这四个图像仅需1个图像配准模型即可，即对于n个图像而言，仅需训练1个图像配准模型，共需执行n-1次配准即可；而在不适用“多模态”，且不满足“微分同胚”的情况下，对于图像A、图像B、图像C、图像D，任意两者之间均须执行2次配准，或需2个图像配准模型，以得到其中一个图像配准至另一个图像的正向配准参数和反向配准参数，即对于n个图像而言，共需执行n(n-1)次配准，或需训练n(n-1)个图像配准模型。
区别于前述实施例，在利用上述方式进行图像编码之前，先获取待配准的多个图像，再将多个图像中的一个作为第一图像，并分别将剩余的至少一个图像作为第二图像。故此，在满足“微分同胚”的基础上，能够使两个图像仅需一次配准流程，即可实现正向和反向的配准，故能够有利于减少配准次数，而在满足“多模态配准”的基础上，能够仅需少量配准次数即可实现多模态图像配准。
请参阅图9，图9是本公开提供的图像配准模型的训练方法一实施例的流程示意图。如图9所示，可以包括如下步骤：
步骤S91：利用图像配准模型的第一编码子网络对第一样本图像进行编码，得到第一样本特征图，并利用图像配准模型的第二编码子网络对第二样本图像进行编码，得到第二样本特征图。
步骤S92：利用图像配准模型的速度场子网络融合第一样本特征图和第二样本特征图，得到样本速度场。
步骤S93：利用图像配准模型的第一解码子网络对第一样本特征图进行解码，得到新的第一样本特征图，并利用图像配准模型的第二解码子网络对第二样本特征图进行解码，得到新的第二样本特征图。
本公开实施例中，新的第一样本特征图的分辨率不同于本次解码前的第一样本特征图的分辨率，且新的第二样本特征图的分辨率不同于本次解码前的第二样本特征图的分辨率。
步骤S94：基于图像配准模型的速度场子网络，利用已得到的样本速度场，对新的第一样本特征图和新的第二样本特征图进行融合，再次得到样本速度场。
步骤S95：基于历次得到的样本速度场，得到用于配准第一样本图像和第二样本图像的样本配准参数。
如前述公开实施例所述，可以基于历次得到的样本速度场，得到用于将第一样本图像配准至第二样本图像的正向样本配准参数。此外，在本公开的一些实施例中，也可以基于历次得到的样本速度场，得到用于将第二样本图像配准至第一样本图像的反向样本配准参数，在此不做限定。
步骤S96：利用样本配准参数对第一样本图像进行处理，得到第一样本图像的样本配准图像。
本公开实施例中，样本配准参数可以是正向样本配准参数，则在此情形下，可以利用正向样本配准参数对第一样本图像进行处理，得到第一样本图像的样本配准图像。此外，在样本配准参数为反向样本配准参数的情况下，可以利用反向样本配准参数对第二图像进行处理，得到第二样本图像的样本配准图像，在此不做限定。
步骤S97：基于第二样本图像和样本配准图像之间的差异，调整图像配准模型的网络参数。
上述步骤S91至步骤S97的实施细节，可参阅前述本公开实施例中的相关步骤。
在本公开的一些实施例中，可以计算第二样本图像和样本配准图像之间损失值，并根据损失值调整图像配准模型的网络参数。
在一个实施场景中，可以采用随机梯度下降(Stochastic Gradient Descent，SGD)、批量梯度下降(Batch Gradient Descent，BGD)、小批量梯度下降(Mini-Batch Gradient Descent，MBGD)等方式，利用损失值对图像配准模型的网络参数进行调整，其中，批量梯度下降是指在每一次迭代时，使用所有样本来进行参数更新；随机梯度下降是指在每一次迭代时，使用一个样本来进行参数更新；小批量梯度下降是指在每一次迭代时，使用一批样本来进行参数更新。
在本公开的一些实施例中，还可以设置一训练结束条件，当满足训练结束条件时，可以结束对图像配准模型的训练。在本公开的一些实施例中，训练结束条件可以包括：损失值小于一预设损失阈值；当前训练次数达到预设次数阈值(例如，500次、1000次等)，在此不做限定。
此外，在样本配准参数为反向样本配准参数的情况下，可以利用反向样本配准参数对第二图像进行处理，得到第二样本图像的样本配准图像，从而可以基于第一样本图像和第二样本图像的样本配准图像之间的差异，调整图像配准模型的网络参数。
区别于前述实施例，通过在多个阶段分别融合不同分辨率的样本特征图，能够得到不同尺度的样本速度场，从而能够基于不同尺度的样本速度场，提高样本配准参数的精度，进而有利于提高图像配准模型的精度。此外，由于在多个阶段分别得到不同尺度的样本速度场，从而能够有利于得到用于将第一样本图像正向地配准至第二样本图像的样本配准参数，也能够有利于得到用于将第二样本图像反向地配准至第一样本图像的样本配准参数，进而能够有利于满足微分同胚。
基于同一技术构思，本公开实施例中还提供了与图像配准方法对应的图像配准装置，由于本公开实施例中的装置解决问题的原理与本公开实施例上述展示方法相似，因此装置的实施可以参见方法的实施。
请参阅图10，图10是本公开提供的图像配准装置100一实施例的框架示意图。图像配准装置100包括：图像编码模块1001、第一融合模块1002、图像解码模块1003、第二融合模块1004和参数获取模块1005，
图像编码模块1001，配置为对第一图像进行编码，得到第一特征图，并对第二图像进行编码，得到第二特征图；
第一融合模块1002，配置为融合第一特征图和第二特征图，得到速度场；
图像解码模块1003，配置为分别对第一特征图和第二特征图进行解码，得到新的第一特征图和新的第二特征图；其中，新的第一特征图的分辨率大于本次解码前的第一特征图的分辨率，且新的第二特征图的分辨率大于本次解码前的第二特征图的分辨率；
第二融合模块1004，配置为利用已得到的速度场，对新的第一特征图和新的第二特征图进行融合，再次得到速度场；
参数获取模块1005，配置为基于多次融合得到的速度场，生成用于配准第一图像和第二图像的配准参数。
在本公开的一些实施例中，图像编码模块1001包括：特征提取子模块，配置为对图像进行特征提取，得到多个通道特征图；权重获取子模块，配置为基于各个通道特征图在多个通道特征图中的重要程度，得到对应通道特征图的注意力权重，特征图加权子模块，配置为分别利用每一通道特征图的注意力权重对对应通道特征图进行加权处理，得到图像的特征图，其中，在图像为第一图像的情况下，通过上述步骤得到的特征图为第一特征图；在图像为第二图像的情况下，通过上述步骤得到的特征图 为第二特征图。
在本公开的一些实施例中，图像配准装置100包括：图像获取模块，配置为获取待配准的多个图像；以及将多个图像中的一个作为第一图像，并分别将剩余的至少一个图像作为第二图像。
在本公开的一些实施例中，多个图像均为医学图像，且多个图像满足以下条件：多个图像是由不同种类的医疗设备扫描得到的；多个图像是由同一种医疗设备在不同扫描时间扫描得到的。
在本公开的一些实施例中，第二融合模块1004包括：转换子模块，配置为对已得到的速度场进行转换，得到位移场；变形子模块，配置为利用位移场对新的第一特征图进行变形，得到变形特征图，融合子模块，配置为融合变形特征图和新的第二特征图，再次得到速度场。
在本公开的一些实施例中，第一融合模块1002包括：拼接子模块，配置为将第一特征图和第二特征图进行拼接，得到拼接特征图；提取子模块，配置为对拼接特征图进行特征提取，得到速度场。
在本公开的一些实施例中，第二融合模块1004，还配置为在满足预设条件的情况下，基于最新得到的第一特征图和第二特征图，重新执行分别对第一特征图和第二特征图进行解码，得到新的第一特征图和新的第二特征图的步骤以及后续步骤。
在本公开的一些实施例中，预设条件包括以下任一者：执行解码的次数小于预设阈值，最近一次执行解码得到的第一特征图或第二特征图的分辨率小于预设分辨率；和/或，新的第一特征图的分辨率大于本次解码前的第一特征图的分辨率，且新的第二特征图的分辨率大于本次解码前的第二特征图的分辨率。
在本公开的一些实施例中，图像配准装置1000包括：图像处理模块，配置为执行以下至少一者：利用配准参数对第一图像进行处理，得到第一图像的配准图像；利用配准参数对第二图像进行处理，得到第二图像的配准图像；利用配准参数对第一图像中至少一个第一像素点进行处理，基于处理后的至少一个第一像素点，得到第二图像中分别与至少一个第一像素点对应的至少一个第二像素点；或者利用配准参数对第二图像中至少一个第二像素点进行处理，基于处理后的至少一个第二像素点，得到第一图像中分别与至少一个第二像素点对应的至少一个第一像素点。
在本公开的一些实施例中，图像编码模块1001，还配置为利用图像配准模型的第一编码子网络对第一图像进行编码，得到第一特征图，并利用图像配准模型的第二编码子网络对第二图像进行编码，得到第二特征图；第一融合模块1002，还配置为利用图像配准模型的速度场子网络融合第一特征图和第二特征图，得到速度场；图像解码模块1003，还配置为利用图像配准模型的第一解码子网络对第一特征图进行解码，得到新的第一特征图，并利用图像配准模型的第二解码子网络对第二特征图进行解码，得到新的第二特征图。
请参阅图11，图11是本公开提供的电子设备110一实施例的框架示意图。电子设备110包括相互耦接的存储器101和处理器102，处理器102配置为执行存储器101中存储的程序指令，以实现上述任一图像配准方法。在本公开的一些实施例中，电子设备110可以包括但不限于：微型计算机、服务器，此外，电子设备110还可以包括笔记本电脑、平板电脑等移动设备，在此不做限定。
在本公开的一些实施例中，处理器102配置为控制其自身以及存储器101以实现上述任一图像配准方法。处理器102还可以称为中央处理单元(Central Processing Unit，CPU)。处理器102可能是一种集成电路芯片，具有信号的处理能力。处理器102还可以是通用处理器、数字信号处理器(Digital Signal Processor，DSP)、专用集成电路(Application Specific Integrated Circuit，ASIC)、现场可编程门阵列(Field-Programmable Gate Array，FPGA)或者其他可编程逻辑器件、分立门或者晶体管逻辑器件、分立硬件组件。通用处理器可以是微处理器或者该处理器也可以是任何常规的处理器等。另外，处理器102可以由集成电路芯片共同实现。
请参阅图12，图12为本公开提供的计算机可读存储介质120一实施例的框架示意图。计算机可读存储介质120存储有能够被处理器运行的程序指令121，程序指令121配置为实现上述任一图像配准方法。
本公开实施例还提供一种计算机程序，计算机程序包括计算机可读代码，在计算机可读代码在电子设备中运行的情况下，电子设备的处理器执行如上述任一实施例所述图像配准方法。
本公开实施例还提供另一种计算机程序产品，该计算机程序产品承载有程序代码，程序代码包括的指令可配置为执行上述方法实施例中所述的图像配准方法。
其中，上述计算机程序产品可以通过硬件、软件或其结合的方式实现。在一些实施例中，所述计算机程序产品可以体现为计算机存储介质，在另一些实施例中，计算机程序产品可以体现为软件产品，例如软件开发包(Software Development Kit，SDK)等等。
本公开实施例中涉及的设备可以是系统、方法和计算机程序产品中的至少之一。计算机程序产品 可以包括计算机可读存储介质，其上载有用于使处理器实现本公开的各个方面的计算机可读程序指令。
计算机可读存储介质可以是可以保持和存储由指令执行设备使用的指令的有形设备。计算机可读存储介质例如可以是但不限于电存储设备、磁存储设备、光存储设备、电磁存储设备、半导体存储设备或者上述的任意合适的组合。计算机可读存储介质的例子(非穷举的列表)包括：便携式计算机盘、硬盘、随机存取存储器(Random Access Memory，RAM)、只读存储器(Read-Only Memory，ROM)、可擦除可编程只读存储器(Electrical Programmable Read Only Memory，EPROM)或闪存、静态随机存取存储器(Static Random-Access Memory，SRAM)、便携式压缩盘只读存储器(Compact Disc Read-Only Memory，CD-ROM)、数字多功能盘(Digital Video Disc，DVD)、记忆棒、软盘、机械编码设备、例如其上存储有指令的打孔卡或凹槽内凸起结构、以及上述的任意合适的组合。这里所使用的计算机可读存储介质不被解释为瞬时信号本身，诸如无线电波或者其他自由传播的电磁波、通过波导或其他传输媒介传播的电磁波(例如，通过光纤电缆的光脉冲)、或者通过电线传输的电信号。
这里所描述的计算机可读程序指令可以从计算机可读存储介质下载到各个计算/处理设备，或者通过网络、例如因特网、局域网、广域网和无线网中的至少之一下载到外部计算机或外部存储设备。网络可以包括铜传输电缆、光纤传输、无线传输、路由器、防火墙、交换机、网关计算机和边缘服务器中的至少之一。每个计算/处理设备中的网络适配卡或者网络接口从网络接收计算机可读程序指令，并转发该计算机可读程序指令，以供存储在各个计算/处理设备中的计算机可读存储介质中。
用于执行本公开操作的计算机程序指令可以是汇编指令、指令集架构(Industry Standard Architecture，ISA)指令、机器指令、机器相关指令、微代码、固件指令、状态设置数据、或者以一种或多种编程语言的任意组合编写的源代码或目标代码，所述编程语言包括面向对象的编程语言—诸如Smalltalk、C++等，以及常规的过程式编程语言，诸如“C”语言或类似的编程语言。计算机可读程序指令可以完全地在用户计算机上执行、部分地在用户计算机上执行、作为一个独立的软件包执行、部分在用户计算机上部分在远程计算机上执行、或者完全在远程计算机或服务器上执行。在涉及远程计算机的情形中，远程计算机可以通过任意种类的网络，包括局域网(Local Area Network，LAN)或广域网(Wide Area Network，WAN)连接到用户计算机，或者，可以连接到外部计算机(例如利用因特网服务提供商来通过因特网连接)。在一些实施例中，通过利用计算机可读程序指令的状态信息来个性化定制电子电路，例如可编程逻辑电路、FPGA或可编程逻辑阵列(Programmable Logic Arrays，PLA)，该电子电路可以执行计算机可读程序指令，从而实现本公开的各个方面。
在本公开的一些实施例中，本公开实施例提供的装置具有的功能或包含的模块可以配置为执行上文方法实施例描述的方法，其实施方式的实现过程可以参照上文方法实施例的描述。
上文对本公开的各个实施例的描述倾向于强调各个实施例之间的不同之处，其相同或相似之处可以互相参考。
在本公开所提供的几个实施例中，应该理解到，所揭露的方法和装置，可以通过其它的方式实现。例如，以上所描述的装置实施方式仅仅是示意性的，例如，模块或单元的划分，仅仅为一种逻辑功能划分，实际实现时可以有另外的划分方式，例如单元或组件可以结合或者可以集成到另一个系统，或一些特征可以忽略，或不执行。另一点，所显示或讨论的相互之间的耦合或直接耦合或通信连接可以是通过一些接口，装置或单元的间接耦合或通信连接，可以是电性、机械或其它的形式。
作为分离部件说明的单元可以是或者也可以不是物理上分开的，作为单元显示的部件可以是或者也可以不是物理单元，即可以位于一个地方，或者也可以分布到网络单元上。可以根据实际的需要选择其中的部分或者全部单元来实现本实施方式方案的目的。
另外，在本公开各个实施例中的各功能单元可以集成在一个处理单元中，也可以是各个单元单独物理存在，也可以两个或两个以上单元集成在一个单元中。上述集成的单元既可以采用硬件的形式实现，也可以采用软件功能单元的形式实现。
集成的单元如果以软件功能单元的形式实现并作为独立的产品销售或使用时，可以存储在一个计算机可读取存储介质中。基于这样的理解，本公开的技术方案本质上或者说对现有技术做出贡献的部分或者该技术方案的全部或部分可以以软件产品公开的形式体现出来，该计算机软件产品存储在一个存储介质中，包括若干指令用以使得一台计算机设备(可以是个人计算机，服务器，或者网络设备等)或处理器(processor)执行本公开各个实施方式方法的全部或部分步骤。而前述的存储介质包括：U盘、移动硬盘、ROM、RAM、磁碟或者光盘等各种可以存储程序代码的介质。
最后应说明的是：以上所述实施例，仅为本公开的具体实施方式，用以说明本公开的技术方案， 而非对其限制，本公开的保护范围并不局限于此，尽管参照前述实施例对本公开进行了详细的说明，本领域的普通技术人员应当理解：任何熟悉本技术领域的技术人员在本公开揭露的技术范围内，其依然可以对前述实施例所记载的技术方案进行修改或可轻易想到变化，或者对其中部分技术特征进行等同替换；而这些修改、变化或者替换，并不使相应技术方案的本质脱离本公开实施例技术方案的精神和范围，都应涵盖在本公开的保护范围之内。因此，本公开的保护范围应所述以权利要求的保护范围为准。
工业实用性
本公开实施例提供了一种图像配准方法、装置、电子设备、存储介质及程序，其中，所述方法由电子设备执行，该方法包括：对第一图像进行编码，得到第一特征图，并对第二图像进行编码，得到第二特征图；融合所述第一特征图和所述第二特征图，得到速度场；分别对所述第一特征图和所述第二特征图进行解码，得到新的第一特征图和新的第二特征图，其中，所述新的第一特征图的分辨率不同于本次解码前的第一特征图的分辨率，且所述新的第二特征图的分辨率不同于本次解码前的第二特征图的分辨率；利用已得到的所述速度场，对所述新的第一特征图和所述新的第二特征图进行融合，以再次得到速度场；以及基于多次所述融合得到的速度场，生成用于配准所述第一图像和所述第二图像的配准参数。

Claims (14)

  1. 一种图像配准方法，所述方法由电子设备执行，所述方法包括：
    对第一图像进行编码，得到第一特征图，并对第二图像进行编码，得到第二特征图；
    融合所述第一特征图和所述第二特征图，得到速度场；
    分别对所述第一特征图和所述第二特征图进行解码，得到新的第一特征图和新的第二特征图，其中，所述新的第一特征图的分辨率不同于本次解码前的第一特征图的分辨率，且所述新的第二特征图的分辨率不同于本次解码前的第二特征图的分辨率；
    利用已得到的所述速度场，对所述新的第一特征图和所述新的第二特征图进行融合，再次得到速度场；以及
    基于多次所述融合得到的速度场，生成用于配准所述第一图像和所述第二图像的配准参数。
  2. 根据权利要求1所述的方法，其中，所述对第一图像进行编码，得到第一特征图，或者，所述对第二图像进行编码，得到第二特征图，包括：
    对图像进行特征提取，得到多个通道特征图；
    基于各个所述通道特征图在所述多个通道特征图中的重要程度，得到对应所述通道特征图的注意力权重；以及
    分别利用每一所述通道特征图的注意力权重对对应所述通道特征图进行加权处理，得到所述图像的特征图；
    其中，在所述图像为所述第一图像的情况下，通过上述步骤得到的所述特征图为所述第一特征图；或者，在所述图像为所述第二图像的情况下，通过上述步骤得到的所述特征图为所述第二特征图。
  3. 根据权利要求2所述的方法，其中，在所述对第一图像进行编码，得到第一特征图，并对第二图像进行编码，得到第二特征图之前，所述方法还包括：
    获取待配准的多个图像；以及
    将所述多个图像中的一个图像作为所述第一图像，并分别将剩余的至少一个图像作为所述第二图像。
  4. 根据权利要求3所述的方法，其中，所述多个图像均为医学图像，且所述多个图像满足以下任一条件：
    所述多个图像是由不同种类的医疗设备扫描得到的；
    所述多个图像是由同一种医疗设备在不同扫描时间扫描得到的。
  5. 根据权利要求1至4任一项所述的方法，其中，所述利用已得到的所述速度场，对所述新的第一特征图和所述新的第二特征图进行融合，再次得到速度场，包括：
    对已得到的所述速度场进行转换，得到位移场；
    利用所述位移场对所述新的第一特征图进行变形，得到变形特征图；以及
    融合所述变形特征图和所述新的第二特征图，再次得到速度场。
  6. 根据权利要求1至5任一项所述的方法，其中，所述融合所述第一特征图和所述第二特征图，得到速度场，包括：
    将所述第一特征图和所述第二特征图进行拼接，得到拼接特征图；以及
    对所述拼接特征图进行特征提取，得到所述速度场。
  7. 根据权利要求1至6任一项所述的方法，其中，在所述基于多次所述融合得到的速度场，生成用于配准所述第一图像和所述第二图像的配准参数之前，所述方法还包括：
    在满足预设条件的情况下，基于最新得到的第一特征图和第二特征图，重新执行所述分别对所述第一特征图和所述第二特征图进行解码，得到新的第一特征图和新的第二特征图的步骤以及后续步骤。
  8. 根据权利要求7所述的方法，其中，所述预设条件包括以下任一者：执行所述解码的次数小于预设阈值，最近一次执行所述解码得到的所述第一特征图或所述第二特征图的分辨率小于预设分辨率；
    和/或，所述新的第一特征图的分辨率大于本次解码前的第一特征图的分辨率，且所述新的第二特征图的分辨率大于本次解码前的第二特征图的分辨率。
  9. 根据权利要求1至8任一项所述的方法，其中，在所述基于多次所述融合得到的速度场，生 成用于配准所述第一图像和所述第二图像的配准参数之后，所述方法还包括以下至少一者：
    利用所述配准参数对所述第一图像进行处理，得到所述第一图像的配准图像；
    利用所述配准参数对所述第二图像进行处理，得到所述第二图像的配准图像；
    利用所述配准参数对所述第一图像中至少一个第一像素点进行处理，基于处理后的所述至少一个第一像素点，得到所述第二图像中分别与所述至少一个第一像素点对应的至少一个第二像素点；或者
    利用所述配准参数对所述第二图像中至少一个第二像素点进行处理，基于处理后的所述至少一个第二像素点，得到所述第一图像中分别与所述至少一个第二像素点对应的至少一个第一像素点。
  10. 根据权利要求1至9任一项所述的方法，其中，所述对第一图像进行编码，得到第一特征图，并对第二图像进行编码，得到第二特征图，包括：
    利用图像配准模型的第一编码子网络对第一图像进行编码，得到第一特征图，并利用所述图像配准模型的第二编码子网络对第二图像进行编码，得到第二特征图；
    所述融合所述第一特征图和所述第二特征图，得到速度场，包括：
    利用所述图像配准模型的速度场子网络融合所述第一特征图和所述第二特征图，得到速度场；
    以及所述分别对所述第一特征图和所述第二特征图进行解码，得到新的第一特征图和新的第二特征图，包括：
    利用所述图像配准模型的第一解码子网络对所述第一特征图进行解码，得到新的第一特征图，并利用所述图像配准模型的第二解码子网络对所述第二特征图进行解码，得到新的第二特征图。
  11. 一种图像配准装置，包括：
    图像编码模块，配置为对第一图像进行编码，得到第一特征图，并对第二图像进行编码，得到第二特征图；
    第一融合模块，配置为融合所述第一特征图和所述第二特征图，得到速度场；
    图像解码模块，配置为分别对所述第一特征图和所述第二特征图进行解码，得到新的第一特征图和新的第二特征图；其中，所述新的第一特征图的分辨率大于本次解码前的第一特征图的分辨率，且所述新的第二特征图的分辨率大于本次解码前的第二特征图的分辨率；
    第二融合模块，配置为利用已得到的所述速度场，对所述新的第一特征图和所述新的第二特征图进行融合，再次得到速度场；
    参数获取模块，配置为基于多次所述融合得到的速度场，生成用于配准所述第一图像和所述第二图像的配准参数。
  12. 一种电子设备，包括相互耦接的存储器和处理器，所述处理器配置为执行所述存储器中存储的程序指令，以实现权利要求1至10任一项所述的图像配准方法。
  13. 一种计算机可读存储介质，其上存储有程序指令，所述程序指令被处理器执行时实现权利要求1至10任一项所述的图像配准方法。
  14. 一种计算机程序，所述计算机程序包括计算机可读代码，在所述计算机可读代码在电子设备中运行的情况下，所述电子设备的处理器执行用于实现如权利要求1至10任一项所述的图像配准方法。
PCT/CN2021/114524 2021-03-26 2021-08-25 图像配准方法、装置、电子设备、存储介质及程序 WO2022198915A1 (zh)

Priority Applications (1)

Application Number Priority Date Filing Date Title
JP2022544835A JP2023522527A (ja) 2021-03-26 2021-08-25 画像レジストレーション方法、装置、電子デバイス、記憶媒体及びプログラム

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202110325843.9 2021-03-26
CN202110325843.9A CN113052882B (zh) 2021-03-26 2021-03-26 图像配准方法及相关装置、电子设备、存储介质

Publications (1)

Publication Number Publication Date
WO2022198915A1 true WO2022198915A1 (zh) 2022-09-29

Family

ID=76515477

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2021/114524 WO2022198915A1 (zh) 2021-03-26 2021-08-25 图像配准方法、装置、电子设备、存储介质及程序

Country Status (3)

Country Link
JP (1) JP2023522527A (zh)
CN (1) CN113052882B (zh)
WO (1) WO2022198915A1 (zh)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116740218A (zh) * 2023-08-11 2023-09-12 南京安科医疗科技有限公司 一种心脏ct成像图像质量优化方法、设备及介质

Families Citing this family (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113052882B (zh) * 2021-03-26 2023-11-24 上海商汤智能科技有限公司 图像配准方法及相关装置、电子设备、存储介质
CN113724307B (zh) * 2021-09-02 2023-04-28 深圳大学 基于特征自校准网络的图像配准方法、装置及相关组件
CN115115676A (zh) * 2022-04-29 2022-09-27 腾讯医疗健康(深圳)有限公司 图像配准方法、装置、设备和存储介质

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111275749A (zh) * 2020-01-21 2020-06-12 沈阳先进医疗设备技术孵化中心有限公司 图像配准及其神经网络训练方法及装置
US20200286257A1 (en) * 2019-03-07 2020-09-10 Mitsubishi Heavy Industries, Ltd. Self-localization device, self-localization method, and non-transitory computer-readable medium
CN111724424A (zh) * 2020-06-24 2020-09-29 上海应用技术大学 图像配准方法
CN112200845A (zh) * 2020-10-22 2021-01-08 清华大学 一种图像配准方法和装置
CN113052882A (zh) * 2021-03-26 2021-06-29 上海商汤智能科技有限公司 图像配准方法及相关装置、电子设备、存储介质

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20200286257A1 (en) * 2019-03-07 2020-09-10 Mitsubishi Heavy Industries, Ltd. Self-localization device, self-localization method, and non-transitory computer-readable medium
CN111275749A (zh) * 2020-01-21 2020-06-12 沈阳先进医疗设备技术孵化中心有限公司 图像配准及其神经网络训练方法及装置
CN111724424A (zh) * 2020-06-24 2020-09-29 上海应用技术大学 图像配准方法
CN112200845A (zh) * 2020-10-22 2021-01-08 清华大学 一种图像配准方法和装置
CN113052882A (zh) * 2021-03-26 2021-06-29 上海商汤智能科技有限公司 图像配准方法及相关装置、电子设备、存储介质

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116740218A (zh) * 2023-08-11 2023-09-12 南京安科医疗科技有限公司 一种心脏ct成像图像质量优化方法、设备及介质
CN116740218B (zh) * 2023-08-11 2023-10-27 南京安科医疗科技有限公司 一种心脏ct成像图像质量优化方法、设备及介质

Also Published As

Publication number Publication date
CN113052882B (zh) 2023-11-24
JP2023522527A (ja) 2023-05-31
CN113052882A (zh) 2021-06-29

Similar Documents

Publication Publication Date Title
WO2022198915A1 (zh) 图像配准方法、装置、电子设备、存储介质及程序
CN111104962A (zh) 图像的语义分割方法、装置、电子设备及可读存储介质
US10467768B2 (en) Optical flow estimation using 4-dimensional cost volume processing
CN109858333B (zh) 图像处理方法、装置、电子设备及计算机可读介质
US10467767B2 (en) 3D segmentation reconstruction from 2D slices
CN113870104A (zh) 超分辨率图像重建
WO2020146911A2 (en) Multi-stage multi-reference bootstrapping for video super-resolution
CN111915480B (zh) 生成特征提取网络的方法、装置、设备和计算机可读介质
WO2023035586A1 (zh) 图像检测方法、模型训练方法、装置、设备、介质及程序
CN112990219B (zh) 用于图像语义分割的方法和装置
US9020273B2 (en) Image processing method, image processor, integrated circuit, and program
US9697584B1 (en) Multi-stage image super-resolution with reference merging using personalized dictionaries
WO2022217876A1 (zh) 实例分割方法及装置、电子设备及存储介质
CN115861131B (zh) 基于图像生成视频、模型的训练方法、装置及电子设备
CN113674146A (zh) 图像超分辨率
CN112396605B (zh) 网络训练方法及装置、图像识别方法和电子设备
WO2022151586A1 (zh) 一种对抗配准方法、装置、计算机设备及存储介质
CN111539287A (zh) 训练人脸图像生成模型的方法和装置
JP7176616B2 (ja) 画像処理システム、画像処理装置、画像処理方法、及び画像処理プログラム
CN112862727B (zh) 一种跨模态图像转换方法及装置
CN117011137A (zh) 基于rgb相似度特征匹配的图像拼接方法、装置及设备
CN114758130B (zh) 图像处理及模型训练方法、装置、设备和存储介质
CN114418845A (zh) 图像分辨率提升方法及装置、存储介质及电子设备
CN116912631B (zh) 目标识别方法、装置、电子设备及存储介质
CN113177483B (zh) 视频目标分割方法、装置、设备以及存储介质

Legal Events

Date Code Title Description
ENP Entry into the national phase

Ref document number: 2022544835

Country of ref document: JP

Kind code of ref document: A

121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21932527

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE