CN117115220B

CN117115220B - Image processing method, service providing method, device, equipment and storage medium

Info

Publication number: CN117115220B
Application number: CN202311121899.8A
Authority: CN
Inventors: 李孜; 田琳; 白晓宇; 吕乐; 闫轲; 金达开
Original assignee: Alibaba Damo Institute Hangzhou Technology Co Ltd
Current assignee: Alibaba Damo Institute Hangzhou Technology Co Ltd
Priority date: 2023-08-31
Filing date: 2023-08-31
Publication date: 2024-04-26
Anticipated expiration: 2043-08-31
Also published as: CN117115220A

Abstract

The embodiment of the invention provides an image processing method, a service providing method, a device, equipment and a storage medium, wherein the method comprises the following steps: and acquiring a reference image and an image to be registered, wherein the reference image and the image to be registered comprise images shot on the same object under different shooting conditions. Next, a reference feature map of the reference image is acquired, the feature map containing semantic information of each information unit in the reference image. And determining a transformation model of the image to be registered according to semantic information in the reference feature map, and registering the image to be registered according to the transformation model to obtain a registered image. Compared with the method for determining the transformation model by using the intensity information or the appearance information of the image, the method can determine the transformation model by using the semantic information of the information unit, so that the transformation model is more accurate, namely the image to be registered can be deformed more reasonably, and the accuracy of image registration is improved.

Description

Image processing method, service providing method, device, equipment and storage medium

Technical Field

The present invention relates to the field of artificial intelligence technologies, and in particular, to an image processing method, a service providing method, an apparatus, a device, and a storage medium.

Background

Image registration (Image registration) is the process of matching images acquired at different times, with different imaging devices, or under different imaging conditions. That is, one image seeks a spatial transformation that spatially coincides with a corresponding point on another image. Such agreement may refer to the same object in the images having the same spatial position on the two images after registration. The photographing conditions may include a photographing apparatus, photographing weather, photographing illuminance, photographing position, photographing angle, and the like.

In practice, how to improve the image registration effect is a problem to be solved.

Disclosure of Invention

In view of this, the embodiments of the present invention provide an image processing method, a service providing method, an apparatus, a device, and a storage medium for improving an image registration effect.

In a first aspect, an embodiment of the present invention provides an image processing method, including:

acquiring a reference feature map of a reference image, wherein the reference feature map comprises semantic information of each information unit in the reference image;

Determining a transformation model corresponding to an image to be registered according to the semantic information, wherein the image to be registered and the reference image comprise images shot on the same object under different shooting conditions;

and registering the images to be registered according to the transformation model to obtain registered images.

In a second aspect, an embodiment of the present invention provides an image processing method, including:

Acquiring a reference image and an image to be registered, which are shot on the same object under different shooting conditions;

Acquiring a reference feature map of the reference image and a feature map to be registered of the image to be registered, wherein the feature map comprises semantic information of each information unit in the image;

Performing correlation processing on the reference feature map and the feature map to be registered to obtain a correlation feature map;

Determining a displacement field with the same size as the image to be registered according to the reference image, the image to be registered and the related feature map;

Smoothing the displacement field by using a velocity field;

And registering the images to be registered according to the smoothing processing result to obtain registered images.

In a third aspect, an embodiment of the present invention provides a service providing method, including:

responding to an input instruction triggered by a processing platform, and acquiring a reference feature map of a reference image, wherein the reference feature map comprises semantic information of each information unit in the reference image;

determining a transformation model corresponding to an image to be registered according to semantic information in the reference feature map, wherein the image to be registered and the reference image comprise images shot on the same object under different shooting conditions;

Registering the image to be registered according to the transformation model;

the registered image is displayed.

In a fourth aspect, an embodiment of the present invention provides a service providing method, including:

Responding to an input instruction for processing flat triggering, and acquiring a reference image and an image to be registered, which are shot for the same object under different shooting conditions;

Smoothing the displacement field by using a velocity field;

registering the images to be registered according to the smoothing result;

the registered image is displayed.

In a fifth aspect, an embodiment of the present invention provides a service providing method, including:

Responding to an input instruction triggered by a training platform, acquiring a first training image, a first training feature image of the first training image, a second training image and a second training feature image of the second training image, wherein the first training image and the second training image comprise images shot on the same object under different shooting conditions;

performing correlation processing on the first training feature map and the second training feature map to obtain a related training feature map, wherein the feature map comprises semantic information of information units in an image;

Inputting the first training image, the second training image and the related training feature map into a prediction model;

determining the smoothness of a target displacement field output by the prediction model;

Determining a first similarity between the second training image and a registration result obtained after the first training image is registered by using the target displacement field;

Determining a second similarity between the second training feature map and the feature map of the registration result;

training the prediction model by taking the first similarity, the second similarity and the smoothness as loss values;

And outputting the prediction model.

In a sixth aspect, an embodiment of the present invention provides an image processing apparatus including:

The characteristic map acquisition module is used for acquiring a reference characteristic map of a reference image, wherein the reference characteristic map comprises semantic information of each information unit in the reference image;

the transformation model determining module is used for determining a transformation model corresponding to an image to be registered according to the semantic information, wherein the image to be registered and the reference image comprise images shot on the same object under different shooting conditions;

And the registration module is used for registering the image to be registered according to the transformation model so as to obtain a registered image.

In a seventh aspect, an embodiment of the present invention provides an electronic device, including a processor and a memory, where the memory is configured to store one or more computer instructions, where the one or more computer instructions, when executed by the processor, implement the image processing method in the first aspect or the second aspect, or perform the service providing method in any one of the third aspect to the fifth aspect. The electronic device may also include a communication interface for communicating with other devices or communication systems.

In an eighth aspect, embodiments of the present invention provide a non-transitory machine-readable storage medium having stored thereon executable code, which when executed by a processor of an electronic device, causes the processor to implement at least the image processing method as in the first or second aspect, or the service providing method as in any of the third to fifth aspects.

In the image processing method provided by the embodiment of the invention, the reference image and the image to be registered are acquired, and the reference image and the image to be registered comprise images shot on the same object under different shooting conditions. Next, a reference feature map of the reference image is acquired, the feature map containing semantic information of each information unit in the reference image. And determining a transformation model of the image to be registered according to semantic information in the reference feature map, and registering the image to be registered according to the transformation model to obtain a registered image.

The purpose of registration is to have the same object in the image have the same spatial position on both images after registration. However, if the transformation model is determined using the intensity information or the appearance information of the image, the two pieces of information do not reflect the semantics of the object in the image, and thus the obtained transformation model is not accurate. The method directly utilizes the semantic information of the information unit to determine the transformation model, and the transformation model is more accurate, so that the accuracy of image registration is improved.

Drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions of the prior art, the following description will briefly explain the drawings used in the embodiments or the description of the prior art, and it is obvious that the drawings in the following description are some embodiments of the present invention, and other drawings can be obtained according to these drawings without inventive effort for a person skilled in the art.

Fig. 1 is a flowchart of an image processing method according to an embodiment of the present invention;

FIG. 2 is a flowchart of another image processing method according to an embodiment of the present invention;

FIG. 3 is a flowchart of another image processing method according to an embodiment of the present invention;

FIG. 4 is a flowchart of another image processing method according to an embodiment of the present invention;

FIG. 5 is a schematic diagram of a model training method according to an embodiment of the present invention;

FIG. 6a is a flowchart of a method for providing services according to an embodiment of the present invention;

FIG. 6b is an interface schematic diagram of a service platform according to an embodiment of the present invention;

FIG. 7 is a flowchart of another service providing method according to an embodiment of the present invention;

FIG. 8a is a flowchart of a method for providing a service according to another embodiment of the present invention;

FIG. 8b is an interface schematic diagram of another service platform according to an embodiment of the present invention;

FIG. 9 is a schematic diagram of a method provided using an embodiment of the present invention in a medical image registration scenario;

fig. 10 is a schematic structural diagram of an image processing apparatus according to an embodiment of the present invention;

Fig. 11 is a schematic structural diagram of an electronic device according to an embodiment of the present invention;

fig. 12 is a schematic structural diagram of another image processing apparatus according to an embodiment of the present invention;

Fig. 13 is a schematic structural diagram of another electronic device according to an embodiment of the present invention;

fig. 14 is a schematic structural diagram of a service providing apparatus according to an embodiment of the present invention;

fig. 15 is a schematic structural diagram of still another electronic device according to an embodiment of the present invention;

fig. 16 is a schematic structural diagram of another service providing apparatus according to an embodiment of the present invention;

Fig. 17 is a schematic structural diagram of still another electronic device according to an embodiment of the present invention;

Fig. 18 is a schematic structural diagram of yet another service providing apparatus according to an embodiment of the present invention;

Fig. 19 is a schematic structural diagram of still another electronic device according to an embodiment of the present invention.

Detailed Description

For the purpose of making the objects, technical solutions and advantages of the embodiments of the present invention more apparent, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention, and it is apparent that the described embodiments are some embodiments of the present invention, but not all embodiments of the present invention. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.

The terminology used in the embodiments of the invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. As used in this application and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise, the "plurality" generally includes at least two, but does not exclude the case of at least one.

It should be understood that the term "and/or" as used herein is merely one relationship describing the association of the associated objects, meaning that there may be three relationships, e.g., a and/or B, may represent: a exists alone, A and B exist together, and B exists alone. In addition, the character "/" herein generally indicates that the front and rear associated objects are an "or" relationship.

The words "if", as used herein, may be interpreted as "at … …" or "at … …" or "in response to a determination" or "in response to an identification", depending on the context. Similarly, the phrase "if determined" or "if identified (stated condition or event)" may be interpreted as "when determined" or "in response to a determination" or "when identified (stated condition or event)" or "in response to an identification (stated condition or event), depending on the context.

It should be noted that, the user information (including but not limited to user equipment information, user personal information, etc.) and the data (including but not limited to data for analysis, stored data, presented data, etc.) related to the present invention are information and data authorized by the user or fully authorized by each party, and the collection, use and processing of the related data need to comply with the related laws and regulations and standards of the related country and region, and provide corresponding operation entries for the user to select authorization or rejection.

It should also be noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a product or system that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such product or system. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a commodity or system comprising such elements.

Some embodiments of the invention will now be described in detail with reference to the accompanying drawings. In the case of no conflict between the embodiments, the following embodiments and features and steps in the embodiments may be combined with each other. In addition, the sequence of steps in the method embodiments described below is only an example and is not strictly limited.

Fig. 1 is a flowchart of an image processing method according to an embodiment of the present invention. The image processing method provided by the embodiment of the invention can be executed by the processing device with the image processing capability. As shown in fig. 1, the method may include the steps of:

s101, acquiring a reference feature map of a reference image, wherein the reference feature map comprises semantic information of each information unit in the reference image.

The processing device may acquire a feature map of the reference image. The reference image may be a medical image or other types of three-dimensional images, and the reference image may also be a two-dimensional image, and the reference feature map may include semantic information of each information unit in the reference image. And when the reference image is specifically a three-bit image, the information units of the image are voxels, and when the reference image is a two-dimensional image, the information units of the image are pixels.

The size of the reference feature map is smaller than the size of the reference image, for example, may be half of the reference image. And the reference feature map may be obtained by extraction from the segmentation model.

Alternatively, when the reference image is in particular a medical image, it may in particular comprise an X-ray image, an ultrasound image, a nuclear magnetic resonance image (Nuclear Magnetic Resonance Imaging, NMRI for short) or the like. X-ray images include, in particular, flat X-ray films, computed X-ray (computed radiography, CR) images, digital X-ray (Digital Radiography, DR) images, and computed tomography (Computed Tomography, CT) images. It is readily understood that these medical images are three-dimensional images.

Alternatively, the segmentation Model may specifically include a SAM (SEGMENT ANYTHING Model) Model, a full convolutional neural network (Fully Convolutional Networks, abbreviated as FCN) Model, a Mask Region convolutional neural network (Mask Region-Convolutional Neural Network, abbreviated as Mask R-CNN) Model, a U-shaped network (Recurrent Residual CNN-based U-Net, abbreviated as R2U-Net) Model based on a recursive residual neural network, and the like.

Alternatively, the above-described segmentation model may be deployed in a processing device, and the reference image may be input into the processing device to output a reference feature map of the reference image from the segmentation model. Alternatively, the segmentation model described above may be deployed in other devices independent of the processing device. The reference image may be input to the other device first, and then the reference feature map output by the other device is input to the processing device for subsequent processing.

S102, determining a transformation model corresponding to an image to be registered according to semantic information, wherein the image to be registered and the reference image comprise images shot on the same object under different shooting conditions.

The reference image and the image to be registered may comprise images taken of the same subject under different photographing conditions. And the reference image and the image to be registered are three-dimensional images or two-dimensional images of the same size. However, because the shooting conditions such as the shooting equipment and/or the shooting angle of the images are different, the same object is easy to cause different positions in the reference image and the image to be registered, and therefore, the processing equipment can further determine a transformation model corresponding to the image to be registered according to semantic information in the reference feature map.

Optionally, when the reference image and the image to be registered may be medical images, the same object contained in the two images may be the same part of the human body or the same tissue structure in the same part, and at this time, the reference image and the image to be registered may be an anatomical map of the human body, and then the anatomical map further displays a plurality of tissue structures contained in the human body. For example, the brain may include brain stem, ventricle, cerebellum, bridge, cerebral vessels, and the like.

Alternatively, the photographing condition may specifically include a device for photographing an image to be registered and a reference image, such as an MRI image photographed by a magnetic resonance device, and a photographing angle, and the reference image may be a CT image photographed by a computer tomograph (i.e., CT machine).

And S103, registering the image to be registered according to the transformation model to obtain a registered image.

Finally, the processing device may register the image to be registered according to the transformation model to obtain a registered image. That is, the processing device may perform a deformation process on the image to be registered according to the transformation model to align the same object in the registered image and the reference image, that is, to make the position of the object in the registered image identical to the position in the reference image. Optionally, registering the image to be registered with the transformation model may specifically include a parametric registration or a non-parametric registration, depending on the representation of the transformation model.

In this embodiment, the processing device acquires a reference image and an image to be registered, both of which include images taken of the same subject under different photographing conditions. Then, a reference feature map of the reference image is acquired, wherein the feature map contains semantic information of each information unit in the reference image. And determining a transformation model of the image to be registered according to semantic information in the reference feature map, and registering the image to be registered according to the transformation model to obtain a registered image.

The purpose of registration is to make the same object (i.e. the image region of the same semantic meaning in the image) positionally aligned in the reference image and the registered image. If the transformation model is determined directly using the intensity information or the appearance information of the image, the obtained transformation model is not accurate because the two pieces of information do not reflect the semantics of the object in the image. The method can directly utilize the semantic information of the information unit to determine the transformation model, so that the transformation model is more accurate, and the accuracy of image registration is improved.

In addition, in practice, special cases such as registration of medical images with large deformations or complex tissue structures often occur when registering medical images. This situation tends to have greater registration difficulty. By using the method of the embodiments of the present invention, that is, using semantic information to register medical images with large deformation or complex tissue structure, a good registration effect can be achieved.

Wherein it is clear from the following description that the same object comprised by the reference image and the image to be registered may be referred to as a target object. Large deformation means that the reference image and the image to be registered both contain the target object, but other objects may also be contained in the reference image or the image to be registered. Other objects in the image may interfere with the registration of the target object when using the intensity information and the appearance information of the image for registration. For example, the reference image is a CT image of the whole body of the human body, and the image to be registered is a CT image of the abdomen of the human body, so that other parts in the whole body CT image will interfere with the registration of the abdomen.

The image processing method provided by the embodiments of the invention uses semantic information when the image processing method is used for registration, and the semantics of other parts in the whole body CT image are obviously different from those of the abdomen, so that the image areas corresponding to other parts cannot participate in the registration process, and interference to the registration of the abdomen cannot be caused.

Wherein, the tissue structure is complex, and the reference image and the image to be registered contain target objects, but the target objects can be continuously moving parts or tissue structures. The images taken when the target object is in different motion states may be greatly different, so that the registration difficulty is increased. For example, the reference image is an abdominal image when the user exhales, the image to be registered is an abdominal image when the user inhales, and the gas inhaled into the abdomen during breathing can interfere with the registration process.

The image processing method provided by the embodiments of the invention uses semantic information when the image processing method is used for registration, but the semantics of the gas in the image are obviously different from those of the abdomen, so that the image area corresponding to the gas in the image to be registered cannot participate in the registration process, and interference to the registration of the abdomen cannot be caused.

In addition, when registering medical images in particular, after performing the image processing method and the service providing system provided by the above-described and the following embodiments of the present invention to complete registration of images, the processing device may optionally further perform the following steps: the registered image and the reference image are shown.

Since the registered image and the reference image may be different types of medical images acquired by different devices and the imaging principles of the different devices are different, the description of the same object by the different types of medical images is focused differently. For example, MRI images can more clearly describe the condition of brain tissue, and CT images can more clearly describe the blood flow condition of cerebral vessels. The professional can compare and observe the two registered images to more accurately know the health state of the object in the images, and further give a judgment result and a treatment scheme corresponding to the judgment result. Alternatively, the presentation of the registered image and the reference image may be performed by the processing device in the above-described embodiment, or may be performed by another device independent of the processing device.

Optionally, the processing device may further perform the following steps after image registration: and carrying out semantic recognition on the registered image and the reference image, and displaying a recognition result.

Based on the identification results obtained in the above steps, a professional can compare and observe the respective identification results of the registered image and the reference image to give a judgment result reflecting the health state of the target object in the image and a treatment scheme corresponding to the judgment result. Alternatively, the device performing the semantic recognition and presentation step may be the processing device in the above embodiment, or may be another device independent of the processing device.

Alternatively, it is mentioned in the embodiment shown in fig. 1 that image registration may be achieved using a transformation model. Alternatively, the transformation model may in particular be represented as an affine transformation matrix, at which point the processing device may perform the embodiment shown in fig. 2 to complete the image registration. Fig. 2 is a flowchart of another image processing method according to an embodiment of the present invention. As shown in fig. 2, the following steps may be included:

s201, acquiring a reference feature map of a reference image, wherein the reference feature map comprises semantic information of each information unit in the reference image.

The specific implementation process of the above step S201 may refer to the specific description of the related steps in the embodiment shown in fig. 1, which is not repeated herein.

S202, acquiring a feature map to be registered of an image to be registered, wherein the feature map to be registered comprises semantic information of each information unit in the image to be registered, and the image to be registered and a reference image comprise images shot on the same object under different shooting conditions.

The image to be registered may also be subjected to feature extraction by the segmentation model to output a feature map to be registered, and the specific process is the same as that of the reference feature map, and may be referred to the description in the embodiment shown in fig. 1, which is not repeated here.

S203, sampling the reference feature map to obtain a sampling set.

The processing device may then further sample the reference feature map to obtain a set of samples. Alternatively, the feature map to be registered may be sampled in a random or uniform manner.

Alternatively, the embodiment does not limit the execution sequence between step S202 and step S203, and the processing device may sample and then acquire the feature map to be registered, or may execute the two steps simultaneously.

S204, determining a second information unit matched with the first information unit in the sampling set in the feature map to be registered, wherein the second information unit is the information unit with the highest semantic information similarity with the first information unit in the feature map to be registered, and the first information unit comprises any information unit in the sampling set.

Based on the sampled set obtained in the above step, the processing device may determine, in the feature map to be registered, a respective matching information unit for each information unit in the sampled set. And because the process of determining the information unit matched with any information unit in the sampling set in the feature map to be registered is the same, the description can be given by taking any information unit in the sampling set, namely the first information unit as an example:

The processing device may calculate a semantic information similarity between semantic information of each information unit in the feature map to be registered and semantic information of a first information unit in the sample set. If the semantic similarity between the second information unit and the first information unit in the feature map to be registered is highest compared with other information units in the feature map to be registered, the first information unit and the second information unit are determined to be matched, and the first information unit and the second information unit can form an information unit pair.

According to the above process, the processing device may perform matching (forward matching) on the to-be-registered feature map based on each information unit in the sample set, so as to obtain at least one information unit pair having a matching relationship.

S205, determining a transformation model which is expressed as an affine transformation matrix according to the positions of the information units with the matching relationship in the reference feature map and the feature map to be registered.

Further, the processing device may determine a transformation model that appears as an affine transformation matrix based on the positions of the information units in which the matching relationship exists in the reference feature map and the feature map to be registered. Alternatively, the positions of the information elements in the feature map of at least one information element pair may each be input as parameters to a linear mathematical model to estimate an affine transformation matrix using the mathematical model.

S206, registering the image to be registered according to the affine transformation matrix to obtain a registered image.

Finally, the processing device may affine transform the image to be registered according to the affine transformation matrix to obtain a registered image. And registration using affine transformation matrices is essentially a parametric registration. Since the same affine transformation can be performed on each information unit in the image to be registered using the affine transformation matrix, step S206 is actually an overall registration, i.e. an overall deformation, of the image to be registered.

In this embodiment, the processing device may match each information unit in the reference feature map and the feature map to be registered by using semantic information, and estimate a transformation model represented as an affine transformation matrix by using the positions of each information unit having a matching relationship in the corresponding feature map, so as to complete image registration by using the matrix. Since the semantic information is also used for the estimation of the affine transformation matrix in the present embodiment, the accuracy of registration can be improved as in the above-described embodiment. In addition, the details and technical effects that can be achieved in this embodiment are referred to in the above embodiments, and are not described herein.

Optionally, after obtaining the reference image, the image to be registered and the feature images corresponding to the reference image and the image to be registered, if the calculation amount is not considered, the processing device may not sample the reference feature image, but directly perform matching (forward matching) on the feature image to be registered with each information unit in the reference feature image as a reference, so as to obtain at least one information unit pair with a matching relationship.

According to the above embodiments, the processing device may determine the affine transformation matrix according to the position of the information unit with the matching relationship in the corresponding feature map, so that the accuracy of the matching relationship may directly affect the accuracy of the affine transformation matrix, and finally affect the registration effect. Optionally, in order to improve accuracy of the matching relationship between the information unit points, after performing forward matching, the processing device may further perform matching (reverse matching) to the reference feature map with reference to each information unit in the feature map to be registered, so as to verify whether the matching relationship in step S204 is correct.

Specifically, the processing device may calculate a semantic information similarity between semantic information of each information unit in the reference feature map and semantic information of a second information unit in the feature map to be registered. And if the semantic information similarity between the third information unit in the reference feature map and the second information unit in the feature map to be registered is highest compared with other information units in the reference feature map, determining that the second information unit is matched with the third information unit. And if the third information unit is identical to the first information unit, the processing device may finally determine that there is a matching relationship between the first information unit and the second information unit. If the first information unit and the third information unit are different, discarding the first information unit.

Through the matching in the forward and reverse directions, the processing device may screen the information unit pair obtained in the step S204, and the forward and reverse matching process may be regarded as a sampling strategy for ensuring the loop consistency (Stable SAMPLING VIA CYCLE Consistency, abbreviated as SSCC).

In this embodiment, after step S204 is performed to achieve forward matching, reverse matching may also be performed in the manner provided in this embodiment. The accuracy of the matching relationship can be improved through forward and reverse matching.

Optionally, in order to further improve accuracy of the affine transformation matrix, the information units with the matching relationship may be screened according to a preset threshold, so as to screen out the information units with semantic information similarity greater than or equal to the preset threshold, and the affine transformation matrix is determined by using positions of the screened information units in the reference feature map and the feature map to be registered.

It should be noted that, the process of screening the information units according to the preset threshold may be performed on the information unit pair obtained by performing forward matching, or may be performed on the information unit pair obtained by performing reverse matching.

In this embodiment, the information unit pairs having the matching relationship are screened according to a preset threshold. Since the screened information units have high similarity semantically, a more accurate affine transformation matrix can be estimated by using the screening result.

Optionally, to further improve the registration effect, the processing device may also perform the steps in the embodiment shown in fig. 3 to perform a fine registration of the image to be registered, i.e. a local deformation of the image to be registered, using a transformation model that appears as a displacement field. Fig. 3 is a flowchart of still another image processing method according to an embodiment of the present invention. As shown in fig. 3, the following steps may be included:

s301, acquiring a reference feature map of the reference image, wherein the reference feature map comprises semantic information of each information unit in the reference image.

S302, acquiring a feature map to be registered of an image to be registered, wherein the feature map to be registered comprises semantic information of each information unit in the image to be registered, and the image to be registered and a reference image comprise images shot on the same object under different shooting conditions.

S303, sampling the reference feature map to obtain a sampling set.

S304, determining a second information unit matched with the first information unit in the sampling set in the feature map to be registered, wherein the second information unit is the information unit with the highest semantic information similarity with the first information unit in the feature map to be registered, and the first information unit comprises any information unit in the sampling set.

The specific implementation process of the steps S301 to S304 may refer to the specific description of the related steps in the embodiment shown in fig. 2, which is not repeated herein.

S305, determining a transformation model which is expressed as a first displacement field according to the positions of the information units with the matching relation in the reference feature map and the feature map to be registered, wherein the reference image and the image to be registered are the same in size, and the size of the first displacement field is smaller than that of the reference image.

The processing device may determine the transformation model expressed as the first displacement field according to the positions in the reference feature map and the feature map to be registered based on the information units having the matching relationship obtained after the execution of steps S301 to S304. Alternatively, the processing device may input at least one information unit pair having a matching relationship and its position in the feature map into a preset algorithm, so as to output the first displacement field according to the position of the at least one information unit pair in the reference feature map and the feature map to be registered by the preset algorithm. Alternatively, the preset algorithm may specifically be a least square method or the like.

Since the size of the reference feature map is smaller than that of the reference image, and the sampling set is the sampling result of the reference feature map, the number of information unit pairs with a matching relationship is also smaller than that of the reference feature map, so that the size of the first displacement field can be smaller than that of the reference image and smaller than that of the image to be registered, for example, 1/8 of that of the reference image.

S306, registering the image to be registered according to the first displacement field to obtain a registered image.

Finally, the processing device may deform the image to be registered according to the first displacement field to obtain a registered image. And compared with an affine transformation matrix, the first displacement field can locally deform the image to be registered, so that fine registration of the image to be registered is realized. And registration with displacement fields is essentially a non-parametric registration.

In this embodiment, the processing device may determine the transformation model expressed as a displacement field from the information units having a matching relationship, thereby achieving fine registration of the images to be registered. Because semantic information of the information units is used in the process of determining the matching relationship, the accuracy of registration can be improved by using the embodiment. In addition, the details of the embodiment which are not described in detail and the technical effects which can be achieved can be referred to the description of the above embodiment, and are not described herein.

In the above embodiment, the processing device may obtain the transformation models of different manifestations by using semantic information, so as to perform parameter registration on the image to be registered alone or perform non-parameter registration alone. To further enhance the registration effect, the processing device may optionally also perform the relevant steps in the embodiments shown in fig. 2 and 3 sequentially to determine the affine transformation matrix and the first displacement field, respectively. The processing device may perform parameter registration on the image to be registered by using the affine transformation matrix to obtain a first transformation result, and perform non-parameter registration on the first transformation result by using the first displacement field, so as to obtain a registered image.

To further enhance the registration effect, the processing device may also perform the steps in the embodiment shown in fig. 4 to more finely register the images to be registered with a larger size deformation field. Fig. 4 is a flowchart of another image processing method according to an embodiment of the present invention. As shown in fig. 4, the following steps may be included:

s401, acquiring a reference image and an image to be registered, which are shot on the same object under different shooting conditions.

S402, acquiring a reference feature map of a reference image and a feature map to be registered of an image to be registered, wherein the feature map comprises semantic information of each information unit in the image.

The processing device acquires the reference image and the image to be registered, and can also acquire the feature images corresponding to the two images. The specific expression form, the acquisition mode and the acquisition mode of the reference image and the image to be registered may be referred to the related description in the embodiment shown in fig. 1, and are not repeated herein.

S403, carrying out correlation processing on the reference feature map and the feature map to be registered so as to obtain a correlation feature map.

The processing device may then perform a correlation process on the reference feature map and the feature map to be registered to obtain a correlation feature map. Alternatively, the correlation process may be direct difference of the two feature maps, or normalized cross-correlation process (Normalized Cross Correlation, abbreviated as NCC) of the two feature maps. The related feature map is also a feature map containing semantic information, and the related feature map can more clearly and directly display the difference of the semantic information between each information unit in the reference feature map and the feature map to be registered.

S404, determining a displacement field with the same size as the image to be registered according to the reference image, the image to be registered and the related feature map.

The processing device may further determine the displacement field from the reference image, the image to be registered and the associated feature map. Alternatively, the reference image, the image to be registered and the related feature map may be taken as inputs to output the displacement field from the predictive model. The displacement field has the same size as the image to be registered, the reference image, i.e. is larger than the first displacement field in the embodiment shown in fig. 3. And in order to distinguish it from the displacement field in the embodiment shown in fig. 3, the displacement field output by the predictive model in embodiments of the present invention may be referred to as a second displacement field. Alternatively, the predictive model may be deployed in a processing device, which may be specifically a CNN backbone model or a conversion (transducer) model, or the like.

S405, smoothing the displacement field with the velocity field.

The processing device may then also smooth the second displacement field using the velocity field. Wherein the velocity field may be a stationary velocity field. The smoothing process may be considered as a process of reducing the second displacement field gradient, the magnitude of which is used to describe the rationality of the deformation of the image. And the smoothing treatment of the displacement field is used for ensuring that the deformation of the image is reasonable after the image registration is carried out according to the displacement field after the smoothing treatment.

And S406, registering the image to be registered according to the smoothing processing result to obtain a registered image.

Finally, the processing device may register the image to be registered using the smoothing result. Because the second displacement field has a larger size than the first displacement field, the use of the second displacement field with a larger size can perform finer deformation on the image to be registered to improve the registration effect. And registration with large-sized displacement fields is essentially a non-parametric registration.

In this embodiment, the correlation feature map can more clearly reflect the difference between the semantic information of the information units in the feature map, and the difference can guide the prediction model to output a more accurate second displacement field, so that the second displacement field can deform the image to be registered in the correct direction. Further, by smoothing the second displacement field, a third displacement field with more reasonable deformation can be obtained, so that the registering effect is improved.

In the embodiment shown in fig. 4, the processing device may register the images to be registered using the second displacement field alone. Optionally, in order to improve the registration effect, the processing device may also sequentially use multiple transformation models, that is, may first transform the image to be registered using the affine transformation matrix, so as to obtain a first transformation result. And registering the first transformation result by using the second displacement field to obtain a registered image.

Specifically, the processing device may perform correlation processing on the reference feature map and the feature map of the first transformation result to obtain a correlation feature map. And inputting the reference image, the first transformation result and the related characteristic diagram into a prediction model so as to realize second displacement field by the prediction model. The processing device may register the first transformation result using this second displacement field.

It should be noted that the second displacement field is actually obtained according to different inputs from the second displacement field in the embodiment shown in fig. 4.

In this embodiment, the processing device may first perform integral deformation on the image to be registered by using two transformation models in sequence, and then use the second displacement field to deform the image to be registered with the information unit as the minimum unit, so as to obtain the registered image.

Optionally, in order to further improve the registration effect, after obtaining the second displacement field using the reference feature map and the feature map of the first transformation result, the processing device may also perform smoothing processing on this second displacement field using the velocity field to obtain the third displacement field. Finally, the processing device may register the first transformation result using this third displacement field.

Alternatively, the velocity field may be used to smooth the first displacement field, and the smoothed first displacement field may be used to register the image to be registered.

In the above embodiments, the large-size displacement field may be output by the prediction model, and then the training process of this prediction model may be described as:

1. The method comprises the steps of obtaining a first training image, a first training feature image of the first training image, a second training image and a second training feature image of the second training image, wherein the first training image and the second training image comprise images shot on the same object under different shooting conditions.

2. And carrying out correlation processing on the first training feature map and the second training feature map to obtain a correlated training feature map, wherein the feature map comprises semantic information of each information unit in the image.

3. The first training image, the second training image and the related training feature map are input into a predictive model.

4. And determining the smoothness of the target displacement field output by the prediction model.

5. And determining a first similarity between the second training image and a registration result obtained after the first training image is registered by using the target displacement field.

6. And determining a second similarity between the second training feature map and the feature map of the registration result.

7. And training a prediction model by taking the first similarity, the second similarity and the smoothness as loss values.

Alternatively, model training may be performed by a processing device or by other devices.

Optionally, in step 2, a correlation processing module may be used to perform processing to obtain a correlation profile (Correlation feature map). In step 4, smoothness may be determined by means of diffusion regularization. In step 6, the first training image and the target displacement field may be input to a spatial transformer (Spatial Tramsform Layer) to output a registration result from the spatial transformer.

In this embodiment, the first similarity may reflect a similarity between the second training image and the intensity information, the semantic information, and the appearance information in the registration result. The second similarity reflects similarity between semantic information in the feature maps of the registration result and the second training image, respectively. The smoothness reflects the rationality of deforming the first training image according to the target displacement field.

Optionally, after step 3, the velocity field may be used to smooth the target displacement field output by the prediction model, and the smoothed displacement field may be further used to calculate the first similarity. At this time, the process of model training may also be as shown in fig. 5.

In this embodiment, similarity between multiple aspects of information and rationality of deformation in an image are considered when the prediction model is trained, so that accuracy of predicting the displacement field by the prediction model can be improved.

Optionally, to improve the registration effect, the processing device may further perform an optimization process on the large-sized second displacement field output by the prediction model. The optimization processing is to ensure that the registered image and the reference image are similar after image registration according to the optimized displacement field.

Alternatively, when the processing device performs image registration by using the second displacement field alone, the processing device may determine the similarity between the feature images of each of the transformation results obtained by deforming the image to be registered by using the second displacement field, optimize the second displacement field, and deform the deformation result by using the optimized second displacement field, so as to finally obtain the image to be registered.

Alternatively, the second transformation result may be obtained when the processing device uses the affine transformation matrix and the second displacement field in succession for image registration. The processing device may determine that the similarity between the reference image and the respective feature map of the second transformation result optimizes the second displacement field to obtain a fourth displacement field. And deforming the second deformation result by using a fourth displacement field to finally obtain an image to be registered.

In yet another alternative, the third transformation result may be obtained when the processing device first performs image registration using the affine transformation matrix and the first displacement field. The processing device may determine the similarity between the feature maps of the reference image and the third transformation result, optimize the second displacement field, and deform the third transformation result by using the optimized second displacement field, so as to finally obtain the image to be registered.

As can be seen from the above embodiments, the processing device may determine the transformation model of various manifestations in different manners, that is, the affine transformation matrix, the first displacement field of small size, and the second displacement field of large size output by the prediction model. Based on these transformation models, the processing device may select one or several of them for image registration.

And optionally, in order to improve the registration effect, before the registration is performed by using the displacement field with any size, the displacement field may be further smoothed so as to perform registration by the smoothed displacement field. Optionally, in order to improve the registration effect, after the displacement field is used for registration, the displacement field may be further optimized according to the registered image, and the optimized displacement field may be used for image registration.

Alternatively, the embodiment of the present invention does not limit the execution order of the smoothing process and the optimization process of the displacement field. And the two can be used sequentially or alternatively.

The above embodiments have described the process of achieving image registration from a flow point of view. On this basis, the image registration may also serve as a service capable of interacting with the user. The service may be provided by a processing platform. The processing platform may provide an image registration service in the manner of the embodiment shown in fig. 6 a. Specifically, fig. 6a is a flowchart of a service providing method according to an embodiment of the present invention. The execution subject of the method may be a processing platform. Alternatively, the processing platform may be deployed at the cloud. As shown in fig. 6a, the method may comprise the steps of:

S501, responding to an input instruction triggered to a processing platform, and acquiring a reference feature map of a reference image, wherein the reference feature map comprises semantic information of each information unit in the reference image.

The service platform may obtain the reference feature map in response to an input instruction from the user. Alternatively, the user may directly input the reference feature map and the image to be registered. Alternatively, just as an alternative operation interface (platform home page) provided by the service platform shown in fig. 6b, the user may select the service content and further input the corresponding reference image and the image to be registered on the operation interface. Then, the service platform can conduct feature extraction on the image to obtain a reference feature map.

S502, determining a transformation model corresponding to the image to be registered according to semantic information in the reference feature map, wherein the reference image and the image to be registered comprise images shot on the same object under different shooting conditions.

And S503, registering the image to be registered according to the transformation model.

S504, the registered image is displayed.

Further, the service platform may perform the above steps S502 to S504 so that the registered image is displayed on the operation interface of the service platform together with the reference image (registration result page).

The specific implementation process of each step may refer to the specific description of the relevant step in each embodiment, which is not repeated herein. In addition, other technical effects achieved by the present embodiment may also be referred to the description in the above embodiments, and will not be described herein.

Fig. 7 is a flowchart of another image processing method according to an embodiment of the present invention. The execution subject of this embodiment is a service platform. As shown in fig. 7, the method may include the steps of:

S601, responding to an input instruction triggered to a processing platform, and acquiring a reference image and an image to be registered, which are shot on the same object under different shooting conditions.

The service platform can acquire an image directly input by a user like responding to an input instruction. An alternative operator interface (platform front page) is provided by the service platform as shown in fig. 6b, on which the user can select the service content and further input the corresponding reference image and image to be registered.

S602, acquiring a reference feature map of a reference image and a feature map to be registered of an image to be registered, wherein the feature map comprises semantic information of each information unit in the image.

The service platform may then also acquire a reference feature map and a feature map to be registered. Alternatively, the user may input the reference image and the image to be registered, and simultaneously input the two feature maps to the service platform. Alternatively, the service may perform feature extraction on the reference image and the image to be registered, which are input by the user, so as to obtain a reference feature map and a feature map to be registered.

And S603, carrying out correlation processing on the reference feature map and the feature map to be registered to obtain a correlation feature map.

S604, determining a displacement field with the same size as the image to be registered according to the reference image, the image to be registered and the related feature map.

S605, smoothing the displacement field with the velocity field.

And S606, registering the image to be registered according to the smoothing processing result.

S607, the registered image is displayed.

The service platform may continue to perform the above steps to ultimately display the registered image and the reference image on the interface. The specific implementation process of each step may refer to the specific description of the relevant step in each embodiment, which is not repeated herein.

In addition, other technical effects achieved by the present embodiment may also be referred to the description in the above embodiments, and will not be described herein.

The processing platform in the above embodiment may further provide a model training service, and the service platform may further train a prediction model for obtaining the large-size second displacement field for the user by means of calculation force of the cloud. Fig. 8a is a flowchart of yet another service providing method according to an embodiment of the present invention. The execution subject of the method may be a training platform. When the user selects the model training service, the service platform in the above embodiment is a training platform. As shown in fig. 8a, the method may comprise the steps of:

S701, responding to an input instruction triggered by a training platform, acquiring a first training image, a first training feature image of the first training image, a second training image and a second training feature image of the second training image, wherein the first training image and the second training image comprise images shot on the same object under different shooting conditions.

The training platform responds to an input instruction of a user and can acquire a first training image and a second training image which are input by the user. Optionally, the first training feature map and the second training feature map may be input by a user, or may be obtained by extracting features from a training image input by the user by the training platform.

S702, performing correlation processing on the first training feature map and the second training feature map to obtain a correlation training feature map, wherein the feature map contains semantic information of information units in the image.

S703, inputting the first training image, the second training image and the related training feature images into a prediction model.

S704, determining the smoothness of the target displacement field output by the prediction model.

S705, determining a first similarity between the second training image and a registration result obtained after the first training image is registered by using the target displacement field.

S706, determining a second similarity between the second training feature map and the feature map of the registration result.

And S707, training a prediction model by taking the first similarity, the second similarity and the smoothness as loss values.

S708, outputting a prediction model.

In this embodiment, the service platform may also respond to the input operation triggered by the user on the interface shown in fig. 8b, so as to execute the above steps, and finally output the prediction model for the user. The predictive model may be downloaded by the user in the form of compressed packets.

Alternatively, in this embodiment, the velocity field may be used to process the displacement field during the training of the predictive model.

The specific implementation process of each step in this embodiment may refer to the specific description of the relevant step in each embodiment, which is not repeated herein. In addition, other technical effects achieved by the present embodiment may also be referred to the description in the above embodiments, and will not be described herein.

In addition, it should be noted that, unlike the users in the embodiments shown in fig. 6a and 7, the users in the embodiment shown in fig. 8a are not users who are required for image registration directly, but users who are required for predictive model training.

Specific implementation procedures of the image processing method and the service providing method provided by the above embodiments will be described below taking medical image registration as an example. The following procedure can also be understood in connection with fig. 9.

The user may acquire an image to be registered and a reference image, which may be a CT image and an MRI image of the human brain, respectively. The user can input a CT image and an MRI image in the platform front page shown in fig. 6 b. After that, the service platform can determine the affine transformation matrix, the small-sized displacement field 1, and the large-sized displacement field 2, respectively, in the manner provided in the above embodiments.

One preferred registration approach: the service platform can deform the CT image by sequentially using the affine transformation matrix, the displacement field 1 and the displacement field 2 so as to finally obtain the registered CT image. The registered CT image and the MRI image serving as a reference can be displayed together in an interface of the service platform for comparison and viewing by professionals.

For clarity of the following description, an image obtained by registering a CT image using an affine transformation matrix may be referred to as an intermediate CT image 1, an image obtained by registering the intermediate CT image 1 using the displacement field 1 may be referred to as an intermediate CT image 2, and an image obtained by registering the intermediate CT image 2 using the displacement field 2 may be referred to as an intermediate CT image 3.

Optionally, to further improve the registration effect, the displacement field 1 and/or the displacement field 2 may be smoothed by using the velocity field, so that the image registration is performed by the smoothed displacement field.

Optionally, to further improve the registration effect, after obtaining the intermediate CT image 2, the service platform may further optimize the displacement field 1 to obtain the displacement field 1' according to the similarity between the feature maps of the features of the intermediate CT image 2 and the MRI image. To re-register the intermediate CT image 2 with the displacement field 1 'again to obtain an intermediate CT image 2'.

Similarly, optionally, to further improve the registration effect, after obtaining the intermediate CT image 3, the service platform may further optimize the displacement field 2 to obtain a displacement field 2' according to the similarity between the feature maps of the features of the intermediate CT image 3 and the MRI image. The intermediate CT image 3 is re-registered with the displacement field 2' to obtain a registered image.

If the displacement field 2 is not optimized, the intermediate CT image 3 is a registered image.

Alternatively, both displacement fields 1 and 2 may be optimized, or alternatively optimized.

Alternatively, the optimization process described above and the smoothing process for the displacement field 2 are not shown in fig. 9.

Details of the registration process, technical effects that can be achieved, and specific training processes for outputting the predictive model of the displacement field 2, which are not described in detail in the above-described registration process, can be found in the above-described related embodiments, and are not described in detail herein

An image processing apparatus of one or more embodiments of the present invention will be described in detail below. Those skilled in the art will appreciate that these image processing devices may be configured using commercially available hardware components through the steps taught by the present solution.

Fig. 10 is a schematic structural diagram of an image processing apparatus according to an embodiment of the present invention, as shown in fig. 10, the apparatus includes:

The feature map obtaining module 11 obtains a reference feature map of a reference image, where the reference feature map includes semantic information of each information unit in the reference image.

The transformation model determining module 12 is configured to determine a transformation model corresponding to an image to be registered according to the semantic information, where the image to be registered and the reference image include images obtained by capturing the same object under different capturing conditions.

And the registration module 13 is used for registering the images to be registered according to the transformation model so as to obtain registered images.

Wherein the reference image and the image to be registered are medical images.

In order to distinguish between the modules in the apparatus provided in the embodiments described below, the following description of the present embodiment may refer to the feature map acquisition module 11 as the first feature map acquisition module 11, and the registration module 13 as the first registration module 13.

Optionally, the first feature map obtaining module 11 is further configured to obtain a feature map to be registered of the image to be registered, where the feature map to be registered includes semantic information of each information unit in the image to be registered.

The transformation model determining module 12 is configured to sample the reference feature map to obtain a sampling set;

Determining a second information unit matched with a first information unit in the sampling set in the feature map to be registered, wherein the second information unit is the information unit with the highest semantic information similarity with the first information unit in the feature map to be registered, and the first information unit comprises any information unit in the sampling set;

And determining a transformation model which is expressed as an affine transformation matrix according to the positions of the information units with the matching relationship in the reference feature map and the feature map to be registered.

Optionally, the transformation model determining module 12 is configured to determine, in the reference feature map, a third information unit that matches the second information unit, where the third information unit is an information unit with the highest semantic information similarity with the second information unit in the reference feature map;

And if the third information unit is the same as the first information unit, determining that the first information unit and the second information unit have a matching relationship.

Optionally, the transformation model determining module 12 is configured to screen the information units with matching relationships according to whether the similarity of semantic information meets a preset threshold; and determining the affine transformation matrix according to the positions of the screened information units in the reference feature map and the feature map to be registered.

Optionally, the transformation model determining module 12 is configured to determine, according to the positions of the information units in the reference feature map and the feature map to be registered, a transformation model that is represented by a first displacement field, where the dimensions of the reference image and the image to be registered are the same, and the dimension of the first displacement field is smaller than the dimension of the reference image.

Optionally, the transformation model determining module 12 is configured to transform the image to be registered by using the affine transformation matrix to obtain a first transformation result;

performing correlation processing on the reference feature map and the feature map of the first transformation result to obtain a correlation feature map;

And inputting the reference image, the first transformation result and the related characteristic map into a prediction model to output a transformation model expressed as a second displacement field by the prediction model, wherein the second displacement field and the image to be registered have the same size.

The first registration module 13 is configured to register the first transformation result by using the second displacement field to obtain a second transformation result.

Optionally, the first registration module 13 is configured to smooth the second displacement field with a velocity field to obtain a third displacement field; registering the first transformation result by using the third displacement field.

Optionally, the apparatus further includes a training module 14, configured to acquire a first training image, a first training feature map of the first training image, a second training image, and a second training feature map of the second training image, where the first training image and the second training image include images captured on the same subject under different capturing conditions;

inputting the first training image, the second training image and the related training feature map into the prediction model;

And training the prediction model by taking the first similarity, the second similarity and the smoothness as loss values.

Optionally, the training module 14 is further configured to smooth the target displacement field with a velocity field; and determining a first similarity between the second training image and a registration result obtained after the first training image is registered by using the smoothing result.

Optionally, the apparatus further comprises: an optimization module 15, configured to determine a similarity between the reference feature map and the feature map of the second transformation result; and optimizing the second displacement field according to the similarity to obtain a fourth displacement field.

The first registration module 13 is configured to perform registration by using the second transformation result of the fourth displacement location, so as to obtain the registered image.

The apparatus shown in fig. 10 may perform the method of the embodiment shown in fig. 1 to 3, and reference is made to the relevant description of the embodiment shown in fig. 1 to 3 for parts of this embodiment not described in detail. The implementation process and technical effects of this technical solution are described in the embodiments shown in fig. 1 to 3, and are not described herein.

In one possible design, the image processing method provided in the above embodiments may be applied to an electronic device, as shown in fig. 11, where the electronic device may include: a first processor 21 and a first memory 22. Wherein the first memory 22 is for storing a program for supporting the electronic device to execute the image processing method provided in the embodiment shown in fig. 1 to 3 described above, the first processor 21 is configured for executing the program stored in the first memory 22.

The program comprises one or more computer instructions, wherein the one or more computer instructions, when executed by the first processor 21, are capable of performing the steps of:

Optionally, the first processor 21 is further configured to perform all or part of the steps in the embodiments shown in fig. 1 to 3.

The electronic device may further include a first communication interface 23 in a structure for the electronic device to communicate with other devices or communication systems.

Fig. 12 is a schematic structural diagram of another image processing apparatus according to an embodiment of the present invention, as shown in fig. 12, the apparatus includes:

A first image acquisition module 31, configured to acquire a reference image and an image to be registered acquired under different photographing conditions for the same subject.

A second feature map obtaining module 32, configured to obtain a reference feature map of the reference image and a feature map to be registered of the image to be registered, where the feature map includes semantic information of each information unit in the image.

And a first correlation processing module 33, configured to perform correlation processing on the reference feature map and the feature map to be registered, so as to obtain a correlation feature map.

A first displacement field determining module 34, configured to determine a displacement field having a size identical to that of the image to be registered according to the reference image, the image to be registered, and the related feature map.

A first smoothing module 35 for smoothing the displacement field with a velocity field.

And a second registration module 36, configured to register the image to be registered according to the smoothing result, so as to obtain a registered image.

The apparatus shown in fig. 12 may perform the method of the embodiment shown in fig. 4, and reference is made to the relevant description of the embodiment shown in fig. 4 for parts of this embodiment not described in detail. The implementation process and the technical effect of this technical solution are described in the embodiment shown in fig. 4, and are not described herein.

In one possible design, the image processing method provided in the foregoing embodiments may be applied to another electronic device, as shown in fig. 13, where the electronic device may include: a second processor 41 and a second memory 42. Wherein the second memory 42 is for storing a program for supporting the electronic device to perform the image processing provided in the embodiment shown in fig. 4 described above, the second processor 41 is configured for executing the program stored in the second memory 42.

The program comprises one or more computer instructions, wherein the one or more computer instructions, when executed by the second processor 41, are capable of performing the steps of:

Smoothing the displacement field by using a velocity field;

Optionally, the second processor 41 is further configured to perform all or part of the steps in the embodiment shown in fig. 4.

The electronic device may further include a second communication interface 43 in its structure for communicating with other devices or communication systems.

Fig. 14 is a schematic structural diagram of a service providing apparatus according to an embodiment of the present invention, as shown in fig. 14, where the apparatus includes:

The third feature map obtaining module 51 is configured to obtain, in response to an input instruction triggered by the processing platform, a reference feature map of a reference image, where the reference feature map includes semantic information of each information unit in the reference image.

The second transformation model determining module 52 is configured to determine a transformation model corresponding to an image to be registered according to semantic information in the reference feature map, where the image to be registered and the reference image include images captured under different capturing conditions for the same object.

And a third registration module 53, configured to register the image to be registered according to the transformation model.

A first display module 54 for displaying the registered images.

The apparatus of fig. 14 may perform the method of the embodiment of fig. 6 a-6 b, and reference is made to the relevant description of the embodiment of fig. 6 a-6 b for parts of this embodiment not described in detail. The implementation process and the technical effect of this technical solution are described in the embodiments shown in fig. 6a to 6b, and are not described here again.

In one possible design, the service providing method provided in the foregoing embodiments may be applied to another electronic device, as shown in fig. 15, where the electronic device may include: a third processor 61 and a third memory 62. Wherein the third memory 62 is for storing a program for supporting the electronic device to execute the service providing method provided in the embodiment shown in fig. 6a to 6b described above, and the third processor 61 is configured for executing the program stored in the third memory 62.

The program comprises one or more computer instructions, wherein the one or more computer instructions, when executed by the third processor 61, are capable of performing the steps of:

in response to an input instruction triggered to a processing platform, acquiring a reference feature map of a reference image, wherein the reference feature map comprises semantic information of each information unit in the reference image

Registering the image to be registered according to the transformation model;

the registered image is displayed.

Optionally, the third processor 61 is further configured to perform all or part of the steps in the embodiments shown in fig. 6 a-6 b.

A third communication interface 63 may also be included in the structure of the electronic device for the electronic device to communicate with other devices or communication systems.

Fig. 16 is a schematic structural diagram of another service providing apparatus according to an embodiment of the present invention, as shown in fig. 16, the apparatus includes:

The second image acquisition module 71 is configured to acquire a reference image and an image to be registered, which are acquired for the same subject under different photographing conditions, in response to an input instruction for processing a flat trigger.

A fourth feature map obtaining module 72, configured to obtain a reference feature map of the reference image and a feature map to be registered of the image to be registered, where the feature map includes semantic information of each information unit in the image.

And a second correlation processing module 73, configured to perform correlation processing on the reference feature map and the feature map to be registered, so as to obtain a correlation feature map.

A second displacement field determining module 74 is configured to determine a displacement field having the same size as the image to be registered according to the reference image, the image to be registered and the correlation feature map.

A second smoothing module 75 for smoothing the displacement field with a velocity field.

And a fourth registration module 76, configured to register the image to be registered according to the smoothing result.

A second display module 77 for displaying the registered images.

The apparatus shown in fig. 16 may perform the method of the embodiment shown in fig. 7, and reference is made to the relevant description of the embodiment shown in fig. 7 for parts of this embodiment not described in detail. The implementation process and the technical effect of this technical solution are described in the embodiment shown in fig. 7, and are not described herein.

In one possible design, the service providing method provided in the above embodiments may be applied to another electronic device, as shown in fig. 17, where the electronic device may include: a fourth processor 81 and a fourth memory 82. Wherein the fourth memory 82 is for storing a program for supporting the electronic device to execute the service providing method provided in the embodiment shown in fig. 7 described above, and the fourth processor 81 is configured for executing the program stored in the fourth memory 82.

The program comprises one or more computer instructions, wherein the one or more computer instructions, when executed by the fourth processor 81, are capable of performing the steps of:

Smoothing the displacement field by using a velocity field;

registering the images to be registered according to the smoothing result;

the registered image is displayed.

Optionally, the fourth processor 81 is further configured to perform all or part of the steps in the embodiment shown in fig. 7 described above.

The electronic device may further include a fourth communication interface 83 in the structure for the electronic device to communicate with other devices or communication systems.

Fig. 18 is a schematic structural diagram of yet another service providing apparatus according to an embodiment of the present invention, as shown in fig. 18, the apparatus includes:

The acquiring module 91 is configured to acquire a first training image, a first training feature map of the first training image, a second training image, and a second training feature map of the second training image in response to an input instruction triggered by the training platform, where the first training image and the second training image include images captured on the same object under different capturing conditions.

And a third correlation processing module 92, configured to perform correlation processing on the first training feature map and the second training feature map, so as to obtain a relevant training feature map, where the feature map includes semantic information of an information unit in the image.

An input module 93, configured to input the first training image, the second training image, and the related training feature map into the prediction model.

And the smoothness determination module 94 is configured to determine smoothness of the target displacement field output by the prediction model.

A similarity determining module 95, configured to determine a first similarity between the second training image and a registration result obtained after the first training image is registered by using the target displacement field; a second similarity between the second training feature map and the feature map of the registration result is determined.

The model training module 96 is configured to train the prediction model by using the first similarity, the second similarity, and the smoothness as loss values.

An output module 97 is configured to output the prediction model.

The apparatus shown in fig. 18 may perform the method of the embodiment shown in fig. 8 a-8 b, and reference is made to the relevant description of the embodiment shown in fig. 8 a-8 b for parts of this embodiment not described in detail. The implementation process and the technical effect of this technical solution are described in the embodiments shown in fig. 8a to 8b, and are not described herein.

In one possible design, the service providing method provided in the foregoing embodiments may be applied to another electronic device, as shown in fig. 19, where the electronic device may include: a fifth processor 101 and a fifth memory 102. The fifth memory 102 is used for storing a program for supporting the electronic device to execute the video recognition method provided in the embodiment shown in fig. 8a to 8b, and the fifth processor 101 is configured to execute the program stored in the fifth memory 102.

The program comprises one or more computer instructions, wherein the one or more computer instructions, when executed by the fifth processor 101, are capable of performing the steps of:

Determining a second similarity between the second training feature map and the registration result; training the prediction model by taking the first similarity, the second similarity and the smoothness as loss values;

And outputting the prediction model.

Optionally, the fifth processor 101 is further configured to perform all or part of the steps in the embodiments shown in fig. 8 a-8 b.

A fifth communication interface 103 may also be included in the structure of the electronic device for the electronic device to communicate with other devices or communication systems.

In addition, an embodiment of the present invention provides a computer storage medium storing computer software instructions for the electronic device, which includes a program for executing the image processing method shown in fig. 1 to 5 or a program for executing the service providing method shown in fig. 6a to 8 b.

Finally, it should be noted that: the above embodiments are only for illustrating the technical solution of the present invention, and are not limiting; although the invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An image processing method, comprising:

Acquiring a reference feature map of a reference image and a feature map to be registered of an image to be registered, wherein the reference feature map comprises semantic information of each information unit in the reference image, and the feature map to be registered comprises semantic information of each information unit in the image to be registered;

Determining a transformation model corresponding to the image to be registered according to the positions of information units with matching relations on semantic information in the reference feature map and the feature map to be registered, wherein the image to be registered and the reference image comprise images shot on the same object under different shooting conditions;

2. The method according to claim 1, wherein the method further comprises:

the method further comprises the steps of:

Sampling the reference feature map to obtain a sampling set;

3. The method according to claim 2, wherein before said determining the transformation model that appears as an affine transformation matrix, the method further comprises:

Determining a third information unit matched with the second information unit in the reference feature map, wherein the third information unit is the information unit with the highest semantic information similarity with the second information unit in the reference feature map;

4. The method according to claim 2, wherein determining a transformation model that appears as an affine transformation matrix based on the locations of the information units in which a matching relationship exists in the reference feature map and the feature map to be registered comprises:

Screening the information units with the matching relationship according to whether the similarity of the semantic information meets a preset threshold value;

and determining the affine transformation matrix according to the positions of the screened information units in the reference feature map and the feature map to be registered.

5. The method according to claim 2, wherein the method further comprises:

And determining a transformation model which is represented as a first displacement field according to the positions of the information units with the matching relation in the reference feature map and the feature map to be registered, wherein the reference image and the image to be registered are the same in size, and the size of the first displacement field is smaller than that of the reference image.

6. The method according to claim 2, wherein the method further comprises:

Transforming the image to be registered by utilizing the affine transformation matrix to obtain a first transformation result;

Performing correlation processing on the reference feature map and the feature map of the first transformation result to obtain a correlation feature map, wherein the correlation feature map reflects the difference of semantic information between each information unit in the reference feature map and the feature map to be registered;

Inputting the reference image, the first transformation result and the related feature map into a prediction model to output a transformation model expressed as a second displacement field by the prediction model, wherein the second displacement field and the image to be registered have the same size;

The registering the image to be registered according to the transformation model comprises the following steps:

registering the first transformation result by using the second displacement field to obtain a second transformation result.

7. The method of claim 6, wherein the registering the first transformation result with the second displacement field comprises:

smoothing the second displacement field by using a velocity field to obtain a third displacement field;

Registering the first transformation result by using the third displacement field.

8. The method of claim 6, wherein the method further comprises:

Acquiring a first training image, a first training feature image of the first training image, a second training image and a second training feature image of the second training image, wherein the first training image and the second training image comprise images shot on the same object under different shooting conditions;

performing correlation processing on the first training feature map and the second training feature map to obtain a correlation training feature map, wherein the feature map comprises semantic information of information units in an image, and the correlation training feature map reflects the difference of semantic information between the information units in the first training feature map and the second training feature map;

9. The method of claim 8, wherein the method further comprises:

smoothing the target displacement field by using a velocity field;

the determining the first similarity between the second training image and the registration result obtained after the first training image is registered by using the target displacement field includes:

And determining a first similarity between the second training image and a registration result obtained after the first training image is registered by using the smoothing result.

10. The method of claim 6, wherein after registering the first transformation result with the second displacement field to obtain a second transformation result, the method further comprises:

Determining the similarity between the reference feature map and the feature map of the second transformation result;

optimizing the second displacement field according to the similarity to obtain a fourth displacement field;

and registering the second transformation result by using the fourth displacement place to obtain the registered image.

11. The method according to any one of claims 1 to 10, wherein the reference image and the image to be registered are medical images.

12. An image processing method, comprising:

Performing correlation processing on the reference feature map and the feature map to be registered to obtain a correlation feature map, wherein the correlation feature map reflects the difference of semantic information between each information unit in the reference feature map and the feature map to be registered;

Smoothing the displacement field by using a velocity field;

13. A service providing method, comprising:

Responding to an input instruction triggered by a processing platform, acquiring a reference feature map of a reference image and a feature map to be registered of an image to be registered, wherein the reference feature map comprises semantic information of each information unit in the reference image, and the feature map to be registered comprises the semantic information of each information unit in the image to be registered;

Determining a transformation model corresponding to an image to be registered according to the positions of an information unit with a matching relation on semantic information in the reference feature image and the feature image to be registered, wherein the image to be registered and the reference image comprise images shot on the same object under different shooting conditions;

Registering the image to be registered according to the transformation model;

the registered image is displayed.

14. A service providing method, comprising:

Performing correlation processing on the reference feature map and the feature map to be registered to obtain a correlation feature map, wherein the feature map comprises semantic information of information units in an image, and the correlation feature map reflects differences of semantic information between the reference feature map and each information unit in the feature map to be registered;

Smoothing the displacement field by using a velocity field;

registering the images to be registered according to the smoothing result;

the registered image is displayed.

15. A service providing method, comprising:

And outputting the prediction model.

16. An image processing apparatus, comprising:

The image registration device comprises a feature map acquisition module, a registration module and a registration module, wherein the feature map acquisition module is used for acquiring a reference feature map of a reference image and a feature map to be registered of an image to be registered, the reference feature map comprises semantic information of each information unit in the reference image, and the feature map to be registered comprises the semantic information of each information unit in the image to be registered;

The transformation model determining module is used for determining a transformation model corresponding to an image to be registered according to the positions of an information unit with a matching relation on semantic information in the reference feature image and the feature image to be registered, wherein the image to be registered and the reference image comprise images shot on the same object under different shooting conditions;

17. An electronic device, comprising: a memory, a processor; wherein the memory has stored thereon executable code which, when executed by the processor, causes the processor to perform the image processing method according to any one of claims 1 to 11 or the service providing method according to any one of claims 12 to 15.

18. A non-transitory machine-readable storage medium having stored thereon executable code, which when executed by a processor of an electronic device, causes the processor to perform the image processing method of any of claims 1 to 11 or the service providing method of any of claims 12 to 15.