WO2023045284A1 - 图像处理方法、装置、计算机设备、存储介质及程序产品 - Google Patents

图像处理方法、装置、计算机设备、存储介质及程序产品 Download PDF

Info

Publication number
WO2023045284A1
WO2023045284A1 PCT/CN2022/082195 CN2022082195W WO2023045284A1 WO 2023045284 A1 WO2023045284 A1 WO 2023045284A1 CN 2022082195 W CN2022082195 W CN 2022082195W WO 2023045284 A1 WO2023045284 A1 WO 2023045284A1
Authority
WO
WIPO (PCT)
Prior art keywords
object instance
image
attribute information
matching
instance
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2022/082195
Other languages
English (en)
French (fr)
Inventor
黄烨翀
陈翼男
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Shanghai Sensetime Intelligent Technology Co Ltd
Original Assignee
Shanghai Sensetime Intelligent Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Shanghai Sensetime Intelligent Technology Co Ltd filed Critical Shanghai Sensetime Intelligent Technology Co Ltd
Publication of WO2023045284A1 publication Critical patent/WO2023045284A1/zh
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/50—Image enhancement or restoration using two or more images, e.g. averaging or subtraction
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/048—Activation functions
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20212—Image combination
    • G06T2207/20221—Image fusion; Image merging
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30004—Biomedical image processing
    • G06T2207/30096—Tumor; Lesion

Definitions

  • the present disclosure relates to the technical field of image processing, and relates to but not limited to an image processing method, device, computer equipment, storage medium and program product.
  • Object detection related algorithms can detect regions of interest from input images, and are widely used in medical image analysis, such as lesion detection and other fields.
  • the existing process of providing medical images to determine the course of a lesion often involves multiple image sources. For example, multiple time-series images are required for follow-up, multiple images of different contrast stages are required for contrast imaging, and multiple images of different modalities are required for multimodal fusion diagnosis. images, etc., which involves the fusion of images; the current fusion method has the problem of large errors.
  • Embodiments of the present disclosure at least provide an image processing method, device, computer equipment, storage medium, and program product.
  • an embodiment of the present disclosure provides an image processing method, including:
  • the first object instance and the second object instance are fused.
  • the fusion of the first object instance and the second object instance can be performed with higher precision.
  • it also includes: acquiring a first original image and a second original image;
  • the first attribute information or the second attribute information includes at least one of the following:
  • the object instance includes: the first object instance and the second object instance.
  • the matching of the first object instance and the second object instance based on the first attribute information and the second attribute information includes:
  • the first attribute information and the second attribute information determine at least one of the following matching information between the first object instance and the second object instance: similarity, matching priority, distance, etc. effective radius;
  • a matching degree between the first object instance and the second object instance is obtained.
  • the obtaining the matching degree between the first object instance and the second object instance based on the matching information includes:
  • the matching degree between the first object instance and the second object instance can be obtained more accurately.
  • Matching the first object instance and the second object instance based on the first attribute information and the second attribute information to obtain the relationship between the first object instance and the second object instance match including:
  • the object instance pair For each object instance pair, according to the first attribute information corresponding to the first object instance included in the object instance pair and the second attribute information corresponding to the second object instance included in the object instance pair, the object instance pair The included first object instance is matched with the second object instance to obtain the matching degree of the object instance pair.
  • merging the first object instance and the second object instance based on the matching degree includes:
  • the object instance group For each object instance group, if the object instance group includes at least a first object instance and at least one second object instance, fuse the first object instance and the second object instance included in the object instance group.
  • the merging the first object instance and the second object instance includes:
  • the first attribute information of the first object instance and the second attribute information of the second object instance are fused.
  • an image processing device including:
  • a first determining module configured to determine first attribute information of a first object instance of the target object in the first image and determine second attribute information of a second object instance of the target object in the second image;
  • a matching module configured to match the first object instance and the second object instance based on the first attribute information and the second attribute information, to obtain the first object instance and the second object The degree of matching between instances;
  • a fusion module configured to fuse the first object instance and the second object instance based on the matching degree.
  • it also includes:
  • An acquisition module configured to acquire the first original image and the second original image
  • a second determination module configured to determine transformation relationship information between the first original image and the second original image
  • a transformation module configured to transform the first original image based on the transformation relationship information to obtain a first image, and use the second original image as the second image; or, based on the transformation relationship information Perform transformation processing on the second original image to obtain a second image, and use the first original image as the first image.
  • the attribute information includes at least one of the following:
  • the object instance includes: the first object instance and the second object instance.
  • the matching module includes:
  • a determining unit configured to determine at least one item of the following matching information between the first object instance and the second object instance based on the first attribute information and the second attribute information: similarity, matching priority level, distance, equivalent radius;
  • a matching unit configured to obtain a matching degree between the first object instance and the second object instance based on the matching information.
  • the matching unit is configured as:
  • the matching module is configured as:
  • the object instance pair For each object instance pair, according to the first attribute information corresponding to the first object instance included in the object instance pair and the second attribute information corresponding to the second object instance included in the object instance pair, the object instance pair The included first object instance is matched with the second object instance to obtain the matching degree of the object instance pair.
  • the fusion module is configured as:
  • the object instance group For each object instance group, if the object instance group includes at least a first object instance and at least one second object instance, fuse the first object instance and the second object instance included in the object instance group.
  • the fusion module is configured as:
  • the first attribute information of the first object instance and the second attribute information of the second object instance are fused.
  • an embodiment of the present disclosure further provides a computer device, including: a processor, a memory, and a bus, the memory stores machine-readable instructions executable by the processor, and when the computer device is running, the processing The processor communicates with the memory through a bus, and when the machine-readable instructions are executed by the processor, the above-mentioned first aspect, or the steps in any possible implementation manner of the first aspect are executed.
  • a computer device including: a processor, a memory, and a bus
  • the memory stores machine-readable instructions executable by the processor, and when the computer device is running, the processing
  • the processor communicates with the memory through a bus, and when the machine-readable instructions are executed by the processor, the above-mentioned first aspect, or the steps in any possible implementation manner of the first aspect are executed.
  • embodiments of the present disclosure further provide a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the above-mentioned first aspect, or any of the first aspects of the first aspect, may be executed. Steps in one possible implementation.
  • the embodiments of the present disclosure further provide a computer program product
  • the computer program product includes a non-transitory computer-readable storage medium storing the computer program
  • the computer program product may be a software installation package
  • the above computer program may be Operate to cause the computer to execute the steps in the above first aspect, or any possible implementation manner of the first aspect.
  • the image processing method, device, computer equipment, storage medium, and program product provided by the embodiments of the present disclosure, by matching the attribute information of the first object instance and the second object instance, the relationship between the first object instance and the second object instance is obtained. , and based on the matching degree, the first object instance and the second object instance are fused. In this way, by matching the attribute information of the first object instance and the second object instance, the matching degree between the first object instance and the object instance can be obtained, so that the first object instance and the second object instance can be matched according to the matching degree. more precise fusion.
  • FIG. 1 shows a flowchart of an image processing method provided by an embodiment of the present disclosure
  • FIG. 2 shows a schematic diagram of image fusion proposed by an embodiment of the present disclosure
  • FIG. 3 shows a flowchart of a method for determining transformation relationship information between a first original image and a second original image provided by an embodiment of the present disclosure
  • FIG. 4 shows a schematic diagram of multi-object instance matching proposed by an embodiment of the present disclosure
  • Fig. 5 shows a schematic diagram of an image processing device provided by an embodiment of the present disclosure
  • FIG. 6 shows a schematic diagram of another image processing device provided by an embodiment of the present disclosure.
  • FIG. 7 shows a schematic diagram of a matching module in an image processing device provided by an embodiment of the present disclosure
  • Fig. 8 shows a schematic diagram of a computer device provided by an embodiment of the present disclosure.
  • the detection performance and results of the target detection algorithm may be different. For example, lesions may disappear or increase during follow-up, and the appearance of lesions in different angiographic stages is different. In addition, the same patient is shooting When different images are used, there are spatial displacements and changes in their own background conditions; these two factors make it impossible to simply combine detection results from different sources.
  • For the detection of multi-source images there are currently two main solutions: one is to perform detection on the images from each source, and the reader establishes a connection by himself; the other is to use a registration algorithm to establish the spatial relationship between the two images , and then mapped from one source to another.
  • the first method does not solve the problem of multi-source image detection, and the second method partially solves the problem, but the current method of multi-source image fusion is generally to directly superimpose the registered images; this image fusion method causes The problem of poor fusion accuracy.
  • an embodiment of the present disclosure provides an image processing method, which obtains the matching degree between the first object instance and the object instance by matching the attribute information of the first object instance and the second object instance, so that the matching degree between the first object instance and the object instance can be obtained according to Matching degree, more accurate fusion of the first object instance and the second object instance.
  • an image processing method disclosed in the embodiment of the present disclosure is firstly introduced in detail.
  • the image processing method provided in the embodiment of the present disclosure is generally executed by a computer device with certain computing capabilities.
  • the image processing method may be implemented by a processor invoking computer-readable instructions stored in a memory.
  • FIG. 1 is a flowchart of an image processing method provided by an embodiment of the present disclosure, the method includes steps S101 to S103, wherein:
  • S101 Determine first attribute information of a first object instance of a target object in a first image and determine second attribute information of a second object instance of the target object in a second image.
  • the first image and the second image are images obtained by shooting the same target object.
  • the first image and the second image are images obtained by shooting the target object at different times;
  • the image obtained by shooting the target object can refer to the same diseased organ or the same body part of the same patient;
  • the first image and the second image can be the angles and distances taken at the same time for the same organ or the same body part Multiple images with different shooting parameters, or multiple images taken at different times for the same organ or the same body part.
  • the first object instance and the second object instance refer to a lesion that may exist on the first image and the second image respectively.
  • the attribute information includes at least one of the following:
  • the object instance includes: the first object instance in the first image and the second object instance in the second image.
  • an object instance is a result obtained through a target detection algorithm, and a certain post-processing and analysis may be performed on the detection result to obtain the object instance and its corresponding attribute information.
  • the target detection algorithm can be any method of detecting a specific target from an image, which is not limited here. When two images are detected, the corresponding target detection algorithms can be different, but the detected target types need to be the same, and the same Type attribute information.
  • the target detection algorithm can be Mask RCNN, Retina Net, etc.
  • the image may be detected by the above target detection algorithm, so as to obtain the attribute information corresponding to the object instance.
  • the position information may be the coordinate information of the object instance in the image, namely (x, y), which may be the coordinate information corresponding to the center point of the bounding box;
  • the size information may be the object instance The parameter information characterizing the size of an object instance, such as the size and radius of the instance;
  • the probability that the object instance belongs to the target object can be the probability or degree that the object instance has the target attribute determined based on a certain measurement standard, for example, a lesion corresponds to The degree of malignancy, etc.
  • the feature data can be coded for the hidden layer, that is, the feature vector corresponding to the object instance extracted by a certain network in the detection network;
  • the grayscale data of the image area corresponding to the object instance refers to the image area The gray value corresponding to each pixel;
  • the radiomics information is the description amount obtained by using the radiomics method to describe the features according to the image and range of the object instance.
  • other attribute information may also be included.
  • target detection processing may also be performed on the image, wherein the target detection method may be a method using semantic segmentation, that is, to determine which type of image each pixel in the image belongs to, and based on As a result of the target detection processing, the position information of the object instance in the image and the probability that the object instance belongs to the target object are obtained, and based on the position of the object instance in the image, determine and Grayscale data and/or radiomics information of the image region corresponding to the object instance.
  • semantic segmentation that is, to determine which type of image each pixel in the image belongs to, and based on
  • the position information of the object instance in the image and the probability that the object instance belongs to the target object are obtained, and based on the position of the object instance in the image, determine and Grayscale data and/or radiomics information of the image region corresponding to the object instance.
  • the first original image and the second original image are respectively original images taken for the target object. Since there is a certain positional deviation between the first original image and the second original image, for example, the overall translation of the second original image relative to the first original image is 10mm, etc., therefore, it is necessary to register the first original image and the second original image , so that the first original image and the second original image are in the same position in space, which facilitates the extraction and matching of later attribute information.
  • the second original image can be registered based on the position information of the first original image to obtain the first image and the transformed second image, or based on the position information of the second original image, the The first original image is registered to obtain the second image and the transformed first image.
  • FIG. 2 is a schematic diagram of image fusion proposed by an embodiment of the present disclosure.
  • Figure 2 it can be concluded that after object instance a is registered based on object instance A, the global spatial relationship 2 corresponding to object instance a is adjusted to the same global spatial relationship 1 as object instance A, and its corresponding coordinates, Dimensions are changed, but other properties not related to space are unchanged.
  • an embodiment of the present disclosure provides a flow chart of a method for determining transformation relationship information between a first original image and a second original image.
  • the method includes:
  • S301 Perform multi-level feature extraction on the first image to obtain the first target feature map corresponding to the multi-level feature extraction; and perform multi-level feature extraction on the second image to obtain the first target feature map corresponding to the multi-level feature extraction respectively The second target feature map of .
  • the first original image and the second original image may include different images taken for the same object at different times or at different angles.
  • the first original image and the second original image may be medical images obtained from multiple shots during one scan of the same lesion of the same patient , or, may be different medical images obtained during scanning the same lesion of the patient at different times, which is not limited here.
  • multi-level feature extraction can be performed on the two images respectively to obtain the first target feature map and the second target feature map respectively corresponding to the multi-level feature extraction.
  • target feature map can adopt the neural network self-learning method to realize the gradient return of the multi-level features of the image, and extract the high-level semantic features of the image.
  • multi-level feature extraction is performed on the image to obtain feature maps respectively corresponding to the multi-level feature extraction, including:
  • the first input data includes: the image, or the encoding feature output by the subsequent level of feature extraction Figure
  • the second input data includes: the decoding feature map output by the previous level of feature extraction
  • the decoding feature map corresponding to this level of feature extraction is determined as the target feature map corresponding to this level of feature extraction.
  • a feature extractor may be used to perform feature extraction on the first original image and the second original image respectively to obtain multi-level features corresponding to the two images.
  • the feature extractor is a feature pyramid network, which is divided into two stages of encoding and decoding, and connects the low-level and high-level networks by skip connections.
  • the feature extractor receives an image as input data, downsamples (also called “downsampling") and extracts features layer by layer in the encoding module, and upsamples (also called “upsampling”) and extracts features layer by layer in the decoding module.
  • the output of each decoding module is fed into an iterative registration network to form a multi-layer, coarse-to-fine pyramid feature.
  • the corresponding input data is the original image, that is, the first original
  • the corresponding input data is the encoding feature map corresponding to the output of the previous level, so that an iterative process of encoding feature extraction can be realized.
  • the lowest level that is, the most "thin” level in the “coarse to fine” structure of the feature "pyramid” network
  • the decoding feature map obtained by the corresponding lowest level encoding feature map
  • the corresponding input data is the decoding feature map corresponding to the output of the next level, and finally output the feature extraction result for the original image, so that the iterative process of decoding feature extraction can be realized.
  • the multi-level feature extraction described in the embodiments of the present disclosure indicates a multi-level feature extraction process from "fine” to "coarse", and each level of feature extraction includes an encoding process and a decoding process corresponding to the level of feature extraction, that is, From the bottom to the top of the "pyramid" network.
  • each level of feature extraction includes an encoding network corresponding to this level of feature extraction and a decoding network corresponding to this level of feature extraction;
  • each encoding network includes: an encoding module and a parallel domain adaptation module.
  • the decoding network corresponding to the first-level encoding network includes: a decoding module.
  • the decoding network corresponding to other encoding networks except the first-level encoding network includes: a decoding module and a gating fusion module.
  • the encoding process is a process from “coarse” to "fine", and the decoding process is sequentially decoded from “fine” to “coarse”.
  • the multi-level feature extraction described in the embodiments of the present disclosure corresponds to the decoding process.
  • the encoding network in each level of feature extraction it is used to perform encoding processing corresponding to the level of feature extraction on the first input data to obtain the encoding feature map corresponding to the level of feature extraction.
  • the decoding network in each level of feature extraction it is used to perform fusion processing on the encoding feature map corresponding to the level of feature extraction and the second input data to obtain a fusion feature map, and perform a fusion process on the fusion feature map with the level
  • the decoding process corresponding to the feature extraction obtains the decoding feature map corresponding to the feature extraction of this level.
  • encoding processing may be performed based on the first input data to obtain an encoding feature map corresponding to each level.
  • performing encoding processing corresponding to feature extraction of this level on the first input data to obtain an encoding feature map corresponding to feature extraction of this level includes:
  • the encoded feature map is obtained based on the downsampled feature map and the attention weights.
  • the encoding module When the first input data is input to the encoding module in each level of feature extraction, the encoding module will down-sample the first input data to obtain the corresponding down-sampling feature map, and input the output down-sampling feature map to each level In the parallel domain adaptation module corresponding to the encoding module.
  • the parallel domain adaptation module can enhance the expressive ability of the encoding network for some specific features in the encoding stage; for example, the image includes human organs; the feature extractor can be trained to enhance the expressive ability of organ texture features.
  • the parallel domain adaptation module After the downsampling feature map is input to the parallel domain adaptation module, the parallel domain adaptation module performs channel attention processing on the downsampling feature map to obtain the attention weights corresponding to each data channel in the downsampling feature map.
  • the above process includes:
  • the attention weights corresponding to each data channel in the downsampled feature map are obtained.
  • the parallel domain adaptation module includes two mechanisms: a channel attention mechanism and a domain awareness mechanism.
  • the channel attention mechanism determines the channel attention weights of multiple channels in the form of group convolution, thereby generating multiple candidate channel weights.
  • the domain-aware mechanism can combine the weights of multiple candidate channels according to the properties of the feature map to obtain the final result, so that when the performance of the first original image and the second original image are very different in the image domain, they can also be extracted through feature extraction.
  • the device obtains similar features, which is convenient for the subsequent registration process.
  • the parallel domain adaptation module is applied to the encoding stage of multi-level feature extraction, for example, in the encoding process from the i+1th level to the ith level, the corresponding parallel domain adaptation module is applied, for example, During the encoding process from level 3 to level 2, the parallel domain adaptation module 3 is applied.
  • FIG. 4 it is an example of a parallel domain adaptation module provided by an embodiment of the present disclosure.
  • the downsampling feature map corresponding to the encoding module of the i-th layer is expressed as (H, C, D, H, W), where N represents the number of feature maps included in a feature extraction (Number of instances in batch) , C represents the number of channels (Channel) of the feature map, D, H, and W represent the length, width and height (Depth, Height, Width) of the feature map, respectively, and B represents the branch of channel attention (Branches of attention).
  • the global average pooling process can be performed on the downsampled feature map, that is, the feature map Perform dimensionality reduction, average all pixels in the spatial dimensions of length, width, and height, and use it to reduce the dimensionality of the data, so that the feature map changes from (N, C, D, H, W) to (N, C), thus the overall information of the channel dimension can be extracted to obtain the first feature subgraph.
  • repeat Repeat
  • flatten Flatten
  • the first feature subgraph Transformed into the form of (N, BC, 1), where repetition refers to repeating the dimensionally reduced data B times, corresponding to B channel attention, where the repetition operation is to change the format of the data to facilitate calculation.
  • the feature map of this form is subjected to B-group convolution, activation and then B-group convolution processing, wherein the activation function can use a linear rectification function (Rectified Linear Unit, ReLU), and finally, the convolution is obtained
  • the channel reorganization is performed on the feature map of (N, C, B) to obtain the feature map of (N, C, B), so that the candidate attention weights corresponding to each data channel can be obtained.
  • the first feature submap is transformed into the form of (N, C), where flattening refers to rearranging the order of data, where flattening
  • the operation is to change the format of the data to facilitate calculation.
  • the feature map of this form is fully connected and activated.
  • the ReLU activation function and the activation function (Softmax) of the deep learning output layer can be used respectively to obtain a feature map of the form (N, B).
  • the feature domain weights of candidate attention corresponding to each data channel can be obtained.
  • the activation function can be an S-type growth curve (Sigmoid), so that the encoding feature map of the i-th level can be obtained.
  • the attention weights corresponding to each data channel in the downsampling feature map can be determined based on the feature domain weights of candidate attention corresponding to each data channel .
  • the image presented are different. Therefore, based on the pass domain
  • the feature domain weight determined by the perception module is used to match the feature domain weights of candidate attention corresponding to each data channel, highlighting the weight of the target position, that is, the weight of the liver position, and weakening other positions, such as muscles and blood. In this way, those positions can be prominently displayed for the target position. Therefore, even when the display effects of the first original image and the second original image are quite different, it can also be obtained by the feature extractor similar features for registration.
  • the channel attention mechanism can adopt multiple sets of parallel channel attention weight determination methods. For example, assuming that there are 12 channels, under normal circumstances, the above 12 channels can be convolved separately to obtain the candidate corresponding to the 12 channels. Attention weight, in order to improve the processing speed, the 12 channels can be divided into three channel attention, each of which includes 4 channels, so that the three channel attention can be convoluted at the same time, which improves the processing speed.
  • the encoding feature map can be obtained based on the downsampling feature map and attention weight, that is, the features of each channel in the downsampling feature map are calculated according to the corresponding
  • the attention weights are reorganized to obtain the encoded feature maps corresponding to each level.
  • the corresponding encoding feature map In the decoding process corresponding to each level of feature extraction, in addition to the second input data input to each level of decoding module, the corresponding encoding feature map also participates in the generation process of the encoding feature map.
  • the upsampled feature map of the previous decoding module has a lower spatial resolution but a higher degree of semantic information expression. Therefore, In order to combine the respective advantages of the upsampled feature map of the previous decoding module of the encoded feature map, in the embodiment of the disclosure, a gated fusion module that improves the fusion effect of high-level low-resolution features and low-level high-resolution features is used in the decoding stage .
  • performing fusion processing on the coded feature map corresponding to the level of feature extraction and the second input data to obtain the fusion feature map includes:
  • the encoding feature map of the encoding module data corresponding to the decoding module of this level is fused with the second feature data input to the decoding module to obtain the proportion of the encoding feature map corresponding to the feature extraction of this level.
  • the weight of the encoded feature map is multiplied by the weight to obtain a second feature submap, which is concatenated with the second feature data input to the decoding module to obtain a fusion feature map.
  • the encoded feature map from the encoding module is multiplied by the weight, and the decoding feature map of the decoding module is concatenated in the channel dimension, and sent to the next decoding module, so that the feature maps from the two sources can be fused more efficiently.
  • the function of the gated fusion module is mainly to determine the weights corresponding to each feature point in the encoding feature map, and extract the corresponding encoding feature map and the second input data based on the features of this level to obtain the The weights corresponding to each feature point in the coded feature map corresponding to level feature extraction, including:
  • weights corresponding to each feature point in the coded feature map corresponding to the level of feature extraction are obtained.
  • the eigenvalue of any feature point in the third eigensubgraph represents the autocorrelation coefficient of the image region corresponding to the feature point.
  • the third feature submap is convolved, normalized, and based on the linear rectification function (Rectified Linear Unit, ReLU) activation processing to obtain the fourth feature subgraph corresponding to the third feature subgraph, where convolution, normalization and activation processing are operations in a convolutional neural network (Convolutional Neural Network, CNN) Model (pattern), the number of occurrences of the above operations represents the depth of the network. The deeper the network, the stronger the expressive ability, and the larger the number of parameters. Here, in order to improve the expressive ability of the network, two or more layers can be selected. Convolution, normalization, and activation processing.
  • the encoding feature map and the local autocorrelation coefficient with the second input data can be obtained based on the fourth feature submap, including:
  • the maximum channel dimension and the average channel dimension are both aimed at the dimensionality reduction operation of the data, which is based on the channel dimension.
  • W becomes (N, 1, C, H, W), where the channel dimension maximum value and the channel dimension average value can be expressed as (N, 1, C, H, W).
  • the above two values can be spliced to obtain the spliced splicing result (N, 2, C, H, W), and then the splicing can be Convolution and normalization processing are performed on the result to obtain the local autocorrelation coefficient between the coded feature map and the second input data.
  • the activation function (sigmoid) can be used to activate the autocorrelation coefficient to obtain the gating activation value corresponding to each feature point in the encoding feature map; the gating activation value is used weights corresponding to each feature point in the coded feature map.
  • the fusion feature map can be obtained by multiplying the coded feature map by the gating activation value obtained through the activation function, and then concatenating it with the second input data.
  • the embodiment of the present disclosure also provides an example of a gating fusion module.
  • the encoded feature map output by the i-level encoding module and the decoded feature map of the data after the i-level decoding module has been down-sampled are spliced, where splicing refers to splicing in the channel dimension, requiring other channel dimensions
  • splicing refers to splicing in the channel dimension, requiring other channel dimensions
  • the same size, such as two data are (N, C1, D, H, W) and (N, C2, D, H, W), the size after splicing is (N, C1+C2, D, H , W).
  • the concatenated feature maps are processed by convolution, normalization, and activation, and then the channel dimension is maximized and the channel dimension is averaged.
  • the above two data are spliced, and convolution, normalization and activation processing are performed again, and after multiplication processing, the decoding feature of the i+1th level is obtained picture.
  • the fused feature map in the decoding stage is obtained, the fused feature map is subjected to decoding processing corresponding to the feature extraction of this level to obtain the decoding feature map corresponding to the feature extraction of this level, and the decoding feature map corresponding to the feature extraction of this level , to determine the target feature map corresponding to the feature extraction of this level, so as to facilitate the subsequent input of the target feature map corresponding to each level into the registration network.
  • the multi-level feature extraction method that is, the pyramid feature method
  • the parallel domain adaptation module and the gated fusion module are used in the encoding and decoding stages to enhance the generalization ability and robustness of the neural network and improve the abstraction ability of the feature extraction process.
  • the method for determining the transformation relationship information between the first original image and the second original image further includes:
  • S302 For each level of feature extraction, based on the first target feature map, the second target feature map corresponding to this level of feature extraction, and the first transformation relationship information corresponding to this level of feature extraction, determine the second transformation relationship corresponding to this level of feature extraction Information; wherein, the first transformation relationship information corresponding to this level of feature extraction includes: the second transformation relationship information corresponding to the previous level of feature extraction, or the original transformation relationship information between the first image and the second image.
  • the first transformation relationship information corresponding to this level of feature extraction includes: the second transformation relationship information corresponding to the previous level feature extraction, or the original transformation relationship information between the first original image and the second original image.
  • the original transformation relationship information is also called the initial deformation relationship.
  • the identity transformation can be used as the initial deformation relationship, that is, the initial image of the first original image and the second original image is used as input data ;
  • the previous linear registration network can solve the problem of poor accuracy and mismatch of Field of View (FoV) when the global deformation is too large.
  • FoV Field of View
  • the second target feature map and the first transformation relationship information corresponding to this level of feature extraction determine the level of feature extraction
  • the corresponding second transformation relationship information includes:
  • the iterative registration framework accepts an initial transformation relation in the form of a deformation field and several pairs of feature maps as input (each stage of registration corresponds to a stage of feature maps).
  • the iterative registration framework consists of multiple stages, and each stage includes a registration module and a combination module.
  • the registration module accepts the feature map pair of this stage and the transformation relationship of the previous stage (for the first stage, it is the initial transformation relationship) as input, and outputs the residual relative to the transformation relationship of the previous step, namely:
  • f( ⁇ ) is the registration module
  • ⁇ represents the applied transformation relationship
  • ⁇ i represents the cumulative transformation relationship in the i-th stage
  • ⁇ i represents the residual of ⁇ i relative to ⁇ i-1 .
  • the superscripts of ⁇ and ⁇ indicate the direction of the transformation relation, namely Represents the transformation relationship from the source image to the target image, Represents the transformation relationship from the target image to the source image.
  • the deformation field refers to adding the regular spatial grid to the predicted deformation field in the image registration to obtain the sampling grid, and using the sampling grid (sampling grid) containing deformation information for the floating image to obtain is the deformed image.
  • the size of the deformation field corresponding to a two-dimensional image with a size of [W, H] is [W, H, 2], where the size of the third dimension is 2, representing the displacement in the x-axis and y-axis directions, respectively.
  • the size of the deformation field corresponding to a three-dimensional image with a size of [D, W, H] is [D, W, H, 3], where the size of the third dimension is 3, respectively represented on the x-axis, y Axis and z-axis displacement.
  • Registration modules are divided into two types: linear transformation and deformation transformation, both of which are the same in input and output, so in application, you can freely combine linear and deformation transformation registration modules according to your needs, such as using a linear registration
  • the quasi-module is connected in series with three deformation transformation modules.
  • the two modules do not directly output ⁇ , but obtain ⁇ according to the transformation, that is, the output is the matrix corresponding to ⁇ .
  • the differential homeomorphism of the transformation process the calculation can be reduced The number of times, and because the output is a matrix, the process is reversible, providing a verification function.
  • the registration process proceeds as follows:
  • the original output of the linear registration module is the forward rotation, scaling, skew matrix Afw and the forward translation vector bfw.
  • Afw can be predicted directly through the network, or the parameters of rotation, scaling, and oblique cutting can be predicted separately, and then combined.
  • the reverse linear transformation relationship is obtained by inverting the linear transformation, namely:
  • the original output of the deformation transformation is the forward deformation relationship V fw in the form of velocity field, and on this basis, the deformation relationship ⁇ in the form of deformation field is obtained by integration, where the deformation relationship ⁇ satisfies the following formula (4) and formula (5) :
  • ⁇ represents the applied deformation relationship
  • the combination module accumulates the ⁇ of each previous step to obtain the cumulative deformation relationship ⁇ , namely (note that for forward and reverse deformations, the order of combination is reversed):
  • the registration neural network is used to perform multi-level fitting residuals, which not only has the advantage of fast registration speed of neural network learning, but also reduces the number of neural network errors. Learning the disadvantages of low accuracy when the global deformation is too large makes the registration speed and accuracy high.
  • the method for determining the transformation relationship information between the first original image and the second original image further includes:
  • the first original image and the second original image may be registered based on the second transformation relation information output after the last level of registration, ie final transformation information.
  • the last level of features can be used to extract the corresponding second transformation relationship information, and the first original image can be transformed to obtain the transformed image of the first original image , and then perform one-to-one position matching between the transformed image and the second original image to obtain a registration result.
  • the registration method involved is applied to a pre-trained registration neural network
  • the registration neural network includes two branch networks of a feature extraction neural network and a multi-level registration neural network
  • the feature extraction The neural network is used to perform multi-level feature extraction processing on the first original image and the second original image
  • the multi-level registration neural network is used to The multi-level features extracted from the image determine the target transformation relationship information between the first original image and the second original image, wherein the target transformation relationship information is used for the first original image and the second original image Two original images were registered.
  • the image processing method proposed by the embodiment of the present disclosure further includes:
  • S102 Based on the first attribute information and the second attribute information, match the first object instance and the second object instance to obtain the relationship between the first object instance and the second object instance match degree.
  • the first object instance and the second object instance may be matched based on the acquired attribute information. That is, based on the first attribute information and the second attribute information, determine at least one of the following matching information between the first object instance and the second object instance: similarity, matching priority, distance , the equivalent radius, and based on the matching information between the first object instance and the second object instance, the matching degree between the first object instance and the second object instance is obtained.
  • this embodiment of the present disclosure provides a method of obtaining the matching degree between the first object instance and the second object instance.
  • the matching information includes: similarity, matching priority, distance and equivalent radius ;
  • the obtaining the matching degree between the first object instance and the second object instance based on the matching information between the first object instance and the second object instance includes:
  • all possible object instances in the two images may be paired, and the similarity, matching priority, distance and equivalent radius between any two object instances may be calculated to form four matrices.
  • A represents the first object instance
  • a represents the second object instance, where:
  • Similarity is a measurement value. When the attributes of two object instances are closer, the value of similarity is greater. In order to be numerically controllable, it can be scaled or truncated to [0,1] or [-1, 1 Scope. Here, taking the radius of an object instance as an example, for the first object instance A and the second object instance a, the similarity can be:
  • the similarity of the radius is between [0,1], and the closer the two object instances are in the radius, the greater the similarity.
  • the similarity between other attribute information can also be calculated.
  • the attribute information includes: when the attribute information of the object instance includes a bounding box, the intersection-over-union (IoU) of the bounding boxes corresponding to the first object instance and the second object instance can be calculated, and This intersection is compared as the degree of similarity between the two; under the situation that the attribute information of object instance comprises contour, can calculate the Dika (Dice) coefficient of the contour corresponding to the first object instance and the second object instance respectively, and The Decca coefficient is used as the similarity between the two; when the attribute information of the object instance includes feature data, for example, in the case of a coded vector, the codes corresponding to the first object instance and the second object instance can be calculated respectively The cosine similarity of the vector, and the value of the cosine similarity is used as the similarity between the two, etc.
  • the similarities corresponding to multiple attributes can be weighted and summed, and the result of the weighted sum can be used as the similarity between the first object instance and the second object instance Spend.
  • the matching priority is also a kind of measurement value, and the higher the importance of the object instance, the greater its value.
  • the matching priority may represent the malignancy probability of the lesion, and the higher the malignancy probability and the larger the lesion, the higher the corresponding matching priority.
  • the matching priority can also be scaled and truncated to achieve numerical controllability. Taking the malignant probability of the lesion as an example, the matching priority can be expressed as:
  • the malignant probability is a value between [0,1]
  • the numerical distribution of the matching priority is between [0.5,1].
  • different matching priority metrics can be set for different attributes, or multiple matching priority metrics can be set for multiple attributes, or multiple matching priority metrics can be weighted, such as lesion deterioration speed, diffusion speed, and the like.
  • the distance is the distance between all possible paired object instances.
  • the Euclidean distance between the centers of the object instances can be used to represent the distance between the two object instances:
  • k represents the k-order norm
  • the second-order norm of the vector is the root sign after summing the squares of all components of the vector.
  • the equivalent radius is usually the average value of the radii or lengths of the diagonals of two object instances in a certain form.
  • its equivalent radius is The geometric mean of the radii of an object instance; when an object instance is expressed as a bounding box, its equivalent radius can be the length of a diagonal line, etc.: where, the equivalent radius is the geometric mean of the radii of two object instances It can be expressed as:
  • four matrices can be obtained, namely similarity matrix, matching priority matrix, distance matrix and equivalent radius matrix.
  • the i-th row and the j-th column represent the corresponding values of the i-th object instance in image 1 to the j-th object instance in image 2.
  • These four matrices can be used for further combinations in subsequent calculations.
  • the similarity matrix and matching priority matrix can be combined into a weighted similarity matrix.
  • the weighted similarity is the similarity after considering the matching priority, so the weighted similarity can be expressed as:
  • Weighted similarity similarity * matching priority.
  • the adjacency matrix is the relative positional relationship between two object instances calculated according to the distance and equivalent radius.
  • the adjacency relationship is used to measure whether two object instances are close or coincident in position. It is negatively related to the distance and positively related to the equivalent radius. For example, it can be expressed as follows:
  • a threshold can also be set. Due to the correction of the adjacency relationship, for example, it can be expressed as follows:
  • the final matching degree can be obtained.
  • the degree of matching determines how good or bad the match between two object instances is.
  • the degree of matching can be expressed as:
  • Matching degree weighted similarity * adjacency.
  • the first object instance Matching with the second object instance to obtain the matching degree between the first object instance and the second object instance includes:
  • the object instance pair For each object instance pair, according to the first attribute information corresponding to the first object instance included in the object instance pair and the second attribute information corresponding to the second object instance included in the object instance pair, the object instance pair The included first object instance is matched with the second object instance to obtain the matching degree of the object instance pair.
  • a plurality of first object instances and a plurality of second object instances may be grouped based on the corresponding matching degrees of the plurality of object instances and a preset matching degree threshold, so as to obtain multiple object instance groups; for each object instance group, if the object instance group includes at least a first object instance and at least one second object instance, the first object instance and the second object included in the object instance group Examples are merged.
  • the final matching result can be obtained from the matrix. That is, based on a threshold, select a pair whose matching degree is greater than the threshold as a feasible pairing, and transform the matrix into an undirected bipartite graph, each node on the graph is an object instance, and the weight of the edge is the matching degree; then According to the weight of the edge from high to low, pairing with replacement or without replacement is performed. Pairing with replacement will fuse all possible connected object instances into a new object instance; pairing without replacement will give priority to pairing two object instances with higher matching degrees. When an object instance is successfully paired, it will not be able to It is then paired with other object instances.
  • whether there is a return method is a configurable item, which is determined according to requirements.
  • one object instance can only be paired with another object instance at most, which is suitable for tracking the follow-up changes of pulmonary nodules, etc.; the pairing with replacement allows many-to-many matching, which is suitable for cross-modality Situations, such as tumor capsules that are not visualized in a particular modality, are not limited in the direction of application.
  • the successfully matched object instances will not continue to be matched, and can be removed from the pool of object instances that allow matching, but the object instances that have not yet been successfully matched. You can also continue to stay in the object instance pool and continue to wait for a match until all object instances in the object instance pool do not meet the matching conditions.
  • FIG. 4 is a schematic diagram of multi-object instance matching proposed by an embodiment of the present disclosure.
  • multiple object instance pairs that meet the threshold are screened out, including 0.7 between Aa, 0.6 between Ba, and 0.6 between Cc, and based on The methods with and without replacement get the final matching result.
  • the similarity, matching priority, position and size relationship, etc. are considered, and the matching accuracy is higher.
  • the image processing method proposed by the embodiment of the present disclosure further includes:
  • the first attribute information of the first object instance and the second attribute information of the second object instance may be fused.
  • attribute fusion can be performed.
  • image 1 is used as the reference space.
  • space-related attributes such as bounding box, center, contour, etc.
  • the coordinate transformation will be performed according to the registration relationship before fusion.
  • merging attributes different merging methods are used according to the type of attributes, which are not limited here. For example, for bounding boxes and outlines, etc., the union of all object instances is obtained; for dimensions, or based on the fused bounding box or outline. Calculate, or obtain the maximum value of all object instances; for the degree of malignancy, etc., obtain the maximum value of all object instances; for the hidden layer coding of the detection module, obtain the mean value of multiple object instances.
  • the coordinates and dimensions of the fused object instance are converted into coordinates and dimensions after registration according to the registration relationship, and the corresponding malignant degree is selected from the maximum value between object instance A and object instance a, and the feature The vectors are also averaged, thus completing the fusion process between the first object instance and the second object instance.
  • merging may be used for fusion to obtain a final result. For example, two-by-two object instances are fused, and then the two-by-two fusion results are fused; or, two object instances are arbitrarily selected for fusion, and the fusion result is fused with another object instance until the fusion with all object instances is completed.
  • the two merging methods depend on whether the multiple sources are in a parallel or hierarchical relationship.
  • multiple images can be linked, such as multi-modality, multi-phase contrast radiography, or multiple time-series follow-up results.
  • the matching method is more flexible, and it can better assist users in joint image reading and diagnosis.
  • the embodiments of the present disclosure are applied to the liver imaging diagnosis platform, for example.
  • Patients will be detected on the results of multi-phase angiography.
  • it can solve the problem of possible missed diagnosis in a single phase.
  • focal nodular hyperplasia is not obvious in the portal phase.
  • Metastatic tumors are often not obvious in the arterial phase, and the relationship of lesions in multiple phases is established for subsequent multi-phase combined diagnosis.
  • It can also be applied to the lung imaging diagnosis platform. According to multiple time-series lung images, nodule follow-up and analysis of nodule volume and sign changes are of great significance for the judgment of patients' condition.
  • the embodiments of the present disclosure first use a target detection algorithm to obtain the detection result corresponding to each image, that is, attribute information of an object instance, and then use a registration algorithm to obtain a spatial transformation relationship between any two images.
  • the similarity, matching priority, distance and equivalent radius of all possible detection target pairs in the two images are calculated sequentially, and the matching degree is calculated by combination to obtain the detection target matching of the two images
  • attribute fusion is performed again to obtain a fused object instance.
  • the matching degree between the first object instance and the object instance can be obtained, so that the first object instance and the second object instance can be matched according to the matching degree. more precise fusion.
  • the embodiment of the present disclosure also provides an image processing device corresponding to the image processing method. Since the problem-solving principle of the device in the embodiment of the present disclosure is similar to the above-mentioned image processing method in the embodiment of the present disclosure, the implementation of the device See the implementation of the method.
  • FIG. 5 is a schematic diagram of an image processing device provided by an embodiment of the present disclosure
  • FIG. 6 is a schematic diagram of another image processing device provided by an embodiment of the present disclosure
  • FIG. 7 It is a schematic diagram of a matching module in an image processing device provided by an embodiment of the present disclosure.
  • the device includes: a first determination module 510, a matching module 520, and a fusion module 530; wherein,
  • the first determination module 510 is configured to determine the first attribute information of the first object instance of the target object in the first image and determine the second attribute information of the second object instance of the target object in the second image;
  • the matching module 520 is configured to match the first object instance and the second object instance based on the first attribute information and the second attribute information to obtain the first object instance and the second object instance The degree of matching between object instances;
  • the fusion module 530 is configured to fuse the first object instance and the second object instance based on the matching degree.
  • An acquisition module 540 configured to acquire the first original image and the second original image
  • the second determination module 550 is configured to determine transformation relationship information between the first original image and the second original image
  • the transformation module 560 is configured to perform transformation processing on the first original image based on the transformation relationship information to obtain a first image, and use the second original image as the second image; or, based on the transformation relationship The information transforms the second original image to obtain a second image, and uses the first original image as the first image.
  • the attribute information includes at least one of the following:
  • the object instance includes: the first object instance and the second object instance.
  • the matching module 520 includes:
  • the determining unit 521 is configured to determine at least one item of the following matching information between the first object instance and the second object instance based on the first attribute information and the second attribute information: similarity, matching priority, distance, equivalent radius;
  • the matching unit 522 is configured to obtain a matching degree between the first object instance and the second object instance based on the matching information between the first object instance and the second object instance.
  • the matching unit 522 is configured as:
  • the matching module 520 is configured as:
  • the object instance pair For each object instance pair, according to the first attribute information corresponding to the first object instance included in the object instance pair and the second attribute information corresponding to the second object instance included in the object instance pair, the object instance pair The included first object instance is matched with the second object instance to obtain the matching degree of the object instance pair.
  • the fusion module 530 is configured to:
  • the object instance group For each object instance group, if the object instance group includes at least a first object instance and at least one second object instance, fuse the first object instance and the second object instance included in the object instance group.
  • the fusion module 530 is configured to:
  • the first attribute information of the first object instance and the second attribute information of the second object instance are fused.
  • the matching degree between the first object instance and the object instance is obtained by matching the attribute information of the first object instance and the second object instance, so that the first object instance and the second object instance can be matched according to the matching degree. Instances are more accurately fused.
  • FIG. 8 is a schematic structural diagram of the computer device provided by the embodiment of the present disclosure, including:
  • processor 801 memory 802, and bus 803; memory 802 is used to store execution instructions, including memory 8021 and external memory 8022; memory 8021 here is also called internal memory, and is used for temporarily storing the operation data in the processor 801, and The data exchanged by the external memory 8022 such as hard disk, the processor 801 exchanges data with the external memory 8022 through the memory 8021, when the computer device is running, the processor 801 communicates with the memory 802 through the bus 803, so that all The processor 801 executes the following instructions:
  • the first object instance and the second object instance are fused.
  • Embodiments of the present disclosure further provide a computer-readable storage medium, on which a computer program is stored, and when the computer program is run by a processor, the steps of the image processing method described in the foregoing method embodiments are executed.
  • the storage medium may be a volatile or non-volatile computer-readable storage medium.
  • the embodiment of the present disclosure also provides a computer program product, the computer program product carries a program code, and the instructions included in the program code can be used to execute the steps of the image processing method described in the above method embodiment, please refer to the above method implementation example.
  • the above-mentioned computer program product may be specifically implemented by means of hardware, software or a combination thereof.
  • the computer program product is embodied as a computer storage medium, and in another optional embodiment, the computer program product is embodied as a software product, such as a software development kit (Software Development Kit, SDK) etc. wait.
  • a software development kit Software Development Kit, SDK
  • the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place, or may be distributed to multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.
  • each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, each unit may exist separately physically, or two or more units may be integrated into one unit.
  • the functions are realized in the form of software function units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium executable by a processor.
  • the technical solution of the present disclosure is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including Several instructions are used to make a computer device (which may be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the methods described in various embodiments of the present disclosure.
  • the aforementioned storage media include: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), magnetic disk or optical disc and other media that can store program codes. .
  • Embodiments of the present disclosure provide an image processing method, device, computer equipment, storage medium, and program product, wherein the method includes: determining the first attribute information of the first object instance of the target object in the first image and the second attribute information of the second object instance of the target object in the second image; based on the first attribute information and the second attribute information, matching the first object instance and the second object instance, A matching degree between the first object instance and the second object instance is obtained; based on the matching degree, the first object instance and the second object instance are fused.
  • the matching degree between the first object instance and the object instance is obtained by matching the attribute information of the first object instance and the second object instance, so that the first object instance and the second object instance can be matched according to the matching degree. Instances are more accurately fused.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • Molecular Biology (AREA)
  • Biophysics (AREA)
  • Computational Linguistics (AREA)
  • Artificial Intelligence (AREA)
  • Evolutionary Computation (AREA)
  • General Health & Medical Sciences (AREA)
  • Biomedical Technology (AREA)
  • Computing Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Health & Medical Sciences (AREA)
  • Image Analysis (AREA)

Abstract

本公开实施例提供了一种图像处理方法、装置、计算机设备、存储介质及程序产品，其中，该方法包括：确定目标对象的第一对象实例在第一图像中的第一属性信息和所述目标对象的第二对象实例在第二图像中的第二属性信息；基于所述第一属性信息和所述第二属性信息，对所述第一对象实例和所述第二对象实例进行匹配，得到所述第一对象实例和所述第二对象实例之间的匹配度；基于所述匹配度，对所述第一对象实例和所述第二对象实例进行融合。本公开实施例通过对第一对象实例和第二对象实例的属性信息进行匹配，得到第一对象实例和对象实例之间的匹配度，从而可以根据匹配度，对第一对象实例和第二对象实例进行更加精确的融合。

Description

图像处理方法、装置、计算机设备、存储介质及程序产品
相关申请的交叉引用
本公开基于申请号为202111129299.7、申请日为2021年09月26日的中国专利申请提出，并要求该中国专利申请的优先权，该中国专利申请的全部内容在此以全文引用的方式引入本公开。
技术领域
本公开涉及图像处理技术领域，涉及但不限于一种图像处理方法、装置、计算机设备、存储介质及程序产品。
背景技术
目标检测相关算法可以从输入图像中检测出感兴趣区域，在医学图像分析，如病灶检测等领域有着广泛的应用。然而，现有提供医学图像确定病灶病程的过程往往涉及到多个图像来源，如随访需求多个时序图像，造影成像需要多个不同造影阶段图像，多模态融合诊断需要多个不同模态的图像等，这就涉及到对图像进行融合；当前的融合方式存在误差较大的问题。
发明内容
本公开实施例至少提供一种图像处理方法、装置、计算机设备、存储介质及程序产品。
第一方面，本公开实施例提供了一种图像处理方法，包括：
确定目标对象的第一对象实例在第一图像中的第一属性信息和确定所述目标对象的第二对象实例在第二图像中的第二属性信息；
基于所述第一属性信息和所述第二属性信息，对所述第一对象实例和所述第二对象实例进行匹配，得到所述第一对象实例和所述第二对象实例之间的匹配度；
基于所述匹配度，对所述第一对象实例和所述第二对象实例进行融合。
这样，可以更高精度地对第一对象实例以及第二对象实例进行融合。
一种可选的实施方式中，还包括：获取第一原始图像和第二原始图像；
确定所述第一原始图像和所述第二原始图像之间的变换关系信息；
基于所述变换关系信息对所述第一原始图像进行变换处理，得到第一图像，并将所述第二原始图像作为所述第二图像；
或者，
基于所述变换关系信息对所述第二原始图像进行变换处理，得到第二图像，并将所述第一原始图像作为所述第一图像。
这样，可以对第一图像与第二图像进行配准，便于后续的处理过程。
一种可选的实施方式中，所述第一属性信息或所述第二属性信息包括以下至少一种：
对象实例的位置信息、尺寸信息、所述对象实例属于所述目标对象的概率、特征数据、与所述对象实例对应的图像区域的灰度数据、影像组学信息；
所述对象实例包括：所述第一对象实例和所述第二对象实例。
这样，基于多种属性信息，可以提高对象实例之间的匹配精度。
一种可选的实施方式中，所述基于所述第一属性信息和所述第二属性信息，对所述第一 对象实例和所述第二对象实例进行匹配，包括：
基于所述第一属性信息和所述第二属性信息，确定所述第一对象实例和所述第二对象实例之间的下述至少一项匹配信息：相似度、匹配优先级、距离、等效半径；
基于所述匹配信息，得到所述第一对象实例和所述第二对象实例之间的匹配度。
一种可选的实施方式中，所述基于所述匹配信息，得到所述第一对象实例和所述第二对象实例之间的匹配度，包括：
将所述匹配优先级作为权值，对所述第一对象实例和所述第二对象实例之间的相似度进行加权处理，得到所述第一对象实例和所述第二对象实例之间的加权相似度；以及
基于所述距离和等效半径，确定所述第一对象实例和所述第二对象实例之间的邻接关系信息；
基于所述加权相似度和所述邻接关系信息，确定所述第一对象实例和所述第二对象实例之间的匹配度。
这样，可以更加精准地得到第一对象实例和第二对象实例之间的匹配度。
一种可选的实施方式中，所述第一对象实例、和所述第二对象实例均有多个；
所述基于所述第一属性信息和所述第二属性信息，对所述第一对象实例和所述第二对象实例进行匹配，得到所述第一对象实例和所述第二对象实例之间的匹配度，包括：
基于多个所述第一对象实例、和多个所述第二对象实例，构成多个对象实例对；
针对每个对象实例对，根据该对象实例对中包括的第一对象实例对应的第一属性信息和该对象实例对中包括的第二对象实例对应的第二属性信息，对该对象实例对中包括的第一对象实例和第二对象实例进行匹配，得到该对象实例对的匹配度。
这样，可以筛选出匹配度满足一定条件的对象实例对，可以提高融合的精度。
一种可选的实施方式中，基于所述匹配度，对所述第一对象实例和所述第二对象实例进行融合，包括：
基于多个对象实例对分别对应的匹配度和预设的匹配度阈值，对多个所述第一对象实例、和多个所述第二对象实例进行分组，得到多个对象实例组；
针对各个对象实例组，在该对象实例组中包括至少第一对象实例和至少一个第二对象实例的情况下，将该对象实例组中包括的第一对象实例和第二对象实例进行融合。
一种可选的实施方式中，所述对所述第一对象实例和所述第二对象实例进行融合，包括：
对所述第一对象实例的第一属性信息、和所述第二对象实例的第二属性信息进行融合。
第二方面，本公开实施例提供了一种图像处理装置，包括：
第一确定模块，配置为确定目标对象的第一对象实例在第一图像中的第一属性信息和确定所述目标对象的第二对象实例在第二图像中的第二属性信息；
匹配模块，配置为基于所述第一属性信息和所述第二属性信息，对所述第一对象实例和所述第二对象实例进行匹配，得到所述第一对象实例和所述第二对象实例之间的匹配度；
融合模块，配置为基于所述匹配度，对所述第一对象实例和所述第二对象实例进行融合。
一种可选的实施方式中，还包括：
获取模块，配置为获取第一原始图像和第二原始图像；
第二确定模块，配置为确定所述第一原始图像和所述第二原始图像之间的变换关系信息；
变换模块，配置为基于所述变换关系信息对所述第一原始图像进行变换处理，得到第一图像，并将所述第二原始图像作为所述第二图像；或者，基于所述变换关系信息对所述第二原始图像进行变换处理，得到第二图像，并将所述第一原始图像作为所述第一图像。
一种可选的实施方式中，所述属性信息包括以下至少一种：
对象实例的位置信息、尺寸信息、所述对象实例属于所述目标对象的概率、特征数据、与所述对象实例对应的图像区域的灰度数据、影像组学信息；
所述对象实例包括：所述第一对象实例和所述第二对象实例。
一种可选的实施方式中，所述匹配模块包括：
确定单元，配置为基于所述第一属性信息和所述第二属性信息，确定所述第一对象实例 和所述第二对象实例之间的下述至少一项匹配信息：相似度、匹配优先级、距离、等效半径；
匹配单元，配置为基于所述匹配信息，得到所述第一对象实例和所述第二对象实例之间的匹配度。
一种可选的实施方式中，所述匹配单元配置为：
将所述匹配优先级作为权值，对所述第一对象实例和所述第二对象实例之间的相似度进行加权处理，得到所述第一对象实例和所述第二对象实例之间的加权相似度；以及
基于所述距离和等效半径，确定所述第一对象实例和所述第二对象实例之间的邻接关系信息；
基于所述加权相似度和所述邻接关系信息，确定所述第一对象实例和所述第二对象实例之间的匹配度。
一种可选的实施方式中，所述第一对象实例、和所述第二对象实例均有多个；
所述匹配模块配置为：
基于多个所述第一对象实例、和多个所述第二对象实例，构成多个对象实例对；
针对每个对象实例对，根据该对象实例对中包括的第一对象实例对应的第一属性信息和该对象实例对中包括的第二对象实例对应的第二属性信息，对该对象实例对中包括的第一对象实例和第二对象实例进行匹配，得到该对象实例对的匹配度。
一种可选的实施方式中，所述融合模块配置为：
基于多个对象实例对分别对应的匹配度和预设的匹配度阈值，对多个所述第一对象实例、和多个所述第二对象实例进行分组，得到多个对象实例组；
针对各个对象实例组，在该对象实例组中包括至少第一对象实例和至少一个第二对象实例的情况下，将该对象实例组中包括的第一对象实例和第二对象实例进行融合。
一种可选的实施方式中，所述融合模块配置为：
对所述第一对象实例的第一属性信息、和所述第二对象实例的第二属性信息进行融合。
第三方面，本公开实施例还提供一种计算机设备，包括：处理器、存储器和总线，所述存储器存储有所述处理器可执行的机器可读指令，当计算机设备运行时，所述处理器与所述存储器之间通过总线通信，所述机器可读指令被所述处理器执行时执行上述第一方面，或第一方面中任一种可能的实施方式中的步骤。
第四方面，本公开实施例还提供一种计算机可读存储介质，该计算机可读存储介质上存储有计算机程序，该计算机程序被处理器运行时执行上述第一方面，或第一方面中任一种可能的实施方式中的步骤。
第五方面，本公开实施例还提供一种计算机程序产品，该计算机程序产品包括存储了计算机程序的非瞬时性计算机可读存储介质，该计算机程序产品可以为一个软件安装包，上述计算机程序可操作来使计算机执行上述第一方面，或第一方面中任一种可能的实施方式中的步骤。
本公开实施例提供的图像处理方法、装置、计算机设备、存储介质及程序产品，通过对第一对象实例和第二对象实例的属性信息进行匹配，得到第一对象实例和第二对象实例之间的匹配度，并基于该匹配度，对第一对象实例和第二对象实例进行融合。这样，通过对第一对象实例和第二对象实例的属性信息进行匹配，得到第一对象实例和对象实例之间的匹配度，从而可以根据匹配度，对第一对象实例和第二对象实例进行更加精确的融合。
为使本公开的上述目的、特征和优点能更明显易懂，下文特举较佳实施例，并配合所附附图，作详细说明如下。
附图说明
为了更清楚地说明本公开实施例的技术方案，下面将对实施例中所需要使用的附图作简单地介绍，此处的附图被并入说明书中并构成本说明书中的一部分，这些附图示出了符合本公开的实施例，并与说明书一起用于说明本公开的技术方案。应当理解，以下附图仅示出了 本公开的某些实施例，因此不应被看作是对范围的限定，对于本领域普通技术人员来讲，在不付出创造性劳动的前提下，还可以根据这些附图获得其他相关的附图。
图1示出了本公开实施例所提供的一种图像处理方法的流程图；
图2示出了本公开实施例提出的图像融合的示意图；
图3示出了本公开实施例提供一种确定第一原始图像和第二原始图像之间的变换关系信息的方法的流程图；
图4示出了本公开实施例提出的多对象实例匹配的示意图；
图5示出了本公开实施例所提供的一种图像处理装置的示意图；
图6示出了本公开实施例所提供的另一种图像处理装置的示意图；
图7示出了本公开实施例所提供的图像处理装置中，匹配模块的示意图；
图8示出了本公开实施例所提供的一种计算机设备的示意图。
具体实施方式
为使本公开实施例的目的、技术方案和优点更加清楚，下面将结合本公开实施例中附图，对本公开实施例中的技术方案进行清楚、完整地描述，显然，所描述的实施例仅仅是本公开一部分实施例，而不是全部的实施例。通常在此处附图中描述和示出的本公开实施例的组件可以以各种不同的配置来布置和设计。因此，以下对在附图中提供的本公开的实施例的详细描述并非旨在限制要求保护的本公开的范围，而是仅仅表示本公开的选定实施例。基于本公开的实施例，本领域技术人员在没有做出创造性劳动的前提下所获得的所有其他实施例，都属于本公开保护的范围。
应注意到：相似的标号和字母在下面的附图中表示类似项，因此，一旦某一项在一个附图中被定义，则在随后的附图中不需要对其进行进一步定义和解释。
本文中术语“和/或”，仅仅是描述一种关联关系，表示可以存在三种关系，例如，A和/或B，可以表示：单独存在A，同时存在A和B，单独存在B这三种情况。另外，本文中术语“至少一种”表示多种中的任意一种或多种中的至少两种的任意组合，例如，包括A、B、C中的至少一种，可以表示包括从A、B和C构成的集合中选择的任意一个或多个元素。
经研究发现，对于不同来源的图像，目标检测算法的检出性能和结果可能不同，如随访中病灶可能会出现消失或新增，不同造影阶段病灶的表现不同等；此外，同一个病人在拍摄不同图像时，存在空间上的位移和自身背景情况的转变；这两方面因素使得在不同来源上的检测结果无法简单地组合。对于多源图像的检测，目前的解决方案主要有两种：其一是在各个来源的图像上各自做检测，由阅片者自行建立联系；其二是使用配准算法建立两图像的空间关系，再从一个来源映射至另一个来源。第一种方法没有解决多源图像检测的问题，第二种方法部分解决了问题，但当前的多源图像融合的方式一般是将配准后的图像进行直接叠加；这种图像的融合方式造成融合精度差的问题。
基于上述研究，本公开实施例提供了一种图像处理方法，通过对第一对象实例和第二对象实例的属性信息进行匹配，得到第一对象实例和对象实例之间的匹配度，从而可以根据匹配度，对第一对象实例和第二对象实例进行更加精确的融合。
为便于对本实施例进行理解，首先对本公开实施例所公开的一种图像处理方法进行详细介绍，本公开实施例所提供的图像处理方法的执行主体一般为具有一定计算能力的计算机设备。在一些可能的实现方式中，该图像处理方法可以通过处理器调用存储器中存储的计算机可读指令的方式来实现。
参见图1所示，图1为本公开实施例提供的一种图像处理方法的流程图，所述方法包括步骤S101～S103，其中：
S101：确定目标对象的第一对象实例在第一图像中的第一属性信息和确定所述目标对象的第二对象实例在第二图像中的第二属性信息。
其中，第一图像以及第二图像为对同一目标对象进行拍摄得到的图像，示例性的，第一 图像和第二图像为不同时刻对目标对象进行拍摄得到的图像；或者，为在不同角度对目标对象进行拍摄得到的图像。示例性的，在医疗领域中，目标对象可以指同一患者的同一病变器官、或者同一身体部位等；第一图像和第二图像可以为针对同一器官或者同一身体部位在同一时刻拍摄的角度、距离等拍摄参数不同的多张图像，又或者可以为同一器官或者同一身体部位在不同时刻拍摄的多张图像。例如，在同一患者的一次检查过程中，为了能够全方位的获取病灶的图像，基于不同的拍摄角度拍摄的多张图像；或者，为了观察到病灶的发展进程，在同一患者的多次检查过程中，针对同一病灶拍摄的多张图像。对应的，所述第一对象实例和第二对象实例指的是分别在第一图像及第二图像上可能存在的一个病灶。
所述属性信息包括以下至少一种：
对象实例的位置信息、尺寸信息、所述对象实例属于所述目标对象的概率、特征数据、与所述对象实例对应的图像区域的灰度数据、影像组学信息；
所述对象实例包括：第一图像中的所述第一对象实例和第二图像中的所述第二对象实例。
在本公开一个实施例中，对象实例是经由目标检测算法进行检测得到的结果，并可能对检测结果进行了一定的后处理与分析，得到对象实例及其对应的属性信息。其中，目标检测算法可以是任意的从图像中检测特定目标的方法，在此不作限定，当检测两个图像时，其对应的目标检测算法可以不同，但检测的目标类型需要相同，能够得到相同类型的属性信息。
示例性的，所述目标检测算法可以为Mask RCNN，视网膜网络(Retina Net)等。
在本公开实施例中，可以通过上述目标检测算法对图像进行检测，从而得到对象实例对应的属性信息。
示例性的，在多种可能的属性信息中，位置信息可以为对象实例在图像中的坐标信息，即(x，y)，可以是包围框的中心点对应的坐标信息；尺寸信息可以为对象实例的大小、半径等表征对象实例尺寸的参数信息；所述对象实例属于所述目标对象的概率可以为基于一定衡量标准确定的、对象实例具有目标属性的概率或是程度，例如一病灶对应的恶性程度等；特征数据可以为隐含层编码，即检测网络里某一网络提取出来与的对象实例对应的特征向量；与所述对象实例对应的图像区域的灰度数据指的是图像区域中各个像素点分别对应的灰度值；影像组学信息是根据对象实例的图像和范围，使用影像组学方法进行特征描述得到的描述量。此外，还可能包括其他的属性信息。
在本公开另一实施例中，还可以对所述图像进行目标检测处理，其中，该目标检测方法可以是利用语义分割的方法，即确定图像中每一个像素归属于哪一类图像，并基于所述目标检测处理的结果，得到所述对象实例在所述图像中的位置信息和所述对象实例属于所述目标对象的概率，以及基于所述对象实例在所述图像中的位置，确定与所述对象实例对应的图像区域的灰度数据、和/或影像组学信息。
在本公开另一实施例中，在确定对象实例的属性信息之前，还包括：
获取第一原始图像和第二原始图像；
确定所述第一原始图像和所述第二原始图像之间的变换关系信息；
基于所述变换关系信息对所述第一原始图像进行变换处理，得到第一图像，并将所述第二原始图像作为所述第二图像；
或者，
基于所述变换关系信息对所述第二原始图像进行变换处理，得到第二图像，并将所述第一原始图像作为所述第一图像。
其中，所述第一原始图像、第二原始图像分别为针对目标对象拍摄的原始图像。由于第一原始图像和第二原始图像之间存在一定的位置偏差，例如第二原始图像相对于第一原始图像整体平移10mm等，因此，需要对第一原始图像和第二原始图像进行配准，以使第一原始图像和第二原始图像在空间上处于相同的位置，便于后期属性信息的提取与匹配。
在一些实施例中，可以基于第一原始图像的位置信息，对第二原始图像进行配准，得到第一图像以及变换处理后的第二图像，也可以基于第二原始图像的位置信息，对第一原始图像进行配准，得到第二图像以及变换处理后的第一图像。
参照图2所示，图2为本公开实施例提出的图像融合的示意图。在图2中，可以得出在对对象实例a基于对象实例A进行配准之后，将对象实例a对应的全局空间关系2调整为与对象实例A相同的全局空间关系1，其对应的坐标、尺寸均发生改变，与空间无关的其他属性则不变。
参见图3所示，为本公开实施例提供一种确定第一原始图像和第二原始图像之间的变换关系信息的方法的流程图，该方法包括：
S301：对第一图像进行多级特征提取，得到与所述多级特征提取分别对应的第一目标特征图；以及对第二图像进行多级特征提取，得到与所述多级特征提取分别对应的第二目标特征图。
其中，所述第一原始图像、第二原始图像可以包括在不同时刻、或者不同角度针对同一对象拍摄的不同图像。例如，将本公开实施例提供的图像配准方法应用于医学领域中时，第一原始图像、第二原始图像可以为针对同一病人的同一病灶的一次扫描过程中的多次拍摄得到的医学影像，又或者，可以为在不同的时间对病人的同一病灶进行扫描的过程中，得到的不同医学影像，在此不作限定。
在本公开实施例中，在确定出第一原始图像以及第二原始图像之后，便可以分别对两个图像进行多级特征提取，得到多级特征提取分别对应的第一目标特征图以及第二目标特征图。其中，多级特征提取例如可以采用神经网络自学习的方法，实现图像多级特征的梯度回传，提取出图像的高层语义特征。
在一些实施例中，对图像进行多级特征提取，得到与所述多级特征提取分别对应的特征图，包括：
针对多级特征提取中的每级特征提取，确定该级特征提取的第一输入数据和第二输入数据；所述第一输入数据包括：所述图像、或者后一级特征提取输出的编码特征图；所述第二输入数据包括：前一级特征提取输出的解码特征图；
对所述第一输入数据进行与该级特征提取对应的编码处理，得到该级特征提取对应的编码特征图；
对该级特征提取对应的编码特征图和所述第二输入数据进行融合处理，得到融合特征图；
对所述融合特征图进行与该级特征提取对应的解码处理，得到该级特征提取对应的解码特征图；
将该级特征提取对应的解码特征图，确定为该级特征提取对应的目标特征图。
在该示例中，可以通过特征提取器分别对第一原始图像和第二原始图像进行特征提取，得到两个图像分别对应的多级特征。特征提取器是一个特征金字塔网络，分为编码和解码两个阶段，并以跳跃连接的方法连接低层和高层网络。特征提取器接收一个图像作为输入数据，在编码模块逐层下采样(又称“降采样”)和提取特征，在解码模块逐层上采样(又称“升采样”)和提取特征。每一个解码模块的输出都会送入迭代配准网络，形成多层、从粗至细的金字塔特征。
对于编码特征提取过程来说，针对于最高级别、即特征“金字塔”网络的“从粗到细”结构中最“粗”的一级，其对应的输入数据为原始的图像，即第一原始图像或是第二原始图像，对于其他级别，其对应的输入数据为对应上一级输出的编码特征图，由此可以实现对于编码特征提取的迭代过程。对于解码特征提取过程来说，其最低级别，即特征“金字塔”网络的“从粗到细”结构中最“细”的一级，是通过对应的最低级别的编码特征图得到的解码特征图，对于其他级别，其对应的输入数据为对应下一级输出的解码特征图，并最终输出针对原始图像的特征提取结果，由此可以实现对于解码特征提取的迭代过程。
本公开实施例所述的多级特征提取，指示从“细”到“粗”的多级特征提取过程，每级特征提取，包括与该级特征提取对应的编码过程和解码过程，也即，从“金字塔”网络的底部到顶部。
在该示例中，每级特征提取包括与该级特征提取对应的编码网络和与该级特征提取对应的解码网络；
其中，每个编码网络包括：编码模块和并行域适应模块。
第一级编码网络对应的解码网络，包括：解码模块。
除第一级编码网络外的其他编码网络对应的解码网络包括：解码模块和门控融合模块。编码过程是从“粗”到“细”的过程，而解码过程是从“细”到“粗”依次进行解码，本公开实施例所述的多级特征提取，与解码过程相对应。
其中，针对各级特征提取中的编码网络，其用于对所述第一输入数据进行与该级特征提取对应的编码处理，得到该级特征提取对应的编码特征图。
针对各级特征提取中的解码网络，其用于对该级特征提取对应的编码特征图和所述第二输入数据进行融合处理，得到融合特征图，并对所述融合特征图进行与该级特征提取对应的解码处理，得到该级特征提取对应的解码特征图。
在确定出在编码过程中每级对应的第一输入数据之后，便可以基于所述第一输入数据进行编码处理，得到每级对应的编码特征图。
在本公开的一个实施例中，所述对所述第一输入数据进行与该级特征提取对应的编码处理，得到该级特征提取对应的编码特征图，包括：
对所述第一输入数据进行下采样处理，得到下采样特征图；
对所述下采样特征图进行通道注意力处理，得到下采样特征图中各个数据通道分别对应的注意力权重；
基于所述下采样特征图和所述注意力权重，得到所述编码特征图。
在第一输入数据输入至每级特征提取中的编码模块时，编码模块会对第一输入数据进行下采样处理，得到对应的下采样特征图，并将输出的下采样特征图输入至每级编码模块对应的并行域适应模块中。其中，该并行域适应模块可以在编码阶段增强编码网络对于某些特定特征的表达能力；例如图像包括人体的器官；则可以通过训练特征提取器，以增强对器官纹理特征的表达能力。将下采样特征图输入至并行域适应模块后，并行域适应模块对下采样特征图进行通道注意力处理，得到下采样特征图中各个数据通道分别对应的注意力权重。上述过程包括：
对所述下采样特征图进行全局平均池化处理，得到第一特征子图；
基于所述第一特征子图，确定各个数据通道分别对应的候选注意力权重；以及基于所述第一特征子图，确定各个数据通道分别对应的候选注意力的特征域权重；
基于各个数据通道分别对应的候选注意力权重和各个数据通道分别对应的候选注意力的特征域权重，得到所述下采样特征图中各个数据通道分别对应的注意力权重。
在本公开实施例中，并行域适应模块包含两个机制：通道注意力机制和域感知机制。在并行域适应模块中，通道注意力机制通过分组卷积的形式、确定了多个通道的通道注意力权重，从而生成多个候选通道权重。域感知机制可以根据特征图的性质，对多个候选通道权重进行组合，得到最后的结果，从而使得第一原始图像和第二原始图像在图像域的表现差异很大时，也能通过特征提取器得到相近的特征，便于后续的配准过程。
在本实施例中，所述并行域适应模块应用于多级特征提取的编码阶段，例如，在由第i+1级至第i级的编码过程中，应用对应的并行域适应模块，例如，在第3级至第2级的编码过程中，应用并行域适应模块3。如图4所示，为本公开实施例所提供的一种并行域适应模块的示例。在这里，假设第i层的编码模块对应的下采样特征图表示为(H，C，D，H，W)，其中，N表示一次特征提取中包括的特征图数量(Number of instances in batch)，C表示特征图的通道数(Channel)，D，H，W分别表示特征图的长宽高(Depth，Height，Width)，B表示通道注意力的支数(Branches of attention)。在将编码特征图输入至该并行域适应模块后，在通过编码模块获取到下采样特征图之后，便可以对该下采样特征图进行全局平均池化处理(Global Average Pooling)，即对特征图进行降维，在长、宽、高这几个空间维度上对所有像素求平均，用于对数据进行降维，使得特征图从(N，C，D，H，W)变成(N，C)，由此可以提取通道维度的总体信息，得到第一特征子图。之后，分别通过两个分支对该第一特征子图进行重复(Repeat)和压平(Flatten)处理，在第一分支中，将第一特征子图进行重复处理 之后，该第一特征子图转变为(N，BC，1)的形式，其中，重复是指把降维后的数据重复B次，对应B个通道注意力，这里重复操作是为了改变数据的格式方便计算。之后，将该形式的特征图进行B组卷积、激活再进行B组卷积的处理过程，其中，所述激活函数可以采用线性整流函数(Rectified Linear Unit，ReLU)，最后，对卷积得到的特征图进行通道重组，得到(N，C，B)的特征图，由此便可以获得各个数据通道分别对应的候选注意力权重。在第二分支中，在将第一特征子图进行压平处理之后，该第一特征子图转变为(N，C)的形式，其中，压平是指重新排列数据的顺序，这里压平操作是为了改变数据的格式方便计算。之后，将该形式的特征图进行全连接以及激活处理，其中，可以分别采用ReLU激活函数，以及深度学习输出层的激励函数(Softmax)，得到(N，B)形式的特征图，由此便可以得到各个数据通道分别对应的候选注意力的特征域权重。在获取到各个数据通道分别对应的候选注意力权重以及特征域权重之后，将上述两个数据进行内积处理，得到(N，C)的特征数据，之后对该数据进行激活处理，其中，该激活函数可以为S型生长曲线(Sigmoid)，这样便可以得到第i级的编码特征图。
在确定出每个通道对应的候选注意力的特征域权重之后，便可以基于各个数据通道分别对应的候选注意力的特征域权重，确定出下采样特征图中各个数据通道分别对应的注意力权重。以人体肝脏的多个图像为例，针对同一病灶，由于扫描拍摄时间有所不同，或是拍摄的角度不同，造成虽然针对的是同一位置，但是呈现出图像有所不同，因此，基于通过域感知模块确定的特征域权重，对各个数据通道分别对应的候选注意力的特征域权重进行配比，突出目标位置所占的权重，即肝脏位置的所占权重，弱化其他位置，如肌肉、血液等位置所占的权重，这样，便可以对于目标位置就那些突出展示，由此，即便在第一原始图像和第二原始图像的展示效果相差较大的情况下，也可以通过特征提取器得到相近的特征，从而进行配准。
这里，通道注意力机制可以采用多组并行的通道注意力权重的确定方法，例如，假设存在12个通道，通常情况下，可以对上述12个通道分别进行卷积，得到12个通道对应的候选注意力权重，为了提高处理速度，可以将该12个通道分为三支通道注意力，每一支中包括4个通道，这样便可以同时对三支通道注意力进行卷积处理，提高了处理速度。
在确定出下采样特征图中各个数据通道分别对应的注意力权重之后，便可以基于下采样特征图和注意力权重，得到编码特征图，即对下采样特征图中各通道的特征按照对应的注意力权重进行重组，得到各级对应的编码特征图。
在与各级特征提取对应的解码过程中，除了输入至每一级解码模块的第二输入数据之外，与之对应的编码特征图也参与了编码特征图的生成过程。
这里，由于编码特征图拥有较高的空间分辨率，但语义信息的表达程度较低，前一个解码模块的上采样特征图的空间分辨率较低，但语义信息的表达程度较高，因此，为了结合编码特征图前一个解码模块的上采样特征图所各自具备的优势，本公开实施例中，在解码阶段采用了改善高层低分辨率特征和低层高分辨率特征融合效果的门控融合模块。
在一些实施例中，所述对该级特征提取对应的编码特征图和所述第二输入数据进行融合处理，得到融合特征图，包括：
基于该级特征提取对应的编码特征图和所述第二输入数据，得到该级特征提取对应的编码特征图中各个特征点分别对应的权重；
基于该级特征提取对应的编码特征图中各个特征点分别对应的权重和所述编码特征图，得到第二特征子图；
将所述第二特征子图和所述第二输入数据进行拼接，得到所述融合特征图。
示例性的，通过门控融合模块，将该级解码模块对应的编码模块数据的编码特征图与输入至该解码模块的第二特征数据进行融合，得到该级特征提取对应的编码特征图所占的权重，将编码特征图乘上该权重，得到第二特征子图，并与输入至该解码模块的第二特征数据进行拼接(Concatenate)，便可以得到融合特征图。这样，来自编码模块的编码特征图在乘权重后，与解码模块的解码特征图在通道维度拼接，送入下一个解码模块，使得两个来源的特征 图能更有效率地融合。
在公开实施例中，门控融合模块的作用主要是确定出编码特征图中各个特征点分别对应的权重，所述基于该级特征提取对应的编码特征图和所述第二输入数据，得到该级特征提取对应的编码特征图中各个特征点分别对应的权重，包括：
将该级特征提取对应的编码特征图和所述第二输入数据进行拼接，得到第三特征子图；
对所述第三特征子图进行卷积处理，得到第四特征子图，并基于所述第四特征子图，得到所述编码特征图、与所述第二输入数据的局域自相关系数；
基于所述局域自相关系数，得到该级特征提取对应的编码特征图中各个特征点分别对应的权重。
其中，所述第三特征子图中任一特征点的特征值表征与该特征点对应的图像区域的自相关系数。
在获取到基于该级特征提取对应的编码特征图以及第二输入数据拼接得到的第三特征子图之后，对该第三特征子图进行卷积、归一化以及基于线性整流函数(Rectified Linear Unit，ReLU)的激活处理，得到与第三特征子图对应的第四特征子图，这里卷积、归一化以及激活处理是卷积神经网络(Convolutional Neural Network，CNN)中的一种操作模型(pattern)，上述操作出现的次数代表了网络的深度，越深的网络表达能力越强，同时参数量也越大，这里，为了提高网络的表达能力，可以选取两层或两层以上的卷积、归一化以及激活处理。
在确定出第四特征子图之后，便可以基于第四特征子图，得到所述编码特征图、与所述第二输入数据的局域自相关系数，包括：
对所述第四特征子图进行通道维最大处理，得到所述第四特征子图的通道维最大值；以及对所述第四特征子图进行通道维平均处理，得到所述第四特征子图的通道维平均值；
对所述通道维最大值以及所述通道维平均值进行拼接处理，得到通道维拼接结果，并对所述通道维拼接结果进行卷积以及归一化处理，得到所述编码特征图、与所述第二输入数据的局域自相关系数。
其中，通道维最大和通道维平均均是针对于数据的降维操作，其基于通道维进行操作的，目的是减少参数量，保留空间上的特征，使得数据从(N，C，D，H，W)变为(N，1，C，H，W)，这里，通道维最大值以及通道维平均值均可表示为(N，1，C，H，W)。在确定出通道维最大值以及通道维平均值之后，便可以将上述两个数值进行拼接处理，得到拼接后的拼接结果(N，2，C，H，W)，之后，便可以对该拼接结果进行卷积以及归一化处理，得到所述编码特征图、与所述第二输入数据的局域自相关系数。
在获取到局域自相关系数之后，可以利用激活函数(sigmoid)对自相关系数进行激活处理，得到所述编码特征图中各个特征点分别对应的门控激活值；所述门控激活值用于表征所述编码特征图中各个特征点对应的权重。这样，在编码特征图乘上通过激活函数得到的门控激活值之后，再与第二输入数据进行拼接，便可以得到所述融合特征图。
本公开实施例还提供一种门控融合模块的示例。在该示例中，将第i级编码模块输出的加编码特征图以及第i级解码模块进行下采样之后数据的解码特征图进行拼接，其中，拼接是指在通道维进行拼接，要求其他通道维的大小一样，如两个数据分别为(N，C1，D，H，W)和(N，C2，D，H，W)，在拼接后的大小为(N，C1+C2，D，H，W)。之后对拼接后的特征图就那些卷积、归一化以及激活处理，并且再对其进行通道维最大以及通道维平均处理。在得到通道维最大值以及通道维平均值之后，对上述两个数据进行拼接，并且再次进行卷积、归一化以及激活处理，在进行相乘处理之后，得到第i+1级的解码特征图。
在获取到解码阶段的融合特征图之后，对所述融合特征图进行与该级特征提取对应的解码处理，得到该级特征提取对应的解码特征图，并将该级特征提取对应的解码特征图，确定为该级特征提取对应的目标特征图，便于后续将每级对应的目标特征图输入至配准网络当中。
由此，在本公开实施例中，使用多级特征提取的方法，即采用金字塔式特征方法，对于神经网络复用程度高，迭代配准速度快。同时，在编码和解码阶段使用并行域适应模块和门控融合模块，增强神经网络的泛化能力和鲁棒性，提升特征提取过程的抽象能力。
承接于上述S301，确定第一原始图像和第二原始图像之间的变换关系信息的方法还包括：
S302：针对每级特征提取，基于该级特征提取对应的第一目标特征图、第二目标特征图和该级特征提取对应的第一变换关系信息，确定该级特征提取对应的第二变换关系信息；其中，该级特征提取对应的第一变换关系信息，包括：前一级特征提取对应的第二变换关系信息、或者第一图像和第二图像之间的原始变换关系信息。
其中，该级特征提取对应的第一变换关系信息，包括：前一级特征提取对应的第二变换关系信息、或者第一原始图像和第二原始图像之间的原始变换关系信息。
示例性的，所述原始变换关系信息又称初始形变关系，当全局的形变较小时，可以使用恒等变换作为初始形变关系，即采用第一原始图像以及第二原始图像的初始图像作为输入数据；当存在较大的全局形变时，则可以使用传统方法中常用的初始化方法，如：基于图像的中心或灰度重心初始化位移，使用灰度的矩估计初始化旋转等。这样，可以解决此前的线性配准网络遇到全局形变过大时的精度不佳乃至视野(Field of View,FoV)不匹配的问题。
在一些实施例中，所述针对每级特征提取，基于该级特征提取对应的第一目标特征图、第二目标特征图和该级特征提取对应的第一变换关系信息，确定该级特征提取对应的第二变换关系信息，包括：
基于所述第一目标特征图、所述第二目标特征图和所述第一变换关系信息，确定该级特征提取相对于上一级特征提取之间的变换残差；
基于所述变换残差和所述第一变换关系信息，确定该级特征提取对应的第二变换关系信息。
其中，迭代配准框架接受一个形变场形式的初始变换关系和若干对特征图作为输入(每一个阶段的配准对应一个阶段的特征图)。迭代配准框架包含多个阶段，每个阶段包括一个配准模块和一个组合模块。配准模块接受该阶段的特征图对，和上一阶段的变换关系(对于第一阶段来说，就是初始变换关系)作为输入，输出相对于上一步变换关系的残差，即：
Figure PCTCN2022082195-appb-000001
其中f(□)为配准模块，
Figure PCTCN2022082195-appb-000002
表示第i阶段，来自目标图像特征提取器输出的特征图，
Figure PCTCN2022082195-appb-000003
表示第i阶段，来自源图像特征提取器输出的特征图，ο表示应用变换关系，Φ i表示第i阶段的累积变换关系，φ i表示Φ i相对于Φ i-1的残差。φ和Φ的上标指示了变换关系的方向，即
Figure PCTCN2022082195-appb-000004
表示从源图像变换到目标图像的变换关系，
Figure PCTCN2022082195-appb-000005
表示从目标图像到源图像的变换关系。
这里，形变场指的是在图像配准中，将规则的空间网格，加上预测出的变形场，得到采样网格，对浮动图像使用包含形变信息的采样网格(sampling grid)，得到的是变形后的图像。一个大小为[W,H]的二维图像对应的形变场的大小是[W,H,2]，其中第三个维度的大小为2，分别表示在x轴和y轴方向的位移。同理，一个大小为[D,W,H]的三维图像对应的形变场的大小是[D,W,H,3]，其中第三个维度的大小为3，分别表示在x轴、y轴和z轴方向的位移。
配准模块分为两种类型：线性变换和形变变换，两者在输入和输出上是相同的，因此在应用时，可以根据需求自由组合线性和形变变换的配准模块，如使用一个线性配准模块和三个形变变换模块串接。为了实现变换过程是微分同胚的，两个模块并不直接输出φ，而是根据变换得到φ，即输出的为φ对应的矩阵，在这里通过实现变换过程是微分同胚的，可以减少运算次数，同时由于输出的为矩阵，使得该过程是可逆的，提供了校验的作用。
配准过程的过程如下：
线性配准模块的原始输出为正向的旋转、放缩、斜切矩阵Afw和正向的平移向量bfw。Afw可以直接通过网络预测，也可以分别预测旋转、放缩、斜切的参数，再组合而成。反向的线性变换关系通过线性变换求逆得到，即：
A bw＝(A fw) -1                                            (2)
b bw＝-(A fw) -1b fw                                          (3)
得到A和b后，应用在网格矩阵(mesh grid)上，再减去网格矩阵，即可得到形变场形式的形变关系φ。
形变变换的原始输出为速度场形式的正向形变关系V fw，在此基础上通过积分得到形变场形式的形变关系φ，其中，该形变关系φ满足下述公式(4)以及公式(5)：
Figure PCTCN2022082195-appb-000006
Figure PCTCN2022082195-appb-000007
其中，ο表示应用形变关系。
组合模块累积之前的每一步的φ得到累积形变关系Φ，即(注意对于正向和反向的形变，组合的顺序是相反的)：
Figure PCTCN2022082195-appb-000008
Figure PCTCN2022082195-appb-000009
其中
Figure PCTCN2022082195-appb-000010
和
Figure PCTCN2022082195-appb-000011
即为初始的形变关系，最终的网络输出同样由组合模块累积得到。
由此，在本公开实施例中，在进行形变初始化之后，再使用配准神经网络网络进行多级拟合残差，即具有神经网络学习具有的配准速度快的优势，又可以减少神经网络学习在全局形变过大时精度低的弊端，使得配准速度快且精度高。
承接于上述S302，确定第一原始图像和第二原始图像之间的变换关系信息的方法还包括：
S303：基于最后一级特征提取对应的第二变换关系信息，对所述第一图像和所述第二图像进行配准。
在一些实施例中，在进行多级迭代配准之后，可以基于最后一级配准之后输出的第二变换关系信息，即最终变换信息，对第一原始图像和第二原始图像进行配准。
在将第一原始图像和第二原始图像进行配准时，例如可以利用最后一级特征提取对应的第二变换关系信息，对第一原始图像进行变换处理，得到第一原始图像经过变换后的图像，然后将经过变换后的图像和第二原始图像进行位置的一一匹配，得到配准结果。
另外，也可以利用最后一级特征提取对应的第二变换关系信息，对第二原始图像进行变换处理，得到第二原始图像经过变换后的图像，然后将经过变换后的图像和第一原始图像进行位置的一一匹配，得到配准结果。
在本公开实施例中，涉及的配准方法应用于预先训练好的配准神经网络，所述配准神经网络包括特征提取神经网络以及多级配准神经网络两个分支网络；所述特征提取神经网络用于对所述第一原始图像和所述第二原始图像分别进行多级特征提取处理；所述多级配准神经网络用于基于从所述第一原始图像和所述第二原始图像提取到的多级特征，确定所述第一原始图像和所述第二原始图像之间目标变换关系信息，其中，所述目标变换关系信息用于对所述第一原始图像和所述第二原始图像进行配准。
承接于上述S101，本公开实施例提出的图像处理方法还包括：
S102：基于所述第一属性信息和所述第二属性信息，对所述第一对象实例和所述第二对象实例进行匹配，得到所述第一对象实例和所述第二对象实例之间的匹配度。
在一些实施例中，在获取到第一属性信息以及第二属性信息之后，便可以基于获取到的属性信息对第一对象实例和第二对象实例进行匹配。即，基于所述第一属性信息和所述第二属性信息，确定所述第一对象实例和所述第二对象实例之间的下述至少一项匹配信息：相似度、匹配优先级、距离、等效半径，并基于所述第一对象实例和所述第二对象实例之间的匹 配信息，得到所述第一对象实例和所述第二对象实例之间的匹配度。
示例性的，本公开实施例提供一种得到第一对象实例和第二对象实例之间匹配度的方式，在该实施例中，匹配信息包括：相似度、匹配优先级、距离和等效半径；
所述基于所述第一对象实例和所述第二对象实例之间的匹配信息，得到所述第一对象实例和所述第二对象实例之间的匹配度，包括：
将所述匹配优先级作为权值，对所述第一对象实例和所述第二对象实例之间的相似度进行加权处理，得到所述第一对象实例和所述第二对象实例之间的加权相似度；以及
基于所述距离和等效半径，确定所述第一对象实例和所述第二对象实例之间的邻接关系信息；
基于所述加权相似度和所述邻接关系信息，确定所述第一对象实例和所述第二对象实例之间的匹配度。
示例性的，可以通过遍历两个图像中所有可能的对象实例进行配对，计算两两对象实例之间的相似度、匹配优先级、距离和等效半径，形成四个矩阵。在本示例中，以A表示第一对象实例，a表示第二对象实例，其中：
(1)相似度是一种度量值，当两个对象实例的属性越接近，相似度的值越大，为了数值上可控，可以放缩或截断到[0,1]或[-1,1]范围。此处以对象实例的半径为例，对于第一对象实例A和第二对象实例a，其相似度可以是：
Figure PCTCN2022082195-appb-000012
这种情况下，其半径的相似度处于[0,1]之间，且两个对象实例在半径上越接近，相似度越大。
类似的，也可以计算其他属性信息之间的相似度。例如在属性信息包括：在对象实例的属性信息包括包围框的情况下，可以计算第一对象实例和第二对象实例分别对应的包围框的交并比(Intersection-over-Union，IoU)，并将该交并比作为两者之间的相似度；在对象实例的属性信息包括轮廓的情况下，可以计算第一对象实例和第二对象实例分别对应的轮廓的迪卡(Dice)系数，并将该迪卡系数作为两者之间的相似度；在对象实例的属性信息包括特征数据的情况下，例如为编码向量的情况下，可以计算第一对象实例和第二对象实例分别对应的编码向量的余弦相似度，并将该余弦相似度的数值作为两者之间的相似度等。
另外，在计算多种属性分别对应的相似度时，可以度多种属性分别对应的相似度进行加权求和，并将加权求和的结果作为第一对象实例和第二对象实例之间的相似度。
(2)匹配优先级是也是一种度量值，当对象实例的重要性越高，其值越大。例如，对于病灶而言，匹配优先度可以表征病灶的恶性概率，其恶性概率越高、病灶越大，对应的匹配优先级越高。同样地，匹配优先级也可以进行放缩和截断，实现数值上可控。此处以病灶的恶性概率为例，匹配优先级可以表示为：
Figure PCTCN2022082195-appb-000013
由于恶性概率是一个[0,1]之间的值，由上述公式可知，匹配优先级的数值分布在[0.5,1]之间。类似的，可以对于不同的属性设置不同的匹配优先级度量，或对多个属性设置匹配优先级度量，或对多个匹配优先级度量进行加权，例如病灶的恶化速度、扩散速度等。
(3)距离为所有可能配对的两对象实例之间的距离，示例性的，可以使用对象实例中心之间的欧式距离表示两对象实例之间的距离：
距离(A，a)＝||中心坐标A-中心坐标a|| 2；
其中，||·|| k表示k阶范数，向量的2阶范数为对向量所有分量求平方和后开根号。
(4)等效半径通常是两个对象实例的半径或对角线的长度在某种形式下的平均数值，例如，当对象实例的表现形式为中心+半径时，其等效半径即为两个对象实例的半径的几何平均数；当对象实例的表现形式为包围框时，其等效半径可以为对角线的长度等：其中，等效半径为两个对象实例的半径的几何平均数可以表示为：
Figure PCTCN2022082195-appb-000014
按照上述方法，可以得到四个矩阵，即相似度矩阵、匹配优先级矩阵、距离矩阵和等效半径矩阵。对于每个矩阵而言，第i行第j列表示图像1的第i个对象实例对于图像2的第j个对象实例的相应的数值。这四个矩阵可以用于在后续的计算过程中进行进一步的组合。
基于获取的四个矩阵，相似度矩阵和匹配优先级矩阵可以组合为加权相似度矩阵。加权相似度是考虑了匹配优先级之后的相似度，由此加权相似度可以表示为：
加权相似度＝相似度*匹配优先级。
邻接关系矩阵为根据距离和等效半径计算的两个对象实例之间的相对位置关系。邻接关系用来衡量两个对象实例间是否在位置上相近或重合，与距离负相关，与等效半径正相关，例如可以表示如下：
Figure PCTCN2022082195-appb-000015
此外，为了便于后续计算，也可以设置一个阈值，由于修正邻接关系，例如可以表示如下：
Figure PCTCN2022082195-appb-000016
根据加权相似度和邻接关系，可以得到最终的匹配度。匹配度决定了两个对象实例之间匹配的优劣程度。匹配度可以表示为：
匹配度＝加权相似度*邻接关系。
由此便可以获取到两个对象实例之间的匹配度。
在本公开另一实施例中，当第一对象实例、和第二对象实例均有多个时，所述基于所述第一属性信息和所述第二属性信息，对所述第一对象实例和所述第二对象实例进行匹配，得到所述第一对象实例和所述第二对象实例之间的匹配度，包括：
基于多个所述第一对象实例、和多个所述第二对象实例，构成多个对象实例对；
针对每个对象实例对，根据该对象实例对中包括的第一对象实例对应的第一属性信息和该对象实例对中包括的第二对象实例对应的第二属性信息，对该对象实例对中包括的第一对象实例和第二对象实例进行匹配，得到该对象实例对的匹配度。
在一些实施例中，可以基于多个对象实例对分别对应的匹配度和预设的匹配度阈值，对多个所述第一对象实例、和多个所述第二对象实例进行分组，得到多个对象实例组；针对各个对象实例组，在该对象实例组中包括至少第一对象实例和至少一个第二对象实例的情况下，将该对象实例组中包括的第一对象实例和第二对象实例进行融合。
在一些实施例中，在得到匹配度矩阵后，可以从矩阵得到最终的匹配结果。即基于一个阈值，选取匹配度大于该阈值的配对为一个可行的配对，并将矩阵转化为一个无向的二部图，图上每一个节点为一个对象实例，边的权重为匹配度；然后按照边的权重从高到低，进行有放回或无放回的配对。有放回的配对会将所有可能连接的对象实例都融合为一个新的对象实例；无放回的配对优先配对匹配度较高的两个对象实例，当某个对象实例成功配对后，将无法再与其他的对象实例进行配对，这里，有无放回的方法为可配置项，是根据需求决定的。在无放回的配对里面，一个对象实例最多仅能与另一个对象实例配对，适用于跟踪肺结节随随访变化等；有放回的配对允许多对多的匹配，适用于跨模态的情况，如肿瘤的包膜在特定模态不显影，在应用方向上不做限定。
示例性的，当一对对象实例匹配成功之后，匹配成功的对象实例不会继续进行匹配，可以将其移出允许匹配的对象实例池中，但是还未匹配成功的对象实例。还可以继续留在对象实例池中，继续等到匹配，直到所有留在对象实例池里面的对象实例都不满足匹配条件。
示例性的，参阅图4，图4为本公开实施例提出的多对象实例匹配的示意图。在得到匹配度的矩阵之后，基于预设的阈值0.5，筛选出满足阈值的多个对象实例对，其中，包括Aa之间的0.7，Ba之间的0.6，以及Cc之间的0.6，并基于有放回及无放回的方法得到最终的 匹配结果。
在本公开实施例中，由于匹配时纳入多个属性的结果，考虑相似度、匹配优先级、位置和大小关系等，在匹配上精度更高。
承接于上述S102，本公开实施例提出的图像处理方法还包括：
S103：基于所述匹配度，对所述第一对象实例和所述第二对象实例进行融合。
在一些实施例中，在确定匹配度之后，可以对所述第一对象实例的第一属性信息、和所述第二对象实例的第二属性信息进行融合。
示例性的，在匹配完成后，便可以进行属性融合。默认以图像1为基准空间。对于与空间有关的属性，如包围框、中心等、轮廓等，会在融合前按照配准关系进行坐标变换。属性融合时，根据属性的类型采用不同的融合方法，在此不作限定，如对于包围框和轮廓等，求取所有对象实例的并集；对于尺寸，或是根据融合后的包围框或轮廓重新计算，或是求取所有对象实例的最大值；对于恶性程度等，求取所有对象实例的最大值；对于检测模块的隐含层编码，求取多个对象实例的均值。参见图2所示，融合后的对象实例，其坐标及尺寸转变为按照配准关系配准之后的坐标及尺寸，其对应的恶性程度选择对象实例A和对象实例a之间的最大值，特征向量也进行了平均处理，由此便完成了针对第一对象实例以及第二对象实例之间的融合过程。
在本公开另一实施例中，当存在多个图像时，可以使用归并的方式进行融合，得到最终的结果。例如两两对象实例进行融合，再将两两融合的结果进行融合；又或者，任意选取两个对象实例进行融合，在将融合的结果与另一对象实例进行融合，直至与全部对象实例融合完成，这两种归并方式取决于多个来源是平行关系还是层级关系。由此，可以联系多个图像，如多模态、多期相造影或多个时序随访的结果，在匹配方式上更加灵活，更好地辅助使用者进行联合阅片与诊断。
本公开实施例例如应用在肝脏影像诊断平台上，患者会在多期造影结果上分别做检出，一方面解决单期相可能漏诊的问题，如局灶性结节增生在门脉期不明显，转移瘤在动脉期常常不明显，并且建立起病灶在多个期相上的关系，用于后续的多期相联合诊断。还可以应用在在肺部影像诊断平台上，根据多个时序的肺部图像，进行结节随访，分析结节的体积和征象变化，对于患者的病情判断具有重要的意义。
本公开实施例对于多个图像，先使用目标检测算法得到每个图像对应的检测结果，即对象实例的属性信息，再使用配准算法得到任意两个图像的空间变换关系。根据检测结果对应的属性和空间相对关系，依次计算两个图像中所有可能检测目标配对的相似度、匹配优先级、距离和等效半径，并通过组合计算匹配度，得到两图像的检测目标匹配结果，再进行属性融合，得到融合后的对象实例。这样，通过对第一对象实例和第二对象实例的属性信息进行匹配，得到第一对象实例和对象实例之间的匹配度，从而可以根据匹配度，对第一对象实例和第二对象实例进行更加精确的融合。
本领域技术人员可以理解，在具体实施方式的上述方法中，各步骤的撰写顺序并不意味着严格的执行顺序而对实施成任何限定，各步骤的实际执行顺序应当以其功能和可能的内在逻辑确定。
基于同一发明构思，本公开实施例中还提供了与图像处理方法对应的图像处理装置，由于本公开实施例中的装置解决问题的原理与本公开实施例上述图像处理方法相似，因此装置的实施可以参见方法的实施。
参照图5、图6、图7所示，图5为本公开实施例提供的一种图像处理装置的示意图；图6为本公开实施例所提供的另一种图像处理装置的示意图；图7为本公开实施例所提供的图像处理装置中，匹配模块的示意图。所述装置包括：第一确定模块510、匹配模块520、融合模块530；其中，
第一确定模块510，配置为确定目标对象的第一对象实例在第一图像中的第一属性信息和确定所述目标对象的第二对象实例在第二图像中的第二属性信息；
匹配模块520，配置为基于所述第一属性信息和所述第二属性信息，对所述第一对象实 例和所述第二对象实例进行匹配，得到所述第一对象实例和所述第二对象实例之间的匹配度；
融合模块530，配置为基于所述匹配度，对所述第一对象实例和所述第二对象实例进行融合。
一种可选的实施方式中，如图6所示，还包括：
获取模块540，配置为获取第一原始图像和第二原始图像；
第二确定模块550，配置为确定所述第一原始图像和所述第二原始图像之间的变换关系信息；
变换模块560，配置为基于所述变换关系信息对所述第一原始图像进行变换处理，得到第一图像，并将所述第二原始图像作为所述第二图像；或者，基于所述变换关系信息对所述第二原始图像进行变换处理，得到第二图像，并将所述第一原始图像作为所述第一图像。
一种可选的实施方式中，所述属性信息包括以下至少一种：
对象实例的位置信息、尺寸信息、所述对象实例属于所述目标对象的概率、特征数据、与所述对象实例对应的图像区域的灰度数据、影像组学信息；
所述对象实例包括：所述第一对象实例和所述第二对象实例。
一种可选的实施方式中，如图7所示，所述匹配模块520包括：
确定单元521，配置为基于所述第一属性信息和所述第二属性信息，确定所述第一对象实例和所述第二对象实例之间的下述至少一项匹配信息：相似度、匹配优先级、距离、等效半径；
匹配单元522，配置为基于所述第一对象实例和所述第二对象实例之间的匹配信息，得到所述第一对象实例和所述第二对象实例之间的匹配度。
一种可选的实施方式中，所述匹配单元522配置为：
将所述匹配优先级作为权值，对所述第一对象实例和所述第二对象实例之间的相似度进行加权处理，得到所述第一对象实例和所述第二对象实例之间的加权相似度；以及
基于所述距离和等效半径，确定所述第一对象实例和所述第二对象实例之间的邻接关系信息；
基于所述加权相似度和所述邻接关系信息，确定所述第一对象实例和所述第二对象实例之间的匹配度。
一种可选的实施方式中，所述第一对象实例、和所述第二对象实例均有多个；
所述匹配模块520配置为：
基于多个所述第一对象实例、和多个所述第二对象实例，构成多个对象实例对；
针对每个对象实例对，根据该对象实例对中包括的第一对象实例对应的第一属性信息和该对象实例对中包括的第二对象实例对应的第二属性信息，对该对象实例对中包括的第一对象实例和第二对象实例进行匹配，得到该对象实例对的匹配度。
一种可选的实施方式中，所述融合模块530配置为：
基于多个对象实例对分别对应的匹配度和预设的匹配度阈值，对多个所述第一对象实例、和多个所述第二对象实例进行分组，得到多个对象实例组；
针对各个对象实例组，在该对象实例组中包括至少第一对象实例和至少一个第二对象实例的情况下，将该对象实例组中包括的第一对象实例和第二对象实例进行融合。
一种可选的实施方式中，所述融合模块530配置为：
对所述第一对象实例的第一属性信息、和所述第二对象实例的第二属性信息进行融合。
本公开实施例通过对第一对象实例和第二对象实例的属性信息进行匹配，得到第一对象实例和对象实例之间的匹配度，从而可以根据匹配度，对第一对象实例和第二对象实例进行更加精确的融合。
关于装置中的各模块的处理流程和各模块之间的交互流程的描述可以参照上述方法实施例中的相关说明，这里不再详述。
对应于图1中的图像处理方法，本公开实施例还提供了一种计算机设备，如图8所示，为本公开实施例提供的计算机设备结构示意图，包括：
处理器801、存储器802、和总线803；存储器802用于存储执行指令，包括内存8021和外部存储器8022；这里的内存8021也称内存储器，用于暂时存放处理器801中的运算数据，以及与硬盘等外部存储器8022交换的数据，处理器801通过内存8021与外部存储器8022进行数据交换，当所述计算机设备运行时，所述处理器801与所述存储器802之间通过总线803通信，使得所述处理器801执行以下指令：
确定目标对象的第一对象实例在第一图像中的第一属性信息和确定所述目标对象的第二对象实例在第二图像中的第二属性信息；
基于所述第一属性信息和所述第二属性信息，对所述第一对象实例和所述第二对象实例进行匹配，得到所述第一对象实例和所述第二对象实例之间的匹配度；
基于所述匹配度，对所述第一对象实例和所述第二对象实例进行融合。
本公开实施例还提供一种计算机可读存储介质，该计算机可读存储介质上存储有计算机程序，该计算机程序被处理器运行时执行上述方法实施例中所述的图像处理方法的步骤。其中，该存储介质可以是易失性或非易失的计算机可读取存储介质。
本公开实施例还提供一种计算机程序产品，该计算机程序产品承载有程序代码，所述程序代码包括的指令可用于执行上述方法实施例中所述的图像处理方法的步骤，可参见上述方法实施例。
其中，上述计算机程序产品可以具体通过硬件、软件或其结合的方式实现。在一个可选实施例中，所述计算机程序产品具体体现为计算机存储介质，在另一个可选实施例中，计算机程序产品具体体现为软件产品，例如软件开发包(Software Development Kit，SDK)等等。
所属领域的技术人员可以清楚地了解到，为描述的方便和简洁，上述描述的系统和装置的具体工作过程，可以参考前述方法实施例中的对应过程。在本公开所提供的几个实施例中，应该理解到，所揭露的系统、装置和方法，可以通过其它的方式实现。以上所描述的装置实施例仅仅是示意性的，例如，所述单元的划分，仅仅为一种逻辑功能划分，实际实现时可以有另外的划分方式，又例如，多个单元或组件可以结合或者可以集成到另一个系统，或一些特征可以忽略，或不执行。另一点，所显示或讨论的相互之间的耦合或直接耦合或通信连接可以是通过一些通信接口，装置或单元的间接耦合或通信连接，可以是电性，机械或其它的形式。
所述作为分离部件说明的单元可以是或者也可以不是物理上分开的，作为单元显示的部件可以是或者也可以不是物理单元，即可以位于一个地方，或者也可以分布到多个网络单元上。可以根据实际的需要选择其中的部分或者全部单元来实现本实施例方案的目的。
另外，在本公开各个实施例中的各功能单元可以集成在一个处理单元中，也可以是各个单元单独物理存在，也可以两个或两个以上单元集成在一个单元中。
所述功能如果以软件功能单元的形式实现并作为独立的产品销售或使用时，可以存储在一个处理器可执行的非易失的计算机可读取存储介质中。基于这样的理解，本公开的技术方案本质上或者说对现有技术做出贡献的部分或者该技术方案的部分可以以软件产品的形式体现出来，该计算机软件产品存储在一个存储介质中，包括若干指令用以使得一台计算机设备(可以是个人计算机，服务器，或者网络设备等)执行本公开各个实施例所述方法的全部或部分步骤。而前述的存储介质包括：U盘、移动硬盘、只读存储器(Read-Only Memory，ROM)、随机存取存储器(Random Access Memory，RAM)、磁碟或者光盘等各种可以存储程序代码的介质。
最后应说明的是：以上所述实施例，仅为本公开的具体实施方式，用以说明本公开的技术方案，而非对其限制，本公开的保护范围并不局限于此，尽管参照前述实施例对本公开进行了详细的说明，本领域的普通技术人员应当理解：任何熟悉本技术领域的技术人员在本公开揭露的技术范围内，其依然可以对前述实施例所记载的技术方案进行修改或可轻易想到变化，或者对其中部分技术特征进行等同替换；而这些修改、变化或者替换，并不使相应技术方案的本质脱离本公开实施例技术方案的精神和范围，都应涵盖在本公开的保护范围之内。因此，本公开的保护范围应所述以权利要求的保护范围为准。
工业实用性
本公开实施例提供了一种图像处理方法、装置、计算机设备、存储介质及程序产品，其中，该方法包括：确定目标对象的第一对象实例在第一图像中的第一属性信息和所述目标对象的第二对象实例在第二图像中的第二属性信息；基于所述第一属性信息和所述第二属性信息，对所述第一对象实例和所述第二对象实例进行匹配，得到所述第一对象实例和所述第二对象实例之间的匹配度；基于所述匹配度，对所述第一对象实例和所述第二对象实例进行融合。本公开实施例通过对第一对象实例和第二对象实例的属性信息进行匹配，得到第一对象实例和对象实例之间的匹配度，从而可以根据匹配度，对第一对象实例和第二对象实例进行更加精确的融合。

Claims (12)

  1. 一种图像处理方法，其中，包括：
    确定目标对象的第一对象实例在第一图像中的第一属性信息和所述目标对象的第二对象实例在第二图像中的第二属性信息；
    基于所述第一属性信息和所述第二属性信息，对所述第一对象实例和所述第二对象实例进行匹配，得到所述第一对象实例和所述第二对象实例之间的匹配度；
    基于所述匹配度，对所述第一对象实例和所述第二对象实例进行融合。
  2. 根据权利要求1所述的图像处理方法，其中，还包括：获取第一原始图像和第二原始图像；
    确定所述第一原始图像和所述第二原始图像之间的变换关系信息；
    基于所述变换关系信息对所述第一原始图像进行变换处理，得到所述第一图像，并将所述第二原始图像作为所述第二图像；
    或者，
    基于所述变换关系信息对所述第二原始图像进行变换处理，得到所述第二图像，并将所述第一原始图像作为所述第一图像。
  3. 根据权利要求1或2所述的图像处理方法，其中，所述第一属性信息或所述第二属性信息包括以下至少一种：
    对象实例的位置信息、尺寸信息、所述对象实例属于所述目标对象的概率、特征数据、与所述对象实例对应的图像区域的灰度数据、影像组学信息；
    所述对象实例包括：所述第一对象实例和所述第二对象实例。
  4. 根据权利要求1-3任一项所述的图像处理方法，其中，所述基于所述第一属性信息和所述第二属性信息，对所述第一对象实例和所述第二对象实例进行匹配，包括：
    基于所述第一属性信息和所述第二属性信息，确定所述第一对象实例和所述第二对象实例之间的下述至少一项匹配信息：相似度、匹配优先级、距离、等效半径；
    基于所述匹配信息，得到所述第一对象实例和所述第二对象实例之间的匹配度。
  5. 根据权利要求4所述的图像处理方法，其中，所述基于所述匹配信息，得到所述第一对象实例和所述第二对象实例之间的匹配度，包括：
    将所述匹配优先级作为权值，对所述第一对象实例和所述第二对象实例之间的相似度进行加权处理，得到所述第一对象实例和所述第二对象实例之间的加权相似度；以及
    基于所述距离和等效半径，确定所述第一对象实例和所述第二对象实例之间的邻接关系信息；
    基于所述加权相似度和所述邻接关系信息，确定所述第一对象实例和所述第二对象实例之间的匹配度。
  6. 根据权利要求1-5任一项所述的图像处理方法，其中，所述第一对象实例和所述第二对象实例均有多个；
    所述基于所述第一属性信息和所述第二属性信息，对所述第一对象实例和所述第二对象实例进行匹配，得到所述第一对象实例和所述第二对象实例之间的匹配度，包括：
    基于多个所述第一对象实例、和多个所述第二对象实例，构成多个对象实例对；
    针对每个对象实例对，根据该对象实例对中包括的第一对象实例对应的第一属性信息和该对象实例对中包括的第二对象实例对应的第二属性信息，对该对象实例对中包括的第一对象实例和第二对象实例进行匹配，得到该对象实例对的匹配度。
  7. 根据权利要求6所述的图像处理方法，其中，基于所述匹配度，对所述第一对象实例和所述第二对象实例进行融合，包括：
    基于多个对象实例对分别对应的匹配度和预设的匹配度阈值，对多个所述第一对象实例、和多个所述第二对象实例进行分组，得到多个对象实例组；
    针对各个对象实例组，在该对象实例组中包括至少第一对象实例和至少一个第二对象实例的情况下，将该对象实例组中包括的第一对象实例和第二对象实例进行融合。
  8. 根据权利要求1-7任一项所述的图像处理方法，其中，所述对所述第一对象实例和所述第二对象实例进行融合，包括：
    对所述第一对象实例的第一属性信息和所述第二对象实例的第二属性信息进行融合。
  9. 一种图像处理装置，其中，包括：
    第一确定模块，配置为确定目标对象的第一对象实例在第一图像中的第一属性信息和确定所述目标对象的第二对象实例在第二图像中的第二属性信息；
    匹配模块，配置为基于所述第一属性信息和所述第二属性信息，对所述第一对象实例和所述第二对象实例进行匹配，得到所述第一对象实例和所述第二对象实例之间的匹配度；
    融合模块，配置为基于所述匹配度，对所述第一对象实例和所述第二对象实例进行融合。
  10. 一种计算机设备，其中，包括：处理器、存储器和总线，所述存储器存储有所述处理器可执行的机器可读指令，当计算机设备运行时，所述处理器与所述存储器之间通过总线通信，所述机器可读指令被所述处理器执行时执行如权利要求1至8任一项所述的图像处理方法的步骤。
  11. 一种计算机可读存储介质，其中，该计算机可读存储介质上存储有计算机程序，该计算机程序被处理器运行时执行如权利要求1至8任一项所述的图像处理方法的步骤。
  12. 一种计算机程序产品，所述计算机程序产品包括存储了计算机程序的非瞬时性计算机可读存储介质，所述计算机程序被计算机读取并执行时，实现如权利要求1至8任一项所述的图像处理方法的步骤。
PCT/CN2022/082195 2021-09-26 2022-03-22 图像处理方法、装置、计算机设备、存储介质及程序产品 Ceased WO2023045284A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202111129299.7 2021-09-26
CN202111129299.7A CN114240809B (zh) 2021-09-26 2021-09-26 图像处理方法、装置、计算机设备及存储介质

Publications (1)

Publication Number Publication Date
WO2023045284A1 true WO2023045284A1 (zh) 2023-03-30

Family

ID=80743004

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2022/082195 Ceased WO2023045284A1 (zh) 2021-09-26 2022-03-22 图像处理方法、装置、计算机设备、存储介质及程序产品

Country Status (2)

Country Link
CN (1) CN114240809B (zh)
WO (1) WO2023045284A1 (zh)

Cited By (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116453104A (zh) * 2023-06-15 2023-07-18 安徽容知日新科技股份有限公司 液位识别方法、装置、电子设备及计算机可读存储介质
CN116503603A (zh) * 2023-05-16 2023-07-28 北京交通大学 一种基于弱监督语义分割和特征弥补的类间遮挡目标检测网络模型的训练方法
CN117253113A (zh) * 2023-09-12 2023-12-19 南京信息工程大学 一种基于车载相机多模态特征融合方法
CN118876055A (zh) * 2024-07-19 2024-11-01 杭州海康机器人股份有限公司 一种运动机构控制方法、装置、电子设备及存储介质
CN121120634A (zh) * 2025-11-12 2025-12-12 昆仑数智科技有限责任公司 基于语义分割的图像处理方法、装置以及设备

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20190043224A1 (en) * 2016-02-16 2019-02-07 Brainlab Ag Determination of Dynamic DRRs
CN112288708A (zh) * 2020-10-27 2021-01-29 推想医疗科技股份有限公司 Ct图像中淋巴结的检测方法、装置、介质及电子设备

Family Cites Families (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2018119766A1 (zh) * 2016-12-28 2018-07-05 上海联影医疗科技有限公司 多模态图像处理系统及方法
CN106683042B (zh) * 2017-01-05 2020-08-11 卡本(深圳)医疗科技有限公司 一种基于非线性最小二乘的多模态医学图像融合方法以及系统
CN108427927B (zh) * 2018-03-16 2020-11-27 深圳市商汤科技有限公司 目标再识别方法和装置、电子设备、程序和存储介质
CN110245679B (zh) * 2019-05-08 2021-12-28 北京旷视科技有限公司 图像聚类方法、装置、电子设备及计算机可读存储介质
CN110533641A (zh) * 2019-08-20 2019-12-03 东软医疗系统股份有限公司 一种多模态医学图像配准方法和装置
CN110837579B (zh) * 2019-11-05 2024-07-23 腾讯科技（深圳）有限公司 视频分类方法、装置、计算机以及可读存储介质
CN112184781A (zh) * 2020-09-14 2021-01-05 中国科学院深圳先进技术研究院 超声图像和ct图像的配准方法、装置及设备
CN113298742A (zh) * 2021-05-20 2021-08-24 广东省人民医院 基于图像配准的多模态视网膜图像融合方法及系统

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20190043224A1 (en) * 2016-02-16 2019-02-07 Brainlab Ag Determination of Dynamic DRRs
CN112288708A (zh) * 2020-10-27 2021-01-29 推想医疗科技股份有限公司 Ct图像中淋巴结的检测方法、装置、介质及电子设备

Cited By (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116503603A (zh) * 2023-05-16 2023-07-28 北京交通大学 一种基于弱监督语义分割和特征弥补的类间遮挡目标检测网络模型的训练方法
CN116503603B (zh) * 2023-05-16 2024-02-23 北京交通大学 一种基于弱监督语义分割和特征弥补的类间遮挡目标检测网络模型的训练方法
CN116453104A (zh) * 2023-06-15 2023-07-18 安徽容知日新科技股份有限公司 液位识别方法、装置、电子设备及计算机可读存储介质
CN116453104B (zh) * 2023-06-15 2023-09-08 安徽容知日新科技股份有限公司 液位识别方法、装置、电子设备及计算机可读存储介质
CN117253113A (zh) * 2023-09-12 2023-12-19 南京信息工程大学 一种基于车载相机多模态特征融合方法
CN118876055A (zh) * 2024-07-19 2024-11-01 杭州海康机器人股份有限公司 一种运动机构控制方法、装置、电子设备及存储介质
CN121120634A (zh) * 2025-11-12 2025-12-12 昆仑数智科技有限责任公司 基于语义分割的图像处理方法、装置以及设备

Also Published As

Publication number Publication date
CN114240809B (zh) 2025-05-27
CN114240809A (zh) 2022-03-25

Similar Documents

Publication Publication Date Title
Fu et al. LungRegNet: an unsupervised deformable image registration method for 4D‐CT lung
Liu et al. Automatic prostate zonal segmentation using fully convolutional network with feature pyramid attention
Kroon et al. MRI modalitiy transformation in demon registration
CN110914866B (zh) 用于在图像分析中进行解剖结构分割的系统和方法
CN114240809A (zh) 图像处理方法、装置、计算机设备及存储介质
Wang et al. Multiscale transunet++: dense hybrid u-net with transformer for medical image segmentation
Paragios et al. Non-rigid registration using distance functions
Pizer et al. Multiscale medial shape-based analysis of image objects
JP2018139693A (ja) 画像分類装置、方法およびプログラム
Corral Acero et al. SMOD-data augmentation based on statistical models of deformation to enhance segmentation in 2D cine cardiac MRI
Khalifa et al. State-of-the-art medical image registration methodologies: A survey
CN115115676A (zh) 图像配准方法、装置、设备和存储介质
Liu et al. Microscopic 3D reconstruction based on point cloud data generated using defocused images
Xu et al. 3D‐SIFT‐Flow for atlas‐based CT liver image segmentation
Cheng et al. Fully automated prostate whole gland and central gland segmentation on MRI using holistically nested networks with short connections
CN120318427A (zh) 一种肝脏病灶三维建模方法
CN112634265A (zh) 基于dnn的胰腺全自动分割模型的构建、分割方法及系统
Preedanan et al. Urinary stones segmentation in abdominal X-ray images using cascaded U-net pipeline with stone-embedding augmentation and lesion-size reweighting approach
Stralen et al. Stenosis detection in X-ray coronary angiography with deep neural networks leveraged by attention mechanisms
CN121033080A (zh) 一种基于SCDU-Net模型的MRI头颈部肿瘤分割方法及系统
CN115713549A (zh) 一种基于深度学习的二维三维医学图像配准方法
Chang et al. Structure-aware independently trained multi-scale registration network for cardiac images
Carvalho et al. Joint intensity‐and‐point based registration of free‐hand B‐mode ultrasound and MRI of the carotid artery
Haq et al. Deformable multisurface segmentation of the spine for orthopedic surgery planning and simulation
Jacob et al. Deep Conditional Shape Models for 3D cardiac image segmentation

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 22871349

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 22871349

Country of ref document: EP

Kind code of ref document: A1

122 Ep: pct application non-entry in european phase

Ref document number: 22871349

Country of ref document: EP

Kind code of ref document: A1

32PN Ep: public notification in the ep bulletin as address of the adressee cannot be established

Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205A DATED 27-11-2024)

122 Ep: pct application non-entry in european phase

Ref document number: 22871349

Country of ref document: EP

Kind code of ref document: A1