WO2022221991A1 - 一种影像数据处理方法、装置、计算机及存储介质 - Google Patents
一种影像数据处理方法、装置、计算机及存储介质 Download PDFInfo
- Publication number
- WO2022221991A1 WO2022221991A1 PCT/CN2021/088127 CN2021088127W WO2022221991A1 WO 2022221991 A1 WO2022221991 A1 WO 2022221991A1 CN 2021088127 W CN2021088127 W CN 2021088127W WO 2022221991 A1 WO2022221991 A1 WO 2022221991A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- feature
- feature map
- image data
- channel
- preset
- Prior art date
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 19
- 238000012545 processing Methods 0.000 claims abstract description 49
- 238000000034 method Methods 0.000 claims abstract description 38
- 238000004422 calculation algorithm Methods 0.000 claims abstract description 28
- 238000005457 optimization Methods 0.000 claims abstract description 18
- 230000000694 effects Effects 0.000 claims abstract description 17
- 238000012549 training Methods 0.000 claims abstract description 16
- 230000010354 integration Effects 0.000 claims abstract description 15
- 238000007781 pre-processing Methods 0.000 claims abstract description 10
- 230000009466 transformation Effects 0.000 claims description 38
- 230000007246 mechanism Effects 0.000 claims description 15
- 238000004364 calculation method Methods 0.000 claims description 9
- 238000002156 mixing Methods 0.000 claims description 9
- 238000011426 transformation method Methods 0.000 claims description 8
- 230000004927 fusion Effects 0.000 abstract description 7
- 239000010410 layer Substances 0.000 description 20
- 230000006870 function Effects 0.000 description 15
- 238000013135 deep learning Methods 0.000 description 10
- 230000011218 segmentation Effects 0.000 description 10
- 238000011160 research Methods 0.000 description 8
- 206010058467 Lung neoplasm malignant Diseases 0.000 description 7
- 201000005202 lung cancer Diseases 0.000 description 7
- 208000020816 lung neoplasm Diseases 0.000 description 7
- 230000008569 process Effects 0.000 description 7
- 238000012216 screening Methods 0.000 description 6
- 238000004458 analytical method Methods 0.000 description 5
- 238000010586 diagram Methods 0.000 description 5
- 238000000605 extraction Methods 0.000 description 5
- 238000003709 image segmentation Methods 0.000 description 5
- 230000006872 improvement Effects 0.000 description 4
- 238000011176 pooling Methods 0.000 description 4
- 239000013598 vector Substances 0.000 description 4
- 230000004913 activation Effects 0.000 description 3
- 238000013136 deep learning model Methods 0.000 description 3
- 238000002059 diagnostic imaging Methods 0.000 description 3
- 201000010099 disease Diseases 0.000 description 3
- 208000037265 diseases, disorders, signs and symptoms Diseases 0.000 description 3
- 238000010801 machine learning Methods 0.000 description 3
- 239000011159 matrix material Substances 0.000 description 3
- 238000005070 sampling Methods 0.000 description 3
- 238000004590 computer program Methods 0.000 description 2
- 238000013461 design Methods 0.000 description 2
- 238000003745 diagnosis Methods 0.000 description 2
- 238000002474 experimental method Methods 0.000 description 2
- 238000004393 prognosis Methods 0.000 description 2
- 238000000844 transformation Methods 0.000 description 2
- 238000011282 treatment Methods 0.000 description 2
- 238000007476 Maximum Likelihood Methods 0.000 description 1
- 206010028980 Neoplasm Diseases 0.000 description 1
- 238000012952 Resampling Methods 0.000 description 1
- 230000006978 adaptation Effects 0.000 description 1
- 239000000654 additive Substances 0.000 description 1
- 230000000996 additive effect Effects 0.000 description 1
- 238000013459 approach Methods 0.000 description 1
- 238000013473 artificial intelligence Methods 0.000 description 1
- 238000013398 bayesian method Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 239000000090 biomarker Substances 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000013527 convolutional neural network Methods 0.000 description 1
- 238000012937 correction Methods 0.000 description 1
- 238000007418 data mining Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000018109 developmental process Effects 0.000 description 1
- 238000003748 differential diagnosis Methods 0.000 description 1
- 238000006073 displacement reaction Methods 0.000 description 1
- 238000011156 evaluation Methods 0.000 description 1
- 238000003384 imaging method Methods 0.000 description 1
- 208000037841 lung tumor Diseases 0.000 description 1
- 238000013508 migration Methods 0.000 description 1
- 230000005012 migration Effects 0.000 description 1
- 239000000203 mixture Substances 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000003058 natural language processing Methods 0.000 description 1
- 238000010606 normalization Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000002685 pulmonary effect Effects 0.000 description 1
- 239000002356 single layer Substances 0.000 description 1
- 230000006641 stabilisation Effects 0.000 description 1
- 238000011105 stabilization Methods 0.000 description 1
- 230000002194 synthesizing effect Effects 0.000 description 1
- 230000009897 systematic effect Effects 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
- 238000002604 ultrasonography Methods 0.000 description 1
- 238000012795 verification Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
Definitions
- Embodiments of the present invention relate to the technical field of image processing, and in particular, to an image data processing method, device, computer, and storage medium.
- a machine learning task T is defined as the modeling of the conditional probability p(x
- domain D refers to a sample space and its distribution.
- Radiomics analysis mainly includes the steps of image acquisition, image segmentation, radiomics parameter extraction, model establishment and verification. Extract massive data from medical images, so as to transform the subjective qualitative data in medical images into objective quantitative data, and carry out data mining analysis [2].
- the rapid development of radiomics has made great achievements in disease diagnosis and differential diagnosis, tumor staging and grading, genotype and phenotype prediction, treatment plan decision-making, efficacy evaluation and prognosis prediction, especially in lung tumors. [3].
- radiomics parameters exhibit biomarker properties beyond traditional medical approaches in disease screening, diagnosis, treatment, and prognosis, they produce large variability and repeatability in multicenter and multidevice imaging.
- the problems of poor repeatability and reproducibility result in the lack of generalization ability of the models constructed based on radiomics parameters, which greatly limits the diagnostic performance of the models, making it difficult to apply them to the practice of real medical scenarios.
- an embodiment of the present invention provides an image data processing method, including:
- a preset batch effect removal algorithm will be used to adjust the depth-omics parameters of the feature map in each channel to obtain a second feature map
- the first feature map and the second feature are input into a preset network integration model for processing to obtain feature optimization parameters.
- the preprocessing of the image data to obtain multi-channel data includes:
- the image data and the image data processed by each transformation method are used as the multi-channel data.
- obtaining a feature map containing feature weights according to the deep omics parameters, and fusing the feature maps containing weights in each channel to obtain a first feature map including:
- the feature maps containing weights in each channel are fused to obtain the first feature map.
- adjusting the depth-omics parameters of the feature map in each channel by using a preset batch effect removal algorithm to obtain a second feature map including:
- the second feature map is obtained by adjusting the data set obtained after mixing with the reference data set as a standard and using a preset ComBat algorithm.
- an embodiment of the present invention further provides an image data processing apparatus, including:
- the acquisition module is used for acquiring the image data to be processed, preprocessing the image data to obtain multi-channel data, and inputting the multi-channel data to the pre-training encoder to obtain the feature maps of multiple channels;
- the processing module is used to extract the deep omics parameters of each of the feature maps, obtain the feature maps containing the feature weights according to the deep omics parameters, and fuse the feature maps containing the weights in each channel to obtain the first feature map;
- the processing module is further configured to adjust the depth-omics parameters of the feature map in each channel by using a preset batch effect removal algorithm to obtain a second feature map;
- the execution module is configured to input the first feature map and the second feature into a preset network integration model for processing to obtain feature optimization parameters.
- the acquisition module includes:
- the first acquisition submodule is used to perform Laplace transform, wavelet transform, image intensity square, image intensity square root, logarithmic transformation, exponential transformation, gradient transformation and local binary mode transformation on the image data respectively;
- the first processing sub-module is configured to use the image data and the image data processed by each transformation method as the multi-channel data.
- processing module includes:
- a second processing submodule configured to calculate the weight of the feature in the feature map by using a preset weight calculation method
- the third processing sub-module is used to calculate the feature map including the weight by using a preset regularized feature weight distribution mechanism
- the first execution sub-module is used for using the attention mechanism to fuse the feature maps containing weights in each channel to obtain the first feature map.
- processing module includes:
- a fourth processing sub-module used for mixing the deep omics parameters as different data sets with a preset data set
- the fifth processing submodule is used to use the center and equipment datasets with the most stable parameters in the group as the reference datasets in the image data;
- the sixth execution sub-module is configured to use the reference data set as a standard to adjust the data set obtained after mixing by using a preset ComBat algorithm to obtain the second feature map.
- execution module includes:
- the sixth processing sub-module is used to input the first feature map and the second feature map into the preset MOE dual-network integration model and output the processed feature map;
- the seventh processing sub-module is used for inputting the processed feature map to a preset decoder to obtain the feature optimization parameter.
- embodiments of the present invention provide a computer device, including a memory and a processor, where computer-readable instructions are stored in the memory, and when the computer-readable instructions are executed by the processor, the processor Steps of the above-mentioned image data processing method are performed.
- embodiments of the present invention provide a storage medium storing computer-readable instructions.
- the computer-readable instructions are executed by one or more processors, the one or more processors execute the above image data processing method. A step of.
- the beneficial effects of the embodiments of the present invention are as follows: the embodiments of the present invention improve medical science in real and complex scenarios by synthesizing from the perspectives of method framework, fusion of radiomics and deep learning methods, feature extraction and screening of deep radiomics, and hybrid expert models, etc. Generalization performance of image segmentation models.
- FIG. 1 is a schematic flowchart of an image data processing method provided by an embodiment of the present invention
- FIG. 2 is a schematic flowchart of a method for preprocessing image data to obtain multi-channel data according to an embodiment of the present invention
- [Correction 07.06.2021 according to Rule 26] 3 is a schematic flowchart of a method for obtaining a feature map including feature weights according to the deep omics parameters provided by an embodiment of the present invention, and fusing the feature maps including weights in each channel to obtain a first feature map;
- FIG. 4 is a schematic flowchart of a method for adjusting the depth-omics parameters of the feature map in each channel by using a preset batch effect removal algorithm provided by an embodiment of the present invention to obtain a second feature map;
- FIG. 5 is a basic structural block diagram of an image data processing apparatus provided by an embodiment of the present invention.
- FIG. 6 is a basic structural block diagram of a computer device provided by an embodiment of the present invention.
- FIG. 1 provides an image data processing method according to an embodiment of the present invention. As shown in FIG. 1, the method specifically includes the following steps:
- the multi-channel data is a variety of expression forms obtained after the influence data is processed by a variety of processing methods.
- the image data includes medical data, such as CT images, ultrasound images, etc., which are applied to the problem of pulmonary nodule segmentation under the condition of multi-center and multi-device.
- Deep learning can be divided into two parts, encoder and decoder.
- the encoder is mainly responsible for extracting the deep features of the image.
- the encoder is a series of convolutional neural networks, which are mainly composed of convolutional layers, pooling layers and activation layers. It mainly classifies and analyzes low-level local pixel values of images to obtain high-level semantic information.
- the decoder is to achieve fixed tasks (including classification, segmentation, recognition, etc.), it collects the semantics of the encoder, understands and compiles, and classifies the pixels with the same semantically similar to complete the segmentation task. Therefore, use a public and similar dataset to train a deep learning model for a certain task.
- the model is effective, keep the weight parameters in the trained model, and use the decoder in it as our pre-training model. Go to Extract the deep-omics parameters of an image. In this way, the feature parameters of different images can be extracted with the same rules without the need to retrain the model and in the case of less training data, and the extracted parameters are more stable.
- the pretrained model is mainly composed of convolutional layers, activation function layers and pooling layers. And referring to the idea of DenseNet, further improvements have been made on the basis of ResNet. Instead of passing only the features of the previous layer to the next layer, the features are multiplexed in multiple layers and passed to the input of each subsequent layer. This makes the model more compact, since the feature map output by any layer in the network can be accessed in all subsequent network layers. This enables the features captured by each network layer to be fully reused, so the network is very compact, and the number of parameters is often less; secondly, it will imply deep supervision. Since the model species contains more shortcut connections, each layer in the network is made.
- any network layer has a direct connection with all subsequent network layers, making it possible for the network layers in two different modules to be connected together through the transformation layer.
- the multi-center problem is discussed for the most mature lung cancer CT image segmentation model.
- an embodiment of the present invention provides a method for preprocessing image data to obtain multi-channel data, including:
- S111 perform Laplace transform, wavelet transform, image intensity square, image intensity square root, logarithmic transformation, exponential transformation, gradient transformation and local binary mode transformation on the image data respectively;
- the Resnet-Unet structure is used as the basic network
- the LUNA16 data set is used to train the Resnet pre-training model
- the phantom image data in the pre-experiment is subjected to Laplace transform, wavelet transform, image intensity Square, square root of image intensity, logarithmic transformation, exponential transformation, gradient transformation, and local binary mode transformations, enter the data of such 9 channels into the pre-training model respectively, and obtain 9*4 feature maps of different scales, 4 features
- the pictures are 1/4, 1/8, 1/16, 1/32 sizes, and the number of channels is 64, 128, 256, and 512, respectively.
- each transformation is an independent information representation dimension, which can expand the original one-dimensional representation into 9 different representations or 9 different channels.
- x, y, z are the three coordinate axis values, respectively, and ⁇ is the standard deviation.
- Wavelet changes. a>0, it becomes a scale factor, and its function is to affect the basic wavelet The function is stretched and ⁇ reflects the displacement.
- Image intensity squared. x and f(x) are the original and filtered image intensities, respectively.
- x and f(x) are the original and filtered image intensities, respectively.
- Exponential transformation. x and f(x) are the original and filtered image intensities, respectively.
- (x c , y c ) is the center pixel
- the brightness is i c
- i p is the difference between adjacent pixels
- s(x) is to calculate the difference between the adjacent position pixel and the middle position pixel
- p is the basis of the middle pixel The number of adjacent pixels.
- an embodiment of the present invention further provides a method for obtaining a feature map including feature weights according to the deep omics parameters, and fusing the feature maps including weights in each channel to obtain a first feature map , including:
- the present invention proposes to return the stable radiomics parameters to the model by means of L1 regularization and utilize them.
- L1 regularization is a commonly used technique in machine learning, and its main purpose is to control model complexity and reduce overfitting.
- the most basic regularization method is to add a penalty term to the original objective function to "penalize" the model with high complexity, as shown in the following formula. It guarantees the sparsity of the model while limiting the complexity of the model, making the model tend to more important features.
- the so-called sparsity means that most of the elements in the model are expected to be 0. Because there are many factors that affect the prediction results, some of the features have no effect on the output at all.
- ⁇ is the weight coefficient vector
- J() is the objective function
- ⁇ ( ⁇ ) is the penalty term, which can be understood as a measure of the "scale” of the model
- the parameter ⁇ controls the control regularity Strengthen or weaken.
- Different ⁇ functions have different preferences for the optimal solution of the weight ⁇ , thus producing different regularization effects.
- the multi-channel and multi-dimensional feature representation can greatly increase the performance of the model, but it will also increase some noise and redundancy in disguised form. Therefore, the attention mechanism is added to the feature fusion step in the model design, and the attention mechanism is used to determine the downstream network. Correlation with multiple representations of the input. Finally, we weight the multi-channel features to get a combined depth feature, as shown in the formula.
- the deep omics parameters are calculated by ICC and CCC, the weight of each feature is calculated by using the calculation result and the ratio of its values, and then a new weighted feature is calculated by using the L1 regularized feature weight distribution mechanism.
- 9*4 feature maps are fused into 4 feature maps, that is, the feature fusion of 9 different input channels, but for the unique jump connection and fusion of Unet, it needs to be fused 4 times, and the features of 4 stages are processed separately. picture.
- This part includes two operations, one is to perform a weighted ADD operation on 9 feature maps to obtain a 1*C feature map F-sum; the second is to perform a Concat operation on 9 feature maps, and then calculate the channel attention weight, Channel attention feature map, and then downsample the 9*C feature map to the 1*C feature map F-Cat. Finally, the two feature maps F-sum and F-cat are Catatized to reduce the dimension to a 1*C feature map.
- U-Net has been proved to be a highly robust deep segmentation model, which is widely used in segmentation tasks of various medical images, and has achieved good results. That's why we chose it as our model. It is mainly composed of convolutional layers, four pooling layers, four upsampling layers and activation functions. The biggest feature is that some semantic features will be lost in the process of upsampling. By splicing, some parts can be recovered. Semantic information, so as to ensure the accuracy of segmentation, and no additional model parameters are introduced in the process.
- Atrous convolution is used.
- the atrous convolution can increase the receptive field of the convolution without adding any parameters, and at the same time does not reduce the resolution of the image, so as to accurately locate the target;
- ResNet is adopted.
- ResNet optimizes the parameters of the deep network through identity transformation without increasing the amount of parameters and ensures the accuracy of the model.
- SPP Spatial Pyramid Pooling
- an embodiment of the present invention also provides a method for using a preset batch effect removal algorithm to adjust the depth-omics parameters of the feature map in each channel to obtain a second feature map.
- the specific method includes:
- the stability analysis based on depth radiomics plus L1 regularization may cause important depth features to be discarded due to instability. Therefore, in order to cope with this situation, a second network model is constructed, which is different from the above
- the first network model used here is an improved batch effect removal algorithm (ComBat).
- ComBat is essentially a method of removing batch effects based on empirical Bayesian methods.
- the representation of each deep omics parameter is shown in the following formula:
- x ijg ⁇ i +X ⁇ i + ⁇ ig + ⁇ ig ⁇ ijg
- x ijg is the value of a single center single device dataset g, patient j, and depth radiomics i.
- ⁇ i is the average value of depth radiomics;
- X is the design matrix of center and equipment parameters;
- ⁇ i is the vector of regression coefficients corresponding to the X matrix,
- ⁇ ijg is the error term assumed to obey a normal distribution;
- ⁇ ig and ⁇ ig are Additive and multiplicative batch effects of deep omics parameters i in dataset g.
- the normalization formula of deep radiomics data is shown in the following formula:
- the improved ComBat algorithm mainly converts the mean and variance of the overall sample in the original algorithm into the mean and variance of the radiomics parameters in the reference dataset. By changing the parameter estimates at the overall level and parameters at the reference dataset level and make adjustments. Its improved data adjustment formula, such as the following formula:
- the mean and variance of the overall data in the batch effect adjustment formula are adjusted to the mean and variance of the reference data set.
- the adjusted data and the reference data set are distributed to the greatest extent. , which also explains why we chose the center and device datasets with the most in-group stable parameters in the phantom data as the reference datasets.
- the embodiment of the present invention also provides a method for inputting the first feature map and the second feature into a preset network integration model for processing to obtain feature optimization parameters, and the method includes:
- the first feature map and the second feature map of the two models are input into the MoE model.
- Set the MoE model hyperparameters and initial parameter values including: the number of underlying models, RMSProp algorithm parameters, batch books and regularization coefficients of small batch stochastic gradient descent, etc.; Judging that the current parameters are the model parameters in the current iteration step, if it is iterative
- the first step is the initial value of the model parameters, otherwise it is the model parameters updated by the RMSProp algorithm in the previous step; in the E step of the EM algorithm, the Q function is calculated, which is a function of the model parameters and the model parameters of the current iteration step; After the final objective optimization function value is determined whether it converges, if it has converged, the current model parameters are output, otherwise it enters the M step of the EM algorithm; in the M step of the EM algorithm, the final objective optimization function is calculated for all parameters.
- the MoE model outputs the integrated feature map.
- the upsampling part adopts the method of bilinear interpolation in order to speed up the calculation.
- MOE dual-network ensemble model can combine multiple models to solve complex problems.
- Each model is called an expert and can solve the problems it is good at under specific conditions, so it will also be obtained under such conditions. higher weight.
- the initial hybrid expert model is mainly trained by the methods of maximum likelihood and gradient ascent. In order to improve the convergence speed of the model, the MoE model based on the expectation maximization algorithm is proposed.
- the embodiment of the present invention mainly adopts a single-layer hybrid expert model architecture, and uses fj to represent the network based on deep omics parameter stability analysis and the network based on deep parameter stability optimization that we use.
- the network that regulates the weights of each expert model is called the gating network.
- the gating network is assumed to be generalized linear. This defines the intermediate variable: where ⁇ j is a weight vector, then the jth output of the gating network is the "softmax" function of ⁇ j as shown in the following formula.
- the mixed expert system can be regarded as a probabilistic Shencheng model, that is, the total probability of generating output y from the input is a mixture of the probabilities of generating y from each component density, where the sum ratio is the value given in the gated network. .
- the parameter ⁇ j in each expert network be the parameter in the gating network, then the total probability is generated by the following formula:
- ⁇ includes the expert model parameter ⁇ j and the gating network parameter v j .
- an embodiment of the present invention further provides an image data processing apparatus, including: a retrieval module 2100 , a processing module 2200 and an execution module 2300 , wherein the obtaining module 2100 is used to obtain the to-be-processed data Image data, preprocessing the image data to obtain multi-channel data, and inputting the multi-channel data to the pre-training encoder to obtain feature maps of multiple channels; the processing module 2200 is used to extract each of the feature maps According to the deep omics parameters, a feature map containing feature weights is obtained according to the deep omics parameters, and the feature maps containing weights in each channel are fused to obtain the first feature map; the processing module 2200 is also used to use the The preset batch effect removal algorithm adjusts the depth-omics parameters of the feature map in each channel to obtain a second feature map; the execution module 2300 is used to input the first feature map and the second feature into the The preset network integration model is processed to obtain feature optimization parameters.
- the obtaining module 2100 is used to obtain the to-be-processed
- the embodiment of the present invention discusses the multi-center problem for the most mature lung cancer CT image segmentation model, from the perspectives of method framework, radiomics and deep learning method fusion, deep radiomics feature extraction and screening, mixed expert model and other perspectives. Research to improve the generalization performance of lung cancer segmentation models in real complex scenes.
- the acquisition module 2100 includes: a first acquisition sub-module, configured to perform Laplace transform, wavelet transform, image intensity square, image intensity square root, logarithmic transformation, exponential transformation on the image data, respectively transformation, gradient transformation and local binary mode transformation; the first processing submodule is used for taking the image data and the image data processed by each transformation method as the multi-channel data.
- the processing module 2200 includes: a second processing sub-module for calculating the weight of the features in the feature map by using a preset weight calculation method; a third processing sub-module for using a preset weight calculation method
- the regularized feature weight distribution mechanism calculates the feature map containing the weight; the first execution sub-module is used to fuse the feature map containing the weight in each channel by using the attention mechanism to obtain the first feature map.
- the processing module 2200 includes: a fourth processing sub-module for mixing the deep omics parameters as different data sets with a preset data set; a fifth processing sub-module for using The center and equipment data sets with the most stable parameters in the group in the image data are used as reference data sets; the sixth execution sub-module is used to use the preset ComBat algorithm as a standard to mix the data sets obtained by using the reference data set as a standard. Adjustment is made to obtain the second feature map.
- the execution module 2300 includes: a sixth processing sub-module, configured to input the first feature map and the second feature map into a preset MOE dual-network integration model and output the processed feature map; a seventh processing sub-module, configured to input the processed feature map to a preset decoder to obtain the feature optimization parameters.
- the embodiment of the present invention discusses the multi-center problem for the most mature lung cancer CT image segmentation model, from the perspectives of method framework, radiomics and deep learning method fusion, deep radiomics feature extraction and screening, mixed expert model and other perspectives. Research to improve the generalization performance of lung cancer segmentation models in real complex scenes.
- FIG. 6 is a block diagram of the basic structure of a computer device according to this embodiment.
- the internal structure of the computer equipment is a schematic diagram.
- the computer device includes a processor, a non-volatile storage medium, a memory, and a network interface connected through a system bus.
- the non-volatile storage medium of the computer device stores an operating system, a database and computer-readable instructions
- the database can store a control information sequence.
- the processor can realize a an image processing method.
- the processor of the computer device is used to provide computing and control capabilities and support the operation of the entire computer device.
- Computer-readable instructions may be stored in the memory of the computer device, and when executed by the processor, the computer-readable instructions may cause the processor to perform an image processing method.
- the network interface of the computer equipment is used for communication with the terminal connection.
- FIG. 6 is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer equipment to which the solution of the present application is applied. Include more or fewer components than shown in the figures, or combine certain components, or have a different arrangement of components.
- the processor is used to execute the specific content of the acquisition module 2100, the processing module 2200, and the execution module 2300 in FIG. 5, and the memory stores program codes and various types of data required to execute the above modules.
- the network interface is used for data transmission between user terminals or servers.
- the memory in this embodiment stores the program codes and data required to execute all sub-modules in the image processing method, and the server can call the server's program codes and data to execute the functions of all the sub-modules.
- the reference feature map is obtained by feature extraction from a high-definition image set in a reference pool. Due to the diversification of images in the high-definition image set, the reference feature map includes all possible local parts. Features, which can provide high-frequency texture information for each low-resolution image not only ensures the richness of features, but also reduces the memory burden. In addition, the reference feature map is searched according to the low-resolution image, and the selected reference feature map can adaptively mask or enhance a variety of different features, so that the details of the low-resolution image are richer.
- the present invention also provides a storage medium storing computer-readable instructions, which, when executed by one or more processors, cause one or more processors to execute the image processing method described in any of the foregoing embodiments A step of.
- the realization of all or part of the processes in the methods of the above embodiments can be accomplished by instructing relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium, and the program is During execution, it may include the processes of the embodiments of the above-mentioned methods.
- the aforementioned storage medium may be a non-volatile storage medium such as a magnetic disk, an optical disk, a read-only storage memory (Read-Only Memory, ROM), or a random access memory (Random Access Memory, RAM) or the like.
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- General Health & Medical Sciences (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Evolutionary Computation (AREA)
- Artificial Intelligence (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Health & Medical Sciences (AREA)
- Image Analysis (AREA)
Abstract
一种影像数据处理方法、装置、计算机及存储介质，应用于图像处理技术领域。方法包括：获取待处理的影像数据，对所述影像数据进行预处理得到多通道数据，并将所述多通道数据输入至预训练编码器得到多个通道的特征图（S110）；提取每个所述特征图的深度组学参数，根据所述深度组学参数得到包含特征权重的特征图，并对每个通道中包含权重的特征图进行融合，得到第一特征图（S120）；将利用预设的批次效应去除算法调整每个通道中所述特征图的深度组学参数，得到第二特征图（S130）；将所述第一特征图和所述第二特征输入至预设的网络集成模型中进行处理，得到特征优化参数（S140）。
Description
本发明实施例涉及图像处理技术领域，尤其是一种影像数据处理方法、装置、计算机及存储介质。
在医学影像人工智能深度学习的方法研究中，乃至在其他通用的深度学习的算法研究中，由于数据偏移、标注稀疏等原因引起的模型泛化性问题成为了研究者们讨论的热点话题之一。随着众多大规模标注的训练数据集(例如ImageNet、IMDB、LIDC-IDRI、DDSM MIAS等)的公开，深度学习在完成计算机视觉与自然语言处理领域中的许多任务上获得了巨大的成功，对于特定任务的表现甚至超过了人类自己。但是，大多数应用场景中特别是医学影像领域，标注数据的获取是一个非常昂贵、耗时、甚至无法完成的过程，加之目前不同医院之间的数据共享和互通程度较低，因此用于训练模型的数据和标注往往是来自公开数据集或者源于单一医院，如果将基于上述数据训练出的模型直接使用到其他医院中去，或者换句话讲就是将模型迁移到无标注或者是稀疏标注的目标域中去，这样的直接迁移会导致模型的准确率大幅度下降。产生这种情况的一个主要原因是数据集偏移(Dataset shift)，从统计学习的观点来看，一个机器学习任务T定义为在一个领域D上的条件概率p(x|y)的建模问题，领域D指一个样本空间及其分布。根据贝叶斯公式，p(x,y)＝p(x|y)p(y)＝p(y|x)p(x)，其中有三个我们可以考虑的概率分布项：输入空间的边缘概率分布p(x),输出空 间的标签分布p(y)，以及表示该机器学习任务的条件概率分布p(x|y)。当源域和目标域的三者之一发生了变化，我们都认为发生了源域与目标域的数据集的分布发生了偏移，即数据集偏移。在我们前期的研究中发现，在复杂医疗场景下的深度学习模型构建过程中，多中心多设备所生成的影像数据偏移也成为了当前医学影像深度学习算法面临的主要挑战问题之一。
对于同样用于分析医学影像的影像组学(radiomics)分析也存在数据集偏移的问题。影像组学分析主要包含图像获取、图像分割、影像组学参数提取和模型建立与验证等步骤，通过计算机高通量定量特征提取的方法实现从医学影像图像(CT、MRI和PET-CT等)中提取海量数据，从而将医学图像中主观性强的定性数据转化为具有客观性的定量数据，并进行数据挖掘分析[2]。影像组学的迅猛发展，在疾病诊断和鉴别诊断、肿瘤分期分级、基因表型预测、治疗方案决策、疗效评估及预后预测等方面取得了较大成果，尤其在肺部肿瘤方面显示出巨大优势[3]。虽然影像组学参数在疾病的筛查、诊断、治疗和预后中展现出了超越传统医疗方式的生物标志物属性，但其在多中心多设备影像上所产生变异性(variability)大、重复性(repeatability)及再现性(reproducibility)差等问题，造成基于影像组学参数所构建的模型泛化能力欠缺，从而大大限制了模型的诊断效能，导致其难以运用到真实医疗场景的实践中。
国内外学者分别针对影像组学特征可重复性问题与深度学习中的数据偏移问题做了大量的研究工作，其中影像组学领域提出了筛选稳定参数、改善信噪比、重采样、超分辨率重建以及Combat补偿等方法；在深度学习多源领域适应领域提出了基于差异的潜伏特征空间变换方法(Discrepancy-based latent space transformation methods)、基 于对抗的潜伏特征空间变换方法(Adversarial latent space transformation methods)和中间域生成方法等[11]。然而这些方法在疾病的多样化以及多中心多设备多参数变化的偏移叠加下，即使使用最成熟的肺癌分割模型，仍存在难以准确捕获目标稳定特征、模型在多源数据测试集中准确率低、模型效率低等诸多问题。因此如何有效提高深度学习模型在多中心多设备数据上的泛化性能具有重要的理论研究意义和广泛的应用前景。
发明内容
为解决上述技术问题，本发明创造的实施例提供一种影像数据处理方法，包括：
获取待处理的影像数据，对所述影像数据进行预处理得到多通道数据，并将所述多通道数据输入至预训练编码器得到多个通道的特征图；
提取每个所述特征图的深度组学参数，根据所述深度组学参数得到包含特征权重的特征图，并对每个通道中包含权重的特征图进行融合，得到第一特征图；
将利用预设的批次效应去除算法调整每个通道中所述特征图的深度组学参数，得到第二特征图；
将所述第一特征图和所述第二特征输入至预设的网络集成模型中进行处理，得到特征优化参数。
进一步地，所述对所述影像数据进行预处理得到多通道数据，包括：
对所述影像数据分别进行拉普拉斯变换、小波变换、图像强度平方、图像强度平方根、对数变换、指数变换、梯度变换和局部二值模式变换；
将所述影像数据以及经每个变换方法处理得到的影像数据作为所述多通道数据。
进一步地，所述根据所述深度组学参数得到包含特征权重的特征图，并对每个通道中包含权重的特征图进行融合，得到第一特征图，包括：
利用预设的权重计算方法计算所述特征图中特征的权重；
采用预设的正则化特征权重分配机制计算包含权重的特征图；
利用注意力机制将每个通道中包含权重的特征图进行融合，得到第一特征图。
进一步地，所述利用预设的批次效应去除算法调整每个通道中所述特征图的深度组学参数，得到第二特征图，包括：
将所述深度组学参数作为不同数据集与预设的数据集进行混合；
以所述影像数据中组内稳定参数最多的中心与设备数据集作为参考数据集；
以所述参考数据集为标准利用预设的ComBat算法对混合后得到的数据集进行调整得到所述第二特征图。
进一步地，所述将所述第一特征图和所述第二特征输入至预设的网络集成模型中进行处理，得到特征优化参数，包括：
将所述第一特征图和所述第二特征图输入至预设的MOE双网络集成模型中输出已处理的特征图；
将已处理的特征图输入至预设的解码器得到所述所述特征优化参数。
为解决上述问题，本发明实施例还提供一种影像数据处理装置，包括：
获取模块，用于获取待处理的影像数据，对所述影像数据进行预 处理得到多通道数据，并将所述多通道数据输入至预训练编码器得到多个通道的特征图；
处理模块，用于提取每个所述特征图的深度组学参数，根据所述深度组学参数得到包含特征权重的特征图，并对每个通道中包含权重的特征图进行融合，得到第一特征图；
所述处理模块，还用于将利用预设的批次效应去除算法调整每个通道中所述特征图的深度组学参数，得到第二特征图；
执行模块，用于将所述第一特征图和所述第二特征输入至预设的网络集成模型中进行处理，得到特征优化参数。
进一步地，所述获取模块包括：
第一获取子模块，用于对所述影像数据分别进行拉普拉斯变换、小波变换、图像强度平方、图像强度平方根、对数变换、指数变换、梯度变换和局部二值模式变换；
第一处理子模块，用于将所述影像数据以及经每个变换方法处理得到的影像数据作为所述多通道数据。
进一步地，所述处理模块包括：
第二处理子模块，用于利用预设的权重计算方法计算所述特征图中特征的权重；
第三处理子模块，用于采用预设的正则化特征权重分配机制计算包含权重的特征图；
第一执行子模块，用于利用注意力机制将每个通道中包含权重的特征图进行融合，得到第一特征图。
进一步地，所述处理模块包括：
第四处理子模块，用于将所述深度组学参数作为不同数据集与预设的数据集进行混合；
第五处理子模块，用于以所述影像数据中组内稳定参数最多的中心与设备数据集作为参考数据集；
第六执行子模块，用于以所述参考数据集为标准利用预设的ComBat算法对混合后得到的数据集进行调整得到所述第二特征图。
进一步地，所述执行模块包括：
第六处理子模块，用于将所述第一特征图和所述第二特征图输入至预设的MOE双网络集成模型中输出已处理的特征图；
第七处理子模块，用于将已处理的特征图输入至预设的解码器得到所述特征优化参数。
为解决上述问题本发明实施例提供一种计算机设备，包括存储器和处理器，所述存储器中存储有计算机可读指令，所述计算机可读指令被所述处理器执行时，使得所述处理器执行上述影像数据处理方法的步骤。
为解决上述问题本发明实施例提供一种存储有计算机可读指令的存储介质，所述计算机可读指令被一个或多个处理器执行时，使得一个或多个处理器执行上述影像数据处理方法的步骤。
本发明实施例的有益效果是：本发明实施例通过从方法框架、影像组学与深度学习方法融合、深度影像组学特征提取与筛选、混合专家模型等角度进行综合，提高真实复杂场景下医学图像分割模型的泛化性能。
为了更清楚地说明本发明实施例中的技术方案，下面将对实施例描述中所需要使用的附图作简单地介绍，显而易见地，下面描述中的附图仅仅是本发明的一些实施例，对于本领域技术人员来讲，在不付 出创造性劳动的前提下，还可以根据这些附图获得其他的附图。
图1为本发明实施例提供的影像数据处理方法的流程示意图；
图2为本发明实施例提供的对影像数据进行预处理得到多通道数据方法的流程示意图；
[根据细则26改正07.06.2021]
图3为本发明实施例提供的根据所述深度组学参数得到包含特征权重的特征图，并对每个通道中包含权重的特征图进行融合，得到第一特征图的方法的流程示意图；
图3为本发明实施例提供的根据所述深度组学参数得到包含特征权重的特征图，并对每个通道中包含权重的特征图进行融合，得到第一特征图的方法的流程示意图；
图4为本发明实施例提供的利用预设的批次效应去除算法调整每个通道中所述特征图的深度组学参数，得到第二特征图方法的流程示意图；
图5为本发明实施例提供的影像数据处理装置基本结构框图；
图6为本发明实施例提供的计算机设备基本结构框图。
为了使本技术领域的人员更好地理解本发明方案，下面将结合本发明实施例中的附图，对本发明实施例中的技术方案进行清楚、完整地描述。
在本发明的说明书和权利要求书及上述附图中的描述的一些流程中，包含了按照特定顺序出现的多个操作，但是应该清楚了解，这些操作可以不按照其在本文中出现的顺序来执行或并行执行，操作的序号如101、102等，仅仅是用于区分开各个不同的操作，序号本身不代表任何的执行顺序。另外，这些流程可以包括更多或更少的操作，并且这些操作可以按顺序执行或并行执行。需要说明的是，本文中的“第一”、“第二”等描述，是用于区分不同的消息、设备、模块等，不代表先后顺序，也不限定“第一”和“第二”是不同的类型。
下面将结合本发明实施例中的附图，对本发明实施例中的技术方 案进行清楚、完整地描述，显然，所描述的实施例仅仅是本发明一部分实施例，而不是全部的实施例。基于本发明中的实施例，本领域技术人员在没有作出创造性劳动前提下所获得的所有其他实施例，都属于本发明保护的范围。
请参照图1，图1为本发明实施例提供一种影像数据处理方法，如图1所示，该方法具体包括如下步骤：
S110、获取待处理的影像数据，对所述影像数据进行预处理得到多通道数据，并将所述多通道数据输入至预训练编码器得到多个通道的特征图；
本发明实施例中，多通道数据为影响数据经多种处理方法处理后得到的多种表达形式。其中，影像数据包括医学数据，例如CT图像，超声图像等，应用于多中心多设备条件下的肺结节分割问题。
S120、提取每个所述特征图的深度组学参数，根据所述深度组学参数得到包含特征权重的特征图，并对每个通道中包含权重的特征图进行融合，得到第一特征图；
本实施例中，通过预训练模型提取图像的深度组学参数。深度学习可以分为两部分，分别为编码器和解码器。编码器主要负责提取图像的深度特征，编码器是一连串的卷积神经网络，该网络主要由卷积层、池化层和激活层组成。它主要对图像的低级局域像素值进行归类和分析，从而获得高阶语义信息。解码器是为了实现固定的任务(包括分类、分割、识别等)，它收集编码器的语义，进行理解和编译，对相同语义相近的像素进行分类，从而完成分割任务。因此，用公开的、相近的数据集训练某个任务的深度学习模型，在模型效果较好的情况下，保留训练好模型中的权重参数，把其中的解码器作为我们的预训练模型，去提取图像的深度组学参数。如此可以在不需要重新训 练模型的情况下，以及在训练数据较少的情况下，能够保证以相同的规则提取不同图像的特征参数，也使得提取的参数更加的稳定。
预训练模型主要由卷积层、激活函数层和池化层组成。并参照了DenseNet的思想，在ResNet的基础上做了进一步改进，不是将前层特征只传递给下一层，而是将特征进行多层复用，传递给后面的每一层输入，如此会使得模型更加紧凑，由于网络中任何一层输出的特征图谱，都能够在后面所有的网络层访问。这使得各个网络层捕获的特征都能够被充分复用，因此网络非常紧凑，参数数量往往也更少；其次，会隐含深度监督，由于模型种含有较多的快捷连接，使得网络中各个层都能独立的接收损失函数的梯度传到，这就是“深度监督”方式；再次，随机的深度和连接。在模型内部，任意一个网络层与后面所有网络层都有直接的连接，使得两个不同的模块中的网络层隔着转换层连接在一起的情况成为可能。
S130、将利用预设的批次效应去除算法调整每个通道中所述特征图的深度组学参数，得到第二特征图；
S140、将所述第一特征图和所述第二特征输入至预设的网络集成模型中进行处理，得到特征优化参数。
本发明实施例提供的影像数据处理方法中，针对最成熟的肺癌CT影像分割模型进行多中心问题的探讨，从方法框架、影像组学与深度学习方法融合、深度影像组学特征提取与筛选、混合专家模型等角度进行系统的研究，提高真实复杂场景下肺癌分割模型的泛化性能。
如图2所示，本发明实施例提供一种对影像数据进行预处理得到多通道数据的方法，包括：
S111、对所述影像数据分别进行拉普拉斯变换、小波变换、图像强度平方、图像强度平方根、对数变换、指数变换、梯度变换和局部 二值模式变换；
S112、将所述影像数据以及经每个变换方法处理得到的影像数据作为所述多通道数据。
本发明实施例中，采用Resnet-Unet结构作为基础网络，利用LUNA16的数据集训练出Resnet的预训练模型，然后对预实验中的体模影像数据进行拉普拉斯变换、小波变换、图像强度平方、图像强度平方根、对数变换、指数变换、梯度变换和局部二值模式等变换，将这样9个通道的数据分别进入预训练模型，得到9*4个不同尺度的特征图，4个特征图分别为1/4、1/8、1/16、1/32尺寸，通道数分别为64、128、256、512。
具体地，通过滤波器或数学变换，如下公式所示，将原图像转换为新的变换影像，输入由原来的单一的原图增加到若干种包含原图部分特征的变换图像，从而增加了输入的多样性。在预实验中我们也发现图像经过不同的变换，其深度组学参数在不同的变换图像中稳定性分布是具有较大差异的。换句话说，每一种变换都是独立的信息表达维度，这可以将原来的一维表达方式扩展为9种不同的表达方式或者称之为9个不同的通道。
高斯拉普拉斯变化。x,y,z分别为三个坐标轴值，σ为标准差。
小波变化。a>0，成为尺度因子，作用是对基本小波
函数做伸缩，τ反映位移。
图像强度平方。x和f(x)分别是原图和滤波后的图像强度。
f(x)＝(cx)
2,其中
图像强度平方根。x和f(x)分别是原图和滤波后的图像强度。
对数变换。x和f(x)分别是原图和滤波后的图像强度。
指数变换。x和f(x)分别是原图和滤波后的图像强度。
梯度变换。f为图像的像素矩阵。
局部二值模式。(x
c,y
c)为中心像素，亮度为i
c，i
p为相邻像素的差，s(x)就是计算相邻位置像素与中间位置像素的差值，p为该中间像素依据的邻近像素的个数。
如图3所示，本发明实施例还提供一种根据所述深度组学参数得到包含特征权重的特征图，并对每个通道中包含权重的特征图进行融合，得到第一特征图的方法，具体包括：
S121、利用预设的权重计算方法计算所述特征图中特征的权重；
S122、采用预设的正则化特征权重分配机制计算包含权重的特征图；
S123、利用注意力机制将每个通道中包含权重的特征图进行融合， 得到第一特征图。
筛选稳定的深度组学参数并应用到网络的后期计算中是提升模型整体泛化性的有效途径。本发明提出将稳定的影像组学参数通过L1正则化的方式返回至模型并加以利用。
通过对前期的多中心体模数据进行深度组学参数提取，再利用ICC及CCC等统计公式去选择更加关注、更加稳定的深度特征。L1正则化是机器学习中一种常用的技术，其主要目的是控制模型复杂度，减小过拟合。最基本的正则化方法是在原目标函数中添加惩罚项，对复杂度高的模型进行“惩罚”，如以下公式所示。它在限制模型复杂度的同时，保证了模型的稀疏性，使模型倾向于更重要的特征。所谓稀疏性，就是希望模型中大部分的元素都是0。因为影响预测结果的因素由很多，但是其中一部分的特征对输出是完全没有影响的，在最小化目标函数的时候虽然加入这些特征会降低训练误差，但是实际应用这些无效的特征信息会干扰输出的正确预测，所以我们引入稀疏将这些特征的权重置为0，从而起到选择有效特征的作用。同时它对噪声的输入几乎是不变的，从而保证了模型的鲁棒性。
J(θ；X；y)＝L
emp(θ；X；y)+αΩ(θ)
Ω(θ)＝||ω||
1
式中X、y训练样本和相应标签，ω为权重系数向量；J()为目标函数，Ω(ω)即为惩罚项，可理解为模型“规模”的某种度量；参数α控制控制正则化强弱。不同的Ω函数对权重ω的最优解有不同的偏好，因而会产生不同的正则化效果。
多通道多维度的特征表示可以大大增加模型的性能，但是同时也会变相增加一些噪声和冗余，因此在模型设计的时候将注意力机制加入了特征融合步骤，利用注意力机制以确定下游网络与输入的多个表 示的相关性。最终，我们将多通道的特征经过加权之后得到一个组合的深度特征，如公式所示。
式中w
i是各个通道的权重系数，F
i为每一个通道输出的特征，F为最终的组合特征。
本发明实施例将深度组学参数进行ICC及CCC的计算，利用计算结果及其值的占比计算出每个特征的权重，之后运用L1正则化的特征权重分配机制计算出新的带权重的9*4个特征图。将9*4个特征图融合为4个特征图，也就是将9个不同输入的通道进行特征融合，但是为了Unet特有的跳跃连接、融合，这里需要融合4次，分别处理4个阶段的特征图。此部分包括两个操作，一个是对9个特征图进行加权ADD操作，得到1*C的特征图F-sum；第二个是对9个特征图进行Concat操作，然后计算通道注意力权重、通道注意力特征图，然后将9*C的特征图降采样到1*C特征图F-Cat。最后将F-sum和F-cat两个特征图Cat起来，进行降维到1*C的特征图。
本发明实施例中，U-Net已经被证明是具有高鲁棒的深度分割模型，它广泛应用于各类医疗影像的分割任务，取得了良好的效果。这也是我们选择它作为我们的模型的原因。它主要由卷积层、四个池化层、四个上采样层以及激活函数等组成，其中最大的特色是将在上采样的过程中会丢失部分语义特征，通过拼接的方式，可以恢复部分的语义信息，从而保证分割的精度，并且过程中不会引入额外的模型参数。
同时，在原来的基础上，对U-Net做了以下几点的改进：1)使用空洞卷积。空洞卷积可以在不增加任何参数的情况下，增大卷积的感受野，同时不降低图像的分辨率，从而精确定位目标；2)采用了 ResNet思想。随着模型的加深和大参数量，使得深层网络出现退化。ResNet在不增加参数量的情况下，通过恒等变换，使得深层网络参数得以优化，保证了模型的精度。3)采用SPP(Spatial Pyramid Pooling)替代原来单一的输入大小。对所给定的输入以不同采样率的卷积并行采样，相当于以多个比例捕捉图像的上下文。同时使得任意大小的特征图都能够转换成固定大小的特征向量，提高了模型的鲁棒性和精度。
如图4所示，本发明实施例还提供一种利用预设的批次效应去除算法调整每个通道中所述特征图的深度组学参数，得到第二特征图的方法，具体方法包括：
S141、将所述深度组学参数作为不同数据集与预设的数据集进行混合；
S142、以所述影像数据中组内稳定参数最多的中心与设备数据集作为参考数据集；
S143、以所述参考数据集为标准利用预设的ComBat算法对混合后得到的数据集进行调整得到所述第二特征图。
本发明实施例采用基于深度影像组学稳定性分析加上L1正则化可能会出现重要深度特征由于不稳定而被丢弃的情况，因此为了应对这种情况构建了第二种网络模型，区别于上述第一种网络模型这里采用的是改进的批次效应去除算法(ComBat)。
ComBat其实质是一种基于经验贝叶斯方法的去除批量效应方法，对于每个深度组学参数的表示如下述公式所示：
x
ijg＝α
i+Xβ
i+γ
ig+δ
igε
ijg
式中x
ijg为单个中心单个设备数据集g、病人j、深度影像组学i的值。α
i为深度影像组学的平均值；X为中心与设备参数的设计矩阵；β
i为对应X矩阵的回归系数的向量，ε
ijg为误差项假设服从正态分布； γ
ig和δ
ig为数据集g中深度组学参数i加性和乘性批次效应。深度影像组学数据标准化公式如以下公式所示：
最终的批次效应调整数据如下述公式所示：
改进的ComBat算法主要是将原始算法中总体样本的平均值和方差转换为参考数据集中影像组学参数的平均值和方差。通过改变整体水平的参数估计值
和
为参考数据集水平的参数
及
进行调整。其改进后的数据调整公式，如以下公式：
式中g＝r为参考数据集，与改进前相比将批次效应调整公式中数据整体的均值和方差调整为参考数据集的均值和方差，调整后的数据与参考数据集分布最大程度重叠，这也解释了我们选择体模数据中组内稳定参数最多的中心与设备数据集作为参考数据集的原因。
本发明实施例还提供一种将第一特征图和第二特征输入至预设的网络集成模型中进行处理，得到特征优化参数的方法，该方法包括：
将所述第一特征图和所述第二特征图输入至预设的MOE双网络集成模型中输出已处理的特征图；
将已处理的特征图输入至预设的解码器得到所述特征优化参数。
本发明实施例中，首先输入两个模型的第一特征图和第二特征图至MoE模型。设定MoE模型超参数和参数初值包含：底层模型数量、RMSProp算法参数、小批量随机梯度下降的批量书和正则化系数等； 判断当前参数为当前迭代步下的模型参数，若为迭代的第一步则为模型参数初值，否则为经过上一步RMSProp算法更新过后的模型参数；在EM算法的E步，计算Q函数，其是关于模型参数和当前迭代步模型参数的函数；在得到最终的目标优化函数值后判断其是否收敛，若已收敛，则输出当前模型参数，否则进入EM算法的M步；在EM算法的M步，将最终目标优化函数分别对所有参数求偏微分，然后用RMSProp算法更新当前参数，回到判断当前参数为当前迭代步下的模型参数这一步。MoE模型输出集成后的特征图，当实现跳跃连接和上采样的解码器部分，我们预采用的是维度分别是(倒序：从最大特征图开始向下的维度)[64,96,128,256,512]。上采样部分为了加快计算采用的是双线性插值的方法。最后利用前期研究中的真实病人数据对模型进行训练与验证。
需要说明的是，MOE双网络集成模型可以将多个模型进行结合来解决复杂问题，每个模型被称为专家，能在特定条件下解决自己擅长的问题，因而也会在这样的条件下获得更高的权重。最初的混合专家模型主要采用最大似然和梯度上升的方法进行训练，为了提升模型的收敛速度基于期望极大算法的MoE模型被提出。
本发明实施例主要采用单层的混合专家模型架构，用f
j来表示我们使用的基于深度组学参数稳定性分析的网络和基于深度参数稳定性优化的网络，对于给定的输入x，每个网络都能独立的给各自的输出：u
j＝f
j(x)。将调控各专家模型权重的网络称为门控网络，在建模时，假设门控网络是广义线性的。由此定义中间变量：
其中υ
j是一个权重向量，则门控网络的第j个输出是ξ
j的“softmax”函数如以下公式所示。
得到专家输出和门控网络后，最后模型的输出为个专家输出的加权和：u＝∑
jg
ju
j。可以将混合专家系统看成是一个概率申城模型，即由输入生成输出y的总概率是来自每个组分密度生成y的概率的混合，其中呼和比例即门控网络中给出的值。设每个专家网络中的参数υ
j为门控网络中的参数，则总概率由下式生成：
P(y|x,θ)＝∑
jg
j(x,v
j)P(y|x,θ
j)
其中θ包括专家模型参数θ
j以及门控网络参数v
j。
如图5所示，为了解决上述问题，本发明实施例还提供一种影像数据处理装置，包括：取模块2100、处理模块2200和执行模块2300，其中，获取模块2100，用于获取待处理的影像数据，对所述影像数据进行预处理得到多通道数据，并将所述多通道数据输入至预训练编码器得到多个通道的特征图；处理模块2200，用于提取每个所述特征图的深度组学参数，根据所述深度组学参数得到包含特征权重的特征图，并对每个通道中包含权重的特征图进行融合，得到第一特征图；处理模块2200，还用于将利用预设的批次效应去除算法调整每个通道中所述特征图的深度组学参数，得到第二特征图；执行模块2300，用于将所述第一特征图和所述第二特征输入至预设的网络集成模型中进行处理，得到特征优化参数。
本发明实施例针对最成熟的肺癌CT影像分割模型进行多中心问题的探讨，从方法框架、影像组学与深度学习方法融合、深度影像组学特征提取与筛选、混合专家模型等角度进行系统的研究，提高真实复杂场景下肺癌分割模型的泛化性能。
在一些实施例中，所述获取模块2100包括：第一获取子模块， 用于对所述影像数据分别进行拉普拉斯变换、小波变换、图像强度平方、图像强度平方根、对数变换、指数变换、梯度变换和局部二值模式变换；第一处理子模块，用于将所述影像数据以及经每个变换方法处理得到的影像数据作为所述多通道数据。
在一些实施例中，所述处理模块2200包括：第二处理子模块，用于利用预设的权重计算方法计算所述特征图中特征的权重；第三处理子模块，用于采用预设的正则化特征权重分配机制计算包含权重的特征图；第一执行子模块，用于利用注意力机制将每个通道中包含权重的特征图进行融合，得到第一特征图。
在一些实施例中，所述处理模块2200包括：第四处理子模块，用于将所述深度组学参数作为不同数据集与预设的数据集进行混合；第五处理子模块，用于以所述影像数据中组内稳定参数最多的中心与设备数据集作为参考数据集；第六执行子模块，用于以所述参考数据集为标准利用预设的ComBat算法对混合后得到的数据集进行调整得到所述第二特征图。
在一些实施例中，所述执行模块2300包括：第六处理子模块，用于将所述第一特征图和所述第二特征图输入至预设的MOE双网络集成模型中输出已处理的特征图；第七处理子模块，用于将已处理的特征图输入至预设的解码器得到所述特征优化参数。
本发明实施例针对最成熟的肺癌CT影像分割模型进行多中心问题的探讨，从方法框架、影像组学与深度学习方法融合、深度影像组学特征提取与筛选、混合专家模型等角度进行系统的研究，提高真实复杂场景下肺癌分割模型的泛化性能。
为解决上述技术问题，本发明实施例还提供计算机设备。具体请参阅图6，图6为本实施例计算机设备基本结构框图。
如图6所示，计算机设备的内部结构示意图。如图6所示，该计算机设备包括通过系统总线连接的处理器、非易失性存储介质、存储器和网络接口。其中，该计算机设备的非易失性存储介质存储有操作系统、数据库和计算机可读指令，数据库中可存储有控件信息序列，该计算机可读指令被处理器执行时，可使得处理器实现一种图像处理方法。该计算机设备的处理器用于提供计算和控制能力，支撑整个计算机设备的运行。该计算机设备的存储器中可存储有计算机可读指令，该计算机可读指令被处理器执行时，可使得处理器执行一种图像处理方法。该计算机设备的网络接口用于与终端连接通信。本领域技术人员可以理解，图6中示出的结构，仅仅是与本申请方案相关的部分结构的框图，并不构成对本申请方案所应用于其上的计算机设备的限定，具体的计算机设备可以包括比图中所示更多或更少的部件，或者组合某些部件，或者具有不同的部件布置。
本实施方式中处理器用于执行图5中获取模块2100、处理模块2200和执行模块2300的具体内容，存储器存储有执行上述模块所需的程序代码和各类数据。网络接口用于向用户终端或服务器之间的数据传输。本实施方式中的存储器存储有图像处理方法中执行所有子模块所需的程序代码及数据，服务器能够调用服务器的程序代码及数据执行所有子模块的功能。
本发明实施例提供的计算机设备，其中的参考特征图是对参考池中的高清图像集进行特征提取得到的，由于高清图像集中图像的多样化，参考特征图中包含了所有可能用到的局部特征，可以为每一张低分辨率图像提供高频纹理信息不仅保证了特征的丰富性，还可以减轻了内存负担。此外，根据低分辨率图像来搜索参考特征图，选择的参考特征图可以自适应的屏蔽或增强多种不同的特征，使低分辨率图像 的细节更加丰富。
本发明还提供一种存储有计算机可读指令的存储介质，所述计算机可读指令被一个或多个处理器执行时，使得一个或多个处理器执行上述任一实施例所述图像处理方法的步骤。
本领域普通技术人员可以理解实现上述实施例方法中的全部或部分流程，是可以通过计算机程序来指令相关的硬件来完成，该计算机程序可存储于一计算机可读取存储介质中，该程序在执行时，可包括如上述各方法的实施例的流程。其中，前述的存储介质可为磁碟、光盘、只读存储记忆体(Read-Only Memory，ROM)等非易失性存储介质，或随机存储记忆体(Random Access Memory，RAM)等。
应该理解的是，虽然附图的流程图中的各个步骤按照箭头的指示依次显示，但是这些步骤并不是必然按照箭头指示的顺序依次执行。除非本文中有明确的说明，这些步骤的执行并没有严格的顺序限制，其可以以其他的顺序执行。而且，附图的流程图中的至少一部分步骤可以包括多个子步骤或者多个阶段，这些子步骤或者阶段并不必然是在同一时刻执行完成，而是可以在不同的时刻执行，其执行顺序也不必然是依次进行，而是可以与其他步骤或者其他步骤的子步骤或者阶段的至少一部分轮流或者交替地执行。
以上所述仅是本发明的部分实施方式，应当指出，对于本技术领域的普通技术人员来说，在不脱离本发明原理的前提下，还可以做出若干改进和润饰，这些改进和润饰也应视为本发明的保护范围。
Claims (10)
- 一种影像数据处理方法，其特征在于，包括：获取待处理的影像数据，对所述影像数据进行预处理得到多通道数据，并将所述多通道数据输入至预训练编码器得到多个通道的特征图；提取每个所述特征图的深度组学参数，根据所述深度组学参数得到包含特征权重的特征图，并对每个通道中包含权重的特征图进行融合，得到第一特征图；将利用预设的批次效应去除算法调整每个通道中所述特征图的深度组学参数，得到第二特征图；将所述第一特征图和所述第二特征输入至预设的网络集成模型中进行处理，得到特征优化参数。
- 根据权利要求1所述的影像数据处理方法，其特征在于，所述对所述影像数据进行预处理得到多通道数据，包括：对所述影像数据分别进行拉普拉斯变换、小波变换、图像强度平方、图像强度平方根、对数变换、指数变换、梯度变换和局部二值模式变换；将所述影像数据以及经每个变换方法处理得到的影像数据作为所述多通道数据。
- 根据权利要求1所述的影像数据处理方法，其特征在于，所述根据所述深度组学参数得到包含特征权重的特征图，并对每个通道中包含权重的特征图进行融合，得到第一特征图，包括：利用预设的权重计算方法计算所述特征图中特征的权重；采用预设的正则化特征权重分配机制计算包含权重的特征图；利用注意力机制将每个通道中包含权重的特征图进行融合，得到 第一特征图。
- 根据权利要求1所述的影像数据处理方法，其特征在于，所述利用预设的批次效应去除算法调整每个通道中所述特征图的深度组学参数，得到第二特征图，包括：将所述深度组学参数作为不同数据集与预设的数据集进行混合；以所述影像数据中组内稳定参数最多的中心与设备数据集作为参考数据集；以所述参考数据集为标准利用预设的ComBat算法对混合后得到的数据集进行调整得到所述第二特征图。
- 根据权利要求1所述的影像数据处理方法，其特征在于，所述将所述第一特征图和所述第二特征输入至预设的网络集成模型中进行处理，得到特征优化参数，包括：将所述第一特征图和所述第二特征图输入至预设的MOE双网络集成模型中输出已处理的特征图；将已处理的特征图输入至预设的解码器得到所述特征优化参数。
- 一种影像数据处理装置，其特征在于，包括：获取模块，用于获取待处理的影像数据，对所述影像数据进行预处理得到多通道数据，并将所述多通道数据输入至预训练编码器得到多个通道的特征图；处理模块，用于提取每个所述特征图的深度组学参数，根据所述深度组学参数得到包含特征权重的特征图，并对每个通道中包含权重的特征图进行融合，得到第一特征图；所述处理模块，还用于将利用预设的批次效应去除算法调整每个通道中所述特征图的深度组学参数，得到第二特征图；执行模块，用于将所述第一特征图和所述第二特征输入至预设的 网络集成模型中进行处理，得到特征优化参数。
- 根据权利要求6所述的影像数据处理装置，其特征在于，所述获取模块包括：第一获取子模块，用于对所述影像数据分别进行拉普拉斯变换、小波变换、图像强度平方、图像强度平方根、对数变换、指数变换、梯度变换和局部二值模式变换；第一处理子模块，用于将所述影像数据以及经每个变换方法处理得到的影像数据作为所述多通道数据。
- 根据权利要求6所述的影像数据处理装置，其特征在于，所述处理模块包括：第二处理子模块，用于利用预设的权重计算方法计算所述特征图中特征的权重；第三处理子模块，用于采用预设的正则化特征权重分配机制计算包含权重的特征图；第一执行子模块，用于利用注意力机制将每个通道中包含权重的特征图进行融合，得到第一特征图。
- 一种计算机设备，包括存储器和处理器，所述存储器中存储有计算机可读指令，所述计算机可读指令被所述处理器执行时，使得所述处理器执行如权利要求1至5中任一项权利要求所述影像数据处理方法的步骤。
- 一种存储有计算机可读指令的存储介质，所述计算机可读指令被一个或多个处理器执行时，使得一个或多个处理器执行如权利要求1至5中任一项权利要求所述影像数据处理方法的步骤。
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| PCT/CN2021/088127 WO2022221991A1 (zh) | 2021-04-19 | 2021-04-19 | 一种影像数据处理方法、装置、计算机及存储介质 |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| PCT/CN2021/088127 WO2022221991A1 (zh) | 2021-04-19 | 2021-04-19 | 一种影像数据处理方法、装置、计算机及存储介质 |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| WO2022221991A1 true WO2022221991A1 (zh) | 2022-10-27 |
Family
ID=83723642
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| PCT/CN2021/088127 WO2022221991A1 (zh) | 2021-04-19 | 2021-04-19 | 一种影像数据处理方法、装置、计算机及存储介质 |
Country Status (1)
| Country | Link |
|---|---|
| WO (1) | WO2022221991A1 (zh) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116310513A (zh) * | 2023-02-14 | 2023-06-23 | 成都泰莱生物科技有限公司 | 基于肺部CT与5hmC标志物融合的肺结节分类方法及产品 |
| CN117495876A (zh) * | 2023-12-29 | 2024-02-02 | 山东大学齐鲁医院 | 基于深度学习的冠状动脉图像分割方法及系统 |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109978850A (zh) * | 2019-03-21 | 2019-07-05 | 华南理工大学 | 一种多模态医学影像半监督深度学习分割系统 |
| US20190254611A1 (en) * | 2018-02-21 | 2019-08-22 | Case Western Reserve University | Predicting disease recurrence following trimodality therapy in non-small cell lung cancer using computed tomography derived radiomic features and clinico-patholigic features |
| CN110427954A (zh) * | 2019-07-26 | 2019-11-08 | 中国科学院自动化研究所 | 基于肿瘤影像的多区域的影像组学特征提取方法 |
| CN111275130A (zh) * | 2020-02-18 | 2020-06-12 | 上海交通大学 | 基于多模态的深度学习预测方法、系统、介质及设备 |
| CN112365980A (zh) * | 2020-11-16 | 2021-02-12 | 复旦大学附属华山医院 | 脑肿瘤多靶点辅助诊断与前瞻性治疗演化可视化方法及系统 |
-
2021
- 2021-04-19 WO PCT/CN2021/088127 patent/WO2022221991A1/zh active Application Filing
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20190254611A1 (en) * | 2018-02-21 | 2019-08-22 | Case Western Reserve University | Predicting disease recurrence following trimodality therapy in non-small cell lung cancer using computed tomography derived radiomic features and clinico-patholigic features |
| CN109978850A (zh) * | 2019-03-21 | 2019-07-05 | 华南理工大学 | 一种多模态医学影像半监督深度学习分割系统 |
| CN110427954A (zh) * | 2019-07-26 | 2019-11-08 | 中国科学院自动化研究所 | 基于肿瘤影像的多区域的影像组学特征提取方法 |
| CN111275130A (zh) * | 2020-02-18 | 2020-06-12 | 上海交通大学 | 基于多模态的深度学习预测方法、系统、介质及设备 |
| CN112365980A (zh) * | 2020-11-16 | 2021-02-12 | 复旦大学附属华山医院 | 脑肿瘤多靶点辅助诊断与前瞻性治疗演化可视化方法及系统 |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116310513A (zh) * | 2023-02-14 | 2023-06-23 | 成都泰莱生物科技有限公司 | 基于肺部CT与5hmC标志物融合的肺结节分类方法及产品 |
| CN116310513B (zh) * | 2023-02-14 | 2023-12-05 | 成都泰莱生物科技有限公司 | 基于肺部CT与5hmC标志物融合的肺结节分类方法及产品 |
| CN117495876A (zh) * | 2023-12-29 | 2024-02-02 | 山东大学齐鲁医院 | 基于深度学习的冠状动脉图像分割方法及系统 |
| CN117495876B (zh) * | 2023-12-29 | 2024-03-26 | 山东大学齐鲁医院 | 基于深度学习的冠状动脉图像分割方法及系统 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Liu et al. | Connecting image denoising and high-level vision tasks via deep learning | |
| CN110889853B (zh) | 基于残差-注意力深度神经网络的肿瘤分割方法 | |
| WO2019120110A1 (zh) | 图像重建方法及设备 | |
| CN110889852B (zh) | 基于残差-注意力深度神经网络的肝脏分割方法 | |
| CN109711426B (zh) | 一种基于gan和迁移学习的病理图片分类装置及方法 | |
| WO2022221991A1 (zh) | 一种影像数据处理方法、装置、计算机及存储介质 | |
| CN110706214B (zh) | 融合条件随机与残差的三维U-Net大脑肿瘤分割方法 | |
| CN110263801B (zh) | 图像处理模型生成方法及装置、电子设备 | |
| CN112465120A (zh) | 一种基于进化方法的快速注意力神经网络架构搜索方法 | |
| CN111968138B (zh) | 基于3d动态边缘不敏感性损失函数的医学图像分割方法 | |
| Chen et al. | Binarized neural architecture search | |
| CN111667483A (zh) | 多模态图像的分割模型的训练方法、图像处理方法和装置 | |
| CN112990359B (zh) | 一种影像数据处理方法、装置、计算机及存储介质 | |
| CN116779091B (zh) | 一种多模态网络互联融合的胸部影像诊断报告自动生成方法 | |
| CN113112534A (zh) | 一种基于迭代式自监督的三维生物医学图像配准方法 | |
| CN111524140A (zh) | 基于cnn和随机森林法的医学图像语义分割方法 | |
| CN113743474A (zh) | 基于协同半监督卷积神经网络的数字图片分类方法与系统 | |
| Calisto et al. | Self-adaptive 2D-3D ensemble of fully convolutional networks for medical image segmentation | |
| Zhao et al. | Deeply supervised active learning for finger bones segmentation | |
| Ma et al. | Attention-based cross-layer domain alignment for unsupervised domain adaptation | |
| Tang et al. | Lesion segmentation and RECIST diameter prediction via click-driven attention and dual-path connection | |
| Karagoz et al. | An unsupervised transfer learning model based on convolutional auto encoder for non-alcoholic steatohepatitis activity scoring and fibrosis staging of liver histopathological images | |
| CN117078941A (zh) | 一种基于上下文级联注意力的心脏mri分割方法 | |
| Huang | Image super-resolution reconstruction based on generative adversarial network model with double discriminators | |
| CN116188396A (zh) | 一种图像分割方法、装置、设备及介质 |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| 121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 21937242 Country of ref document: EP Kind code of ref document: A1 |
|
| NENP | Non-entry into the national phase |
Ref country code: DE |
|
| 122 | Ep: pct application non-entry in european phase |
Ref document number: 21937242 Country of ref document: EP Kind code of ref document: A1 |