CN109493308B - Medical image synthesis and classification method for generating confrontation network based on condition multi-discrimination - Google Patents

Abstract

The invention discloses a medical image synthesis and classification method for generating a confrontation network based on condition multi-discrimination, which comprises the following steps: firstly, segmenting a lesion Region in a Computed Tomography (CT) image, and extracting a Region of Interest (ROIs) of the lesion; secondly, preprocessing data of the lesion ROIs extracted from the first step; thirdly, designing a model architecture for generating a confrontation Network (CMDGAN) based on Multi-condition judgment, and training the model architecture by using the image in the second step to obtain a generated model; fourthly, performing synthetic data enhancement on the extracted lesion ROIs by using the generated model obtained in the third step; and fifthly, designing and training a multi-scale residual error Network (MRNet for short). The method provided by the invention can generate a high-quality synthetic medical image data set, and the classification accuracy of the classification network on the test image is higher, so that auxiliary diagnosis can be better provided for medical workers.

Description

Medical image synthesis and classification method for generating confrontation network based on condition multi-discrimination

Technical Field

The invention relates to a medical image synthesis and classification for conditional multi-discriminant generation of confrontation networks

Background

In recent years, the classification effect of images has surpassed that of human beings in many aspects by relying on the powerful hierarchical feature extraction capability of a Convolutional Neural Network (CNN), and one main reason is to train a deep Neural Network by using a large-scale labeled data set, such as a computer vision task like handwriting recognition. However, there is still a great space for improvement in some aspects such as the classification problem in the specific field of medical images, and one of the biggest challenges in this is that a large-scale medical image data set cannot be obtained, and a deep neural network trained using a small-scale data set cannot learn some features in the medical images, so that the classification effect is poor.

One of the biggest challenges facing the field of medical imaging is how to handle small-scale datasets and a limited number of labeled samples, since in some supervised learning algorithms, usually labeled data and a larger training set are used to train neural networks. However, in the medical field, a large-scale medical image data set is difficult to obtain, and researchers usually use some classical image enhancement methods to expand the medical image data set, so that the disadvantages caused by a small-scale data set are overcome. However, the traditional image expansion method mainly comprises translation, rotation, inversion and scaling, which are only simple modifications to the data set image, and the training process of improving the network by using the classical image enhancement technology is a standard process. Little additional information is obtained by modifying the image in a small scale using classical means. The method does not greatly contribute to the training effect of the neural network.

At the same time, some radiologists with expertise in data and image tasks are usually required to perform annotations in medical imaging tasks, but most image annotations of medical images are time consuming, especially for some precise annotations, such as segmentation of an organ or a diseased region into multiple 2D slices or 3D volume views. Although some of the presently disclosed data sets are available on the web, most data sets are limited in their size and are only suitable for certain medical problems, such as only certain organs or certain diseases, and collecting a medical data set is a complex and expensive process requiring multi-faceted collaboration.

In summary, a medical image synthesis and classification technology based on a Conditional Multi-discriminant generation countermeasure Network (CMDGAN for short) is proposed, two networks are used for training in a countermeasure process, meanwhile, a plurality of discriminant networks with added conditions are used for comprehensively judging generated images, and further generation of high-quality images is guided, generated comprehensive data enhancement of high-quality examples is a novel and complex data enhancement type, a data set using the synthetic data type has more variability, and the synthesized data set can better improve a training process of a deep neural Network.

The invention content is as follows:

the method aims to solve the problem that the existing network model has poor medical image classification effect, such as incapability of obtaining a large number of medical images, no real label in the images and the like. The invention provides a medical image synthesis and classification method named CMDGAN, which is used for synthesizing a high-quality data set and training a classification model of a medical image. The invention mainly comprises the following steps: the method comprises the steps of automatically constructing the concept of medical image synthesis and classification, constructing a flow of accurate extraction of a medical CT image lesion area, using CMDGAN for the synthesis data enhancement of the medical lesion area image, constructing a flow of a medical image classification network, using depth characteristics for the process of real medical image classification, and preprocessing the flow of a data set.

A medical image synthesis and classification method for generating a confrontation network based on conditional multiple discrimination is characterized in that: at least comprises the following steps:

firstly, according to the labeling of a pathological change Region in a Computed Tomography (CT) image by a radiologist, carrying out image segmentation on a medical CT image, and extracting a pathological change Region of interest (ROIs);

step two, carrying out data preprocessing on the lesion ROIs extracted in the step one to balance image data among different lesions;

step three, designing a model architecture for generating a countermeasure Network (CMDGAN for short) based on Multi-condition judgment, and performing guide training on the model architecture by using the preprocessed image obtained in the step two to obtain a generated model;

step four, performing synthetic data enhancement on the extracted lesion ROIs by using the generation model obtained in the step three to obtain a high-quality lesion ROIs data set;

and step five, designing a multi-scale residual error Network (MRNet for short), and training the Network by using the synthesized lesion ROIs data set obtained in the step four to obtain a high-precision classification neural Network.

Has the advantages that:

compared with the prior art, the design scheme of the invention can achieve the following technical effects:

1. the countermeasure network is generated by adopting multi-condition judgment to synthesize the medical image, the synthesized ROIs lesion data set has more additional information compared with the original data set, the variability is more, the generated comprehensive data enhancement of the high-quality example is a novel and complex data enhancement type, and compared with the classification network trained by the traditional data enhancement method, the accuracy of the classification network trained by adopting the lesion data set synthesized by the method is higher;

2. the confrontation network is generated by using multiple judgment conditions to synthesize the medical images, so that a large number of labeled lesion ROIs can be obtained, and a radiologist with professional knowledge is not required to label the lesion ROIs, so that a large amount of time and extra expenses are saved in the preprocessing stage of the data set;

3. a medical image data set with a specific field can be acquired in a short time, the quality of the synthesized medical image is high, and the classification network trained by the data set has better performance on a test data set;

4. the method for synthesizing the medical image by adopting the multi-condition discrimination generation countermeasure network has excellent performance on one lesion and strong applicability on many other organs or lesions.

5. The MRNet trained by the synthetic data is used for classifying the medical images, and the classification effect is much better than that of other classification networks, the classification precision is high, and the sensitivity and the specificity are better.

Description of the drawings:

FIG. 1. conditional multiple-decision generation of a countermeasure network architecture diagram

FIG. 2 is a flow chart of conditional multi-decision generation of confrontation network composite image

FIG. 3 is a flow chart of DUTFCN extraction of lesion ROIs

FIG. 4 is a diagram of a multi-scale residual network architecture

FIG. 5 is a flowchart of the overall process of conditional multi-discriminant generation of confrontation network composite image and classification

The specific implementation mode is as follows:

firstly, according to the labeling of a pathological change Region in a Computed Tomography (CT) image by a radiologist, image segmentation is carried out on a medical CT image, and a pathological change Region of interest (ROIs) is extracted.

(1) Extracting a suspected lesion area: taking thyroid lesion as an example, a professional radiologist analyzes a thyroid CT image, marks the edge of each lesion and determines a corresponding diagnosis result, and simultaneously, the determination is performed through physical examination and clinical random access, and a suspected lesion area is finally obtained.

(2) Characteristic analysis: the gray value of the suspected lesion area is different from that of the normal tissue, so that the suspected lesion area needs to be subjected to feature analysis to better determine the lesion area, wherein the analyzed features mainly comprise an average gray value, a gray value standard deviation, a diameter and the like.

(3) Extracting final ROIs by utilizing the gray features: and accurately segmenting the image by adopting a Deep U-Type Full Convolutional Neural Network (DUTFCN for short), and extracting final ROIs. The idea of DUTFCN is to use a full convolution neural network to segment biomedical images accurately without any full connectivity layer, so that the loss of position information is reduced, and the input of the network can use pictures of any size. The sampling method is adopted to replace the pooling operation to increase the resolution of the output. The network consists of a contraction sub-network and an expansion sub-network, wherein the contraction sub-network is used for capturing context characteristics, the expansion sub-network is used for positioning, the contraction sub-network and the expansion sub-network are in quick connection in time, the bottom layer characteristics and the high layer characteristics are fused, loss information in pooling is supplemented, a better segmentation effect is obtained, meanwhile, a loss function uses softmax and a cross loss entropy function, and the softmax function formula is as follows:

wherein, a_k(x) And expressing the activation function of the Kth characteristic channel on the Xth pixel point. K is the number of classes, p_k(x) Is approximately the mostA large function (approximated maximum function). While the weighted cross entropy is defined as:

where l ═ {1,2, …, K } denotes the correct label for each pixel, and the weighting function W is defined as:

wherein, w_cWeight graph with function representing the frequency of the equilibrium class, d₁Indicates the distance to the nearest ROI boundary, d₂Indicating the distance to the second near ROI boundary. The model does not need a large amount of training data and can be well converged.

And step two, carrying out data preprocessing on the lesion ROIs extracted in the step one to balance the image data among different lesions.

(1) Sample equalization: and (3) carrying out sample equalization on the accurate thyroid lesion ROIs divided by the U-FCN full convolution network in the step one, and adjusting the ratio of the images of the four types of thyroid lesion ROIs to be 1:1:1: 1.

(2) Generating a data set to be synthesized: and (4) carrying out sample equalization on the thyroid lesion ROIs to obtain a presynthesized data set sample.

(3) Generating a classification test data set: after sample equalization, a classification test data set is obtained for verifying the accuracy of the classification effect.

And step three, designing a model architecture for generating a countermeasure Network (CMDGAN for short) based on Multi-condition judgment, and performing guide training on the model architecture by using the preprocessed image obtained in the step two to obtain a generated model.

(1) In the CMDGAN, there are two generator inputs, respectively a class label C and noise Z corresponding to each generated sample, for generating a false sample.

(2) Real sampleCategories as true data tags X_real(data) and false sample X generated by the generator_fakeAnd simultaneously, each discrimination network has two parts of output of discrimination picture true and false and type corresponding to true.

(3) The outputs of multiple discriminant networks are simultaneously input into an averaging function, the output of the function is the final real result, and the function is as follows:

(4) and (3) training a conditional multi-discriminant generation countermeasure network (CMDGAN) by using the thyroid lesion ROIs image data to obtain a synthesized thyroid image lesion data set, wherein the thyroid image lesion data set mainly comprises four diseases, namely goiter, hyperthyroidism, thyroiditis and thyroid tumor. The training process is a process of mutually 'gaming' the convolutional neural networks of the generation network and the discrimination network, the object of the generation network G is to generate a real picture to deceive the discrimination network D as much as possible, the object of the D is to separate the picture generated by the G from the real picture as much as possible, and finally, the result of the game is that the generation network G can generate a picture G (z) which is 'false and true' under the most ideal condition. Since it is finally difficult to determine whether the generated picture is real or not in the discrimination network D, D (G (z)) is 0.5, and one generation formula model G is finally obtained, and a composite image of different types of lesions is generated using the model G.

Defining 1, generating a network: g is a network of generated pictures whose input is a random noise z from which the pictures are generated, denoted G (z).

Definition 2, network discrimination: and D is a judging network used for judging the truth of a picture. The input parameter is X, X represents a picture, and the output D (X) is used for representing the probability that X is a real picture, if the probability is 1, 100 percent of the picture is real, and the output is 0, the picture cannot be real.

(5) Training two networks by gradient descent method

for training iteration number do:

for k steps do:

from the prior distribution of noise p_gM noise samples { z) are sampled in (z)⁽¹⁾,…,z^(m)}；

In generating a sample distribution p_data(x) To select m samples { x⁽¹⁾,…,x^(m)}；

A fixed generator G for training a discrimination network D to discriminate a real sample and generate a sample as accurately as possible;

continuously updating the parameters of the discrimination network using gradient descent:

end for

a prior noise p_g(z) m noise samples { z) are selected⁽¹⁾,…,z^(m)}：

The parameters of the generating network are continuously updated using gradient descent:

end for

gradient-based updates can use any standard gradient learning rule, using momentum in experiments to avoid falling into local optima.

(6) Optimizing a minimum loss function

The objective function for each discriminating network has two parts, respectively the log-likelihood L of the correct source_sAnd the log-likelihood of the correct class L_cThe corresponding formula is as follows:

L_s＝E[logP(S＝real|X_real)]+E[logP(S＝fake|X_fake)] (7)

L_c＝E[logP(C＝c|X_real)]+E[logP(C＝c|X_fake)] (8)

the loss function is as follows:

wherein p is_data(x) Representing the true sample distribution, p_z(z) represents a random single distribution value. By continuously optimizing the minimum loss function, a generative model can be finally obtained.

And step four, performing synthetic data enhancement on the extracted lesion ROIs by using the generation model obtained in the step three to obtain a high-quality lesion ROIs data set.

(1) Random points are plotted in the underlying space, i.e., a random noise is generated.

(2) The noise is input to the above-described generation model, and the distribution of the random noise is learned to generate an image.

(3) The generated image is mixed with the actual image.

(4) The mixed image is input to discrimination for discrimination, and the corresponding target is used.

(5) Finally, the judging network cannot identify the authenticity of the images of the birthday, and then a large number of high-quality medical images can be synthesized.

And step five, designing a multi-scale residual error Network (MRNet for short), and training the Network by using the synthesized lesion ROIs data set obtained in the step four to obtain a high-precision classification neural Network.

(1) The multi-scale residual error network is formed by using various convolution modes and a large number of residual error blocks, the network depth is increased, the parameter number of the network is reduced as far as possible, and meanwhile, the loss of a large amount of information caused by dimension reduction operation of a 1 x 1 convolution kernel is prevented.

Definition 1, residual block: assuming a two-layer neural network, activation at layer l yields a^[l+1]Activating again to obtain a^[l+2]The calculation process is from a^[l]At the beginning, the fast residual will be a^[l]Is put in linear activation z^[l+2]Then, before nonlinear activation, a is finally obtained^[l+2]。

(2) According to the characteristics of the synthesized thyroid lesion ROIs, the network is a classification network containing more residual blocks, and a dropout layer is added to prevent overfitting, so that the problems of gradient disappearance and gradient explosion in the training process can be effectively prevented, and the training can be more quickly and stably converged.

(3) And training the classification network according to the synthesized medical image obtained in the step four, testing on the test set, and obtaining a more accurate classification result and higher specificity and sensitivity of the image through the classification network trained by the synthesized medical image.

Claims (1)

1. A medical image synthesis and classification method based on conditional multi-discriminant generation countermeasure network is characterized in that: at least comprises the following steps:

firstly, a radiologist obtains a suspected lesion area on a Computed Tomography (CT) image by combining biopsy and clinic, then performs characteristic analysis, distinguishes the suspected lesion from normal tissues by the image characteristics of average gray value, standard deviation of gray value and diameter, designs a deep U-shaped full convolution neural network DUTFCN, the DUTFCN does not have any full connecting layer, reduces the loss of position information, can use pictures with any size for network input, adopts an upper sampling method to replace pooling operation to increase the output resolution, the network consists of a contraction sub-network and an expansion sub-network, the contraction sub-network is used for capturing context characteristics, the expansion sub-network is used for positioning, the two are in quick connection, the bottom layer characteristics and the high layer characteristics are fused, the lost information in pooling is supplemented, the characteristics are input into the DUTFCN to accurately segment the CT image, extracting a Region of interest (Region of interest, ROIs for short) of a lesion;

step two, adjusting the number of the ROIs samples extracted in the step one to ensure that the number of the samples in each category is equal, so that the image data among different focuses is balanced;

designing a CMDGAN model architecture based on condition Multi-discrimination generation countermeasure Network (CMDGAN for short), wherein the CMDGAN has two types of input of labels and noise, real sample types are used as real data labels and false samples generated by a generator and are simultaneously input into a plurality of discrimination networks to be respectively discriminated, a Network target function is discriminated to have two parts of log-likelihood of a correct source and log-likelihood of a correct type, a thyroid lesion image data set is adopted to train the CMDGAN, the data set comprises four diseases of goiter, hyperthyroidism, thyroiditis and thyroid tumor, a gradient descent algorithm aiming at the CMDGAN is designed to synchronously train and generate and Discriminate two networks, momentum is used for gradient updating to avoid falling into local optimality, and a generation model is finally obtained;

step four, performing synthetic data enhancement on the extracted lesion ROIs by using the generation model obtained in the step three to obtain a high-quality lesion ROIs data set;

and step five, designing a multi-scale residual error Network (MRNet for short), training the Network by using the synthesized lesion ROIs data set obtained in the step four to obtain a high-precision classification neural Network, adding a dropout layer to prevent overfitting, and finally realizing medical image classification.

Images