CN116645380A - Automatic segmentation method of tumor area in CT images of esophageal cancer based on two-stage progressive information fusion - Google Patents

Abstract

The invention relates to an automatic segmentation method for tumor areas of an esophageal cancer CT image based on two-stage progressive information fusion, which solves the defect that the automatic segmentation of the esophageal cancer CT image is difficult to realize compared with the prior art. The invention comprises the following steps: acquiring and preprocessing an esophageal cancer CT image; constructing an esophageal cancer CT image segmentation model; training of an esophageal cancer CT image segmentation model; obtaining and preprocessing a CT image of the esophageal cancer to be segmented; and obtaining the esophageal cancer CT image segmentation result. Based on the characteristics of large noise, low resolution and artifacts of the esophageal CT image, the invention proposes the characteristics extracted by using the image super-resolution reconstruction network, and then gradually blends the characteristics into the segmentation network, thereby effectively enhancing the quality of the esophageal CT image, enabling the network to extract more abundant detail characteristics, effectively carrying out segmentation and sketching of an esophageal cancer target region and improving the accuracy and efficiency of segmentation.

Description

Automatic segmentation of tumor area in CT images of esophageal cancer based on two-stage progressive information fusion method

technical field

The invention relates to the technical field of medical image segmentation, in particular to an automatic tumor area segmentation method for CT images of esophageal cancer based on two-stage progressive information fusion.

Background technique

Esophageal cancer is an aggressive male-dominated malignancy, including squamous cell esophageal carcinoma and adenoid esophageal carcinoma, which have different pathological features and distribution. Globally, squamous cell esophageal carcinoma remains the most common type. At present, the technique of esophagectomy and three-field lymphadenectomy has reached the limit of local control, and it is necessary to wait for the further development of the technique. In addition, esophageal cancer is very aggressive, and complications such as lymphatic and hematogenous metastasis may occur in the early stage. Since the early symptoms of the disease are not obvious, most patients have difficulty in eating, hoarseness and other symptoms when they go to the doctor, the tumor is already in the advanced stage, and the opportunity for surgical resection is missed. For patients with esophageal cancer who cannot receive surgical treatment, single methods such as chemotherapy and targeted therapy are not effective, and the 5-year survival rate is low. In contrast, multimodal comprehensive treatment methods such as chemotherapy, radiotherapy, and endoscopic therapy are gradually becoming mainstream, which can provide long-term survival opportunities for some patients with advanced esophageal cancer, and play an important role in cases where surgical resection is not suitable. A major problem with radiation therapy is determining the location of the tumor, which requires a tool that can aid in localization. Therefore, computed tomography technology is widely used in the process of radiotherapy planning.

Precise radiotherapy requires precise determination and delineation of the target volume for radiotherapy. At present, the delineation of the target area in radiotherapy is mainly done manually by experienced doctors and physicists, and the accuracy depends on the experience level of doctors. However, this work is tedious and time-consuming, and it may take an experienced doctor two days to complete the annotation of a set of images.

Therefore, how to automatically delineate the target area on medical images has become a hot issue in the field of computer vision. Due to the complexity of the esophageal organs, there is no mature and feasible solution for the delineation of radiation therapy target volume. In order to improve the work efficiency of doctors and realize the precise treatment of esophageal cancer, it is an urgent problem to study and realize the automatic delineation of tumor target volume of esophageal cancer.

The medical image segmentation of esophageal cancer based on deep learning is a breakthrough technology, which can well realize the automatic classification, identification and segmentation tasks of organs and tumor targets, and minimize the internal information of the image that is difficult for doctors to discover. In the diagnosis of esophageal cancer medical image segmentation, with the help of AI image assistance, surgeons can quickly and effectively detect cancer and save diagnosis time. Recent studies have also demonstrated the satisfactory robustness and potential of this technique.

However, automatic delineation of tumor or clinical targets in esophageal cancer is a challenging task. The segmentation of the tumor area depends on the contrast difference between the tumor and the surrounding tissue in the CT image. Due to the diversity of esophageal cancer lesions, the variability of the location and the complexity of the surrounding tissue, it is difficult for the traditional deep learning algorithm to capture all the details and details of the tumor. feature.

Therefore, how to automatically segment CT images of esophageal cancer with complex lesion image differences has become an urgent technical problem to be solved.

Contents of the invention

The purpose of the present invention is to solve the defect in the prior art that it is difficult to automatically segment CT images of esophageal cancer, and provide a method for automatic tumor area segmentation of CT images of esophageal cancer based on two-stage progressive information fusion to solve the above problems.

In order to achieve the above object, the technical scheme of the present invention is as follows:

A method for automatic segmentation of tumor regions in CT images of esophageal cancer based on two-stage progressive information fusion, comprising the following steps:

11) Acquisition and preprocessing of CT images of esophageal cancer: Acquire CT images in DICOM format, perform data enhancement processing on the CT image data of the cervical esophagus and abdominal esophagus in the CT images, and perform slice processing on all CT images, namely Intercept the 3D CT images in DICOM format to obtain 2D CT image slices in .jpg format and binarized label images in .png format to form an esophageal cancer CT image dataset;

12) Build a CT image segmentation model for esophageal cancer: build a CT image segmentation model for esophageal cancer based on two-stage progressive information fusion technology;

13) Training of the esophageal cancer CT image segmentation model: input the esophageal cancer CT image data set into the esophageal cancer CT image segmentation model for training;

14) Acquisition and preprocessing of CT images of esophageal cancer to be segmented;

15) Obtaining the segmentation result of esophageal cancer CT image: Input the preprocessed esophageal cancer CT image to be segmented into the trained esophageal cancer CT image segmentation model to obtain the segmented esophageal cancer CT image.

Described construction esophagus cancer CT image segmentation model comprises the following steps:

21) Set the CT image segmentation model of esophageal cancer to include the Swin Transformer network model and the TransResUNet convolutional neural network model for super-resolution reconstruction. The feature map output by the Swin Transformer for super-resolution reconstruction and the original image are progressively stitched together. Information fusion, and then input to the TransResUNet convolutional neural network model for segmentation to obtain the final segmentation map;

22) Set the Swin Transformer network model, which includes 6 Swin with residual

Transformer module RSTB and a residual connection structure, where RSTB consists of 6 Swin

Transformer Layer consists of a convolution and a residual connection;

23) Set the TransResUNet convolutional neural network model:

231) Set the TransResUNet convolutional neural network model to include a downsampling encoder module for feature extraction, a feature pyramid ASPP module for obtaining receptive fields of different scales, and an upsampling decoder module for restoring image resolution ;

232) Set the downsampling encoder module to include 4 continuous downsampling structures with residuals;

The downsampling structure with residuals includes branch A and branch B, where branch A is a convolutional layer with a convolution kernel of 3×3, a batch normalization layer, a LeakyRelu layer, a Transformer Encoder Block layer, and a volume The product kernel is a 3×3 convolutional layer, and a batch normalization layer is stacked as the branch A of the residual structure;

Branch B is a convolution layer with a convolution kernel of 1×1, and a batch normalization layer is stacked as another branch B of the residual structure;

The two branches perform addition operations, and finally pass through a LeakyRelu layer;

233) set feature pyramid ASPP module, it comprises:

The first operation module: a convolution layer with a convolution kernel of 1×1;

The second operation module: a convolutional layer with a convolution kernel of 3×3 and a hole rate of 6;

The third operation module: a convolutional layer with a convolution kernel of 3×3 and a hole rate of 12;

The fourth operation module: a convolutional layer with a convolution kernel of 3×3 and a hole rate of 18;

The fifth operation module: an adaptive average pooling layer, a convolution layer with a convolution kernel of 1×1, and an upsampling operation for stacking;

These five operation modules are connected in parallel, and the five feature maps obtained are spliced, and then passed through a convolution layer with a convolution kernel of 1×1;

235) Set an upsampling decoder module, including 4 continuous upsampling structures with residuals, and a splicing structure with output branches of four downsampling structures with residuals of the encoder;

The four downsampling structures with residuals of the encoder obtain four outputs of different sizes,

The output of the fourth dimension is input to the feature pyramid ASPP module and then spliced to the first input of the decoder,

The output of the third dimension is concatenated to the second input of the decoder,

The output of the second dimension is concatenated to the third input of the decoder,

The output of the first dimension is concatenated to the fourth input of the decoder;

The decoder is structured as four consecutive upsampling blocks with residuals,

The upsampling block structure with residuals is:

A double upsampling operation, a splicing operation corresponding to the encoder layer, a convolution layer with a convolution kernel of 3×3, a batch normalization layer, a LeakyRelu layer, and a convolution with a convolution kernel of 3×3 layer stacked with a batch normalization layer as a branch of the residual structure,

And a convolution layer with a convolution kernel of 1×1 stacked with a batch normalization layer as another branch of the residual structure;

The two branches perform an addition operation, and finally pass through a LeakyRelu layer in a decoder module.

The training of described esophagus cancer CT image segmentation model comprises the following steps:

31) Input the esophageal cancer CT image data set into the Swin Transformer network model of the esophageal cancer CT image segmentation model, and output the feature map from the Swin Transformer network model;

Input to the Swin Transformer network model for super-resolution reconstruction, perform a convolution layer with a convolution kernel size of 1×1, go through 6 consecutive RSTB modules, a residual connection operation, and a convolution kernel size of 1 ×1 convolutional layer, a LeakyRelu layer, an upsampling operation, and a convolutional layer with a convolution kernel size of 1×1 to obtain a feature map;

32) Perform progressive information fusion of the feature map output by the Swin Transformer and the original image through splicing operations to obtain the spliced feature map;

33) input the spliced feature map into the TransResUNet convolutional neural network model;

34) The spliced feature maps are trained in the downsampling encoder module:

341) Input the spliced feature map to the first downsampling structure with residuals, branch A of the first downsampling structure with residuals is a convolutional layer with a convolution kernel of 3×3, a batch A normalization layer, a LeakyRelu layer, a Transformer Encoder Block layer, a convolutional layer with a convolution kernel of 3×3, and a batch normalization layer are stacked; the first branch B of the downsampling structure with residuals is A convolution layer with a convolution kernel of 1×1 and a batch normalization layer are stacked;

The two branches perform addition operations, and finally execute a LeakyRelu layer to obtain the first downsampling output;

342) The first downsampling output is sent to the second downsampling structure with residuals, the branch A and branch B of the second downsampling structure with residuals are added, and finally a LeakyRelu layer is executed to obtain the first Two downsampling outputs;

343) The second downsampling output is sent to the third downsampling structure with residuals, the branch A and branch B of the third downsampling structure with residuals are added, and finally a LeakyRelu layer is executed to obtain the first Three downsampling outputs;

344) The third downsampling output is sent to the fourth downsampling structure with residuals, the branch A and branch B of the fourth downsampling structure with residuals are added, and finally a LeakyRelu layer is executed to obtain the first Four downsampling outputs;

35) The above four down-sampling outputs are input to the decoder module, and up-sampling is performed to restore the resolution of the image;

36) the fourth down-sampling output is input to the first input of the decoder after being input to the feature pyramid ASPP module;

37) For the first input of the decoder, perform a double upsampling operation, a concatenation operation corresponding to the encoder layer, a convolution layer with a convolution kernel of 3×3, a batch normalization layer, and a LeakyRelu layer , a convolution layer with a convolution kernel of 3×3 and a batch normalization layer are stacked as a branch of the residual structure;

And a convolution layer with a convolution kernel of 1×1 stacked with a batch normalization layer as another branch of the residual structure;

The two branches perform an addition operation, and finally pass through a LeakyRelu layer in a decoder module to obtain the second input of the decoder;

38) splicing the third downsampled output to the second input of the decoder,

For the second input of the decoder, perform a double upsampling operation, a concatenation operation corresponding to the encoder layer, a convolution layer with a convolution kernel of 3×3, a batch normalization layer, a LeakyRelu layer, a A convolution layer with a convolution kernel of 3×3 is stacked with a batch normalization layer as a branch of the residual structure;

And a convolution layer with a convolution kernel of 1×1 stacked with a batch normalization layer as another branch of the residual structure;

The two branches perform addition operations, and finally pass through the LeakyRelu layer in a decoder module to obtain the third input of the decoder;

39) splicing the second downsampled output to the third input of the decoder,

For the third input of the decoder, perform a double upsampling operation, a concatenation operation corresponding to the encoder layer, a convolution layer with a convolution kernel of 3×3, a batch normalization layer, a LeakyRelu layer, a A convolution layer with a convolution kernel of 3×3 is stacked with a batch normalization layer as a branch of the residual structure;

And a convolution layer with a convolution kernel of 1×1 stacked with a batch normalization layer as another branch of the residual structure;

The two branches perform addition operations, and finally pass through a LeakyRelu layer in a decoder module to obtain the fourth input of the decoder;

310) splicing the first downsampled output to the fourth input of the decoder,

For the fourth input of the decoder, perform a double upsampling operation, a concatenation operation corresponding to the encoder layer, a convolution layer with a convolution kernel of 3×3, a batch normalization layer, a LeakyRelu layer, a A convolution layer with a convolution kernel of 3×3 is stacked with a batch normalization layer as a branch of the residual structure;

And a convolution layer with a convolution kernel of 1×1 stacked with a batch normalization layer as another branch of the residual structure;

The two branches perform addition operations, and finally pass through a LeakyRelu layer in a decoder module, and then pass through a double upsampling and a convolutional layer with a convolution kernel of 1×1 to obtain the final output of TransResUNet;

311) Forward propagation to obtain the segmentation probability;

312) Using the cross-entropy loss function and the Dice loss function as the loss function of the esophageal cancer CT image segmentation model, the segmentation probability is calculated to obtain the segmentation loss, and its expression is as follows:

Among them, C in the cross-entropy loss function CE(p,q) represents the number of categories, p _i is the real value, and q _i is the predicted value; A and B in the DiceLoss formula represent the mask corresponding to the real label and the model prediction label respectively Matrix, |A∩B| is the intersection between A and B, and |A| and |B| represent the number of A and B elements, respectively, where the coefficient of the numerator is 2, because the denominator has repeated calculations of A and B The reason for the common elements between;

313) Use the L1 loss function as the loss function of the Swin Transformer network model to perform super-resolution reconstruction on CT images of esophageal cancer, and its expression is as follows:

Among them, N represents the number of samples, y _i is the true label of the i-th sample f(xi ₎ is the model prediction value of the i-th sample;

314) Determine the gradient vector by backpropagating the loss value, and update the parameters of the esophageal cancer CT image segmentation model;

315) Judging whether the set number of training rounds is reached, if so, complete the training of the esophageal cancer CT image segmentation model, otherwise continue the training.

Beneficial effect

Compared with the prior art, the automatic segmentation method of tumor area in esophageal cancer CT images based on two-stage progressive information fusion of the present invention is based on the characteristics of large noise, low resolution and artifacts in esophageal CT images, and proposes to use image super-resolution The features extracted by the high-rate reconstruction network are gradually fused into the segmentation network, which effectively enhances the quality of esophageal CT images, enables the network to extract more detailed features, and can effectively segment and outline esophageal cancer targets. Improve the accuracy and efficiency of segmentation.

Due to the variable location of esophageal tumors, the complex anatomical structure of the tumor, the blurred boundaries of the tumor target area, and the large individual differences, the improved TransformerResUNet proposed by the present invention can extract long-distance dependent features, thereby improving the segmentation accuracy of the tumor target area and improving the robustness of the model. sex. The invention adds an image super-resolution reconstruction branch, a Transformer Encoder Block module with long-distance dependent feature extraction, and an ASPP module with multi-scale feature fusion, which enhances the feature extraction capability of the network.

Description of drawings

Fig. 1 is a method sequence diagram of the present invention;

Fig. 2 is the structural diagram of the esophageal cancer CT image segmentation model involved in the present invention;

Fig. 3 is a structural diagram of the TransResUNet convolutional neural network model involved in the present invention;

Fig. 4 is the feature pyramid ASPP module structural diagram involved in the present invention;

Fig. 5 is the Swin Transformer network model structural diagram involved in the present invention;

Figure 6a and Figure 7a are CT images of esophageal cancer in the prior art;

Figure 6b and Figure 7b are the segmented label images of Figure 6a and Figure 7a respectively;

Fig. 6c and Fig. 7c are the automatically segmented images generated by the method of the present invention for Fig. 6a and Fig. 7a respectively;

Figure 6d and Figure 7d are the automatically segmented images generated by using the ResUNet network for Figure 6a and Figure 7a respectively;

Figure 6e and Figure 7e are the automatically segmented images generated by the UNet network for Figure 6a and Figure 7a respectively.

Detailed ways

In order to have a further understanding and understanding of the structural features of the present invention and the achieved effects, the preferred embodiments and accompanying drawings are used for a detailed description, as follows:

As shown in Fig. 1, a kind of method for automatic segmentation of tumor area in CT image of esophagus cancer based on two-stage progressive information fusion described in the present invention comprises the following steps:

The first step is the acquisition and preprocessing of CT images of esophageal cancer: obtain CT images in DICOM format, perform data enhancement processing on the CT image data of the cervical esophagus and abdominal esophagus in the CT images, and perform slice processing on all CT images , which is to intercept the 3D CT image in DICOM format to obtain 2D CT image slices in .jpg format and binarized label images in .png format to form an esophageal cancer CT image dataset.

The second step is to build a CT image segmentation model for esophageal cancer: to build a CT image segmentation model for esophageal cancer based on two-stage progressive information fusion technology.

CT images in DICOM format store rich medical imaging information of patients, but they are not suitable for training deep learning networks. Therefore, it is necessary to convert the data in DICOM format, and extract the original image and label information from it for model training and subsequent data analysis. Deep learning algorithms usually use RGB format as data input, so it is necessary to convert medical images in DICOM format to commonly used RGB format, and make a dataset that meets the requirements of deep learning. The conversion process includes two key steps: first, it is necessary to read the metadata of the original DICOM format file, analyze each slice of the patient individually, extract pixels for normalization, and set them between 0 and 1. In order to facilitate storage and subsequent deep learning analysis, it is mapped to between 0 and 255 and stored as image slice data with a size of 512×512. In addition, in the tumor segmentation task of deep learning, it is necessary to use the tumor region labels manually drawn by physicists. The reading of this process requires careful reading of the metadata and finding the outline corresponding to the label naming. Set its pixel value to 1 and the rest to 0.

Due to the lack of CT image data in the cervical and abdominal regions of the esophagus in the data set, data enhancement to expand the data set can enhance the robustness of the network; at the same time, because the esophageal tumor area is small and the shape is relatively irregular CT images are noisy, have low resolution, and have artifacts. Building a CT image segmentation model for esophageal cancer based on two-stage progressive information fusion technology can effectively optimize this problem and improve the effect of the model.

(1) As shown in Figure 2, the CT image segmentation model of esophageal cancer is set to include the SwinTransformer network model for super-resolution reconstruction and the TransResUNet convolutional neural network model. The feature map output by SwinTransformer for super-resolution reconstruction is the same as the original image The splicing operation performs progressive information fusion, and then input to the TransResUNet convolutional neural network model for segmentation to obtain the final segmentation map.

(2) As shown in Figure 5, set the Swin Transformer network model, which includes 6 SwinTransformer modules RSTB with residuals and a residual connection structure, where RSTB consists of 6 Swin Transformer Layers and a convolution, a residual Connection composition.

(3) Set the TransResUNet convolutional neural network model:

A1) As shown in Figure 2, the TransResUNet convolutional neural network model is set to include a downsampling encoder module for feature extraction, a feature pyramid ASPP module for obtaining receptive fields of different scales, and a module for restoring image resolution The upsampling decoder module;

A2) Set the downsampling encoder module to include 4 continuous downsampling structures with residuals;

The downsampling structure with residuals includes branch A and branch B, where branch A is a convolutional layer with a convolution kernel of 3×3, a batch normalization layer, a LeakyRelu layer, a Transformer Encoder Block layer, and a volume The product kernel is a 3×3 convolutional layer, and a batch normalization layer is stacked as the branch A of the residual structure;

Branch B is a convolution layer with a convolution kernel of 1×1, and a batch normalization layer is stacked as another branch B of the residual structure;

The two branches perform addition operations, and finally pass through a LeakyRelu layer;

A3) as shown in Figure 4, set feature pyramid ASPP module, it comprises:

The first operation module: a convolution layer with a convolution kernel of 1×1;

The second operation module: a convolutional layer with a convolution kernel of 3×3 and a hole rate of 6;

The third operation module: a convolutional layer with a convolution kernel of 3×3 and a hole rate of 12;

The fourth operation module: a convolutional layer with a convolution kernel of 3×3 and a hole rate of 18;

The fifth operation module: an adaptive average pooling layer, a convolution layer with a convolution kernel of 1×1, and an upsampling operation for stacking;

These five operation modules are connected in parallel, and the five feature maps obtained are spliced, and then passed through a convolution layer with a convolution kernel of 1×1;

A4) Set the upsampling decoder module, including 4 continuous upsampling structures with residuals, and the splicing structure with the output branches of the four downsampling structures with residuals of the encoder;

The four downsampling structures with residuals of the encoder obtain four outputs of different sizes,

The output of the fourth dimension is input to the feature pyramid ASPP module and then spliced to the first input of the decoder,

The output of the third dimension is concatenated to the second input of the decoder,

The output of the second dimension is concatenated to the third input of the decoder,

The output of the first dimension is concatenated to the fourth input of the decoder;

The decoder is structured as four consecutive upsampling blocks with residuals,

The upsampling block structure with residuals is:

A double upsampling operation, a splicing operation corresponding to the encoder layer, a convolution layer with a convolution kernel of 3×3, a batch normalization layer, a LeakyRelu layer, and a convolution with a convolution kernel of 3×3 layer stacked with a batch normalization layer as a branch of the residual structure,

And a convolution layer with a convolution kernel of 1×1 stacked with a batch normalization layer as another branch of the residual structure;

The two branches perform an addition operation, and finally pass through a LeakyRelu layer in a decoder module.

The third step is the training of the esophageal cancer CT image segmentation model: input the esophageal cancer CT image data set into the esophageal cancer CT image segmentation model for training.

During the training process, the Swin Transformer network model performs super-resolution reconstruction on the original image, obtains richer features and stitches them into the original image, enhances the boundary features of the original image, and then inputs it into the TransResUNet convolutional neural network model for image segmentation. This two-stage progressive information fusion framework can achieve better segmentation accuracy.

The training of the CT image segmentation model for esophageal cancer includes the following steps:

(1) Input the esophageal cancer CT image data set into the Swin Transformer network model of the esophageal cancer CT image segmentation model, and output the feature map from the Swin Transformer network model;

Input to the Swin Transformer network model for super-resolution reconstruction, perform a convolution layer with a convolution kernel size of 1×1, go through 6 consecutive RSTB modules, a residual connection operation, and a convolution kernel size of 1 ×1 convolutional layer, a LeakyRelu layer, an upsampling operation, and a convolutional layer with a convolution kernel size of 1×1 to obtain a feature map.

(2) The feature map output by the Swin Transformer and the original image are progressively fused through a splicing operation to obtain the spliced feature map.

(3) Input the spliced feature map into the TransResUNet convolutional neural network model.

(4) The spliced feature maps are trained in the downsampling encoder module:

B1) The spliced feature map is input to the first downsampling structure with residuals, branch A of the first downsampling structure with residuals is a convolutional layer with a convolution kernel of 3×3, a batch A normalization layer, a LeakyRelu layer, a Transformer Encoder Block layer, a convolutional layer with a convolution kernel of 3×3, and a batch normalization layer are stacked; the first branch B of the downsampling structure with residuals is A convolution layer with a convolution kernel of 1×1 and a batch normalization layer are stacked;

The two branches perform addition operations, and finally execute a LeakyRelu layer to obtain the first downsampling output;

B2) Send the output of the first downsampling to the second downsampling structure with residuals, and perform addition operations on branch A and branch B of the second downsampling structure with residuals, and finally execute a LeakyRelu layer to obtain the first Two downsampling outputs;

B3) The second downsampling output is sent to the third downsampling structure with residuals, the branch A and branch B of the third downsampling structure with residuals are added, and finally a LeakyRelu layer is executed to obtain the first Three downsampling outputs;

B4) The third downsampling output is sent to the fourth downsampling structure with residuals, the branch A and branch B of the fourth downsampling structure with residuals are added, and finally a LeakyRelu layer is executed to obtain the first Four downsampled outputs.

(5) The above four down-sampling outputs are input to the decoder module to perform up-sampling to recover the resolution of the image.

(6) The fourth downsampling output is input to the feature pyramid ASPP module and then spliced to the first input of the decoder.

(7) For the first input of the decoder, perform a double upsampling operation, a splicing operation corresponding to the encoder layer, a convolution layer with a convolution kernel of 3×3, a batch normalization layer, and a LeakyRelu Layer, a convolutional layer with a convolution kernel of 3×3 and a batch normalization layer are stacked as a branch of the residual structure;

And a convolution layer with a convolution kernel of 1×1 stacked with a batch normalization layer as another branch of the residual structure;

The two branches perform an addition operation, and finally pass through a LeakyRelu layer in a decoder module to obtain the second input of the decoder.

(8) Splicing the third downsampled output to the second input of the decoder,

For the second input of the decoder, perform a double upsampling operation, a concatenation operation corresponding to the encoder layer, a convolution layer with a convolution kernel of 3×3, a batch normalization layer, a LeakyRelu layer, a A convolution layer with a convolution kernel of 3×3 is stacked with a batch normalization layer as a branch of the residual structure;

And a convolution layer with a convolution kernel of 1×1 stacked with a batch normalization layer as another branch of the residual structure;

The two branches perform an addition operation, and finally pass through a LeakyRelu layer in a decoder module to obtain the third input of the decoder.

(9) Splicing the second downsampling output to the third input of the decoder,

For the third input of the decoder, perform a double upsampling operation, a concatenation operation corresponding to the encoder layer, a convolution layer with a convolution kernel of 3×3, a batch normalization layer, a LeakyRelu layer, a A convolution layer with a convolution kernel of 3×3 is stacked with a batch normalization layer as a branch of the residual structure;

And a convolution layer with a convolution kernel of 1×1 stacked with a batch normalization layer as another branch of the residual structure;

The two branches perform an addition operation, and finally pass through a LeakyRelu layer in a decoder module to obtain the fourth input of the decoder.

(10) Splicing the first downsampled output to the fourth input of the decoder,

For the fourth input of the decoder, perform a double upsampling operation, a concatenation operation corresponding to the encoder layer, a convolution layer with a convolution kernel of 3×3, a batch normalization layer, a LeakyRelu layer, a A convolution layer with a convolution kernel of 3×3 is stacked with a batch normalization layer as a branch of the residual structure;

And a convolution layer with a convolution kernel of 1×1 stacked with a batch normalization layer as another branch of the residual structure;

The two branches perform addition operations, and finally pass through a LeakyRelu layer in a decoder module, and then pass through a double upsampling and a convolutional layer with a convolution kernel of 1×1 to obtain the final output of TransResUNet.

(11) Forward propagation to obtain the segmentation probability;

(12) Using the cross-entropy loss function and the Dice loss function as the loss function of the esophageal cancer CT image segmentation model, the segmentation probability is calculated to obtain the segmentation loss, and its expression is as follows:

Among them, C in the cross-entropy loss function CE(p,q) represents the number of categories, p _i is the real value, and q _i is the predicted value; A and B in the DiceLoss formula represent the mask corresponding to the real label and the model prediction label respectively Matrix, |A∩B| is the intersection between A and B, and |A| and |B| represent the number of A and B elements, respectively, where the coefficient of the numerator is 2, because the denominator has repeated calculations of A and B The reason for the common elements between;

(13) Using the L1 loss function as the loss function of the Swin Transformer network model to perform super-resolution reconstruction on CT images of esophageal cancer, the expression is as follows:

Among them, N represents the number of samples, y _i is the true label of the i-th sample f(xi ₎ is the model prediction value of the i-th sample;

(14) Determine the gradient vector through loss value backpropagation, and update the parameters of the esophageal cancer CT image segmentation model;

(15) Determine whether the set number of training rounds is reached, if so, complete the training of the esophageal cancer CT image segmentation model, otherwise continue training.

The fourth step is to obtain and preprocess the CT image of esophageal cancer to be segmented.

The fifth step is to obtain the segmentation result of esophageal cancer CT image: input the preprocessed esophageal cancer CT image to be segmented into the trained esophageal cancer CT image segmentation model to obtain the segmented esophageal cancer CT image.

As shown in Fig. 6a and Fig. 7a, they are CT slice images of two patients with esophageal cancer, and Fig. 6b and Fig. 7b are labels corresponding to the CT slice images of two patients with esophageal cancer. It can be seen from Fig. 6c and Fig. 7c that, compared with the ResUNet network model described in Fig. 6d and Fig. 7d, and the UNet network model described in Fig. 6e and Fig. 7e, the method of the present invention has more complete automatic segmentation boundary information, which is different from Labels have good consistency.

DSC represents the Deiss similarity coefficient, and its value is between [0,1]. The larger the value, the higher the accuracy, HD represents the Hausdorff distance, and the smaller the value, the higher the coincidence of the boundary. In order to conduct unbiased experiments, all experiments are performed with the same training initial parameters. As can be seen in Table 1, compared with the classic UNet, DSC and HD indicators, the method of the present invention has increased by 0.19 and 7.88, respectively, compared with ResUNet, DSC and HD indicators, respectively increased by 0.09 and 7.88.

Table 1 Comparison table of segmentation accuracy of the method described in the present invention compared with the classic U-Net network and ResUNet on DSC and HD indicators

The basic principles, main features and advantages of the present invention have been shown and described above. Those skilled in the art should understand that the present invention is not limited by the above-mentioned embodiments. What are described in the above-mentioned embodiments and the description are only the principles of the present invention. Variations and improvements, which fall within the scope of the claimed invention. The scope of protection required by the present invention is defined by the appended claims and their equivalents.

Claims (3)

1. An automatic segmentation method for tumor areas of esophageal cancer CT images based on two-stage progressive information fusion is characterized by comprising the following steps:

11 Acquisition and pretreatment of esophageal cancer CT images: acquiring a CT image in a DICOM format, performing data enhancement processing on CT image data of an esophageal neck region and an esophageal abdominal region in the CT image, and performing slicing processing on all the CT images, namely performing interception operation on the three-dimensional CT image in the DICOM format to obtain a two-dimensional CT image slice in the jpg format and a binarization tag image in the png format to form an esophageal cancer CT image dataset;

12 Constructing an esophageal cancer CT image segmentation model: constructing an esophageal cancer CT image segmentation model based on a two-stage progressive information fusion technology;

13 Training of esophageal cancer CT image segmentation model): inputting the esophageal cancer CT image data set into an esophageal cancer CT image segmentation model for training;

14 Obtaining and preprocessing a CT image of the esophageal cancer to be segmented;

15 Acquisition of esophageal cancer CT image segmentation results: inputting the preprocessed esophageal cancer CT image to be segmented into a trained esophageal cancer CT image segmentation model to obtain a segmented esophageal cancer CT image.

2. The automatic segmentation method for the tumor area of the esophageal cancer CT image based on the two-stage progressive information fusion according to claim 1, wherein the construction of the esophageal cancer CT image segmentation model comprises the following steps:

21 Setting a esophageal cancer CT image segmentation model comprising a Swin Transformer network model and a TransResunet convolutional neural network model for super-resolution reconstruction, wherein the Swin is reconstructed by super-resolution

The feature map output by the transducer and the original map are subjected to progressive information fusion through splicing operation, and then input into a TransResunet convolutional neural network model for segmentation to obtain a final segmentation map;

22 A Swin transducer network model is set, which comprises 6 Swin transducer modules RSTB with residual errors and a residual error connection structure, wherein the RSTB consists of 6 Swin Transformer Layer and one convolution and one residual error connection;

23 Setting a TransResUNet convolutional neural network model:

231 Setting a TransResunet convolutional neural network model, wherein the TransResunet convolutional neural network model comprises a downsampling encoder module for feature extraction, a feature pyramid ASPP module for obtaining different scale receptive fields, and an upsampling decoder module for recovering image resolution;

232 Setting up a downsampling encoder module comprising 4 consecutive downsampling structures with residuals;

the downsampling structure with the residual comprises a branch A and a branch B, wherein the branch A is a convolution layer with a convolution kernel of 3 multiplied by 3, a batch normalization layer, a LeakyRelu layer, a Transformer Encoder Block layer, a convolution layer with a convolution kernel of 3 multiplied by 3, and a batch normalization layer stack which is used as the branch A of the residual structure;

the branch B is a convolution layer with a convolution kernel of 1 multiplied by 1, and a batch of normalization layers are stacked to be used as another branch B of a residual structure;

the two branches are added, and finally pass through a LeakyRelu layer;

233 A set feature pyramid ASPP module comprising:

a first operation module: a convolution kernel is a1 x 1 convolution layer;

a second operation module: a convolution layer with a3 x 3 cavity rate of 6;

and a third operation module: a convolution layer with a3 x 3 cavity rate of 12;

fourth operation module: a convolution layer with a3 x 3 void fraction of 18;

a fifth operation module: an adaptive average pooling layer, a convolution layer with a convolution kernel of 1×1, and an up-sampling operation;

the five operation modules are connected in parallel, the obtained 5 feature images are spliced, and the five feature images are subjected to convolution kernel to form a convolution layer of 1 multiplied by 1;

234 Setting up a upsampling decoder module comprising 4 consecutive upsampling structures with residuals and a splice structure with the output branches of the four downsampling structures with residuals of the encoder;

the four residual downsampling structures of the encoder result in four different sized outputs respectively,

the output of the fourth size, input to the feature pyramid ASPP module and spliced to the first input of the decoder,

the output of the third size is spliced to the second input of the decoder,

the output of the second size is spliced to a third input of the decoder,

the output of the first size is spliced to a fourth input of the decoder;

the decoder structure is four consecutive upsampled blocks with residuals,

the upsampled block structure with residual is:

a double up-sampling operation, a corresponding encoder layer splicing operation, a convolution layer with a convolution kernel of 3 x 3, a batch normalization layer, a LeakyRelu layer, a convolution layer with a convolution kernel of 3 x 3 stacked with a batch normalization layer as a branch of the residual structure,

and a convolution layer with a convolution kernel of 1×1 is stacked with a batch of normalization layers as another branch of the residual structure;

the two branches perform an addition operation and finally pass through the LeakyRelu layer in one decoder block.

3. The automatic segmentation method for the tumor area of the esophageal cancer CT image based on the two-stage progressive information fusion according to claim 1, wherein the training of the esophageal cancer CT image segmentation model comprises the following steps:

31 Inputting the esophageal cancer CT image data set into a Swin Transformer network model of the esophageal cancer CT image segmentation model, and outputting a feature map from the Swin Transformer network model;

inputting to a Swin transform network model for super-resolution reconstruction, executing a convolution layer with a convolution kernel size of 1×1, performing a residual error connection operation on the convolution layer with the convolution kernel size of 1×1, performing an upsampling operation on the convolution layer with the convolution kernel size of 1×1 through 6 continuous RSTB modules, and obtaining a feature map;

32 Performing progressive information fusion on the feature map output by the Swin Transformer and the original map through splicing operation to obtain a spliced feature map;

33 Inputting the spliced characteristic diagram into a TransResunet convolutional neural network model;

34 Training the spliced feature map in a downsampling encoder module:

341 Inputting the spliced feature map into a first downsampling structure with residual errors, wherein a branch A of the first downsampling structure with residual errors is a convolution layer with a convolution kernel of 3 multiplied by 3, a batch normalization layer, a LeakyRelu layer, a Transformer Encoder Block layer, a convolution layer with a convolution kernel of 3 multiplied by 3 and a batch normalization layer stack; the first branch B with the residual downsampling structure is a convolution layer with a convolution kernel of 1 multiplied by 1 and a batch normalization layer stack;

the two branches perform addition operation, and finally a LeakyRelu layer is executed to obtain a first downsampled output;

342 Feeding the first downsampled output into a second downsampled structure with residual errors, adding the branch A and the branch B of the second downsampled structure with residual errors, and finally executing a LeakyRelu layer to obtain a second downsampled output;

343 Feeding the second downsampled output into a third downsampled structure with residual errors, adding the branch A and the branch B of the third downsampled structure with residual errors, and finally executing a LeakyRelu layer to obtain a third downsampled output;

344 Feeding the third downsampled output into a fourth downsampled structure with residual errors, adding the branch A and the branch B of the fourth downsampled structure with residual errors, and finally executing a LeakyRelu layer to obtain a fourth downsampled output;

35 The four downsampled outputs are input to a decoder module to carry out upsampling to recover the resolution of the image;

36 A fourth downsampled output is input to the feature pyramid ASPP module and spliced to the first input of the decoder;

37 For a first input of the decoder, performing a double up-sampling operation, a corresponding encoder layer splicing operation, a convolution layer with a convolution kernel of 3 x 3, a batch normalization layer, a LeakyRelu layer, a convolution layer with a convolution kernel of 3 x 3 stacked with a batch normalization layer as a branch of the residual structure;

and a convolution layer with a convolution kernel of 1×1 is stacked with a batch of normalization layers as another branch of the residual structure;

the two branches perform addition operation, and finally pass through a LeakyRelu layer in one decoder module to obtain a second input of the decoder;

38 A third downsampled output is spliced to a second input of the decoder,

performing a double up-sampling operation, a corresponding encoder layer splicing operation, a convolution layer with a convolution kernel of 3 x 3, a batch normalization layer, a leak relu layer, a convolution layer with a convolution kernel of 3 x 3 stacked with a batch normalization layer as a branch of the residual structure for a second input to the decoder;

and a convolution layer with a convolution kernel of 1×1 is stacked with a batch of normalization layers as another branch of the residual structure;

the two branches perform addition operation, and finally pass through a LeakyRelu layer in one decoder module to obtain a third input of the decoder;

39 A second downsampled output is spliced to a third input of the decoder,

performing a double up-sampling operation, a corresponding encoder layer splicing operation, a convolution layer with a convolution kernel of 3 x 3, a batch normalization layer, a leak relu layer, a convolution layer with a convolution kernel of 3 x 3 stacked with a batch normalization layer as a branch of the residual structure for a third input to the decoder;

and a convolution layer with a convolution kernel of 1×1 is stacked with a batch of normalization layers as another branch of the residual structure;

the two branches perform addition operation, and finally pass through a LeakyRelu layer in a decoder module to obtain a fourth input of the decoder;

310 A) the first downsampled output is spliced to a fourth input of the decoder,

performing a double up-sampling operation, a corresponding encoder layer splicing operation, a convolution layer with a convolution kernel of 3 x 3, a batch normalization layer, a leak relu layer, a convolution layer with a convolution kernel of 3 x 3 stacked with a batch normalization layer as a branch of the residual structure for a fourth input to the decoder;

and a convolution layer with a convolution kernel of 1×1 is stacked with a batch of normalization layers as another branch of the residual structure;

the two branches are added, and finally, a final output of the TransResunet is obtained through a LeakyRelu layer in a decoder module, a convolution layer which is subjected to double up-sampling and a convolution kernel of 1 multiplied by 1;

311 Forward propagation to obtain segmentation probability;

312 Using the cross entropy loss function and the Dice loss function as the loss function of the esophageal cancer CT image segmentation model, calculating the segmentation probability to obtain segmentation loss, wherein the expression is as follows:

wherein C in the cross entropy loss function CE (p, q) represents the number of categories, p _i Is true value, q _i Is a predicted value; a and B in the Dice Loss formula respectively represent mask matrixes corresponding to the real label and the model prediction label, the A and B are intersections between A and B, the A and B respectively represent the number of the A and B elements, wherein the coefficient of the molecule is 2 because of repeated calculation of common elements between A and B in denominator;

313 Using the L1 loss function as the loss function of the Swin transducer network model to reconstruct the esophageal cancer CT image in super resolution, the expression is as follows:

wherein N represents the number of samples, y _i The real label f (x _i ) Is the model predictive value of the ith sample;

314 Determining gradient vectors through back propagation of loss values, and updating the parameters of the esophageal cancer CT image segmentation model;

315 Judging whether the set training round number is reached, if so, completing the training of the esophageal cancer CT image segmentation model, otherwise, continuing the training.