WO2022100495A1

WO2022100495A1 - Method for automatically segmenting ground-glass pulmonary nodule and computer device

Info

Publication number: WO2022100495A1
Application number: PCT/CN2021/128438
Authority: WO
Inventors: 黄钢; 聂生东; 董婷
Original assignee: 上海健康医学院; 上海理工大学
Priority date: 2020-11-11
Filing date: 2021-11-03
Publication date: 2022-05-19
Also published as: CN112465830B; CN112465830A

Abstract

The present invention relates to a method for automatically segmenting a ground-glass pulmonary nodule and a computer device. Said method comprises the following steps: acquiring medical image raw data acquired by a computer tomography device; pre-processing the medical image raw data; and taking the pre-processed image as an input of a trained fully convolutional residual network based on an ASPP structure and an attention mechanism, to obtain a ground-glass pulmonary nodule segmentation result, wherein the fully convolutional residual network based on the ASPP structure and the attention mechanism takes a plurality of Conv2D convolutional layers as a basic architecture, a residual module and an attention module being provided between adjacent Conv2D convolutional layers, and an ASPP structure is provided in the fully convolutional residual network to capture multi-scale information of the ground-glass pulmonary nodule. Compared with the prior art, the present invention has the advantages of rapidness, accuracy, etc.

Description

A method and computer equipment for automatic segmentation of ground-glass pulmonary nodules

technical field

The invention relates to the technical field of deep learning and computed tomography image processing, in particular to a ground-glass-like lung nodule automatic segmentation method and computer equipment based on a fully convolutional residual network.

Background technique

Lung cancer is the leading cause of cancer-related deaths worldwide. Today, computed tomography (CT) has become the technology of choice for detection and monitoring of early-stage lung cancer due to its fast imaging speed and high image resolution. The imaging manifestation of early lung cancer is the formation of pulmonary nodules, and accurate and rapid segmentation of pulmonary nodules is an indispensable preprocessing in the classification of benign and malignant. Ground-glass pulmonary nodules are a special type of nodules. Compared with solid nodules, ground-glass pulmonary nodules are characterized by blurred borders, irregular shapes, uneven intensity, and low contrast with surrounding normal tissues. ignored by doctors. Therefore, the segmentation and diagnosis of ground-glass pulmonary nodules has always been the focus and difficulty in the field of medical image segmentation. Accurate segmentation of ground-glass pulmonary nodules can provide an important basis for medical imaging evaluation and treatment plan formulation, and is of great significance to improve the monitoring efficiency of early lung cancer.

Most ground-glass lung nodule segmentation methods use unsupervised methods. Traditional unsupervised methods can be divided into clustering, deformable models, segmentation methods based on random walk (Random Walk, RW) and Markov random field (MRF) theory, such as the literature published in the Journal of Automation. "Ground-glass lung nodule segmentation based on sparse representation and random walks". Although these methods have advantages in segmenting ground-glass pulmonary nodules in complex backgrounds, they are also highly dependent on manual intervention. For example, clustering and segmentation methods based on RW theory overly rely on the selection of seed points; deformable models rely on The position of the initial contour is sensitive to noise; the segmentation method based on MRF theory easily consumes unnecessary computation.

SUMMARY OF THE INVENTION

The purpose of the present invention is to provide a fast and accurate automatic segmentation method and computer equipment for ground glass-like pulmonary nodules based on a fully convolutional residual network in order to overcome the above-mentioned defects in the prior art.

The object of the present invention can be realized through the following technical solutions:

An automatic segmentation method for ground-glass pulmonary nodules, the method includes the following steps:

Obtain the raw data of medical images collected by computed tomography equipment;

preprocessing the medical image raw data;

The preprocessed image is used as the input of the trained fully convolutional residual network based on ASPP structure and attention mechanism to obtain the segmentation results of ground glass lung nodules;

The fully convolutional residual network based on the ASPP structure and attention mechanism is based on multiple Conv2D convolutional layers, and a residual module and an attention module are set between adjacent Conv2D convolutional layers. The ASPP structure is set up in the residual network to capture the multi-scale information of ground glass-like lung nodules.

Further, the training process of the fully convolutional residual network based on the ASPP structure and attention mechanism includes:

Obtain the raw data of lung medical images marked with ground glass pulmonary nodules, and form training samples after preprocessing;

Select the initial network parameters, construct a fully convolutional residual network based on the ASPP structure and attention mechanism, and determine the class probability of each lung medical image in the training sample based on the fully convolutional residual network;

Calculate the error between each lung medical image and the corresponding label in the training sample based on a preset loss function;

based on the error, updating the network parameters of the fully convolutional residual network;

After the error meets a preset condition, a trained fully convolutional residual network is obtained.

Further, the preprocessing is specifically:

The CT value of the original data is adjusted to a set range, and saved as a grayscale image, a region of interest is extracted from the grayscale image, and one-hot encoding is performed on the region of interest.

Further, the preprocessing is specifically:

The CT value of the original data is adjusted to a set range, and saved as a grayscale image, a region of interest is extracted from the grayscale image and the corresponding label data, and one-hot encoding is performed on the region of interest.

Further, the setting range is [-1000, 400] Hu.

Further, the fully convolutional residual network based on the ASPP structure and attention mechanism also includes a long skip connection layer for fusing low-level features and high-level features.

Further, the fully convolutional residual network based on the ASPP structure and the attention mechanism is divided into a low-level sub-network and a high-level sub-network. Set up with a ConvTranspose2D convolutional layer.

Further, the ASPP structure is arranged between the lower-level sub-network and the higher-level sub-network.

Further, among the multiple Conv2D convolutional layers, the output layer adopts the Sigmoid activation function, and the remaining layers adopt the Relu activation function.

The present invention also provides a computer device, comprising:

one or more processors;

memory; and

One or more programs stored in memory, the one or more programs including instructions for performing the method of automatic segmentation of ground glass pulmonary nodules as described above.

The traditional pulmonary nodule segmentation method needs to segment the lung parenchyma first, then extract the region of interest, and then design a targeted algorithm to complete the segmentation. The contrast is low, and some nodules are adhered to structures such as blood vessels and pleura. Traditional methods are difficult to effectively segment such pulmonary nodules.

Compared with the prior art, the present invention has the following beneficial effects:

1. The present invention adopts the trained full convolutional residual network based on the ASPP structure and the attention mechanism to directly perform feature extraction on the original data information, and directly segment the target according to the obtained features, which has high efficiency and does not rely on manual intervention.

2. The fully convolutional residual network constructed by the present invention has an ASPP structure and considers the attention mechanism, wherein the ASPP structure can capture the multi-scale information of ground glass-like lung nodules and effectively extract multi-scale information from high-level feature maps. The receptive field feature improves the model's ability to deal with nodules of different sizes; the attention mechanism propagates the spatial information in the encoding layer to the decoding layer, and at the same time reduces the loss of information in the forward propagation process, effectively reducing the nodule features in the transmission process. The loss of information; adding the batch normalization layer to the residual structure accelerates the network training speed, and at the same time avoids the gradient disappearance and performance degradation problems caused by the deepening of the network, and effectively improves the accuracy of the final image segmentation.

3. The present invention is also provided with a long jump connection layer for effectively fusing low-level features and high-level features, so as to minimize the loss of effective information of the entire network.

4. The present invention further sets the MaxPooling2D pooling layer and the ConvTranspose2D convolutional layer in the fully convolutional residual network, wherein the MaxPooling2D pooling layer compresses the features extracted by the convolution operation, on the one hand, it makes the features smaller and simplifies the network calculation. On the other hand, feature compression is performed to extract main features; the ConvTranspose2D convolution layer restores the feature map to the original resolution size, which is convenient for end-to-end segmentation prediction.

5. The present invention can segment various types of ground glass pulmonary nodules, and has good universality.

6. The present invention constructs a fully convolutional residual network based on the ASPP structure and attention mechanism, and adopts the batch normalization layer added to the residual module, which speeds up the network training speed and avoids the gradient disappearance and performance degradation caused by the deepening of the network. question.

7. The method of the present invention has the advantages of high calculation accuracy, fast time, good robustness, etc., and can obtain reliable and stable results.

Description of drawings

Fig. 1 is the flow chart of the method of the present invention;

Figure 2 is a framework diagram of a fully convolutional residual network based on the ASPP structure and attention mechanism;

Figure 3 is an ASPP structural diagram;

Figure 4 is the structure diagram of the attention mechanism;

Figure 5 is a residual structure diagram;

Figure 6 shows the results of the fully convolutional residual network based on the ASPP structure and attention mechanism, in which Figures (a) and (b) represent the comparison of the loss rate of the training set and the validation set and the overlap rate of the training set and the validation set, respectively. Compared.

Detailed ways

The present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. This embodiment is implemented on the premise of the technical solution of the present invention, and provides a detailed implementation manner and a specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

Example 1

The invention provides an automatic segmentation method for ground glass-like pulmonary nodules, which is an automatic medical image processing method. The method includes the following steps: obtaining medical image raw data collected by a computer tomography device; Preprocessing; take the preprocessed image as the input of the trained full convolutional residual network (ResAANet) based on ASPP (Atrous Spatial Pyramid Pooling) structure and attention mechanism to obtain ground glass Pulmonary nodule segmentation results. In this method, the fully convolutional residual network based on the ASPP structure and attention mechanism is based on multiple Conv2D convolutional layers, and a residual module and an attention module are set between adjacent Conv2D convolutional layers. The ASPP structure is set in the convolutional residual network to capture the multi-scale information of ground-glass lung nodules, and the original data can be directly extracted effectively to obtain segmentation results quickly and accurately. The segmentation process does not depend on manual intervention. The supervised method based on deep learning mainly trains nodules and label images through neural network, automatically extracts relevant features of nodules, and automatically completes segmentation.

As shown in Figure 1, the training process of the fully convolutional residual network based on the ASPP structure and attention mechanism includes:

Step 1, acquiring raw data of lung medical images of ground-glass-like pulmonary nodules collected by a computer tomography device.

In step 2, the labeling data corresponding to the original data is obtained, and the labeling is the nodule type information manually labelled by the radiologist.

In step 3, preprocessing is performed to form training samples, and all training samples are divided into three parts: training set, validation set and test set to verify the performance of the trained network.

Step 4, select the initial network parameters, construct a fully convolutional residual network based on the ASPP structure and attention mechanism, and determine the class probability of each lung medical image in the training sample based on the fully convolutional residual network; The set loss function calculates the error between each lung medical image and the corresponding label in the training sample; based on the error, the network parameters of the fully convolutional residual network are updated.

Step 5: After the error meets the preset condition, a trained fully convolutional residual network is obtained.

By training the known ground-glass-like lung nodule data and labeled data, the obtained network is used for mask prediction of the ground-glass-like lung nodule data in the test set, and the obtained segmentation mask is compared with the known labeled data. , to judge the reliability and stability of the constructed fully convolutional residual network.

The preprocessing in the training process needs to be performed on the original data and the labeled data at the same time, specifically: adjusting the CT value of the original data to a set range, saving it as a grayscale image, and extracting lung nodules from the grayscale image. The region of interest (region of interest ROI) is the center, and one-hot encoding is performed on the region of interest. In this embodiment, the setting range is [-1000, 400] Hu, the grayscale image is 8 bits, and the size of the region of interest is 256×256.

In a preferred embodiment, the fully convolutional residual network based on the ASPP structure and attention mechanism further includes a long skip connection layer for fusing low-level features and high-level features.

In another preferred embodiment, the fully convolutional residual network based on the ASPP structure and attention mechanism is divided into a low-level sub-network (encoding) and a high-level sub-network (decoding), and MaxPooling2D pools are set at intervals in the low-level sub-network The high-level sub-network is provided with a ConvTranspose2D convolutional layer at intervals.

In another preferred embodiment, among the multiple Conv2D convolutional layers, the output layer adopts the Sigmoid activation function, and the remaining layers adopt the Relu activation function.

The fully convolutional residual network based on ASPP structure and attention mechanism constructed in this embodiment includes Conv2D convolution layer, MaxPooling2D pooling layer, ConvTranspose2D convolution layer, ASPP structure, attention module, residual module and long skip connection layer , among which, the Conv2D convolution layer is used to complete the extraction of ground glass-like lung nodule features; the MaxPooling2D pooling layer compresses the features extracted by the convolution operation, on the one hand, the features become smaller and the network computational complexity is simplified, on the other hand Feature compression is performed to extract the main features; the ConvTranspose2D convolutional layer restores the feature map to the original resolution to complete the end-to-end segmentation prediction; the ASPP structure is used to capture the multi-scale information of ground-glass lung nodules; the attention mechanism will encode The spatial information in the layer is propagated to the decoding layer, and at the same time, the loss of information in the forward propagation process is reduced; the residual module deepens the network depth, realizes information cross-channel fusion, and avoids the network gradient caused by the residual structure deepening the network depth. The problem of disappearance and performance degradation; the long skip connection layer effectively fuses low-level features and high-level features to minimize the loss of effective information in the entire network.

The fully convolutional residual network based on the ASPP structure and attention mechanism has the ability of automatic learning. First, the feature extraction is performed on the input data through the Conv2D convolution layer. At the same time, in order to reduce the number of features of the convolution layer, the operation parameters are reduced. To speed up the calculation, apply the MaxPooling2D pooling layer to compress the features, extract the main features, and then use the ConvTranspose2D convolutional layer to restore the feature map to the original resolution size, and finally use the Sigmoid activation function to complete the probability prediction of pixel samples, and take 0.5 The threshold for generating prediction masks.

The structure of the fully convolutional residual network in this embodiment is shown in FIG. 2 , specifically:

(1) Conv2D convolution layer: The size of the convolution kernel of Conv1~Conv8 is (3,3), and the number of convolution kernels is shown in Table 1; the size of the convolution kernel of Conv9 is (1,1), the convolution kernel The number of kernels is 1; each convolution step size is (1,1), and each convolution layer contains a batch normalization layer and a Relu activation function;

Table 1 Information on the size and number of convolution kernels of the Conv2D convolutional layer

(2) MaxPooling2D pooling layer: pooling layers are applied on the 5th, 10th, 15th, and 20th layers, respectively, and the window size is set to (2,2);

(3) ConvTranspose2D convolution layer: Deconvolution layers are applied on the 22nd, 27th, 32nd, and 37th layers, respectively. The size of the convolution kernel is (3, 3), and the number of convolution kernels is 128, 64, and 32 respectively. , 16;

(4) ASPP structure: the atrous convolution spatial pyramid pooling structure is applied on the 21st layer, and the parallel atrous convolution sampling rates are set to 1, 6, 12, and 18 respectively;

(5) Attention mechanism: add an attention module after every two residual blocks;

(6) Residual module: Two residual modules with the same structure are added after each convolutional layer of Conv1 to Conv8;

(7) Long skip connection: fully connect the outputs of the 4th and 37th, 9th and 32nd, 14th and 27th, and 19th and 22nd layers;

(8) The ground glass-like lung nodule pixels were classified using the Sigmoid activation function, and a threshold of 0.5 was taken to generate a mask.

Simulation:

In the experiment, the preprocessed ground-glass lung nodule data and labeled data were first read, and then the pre-written program was input to the neural network for model training and verification. This experiment involved 794 ground-glass pulmonary nodules from 428 cases, of which 509 were used as training set, 56 as validation set, and 229 as test set. The similarity coefficient and overlap rate are used to evaluate the segmentation results.

The simulation results are shown in Figure 6 and Table 2.

Table 2 Dice similarity coefficient, overlap rate index information

By comparing the loss rate and the overlap rate of the training set and the validation set, it is found that the loss continues to decrease, the network is continuously optimized, and the final training and validation loss values are basically the same. , there is no over-fitting or under-fitting phenomenon, and the overlap rate reaches 71.98%. This result shows that the present invention is effective in the segmentation of ground-glass-like lung nodules.

It can be seen from the results in FIG. 6 and Table 2 that the method of the present invention can rapidly and accurately segment ground-glass pulmonary nodules.

Example 2

This embodiment provides a computer device, comprising one or more processors, a memory, and one or more programs stored in the memory, the one or more programs including a method for executing the ground glass described in Embodiment 1 Instructions for an automated segmentation method for pulmonary nodules.

The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make many modifications and changes according to the concept of the present invention without creative efforts. Therefore, all technical solutions that can be obtained by those skilled in the art through logical analysis, reasoning or limited experiments on the basis of the prior art according to the concept of the present invention shall fall within the protection scope determined by the claims.

Claims

An automatic segmentation method for ground-glass pulmonary nodules, characterized in that the method comprises the following steps:

Obtain the raw data of medical images collected by computed tomography equipment;

preprocessing the medical image raw data;

The preprocessed image is used as the input of the trained fully convolutional residual network based on ASPP structure and attention mechanism to obtain the segmentation results of ground glass lung nodules;

The fully convolutional residual network based on the ASPP structure and attention mechanism is based on multiple Conv2D convolutional layers, and a residual module and an attention module are set between adjacent Conv2D convolutional layers. The ASPP structure is set up in the residual network to capture the multi-scale information of ground glass-like lung nodules.
The automatic segmentation method for ground-glass-like pulmonary nodules according to claim 1, wherein the training process of the fully convolutional residual network based on the ASPP structure and the attention mechanism comprises:

Obtain the raw data of lung medical images marked with ground glass pulmonary nodules, and form training samples after preprocessing;

Select the initial network parameters, construct a fully convolutional residual network based on the ASPP structure and attention mechanism, and determine the class probability of each lung medical image in the training sample based on the fully convolutional residual network;

Calculate the error between each lung medical image and the corresponding label in the training sample based on a preset loss function;

based on the error, updating the network parameters of the fully convolutional residual network;

After the error meets a preset condition, a trained fully convolutional residual network is obtained.
The automatic segmentation method for ground glass-like pulmonary nodules according to claim 1, wherein the preprocessing is specifically:

The CT value of the original data is adjusted to a set range, and saved as a grayscale image, a region of interest is extracted from the grayscale image, and one-hot encoding is performed on the region of interest.
The automatic segmentation method for ground glass-like pulmonary nodules according to claim 2, wherein the preprocessing is specifically:

The CT value of the original data is adjusted to a set range, and saved as a grayscale image, a region of interest is extracted from the grayscale image and the corresponding label data, and one-hot encoding is performed on the region of interest.
The automatic segmentation method for ground glass-like pulmonary nodules according to claim 3 or 4, wherein the set range is [-1000, 400] Hu.
The automatic segmentation method for ground glass-like pulmonary nodules according to claim 1, wherein the fully convolutional residual network based on the ASPP structure and attention mechanism further comprises a long jump for fusing low-level features and high-level features connection layer.
The automatic segmentation method for ground glass-like pulmonary nodules according to claim 1, wherein the fully convolutional residual network based on the ASPP structure and attention mechanism is divided into a low-level sub-network and a high-level sub-network, and the low-level sub-network MaxPooling2D pooling layers are set at intervals in the sub-network, and ConvTranspose2D convolution layers are set at intervals in the high-level sub-network.
The automatic segmentation method for ground-glass-like pulmonary nodules according to claim 7, wherein the ASPP structure is arranged between a low-level sub-network and a high-level sub-network.
The method for automatic segmentation of ground-glass-like pulmonary nodules according to claim 1, wherein among the plurality of Conv2D convolutional layers, the output layer adopts a Sigmoid activation function, and the remaining layers adopt a Relu activation function.
A computer device, comprising:

one or more processors;

memory; and

One or more programs stored in memory, the one or more programs comprising instructions for performing the method for automatic segmentation of ground glass pulmonary nodules as claimed in any one of claims 1-9.