WO2022166800A1

WO2022166800A1 - Deep learning network-based automatic delineation method for mediastinal lymphatic drainage region

Info

Publication number: WO2022166800A1
Application number: PCT/CN2022/074510
Authority: WO
Inventors: 魏军; 卢旭玲; 刘守亮; 沈烁
Original assignee: 广州柏视医疗科技有限公司; 广州柏视数据科技有限公司
Priority date: 2021-02-02
Filing date: 2022-01-28
Publication date: 2022-08-11
Also published as: CN112950651A; CN112950651B

Abstract

Disclosed in the present invention is a deep learning network-based automatic delineation method for a mediastinal lymphatic drainage region, suitable for CT images. The automatic delineation method comprises the following steps: S1: collecting CT image data and a mediastinal lymphatic drainage region image manually marked by a doctor, and preprocessing the CT image data and the mediastinal lymphatic drainage region image manually marked by the doctor; S2: grouping the preprocessed CT image data to obtain a training set, a validation set, and a test set; S3: performing data augmentation on the raining set, the validation set, and the test set; S4: constructing a deep learning segmentation model; and S5: inputting the CT image data and the mediastinal lymphatic drainage region image manually marked by the doctor in the training set into the constructed deep learning segmentation model, after iterative convergence of training, saving the segmentation model of the mediastinal lymphatic drainage region, and then performing recognition and prediction on the mediastinal lymphatic drainage region to obtain a probability graph of each subregion of the mediastinal lymphatic drainage region. A network may locate and segment a small drainage region well.

Description

Automatic delineation method of mediastinal lymphatic drainage area based on deep learning network

technical field

The present invention relates to the field of medical images, in particular to an automatic delineation method for the mediastinal lymphatic drainage area based on a deep learning network.

Background technique

In the field of radiotherapy, precise tumor radiotherapy technology can effectively improve the efficacy of patients and reduce toxic and side effects, and precise radiotherapy relies on precise target contours. During the delineation of the target area, the target area must be delineated carefully with reference to the drainage area of the drainage area. In addition, the mediastinal lymphatic drainage area also plays a very important role in the formulation of clinical staging and treatment principles for patients with lung cancer. Therefore, the automatic delineation of the drainage area has very important clinical significance. This method is helpful for clinicians to delineate the mediastinal drainage area quickly, accurately and with high consistency.

At present, the mediastinal drainage area is completely delineated manually by clinicians. This approach has the following disadvantages:

First, the delineation speed is slow, which consumes a lot of precious time of the doctor; second, the accuracy of delineation depends on the clinical experience of the doctor, and requires a lot of prior clinical knowledge; difference. Fourth, there is inevitable human error. Therefore, on the basis of digital radiotherapy, how to quickly, accurately and consistently help doctors delineate the lymphatic drainage area is extremely important.

The information disclosed in this Background section is only for enhancement of understanding of the general background of the invention and should not be taken as an acknowledgement or any form of suggestion that this information forms the prior art already known to a person of ordinary skill in the art.

SUMMARY OF THE INVENTION

The purpose of the present invention is to provide an automatic delineation method for the mediastinal lymphatic drainage area based on a deep learning network. By introducing a multi-scale non-local attention module, the network can better locate and segment small drainage areas, and at the same time, the network can better The capture of distant anatomical information improves under-segmentation or over-segmentation problems.

In order to achieve the above purpose, the present invention provides an automatic delineation method for the mediastinal lymphatic drainage area based on a deep learning network, which is suitable for CT images. Mediastinal lymphatic drainage area images, and preprocess CT image data and mediastinal lymphatic drainage area images manually marked by doctors; Step S2: Group the preprocessed CT image data to obtain a training set, a validation set and a test set; Step S3: Data enhancement is performed on the training set, the validation set and the test set; step S4: constructing a deep learning segmentation model; and step S5: inputting the CT image data in the training set and the images of the mediastinal lymphatic drainage area manually marked by the doctor into the deep learning that has been constructed For the segmentation model, after the training iteration converges, the segmentation model of the mediastinal lymphatic drainage area is saved, and then the mediastinal lymphatic drainage area is identified and predicted, and the probability map of each partition of the mediastinal lymphatic drainage area is obtained.

In a preferred embodiment, the preprocessing CT image data in step S1 and the images of the mediastinal lymphatic drainage area manually marked by the doctor include the following steps: step S11: collecting a large number of multimodal and multidistribution CT three-dimensional images and corresponding clinical The contour map drawn manually by the doctor; Step S12: Resampling the CT three-dimensional image and the image of the mediastinal lymphatic drainage area manually marked by the doctor to generate an image with the same physical scale; Step S13: Obtaining the three-dimensional lung area and mediastinal position , crop the three-dimensional CT image to a fixed size according to the lung area and mediastinal position; and step S14: normalize the pixel values of the two-dimensional CT image, and generate a multi-distribution CT image input segmentation network according to the lung window and the mediastinal window.

In a preferred embodiment, the data enhancement in step S3 includes: random flip, random rotation, random distortion, random noise, random affine transformation, and random pruning.

In a preferred embodiment, step S4 includes the following steps: Step S41 : constructing a network structure sub-module of the segmentation model, including: first, convolution operations 2 times and downsampling 1 time to extract the feature map of the module , second, construct up-sampling 1 time and convolution operation 2 times to restore its original resolution, and use the skip structure to fuse feature maps of different scales; Step S42: Build the network structure attention module of the segmentation model, Including: pyramid downsampling the key values and eigenvalues in the attention module to reduce a lot of calculations, obtain multi-scale key values and eigenvalues, and then construct a convolution operation to simulate the relationship between the key value and the query value. attention relationship, and finally query the concerned feature map under the attention relationship, the attention module can capture the long-distance pixel dependency and extract the features of the multi-scale pyramid; and step S43: construct the network segmentation model network structure, and reuse the extraction step The feature sub-module of S41 is performed 4 times in order to have a larger receptive field and sufficient network capacity; the attention module of step S42 is inserted into each extracted feature sub-module, so that the network can extract long-distance dependencies and expand the network receptive field. At the same time, the multi-scale information captured by the attention module can be effectively extracted at each layer; then the recovered spatial resolution sub-model is reused 4 times; short connections are used between each module so that the network can be better reversed Propagation and feature fusion.

In a preferred embodiment, step S5 includes the following steps: Step S51: After a large number of patients are processed in the steps, the obtained data-enhanced images are input into the deep learning network. During the input process, the lungs obtained by processing steps S1 to S3 The region controls the number of CT layers of the input patient to reduce the input non-pulmonary regions; Step S52: randomly input the data-enhanced images into the network in groups, until the evaluation standard on the validation set no longer fluctuates greatly, and save the data with good performance on the validation set Model; Step S53: input the cases in the test set to the deep learning segmentation network that has been trained to obtain N partitions after being processed according to steps S1 to S3, and use the softmax function to convert the feature maps of the N partitions obtained into segmentation semantic probabilities Figure, and then use a fixed threshold to generate a binary image from the probability map; and step S54: evaluate the mutual relationship of the N partitions, obtain a mutual relationship table, and correct each partition, if a partition does not meet the outline standard defined by the doctor , the partition is processed by the correction procedure; if there is no relationship in the relationship table between a partition and other partitions, the partition is also processed by the correction procedure; until N partitions meet the clinician's delineation criteria, the final mediastinal lymph node is obtained. Drainage area segmentation results.

Compared with the prior art, the automatic delineation method of the mediastinal lymphatic drainage area based on the deep learning network of the present invention has the following beneficial effects: by introducing a multi-scale non-local attention mechanism, the network can better locate and segment the small drainage area, At the same time, the network can better capture the long-distance deconstructed structural information to improve the problem of under-segmentation or over-segmentation; the segmentation model of the mediastinal lymphatic drainage area can help doctors to delineate the target area and lymph nodes more accurately, and also confirm the clinical stage and The development of treatment plans provides a certain basis, which can greatly reduce the burden on doctors and improve the survival rate of patients.

Description of drawings

1 is a schematic flowchart of an automatic delineation method according to an embodiment of the present invention;

FIG. 2 is a schematic diagram of a deep learning network structure of an automatic delineation method according to an embodiment of the present invention.

Detailed ways

The specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings, but it should be understood that the protection scope of the present invention is not limited by the specific embodiments.

Unless expressly stated otherwise, throughout the specification and claims, the term "comprising" or its conjugations such as "comprising" or "comprising" and the like will be understood to include the stated elements or components, and Other elements or other components are not excluded.

As shown in FIG. 1 , according to a preferred embodiment of the present invention, an automatic delineation method for a mediastinal lymphatic drainage area based on a deep learning network is suitable for CT images, and the automatic delineation method includes the following steps: Step S1: Collect CT image data and The images of the mediastinal lymphatic drainage area manually annotated by the doctor, and the CT image data and the images of the mediastinal lymphatic drainage area manually annotated by the doctor are preprocessed. Step S2: Group the preprocessed CT image data to obtain a training set, a verification set and a test set. Step S3: Data enhancement is performed on the training set, the validation set and the test set. Step S4: Build a deep learning segmentation model. And step S5: input the CT image data in the training set and the images of the mediastinal lymphatic drainage area manually marked by the doctor into the deep learning segmentation model that has been constructed, after the training iteration converges, save the segmentation model of the mediastinal lymphatic drainage area, and then perform mediastinal lymphatic drainage. Zone identification and prediction, resulting in a probability map for each zone of the mediastinal lymphatic drainage zone.

In some embodiments, the preprocessing of the CT image data and the images of the mediastinal lymphatic drainage area manually marked by the doctor in step S1 includes the following steps: Step S11 : collecting a large number of multi-modality and multi-distribution CT three-dimensional images and corresponding manual by the clinician Outline drawing. Step S12: Resampling the CT three-dimensional images and the images of the mediastinal lymphatic drainage area manually marked by the doctor to generate images with the same physical scale. Step S13: Acquire the three-dimensional lung area and the mediastinal position, and crop the three-dimensional CT image to a fixed size according to the lung area and the mediastinal position. And step S14: normalize the pixel values of the two-dimensional CT image, and generate a multi-distribution CT image input segmentation network according to the lung window and the mediastinal window.

Normalized calculation method:

lower=c-w/2;

higher=c+w/2;

x[x<lower]=0;

x[x>higher]=higher;

x=(x-lower)/(higher-lower);

where x is the CT pixel matrix, c is the window level, and w is the window width.

In some embodiments, the data enhancement in step S3 includes: random flip, random rotation, random distortion, random noise, random affine transformation, and random pruning.

In some embodiments, step S4 includes the following steps: Step S41: constructing a network structure sub-module of the segmentation model, including: first, two convolution operations and one downsampling are used to extract the feature map of the module; Second, construct upsampling 1 time and convolution operation 2 times to restore its original resolution, and use skip structure to fuse feature maps of different scales. Step S42: constructing a network structure attention module of the segmentation model, including: pyramid downsampling the key values and eigenvalues in the attention module to reduce a large amount of calculations, obtain multi-scale key values and eigenvalues, and then construct a convolution The operation is used to simulate the attention relationship between the key value and the query value, and finally the focused feature map is queried under the attention relationship. The attention module can capture the long-distance pixel dependencies and extract the features of the multi-scale pyramid. And step S43: construct the network segmentation model network structure, and reuse the feature sub-module of extraction step S41 4 times, so as to have a larger receptive field and sufficient network capacity; insert the attention of step S42 into each extraction feature sub-module module, so that the network can extract long-distance dependencies and expand the network receptive field, and the multi-scale information captured by the attention module can be effectively extracted at each layer. The recovered spatial resolution submodel is then reused 4 times. Use short connections between each module so that the network can have better backpropagation and feature fusion.

In some embodiments, step S5 includes the following steps: Step S51: After a large number of patients are processed in the steps, the obtained data-enhanced images are input into the deep learning network. During the input process, the lung area control obtained by the processing in steps S1 to S3 Enter the number of CT slices of the patient and reduce the input of non-pulmonary areas.

How to calculate the training error:

L _loss =L _IOU +a*L _AC , where a is the balance factor;

Among them, N refers to the total amount of data, pi represents the _i -th pixel in the prediction result image, and qi represents the _i -th pixel in the gold standard image;

Among them, N refers to the total amount of data, p _ij represents the pixel point in the i-th row and the j-th column in the prediction result image, and n is the total number of pixels.

Step S52: The data-enhanced images are randomly input into the network in groups, until the evaluation standard on the validation set no longer fluctuates greatly, and the model with good performance on the validation set is saved.

Evaluation standard calculation method:

Among them, N refers to the total amount of data, pi represents the _ith pixel in the prediction result image, and qi represents the _ith pixel in the gold standard image.

Step S53: the cases in the test set are processed according to steps S1 to S3 and input to the deep learning segmentation network that has been trained to obtain N partitions, and the feature maps of the N partitions obtained are converted into segmentation semantic probability maps using the softmax function, Then use a fixed threshold to make the probability map generate a binary image. And step S54: evaluate the mutual relationship of the N partitions, obtain a mutual relationship table, and correct each partition. If a partition does not meet the delineation standard defined by the doctor, then process the partition through a correction procedure; if a partition and If other partitions do not have the relationship in the mutual relationship table, the partition is also processed through the correction procedure; until the N partitions meet the clinician's delineation criteria, the final segmentation result of the mediastinal lymphatic drainage area is obtained.

To sum up, the automatic delineation method of the mediastinal lymphatic drainage area based on the deep learning network of the present invention has the following advantages: by introducing a multi-scale non-local attention mechanism, the network can better locate and segment the small drainage area, and at the same time, the network can Better capture of long-distance anatomical structure information to improve the problem of under-segmentation or over-segmentation; the segmentation model of the mediastinal lymphatic drainage area can help doctors to more accurately delineate the target area and lymph nodes, and also provide doctors with clinical staging and treatment plans. It can greatly reduce the burden on doctors and improve the survival rate of patients.

The foregoing descriptions of specific exemplary embodiments of the present invention have been presented for purposes of illustration and description. These descriptions are not intended to limit the invention to the precise form disclosed, and obviously many changes and modifications are possible in light of the above teachings. The exemplary embodiments were chosen and described for the purpose of explaining certain principles of the invention and their practical applications, to thereby enable one skilled in the art to make and utilize various exemplary embodiments and various different aspects of the invention. Choose and change. The scope of the invention is intended to be defined by the claims and their equivalents.

Claims

An automatic delineation method for a mediastinal lymphatic drainage area based on a deep learning network, which is suitable for CT images, wherein the automatic delineation method comprises the following steps:

Step S1: collecting CT image data and images of the mediastinal lymphatic drainage area manually marked by the doctor, and preprocessing the CT image data and the images of the mediastinal lymphatic drainage area manually marked by the doctor;

Step S2: grouping the preprocessed CT image data to obtain a training set, a verification set and a test set;

Step S3: performing data enhancement on the training set, the verification set and the test set;

Step S4: constructing a deep learning segmentation model; the step S4 includes the following steps:

Step S41 : constructing a network structure sub-module of the segmentation model, including: first, 2 convolution operations and 1 downsampling to extract the feature map of the module; secondly, 1 upsampling and 2 convolution operations are constructed Second, it is used to restore its original resolution and use the jump structure to fuse feature maps of different scales; the down module of this network uses trilinear interpolation to downsample, and the up module uses a deconvolution module with holes to upsample;

Step S42: constructing a network structure attention module of the segmentation model, including: pyramid downsampling the key values and eigenvalues in the attention module to reduce a large amount of calculations, obtain multi-scale key values and eigenvalues, and then construct The convolution operation is used to simulate the attention relationship between the key value and the query value, and finally the focused feature map is queried under the attention relationship. The attention module can capture long-distance pixel dependencies and extract multi-scale The characteristics of the pyramid; the matrices involved in the calculation are Q(Query), K(Key), and V(Value). In order to accelerate the image attention mechanism, the Q value and V value have undergone multi-scale downsampling operations; in order to accelerate convergence, the Q value The K value has undergone a convolution operation before calculating similarity; the similarity calculation function is different. Our similarity calculation function is as follows,
as well as

Step S43: Construct the network structure of the network segmentation model, and repeat the extraction of the network structure sub-module described in the step S41 4 times, so as to have a larger receptive field and sufficient network capacity; The attention module of step S42 is inserted into the module, so that the network extracts long-distance dependencies, expands the network receptive field, and the multi-scale information captured by the attention module can be effectively extracted in each layer; then reuse the recovered spatial resolution submodel 4 times; use short connections between each module so that the network can have better backpropagation and feature fusion; and

Step S5: Input the CT image data in the training set and the images of the mediastinal lymphatic drainage area manually marked by the doctor into the deep learning segmentation model that has been constructed, and after the training iteration converges, save the segmentation model of the mediastinal lymphatic drainage area, Then carry out the identification and prediction of the mediastinal lymphatic drainage area, and obtain the probability map of each partition of the mediastinal lymphatic drainage area, and the step S5 includes the following steps:

Step S51: After a large number of patients are processed in the steps, the obtained data-enhanced images are input into the deep learning network. During the input process, the lung regions obtained through the steps S1 to S3 are controlled to input the CT layers of the patients to reduce the input Non-lung area, the training error calculation method of the training set is:

L loss =L IOU +a*L AC , where a is the balance factor;

Among them, N refers to the total amount of data, pi represents the i -th pixel in the prediction result image, and qi represents the i -th pixel in the gold standard image;

Among them, N refers to the total amount of data, p ij represents the pixel point in the i-th row and the j-th column in the prediction result image, and n is the total number of pixels;

Step S52: Randomly input the data-enhanced images into the network in groups until the evaluation criteria on the verification set no longer fluctuate greatly, and save the model with good performance on the verification set. The calculation method of the evaluation criteria in the verification set is:

Among them, N refers to the total amount of data, pi represents the i -th pixel in the prediction result image, and qi represents the i -th pixel in the gold standard image;

Step S53: Input the cases in the test set into the deep learning segmentation network that has been trained to obtain N partitions after being processed according to the steps S1 to S3, and use the softmax function to convert the obtained feature maps of the N partitions into a segmentation semantic probability map, and then using a fixed threshold to generate a binary image from the probability map; and

Step S54: Evaluate the mutual relationship of the N partitions, obtain a mutual relationship table, and correct each partition. If a partition does not meet the delineation standard defined by the doctor, process the partition through a correction procedure; If there is no relationship in the mutual relationship table with other partitions, the partition is also processed through the correction procedure; until the N partitions meet the clinician's delineation criteria, the final segmentation result of the mediastinal lymphatic drainage area is obtained.
The automatic delineation method for the mediastinal lymphatic drainage area based on a deep learning network according to claim 1, wherein the preprocessing of the CT image data and the mediastinal lymphatic drainage manually marked by a doctor in the step S1 Area images include the following steps:

Step S11: collecting a large number of multi-modal and multi-distribution CT three-dimensional images and corresponding contour maps manually drawn by clinicians;

Step S12: resampling the CT three-dimensional image and the image of the mediastinal lymphatic drainage area manually marked by the doctor to generate an image with the same physical scale;

Step S13: after obtaining the three-dimensional lung area, use the threshold method to segment the body area, then obtain the mediastinal area by morphological method based on the lung area and the body area, and use the mediastinal area to crop the three-dimensional CT image to a fixed size; and

Step S14: Normalize the pixel values of the two-dimensional CT image, and generate a multi-distribution CT image input segmentation network according to the lung window and the mediastinal window.
The automatic delineation method for the mediastinal lymphatic drainage area based on a deep learning network according to claim 1, wherein the data enhancement in the step S3 comprises: random flip, random rotation, random distortion, random noise, random Affine transformation, random trimming.