KR20210020629A - Tumor automatic segmentation based on deep learning in a medical image - Google Patents

Abstract

The present invention provides a deep learning-based automatic tumor segmentation method in a medical image which can accurately perform automatic segmentation of a tumor. The deep learning-based automatic tumor segmentation method in a medical image comprises: a step of inputting a plurality of sagittal images, a plurality of coronal images, and a plurality of axial images of a two-dimensional medical video including a tumor into a two-dimensional deep neural network to generate a plurality of prediction maps and a plurality of label data of a region of interest through the two-dimensional deep neural network; a step of combining the generated plurality of label data and the generated plurality of prediction maps by using a distance probability map to generate a preliminary shape model of the region of interest; and a step of inputting the preliminary shape model and a three-dimensional medical video including the tumor into a three-dimensional deep neural network to acquire a segmentation result of the region of interest through the three-dimensional deep neural network.

Description

Automatic tumor segmentation method based on deep learning in medical images{TUMOR AUTOMATIC SEGMENTATION BASED ON DEEP LEARNING IN A MEDICAL IMAGE}

The present invention relates to a method for automatic tumor segmentation based on deep learning in medical images, and more particularly, to a method for automatic tumor segmentation using a 2D deep neural network and a 3D deep neural network.

In radiation therapy, it is important to accurately inject radiation into the tumor while minimizing the amount of radiation injected into the surrounding normal tissues. To this end, image-guided radiation therapy is widely used in which an image of a patient immediately before treatment is re-acquired and compared with the patient's posture and tumor position at the time of treatment plan to correct the patient's position or correct the treatment plan to increase the accuracy of treatment.

It is essential to measure the tumor size in order to evaluate the prognosis and treatment response of cancer. Tumor size is currently quantified using (a) one-dimensional RECIST (b) two-dimensional WHO or (c) 3D volume as shown in FIG. 1, but line-length measurement may be misleading compared to 3D measurement of tumor. There is still unproven evidence suggesting that. In addition, volumetric segmentation is required to analyze image features and determine tumor volume for treatment planning. However, as shown in FIG. 2, tumors have various locations ((a) apex of the lung, (b) chest wall, (c) mediastinum, (d) base of the lung) and the size of the tumor. As it varies from 1cm to 18cm, it is difficult to perform the division work.

iηek et al., "3D U-Net: Learning Dense Volumetric Segmentation from Sparse Annotation", MICCAI 2016. "Chest computed tomography display preferences: survey of thoracic radiologists.", Investigate

iηek et al., "3D U-Net: Learning Dense Volumetric Segmentation from Sparse Annotation", MICCAI 2016.

The problem to be solved by the present invention is to input the pre-shape model generated through the 2D deep neural network and the bounding volume of the abdomen into the 3D deep neural network, so that the tumor is located at the boundary between It is to provide an automatic tumor segmentation method that accurately divides the stomach.

The problems to be solved by the present invention are not limited to the problems mentioned above, and other problems that are not mentioned will be clearly understood by those skilled in the art from the following description.

In the medical image according to an aspect of the present invention for solving the above-described problems, a method for automatic tumor segmentation based on deep learning includes a plurality of axial images and a plurality of coronal images of a 2D medical image including a tumor. And inputting a plurality of sagittal images into a two-dimensional deep neural network. Generating a plurality of label data and a plurality of prediction maps of a region of interest through the 2D deep neural network; Generating a prior shape model of the region of interest by combining the generated plurality of label data and the plurality of prediction maps using a distance probability map; And inputting the 3D medical image including the tumor and the pre-shape model to a 3D deep neural network. And obtaining a result of segmentation of the region of interest through the 3D deep neural network.

Other specific details of the present invention are included in the detailed description and drawings.

According to the present invention as described above, it has various effects as follows.

The present invention can accurately perform automatic segmentation of tumors.

In addition, the present invention can prevent over-segmentation in a 3D deep neural network by using a pre-shape model.

The effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the following description.

1 and 2 are diagrams showing a medical image including a conventional tumor.
3 is a flowchart illustrating a method of automatically segmenting a tumor according to an embodiment of the present invention.
4 is a block diagram illustrating a method of automatically segmenting a tumor according to an embodiment of the present invention.
5 is an exemplary diagram for explaining 2D and 3D U-Net according to an embodiment of the present invention.
6 is an exemplary diagram illustrating a distance probability map according to an embodiment of the present invention.
7 is a table for explaining the effect of the automatic tumor segmentation method according to an embodiment of the present invention.
8 is an exemplary diagram for explaining the effect of a method for auto-segmenting a tumor according to an embodiment of the present invention.

Advantages and features of the present invention, and a method of achieving them will become apparent with reference to the embodiments described below in detail together with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but may be implemented in a variety of different forms, only the present embodiments are intended to complete the disclosure of the present invention, It is provided to fully inform the technician of the scope of the present invention, and the present invention is only defined by the scope of the claims.

The terms used in the present specification are for describing exemplary embodiments and are not intended to limit the present invention. In this specification, the singular form also includes the plural form unless specifically stated in the phrase. As used in the specification, “comprises” and/or “comprising” do not exclude the presence or addition of one or more other elements other than the mentioned elements. Throughout the specification, the same reference numerals refer to the same elements, and “and/or” includes each and all combinations of one or more of the mentioned elements. Although "first", "second", and the like are used to describe various elements, it goes without saying that these elements are not limited by these terms. These terms are only used to distinguish one component from another component. Therefore, it goes without saying that the first component mentioned below may be the second component within the technical idea of the present invention.

Unless otherwise defined, all terms (including technical and scientific terms) used in the present specification may be used as meanings that can be commonly understood by those of ordinary skill in the art to which the present invention belongs. In addition, terms defined in a commonly used dictionary are not interpreted ideally or excessively unless explicitly defined specifically.

Spatially relative terms "below", "beneath", "lower", "above", "upper", etc., as shown in the figure It can be used to easily describe the correlation between a component and other components. Spatially relative terms should be understood as terms including different directions of components during use or operation in addition to the directions shown in the drawings. For example, if a component shown in a drawing is turned over, a component described as "below" or "beneath" of another component will be placed "above" the other component. I can. Accordingly, the exemplary term “below” may include both directions below and above. Components may be oriented in other directions, and thus spatially relative terms may be interpreted according to orientation.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings.

Semantic segmentation, like general image segmentation, is not limited to simply dividing into similar areas in terms of certain features or calculated attributes, but a technology that divides semantically the same part and determines which category the part belongs to. Say.

That is, it may correspond to a technology for classifying which category of all pixels of an image belongs within a predefined category, and may correspond to pixelwise classification.

The semantic image segmentation method is largely divided into two categories. The first is a technique of extracting hand-craft features from the input image, dividing it into super-pixel units, and then dividing the image based on meaning. In more detail, a user may directly design and extract features that may serve as clues by analyzing given image data. Thereafter, segmentation may be performed in units of super pixels based on patterns of the extracted features. This process can lead to improved accuracy and speed. Thereafter, semantic image segmentation is performed using a support vector machine for each super pixel, and it is possible to determine which classification a corresponding pixel or super pixel belongs to. This method has a disadvantage in that the range of use of the system is limited because, when the type of image input to the system is changed, a corresponding handmade feature must be redesigned each time, and the manual feature extraction has a disadvantage in that the processing speed is slow.

The second is a technique of extracting features using deep learning, and then classifying them in pixel units. As the performance of deep learning-based classification has been proven to be excellent, an approach using a convolutional neural network (CNN) structure has been proposed for semantic-based image segmentation. FCN (Fully Convolutional Networks), which has changed the CNN structure, shows excellent performance in image segmentation. After the superpixel division is performed, the CNN filter is trained using the training dataset, the image is segmented, and post-processing such as CRF (Conditional Random Field) can be performed.

Hereinafter, the segmentation described in the present invention may mean semantic image segmentation, and a semantic image segmentation method based on deep learning may be applied.

3 is a flowchart illustrating a method of automatically segmenting a tumor according to an embodiment of the present invention. 4 is a block diagram illustrating a method of automatically segmenting a tumor according to an embodiment of the present invention. 5 is an exemplary diagram for explaining 2D and 3D U-Net according to an embodiment of the present invention. 6 is an exemplary diagram illustrating a distance probability map according to an embodiment of the present invention. 7 is a table for explaining the effect of the automatic tumor segmentation method according to an embodiment of the present invention. 8 is an exemplary diagram for explaining the effect of the method of automatically segmenting a tumor according to an embodiment of the present invention.

The present invention may be implemented as an automatic tumor segmentation system, and may include an image input unit, a 2D deep learning unit, a pre-shape model generation unit, an image editing unit, and a 3D deep learning unit.

3 to 8, in an embodiment, in operation 31, the image editing unit applies intensity normalization and spacing normalization to a medical image, thereby providing a plurality of axial images, It is possible to obtain a coronal image and a plurality of sagittal images. For example, if the location where the tumor is placed is the lung, pre-treatment may be required to identify the surrounding area due to the nature of the lung, and the inside of the chest cavity is divided into four areas: upper / chest cavity / chest wall attachment / lower area, and It is possible to perform a treatment process suitable for the characteristics. For example, intensity normalization can be applied to medical images based on the values of windowing width 1500 and windowing level -600, and spacing normalization ^{can be applied to all sizes of medical images with 0.54X0.54mm 2.} I can.

In an embodiment, in operation 32, the image input unit includes a plurality of axial images 110, a plurality of coronal images 120, and a plurality of sagittals of the medical image 100 including the tumor. By inputting the image 130 to the 2D deep neural network 200, the 2D deep learning unit provides a plurality of label data 310 of a region of interest through the 2D deep neural network 200. 320 and 330 and a plurality of prediction maps 410, 420, and 430 may be generated. For example, the 2D deep neural network 200 may include at least one of a 2D fully convolutional network (FCN) and a 2D U-net. For example, the medical image includes a 2D medical image (eg, x-ray image) and/or a 3D medical image (eg, CT image, MRI, PET image), and there is no particular limitation if it is a medical image. “Image” refers to a subject collected by computed tomography (CT), magnetic resonance imaging (MRI), ultrasound, or any other medical imaging system known in the art. It may be a medical image of. The medical image 100 is voxel data and is composed of a plurality of slices, that is, a plurality of unit images. For example, the region of interest may be a region in which a tumor is placed.

In one embodiment, as shown in Fig. 5, the 2D U-net may correspond to the convolutional network structure proposed in “3D U-Net: Learning Dense Volumetric Segmentation from Sparse Annotation”, MICCAI 2016. For example For example, the 2D U-Net may include a contracting path 40 shown on the left and an expansive path 50 shown on the right, which follows a typical structure of a convolutional neural network. Bar, this involves the iterative application of two 3x3 convolutions (unpadded convolutions), each of which contains a rectified linear unit (ReLU) and stride for downsampling. A 2x2 maximum pooling operation of (stride) 2 follows: In each downsampling step, the number of feature channels is doubled, and every step in the expansion path ups the feature map. A 2x2 convolution (“up-convolution”) that cuts the number of upsampling and subsequent feature channels in half, concatenation with feature maps from the correspondingly cropped contraction path, and 2 It consists of 3x3 convolutions of times, each of which is followed by a ReLU of 3x3 convolutions of 2. The aforementioned truncation is necessary because of the loss of border pixels in all convolutions, in the final layer. A 1x1 convolution is used to map each 64 dimensional (64-component) feature vector to a desired number of classes In this example neural network, 22 convolutional layers are included, which are arbitrary. So that the segmentation map in the output is neatly connected, It will be appreciated by those skilled in the art that it is important to select the input tile size so that all 2x2 max-pooling operations are applied to layers with even x and y sizes.

In an embodiment, the plurality of prediction maps 410, 420, 430, and prediction maps have a lower resolution than the input medical image, but may have a plurality of channels. Through the generated prediction maps 410, 420, 430, and prediction maps, they are shared with each other in the region classification step and the object detection step to classify the region and detect the region of interest (any one of the abdominal organs). That is, pixel labeling may be performed from a plurality of prediction maps 410, 420, 430, and prediction maps to distinguish an image region. In this case, pixel labeling may be performed by attaching the same number to all adjacent pixels having similar features of the input image. In addition, the pixel labeling result included in the plurality of label data 310, 320, and 330 may include probability distribution information for each pixel of a class to be classified of an image.

In one embodiment, the generation of a plurality of label data (310, 320, 330) and a plurality of prediction maps (410, 420, 430, Prediction map) may use a convolution neural network (CNN). However, the present invention is not limited thereto, and may vary depending on the purpose of use of the image processing apparatus, such as a deep neural network (DNN) or a recurrent neural network (RNN). Here, as an example, a plurality of label data 310, 320, 330 and a plurality of prediction maps 410, 420, 430, and prediction maps may be generated according to a 2D fully convolutional network (FCN).

In an embodiment, in operation 33, the pre-shape model generation unit converts the generated label data 310, 320, 330 and the plurality of prediction maps 410, 420, and 430 into a distance probability map 500. By combining by using, the pre-shape model 700 of the region of interest may be generated. For example, the pre-shape model 700 may include 3D spatial shape information of the region of interest in the form of a probability map. For example, the pre-shape model 700 may include information that is meaningful for classifying an area in the medical image 100 and detecting an object, and may include, for example, luminous intensity, color, outline, etc. of the medical image 100. .

In an embodiment, the distance probability map 500 is applied with a maximum probability value voting method, and the pre-shape model 700 selects a maximum probability value among a plurality of prediction maps having a reference probability value or more. It can be generated by combining a plurality of prediction maps. For example, the reference probability value may be 0.5, and the pre-shape model 700 may be generated by connecting components labeled with only maximum probability values while being 0.5 or more. Since tumors vary in location and size, the error range can be reduced by using the maximum probability value instead of using the average probability. Meanwhile, the probability value here may mean a probability value close to a tumor as shown in FIG. 6.

In an embodiment, in operation 34, the image editing unit may crop at least a portion of the remaining area of the 3D medical image 800 except for the ROI. For example, in order to reduce the operation process of the 3D deep neural network 900, a part of the boundary area in the 3D medical image 800 may be cropped. Specifically, for example, by using the brightness value information, a border portion completely different from the brightness value information of the tumor and surrounding organs may be cropped.

In an embodiment, in operation 35, the image input unit may input a 3D medical image 800 including abdominal organs and a pre-shape model 700 to the 3D deep neural network 900. For example, the 3D deep neural network 900 may include a 3D U-net. Also, unlike this, the cropped pre-shape model and the cropped 3D medical image 810 may be input to the 3D deep neural network 900. Meanwhile, the 3D U-net of the 3D deep neural network 900 is omitted because it overlaps with the description of FIG. 5.

In an embodiment, in operation 36, the 3D deep learning unit may obtain the segmentation result 10 of the ROI through the 3D deep neural network 900. Here, the segmentation result 10 may be an image in which a tumor corresponding to the region of interest is segmented.

That is, the present invention considers the entire context information of the region in which the region of interest is arranged through the 2D deep neural network 200 and the pre-shape model 700, and uses the pre-shape model 700 to generate a 3D medical image. Through this, it is possible to increase the speed and accuracy of tumor segmentation by reducing the operation to grasp the regional context information where the region of interest is located.

On the other hand, the data set used to confirm the effect of the present invention includes 222 medical images obtained for radiation therapy in patients, of which 150 are for learning, 38 for validation, and 34 for testing. Divided.

In order to evaluate the accuracy of the division result (10), a Dice similarity coefficient (DSC) was calculated and compared through Equation 1.

[Equation 1]

At this time, TP (True Positive) is the number of pixels in the automatically segmented region in the manually segmented long-term region, TN (True Negative) is the number of pixels in the region that is not automatically segmented in the non-manually segmented long term region, and FP (False Positive). Denotes the number of pixels in a region that is automatically divided in a non-manually divided long-term region, and FN (False Negative) denotes the number of pixels in a region that is not automatically divided in the manually divided long-term region.

7 and 8, Class 1 is an isolated tumor, Class 2 is a tumor attached to the chest wall, Class 3 is a tumor attached to mediastinum, and Class 4 is a tumor surrounded by surrounding structures at the upper and lower ends of the lung. to be. As shown in FIG. 7, it can be seen that the DSC of the coupling-Net (2.5D+3D with shape-enhanced prior (pre-shape model)) of the present invention is the highest at 70.71% when all classes are considered. Meanwhile, here, 2.5D may mean forming label data and a prediction map through a 2D deep neural network.

In addition, as shown in Figure 8 ((a) Original image, (b) 2D segmentation result (c) 2.5D segmentation result, (d) 3D segmentation result, (e) Coupling-net), red is ground-truth. Blue means over segmentation, and green means under segmentation, so the coupling-Net of the present invention with the smallest green and blue lines and the clearest boundary in red yields the highest level of tumor segmentation in all classes. can confirm.

In a medical image according to an embodiment of the present invention, a method for automatic tumor segmentation based on deep learning includes a plurality of axial images, a plurality of coronal images, and a plurality of thalamus ( Sagittal) by inputting images into a 2D deep neural network. Generating a plurality of label data and a plurality of prediction maps of a region of interest through the 2D deep neural network; Generating a prior shape model of the region of interest by combining the generated plurality of label data and the plurality of prediction maps using a distance probability map; And inputting the 3D medical image including the tumor and the pre-shape model to a 3D deep neural network. And obtaining a result of dividing the region of interest through the 3D deep neural network.

According to various embodiments, by applying intensity normalization and spacing normalization to the 2D medical image, the plurality of axial images, the plurality of coronal images, and the plurality of sagittal images are applied. It may further include the step of obtaining the (Sagittal) image.

According to various embodiments of the present disclosure, the step of cropping at least a portion of a region other than the region of interest in the 3D medical image; And inputting the cropped 3D medical image to the 3D deep neural network.

According to various embodiments of the present disclosure, by inputting the pre-shape model and the cropped 3D medical image into the 3D deep neural network. And obtaining a result of dividing the region of interest through the 3D deep neural network.

According to various embodiments, a maximum probability value voting method is applied to the distance probability map, and the prior shape model is a plurality of prediction maps having a maximum probability value among a plurality of prediction maps having a reference probability value or more. It can be created by combining prediction maps.

According to various embodiments, the region of interest may be a region in which the tumor is disposed.

According to various embodiments, the pre-shape model may include 3D spatial shape information of the ROI in the form of a probability map.

According to various embodiments, the 2D deep neural network may include at least one of a 2D fully convolutional network (FCN) and a 2D U-net.

According to various embodiments, the 3D deep neural network may include a 3D U-net.

The steps of a method or algorithm described in connection with an embodiment of the present invention may be implemented directly in hardware, implemented as a software module executed by hardware, or a combination thereof. Software modules include Random Access Memory (RAM), Read Only Memory (ROM), Erasable Programmable ROM (EPROM), Electrically Erasable Programmable ROM (EEPROM), Flash Memory, hard disk, removable disk, CD-ROM, or It may reside on any type of computer-readable recording medium well known in the art to which the present invention pertains.

In the above, embodiments of the present invention have been described with reference to the accompanying drawings, but those of ordinary skill in the art to which the present invention pertains can be implemented in other specific forms without changing the technical spirit or essential features. You can understand. Therefore, the embodiments described above are illustrative in all respects, and should be understood as non-limiting.

100: medical image
200: 2D deep neural network
310,320,330: label data
410,420,430: prediction map
700: pre-shape model
800: 3D medical image
900: 3D deep neural network

Claims (10)

In the method of automatic tumor segmentation based on deep learning in medical images,
By inputting a plurality of axial images, a plurality of coronal images, and a plurality of sagittal images of a 2D medical image containing a tumor into a 2D deep neural network, through the 2D deep neural network. Generating a plurality of label data and a plurality of prediction maps of a region of interest;
Generating a prior shape model of the region of interest by combining the generated plurality of label data and the plurality of prediction maps using a distance probability map; And
And inputting the 3D medical image including the tumor and the pre-shape model to a 3D deep neural network, and obtaining a result of segmentation of the region of interest through the 3D deep neural network. Automatic tumor segmentation method based on deep learning in medical imaging. The method of claim 1,
The plurality of axial images, the plurality of coronal images, and the plurality of sagittal images are obtained by applying intensity normalization and spacing normalization to the 2D medical image. Automatic tumor segmentation method based on deep learning in medical images, further comprising the step of: The method of claim 1,
Cropping at least a portion of the remaining area other than the ROI in the 3D medical image; And
And inputting the cropped 3D medical image to the 3D deep neural network. A method for automatically segmenting tumors based on deep learning in medical images, further comprising. The method of claim 3,
And inputting the pre-shape model and the cropped 3D medical image to the 3D deep neural network, and obtaining a result of segmentation of the ROI through the 3D deep neural network. Automatic tumor segmentation method based on deep learning in images. The method of claim 1,
The distance probability map is applied with a maximum probability value voting method,
The pre-shape model is generated by combining a plurality of prediction maps having a maximum probability value among a plurality of prediction maps having a reference probability value or more. The method of claim 1,
The region of interest is a region in which the tumor is placed. A method of automatically segmenting tumors based on deep learning in medical images. The method of claim 1,
The pre-shape model includes 3D spatial shape information of the region of interest in the form of a probability map. A method for automating tumor segmentation based on deep learning in medical images. The method of claim 1,
The 2D deep neural network includes at least one of a 2D fully convolutional network (FCN) and a 2D U-net. A method for automating tumor segmentation based on deep learning in medical images. The method of claim 1,
The 3D deep neural network is a method of automatically segmenting tumors based on deep learning in medical images, characterized in that the 3D U-net is included. A tumor auto-segmentation program based on deep learning in medical images, which is combined with a computer that is hardware and stored in a medium to execute the method of any one of claims 1 to 9.

Images