KR102349515B1 - Tumor automatic segmentation based on deep learning in a medical image - Google Patents

Abstract

An automatic tumor segmentation method based on deep learning in medical images is provided. The automatic tumor segmentation method based on deep learning in the medical image is a two-dimensional deep neural by entering into the network. generating a plurality of label data and a plurality of prediction maps of a region of interest through the two-dimensional deep neural network; generating a prior shape model of the ROI by combining the generated plurality of label data and the plurality of prediction maps using a distance probability map; and inputting the 3D medical image including the tumor and the pre-shape model into a 3D deep neural network. and obtaining a segmentation result of the region of interest through the 3D deep neural network.

Description

Automated tumor segmentation method based on deep learning in medical images {TUMOR AUTOMATIC SEGMENTATION BASED ON DEEP LEARNING IN A MEDICAL IMAGE}

The present invention relates to an automatic tumor segmentation method based on deep learning in a medical image, and more particularly, to a tumor automatic segmentation method using a two-dimensional deep neural network and a three-dimensional deep neural network.

In radiation therapy, it is important to accurately inject radiation into the tumor while minimizing the amount of radiation to the surrounding normal tissue. To this end, image-guided radiation therapy, which obtains an image of the patient immediately before treatment and compares it with the patient's posture and tumor position at the time of the treatment plan, corrects the patient's position or revises the treatment plan to improve the accuracy of treatment.

It is essential to measure the tumor size for the evaluation of cancer prognosis and treatment response. Tumor size is currently quantified using (a) one-dimensional RECIST (b) two-dimensional WHO or (c) 3D volume as shown in Figure 1, but compared to 3D measurements of tumors, line-length measurements may be misleading. There is still unsubstantiated evidence suggesting that In addition, volumetric segmentation is required to analyze image features and determine tumor volume for treatment planning. However, as shown in FIG. 2, the tumor is located in various locations ((a) apex of the lung, (b) chest wall, (c) mediastinum, (d) base of the lung) and the size of the tumor is As it varied from 1 cm to 18 cm, there was a problem in that it was difficult to perform the division operation.

iηek et al., "3D U-Net: Learning Dense Volumetric Segmentation from Sparse Annotation", MICCAI 2016. "Chest computed tomography display preferences: survey of thoracic radiologists.", Investigate

iηek et al., “3D U-Net: Learning Dense Volumetric Segmentation from Sparse Annotation”, MICCAI 2016.

The problem to be solved by the present invention is to input the pre-shape model generated through the two-dimensional deep neural network and the bounding volume of the abdomen into the three-dimensional deep neural network, so that the tumor is located at the boundary with the surroundings regardless of the location and size. It is to provide an automatic tumor segmentation method that accurately divides the stomach.

The problems to be solved by the present invention are not limited to the problems mentioned above, and other problems not mentioned will be clearly understood by those skilled in the art from the following description.

A method for automatically segmenting a tumor based on deep learning in a medical image according to an aspect of the present invention for solving the above-mentioned problems is a plurality of axial images and a plurality of coronal images of a two-dimensional medical image including a tumor. and inputting multiple sagittal images into a two-dimensional deep neural network. generating a plurality of label data and a plurality of prediction maps of a region of interest through the two-dimensional deep neural network; generating a prior shape model of the ROI by combining the generated plurality of label data and the plurality of prediction maps using a distance probability map; and inputting the 3D medical image including the tumor and the pre-shape model into a 3D deep neural network. and obtaining a segmentation result of the region of interest through the 3D deep neural network.

Other specific details of the invention are included in the detailed description and drawings.

According to the present invention as described above, it has various effects as follows.

The present invention can accurately perform automatic segmentation of tumors.

In addition, the present invention can prevent over-segmentation in a 3D deep neural network by using a pre-shape model.

Effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the following description.

1 and 2 are diagrams showing a conventional medical image including a tumor.
3 is a flowchart illustrating an automatic tumor segmentation method according to an embodiment of the present invention.
4 is a block diagram illustrating an automatic tumor segmentation method according to an embodiment of the present invention.
5 is an exemplary diagram for explaining 2D and 3D U-Net according to an embodiment of the present invention.
6 is an exemplary diagram for explaining a distance probability map according to an embodiment of the present invention.
7 is a table for explaining the effect of the automatic tumor segmentation method according to an embodiment of the present invention.
8 is an exemplary view for explaining the effect of the automatic tumor segmentation method according to an embodiment of the present invention.

Advantages and features of the present invention and methods of achieving them will become apparent with reference to the embodiments described below in detail in conjunction with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but may be implemented in various different forms, and only the present embodiments allow the disclosure of the present invention to be complete, and those of ordinary skill in the art to which the present invention pertains. It is provided to fully understand the scope of the present invention to those skilled in the art, and the present invention is only defined by the scope of the claims.

The terminology used herein is for the purpose of describing the embodiments and is not intended to limit the present invention. In this specification, the singular also includes the plural unless specifically stated otherwise in the phrase. As used herein, “comprises” and/or “comprising” does not exclude the presence or addition of one or more other components in addition to the stated components. Like reference numerals refer to like elements throughout, and "and/or" includes each and every combination of one or more of the recited elements. Although "first", "second", etc. are used to describe various elements, these elements are not limited by these terms, of course. These terms are only used to distinguish one component from another. Accordingly, it goes without saying that the first component mentioned below may be the second component within the spirit of the present invention.

Unless otherwise defined, all terms (including technical and scientific terms) used herein will have the meaning commonly understood by those of ordinary skill in the art to which this invention belongs. In addition, terms defined in a commonly used dictionary are not to be interpreted ideally or excessively unless specifically defined explicitly.

Spatially relative terms "below", "beneath", "lower", "above", "upper", etc. It can be used to easily describe the correlation between a component and other components. Spatially relative terms should be understood as terms including different directions of components during use or operation in addition to the directions shown in the drawings. For example, when a component shown in the drawing is turned over, a component described as “beneath” or “beneath” of another component may be placed “above” of the other component. can Accordingly, the exemplary term “below” may include both directions below and above. Components may also be oriented in other orientations, and thus spatially relative terms may be interpreted according to orientation.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

Semantic segmentation, like general image segmentation, does not stop at dividing into similar regions in terms of certain features or calculated properties, but also semantically divides the same parts and determines what category the parts belong to. say

That is, it may correspond to a technique of classifying all pixels of an image to which category they belong within a predefined category, and may correspond to pixelwise classification.

Methods for semantic image segmentation are largely divided into two categories. The first is a technique of extracting hand-crafted features from an input image, dividing it into super-pixel units, and then segmenting the image based on meaning. More specifically, the user can design and extract features that can be clues by analyzing the given image data. Thereafter, segmentation may be performed in units of super pixels based on the pattern of the extracted features. This process can lead to improvements in accuracy and speed. Thereafter, semantic image segmentation may be performed using a support vector machine in units of each super-pixel to determine which category the corresponding pixel or super-pixel belongs to. This method has a disadvantage in that the range of system utilization is limited because homemade features must be redesigned every time the type of image input to the system changes, and homemade feature extraction also has a disadvantage in that the processing speed is slow.

The second is a technique for extracting features using deep learning and then classifying them in units of pixels based on this. As the performance of deep learning-based classification has been proven to be excellent, an approach using a convolutional neural network (CNN) structure has been proposed in semantic-based image segmentation. Fully Convolutional Networks (FCNs), which have changed the CNN structure, show excellent performance in image segmentation. After segmentation in units of superpixels, a CNN filter is trained using the training dataset, and the image is segmented, followed by post-processing such as CRF (Conditional Random Field).

Hereinafter, segmentation described in the present invention may mean semantic image segmentation, and a deep learning-based semantic image segmentation method may be applied.

3 is a flowchart illustrating an automatic tumor segmentation method according to an embodiment of the present invention. 4 is a block diagram illustrating an automatic tumor segmentation method according to an embodiment of the present invention. 5 is an exemplary diagram for explaining 2D and 3D U-Net according to an embodiment of the present invention. 6 is an exemplary diagram for explaining a distance probability map according to an embodiment of the present invention. 7 is a table for explaining the effect of the automatic tumor segmentation method according to an embodiment of the present invention. 8 is an exemplary view for explaining the effect of the automatic tumor segmentation method according to an embodiment of the present invention.

The present invention may be implemented as an automatic tumor segmentation system, and may include an image input unit, a two-dimensional deep learning unit, a pre-shape model generation unit, an image editing unit, and a three-dimensional deep learning unit.

3 to 8 , in an embodiment, in operation 31 , the image editing unit applies intensity normalization and spacing normalization to a medical image to obtain a plurality of axial images, a plurality of A coronal image and a plurality of sagittal images may be acquired. For example, if the location of the tumor is the lung, pretreatment for identification with the surrounding area may be required due to the characteristics of the lung, and the inside of the chest cavity is divided into 4 areas: upper / thoracic sinus / chest wall attachment / lower region A treatment process suitable for the characteristics may be performed. For example, intensity normalization can be applied to the medical image based on the windowing width 1500 and windowing level -600 values, and spatial normalization ^{can be applied to all sizes of the medical image to 0.54X0.54mm 2 .} can

In an embodiment, in operation 32 , the image input unit performs a plurality of axial images 110 , a plurality of coronal images 120 , and a plurality of sagittal images of the medical image 100 including the tumor. By inputting the image 130 into the two-dimensional deep neural network 200, the two-dimensional deep learning unit uses a plurality of label data 310 of a region of interest through the two-dimensional deep neural network 200; 320 and 330) and a plurality of prediction maps 410, 420, 430, and prediction maps may be generated. For example, the 2D deep neural network 200 may include at least one of a 2D fully convolutional network (FCN) and a 2D U-net. For example, the medical image includes a 2D medical image (eg, an x-ray image) and/or a 3D medical image (eg, a CT image, an MRI, or a PET image), and there is no particular limitation if it is a medical image. "Image" means a subject collected by computed tomography (CT), magnetic resonance imaging (MRI), ultrasound or any other medical imaging system known in the art. may be a medical image of The medical image 100 is voxel data and includes a plurality of slices, that is, a plurality of unit images. For example, the region of interest may be a region in which a tumor is disposed.

In one embodiment, as shown in Fig. 5, 2D U-net may correspond to the convolutional network structure proposed in "3D U-Net: Learning Dense Volumetric Segmentation from Sparse Annotation", MICCAI 2016. Example For example, a 2D U-Net may include a contracting path (40, contracting path) shown on the left and an expansive path (50, expansive path) shown on the right. Bar, this involves the iterative application of two 3x3 convolutions (unpadded convolutions), each of which has a rectified linear unit (ReLU) and a stride for downsampling. This is followed by a 2x2 max pooling operation of (stride) 2. For each downsampling step, the number of feature channels is doubled.Every step in the dilation path is an upsampling of the feature map. 2x2 convolution (“up-convolution”) that halves the sampling and subsequent number of feature channels, corresponding concatenation with the feature map from the cropped shrinkage path, and 2 It consists of 3x3 convolutions of 2 times, each of which is followed by a ReLU. The aforementioned truncation is necessary because of the loss of border pixels in every convolution. A 1x1 convolution is used to map each 64-component feature vector to a desired number of classes In this example neural network, 22 convolutional layers are included, which are arbitrary. So that the segmentation map that comes out as an output is connected neatly, One of ordinary skill in the art will appreciate that it is important to choose the input tile size so that all 2x2 max-pooling operations are applied to layers with an even number of x and y sizes.

In an embodiment, the plurality of prediction maps 410 , 420 , 430 , may have a plurality of channels although the resolution is lower than that of the input medical image. Through the plurality of generated prediction maps 410, 420, 430, prediction maps, the regions are shared in the region classification step and the object detection step to classify regions, and a region of interest (one organ among abdominal organs) can be detected. That is, by performing pixel labeling from a plurality of prediction maps 410 , 420 , 430 and prediction maps, regions of an image may be divided. In this case, pixel labeling may be performed by attaching the same number to all adjacent pixels having similar features of the input image. In addition, the pixel labeling result included in the plurality of label data 310 , 320 , and 330 may include probability distribution information for each pixel with respect to the class to be classified of the image.

In one embodiment, the generation of a plurality of label data (310, 320, 330) and a plurality of prediction maps (410, 420, 430, prediction map) may use a convolutional neural network (CNN). However, the present invention is not limited thereto and may vary depending on the purpose of use of the image processing apparatus, such as a deep neural network (DNN) or a recurrent neural network (RNN). Here, as an example, a plurality of label data 310 , 320 , 330 and a plurality of prediction maps 410 , 420 , 430 and prediction maps may be generated according to a 2D fully convolutional network (FCN).

In an embodiment, in operation 33, the pre-shape model generator uses the generated plurality of label data 310, 320, 330 and the plurality of prediction maps 410, 420, 430 as a distance probability map (500). By combining using , the pre-shape model 700 of the region of interest can be generated. For example, the prior shape model 700 may include 3D spatial shape information of the ROI in the form of a probability map. For example, the pre-shape model 700 may include information meaningful for classifying regions and detecting an object in the medical image 100 , and may include, for example, luminance, color, outline, etc. of the medical image 100 . .

In an embodiment, a maximum probability value voting method is applied to the distance probability map 500 , and the dictionary shape model 700 selects a maximum probability value from among a plurality of prediction maps having a reference probability value or more. It can be generated by combining a plurality of prediction maps having For example, the reference probability value may be 0.5, and the pre-shape model 700 may be generated by connecting components labeled with only the maximum probability values that are 0.5 or more. Since tumors vary in location and size, the margin of error can be reduced by using the maximum probability value rather than the average probability. Meanwhile, as shown in FIG. 6 , the probability value may mean a probability value close to a tumor.

In an embodiment, in operation 34 , the image editing unit may crop at least a portion of a region other than a region of interest in the 3D medical image 800 . For example, in order to reduce the operation process of the 3D deep neural network 900 , a portion of the boundary region in the 3D medical image 800 may be cropped. Specifically, for example, a boundary portion completely different from the brightness value information of the tumor and surrounding organs may be cropped using brightness value information.

In an embodiment, in operation 35 , the image input unit may input the 3D medical image 800 including the abdominal organs and the pre-shape model 700 to the 3D deep neural network 900 . For example, the 3D deep neural network 900 may include a 3D U-net. Alternatively, the cropped pre-shape model and the cropped 3D medical image 810 may be input to the 3D deep neural network 900 . On the other hand, the 3D U-net of the 3D deep neural network 900 overlaps with the description of FIG. 5 and thus is omitted.

In an embodiment, in operation 36 , the 3D deep learning unit may obtain the segmentation result 10 of the ROI through the 3D deep neural network 900 . Here, the segmentation result 10 may be an image in which the tumor corresponding to the region of interest is segmented.

That is, the present invention considers the entire context information of the region in which the region of interest is arranged through the two-dimensional deep neural network 200 and the pre-shape model 700, and uses the pre-shape model 700 to generate a 3D medical image. Through this, it is possible to increase the speed and accuracy of tumor segmentation by reducing the computation of understanding the contextual information of the region where the region of interest is located.

Meanwhile, the data set used to confirm the effect of the present invention includes 222 medical images obtained for radiation treatment in patients, of which 150 are used for learning, 38 are used for validation, and 34 are used for testing. split

In order to evaluate the accuracy of the division result (10), a Dice similarity coefficient (DSC) was calculated through Equation 1 and compared.

[Equation 1]

In this case, TP (True Positive) is the number of pixels in the auto-segmented region in the manually divided organ region, TN (True Negative) is the number of pixels in the non-auto-divided region in the non-manually divided organ region, FP (False Positive) is the number of pixels in an automatically segmented region other than the manually segmented organ region, and FN (False Negative) denotes the number of pixels in the non-automatically segmented region in the manually segmented organ region.

7 and 8, Class 1 is an isolated tumor, Class 2 is a tumor attached to the chest wall, Class 3 is a tumor attached to the mediastinum, and Class 4 is a tumor surrounded by surrounding structures in the upper and lower parts of the lung to be. As shown in FIG. 7 , it can be confirmed that the DSC of the coupling-Net (2.5D+3D with shape-enhanced prior) of the present invention is the highest at 70.71% when all classes are considered. Meanwhile, here, 2.5D may mean forming label data and a prediction map through a two-dimensional deep neural network.

In addition, as shown in FIG. 8 ((a) Original image, (b) 2D segmentation result (c) 2.5D segmentation result, (d) 3D segmentation result, (e) Coupling-net), red is ground-truth , blue means over segmentation, and green means under segmentation, so the coupling-Net of the present invention has the highest level of tumor segmentation in all classes, with the fewest green and blue and sharpest red borders. can confirm.

In a method for automatic tumor segmentation based on deep learning in a medical image according to an embodiment of the present invention, a plurality of axial images, a plurality of coronal images, and a plurality of sagittal ( Sagittal) by inputting the image into a two-dimensional deep neural network. generating a plurality of label data and a plurality of prediction maps of a region of interest through the two-dimensional deep neural network; generating a prior shape model of the ROI by combining the generated plurality of label data and the plurality of prediction maps using a distance probability map; and inputting the 3D medical image including the tumor and the pre-shape model into a 3D deep neural network. and obtaining a segmentation result of the region of interest through the 3D deep neural network.

According to various embodiments, by applying intensity normalization and spacing normalization to the 2D medical image, the plurality of axial images, the plurality of coronal images, and the plurality of sagittal images are applied. (Sagittal) may further include the step of acquiring an image.

According to various embodiments of the present disclosure, the method may include cropping at least a portion of a region other than the ROI in the 3D medical image; and inputting the cropped 3D medical image to the 3D deep neural network.

According to various embodiments, by inputting the pre-shape model and the cropped 3D medical image to the 3D deep neural network. and obtaining a segmentation result of the region of interest through the 3D deep neural network.

According to various embodiments, a maximum probability value voting method is applied to the distance probability map, and the dictionary shape model includes a plurality of prediction maps having a maximum probability value from among a plurality of prediction maps having a reference probability value or more. It can be generated by combining prediction maps.

According to various embodiments, the region of interest may be a region in which the tumor is disposed.

According to various embodiments, the pre-shape model may include 3D spatial shape information of the ROI in the form of a probability map.

According to various embodiments, the 2D deep neural network may include at least one of a 2D fully convolutional network (FCN) and a 2D U-net.

According to various embodiments, the 3D deep neural network may include a 3D U-net.

The steps of a method or algorithm described in relation to an embodiment of the present invention may be implemented directly in hardware, as a software module executed by hardware, or by a combination thereof. A software module may include random access memory (RAM), read only memory (ROM), erasable programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory, hard disk, removable disk, CD-ROM, or It may reside in any type of computer-readable recording medium well known in the art to which the present invention pertains.

As mentioned above, although embodiments of the present invention have been described with reference to the accompanying drawings, those skilled in the art to which the present invention pertains know that the present invention may be embodied in other specific forms without changing the technical spirit or essential features thereof. you will be able to understand Therefore, it should be understood that the embodiments described above are illustrative in all respects and not restrictive.

100: medical image
200: 2D deep neural network
310,320,330: label data
410,420,430: prediction map
700: pre-shape model
800: 3D medical image
900: 3D deep neural network

Claims (10)

In the automatic tumor segmentation method based on deep learning in medical images,
A plurality of axial images, a plurality of coronal images, and a plurality of sagittal images of a two-dimensional medical image containing a tumor are input to a two-dimensional deep neural network, and through the two-dimensional deep neural network generating a plurality of label data and a plurality of prediction maps of a region of interest;
generating a prior shape model of the ROI by combining the generated plurality of label data and the plurality of prediction maps using a distance probability map;
cropping a portion of a region other than the region of interest in the 3D medical image including the tumor; and
inputting the pre-shape model and a part of the cropped 3D medical image into a 3D deep neural network, and obtaining a segmentation result of the region of interest through the 3D deep neural network;
The cropping step is
In order to reduce the calculation process of the 3D deep neural network, a boundary portion of a portion of the remaining region that is completely different from brightness value information of the region of interest in the 3D medical image is cropped,
The pre-shape model includes information on luminance, color, and outline of the medical image, which is information meaningful for classifying regions and detecting an object in the medical image,
The segmentation result is an image in which the tumor corresponding to the region of interest is segmented,
The acquisition step is
The entire context information of the region in which the region of interest is disposed is considered through the two-dimensional deep neural network and the dictionary shape model, and contextual information of the region in which the region of interest is located through the 3D medical image using the dictionary shape model An automatic tumor segmentation method based on deep learning in a medical image, characterized in that the segmentation result is obtained by increasing the speed and accuracy of the tumor segmentation by reducing the operation to identify the tumor. According to claim 1,
Obtaining the plurality of axial images, the plurality of coronal images and the plurality of sagittal images by applying intensity normalization and spacing normalization to the two-dimensional medical image An automatic tumor segmentation method based on deep learning in a medical image, characterized in that it further comprises the step of: delete delete According to claim 1,
In the distance probability map, a maximum probability value voting method is applied,
The dictionary shape model is an automatic tumor segmentation method based on deep learning in a medical image, characterized in that it is generated by combining a plurality of prediction maps having a maximum probability value among a plurality of prediction maps having a reference probability value or more. According to claim 1,
The region of interest is an automatic tumor segmentation method based on deep learning in a medical image, characterized in that the region in which the tumor is disposed. According to claim 1,
wherein the pre-shape model includes the three-dimensional spatial shape information of the region of interest in the form of a probability map. According to claim 1,
The two-dimensional deep neural network is a 2D fully convolutional network (FCN) and a 2D U-net automatic tumor segmentation method based on deep learning in a medical image, characterized in that it comprises at least one. According to claim 1,
The 3D deep neural network is an automatic tumor segmentation method based on deep learning in a medical image, characterized in that it includes a 3D U-net. In combination with a computer that is hardware, stored in a medium to execute the method of claim 1, an automatic tumor segmentation program based on deep learning in a medical image.

Images