KR102591587B1 - Medical image segmentation apparatus and method for segmentating medical image - Google Patents

Abstract

A medical image segmentation apparatus and a medical image segmentation method are provided. The medical image segmentation method includes acquiring different types of medical images of abdominal organs, inputting the different types of medical images into a two-dimensional convolutional neural network to generate a probability map of specific organ information for each medical image. Outputting and performing localization of a specific organ through a first majority vote on the obtained plurality of specific organ information probability maps, converting the plurality of localized medical images into 3D medical images, and converting the plurality of converted medical images into 3D medical images. Inputting 3D medical images into a 3D convolutional neural network and outputting a probability map of the average value and a probability map of uncertainty for a plurality of 3D medical images, respectively, and the probability of the average value of the plurality of 3D medical images. It includes performing cooperative learning using a map and probability map of uncertainty, and dividing a specific organ through a second majority vote on the cooperatively learned result.

Description

Medical image segmentation apparatus and medical image segmentation method {MEDICAL IMAGE SEGMENTATION APPARATUS AND METHOD FOR SEGMENTATING MEDICAL IMAGE}

The present invention relates to a medical image segmentation device and a medical image segmentation method. More specifically, it relates to an automatic pancreas segmentation method based on hierarchical networks and cooperative learning that takes into account uncertainty of the pancreas in abdominal CT images, and a device implementing the same.

According to the 2017 National Cancer Registry statistics published by the Ministry of Health and Welfare and the Central Cancer Registry in 2019, pancreatic cancer ranked 8th with an incidence rate of 3% among all cancer types, and the incidence rate has been continuously increasing over the past 10 years.

Pancreatic cancer has no characteristic symptoms from the beginning to the advanced stage, the tumor growth rate is fast, and it is easy to metastasize to surrounding tissues. In most cases, surgical resection is performed to increase the long-term survival rate. Before pancreatic cancer resection, prior information on the shape of the pancreas is important to identify the pancreas and detect pancreatic cancer, so pancreas segmentation in abdominal CT images is an essential preliminary step.

Figure 3 shows the characteristics of the pancreas in an abdominal CT image. As shown in (a) of Figure 3, the pancreas has a brightness value similar to surrounding organs such as the stomach, liver, spleen, small intestine, and portal vein, and (b) of Figure 3 Likewise, the shape of the pancreas seen in each image slice is different, and as shown in Figure 3 (c), the shape of the pancreas varies from patient to patient.

There are various studies based on deep convolutional neural networks (CNN), but they have common limitations in that under-segmentation and over-segmentation problems occur in the uncertain border area of the pancreas.

The problem to be solved by the present invention is to provide a medical image segmentation apparatus and method capable of segmenting a specific device in a medical image in an optimized manner.

The problem to be solved by the present invention is to effectively detect the location of the pancreas in abdominal CT images through a localization step based on a deep convolutional neural network, and improve the segmentation accuracy of the pancreas by considering information about uncertain areas within the area. The aim is to provide a medical image segmentation apparatus and method capable of doing so.

The problems to be solved by the present invention are not limited to the problems mentioned above, and other problems not mentioned can be clearly understood by those skilled in the art from the description below.

A medical image segmentation method according to one aspect of the present invention for solving the above-described problem is a method performed by an apparatus, comprising: acquiring different types of medical images in which abdominal organs are imaged; Input medical images into a two-dimensional convolutional neural network to output a probability map of specific organ information for each medical image, and perform localization of specific organs through first majority voting on the obtained probability maps of multiple specific organ information. Step: Converting a plurality of localized medical images into 3D medical images, inputting the converted 3D medical images into a 3D convolutional neural network, and generating a probability map of the average value of the plurality of 3D medical images. and outputting a probability map of uncertainty, respectively, and performing collaborative learning using the probability map of the average value and the probability map of uncertainty for a plurality of 3D medical images, through a second majority vote on the cooperatively learned results. It involves dividing specific organs.

In an embodiment, the different types of medical images include two-dimensional medical images in a transverse plane, a coronal plane, and a sagittal plane.

In an embodiment, the method further includes performing data preprocessing on the different types of medical images, and performing the data preprocessing includes brightness value normalization and spatial normalization on the different types of medical images. It is characterized by performing normalization.

In an embodiment, the spatial normalization is characterized by dividing the minimum pixel value from the pixel size of the original image and converting the image using nearest interpolation.

In an embodiment, the 2D convolutional neural network and the 3D convolutional neural network are characterized by being composed of a contraction path and an expansion path.

In an embodiment, the contraction path sequentially performs convolution, batch normalization, activation function, and max pooling on the input medical images.

In an embodiment, in the expansion path, up-convolution and concatenation operations are performed to increase the size of the image reduced in the contraction path and to reduce the number of channels increased in the contraction path.

In an embodiment, the different types of medical images are three-section image information, and the step of performing the localization involves localizing the specific organ using spatial information about the three-section image information. It is characterized by:

In an embodiment, the plurality of 3D medical images input to the 3D convolutional neural network are images obtained by converting different types of localized medical images into 3D medical images.

In an embodiment, the output step involves inputting a plurality of 3D medical images into a dropout layer to produce N dropout results (N is a natural number) of the same size as the image input to the 3D convolutional neural network. It is characterized by outputting and extracting a probability map of the average value and a probability map of uncertainty using the dropout result.

In an embodiment, the step of segmenting the specific organ reduces the uncertain region that occurs when segmenting a specific organ in a medical image based on the probability map of uncertainty, and improves segmentation accuracy in the uncertain region through cooperative learning. It is characterized by increasing .

A medical image segmentation device according to another embodiment of the present invention includes a medical image acquisition unit that acquires different types of medical images of abdominal organs, and inputs the different types of medical images into a two-dimensional convolutional neural network to A control unit outputting a specific organ information probability map for medical images and performing localization of a specific organ through a first majority vote on the obtained plurality of specific organ information probability maps, wherein the control unit performs localization. Convert the plurality of medical images into 3D medical images, input the converted 3D medical images into a 3D convolutional neural network, and create a probability map of the average value and a probability map of uncertainty for the plurality of 3D medical images. Output each, perform cooperative learning using the probability map of the average value and the probability map of uncertainty for a plurality of 3D medical images, and segment specific organs through a second majority vote on the cooperatively learned results. It is characterized by

In an embodiment, the specific organ includes a pancreas, and the different types of medical images include two-dimensional medical images of a transverse section, a coronal section, and a sagittal section.

In an embodiment, the control unit performs data preprocessing on the different types of medical images, and performs brightness value normalization and spatial normalization on the different types of medical images.

In an embodiment, the spatial normalization is characterized by dividing the minimum pixel value from the pixel size of the original image and converting the image using nearest interpolation.

In an embodiment, the 2D convolutional neural network and the 3D convolutional neural network are characterized by being composed of a contraction path and an expansion path.

In an embodiment, the contraction path sequentially performs convolution, batch normalization, activation function, and max pooling on the input medical images.

In an embodiment, in the expansion path, up-convolution and concatenation operations are performed to increase the size of the image reduced in the contraction path and to reduce the number of channels increased in the contraction path.

In an embodiment, the different types of medical images are three-section image information, and the control unit localizes the specific organ using spatial information about the three-section image information. .

In an embodiment, the plurality of 3D medical images input to the 3D convolutional neural network are images obtained by converting different types of localized medical images into 3D medical images.

In an embodiment, the control unit inputs a plurality of 3D medical images into a dropout layer and outputs N dropout results (N is a natural number) of the same size as the image input to the 3D convolutional neural network. , Characterized in extracting a probability map of the average value and a probability map of uncertainty using the dropout result.

In an embodiment, the control unit reduces the uncertain region that occurs when segmenting a specific organ in a medical image based on the probability map of uncertainty and increases segmentation accuracy in the uncertain region through cooperative learning. Do it as

In addition to this, another method for implementing the present invention, another system, and a computer-readable recording medium recording a computer program for executing the method may be further provided.

Other specific details of the invention are included in the detailed description and drawings.

According to the present invention, a new medical image that can automatically extract the location information of the pancreas using a 2.5-dimensional segmentation network and then accurately segment the pancreas using a 3-dimensional segmentation network that takes into account information about uncertain areas in the localized area. A partitioning method can be provided.

The effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description below.

1 is a conceptual diagram illustrating a medical image segmentation apparatus according to an embodiment of the present invention.
Figure 2 is a flowchart explaining a medical image segmentation method according to an embodiment of the present invention.
Figure 3 is a conceptual diagram for explaining the characteristics of the pancreas in an abdominal CT image, which is one of the medical images.
FIGS. 4A, 4B, 4C, 5, 6, and 7 are conceptual diagrams for explaining the medical image segmentation method shown in FIG. 2.

The advantages and features of the present invention and methods for achieving them will become clear by referring to the embodiments described in detail below along with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below and may be implemented in various different forms. The present embodiments are merely provided to ensure that the disclosure of the present invention is complete and to provide a general understanding of the technical field to which the present invention pertains. It is provided to fully inform the skilled person of the scope of the present invention, and the present invention is only defined by the scope of the claims.

The terminology used herein is for describing embodiments and is not intended to limit the invention. As used herein, singular forms also include plural forms, unless specifically stated otherwise in the context. As used in the specification, “comprises” and/or “comprising” does not exclude the presence or addition of one or more other elements in addition to the mentioned elements. Like reference numerals refer to like elements throughout the specification, and “and/or” includes each and every combination of one or more of the referenced elements. Although “first”, “second”, etc. are used to describe various components, these components are of course not limited by these terms. These terms are merely used to distinguish one component from another. Therefore, it goes without saying that the first component mentioned below may also be a second component within the technical spirit of the present invention.

Unless otherwise defined, all terms (including technical and scientific terms) used in this specification may be used with meanings commonly understood by those skilled in the art to which the present invention pertains. Additionally, terms defined in commonly used dictionaries are not interpreted ideally or excessively unless clearly specifically defined.

Hereinafter, embodiments of the present invention will be described in detail with reference to the attached drawings.

Prior to explanation, the meaning of terms used in this specification will be briefly explained. However, since the explanation of terms is intended to aid understanding of the present specification, it should be noted that if it is not explicitly described as limiting the present invention, it is not used in the sense of limiting the technical idea of the present invention.

In this specification, 'medical image segmentation device' includes all various devices that can perform computational processing and provide results to the user.

For example, 'medical image segmentation device' is used not only for computers, desktop PCs, and laptops (Note Books), but also for smart phones, tablet PCs, cellular phones, and PCS phones (Personal Communication). Service phone), synchronous/asynchronous IMT-2000 (International Mobile Telecommunication-2000) mobile terminal, Palm PC (Palm Personal Computer), personal digital assistant (PDA), etc. may also be applicable.

Additionally, the 'medical image segmentation device' can receive requests from clients and communicate with a server that performs information processing.

A medical image segmentation apparatus according to an embodiment of the present invention may be implemented to include at least one of the components described in FIG. 1 .

1 is a conceptual diagram illustrating a medical image segmentation apparatus according to an embodiment of the present invention.

Referring to FIG. 1 , the medical image segmentation apparatus 100 of the present invention may include a medical image acquisition unit 110 that acquires a medical image in which abdominal organs are imaged.

Here, the medical image of abdominal organs may include an abdominal CT (Computed Tomography) image.

Specifically, the medical image acquisition unit 110 may acquire (collect, photograph) different types of medical images in which abdominal organs are imaged.

Here, the different types of medical images may include two-dimensional medical images (two-dimensional abdominal CT images) in the transverse plane, coronal plane, and sagittal plane.

The medical image acquisition unit 110 may include an imaging device configured to capture a medical image or a communication unit configured to receive the captured medical image from an external device.

The memory 120 may be configured to store data, images, various information, and application programs created/managed by the medical image acquisition unit 110 and the control unit 130.

The memory 120 is electrically connected to the control unit 130. The memory 120 can store basic data for the unit, control data for controlling the operation of the unit, and input/output data. In terms of hardware, the memory 120 may be a variety of storage devices such as ROM, RAM, EPROM, flash drive, hard drive, etc. The memory 120 may store various data for the overall operation of the medical image segmentation device 100, such as programs for processing or controlling the control unit 130.

In addition to operations related to application programs, the control unit 130 typically controls the overall operation of the medical image segmentation device 100. The control unit 130 can provide or process appropriate information or functions to the user by processing signals, data, information, etc. input or output through the components discussed above, or by running an application program stored in memory.

Hereinafter, a method for segmenting specific organs in a medical image will be examined in more detail with reference to the attached drawings.

Figure 2 is a flowchart explaining a medical image segmentation method according to an embodiment of the present invention.

Figure 3 is a conceptual diagram for explaining the characteristics of the pancreas in an abdominal CT image, which is one of the medical images, and Figures 4a, 4b, 4c, 5, 6, and 7 illustrate the medical image segmentation method seen in Figure 2. This is a concept diagram for doing this.

Referring to FIG. 2, the control unit 130 of the medical image segmentation device may control the medical image acquisition unit 110 to acquire different types of medical images (abdominal CT images) captured by an abdominal device (S210). ).

The control unit 130 inputs the different types of medical images into a two-dimensional convolutional neural network, outputs a specific organ information probability map for each medical image, and generates a first probability map for the plurality of acquired specific organ information probability maps. Localization of specific organs can be performed through majority voting (S220).

The control unit 130 converts a plurality of localized medical images into 3D medical images, inputs the converted 3D medical images into a 3D convolutional neural network, and calculates the average value of the plurality of 3D medical images. The probability map of and the probability map of uncertainty can be output respectively (S230).

The control unit 130 performs cooperative learning using a probability map of the average value and a probability map of uncertainty for a plurality of 3D medical images, and divides a specific organ through a second majority vote on the cooperatively learned results. (S240).

Hereinafter, for convenience of explanation, medical images and abdominal CT images will be used interchangeably, and the specific organ will be described as an example of the pancreas.

As shown in Figure 3, the pancreas has a similar brightness value to surrounding organs such as the stomach, liver, spleen, small intestine, and portal vein, as shown in (a), and the shape of the pancreas visible in each image slice is different, as shown in (b). As shown in (c), the shape of the pancreas varies from patient to patient.

Accordingly, the present invention can provide a method for segmenting the pancreas in an optimized manner from an abdominal CT image.

The control unit 130 of the medical image segmentation device may perform automatic pancreas segmentation in the abdominal CT image.

Referring to FIG. 4A, the medical image segmentation apparatus of the present invention can perform automatic pancreas localization using network input image preprocessing and a two-dimensional segmentation network based on a deep convolutional neural network. Additionally, the medical image segmentation device can perform automatic pancreas segmentation using a 3D segmentation network that also considers information about uncertain regions.

Meanwhile, the present invention can quantify the degree of uncertainty that may occur in an image, apply it to a segmentation model, and improve segmentation performance through cooperative learning.

For example, as shown in FIG. 4B, the present invention quantifies uncertainty using a 3D U-net (3D segmentation network) that considers the uncertainty of a specific device to quantify information about the uncertain area in the image. , this can be reflected in the segmentation model (i.e., automatic pancreas segmentation). In addition, as shown in FIG. 4C, the present invention provides information on specific organs (e.g., axial plane, coronal plane, sagittal plane) for different types of images (eg, axial plane, coronal plane, sagittal plane). For example, uncertainty about the location of the pancreas can be quantified, and automatic pancreas segmentation can be performed for each image through co-training's prediction.

Through this, the present invention can improve segmentation performance by quantifying uncertainty and performing cooperative learning.

First, let's look at the data preprocessing process as follows.

The controller 130 may perform data preprocessing on different types of acquired medical images.

Differences between data resulting from differences in imaging equipment and imaging protocols for abdominal CT images may reduce the segmentation performance of the network.

Accordingly, the control unit 130 performs brightness value normalization and spatial normalization to reduce differences between abdominal CT images before performing automatic segmentation.

That is, as seen above, the controller 130 can perform brightness value normalization and spatial normalization on different types of two-dimensional medical images, namely, the transverse, coronal, and sagittal planes.

In order to match the range between the brightness values of images with different distributions in abdominal CT images, the brightness values of the images are normalized to between 0 and 255 through Equation 1.

At this time, I_org is the brightness value of the original image, I_max and I_min are the minimum and maximum threshold values, respectively. The minimum threshold is set to include the brightness value of the background area, and the maximum threshold is set to include the brightness value of the bone area. . The values corresponding to the threshold can be calculated as -174HU and 326HU, respectively, through experiment.

To reduce the difference in pixel space range between patients, spatial normalization of the experimental data set is performed from the pixel spacing range of the experimental data, 0.6641mm to 0.9766mm, to 0.6641mm, which corresponds to the minimum pixel size.

Spatial normalization divides the minimum pixel value from the pixel size of the original image and transforms the image using nearest interpolation.

Thereafter, the control unit 130 may perform automatic pancreas localization based on the three-section 2.5D U-NET.

The U-net refers to a convolutional neural network developed for biomedical image segmentation and includes a two-dimensional convolutional neural network (2D U-net) and a three-dimensional convolutional neural network (3D U-net). .

As shown in FIG. 5, U-net may have a network shape similar to that of the alphabet U.

U-net is a segmentation network that shows excellent effectiveness in biomedical image segmentation.

While the existing CNN was mainly used for simple classification, U-net has the advantage of being capable of not only classification but also segmentation.

In addition, U-net has the advantage of being able to solve the problems of the existing segmentation method by using an overlap-tile strategy during segmentation.

The overlap tile strategy refers to cutting large-sized images into patches and inputting them as input.

Additionally, U-net has the advantage of not falling into a trade-off depending on patch size.

In general, when using a large-sized patch, it has a great effect on context recognition by recognizing a wide range of images at once, but localization accuracy is reduced, and when using a small-sized patch, localization accuracy is reduced. increases, but the context that can be recognized becomes narrower.

However, in U-net's network structure, context recognition is performed on the contracting path and localization is performed on the expanding path, resulting in trade-offs depending on patch size ( It has the advantage of not falling into a trade-off.

The convolutional neural network described in this specification may refer to a deep convolutional neural network (DCNN) and may be the U-net described above.

The convolutional neural network, that is, U-net, will be described with reference to FIG. 5 as follows.

In a U-net network, the contracting path (red box) corresponding to the convolution encoder and the expanding path (expanding path) corresponding to the convolution decoder can be symmetrical around the center.

When performing upsampling in the expansion path, the U-net network copies and crops the features of the contraction path for accurate localization and concatenation. A structure that does this can be achieved.

Specifically, the contracting path has a convolution network structure, uses the ReLU function as an activation function, reduces the size using 2x2 max pooling, and moves on to the next block.

The U-net network uses twice as many channels as the previous number of feature channels to extract features after moving to the next block.

The Expanding Path uses an Up-convolution block to increase the size reduced during the contracting process.

You can use 2x2 up-convolution and use ReLU as the activation function.

The U-net network can use channels twice as small as the number of previous feature channels for localization after moving to the next block.

In addition, the U-net network has a structure that performs image compensation processing and more accurate localization by copying and cropping the features of the contracting path and concatenating them.

The U-net network can classify images into two classes using 1x1 convolution in the last final layer.

The medical image segmentation apparatus and method of the present invention can automatically segment the pancreas from an abdominal CT image using a convolutional neural network including the U-net described above.

Conventionally, the area occupied by the pancreas in an abdominal CT image is small, so there is a limit to under-segmentation of the pancreas when pancreas segmentation is performed using the entire area of the abdominal CT image.

Therefore, it is necessary to effectively detect the location of the pancreas in abdominal CT images and then provide pancreatic region information that can focus on regional context information in the segmentation step.

The medical image segmentation device and method of the present invention is a method of extracting pancreas location information using a 2-dimensional U-Net-based segmentation network using abdominal CT images that can utilize 2.5-dimensional plan information that can infer the spatial shape of the pancreas. can be provided.

The control unit 130 inputs the preprocessed medical images into a convolutional neural network (e.g., U-net network) for biomedical image segmentation, and selects a specific organ (e.g., a specific organ) from the convolutional neural network for the input medical images. For example, a probability map of shape information of the pancreas can be output.

Referring to stage 1 of FIG. 4B, the control unit 130 inputs the different types of medical images into a two-dimensional convolutional neural network, outputs a probability map of specific organ information for each medical image, and outputs a probability map of specific organ information for each medical image. The location of a specific organ can be performed through a first majority vote on the probability map of specific organ information. The control unit 130 performs data preprocessing on the input image, transverse section (axial plane) and coronal plane with 512 × 512 resolution. Two-dimensional abdominal CT images in the coronal plane and sagittal plane can be used.

The convolutional neural network may be a two-dimensional U-net.

The convolutional neural network may be composed of a contraction path and an expansion path.

In the contraction path, convolution, batch normalization, activation function performance, and max pooling can be sequentially performed on the input medical image.

Specifically, a 2D U-Net can be composed of a contracting path and an expanding path.

The control unit 130 performs two 3×3 convolutions, batch normalization (BN), and rectified linear unit (ReLU) functions on the shrinkage path, and then performs 2×2 max pooling (max) with a stride of 2. The step of sequentially performing -pooling can be repeated six times.

Through 2×2 max pooling performed at each stage of the contraction path, the size of the feature map is reduced by half and the number of channels in the feature map is doubled.

In the expansion path, up-convolution and concatenation operations may be performed to increase the size of the image reduced in the contraction path and reduce the number of channels increased in the contraction path.

Specifically, the control unit 130 performs a 2×2 up-convolution on the expansion path and then performs a skip connection to combine the last feature map of the expansion path and the equivalent contraction path. And the steps of performing two 3×3 convolutions, batch normalization, and the ReLU function can be repeated six times.

Through the 2×2 up convolution performed at each stage of the expansion path, the size of the feature map is doubled and the number of channels in the feature map is reduced by half.

The control unit 130 finally performs 1×1 convolution and outputs a probability map of pancreas shape information of the same size as the input image. Probability maps have probability values between 0 and 1.

The control unit 130 may localize the specific organ by applying first majority voting to the input medical images.

Specifically, the control unit 130 performs majority voting (MV) on N (N is a natural number) (e.g., three) segmentation results obtained using input images of the transverse, coronal, and sagittal planes. ) technique can be used to localize the pancreas.

The control unit 130 may set the positioning area to include a blank space to reduce positioning errors due to underfitting that occurs in the majority voting results.

At this time, the value corresponding to the margin is set to 30 pixels for each axis through experiment. Figure 6 is the result of automatic pancreas localization in the abdominal CT image, and Figure 6(a), Figure 6(b), and Figure 6(c) are the localization results when only transverse, coronal, and sagittal input images are used, respectively. , and Figure 6(d) shows the localization results when using the multiple voting method for three-section image information.

When localization is performed using only two-dimensional plane information of one cross-section, low localization accuracy is shown due to incorrect division into other areas caused by not considering spatial information. However, the present invention shows low localization accuracy in FIG. 6(d). As shown, accurate localization to the pancreas region can be performed by additionally considering spatial information using a majority voting method.

Thereafter, the medical image segmentation apparatus and method of the present invention can provide a method for automatically segmenting the pancreas by taking uncertainty into account.

Referring to Stage 2 of FIG. 4B, the controller 130 of the present invention converts a plurality of located medical images into 3D medical images and inputs the converted 3D medical images into a 3D convolutional neural network. Thus, a probability map of the average value and a probability map of uncertainty for a plurality of 3D medical images can be output, respectively.

In addition, the control unit 130 performs cooperative learning using a probability map of the average value and a probability map of uncertainty for a plurality of 3D medical images, and selects a specific organ through a second majority vote on the cooperatively learned results. It can be divided.

The different types of medical images described above may be 3D medical images.

At this time, the convolutional neural network may be a 3D convolutional neural network (i.e., 3D U-net).

The plurality of 3D medical images input to the 3D convolutional neural network may be images obtained by converting different types of localized medical images into 3D medical images. That is, the plurality of 3D medical images may include a 3D image of a positioned transverse section, a 3D image of a located coronal plane, and a 3D image of a located sagittal plane.

Compared to the prior art of FIG. 4A, the present invention, referring to FIG. 4B, does not input only one 3D medical image into a 3D convolutional neural network (or dropout layer), but instead inputs a plurality of 3D medical images for cooperative learning. Medical images can be input into a 3D convolutional neural network (or dropout layer), and a probability map of the average value and a probability map of uncertainty can be output for each 3D medical image.

Specifically, the control unit 130 inputs a plurality of 3D medical images into a dropout layer and outputs N dropout results (N is a natural number) of the same size as the image input to the 3D convolutional neural network. , a probability map of the average value and a probability map of uncertainty can be extracted using the dropout result. Referring to FIG. 4c, the control unit 130 of the present invention controls each 3D medical image (Axial plane, Coronal plane). Collaborative learning can be performed using the probability map of the average value and the probability map of uncertainty calculated for the sagittal plane.

Using the probability map of the average value and the probability map of uncertainty extracted from each 3D medical image, a prediction value for each 3D medical image is calculated, and cooperative learning is performed using this to obtain a cooperative learning prediction value (Co -trainins' prediction) can be calculated.

Based on the probability map of uncertainty, the control unit 130 can reduce the uncertain region that occurs when segmenting a specific organ in a medical image and increase segmentation accuracy in the uncertain region through cooperative learning.

Specifically, the present invention can provide a 3D U-Net-based automatic pancreas segmentation method that takes uncertainty into account.

When segmenting the pancreas using a 2D network that does not consider spatial information, the shape of the pancreas visible in each slice is different, resulting in under-segmentation in the head or tail of the pancreas and lower segmentation accuracy at the border of the pancreas. there is.

Additionally, when the pancreas is divided into a 3D network, leakage occurs into surrounding organs that have a similar shape to the pancreas, and under- or over-segmentation occurs at the uncertain border of the pancreas.

Therefore, the present invention considers spatial information around the pancreas, prevents leakage to surrounding organs with similar shapes, and prevents under-segmentation and over-segmentation in uncertain areas of the pancreas caused by the positional and morphological diversity of the pancreas. To prevent this, the pancreas can be segmented using a 3D segmentation network that also considers information about uncertain regions.

For this purpose, the medical image segmentation device of the present invention can use a 3D abdominal CT image converted to 96×96×96 resolution from a localized abdominal CT image using a 2D U-Net-based network as an input image. .

A 3D U-Net can be composed of a contraction path and an expansion path in the same way as a 2D U-Net.

In the contraction path, the steps of performing two 3×3×3 convolutions, batch normalization, and the ReLU function, followed by sequentially performing 2×2×2 max pooling with a stride of 2, are repeated four times.

Through 2×2×2 max pooling performed at each stage of the contraction path, the size of the feature map is reduced by half and the number of channels in the feature map is doubled.

In the expansion path, a 2×2×2 up-convolution is performed, followed by a concatenation operation that combines the last feature maps of the expansion path and the equivalent contraction path, followed by two 3×3×3 convolutions, batch normalization, and the ReLU function. Repeat the steps four times.

In order to reduce the uncertain area that occurs when dividing the pancreas, a dropout layer is added to the input before performing a final 1×1×1 convolution and outputting a 3D pancreas shape probability map of the same size as the input image. Output N dropout results (N is a natural number) of the same size as the image. The output N dropout results (N is a natural number) are used to output a probability map of the average value and a probability map of uncertainty extracted from each 3D medical image.

The probability map of the average value can be derived through formula (1) in FIG. 4C, and the probability map of uncertainty can be derived through formula (2) in FIG. 4C.

Using the probability map of the average value and the probability map of uncertainty extracted from each 3D medical image, a prediction value (formula (3) in Figure 4c) is calculated for each 3D medical image, and this is used to cooperate By performing learning, a cooperative learning prediction value (Co-trainings' prediction) (equation (4) in FIG. 4C) can be calculated.

Through this configuration, the present invention not only takes into account local context information and spatial information when segmenting the pancreas using a 3D segmentation network that takes uncertainty into account, but also reduces the problems of under-segmentation and over-segmentation that occur in uncertain areas. there is.

The present invention can provide a D CNN-based automatic pancreas segmentation method that takes into account uncertain regions due to the positional and morphological variability of the pancreas in abdominal CT images.

According to the present invention, a 2.5D network based on three orthogonal planar images can automatically locate the pancreas of the abdominal organ.

Additionally, according to the present invention, the 3D network considering the uncertainty of the present invention can be reduced through segmentation in the uncertain region of the pancreas.

Additionally, the 3D network considering joint training of the present invention can improve segmentation accuracy in uncertain regions of the pancreas.

In other words, the present invention can provide an automatic pancreas segmentation method using a stepped network that takes into account uncertainty of the pancreas in abdominal CT images, and can provide a 2D ensemble network that can accurately position the pancreas by considering information in three orthogonal planes. there is. Additionally, our 3D network can consider local spatial information, pancreatic uncertainty, and joint training to reduce under- and over-segmented areas.

The following experiments and result analysis were performed for this configuration.

The data used in the experiment were portal-venous abdominal CT images of a total of 82 patients distributed by the National Institutes of Health Clinical Center (NIH). The age of each patient was 18 to 76 years old. It consists of 53 men and 27 women. All data were obtained from 3D cross-sectional abdominal CT images taken by Phillips and Siemens at a tube voltage of 120 kVp. The number of slices in the abdominal CT images of a total of 82 people was 181~466, the image resolution was 512×512, the pixel size was 0.6641~0.9766mm2, and the slice interval was 0.5mm~1.0mm.

The environment for the experiment was a PC equipped with an Inter(R) Core(TM) i7-8700 3.20Hz CPU, 32GB memory, and an NVIDIA GeForce RTX 2080Ti graphics card under the Windows 10 (64-bit) operating system. This proposed method was implemented using Python-based deep learning libraries Tensorflow and Keras, and 2D and 3D U-Net were used by modifying open source packages. The learning rate used to learn the 2D segmentation network was 0.0001 and the batch size was 1, and for optimization, a total of 100 iterative learning was performed using the Adam optimizer. The loss function used cross-entropy. The initial value of the learning rate used for learning the 3D segmentation network was 0.0001, and if the loss function of the validation data set did not decrease, it was decreased by 0.8 times. The batch size was 1, and the Adam optimizer was used for optimization. The experiment was conducted using a hold-out cross-validation method. For the experiment, the data set was composed of 52 training data sets and 16 test data sets, and was divided into 14 validation data sets to prevent overfitting.

Quantitative and qualitative evaluations were performed to evaluate the performance of this proposed method. For quantitative evaluation, DSC, sensitivity (recall), and positive predictive value (Precision) between the manual segmentation results, comparison method, and proposed method results were measured as shown in Equation 2.

At this time, TP is the area automatically divided into the pancreas from the area manually divided into the pancreas, TN is the area automatically divided from the area manually segmented into the non-pancreas to the non-pancreas area, and FP is the area automatically divided into the non-pancreas area. The area is automatically segmented from the manually segmented area into the pancreas, and FN refers to the area automatically segmented from the area manually segmented into the pancreas to the non-pancreas area.

As a comparison method, the pancreas segmentation methods using transverse, coronal, and sagittal planes in a two-dimensional U-Net structure are Method A, Method B, and Method C, respectively, and method C is used to segment the pancreas using transverse, coronal, and sagittal planes in a two-dimensional U-Net structure. Comparative analysis of the performance of the majority voting method (Method D) of the pancreas segmentation results using , the pancreas segmentation method (Method E) using a 3D input image located through the majority voting method, and the proposed method (Method F) considering uncertainty. did.

Table 1 shows the quantitative evaluation results of pancreatic segmentation results.

Method A, Method B, and Method C, which trained a 2D segmentation network using cross-sectional images, showed overall low performance with an average DSC of 74.41%. On the other hand, in the case of Method D, 2.5-dimensional spatial information is inferred by combining 2-dimensional plane information of a single cross-section and by using a majority voting method, the over-segmented area is reduced, resulting in a sensitivity of 1.94% compared to Method A, Method B, and Method C, respectively. p, 5.58%p, and 3.42%p improved performance, but showed a tendency for under-segmentation, showing the highest positive predictive value. In the case of Method E, sensitivity increased by 3.29%p by preventing undersegmentation by considering spatial information in a limited area compared to Method D, but the positive predictive value decreased by 8.19%p due to oversegmentation at the uncertain border of the pancreas. It was confirmed that when Method F was applied, the DSC, sensitivity, and positive predictive value increased by 3.95%p, 2.83%p, and 4.13%p, respectively, compared to Method E by considering information about the uncertain area.

Figure 7 shows the segmentation results by comparison method according to the location of the pancreas in the abdominal CT image. For qualitative evaluation of this proposed method, the results of the comparative method and the proposed method were compared with the manual results, respectively.

In Figure 7, (a) to (g) show qualitative evaluations corresponding to Method A to Method G, respectively, where red indicates the area overlapping with the manual segmentation result, blue indicates the over-segmented area, and green indicates the under-segmented area.

In Method A, Method B, and Method C, leakage occurred in surrounding areas with similar shapes and brightness values, such as the chest, liver, spleen, and small intestine, and undersegmentation occurred in the head and tail of the pancreas. In the case of Method D, leakage to surrounding organs arising from segmentation results in the transverse, coronal, and sagittal planes was resolved through majority voting, but it was confirmed that undersegmentation still existed in the tail of the pancreas. In the case of Method E, leakage to surrounding organs was reduced using spatial information within a limited area, but it was confirmed that there was leakage into the small intestine, which has a shape similar to the pancreas, and that the pancreas was under- and over-segmented at the uncertain border. In the case of Method F, it was confirmed that it was most similar to the manual segmentation results by preventing leakage to surrounding organs that occurred in Method A, Method B, and Method C and dividing the under-segmented area in Method E.

Through these experiments and configurations, the present invention provides a method of automatically extracting the location information of the pancreas using a 2.5-dimensional segmentation network and then accurately segmenting the pancreas using a 3-dimensional segmentation network that takes into account information about uncertain areas in the located area. can be provided.

In addition, the present invention uses a two-dimensional segmentation network to obtain N segmentation results (N is a natural number) obtained from abdominal CT images in the transverse, coronal, and sagittal planes in order to obtain accurate location information of the pancreas in the abdominal CT image. It was improved through majority voting, and by changing the weight of the loss function according to uncertainty, it is possible to overcome under-segmentation in uncertain areas caused by morphological characteristics of the pancreas in localized 3D abdominal CT images.

In addition, the present invention showed a DSC of 83.50% between the pancreas segmentation results of the proposed method and the manual segmentation results compared to the comparative method, and it can be confirmed that accurate segmentation of the pancreas is possible compared to other comparative methods.

The steps of the method or algorithm described in connection with embodiments of the present invention may be implemented directly in hardware, implemented as a software module executed by hardware, or a combination thereof. The software module may be RAM (Random Access Memory), ROM (Read Only Memory), EPROM (Erasable Programmable ROM), EEPROM (Electrically Erasable Programmable ROM), Flash Memory, hard disk, removable disk, CD-ROM, or It may reside on any type of computer-readable recording medium well known in the art to which the present invention pertains.

Above, embodiments of the present invention have been described with reference to the attached drawings, but those skilled in the art will understand that the present invention can be implemented in other specific forms without changing its technical idea or essential features. You will be able to understand it. Therefore, the embodiments described above should be understood in all respects as illustrative and not restrictive.

Claims (23)

In a method performed by a device,
Obtaining different types of medical images of abdominal organs;
By inputting the different types of medical images into a two-dimensional convolutional neural network, a specific organ information probability map for each medical image is output, and a specific organ is selected through a first majority vote on the obtained plurality of specific organ information probability maps. performing positioning;
Convert multiple localized medical images into 3D medical images, input the multiple converted 3D medical images into a 3D convolutional neural network, and create a probability map of the average value and uncertainty of the multiple 3D medical images. Outputting each probability map; and
It includes performing cooperative learning using a probability map of the average value and a probability map of uncertainty for a plurality of 3D medical images, and dividing a specific organ through a second majority vote on the cooperatively learned result,
The output step is,
Input a plurality of 3D medical images into a dropout layer to output N dropout results (N is a natural number) of the same size as the image input to the 3D convolutional neural network, and use the dropout results to calculate the average value. A medical image segmentation method characterized by extracting a probability map of and a probability map of uncertainty. According to claim 1,
A medical image segmentation method, wherein the different types of medical images include two-dimensional medical images in the transverse section, coronal section, and sagittal section. According to claim 1,
Further comprising performing data preprocessing on the different types of medical images,
The step of performing the data preprocessing is,
A medical image segmentation method characterized by performing brightness value normalization and spatial normalization on the different types of medical images. According to claim 3,
The spatial normalization is,
A medical image segmentation method characterized by dividing the minimum pixel value from the pixel size of the original image and converting the image using nearest neighbor interpolation. According to claim 1,
A medical image segmentation method, wherein the 2D convolutional neural network and the 3D convolutional neural network are composed of a contraction path and an expansion path. According to claim 5,
In the contraction path,
A medical image segmentation method characterized by sequentially performing convolution, batch normalization, activation function performance, and max pooling on the input medical images. According to claim 5,
In the above expansion path,
A medical image segmentation method comprising performing up-convolution and concatenation operations to increase the size of the image reduced in the contracted path and reduce the number of channels increased in the contracted path. According to claim 1,
The different types of medical images are three-section image information,
The step of performing the positioning is,
A medical image segmentation method, characterized in that localization of the specific organ is performed using spatial information about the three-section image information. According to claim 1,
A plurality of 3D medical images input to the 3D convolutional neural network are:
A medical image segmentation method characterized by converting different types of medical images on which localization has been performed into three-dimensional medical images. delete According to claim 1,
The step of dividing the specific organ is,
Based on the probability map of uncertainty, the uncertain area that occurs when segmenting a specific organ in a medical image is reduced,
A medical image segmentation method characterized by increasing segmentation accuracy in uncertain areas through cooperative learning. a medical image acquisition unit that acquires different types of medical images of abdominal organs; and
By inputting the different types of medical images into a two-dimensional convolutional neural network, a specific organ information probability map for each medical image is output, and a specific organ is selected through a first majority vote on the obtained plurality of specific organ information probability maps. It includes a control unit that performs positioning,
The control unit,
Convert multiple localized medical images into 3D medical images, input the multiple converted 3D medical images into a 3D convolutional neural network, and create a probability map of the average value and uncertainty of the multiple 3D medical images. Probability maps are output respectively,
Collaborative learning is performed using a probability map of the average value and a probability map of uncertainty for a plurality of 3D medical images, and a specific organ is segmented through a second majority vote on the cooperatively learned results.
Input a plurality of 3D medical images into a dropout layer to output N dropout results (N is a natural number) of the same size as the image input to the 3D convolutional neural network, and use the dropout results to calculate the average value. A medical image segmentation device characterized by extracting a probability map of and a probability map of uncertainty. According to claim 12,
The specific organs include the pancreas,
A medical image segmentation device, wherein the different types of medical images include two-dimensional medical images in the transverse section, coronal section, and sagittal section. According to claim 12,
The control unit,
Perform data preprocessing on the different types of medical images,
A medical image segmentation device that performs brightness value normalization and spatial normalization on the different types of medical images. According to claim 14,
The spatial normalization is,
A medical image segmentation device characterized by dividing the minimum pixel value from the pixel size of the original image and converting the image using nearest neighbor interpolation. According to claim 12,
A medical image segmentation device, wherein the 2D convolutional neural network and the 3D convolutional neural network are composed of a contraction path and an expansion path. According to claim 16,
In the contraction path,
A medical image segmentation device that sequentially performs convolution, batch normalization, activation function, and max pooling on the input medical images. According to claim 16,
In the above expansion path,
A medical image segmentation device that performs up-convolution and concatenation operations to increase the size of the image reduced in the contracted path and reduce the number of channels increased in the contracted path. According to claim 12,
The different types of medical images are three-section image information,
The control unit,
A medical image segmentation device that localizes the specific organ using spatial information about the three-section image information. According to claim 12,
A plurality of 3D medical images input to the 3D convolutional neural network are:
A medical image segmentation device characterized by converting different types of medical images on which localization has been performed into three-dimensional medical images. delete According to claim 12,
The control unit,
Based on the probability map of uncertainty, the uncertain area that occurs when segmenting a specific organ in a medical image is reduced,
A medical image segmentation device that increases segmentation accuracy in uncertain areas through cooperative learning. A computer program combined with a computer as hardware and stored in a computer-readable recording medium to perform the method of any one of claims 1 to 9 and 11.