KR20230030935A - Medical image segmentation apparatus and method for segmentating medical image - Google Patents

Abstract

Provided are a medical image segmenting device and a medical image segmenting method. The medical image segmenting method comprises: a step of obtaining different types of medical images in which abdominal organs are photographed; a step of outputting a specific organ information probability map for each medical image by inputting the different types of medical images to a 2D convolutional neural network, and performing positioning of a specific organ through a first majority vote for a plurality of obtained specific organ information probability maps; a step of converting the plurality of positioned medical images into a 3D medical image, inputting a plurality of converted medical images to a 3D convolutional neural network, and individually outputting a probability map and an uncertainty probability map of an average value for the plurality of 3D medical images; and a step of performing cooperative learning by using the probability map and the uncertainty probability map of the average value for the plurality of 3D medical images, and segmenting the specific organ through a second majority vote for a cooperative learned result. The present invention accurately segments pancreas by using a 3D segmenting network.

Description

Medical image segmentation apparatus and medical image segmentation method

The present invention relates to a medical image segmentation apparatus and a medical image segmentation method, and more particularly, to an automatic pancreas segmentation method based on a hierarchical network and cooperative learning considering uncertainty of the pancreas in an abdominal CT image, and an apparatus implementing the same.

According to the 2017 National Cancer Registry statistics released by the Ministry of Health and Welfare and the Central Cancer Registry Headquarters in 2019, pancreatic cancer ranked 8th with an incidence rate of 3% among all cancer types, and the incidence rate has been continuously increasing over the past 10 years.

Pancreatic cancer has no characteristic symptoms from the beginning to the advanced stage, the growth rate of carcinoma is fast, and metastasis to surrounding tissues is easy, and most of them are surgically resected to increase long-term survival rate. Before pancreatic cancer resection, pancreatic segmentation is an essential preliminary step in abdominal CT imaging because prior information on pancreatic morphology is important for identifying the pancreas and detecting pancreatic cancer.

FIG. 3 shows the characteristics of the pancreas in an abdominal CT image. As shown in FIG. Similarly, the shape of the pancreas seen in each image slice is different, and as shown in FIG. 3 (c), there is a feature that the shape of the pancreas varies for each patient.

There are various studies based on deep convolutional neural networks (CNNs), but in common, there are limitations in that under-segmentation and over-segmentation problems occur in the uncertain boundary part of the pancreas.

An object to be solved by the present invention is to provide a medical image segmentation apparatus and method capable of segmenting a specific device in a medical image in an optimized manner.

The problem to be solved by the present invention is to effectively detect the location of the pancreas in an abdominal CT image through a positioning step based on a deep convolutional neural network, and to improve pancreas segmentation accuracy by considering information about an uncertain region within the corresponding region. It is an object of the present invention to provide a medical image segmentation apparatus and method capable of doing this.

The problems to be solved by the present invention are not limited to the problems mentioned above, and other problems not mentioned will be clearly understood by those skilled in the art from the description below.

A method for segmenting a medical image according to an aspect of the present invention for solving the above problems is a method performed by an apparatus, comprising the steps of acquiring different types of medical images in which abdominal organs are photographed, and the different types of medical images. By inputting medical images into a 2D convolutional neural network, a probability map of specific organ information for each medical image is output, and positioning of a specific organ is performed through a first majority vote on the obtained plurality of specific organ information probability maps. converting the plurality of localized medical images into 3D medical images, inputting the converted plurality of 3D medical images to a 3D convolutional neural network, and obtaining a probability map of average values of the plurality of 3D medical images. and outputting a probability map of uncertainty, performing cooperative learning using the probability map of the average value of a plurality of 3D medical images and the probability map of uncertainty, and through a second majority vote on the cooperatively learned result. Segmenting a specific organ.

In an embodiment, the different types of medical images include cross-section, coronal, and sagittal two-dimensional medical images.

In an embodiment, the method may further include performing data pre-processing on the different types of medical images, wherein the performing of the data pre-processing may include normalizing brightness values and spatial resolution values of the different types of medical images. It is characterized by performing normalization.

In an embodiment, the spatial normalization is characterized in that the image is transformed using a nearest-nearest interpolation method by dividing a minimum pixel value from the pixel size of the original image.

In an embodiment, the 2D convolutional neural network and the 3D convolutional neural network are characterized in that they consist of a contraction path and an expansion path.

In an embodiment, in the contraction path, convolution, batch normalization, activation function execution, and max pooling are sequentially performed on the input medical images.

In an embodiment, in the expansion path, up convolution and concatenation operations are performed to increase the size of the reduced image in the contraction path and reduce the increased number of channels in the contraction path.

In an embodiment, the different types of medical images are three-section image information, and the positioning of the specific organ is performed using spatial information on the three-section image information. It is characterized by doing.

In an embodiment, the plurality of 3D medical images input to the 3D convolutional neural network are images obtained by converting different types of medical images on which localization is performed into 3D medical images, respectively.

In an embodiment, the outputting may include N (N is a natural number) dropout results having the same size as the image input to the 3D convolutional neural network by inputting a plurality of 3D medical images to a dropout layer. and extracting a probability map of average values and a probability map of uncertainty using the dropout result.

In an embodiment, the segmenting of the specific organ reduces an uncertain region generated when segmenting a specific organ in a medical image based on the probability map of uncertainty, and segmentation accuracy in the uncertain region through cooperative learning. It is characterized by increasing.

An apparatus for segmenting a medical image according to another embodiment of the present invention includes a medical image acquisition unit that acquires different types of medical images of abdominal organs and inputs the different types of medical images to a 2D convolutional neural network to obtain each of the different types of medical images. A control unit outputting a probability map of specific organ information for medical images and performing localization of a specific organ through a first majority vote on the obtained plurality of specific organ information probability maps, wherein the control unit comprises: The plurality of 3D medical images obtained are converted into 3D medical images, and the converted plurality of 3D medical images are input to a 3D convolutional neural network to obtain a probability map of average values and a probability map of uncertainty for the plurality of 3D medical images. , performing cooperative learning using a probability map of the average value and a probability map of uncertainty for a plurality of 3D medical images, and dividing a specific organ through a second majority vote on the cooperatively learned result. to be characterized

In an embodiment, the specific organ includes the pancreas, and the different types of medical images include transverse, coronal, and sagittal two-dimensional medical images.

In an embodiment, the controller may perform data pre-processing on the different types of medical images, and perform brightness value normalization and space normalization on the different types of medical images.

In an embodiment, the spatial normalization is characterized in that the image is transformed using a nearest-nearest interpolation method by dividing a minimum pixel value from the pixel size of the original image.

In an embodiment, the 2D convolutional neural network and the 3D convolutional neural network are characterized in that they consist of a contraction path and an expansion path.

In an embodiment, in the contraction path, convolution, batch normalization, activation function execution, and max pooling are sequentially performed on the input medical images.

In an embodiment, in the expansion path, up convolution and concatenation operations are performed to increase the size of the reduced image in the contraction path and reduce the increased number of channels in the contraction path.

In an embodiment, the different types of medical images are three-section image information, and the controller performs positioning of the specific organ using spatial information on the three-section image information. .

In an embodiment, the plurality of 3D medical images input to the 3D convolutional neural network are images obtained by converting different types of medical images on which localization is performed into 3D medical images, respectively.

In an embodiment, the controller inputs a plurality of 3D medical images to a dropout layer and outputs N (N is a natural number) dropout results having the same size as the image input to the 3D convolutional neural network. , It is characterized in that a probability map of the average value and a probability map of uncertainty are extracted using the dropout result.

In an embodiment, the control unit reduces an uncertain region generated when a specific organ is segmented in a medical image based on the probability map of uncertainty, and increases segmentation accuracy in the uncertain region through cooperative learning. to be

In addition to this, another method for implementing the present invention, another system, and a computer readable recording medium recording a computer program for executing the method may be further provided.

Other specific details of the invention are included in the detailed description and drawings.

According to the present invention, in the present invention, after automatically extracting location information of the pancreas with a 2.5-dimensional segmentation network, a new medical image capable of accurately segmenting the pancreas using a 3-dimensional segmentation network that takes into account information on uncertain regions in the localized area Split method can be provided.

The effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description below.

1 is a conceptual diagram illustrating an apparatus for segmenting a medical image according to an embodiment of the present invention.
2 is a flowchart illustrating a method for segmenting a medical image according to an embodiment of the present invention.
3 is a conceptual diagram for explaining characteristics of the pancreas in an abdominal CT image, which is one of medical images.
4A, 4B, 4C, 5, 6, and 7 are conceptual diagrams for explaining the medical image segmentation method described in FIG. 2 .

Advantages and features of the present invention, and methods of achieving them, will become clear with reference to the detailed description of the following embodiments taken in conjunction with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but may be implemented in various different forms, only these embodiments are intended to complete the disclosure of the present invention, and are common in the art to which the present invention belongs. It is provided to fully inform the person skilled in the art of the scope of the invention, and the invention is only defined by the scope of the claims.

Terminology used herein is for describing the embodiments and is not intended to limit the present invention. In this specification, singular forms also include plural forms unless specifically stated otherwise in a phrase. As used herein, "comprises" and/or "comprising" does not exclude the presence or addition of one or more other elements other than the recited elements. Like reference numerals throughout the specification refer to like elements, and “and/or” includes each and every combination of one or more of the recited elements. Although "first", "second", etc. are used to describe various components, these components are not limited by these terms, of course. These terms are only used to distinguish one component from another. Accordingly, it goes without saying that the first element mentioned below may also be the second element within the technical spirit of the present invention.

Unless otherwise defined, all terms (including technical and scientific terms) used in this specification may be used with meanings commonly understood by those skilled in the art to which the present invention belongs. In addition, terms defined in commonly used dictionaries are not interpreted ideally or excessively unless explicitly specifically defined.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

Prior to the description, the meaning of the terms used in this specification will be briefly described. However, it should be noted that the description of terms is intended to help the understanding of the present specification, and is not used in the sense of limiting the technical spirit of the present invention unless explicitly described as limiting the present invention.

In this specification, the 'medical image segmentation device' includes all various devices capable of providing results to users by performing calculation processing.

For example, a 'medical image segmentation device' is not only a computer, desktop PC, and notebook (note book), but also a smart phone, tablet PC, cellular phone, and PC phone (Personal Communication). Service phone), synchronous/asynchronous IMT-2000 (International Mobile Telecommunication-2000) mobile terminal, Palm PC (Palm Personal Computer), Personal Digital Assistant (PDA), etc. may also be applicable.

Also, the 'medical image segmentation device' may receive a request from a client and communicate with a server that processes information.

A medical image segmentation apparatus according to an embodiment of the present invention may be implemented to include at least one of the components described in FIG. 1 .

1 is a conceptual diagram illustrating an apparatus for segmenting a medical image according to an embodiment of the present invention.

Referring to FIG. 1 , the medical image segmentation apparatus 100 according to the present invention may include a medical image capture unit 110 that acquires a medical image of an abdominal organ photographed.

Here, the medical image of the abdominal organs may include an abdominal computed tomography (CT) image.

Specifically, the medical image acquisition unit 110 may acquire (collect, photograph) different types of medical images in which abdominal organs are imaged.

Here, the different types of medical images may include axial plane, coronal plane, and sagittal plane 2D medical images (2D abdominal CT images).

The medical image acquisition unit 110 may include a photographing device configured to capture a medical image or a communication unit configured to receive the captured medical image from an external device.

The memory 120 may be configured to store data, images, various types of information, and application programs generated/managed by the medical image acquisition unit 110 and the controller 130 .

The memory 120 is electrically connected to the control unit 130 . The memory 120 may store basic data for the unit, control data for controlling the operation of the unit, and input/output data. The memory 120 may be a variety of storage devices such as ROM, RAM, EPROM, flash drive, hard drive, etc. in terms of hardware. The memory 120 may store various data for overall operations of the medical image segmentation apparatus 100, such as programs for processing or control by the controller 130.

The controller 130 controls general operations of the medical image segmentation apparatus 100 in addition to operations related to application programs. The control unit 130 may provide or process appropriate information or functions to a user by processing signals, data, information, etc. input or output through the components described above or by driving an application program stored in a memory.

Hereinafter, a method of segmenting a specific organ in a medical image will be described in more detail with reference to the accompanying drawings.

2 is a flowchart illustrating a method for segmenting a medical image according to an embodiment of the present invention.

3 is a conceptual diagram for explaining the characteristics of the pancreas in an abdominal CT image, which is one of the medical images, and FIGS. 4a, 4b, 4c, 5, 6, and 7 explain the medical image segmentation method described in FIG. 2. It is a concept for

Referring to FIG. 2 , the controller 130 of the medical image segmentation device may control the medical image acquisition unit 110 to acquire different types of medical images (abdominal CT images) captured by the abdominal device (S210). ).

The controller 130 inputs the different types of medical images to a 2D convolutional neural network, outputs a probability map of specific organ information for each medical image, and generates a first information probability map for a plurality of obtained specific organ information probability maps. Positioning of a specific organ can be performed through a majority vote (S220).

The controller 130 converts the plurality of positioned medical images into 3D medical images, inputs the converted plurality of 3D medical images to the 3D convolutional neural network, and inputs an average value of the plurality of 3D medical images. A probability map of and a probability map of uncertainty may be output respectively (S230).

The control unit 130 performs cooperative learning using a probability map of average values and a probability map of uncertainty for a plurality of 3D medical images, and divides a specific organ through a second majority vote on the cooperatively learned result. It can (S240).

Hereinafter, for convenience of description, a medical image and an abdominal CT image will be used interchangeably, and a specific organ will be described as being the pancreas as an example.

As shown in FIG. 3, the pancreas has a brightness value similar to that of the surrounding organs such as the stomach, liver, spleen, small intestine, and portal vein as shown in (a), and the shape of the pancreas seen in each image slice is different as shown in (b). As shown in (c), the shape of the pancreas varies from patient to patient.

Accordingly, the method of segmenting the pancreas in an optimized method in an abdominal CT image of the present invention can be provided.

The controller 130 of the medical image segmentation apparatus may automatically segment the pancreas in the abdominal CT image.

Referring to FIG. 4A , the apparatus for segmenting a medical image according to the present invention may perform automatic positioning of the pancreas using network input image preprocessing and a 2D segmentation network based on a deep convolutional neural network. In addition, the medical image segmentation apparatus may perform automatic segmentation of the pancreas using a 3D segmentation network in which information on an uncertain region is also considered.

Meanwhile, in the present invention, the degree of uncertainty that may occur in an image is quantified and applied to a segmentation model, and segmentation performance can be improved through cooperative learning.

For example, as shown in FIG. 4B, the present invention quantifies uncertainty using a 3D U-net (three-dimensional segmentation network) considering the uncertainty of a specific device in order to quantify information about an uncertain region in an image, and , it can be reflected in the segmentation model (ie, automatic segmentation of the pancreas). In addition, as shown in FIG. 4C, the present invention provides specific organs (for example, axial plane, coronal plane, and sagittal plane) of different types of images. For example, uncertainty about the location of the pancreas may be quantified, and automatic segmentation of the pancreas may be performed for each image through co-training's prediction.

Through this, the present invention can improve segmentation performance by performing quantification of uncertainty and performing cooperative learning.

First, the data pre-processing process will be described as follows.

The controller 130 may perform data pre-processing on acquired medical images of different types.

Differences between data caused by differences in imaging equipment and imaging protocols for abdominal CT images can reduce the segmentation performance of the network.

Accordingly, the controller 130 performs brightness normalization and spatial normalization to reduce the difference between abdominal CT images before performing automatic segmentation.

That is, as described above, the controller 130 may perform brightness normalization and space normalization on 2D medical images of different types, such as transverse, coronal, and sagittal planes.

In the abdominal CT image, in order to match the range of brightness values of images having different distributions, the brightness values of the images are normalized to between 0 and 255 through Equation 1.

At this time, I_org is the brightness value of the original image, and I_max and I_min are the minimum and maximum thresholds, respectively. The minimum threshold is set to include the brightness value of the background area, and the maximum threshold is set to include the brightness value of the bone area. . Values corresponding to the threshold may be calculated as -174HU and 326HU, respectively, through experiments.

In order to reduce the difference in pixel space range between patients, spatial normalization of the experimental data set is performed from the pixel spacing range of 0.6641 mm to 0.9766 mm in the experimental data to 0.6641 mm corresponding to the minimum pixel size.

Spatial normalization transforms an image by dividing the minimum pixel value from the pixel size of the original image using nearest interpolation.

Thereafter, the controller 130 may perform pancreas automatic positioning based on the three-section 2.5D U-NET.

The U-net refers to a convolutional neural network developed for biomedical image segmentation, and includes a 2D convolutional neural network (2D U-net) and a 3D convolutional neural network (3D U-net). .

As shown in FIG. 5, U-net may have a network shape similar to the alphabet U.

U-net is a segmentation network (segmentation network) that shows excellent effects in biomedical image segmentation.

If existing CNNs were mainly used for simple classification, U-net has the strength of being able to perform segmentation as well as classification.

In addition, U-net has the advantage of being able to solve the problems of the existing segmentation method by using an overlap-tile strategy during segmentation.

The overlap tile strategy refers to cutting a large-sized image in patch units and inputting it as an input.

In addition, U-net has the advantage of not falling into a trade-off according to the patch size.

In general, when large-sized patches are used, a wide range of images are recognized at once, which has a great effect on context recognition, but localization accuracy is reduced. When small-sized patches are used, localization accuracy is greatly reduced. increases, but the recognizable context narrows.

However, in the network structure of U-net, context recognition is performed on the contracting path and localization is performed on the expanding path, resulting in a trade-off according to the patch size ( It has the advantage of not falling into a trade-off.

The convolutional neural network described herein may mean a deep convolutional neural network (DCNN), and may be the U-net described above.

Referring to FIG. 5, the convolutional neural network, that is, U-net, will be described as follows.

In the U-net network, a contracting path (red box) corresponding to the convolution encoder and an expanding path corresponding to the convolution decoder may form a symmetrical shape around the center.

When upsampling is performed in the expansion path, the U-net network copies and crops the features of the contraction path for accurate localization, and concatenation structure can be achieved.

Specifically, the contracting path has a convolution network structure, uses the ReLU function as an activation function, reduces the size using 2x2 max pooling, and moves on to the next block.

After moving to the next block, the U-net network uses twice as many channels as the number of previous feature channels for feature extraction.

The expanding path uses an up-convolution block that increases the size reduced in the contracting process.

You can use 2x2 up-convolution and use ReLU as the activation function.

After moving to the next block, the U-net network can use twice as many channels for localization as the number of previous feature channels.

In addition, the U-net network has a structure that performs image compensation processing and more accurate localization by copying and cropping the features of the contracting path and concatenating them.

The U-net network can classify images into two classes using 1x1 convolution in the last final layer.

The medical image segmentation apparatus and method of the present invention can automatically segment the pancreas in an abdominal CT image using the convolutional neural network including the U-net described above.

Conventionally, since the area occupied by the pancreas in an abdominal CT image is small, when pancreas segmentation is performed using the entire area of the abdominal CT image, there is a limitation in that the pancreas is undersegmented.

Therefore, it is necessary to provide pancreatic region information that can focus on regional context information in the segmentation step after effectively detecting the location of the pancreas in the abdominal CT image.

The medical image segmentation apparatus and method of the present invention is a method for extracting pancreas location information with a 2D U-Net based segmentation network using an abdominal CT image capable of utilizing 2.5D plane information capable of inferring the spatial shape of the pancreas. can provide

The controller 130 inputs the preprocessed medical images to a convolutional neural network (eg, U-net network) for biomedical image segmentation, and from the convolutional neural network to the input medical images, a specific organ (eg, For example, a shape information probability map of the pancreas) may be output.

Referring to stage 1 of FIG. 4B, the control unit 130 inputs the different types of medical images to a 2D convolutional neural network, outputs a probability map of specific organ information for each medical image, and outputs a plurality of acquired The positioning of a specific organ can be performed through a first majority vote on the specific organ information probability map. The control unit 130 performs data preprocessing on the input image, 512×512 resolution axial plane, coronal Two-dimensional abdominal CT images of the coronal plane and the sagittal plane can be used.

The convolutional neural network may be a two-dimensional U-net.

The convolutional neural network may include a contraction path and an expansion path.

In the contraction path, convolution, batch normalization, activation function execution, and max pooling may be sequentially performed on the input medical image.

Specifically, the 2D U-Net may be composed of a contracting path and an expanding path.

The control unit 130 performs two 3 × 3 convolutions, batch normalization (BN), and rectified linear unit (ReLU) functions in the contraction path, and then performs 2 × 2 max pooling (max with a stride of 2) -pooling) can be repeated six times.

Through 2×2 max pooling performed at each stage of the contraction path, the size of the feature map is reduced by half and the number of channels of the feature map is doubled.

In the expansion path, up convolution and concatenation operations are performed to increase the size of the reduced image in the contraction path and reduce the increased number of channels in the contraction path.

Specifically, the control unit 130 performs a 2×2 up-convolution on the expansion path, and then performs a skip connection combining the last feature map of the expansion path and the equivalent contraction path. and performing two 3×3 convolutions, batch normalization, and the ReLU function can be repeated six times.

Through 2×2 up convolution performed at each stage of the extension path, the size of the feature map is doubled and the number of channels of the feature map is reduced by half.

The controller 130 finally performs 1×1 convolution and outputs a pancreas shape information probability map having the same size as the input image. The probability map has a probability value between 0 and 1.

The controller 130 may perform the positioning of the specific organ by applying a first majority vote to input medical images.

Specifically, the controller 130 performs a majority vote (MV) on N (N is a natural number) (e.g., three) division results obtained using input images of a transverse plane, a coronal plane, and a sagittal plane. ) technique can be used to perform pancreatic localization.

The control unit 130 may set the positioning area to include blank spaces in order to reduce positioning errors due to underfitting resulting from the majority vote.

At this time, the value corresponding to the margin is set to 30-pixels for each axis through experiments. Figure 6 shows the result of pancreas automatic positioning in the abdominal CT image, and Figures 6(a), 6(b), and 6(c) show the positioning results when using only transverse, coronal, and sagittal input images, respectively. 6(d) shows the positioning result when the majority voting method is used for the three-section image information.

In the case of performing positioning using only 2D plane information of one cross section, low positioning accuracy is shown due to incorrect segmentation into other regions caused by not considering spatial information, whereas the present invention, in FIG. 6(d) As shown, accurate positioning to the pancreatic region can be performed by additionally considering spatial information using a majority voting method.

Then, the medical image segmentation apparatus and method of the present invention may provide a method of automatically segmenting the pancreas in consideration of uncertainty.

Referring to Stage 2 of FIG. 4B, the controller 130 of the present invention converts a plurality of positioned medical images into a 3D medical image, and inputs the converted plurality of 3D medical images to a 3D convolutional neural network. Thus, a probability map of an average value and a probability map of uncertainty for a plurality of 3D medical images may be output, respectively.

In addition, the controller 130 performs cooperative learning using a probability map of an average value and a probability map of uncertainty for a plurality of 3D medical images, and selects a specific organ through a second majority vote on the cooperatively learned result. can be divided

The different types of medical images described above may be 3D medical images.

In this case, the convolutional neural network may be a 3D convolutional neural network (ie, a 3D U-net).

The plurality of 3D medical images input to the 3D convolutional neural network may be images obtained by converting medical images of different types on which localization is performed into 3D medical images, respectively. That is, the plurality of 3D medical images may include a positioned 3D image of a cross section, a 3D image of a positioned coronal plane, and a 3D image of a positioned sagittal plane.

Compared to the prior art of FIG. 4A, the present invention, referring to FIG. 4B, does not input only one 3D medical image to a 3D convolutional neural network (or dropout layer), but instead uses a plurality of 3D medical images for cooperative learning. A medical image may be input to a 3D convolutional neural network (or dropout layer), and a probability map of an average value and a probability map of uncertainty may be output for each 3D medical image.

Specifically, the controller 130 inputs a plurality of 3D medical images to the dropout layer and outputs N (N is a natural number) dropout results having the same size as the image input to the 3D convolutional neural network. , a probability map of average values and a probability map of uncertainty may be extracted using the dropout result. Referring to FIG. , Sagittal plane), cooperative learning can be performed using the probability map of the calculated average value and the probability map of uncertainty.

A prediction value for each 3D medical image is calculated using the probability map of the average value and the probability map of uncertainty extracted from each 3D medical image, and cooperative learning is performed using the predicted value (Co). -trainingins' prediction) can be calculated.

Based on the probability map of uncertainty, the control unit 130 may reduce an uncertain region generated when segmenting a specific organ in a medical image, and increase segmentation accuracy in an uncertain region through cooperative learning.

Specifically, the present invention can provide a method for automatically segmenting the pancreas based on 3D U-Net in consideration of uncertainty.

When the pancreas is segmented using a 2D network that does not consider spatial information, the shape of the pancreas seen in each slice is different, resulting in under-segmentation in the head or tail of the pancreas, and poor segmentation accuracy at the border of the pancreas. there is.

In addition, when the pancreas is divided into a 3D network, leakage occurs to surrounding organs having a shape similar to that of the pancreas, and under- or over-segmentation occurs at an uncertain boundary portion of the pancreas.

Therefore, in the present invention, the spatial information around the pancreas is considered, leakage to nearby organs having a similar shape is prevented, and under- and over-segmentation in the uncertain region of the pancreas caused by the positional and morphological diversity of the pancreas is prevented. To prevent this, the pancreas can be segmented using a 3D segmentation network that also considers information about uncertain regions.

To this end, the medical image segmentation apparatus of the present invention may use a 3D abdominal CT image obtained by converting an abdominal CT image positioned using a 2D U-Net-based network to a resolution of 96×96×96 as an input image. .

A 3D U-Net can be composed of a contraction path and an expansion path in the same way as a 2D U-Net.

In the contraction path, the step of sequentially performing 2x2x2 max pooling with a stride of 2 is repeated four times after performing two 3x3x3 convolutions, batch normalization, and the ReLU function.

Through 2×2×2 max pooling performed at each stage of the contraction path, the size of the feature map is reduced by half and the number of channels of the feature map is doubled.

In the extension path, after performing a 2×2×2 up convolution, a concatenation operation that combines the last feature map of the expansion path and the equivalent contraction path, two 3×3×3 convolutions, batch normalization, and the ReLU function Repeat the steps to perform four times.

In order to reduce the uncertain region generated during pancreas segmentation, a 1×1×1 convolution is finally performed to output a 3D pancreas shape probability map of the same size as the input image, before adding a dropout layer to input the input image. Output N (N is a natural number) dropout results of the same size as the image. The output N dropout results (N is a natural number) are used to output a probability map of average values extracted from each 3D medical image and a probability map of uncertainty.

The probability map of the average value can be derived through Equation (1) in FIG. 4C, and the probability map of uncertainty can be derived through Equation (2) in FIG. 4C.

Using the probability map of the average value and the probability map of uncertainty extracted from each 3D medical image, a prediction value (Equation (3) in FIG. 4C) is calculated for each 3D medical image, and the cooperation is used to Learning can be performed to calculate a cooperative learning prediction value (Co-trainingins' prediction) (Equation (4) in FIG. 4C).

Through this configuration, when the pancreas is segmented using a 3D segmentation network considering uncertainty, the present invention can consider regional context information and spatial information, as well as reduce under-segmentation and over-segmentation problems that occur in uncertain regions. there is.

The present invention can provide a D CNN-based automatic pancreas segmentation method that considers the uncertain region due to the positional and morphological variability of the pancreas in an abdominal CT image.

According to the present invention, a 2.5D network based on three orthogonal plane images can automatically find the pancreas of an abdominal organ.

In addition, according to the present invention, the 3D network considering the uncertainty of the present invention can be reduced through segmentation in the uncertain region of the pancreas.

In addition, the 3D network considering the joint training of the present invention can improve the segmentation accuracy in the uncertain region of the pancreas.

That is, the present invention can provide an automatic pancreas segmentation method using a cascade network in consideration of pancreatic uncertainty in an abdominal CT image, and can provide a 2D ensemble network that can accurately position the pancreas in consideration of three orthogonal plane information. there is. In addition, the 3D network of the present invention can reduce the area under and over the subdivided area by considering local spatial information, pancreatic uncertainty and joint training.

For this configuration, the following experiments and result analysis were performed.

The data used in the experiment were portal-venous abdominal CT images of a total of 82 patients distributed by The National Institutes of Health Clinical Center (NIH). It consists of 53 males and 27 females. For all data, three-dimensional cross-sectional abdominal CT images taken by Phillips and Siemens at a tube voltage of 120 kVp were used. The number of slices in the abdominal CT images of 82 patients was 181 to 466, the image resolution was 512 × 512, the pixel size was 0.6641 to 0.9766 mm2, and the slice interval was 0.5 mm to 1.0 mm.

The environment for the experiment was conducted using a PC equipped with Inter(R) Core(TM) i7-8700 3.20Hz CPU, 32GB memory, and NVIDIA GeForce RTX 2080Ti graphics card under Windows 10 (64-bit) operating system. This proposed method was implemented using Python-based deep learning libraries Tensorflow and Keras, and 2D and 3D U-Nets were used by modifying open source packages. A learning rate of 0.0001 and a batch size of 1 were used for 2D segmentation network learning, and a total of 100 iterations of learning were performed using the Adam optimizer for optimization. The loss function used cross-entropy. The initial value of the learning rate used in learning the 3D segmentation network is 0.0001, which is reduced by 0.8 times if the loss function of the verification data set does not decrease, the batch size was 1, and the Adam optimizer was used for optimization. The experiment was performed in a hold-out cross-validation method. For the experiment, the data set was composed of 52 training data sets and 16 test data sets, and was divided into 14 validation data sets to prevent overfitting.

Quantitative and qualitative evaluations were performed to evaluate the performance of the proposed method. For quantitative evaluation, DSC, sensitivity (recall), and positive predictive value (precision) between the results of manual segmentation, comparison method, and proposed method were measured as shown in Equation 2.

At this time, TP is the region automatically divided into the pancreas from the region manually divided into the pancreas, TN is the region manually divided into the non-pancreatic region and automatically divided into the non-pancreatic region, and FP is the region divided into the non-pancreatic region. It is a region automatically divided into a pancreas in a manually divided region, and FN means a region automatically divided into a non-pancreatic region in a manually divided region into a pancreas.

As a comparison method, the method of dividing the pancreas using the cross section, coronal plane, and sagittal plane in the 2D U-Net structure is Method A, Method B, and Method C, respectively. Comparison and analysis of the performance of the majority voting method (Method D) of pancreas segmentation results using the majority voting method, the pancreas segmentation method (Method E) using the 3D input image positioned through the majority voting method, and the proposed method (Method F) considering uncertainty did

Table 1 shows the results of quantitative evaluation of pancreas segmentation results.

In the case of Method A, Method B, and Method C, which trained a 2D segmentation network using cross-sectional images, the average DSC was 74.41%, showing low overall performance. On the other hand, in the case of Method D, 2.5-dimensional space information is inferred by combining 2-dimensional plane information of a single cross section, and the over-segmentation area is reduced by using a majority voting method, resulting in 1.94% sensitivity compared to Method A, Method B, and Method C, respectively. p, 5.58%p, 3.42%p improved performance, but showed the highest positive predictive value, showing a tendency to under-split. In the case of Method E, under-segmentation was prevented by considering spatial information in a limited area compared to Method D, and the sensitivity increased by 3.29%p. It was confirmed, and when Method F was applied, it was confirmed that DSC, sensitivity, and positive predictive value increased by 3.95%p, 2.83%p, and 4.13%p, respectively, compared to Method E by considering the information on the uncertain area together.

7 is a result of segmentation by comparison method according to pancreas location in an abdominal CT image. For the qualitative evaluation of the proposed method, the results of the comparison method and the proposed method were compared with manual results.

7, (a) to (g) show qualitative evaluations corresponding to Methods A to Method G, respectively, where red indicates an area overlapping with the manual segmentation result, blue indicates an over-segmented area, and green indicates an under-segmented area.

In the case of Method A, Method B, and Method C, leakage occurred to surrounding areas with similar shapes and brightness values, such as the thorax, liver, spleen, and small intestine, and under-segmentation occurred in the head and tail of the pancreas. In the case of Method D, leakage to the surrounding organs that occurred in the transverse, coronal, and sagittal slices was resolved through a majority vote, but it was confirmed that under-segmentation still existed in the tail of the pancreas. In the case of Method E, leakage into the surrounding organs was reduced using spatial information within a limited area, but leakage into the small intestine, which has a shape similar to that of the pancreas, was present, and under- and over-segmentation was confirmed at the uncertain border portion of the pancreas. In the case of Method F, leakage to the surrounding organs that occurred in Methods A, Method B, and Method C was prevented, and the under-segmented area was segmented in Method E, so it was confirmed that the result was the most similar to the manual segmentation result.

Through these experiments and configurations, the present invention provides a method for accurately segmenting the pancreas using a 3-dimensional segmentation network that automatically extracts location information of the pancreas with a 2.5-dimensional segmentation network and then considers information on uncertain regions in the localized area. can provide

In addition, the present invention, in order to obtain accurate location information of the pancreas in an abdominal CT image using a 2-dimensional segmentation network, N (N is a natural number) segmentation results obtained from transverse, coronal, and sagittal abdominal CT images It was improved through a majority vote, and by changing the weight of the loss function according to the uncertainty, it is possible to overcome the under-segmentation in the uncertain region caused by the morphological features of the pancreas in the localized 3D abdominal CT image.

In addition, in the present invention, compared to the comparison method, the DSC between the pancreas division result of the proposed method and the manual division result showed 83.50%, and it could be confirmed that the pancreas division was possible more accurately than other comparative methods.

Steps of a method or algorithm described in connection with an embodiment of the present invention may be implemented directly in hardware, implemented in a software module executed by hardware, or implemented by a combination thereof. A software module may include random access memory (RAM), read only memory (ROM), erasable programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory, hard disk, removable disk, CD-ROM, or It may reside in any form of computer readable recording medium well known in the art to which the present invention pertains.

Although the embodiments of the present invention have been described with reference to the accompanying drawings, those skilled in the art to which the present invention pertains can be implemented in other specific forms without changing the technical spirit or essential features of the present invention. you will be able to understand Therefore, it should be understood that the embodiments described above are illustrative in all respects and not restrictive.

Claims (23)

In the method performed by the device,
acquiring different types of medical images in which abdominal organs are imaged;
The different types of medical images are input into a 2D convolutional neural network to output a probability map of specific organ information for each medical image, and a specific organ information probability map is obtained through a first majority vote on a plurality of specific organ information probability maps. performing positioning of;
A plurality of localized medical images are converted into 3D medical images, and the converted plurality of 3D medical images are input to a 3D convolutional neural network, and a probability map of an average value of the plurality of 3D medical images and uncertainty outputting probability maps, respectively; and
A medical image comprising performing cooperative learning using a probability map of average values and a probability map of uncertainty for a plurality of 3D medical images, and segmenting a specific organ through a second majority vote on a cooperatively learned result. Split method. According to claim 1,
The medical image segmentation method of claim 1 , wherein the different types of medical images include 2-dimensional medical images of a transverse plane, a coronal plane, and a sagittal plane. According to claim 1,
Further comprising performing data pre-processing on the different types of medical images;
The step of performing the data preprocessing,
The method of segmenting a medical image, characterized in that performing brightness value normalization and spatial normalization on the different types of medical images. According to claim 3,
The spatial normalization,
A method of segmenting a medical image, characterized in that the image is converted using a nearest-nearest interpolation method by dividing a minimum pixel value from a pixel size of an original image. According to claim 1,
The medical image segmentation method, characterized in that the 2-dimensional convolutional neural network and the 3-dimensional convolutional neural network are composed of a contraction path and an expansion path. According to claim 5,
In the contraction path,
The method of segmenting a medical image, characterized in that sequentially performing convolution, batch normalization, execution of an activation function, and max pooling on the input medical images. According to claim 5,
In the extension path,
The method of segmenting a medical image, characterized in that by performing an up-convolution and concatenation operation, the size of the reduced image in the contraction path is increased, and the increased number of channels in the contraction path is reduced. According to claim 1,
The different types of medical images are three-section image information,
The step of performing the positioning is,
The method of segmenting a medical image, characterized in that performing the positioning of the specific organ using spatial information of the three-section image information. According to claim 1,
A plurality of 3D medical images input to the 3D convolutional neural network,
A medical image segmentation method characterized in that the images are images obtained by converting different types of medical images on which positioning has been performed into 3D medical images, respectively. According to claim 9,
The outputting step is
A plurality of 3D medical images are input to the dropout layer, N (N is a natural number) dropout results having the same size as the image input to the 3D convolutional neural network are output, and an average value is obtained using the dropout results. A medical image segmentation method characterized by extracting a probability map of and a probability map of uncertainty. According to claim 10,
The step of dividing the specific organ,
Based on the probability map of uncertainty, reducing an uncertain region generated when segmenting a specific organ in a medical image;
A medical image segmentation method characterized by increasing segmentation accuracy in an uncertain area through cooperative learning. a medical image acquisition unit that acquires different types of medical images in which abdominal organs are photographed; and
The different types of medical images are input into a 2D convolutional neural network to output a probability map of specific organ information for each medical image, and a specific organ information probability map is obtained through a first majority vote on a plurality of specific organ information probability maps. Including a control unit for performing the positioning of,
The control unit,
A plurality of localized medical images are converted into 3D medical images, and the converted plurality of 3D medical images are input to a 3D convolutional neural network, and a probability map of an average value of the plurality of 3D medical images and uncertainty Each probability map is output,
A medical image characterized by performing cooperative learning using a probability map of average values and a probability map of uncertainty for a plurality of 3D medical images, and segmenting a specific organ through a second majority vote on the cooperatively learned result. split device. According to claim 12,
The specific organ includes the pancreas,
The medical image segmentation apparatus of claim 1 , wherein the different types of medical images include 2-dimensional medical images of a transverse plane, a coronal plane, and a sagittal plane. According to claim 12,
The control unit,
performing data pre-processing on the different types of medical images;
The medical image segmentation device characterized in that performing brightness normalization and spatial normalization on the different types of medical images. 15. The method of claim 14,
The spatial normalization,
A medical image segmentation device characterized by converting an image by dividing a minimum pixel value from a pixel size of an original image using a nearest-nearest interpolation method. According to claim 12,
The medical image segmentation apparatus, characterized in that the 2-dimensional convolutional neural network and the 3-dimensional convolutional neural network are composed of a contraction path and an expansion path. 17. The method of claim 16,
In the contraction path,
The medical image segmentation apparatus characterized in that it sequentially performs convolution, batch normalization, execution of an activation function, and max pooling on the input medical images. 17. The method of claim 16,
In the extension path,
The medical image segmentation apparatus, characterized in that, by performing up convolution and concatenation operations, the size of the reduced image in the contraction path is increased, and the increased number of channels in the contraction path is reduced. According to claim 12,
The different types of medical images are three-section image information,
The control unit,
The medical image segmentation apparatus characterized in that the localization of the specific organ is performed using spatial information of the three-section image information. According to claim 12,
A plurality of 3D medical images input to the 3D convolutional neural network,
An apparatus for segmenting medical images, characterized in that they are images obtained by converting different types of medical images on which positioning has been performed into 3D medical images. 21. The method of claim 20,
The control unit,
A plurality of 3D medical images are input to the dropout layer, N (N is a natural number) dropout results having the same size as the image input to the 3D convolutional neural network are output, and an average value is obtained using the dropout results. A medical image segmentation device characterized in that for extracting a probability map of and a probability map of uncertainty. According to claim 21,
The control unit,
Based on the probability map of uncertainty, reducing an uncertain region generated when segmenting a specific organ in a medical image;
A medical image segmentation device characterized by increasing segmentation accuracy in an uncertain area through cooperative learning. A computer program stored in a computer-readable recording medium to perform the method of any one of claims 1 to 11 by being combined with a computer that is hardware.

Images