KR20220129406A - A method and apparatus for image segmentation using residual convolution based deep learning network - Google Patents

Abstract

The present invention relates to image segmentation method and apparatus using a residual convolution-based deep learning network. The image segmentation method using a residual convolution-based deep learning network, according to an embodiment of the present invention, comprises the steps of: (a) obtaining input information; (b) inputting, into a first fire module comprising a squeeze layer composed of a first convolution filter and an expand layer composed of a second convolution filter and a third convolution filter, the input information; (c) concatenating output information of the first fire module and the input information obtained through residual connection; and (d) performing encoding by max pooling a result value calculated by the concatenation. The present invention can perform smooth learning of a deep learning network architecture through a residual unit.

Description

A method and apparatus for image segmentation using residual convolution based deep learning network}

The present invention relates to an image segmentation method and apparatus, and more particularly, to an image segmentation method and apparatus using a residual convolution-based deep learning network.

Magnetic resonance imaging (MRI) has high contrast and relatively high resolution. Therefore, the new technology is widely used to examine the human brain in clinical applications and scientific research.

In this case, automatic division of brain tissue into white matter (WM), gray matter (GM) and cerebrospinal fluid (CSF) is very important in the aforementioned work.

Accurate tissue segmentation is challenging due to the complex brain structure and tissue heterogeneity caused by MRI noise, deflection fields, and partial volume effects.

To solve this problem, a strategy using deep learning networks is used for segmentation work due to related advantages, but studies on this are insufficient.

[Patent Document 1] Korean Patent No. 10-2089014

The present invention was created to solve the above problems, and an object of the present invention is to provide an image segmentation method and apparatus using a residual convolution-based deep learning network.

Another object of the present invention is to provide an image segmentation method and apparatus using a fire module including a squeeze layer and an extension layer.

Another object of the present invention is to provide an image segmentation method and apparatus for concatenating output information of a fire module and input information through residual concatenation.

Objects of the present invention are not limited to the objects mentioned above, and other objects not mentioned will be clearly understood from the description below.

In order to achieve the above objects, an image segmentation method using a residual convolution-based deep learning network according to an embodiment of the present invention includes the steps of: (a) obtaining input information; (b) a first fire module including a squeeze layer composed of a first convolution filter and an expand layer composed of a second convolution filter and a third convolution filter for the input information to enter into; (c) concatenating the output information of the first fire module and the input information obtained through residual connection; and (d) performing encoding by max pooling the result values calculated by concatenation.

In an embodiment, the step (a) includes: obtaining an input image; and generating the input information in the form of a patch by dividing the input image.

In an embodiment, the first convolution filter and the second convolution filter may have a first kernel size, and the third convolution filter may have a second kernel size.

In an embodiment, the step (b) may include generating an output value of a first convolution filter of the squeeze layer; inputting the generated output value to each of a second convolution filter and a third convolution filter configured in parallel of the extension layer; and generating output information of the first fire module by concatenating the respective output values of the second convolution filter and the third convolution filter.

In an embodiment, the image segmentation method using the residual convolution-based deep learning network includes, after the step (d), max unpooling the encoding value calculated by performing the encoding; inputting the result calculated by the max unpooling to a second fire module including a squeeze layer composed of a first convolution filter and an extension layer composed of a second convolution filter and a third convolution filter; and performing decoding by concatenating the output information of the second fire module and the result calculated by the max unpooling obtained through residual concatenation.

In an embodiment, the image segmentation method using the residual convolution-based deep learning network, after performing the decoding, inputs the decoded value calculated by performing the decoding to a classification layer to obtain the final output value Calculating; may include.

In an embodiment, the classification layer may be composed of a softmax activation function.

In an embodiment, an image segmentation apparatus using a residual convolution-based deep learning network includes: an acquisition unit configured to acquire input information; The input information is input to a first fire module including a squeeze layer composed of a first convolution filter and an expand layer composed of a second convolution filter and a third convolution filter, and , a control unit that concatenates the output information of the first fire module and the input information obtained through residual connection, and performs encoding by max pooling the concatenated result value. ; may be included.

In an embodiment, the acquiring unit may acquire an input image, and the controller may generate the input information in the form of a patch by dividing the input image.

In an embodiment, the first convolution filter and the second convolution filter may have a first kernel size, and the third convolution filter may have a second kernel size.

In an embodiment, the control unit generates an output value of the first convolution filter of the squeeze layer, and inputs the generated output value to each of a second convolution filter and a third convolution filter configured in parallel of the extension layer, , by concatenating output values of each of the second convolution filter and the third convolution filter, output information of the first fire module may be generated.

In an embodiment, the control unit max unpools the encoding value calculated by performing the encoding, and uses a squeeze layer composed of a first convolution filter and a second convolution filter for a result value calculated by max unpooling. Input to a second Fire module including an extension layer composed of a convolution filter and a third convolution filter, and connect the output information of the second Fire module and the result calculated by max unpooling obtained through residual concatenation ( concatenate) to perform decoding.

In an embodiment, the controller may input a decoded value calculated by performing the decoding to a classification layer to calculate a final output value.

In an embodiment, the classification layer may be composed of a softmax activation function.

Specific details for achieving the above objects will become clear with reference to the embodiments to be described in detail below in conjunction with the accompanying drawings.

However, the present invention is not limited to the embodiments disclosed below, and may be configured in various different forms, and those of ordinary skill in the art to which the present invention pertains ( Hereinafter, "a person skilled in the art") is provided to fully inform the scope of the invention.

According to an embodiment of the present invention, it is possible to smoothly learn the deep learning network architecture through a residual unit.

According to an embodiment of the present invention, feature collection using a residual convolutional layer can ensure better feature representation in a segmented network.

According to an embodiment of the present invention, it is possible to design a design more efficiently with fewer network parameters and better segmentation accuracy for brain MRI.

The effects of the present invention are not limited to the above-described effects, and potential effects expected by the technical features of the present invention will be clearly understood from the following description.

1 is a diagram illustrating a residual connection according to an embodiment of the present invention.
2 is a diagram illustrating an image segmentation process using a residual convolution-based deep learning network according to an embodiment of the present invention.
3 is a diagram illustrating an image segmentation process using a residual convolution-based deep learning network according to an embodiment of the present invention.
4A is a diagram illustrating a fire module according to an exemplary embodiment of the related art.
4B is a diagram illustrating a fire module according to an embodiment of the present invention.
5 is a diagram illustrating a classification result image according to an embodiment of the present invention.
6 is a diagram illustrating a comparison of classification results based on a first data set according to an embodiment of the present invention.
7 is a diagram illustrating a comparison of classification results based on a second data set according to an embodiment of the present invention.
8 is a diagram illustrating a comparison of convolution units according to an embodiment of the present invention.
9 is a diagram illustrating an image segmentation method using a residual convolution-based deep learning network according to an embodiment of the present invention.
10 is a diagram illustrating an image segmentation apparatus using a residual convolution-based deep learning network according to an embodiment of the present invention.

Since the present invention can have various changes and can have various embodiments, specific embodiments are illustrated in the drawings and described in detail.

Various features of the invention disclosed in the claims may be better understood upon consideration of the drawings and detailed description. The apparatus, methods, preparations, and various embodiments disclosed herein are provided for purposes of illustration. The disclosed structural and functional features are intended to enable those skilled in the art to specifically practice the various embodiments, and are not intended to limit the scope of the invention. The terms and sentences disclosed are for the purpose of easy-to-understand descriptions of various features of the disclosed invention, and are not intended to limit the scope of the invention.

In describing the present invention, if it is determined that a detailed description of a related known technology may unnecessarily obscure the gist of the present invention, the detailed description thereof will be omitted.

Hereinafter, an image segmentation method and apparatus using a residual convolution-based deep learning network according to an embodiment of the present invention will be described.

1 is a diagram illustrating a residual connection according to an embodiment of the present invention.

Referring to FIG. 1 , a block diagram of a residual connection or a shortcut connection may be identified. That is, residual concatenation may mean skipping one or more layers.

Residual concatenation allows information to be passed from the previous layer to the next layer without loss of information and avoids the vanishing gradient problem.

2 is a diagram illustrating an image segmentation process using a residual convolution-based deep learning network according to an embodiment of the present invention.

Referring to FIG. 2 , in order to identify white matter (WM), gray matter (GM), and cerebrospinal fluid (CSF) in brain MRI, local detail may be more important than global information. Therefore, patch-based input for training of residual convolution-based deep learning networks can be used to better capture local details and obtain accurate tissue segmentation.

In operation 210, an input image composed of a plurality of slices may be acquired. For example, the input image may include a brain MRI image composed of a cross-sectional slice.

In operation 220, at least one input image may be extracted from among the plurality of input images. For example, at least one input image among a plurality of slice input images may be extracted.

In operation 230, the input image may be divided into a plurality of patch types of input information. In one embodiment, the dimensions of each input image may be composed of height×width×slice (H×W×S). For example, you can pad the HxW of each slice with zeros and resize it to 256x256. For example, an input image in the form of each slice can be divided into 4 uniform patches. For example, the size of each divided patch may be 128×128.

In step 240, these patches may be input to the residual convolution-based deep learning network for training to obtain predicted segmentation results for the test data.

That is, according to the present invention, smaller input patches can better reflect local details, and can train a residual convolution-based deep learning network to produce higher segmentation accuracy. Therefore, it is possible to divide the input MRI slice into uniform patches to capture fine details and feed these patches to residual convolution-based deep learning network training.

Therefore, we can use patch-wise segmentation of each input slice, which is used to train residual convolution-based deep learning networks and provide better segmentation accuracy.

For example, according to the present invention, an improved U-SegNet model by integrating U-SegNet with a fire module and residual convolution to segment brain tissue in magnetic resonance imaging (MRI) is provided. can be used

The residual convolution-based deep learning network according to the present invention utilizes the power, residual connection, and squeeze of U-SegNet and extends the convolution layer in the Fire module for segmentation of brain MRI. can

3 is a diagram illustrating an image segmentation process using the residual convolution-based deep learning network 300 according to an embodiment of the present invention. 4B is a diagram illustrating a fire module according to an embodiment of the present invention.

Referring to FIG. 3 , the residual convolution-based deep learning network 300 may include an encoder path, a decoder path, and a classification path. In addition, according to the present invention, a fire module (ie, the first fire module 310 and the second fire module 322) can be used to reduce the number of learnable parameters and the computational complexity in the encoder path and the decoder path. .

In the case of the encoder path, the input information in the form of a patch is composed of a squeeze layer 410 composed of a first convolution filter 412 , a second convolution filter 422 , and a third convolution filter 424 . An input may be performed to the first fire module 310 including an expand layer 420 .

In an embodiment, the first convolution filter 412 and the second convolution filter 422 may be configured with a first kernel size, and the third convolution filter 424 may be configured with a second kernel size. For example, the first kernel size includes a 1x1 kernel size, and the second kernel size includes a 3x3 kernel size, but is not limited thereto.

In an embodiment, the output value of the first convolution filter 412 of the squeeze layer 410 is generated, and the generated output value is used in parallel with the second convolution filter 422 and the third convolutional filter 422 of the extension layer 420 . The output information of the first fire module 310 is input to each of the convolution filters 424, and the output values of the second convolution filter 422 and the third convolution filter 424 are concatenated (426) respectively. can create

Output information of the first fire module 310 and input information obtained through residual connection may be concatenated 312 . Output data of layers separated from each other through the connection 312 may be collected through one single layer. In one embodiment, the operation of concatenation 312 may mean expanding two independent dimensions by merging them into one dimension.

In addition, encoding may be performed by max pooling (314) the result value calculated by the concatenation (312).

In one embodiment, the encoder path may consist of a plurality of first fire modules 310 and connections 312 and max pooling 314 .

In the case of the decoder path, a squeeze layer composed of a first convolution filter 412 for max unpooling (320) an encoding value calculated by performing encoding, and max unpooling (320) for a result calculated by the first convolution filter (412) The input may be performed to the second fire module 322 including the extension layer 420 including the 410 and the second convolution filter 422 and the third convolution filter 424 .

In an embodiment, decoding may be performed by concatenating ( 324 ) output information of the second fire module 322 and a result value calculated by max unpooling 320 obtained through residual concatenation.

In one embodiment, a skip connection that transfers information from one encoder (eg, the first fire module 310) to a decoder (eg, the second fire module 322) or a polling index transfer is performed. can

In the case of a classification path, after decoding is performed, a decoded value calculated by performing decoding may be input to a classification layer 330 to calculate a final output value. In one embodiment, the classification layer 330 may consist of a softmax activation function.

According to the present invention, the combination of fire module and residual connection is a new extension of the proposed U-SegNet model. Residual connections in the network allow passing important information from previous layers to the next.

According to the present invention, the convolutional layers in the encoder and decoder paths of the existing U-SegNet are replaced by fire modules in the proposed method, reducing the parameter and computational complexity.

According to the present invention, the proposed uniform patching input helps the network to focus on the details of individual patches and better retain local spatial information.

According to the present invention, our proposed patch-based residual-based squeeze U-SegNet model can easily obtain accuracy gains, showing much better results than conventional methods.

4A is a diagram illustrating a fire module according to an exemplary embodiment of the related art. 4B is a diagram illustrating a fire module according to an embodiment of the present invention.

필터를 포함하는 각 컨볼루션 블록이 있는 종래의 U-Net의 컨볼루션 레이어를 확인할 수 있으며, 높이x너비x채널의 특징 맵을 입력으로 사용할 수 있다. Referring to Figure 4a,

It is possible to check the convolutional layer of the conventional U-Net with each convolution block including the filter, and the feature map of height x width x channel can be used as input.

Referring to FIG. 4B , the fire model 400 according to the present invention may be composed of two parts: (i) a squeeze layer and (ii) an expand layer. For example, the fire model 400 may include a first fire model 310 and a second fire model 322 .

/4인 하나의 제1 컨볼루션 레이어(412)로 구성될 수 있다. 여기서

는 U-Net에서 사용되는 컨볼루션 필터의 수를 나타낼 수 있다. The squeeze layer 410 has a kernel size of 1x1 and an output channel

It may be composed of one first convolutional layer 412 equal to /4. here

may represent the number of convolution filters used in U-Net.

/2 출력 채널을 사용할 수 있다. The output of the squeeze layer 410 is fed to the extension layer 420, which consists of two parallel convolutions with kernel sizes of 3x3 and 1x1, namely the second convolutional layer 422 and the third convolutional layer 424. consists of , and each convolution is

/2 output channels are available.

Accordingly, the outputs of the parallel convolution may be connected to form the output of the fire module 400 .

In one embodiment, there is a series of fire modules in the encoder path, wherein the output of each fire module may be coupled with an input forming a residual connection.

을 입력 샘플로 표시하여, 이 경우, l은 잔차 컨볼루션 네트워크(residual convolutional network, RCNN)의 레이어를 나타낼 수 있다. 스퀴즈 블록의 컨볼루션 출력은 하기 <수학식 1>과 같이 제공됩니다.for example,

By representing as an input sample, in this case, l may represent a layer of a residual convolutional network (RCNN). The convolutional output of the squeeze block is provided as in <Equation 1> below.

는 파이어 모듈(400)의 스퀴즈 레이어(410)의 출력,

은 커널 가중치, 위 첨자 [1x1]은 각 레이어와 관련된 컨볼루션 커널의 크기,

은 바이어스 항을 나타낸다. here,

is the output of the squeeze layer 410 of the fire module 400,

is the kernel weight, the superscript [1x1] is the size of the convolution kernel associated with each layer,

is the bias term.

에 제공되며 하기 <수학식 2>로 표현될 수 있다. In one embodiment, the convolution output is a standard ReLU activation function

is provided and can be expressed by the following <Equation 2>.

In one embodiment, the output of the squeeze layer 410 is supplied in parallel to the 1x1 and 3x3 kernels of the extension layer 420 and the result is connected to generate the output of the fire module 400, and can be expressed together.

는 네트워크의 l 번째 파이어 모듈(400)의 출력을 나타낸다. 파이어 모듈(400)의 출력은 입력

과 연결되어 잔차 연결을 형성할 수 있다. here,

denotes the output of the l-th fire module 400 of the network. The output of the fire module 400 is an input

can be connected to form a residual link.

In an embodiment, the output of a residual convolutional network (RCNN) block may be calculated using Equation 4 below.

은 숏컷 연결에 의해 수행되고, 요소 별 추가(element-wise addition)는 학습할 잔차 매핑을 나타낼 수 있다. here,

is performed by shortcut concatenation, and element-wise addition may indicate a residual mapping to be learned.

은 RCNN 블록의 입력 샘플을 나타냅니다.

샘플은 각각 인코딩 및 디코딩 컨볼루션 단위에서 즉시 연속되는 맥스 풀링(max-pooling) 또는 언풀링 레이어(un-pooling layer)에 대한 입력으로 사용될 수 있다. here,

represents the input sample of the RCNN block.

Samples can be used as input to a max-pooling or un-pooling layer that is immediately continuous in encoding and decoding convolutional units, respectively.

In an embodiment, as shown in Equation 5 below, a residual block with max pooling may represent an encoder device.

Here, the pooling index is stored during each max pooling operation and may be used to pool the feature map in the decoder.

For the decoder path, the max pooling of the encoder path can be replaced with the max unpooling operation to recover the original input image.

/4 출력 채널이 있는 1x1 컨볼루션으로 구성될 수 있다. The decoder may use the second fire module 322 to reduce the model parameters. The second fire module 322, as in the encoder

It can be configured as a 1x1 convolution with /4 output channels.

/2 출력 채널이 연결되어 제2 파이어 모듈(322)의 출력을 형성할 수 있다. The output of the 1x1 convolution is fed into two parallel convolutions with 3x3 and 1x1 kernel sizes, each

The /2 output channel may be connected to form an output of the second fire module 322 .

Similar to the encoder, an output of the second fire module 322 may be connected to an input to form a residual block. The residual output may be unpooled with respect to the pooling index used to localize the feature map during max unpooling.

In an embodiment, the feature map obtained from the encoding layer including the context information may be connected with the feature map of the corresponding decoding layer forming a skip connection as shown in Equation 6 below.

Such skip linking can use both high and low resolution feature information and focus on the most relevant information while performing upsampling operations.

For the classification path, it may consist of a 1x1 convolutional layer with a softmax activation function that outputs a reconstructed image.

For example, the softmax layer can predict four output classes: GM, WM, CSF, and background. The input image can be classified into one of four output classes based on this feature representation. In one embodiment, cross entropy loss may be used to measure the loss of a deep learning network.

The softmax layer can learn the representation decoder (l) and interpret it as an output class. A probability score y' may be assigned to an output class.

In an embodiment, when the number of output classes is defined as c, it can be expressed as in Equation 7 below.

In an embodiment, the cross entropy loss function may be used to calculate the network cost as shown in Equation 8 below.

Here, y and y' represent the ground truth and predicted distribution scores for each class i.

That is, according to the present invention, the number of learnable parameters and computational requirements is reduced using a Fire module including a squeeze layer composed of a 1x1 convolution filter and an extension layer composed of 1x1 and 3x3 filters, and consequently a smaller efficient deep learning network model can be formed.

The proposed method is tested with two sets of brain MRI. For example, the first data set may include T1-weighted brain MRIs of 416 subjects obtained from the OASIS database. Of the total of 416, the first 30 were used for model training and the remaining 20 could be used as a test data set.

The second data set may include MRI of an Internet Brain Segmentation Repository (IBSR) data set. The training dataset contains 12 subjects with manually annotated and verified ground truth labels, the remaining 6 being able to test the model.

5 is a diagram illustrating a classification result image according to an embodiment of the present invention.

Referring to FIG. 5 , the division results for the axis, coronal plane, and sagittal plane of the first data set may be confirmed. 5(a) is the original input image, (b) is a ground truth segmentation map, (c) is a predicted classification map, (d) is a predicted CSF (binary map), (e) ) denotes predicted GM (binary map), and (f) denotes predicted WM (binary map).

That is, it can be confirmed from the results that the method according to the present invention achieves well-refined results for GM, WM, and CSF.

In addition, in order to objectively measure the segmentation performance, the performance of the method according to the present invention can be evaluated using the quantitative indicators detailed in Table 1 below.

Dice similarity
coefficient (DSC)

Jaccard Index (JI)

Hausdorff
distance (HD)

Mean square error
(MSE)

및

Ground Truth 세트 및 예측된 분할 세트(즉, 각 세트의 요소 수)의 카디널리티를 나타낸다. DSC and JI are commonly used to compare volumes based on redundancy and can be used to compare results from ground truth and automated segmentation methods. DSC can be defined as twice the number of elements common to both sets divided by the sum of the number of elements in each set. here

and

It represents the cardinality of the ground truth set and the predicted split set (i.e. the number of elements in each set).

JI can be expressed in DSC as described in <Table 1>. The DSC and JI metrics may determine a match between the predicted segmentation map and the corresponding ground truth segmentation map.

In addition, segmentation performance can be measured in terms of mean square error (MSE), which is a mean square difference between the original x value and the predicted Y value.

HD can be used to determine dissimilarity between two sets in metric space. The two sets of small HDs can look almost identical.

HD and MSE can be calculated as shown in <Table 1>. Here, D means the Euclidean distance of two pixels, and R and C may represent the image height and width, respectively.

To compare the segmentation performance for different network architectures, conventional U-Net, SegNet and U-SegNet models can be trained on the same data set.

6 is a diagram illustrating a comparison of classification results based on a first data set according to an embodiment of the present invention. 7 is a diagram illustrating a comparison of classification results based on a second data set according to an embodiment of the present invention.

6 and 7 , it is possible to compare the split results for the conventional U-Net, SegNet, and U-SegNet with the method according to the present invention (Proposed Method).

That is, it can be seen that the quality of the segmentation map generated by the method according to the present invention is much higher than that of other existing methods.

U-Net has a shallow network and may not be sufficient to capture image spatial information. Complex image textures can significantly reduce the segmentation accuracy of U-Net and SegNet. In particular, it can be seen in the feature maps generated by SegNet and U-Net of FIGS. 6 (c) and 7 (c), and the highlighted area caused a misclassification due to the concentration of a specific tissue. Nevertheless, it can be seen that U-SegNet produces better segmentation results with the combination of index and skip connection, but does not capture fine details.

In the highlighted red box, U-SegNet can usually see subsegments of WM and CSF organizations. Integrating residual linkage can overcome these limitations, reduce overfitting, and improve segmentation performance.

In addition, uniform patches can focus on relevant regions and capture better feature representations, improving the performance of the proposed method.

This improved segmentation can be observed in the results obtained with the method according to the invention. Similar results can be observed for segmentation obtained from IBSR images.

In particular, it can be seen that the residual convolution-based deep learning network according to the present invention can obtain detailed information than other architectures. These visual results confirm that the method according to the present invention can robustly recover coarser and finer subdivision details while bypassing the distraction of ambiguous areas.

Quantitative analysis of the method according to the present invention was performed in comparison with the existing SegNet, U-Net, and U-SegNet methods, and the results can be shown in <Table 2> to <Table 5>.

OASIS dataset Axial plane parameter W.M. SegNet U-net U-Segnet Proposed
Method DSC (%) 87.36 92.18 93.36 96.09 JI (%) 77.18 85.50 87.56 92.48 HD 5.09 4.40 4.2 3.4 Coronal plane DSC (%) 82.12 93.14 94.12 96.37 JI (%) 69.23 87.20 89.63 91.17 HD 5.4 4.14 3.9 3.2 Sagittal plane DSC (%) 82.42 92.44 93.25 96.33 JI (%) 69.54 86.34 87.65 91.38 HD 7.2 4.3 4.0 3.35 IBSR dataset Axial plane DSC (%) 72.86 89.45 90.35 91.76 JI (%) 65.34 81.56 82.49 84.23 HD 6.51 5.14 4.6 4.2 Coronal plane DSC (%) 70.15 88.45 89.36 90.47 JI (%) 62.35 79.38 80.14 82.53 HD 6.3 5.45 5.5 4.6 Sagittal plane DSC (%) 71.53 86.84 87.32 89.52 JI (%) 63.41 78.63 79.62 81.42 HD 6.49 5.75 5.4 4.82

OASIS dataset Axial plane parameter GM SegNet U-net U-Segnet Proposed
Method DSC (%) 84.93 90.32 82.05 94.04 JI (%) 72.20 82.35 85.27 90.55 HD 5.7 4.3 4.0 3.7 Coronal plane DSC (%) 78.21 82.25 93.53 94.94 JI (%) 64.16 85.45 87.85 90.46 HD 4.6 4.2 4.12 3.42 Sagittal plane DSC (%) 80.23 91.13 92.56 95.05 JI (%) 67.57 83.26 85.69 90.53 HD 5.9 5.2 4.2 3.49 IBSR dataset Axial plane DSC (%) 75.63 91.53 92.20 93.83 JI (%) 67.32 85.41 86.15 87.52 HD 6.53 4.87 4.2 4.0 Coronal plane DSC (%) 73.65 90.23 91.45 92.85 JI (%) 65.42 83.56 84.16 85.61 HD 6.21 5.17 4.8 4.53 Sagittal plane DSC (%) 74.62 89.46 90.44 91.71 JI (%) 66.85 81.53 82.15 84.45 HD 6.36 5.77 5.3 4.25

OASIS dataset Axial plane parameter CSF SegNet U-net U-Segnet Proposed
Method DSC (%) 80.67 89.82 91.64 93.88 JI (%) 67.85 81.53 84.57 90.26 HD 4.9 4.6 4.1 3.6 Coronal plane DSC (%) 74.03 89.53 91.46 93.62 JI (%) 61.26 82.36 84.25 89.27 HD 4.6 4.1 4.15 3.81 Sagittal plane DSC (%) 77.45 88.63 92.15 94.31 JI (%) 63.51 81.26 85.36 89.44 HD 6.3 4.4 4.15 3.56 IBSR dataset Axial plane DSC (%) 68.42 84.34 84.95 85.64 JI (%) 59.32 75.85 75.98 77.16 HD 6.3 4.4 4.3 4.86 Coronal plane DSC (%) 66.54 83.65 84.15 85.83 JI (%) 57.32 76.86 76.94 77.86 HD 6.84 5.54 5.2 4.9 Sagittal plane DSC (%) 65.49 80.75 81.19 83.56 JI (%) 54.86 73.96 74.10 75.28 HD 6.99 5.83 5.6 5.1

MSE SegNet U-net U-Segnet Proposed
Method OASIS 0.021 0.008 0.006 0.003 IBSR 0.013 0.009 0.007 0.005

The deep learning network according to the present invention achieved average 10%, 3.9%, and 2.3% improvement in DSC compared to the conventional SegNet, U-Net, and U-SegNet methods, and lower than SegNet, U-Net and U-SegNet. We achieved an MSE value of 0.003.

This performance difference can be explained by the fact that SegNet only stores max pooling indexes. That is, the position of the maximum feature value in each pooling window is memorized for each encoder map and can be used in a lower-level feature map for upsampling. Therefore, translation invariance can often be compromised.

In addition, U-SegNet tends to be insensitive to fine details and has difficulty in identifying boundaries between adjacent tissues such as WM and GM. The segmentation map generated by such an existing model may have a relatively low resolution due to the pooling layer in the encoder stage. Therefore, it may be necessary to remove the pooling layer to maintain high spatial resolution.

However, since convolution is a local operation, SegNet, U-Net, and U-SegNet models may not be able to learn global features from the image without layer pooling.

The method according to the present invention using the residual convolution based deep learning network combined with the Fire module can be a potential solution to the above-mentioned problem and can produce improved segmentation accuracy.

1x1, 3x3 kernels and convolutional inputs connected together in the Fire module can capture the global context without reducing the resolution of the segmentation map.

The information obtained from the Fire module can be further combined with input through residual linkage to improve feature representation.

In this way, global information can be exchanged between layers without sacrificing resolution and the blurring of the semantic segmentation map can be reduced.

Uniform input patching also allows the network to focus more on local details. As a result of selective integration of spatial information through uniform patching, feature maps and residual connections can help to efficiently capture contextual information.

<Table 6> may indicate the learnable parameters and calculation time consumed by the method according to the present invention compared to the existing method.

Models #learnable
parameters computation
time(hrs) SegNet 3,502,028 2.9 U-Net 4,832,324 3.5 U-SegNet 4,279,172 3.1 Proposed method 4,290,717 3.6

In one embodiment, a smaller model can be built by arranging a series of fire modules consisting of a squeeze layer with only a 1x1 convolution filter.

A 1x1 filter in the squeeze layer can reduce parameters by downsampling the input channel before being fed as input to the extension layer.

The 1x1 filter in the extension layer combines channels and performs cross-channel pooling, but cannot recognize spatial structures.

The 3x3 convolution filter in the extension layer can discriminate spatial representations, and combining these two size filters allows the model to be more expressive while working with fewer parameters.

Therefore, the Fire module can reduce the parameter map to build a smaller CNN network, which can reduce the computational load and maintain higher accuracy.

The method according to the present invention includes more layers than the conventional method, but has fewer parameters than U-Net and may require the same number of parameters as the U-SegNet model. The training time of the method according to the present invention is 3.6 hours, which can be almost equal to that of the U-Net model.

8 is a diagram illustrating a comparison of convolution units according to an embodiment of the present invention.

Referring to FIG. 8 , an ablation study on a method integrated with different convolutional units for the method according to the present invention can be performed to confirm the effect of each selection on the segmentation performance.

8, (a) is a forward convolution unit, (b) is a squeeze convolution unit, (c) is a residual convolution unit, (d) is a squeeze residual convolution unit (Proposed method).

In one embodiment, all these convolutional units may be implemented considering U-SegNet as a basic network. The first model can be constructed as a forward convolution with ReLU activation in both the encoder and decoder of the U-SegNet network.

The second model can be constructed by replacing the forward convolution with a fire module including a squeeze and extension layer as shown in FIG. 4(b).

The third model may be formed by connecting the forward convolutional output to the input by forming a residual connection as shown in FIG. 8(c).

In the U-SegNet network, which is the method according to the present invention, a combination of two fire modules and residual connection can be used.

In an embodiment, <Table 7> to <Table 10> may represent the performance of these four convolutional units in terms of DSC scores, number of model parameters, and time required for model training.

Models GM DSC JI HD U-SegNet with forward convolution unit 92.05 85.27 93.36 U-SegNet with Fire module 93.72 86.06 93.65 U-SegNet with residual connection 94.09 88.58 95.11 Proposed Method 95.04 90.55 96.09

Models W.M. DSC JI HD U-SegNet with forward convolution unit 93.36 87.56 4.2 U-SegNet with Fire module 93.65 88.42 2.8 U-SegNet with residual connection 95.11 90.68 3.4 Proposed Method 96.09 92.48 3.4

Models CSF DSC JI HD MSE U-SegNet with forward convolution unit 91.64 84.57 4.1 0.006 U-SegNet with Fire module 92.15 85.66 2.0 0.006 U-SegNet with residual connection 93.53 89.99 3.3 0.005 Proposed Method 94.88 90.26 2.8 0.003

computation
time (5 epochs) #learnable
parameters U-SegNet with forward convolution unit 3.1 hours 4,279,172 U-SegNet with Fire module 1.7 hours 768,788 U-SegNet with residual connection 14 hours 21,138,205 Proposed Method 3.6 hours 4,290,717

The model with simple forward convolution shows an overall DSC score of 92.35% with 4 million parameters, confirming that it requires 3.1 hours of model training.

It can be seen that the same model replaced by the Fire module increased the accuracy by 0.5%, but reduced the learnable parameters by 5.5 times and reduced the model training time by 50%.

Unlike forward convolution, shortcut connections are always active and gradients can easily propagate back, which can improve accuracy.

However, this residual-based model has 21 million parameters and it can take 14 hours to train four times more models than the reference U-SegNet.

Residual concatenation gives better accuracy, but may not be a good choice for image segmentation tasks due to large parameter generation and high time complexity.

To overcome this problem, the present invention can use the Fire module in the residual network. It can be seen that the Fire module-based network reduces the computation time for model training while maintaining the network accuracy, thereby significantly reducing the requirements for learnable parameters.

Fire module with 1x1 convolution on squeeze layer helps reduce dimensionality and helps speed up model convergence, extension layer is combined with 1x1 and 3x3 in parallel to help avoid gradient loss caused by squeeze layer this can be

Cooperating with the Fire module and residual concatenation, it can be seen that the network improved by 2.5% compared to the baseline U-SegNet with an overall DSC score of 95% and a training time of only 3.6 hours.

Therefore, the shortened connection allows the residual information to be transmitted through each network layer, thereby increasing the segmentation accuracy, and the Fire module reduces the model parameters and computation time, confirming that the proposed method is superior to the existing method.

We also experiment with the effect of patch size on training time and segmentation performance, experiments can be performed on the OASIS dataset for three different patch sizes (128×128, 64×64 and 32×32).

In an embodiment, <Table 11> may indicate the segmentation performance of the DSC for various patch sizes.

Patch
size DSC HI Computation time (hours) W.M. GM CSF W.M. GM CSF 128x128 96.33 95.44 94.81 92.93 91.28 90.14 3.6 64x64 96.35 95.48 94.89 92.95 91.37 90.28 5.4 32x32 96.35 95.51 95.12 92.98 91.47 90.34 7.2

It can be seen that the smaller the patch size, the better the performance. This may be because small patches generate more training data for the network to train.

Moreover, the local area can be reconstructed more accurately. Also, if the patch size is 128×128, it takes 3.6 hours to train the model, whereas for a 32×32 patch with almost the same accuracy, the training time can be doubled.

Therefore, it can be confirmed that the 128x128 patch size provides an appropriate balance between the DSC score and the computation time required for model training according to the results in <Table 11>.

9 is a diagram illustrating an image segmentation method using a residual convolution-based deep learning network according to an embodiment of the present invention.

Referring to FIG. 9 , step S901 is a step of acquiring input information.

In an embodiment, the input information may be generated in the form of a patch by acquiring an input image and dividing the input image. Here, the input image may include a slice image of an object. Also, the input information may include a patch image obtained by dividing a slice image into a plurality of patch types.

In step S903, the input information is expanded with the squeeze layer 410 composed of the first convolution filter 412, the second convolution filter 422 and the third convolution filter 424. It is a step of inputting the first fire module 310 including the layer) 420 .

In an embodiment, the first convolution filter 412 and the second convolution filter 422 may be configured with a first kernel size, and the third convolution filter 424 may be configured with a second kernel size. For example, the first kernel size includes a 1x1 kernel size, and the second kernel size includes a 3x3 kernel size, but is not limited thereto.

In an embodiment, the output value of the first convolution filter 412 of the squeeze layer 410 is generated, and the generated output value is used in parallel with the second convolution filter 422 and the third convolutional filter 422 of the extension layer 420 . The output information of the first fire module 310 is input to each of the convolution filters 424, and the output values of the second convolution filter 422 and the third convolution filter 424 are concatenated (426) respectively. can create

Step S905 is a step of concatenating ( 312 ) output information of the first fire module 310 and input information obtained through residual connection.

Step S907 is a step of performing encoding by max-pooling (314) the result value calculated by the connection (312).

In one embodiment, after step S907, the encoding value calculated by performing encoding is max unpooled (320), and the result calculated by max unpooling (320) is applied to the first convolution filter (412). ) is input to the second fire module 322 including the squeeze layer 410 composed of Decoding may be performed by concatenating ( 324 ) the output information of the module 322 and the result calculated by max unpooling 320 obtained through residual concatenation.

In an embodiment, after decoding is performed, a decoded value calculated by performing decoding may be input to a classification layer 330 to calculate a final output value.

In one embodiment, the classification layer 330 may consist of a softmax activation function.

10 is a diagram illustrating an image segmentation apparatus 1000 using a residual convolution-based deep learning network according to an embodiment of the present invention.

Referring to FIG. 10 , the image dividing apparatus 1000 may include an acquisition unit 1010 , a control unit 1020 , and a storage unit 1030 .

The acquisition unit 1010 may acquire input information.

In an embodiment, the acquisition unit 1010 may include a communication unit or a photographing unit. For example, the acquisition unit 1010 may include a magnetic resonance imaging (MRI) camera. Also, the acquisition unit 1010 may include a communication unit that receives data from an external electronic device.

In an embodiment, the communication unit may include at least one of a wired communication module and a wireless communication module. All or part of the communication unit may be referred to as a 'transmitter', 'receiver' or 'transceiver'.

The control unit 1020 includes a squeeze layer 410 composed of a first convolution filter 412 and an extension layer 420 composed of a second convolution filter 422 and a third convolution filter 424 for input information. may be input to the first fire module 310 .

In addition, the controller 1020 connects (312) the output information of the first fire module 310 and the input information obtained through the residual connection, and max-pools (314) the result calculated by the connection (312). encoding can be performed.

In an embodiment, the controller 1020 may include at least one processor or microprocessor, or may be a part of the processor. Also, the controller 1020 may be referred to as a communication processor (CP). The controller 1020 may control the operation of the image dividing apparatus 1000 according to various embodiments of the present disclosure.

The storage unit 1030 may store input information. Also, the storage unit 1030 may store a residual convolution-based deep learning network.

In an embodiment, the storage unit 1030 may be configured as a volatile memory, a non-volatile memory, or a combination of a volatile memory and a non-volatile memory. In addition, the storage unit 1030 may provide the stored data according to the request of the control unit 1020 .

Referring to FIG. 10 , the image dividing apparatus 1000 may include an acquisition unit 1010 , a control unit 1020 , and a storage unit 1030 . In various embodiments of the present disclosure, the image segmentation apparatus 1000 is not essential to the components illustrated in FIG. 10 , and thus may be implemented as having more or fewer components than those illustrated in FIG. 10 . have.

The above description is merely illustrative of the technical idea of the present invention, and various changes and modifications may be made by those skilled in the art without departing from the essential characteristics of the present invention.

The various embodiments disclosed herein may be performed out of order, and may be performed simultaneously or separately.

In an embodiment, at least one step may be omitted or added in each figure described herein, may be performed in the reverse order, or may be performed simultaneously.

The embodiments disclosed in the present specification are not intended to limit the technical spirit of the present invention, but to illustrate, and the scope of the present invention is not limited by these embodiments.

The protection scope of the present invention should be interpreted by the claims, and all technical ideas within the scope equivalent thereto should be understood to be included in the scope of the present invention.

300: Residual Convolution Based Deep Learning Network
310: first fire module
312: connect
314: Max Pooling
320: max unpooling
322: second fire module
324: connect
330: classification layer
400: fire model
410: squeeze layer
412: first convolutional layer
420: extension layer
422: second convolutional layer
424: third convolution layer
426: connect
1000: video splitting device
1010: Acquisition Department
1020: control unit
1030: storage

Claims (14)

(a) obtaining input information;
(b) a first fire module including a squeeze layer composed of a first convolution filter and an expand layer composed of a second convolution filter and a third convolution filter for the input information to enter into;
(c) concatenating the output information of the first fire module and the input information obtained through residual connection; and
(d) performing encoding by max pooling the result values calculated by concatenation;
containing,
An image segmentation method using a residual convolution-based deep learning network.
According to claim 1,
The step (a) is,
acquiring an input image; and
generating the input information in the form of a patch by dividing the input image;
containing,
An image segmentation method using a residual convolution-based deep learning network.
According to claim 1,
The first convolution filter and the second convolution filter are configured with a first kernel size,
The third convolution filter is composed of a second kernel size,
An image segmentation method using a residual convolution-based deep learning network.
According to claim 1,
Step (b) is,
generating an output value of a first convolution filter of the squeeze layer;
inputting the generated output value to each of a second convolution filter and a third convolution filter configured in parallel of the extension layer; and
generating output information of the first fire module by concatenating output values of each of the second convolution filter and the third convolution filter;
containing,
An image segmentation method using a residual convolution-based deep learning network.
According to claim 1,
After step (d),
max unpooling the encoding value calculated by performing the encoding;
inputting the result calculated by the max unpooling to a second fire module including a squeeze layer composed of a first convolution filter and an extension layer composed of a second convolution filter and a third convolution filter; and
performing decoding by concatenating the output information of the second fire module and the result value calculated by max unpooling obtained through residual concatenation;
containing,
An image segmentation method using a residual convolution-based deep learning network.
6. The method of claim 5,
After performing the decoding step,
calculating a final output value by inputting the decoded value calculated by performing the decoding to a classification layer;
containing,
An image segmentation method using a residual convolution-based deep learning network.
7. The method of claim 6,
The classification layer is composed of a softmax activation function,
An image segmentation method using a residual convolution-based deep learning network.
an acquisition unit for acquiring input information;
The input information is input to a first fire module including a squeeze layer composed of a first convolution filter and an expand layer composed of a second convolution filter and a third convolution filter, and ,
concatenates the output information of the first fire module and the input information obtained through residual connection;
a control unit for performing encoding by max pooling the result values calculated by the connection;
containing,
An image segmentation device using a residual convolution-based deep learning network.
9. The method of claim 8,
The acquisition unit acquires an input image,
The control unit divides the input image to generate the input information in the form of a patch,
An image segmentation device using a residual convolution-based deep learning network.
9. The method of claim 8,
The first convolution filter and the second convolution filter are configured with a first kernel size,
The third convolution filter is composed of a second kernel size,
An image segmentation device using a residual convolution-based deep learning network.
9. The method of claim 8,
The control unit is
generating an output value of the first convolution filter of the squeeze layer;
inputting the generated output value to each of a second convolution filter and a third convolution filter configured in parallel of the extension layer,
generating output information of the first fire module by concatenating the output values of each of the second convolution filter and the third convolution filter;
An image segmentation device using a residual convolution-based deep learning network.
9. The method of claim 8,
The control unit is
Max unpooling the encoding value calculated by performing the encoding,
The result calculated by the max unpooling is input to a second fire module including a squeeze layer composed of a first convolution filter and an extension layer composed of a second convolution filter and a third convolution filter,
decoding is performed by concatenating the output information of the second fire module and the result value calculated by max unpooling obtained through residual concatenation,
An image segmentation device using a residual convolution-based deep learning network.
13. The method of claim 12,
The control unit is
Inputting the decoded value calculated by performing the decoding to a classification layer to calculate a final output value,
An image segmentation device using a residual convolution-based deep learning network.
14. The method of claim 13,
The classification layer is composed of a softmax activation function,
An image segmentation device using a residual convolution-based deep learning network.

Images