KR102561214B1 - A method and apparatus for image segmentation using global attention - Google Patents

Abstract

The present invention relates to a method and apparatus for segmenting an image using global attention. An image segmentation method using global attention according to an embodiment of the present invention includes the steps of (a) obtaining input information; (b) a first fire module including a squeeze layer composed of a first convolution layer, an expand layer composed of a second convolution layer and a third convolution layer, inputting the input information to a fire module; (c) inputting output information and multi-scale input information of the first fire module to a first global attention module including global average pooling; and (d) performing encoding by max pooling output information of the first global attention module.

Description

Method and apparatus for image segmentation using global attention {A method and apparatus for image segmentation using global attention}

The present invention relates to an image segmentation method and apparatus, and more particularly, to an image segmentation method and apparatus using global attention.

Magnetic resonance imaging (MRI) has high contrast and relatively high resolution. Therefore, the new technique is widely used to examine the human brain in clinical applications and scientific research.

In this case, automatic segmentation of brain tissue into white matter (WM), gray matter (GM), and cerebrospinal fluid (CSF) is very important in the aforementioned task.

Accurate tissue segmentation is challenging due to complex brain structures and tissue heterogeneity due to MRI noise, deflection fields, and partial volume effects.

Strategies using deep learning networks to solve these problems are being used for segmentation tasks due to their associated advantages, but research on them is insufficient.

[Patent Document 1] Korea Patent Registration No. 10-2089014

The present invention has been created to solve the above problems, and an object of the present invention is to provide a method and apparatus for segmenting an image using global attention.

Another object of the present invention is to provide an image segmentation method and apparatus using multi-scale global attention and a patch method input integrated with a fire module.

The objects of the present invention are not limited to the objects mentioned above, and other objects not mentioned will be clearly understood from the description below.

In order to achieve the above objects, an image segmentation method using global attention according to an embodiment of the present invention includes the steps of (a) obtaining input information; (b) a first fire module including a squeeze layer composed of a first convolution layer, an expand layer composed of a second convolution layer and a third convolution layer, inputting the input information to a fire module; (c) inputting output information and multi-scale input information of the first fire module to a first global attention module including global average pooling; and (d) performing encoding by max pooling output information of the first global attention module; can include

In an embodiment, step (a) may include obtaining an input image; and generating the input information in the form of a patch by dividing the input image.

In an embodiment, the first convolution layer and the second convolution layer may have a first kernel size, and the third convolution layer may have a second kernel size.

In an embodiment, the step (b) may include generating an output value of a first convolution layer of the squeeze layer; inputting the generated output value to each of a second convolution layer and a third convolution layer configured in parallel of the enhancement layer; and generating output information of the first fire module by concatenating output values of each of the second convolution layer and the third convolution layer.

In an embodiment, the image segmentation method using global attention may include generating input information for a second convolution layer by performing max pooling on the input information for the first convolution layer before the step (c); The method may further include generating multi-scale input information for the second layer by inputting input information for the second convolution layer to each of a fourth convolution layer and a fifth convolution layer configured in parallel.

In an embodiment, the step (c) may include: pooling the global average of the output information of the first fire module; inputting a resultant value of the global average pooling to a sixth convolution layer; Generating an attention coefficient by performing upsampling based on a result value of the sixth convolution layer and a result value generated by inputting the multi-scale input information to a seventh convolution layer; and generating output information of the first global attention module using the attention coefficient and output information of the first fire module.

In an embodiment, the image segmentation method using global attention may, after step (d), output information of a first fire module for the first convolution layer and output information of a second fire module for the second convolution layer Inputting information to a second global attention module for the first convolution layer; Further, the second fire module may include a squeeze layer composed of a first transposed convolution layer, an extension layer composed of a second transposed convolution layer, and a third transposed convolution layer.

In an embodiment, in the image segmentation method using global attention, after step (d), output information of the second global attention module and output information of the second fire module for the second layer are upsampled; and calculating a final output value by inputting the decoded value calculated by performing the decoding to a classification layer. may further include.

In an embodiment, the step of inputting the input to the second global attention module may include generating an output value of a first transpose convolution layer of the squeeze layer; inputting the generated output value to a second transpose convolution layer and a third transpose convolution layer configured in parallel of the enhancement layer, respectively; and generating output information of the second fire module by concatenating output values of the second transpose convolution layer and the third transpose convolution layer, respectively.

In an embodiment, an apparatus for segmenting an image using global attention may include: an acquisition unit that obtains input information; and inputting the input information to a first fire module including a squeeze layer composed of a first convolution layer and an expand layer composed of a second convolution layer and a third convolution layer, and inputting the output information and multi-scale input information of the first fire module to a first global attention module including global average pooling, and output information of the first global attention module to the max It may include a control unit that performs encoding by pooling (max pooling).

In an embodiment, the acquiring unit may acquire an input image, and the control unit may generate the input information in a patch form by dividing the input image.

In an embodiment, the first convolution layer and the second convolution layer may have a first kernel size, and the third convolution layer may have a second kernel size.

In an embodiment, the controller may generate an output value of the first convolution layer of the squeeze layer, input the generated output value to each of the second convolution layer and the third convolution layer configured in parallel of the enhancement layer, and concatenate the output values of the second convolution layer and the third convolution layer to generate output information of the first fire module.

In an embodiment, the control unit may perform max pooling on the input information for the first convolution layer to generate input information for the second convolution layer, and generate multi-scale input information for the second convolution layer by inputting the input information for the second convolution layer to each of the fourth convolution layer and the fifth convolution layer configured in parallel.

In an embodiment, the control unit performs global average pooling on output information of the first fire module, inputs a resultant value of the global average pooling to a sixth convolution layer, and generates an attention coefficient by performing upsampling based on a resultant value generated by inputting the resultant value of the sixth convolution layer and the multi-scale input information to a seventh convolution layer.

In an embodiment, the controller inputs output information of a first fire module for the first convolution layer and output information of a second fire module for the second convolution layer to a second global attention module for the first convolution layer, and the second fire module includes a squeeze layer composed of a first transposed convolution layer, a second transposed convolution layer, and an extension layer composed of a third transposed convolution layer. there is

In an embodiment, the control unit may perform decoding by inputting a result value generated by upsampling output information of the second global attention module and output information of the second fire module for the second convolution layer to a second fire module for the first convolution layer, and inputting the decoded value calculated by performing the decoding to a classification layer to calculate a final output value.

In an embodiment, the control unit may generate an output value of the first transpose convolution layer of the squeeze layer, and input the generated output value to the second transpose convolution layer and the third transpose convolution layer configured in parallel of the enhancement layer, respectively.

Specific details for achieving the above objects will become clear with reference to embodiments to be described later in detail in conjunction with the accompanying drawings.

However, the present invention is not limited to the embodiments disclosed below, but may be configured in a variety of different forms, so that the disclosure of the present invention is complete and those skilled in the art to which the present invention belongs (hereinafter "ordinary technician") is provided to fully inform the scope of the invention.

According to one embodiment of the present invention, a network operating on patchy inputs integrated with multi-scale global attention and fire modules can perform efficient brain MRI segmentation.

The effects of the present invention are not limited to the above-mentioned effects, and the potential effects expected by the technical features of the present invention will be clearly understood from the description below.

1 is a diagram illustrating an image segmentation process using a global attention-based deep learning network according to an embodiment of the present invention.
2A is a diagram illustrating an encoder fire module according to a conventional embodiment.
2B is a diagram illustrating a decoder fire module according to a conventional embodiment.
2c is a diagram illustrating an encoder fire module according to an embodiment of the present invention.
2d is a diagram illustrating a decoder fire module according to an embodiment of the present invention.
3 is a diagram illustrating an image segmentation process using a global attention-based deep learning network according to an embodiment of the present invention.
4 is a diagram illustrating a global attention module according to an embodiment of the present invention.
5 is a diagram illustrating a classification result based on a first data set according to an embodiment of the present invention.
6 is a diagram illustrating a classification result based on a second data set according to an embodiment of the present invention.
7 is a diagram illustrating comparison of classification results based on a first data set according to an embodiment of the present invention.
8 is a diagram illustrating comparison of classification results based on a second data set according to an embodiment of the present invention.
9 is a diagram illustrating comparison between the number of learning parameters and calculation time according to an embodiment of the present invention.
10 is a diagram illustrating an image segmentation method using global attention according to an embodiment of the present invention.
11 is a diagram showing a functional configuration of an image segmentation apparatus using global attention according to an embodiment of the present invention.

Since the present invention can make various changes and have various embodiments, specific embodiments will be illustrated in the drawings and described in detail.

Various features of the invention disclosed in the claims may be better understood in consideration of the drawings and detailed description. Devices, methods, manufacturing methods, and various embodiments disclosed in the specification are provided for illustrative purposes. The disclosed structural and functional features are intended to enable a person skilled in the art to specifically implement various embodiments, and are not intended to limit the scope of the invention. The disclosed terms and phrases are intended to provide an easy-to-understand description of the various features of the disclosed invention, and are not intended to limit the scope of the invention.

In describing the present invention, if it is determined that a detailed description of related known technologies may unnecessarily obscure the subject matter of the present invention, the detailed description will be omitted.

Hereinafter, an image segmentation method and apparatus using global attention according to an embodiment of the present invention will be described.

1 is a diagram illustrating an image segmentation process using a global attention-based deep learning network according to an embodiment of the present invention.

Referring to FIG. 1 , in step S110 , an input image (eg, MRI scan data) may be collected along with a corresponding ground truth. For example, each scan has a dimension height width slice (H W S) can be. H for each slice Fill W with zeros and set the size to 256 256 can be adjusted.

In step S120, at least one (eg, 48) slices may be extracted from a specific (eg, 10th) slice at a slice interval of a predetermined number (eg, 3).

In one embodiment, each slice is divided into 4 uniform patches and these patches can be provided as input to the global attention-based deep learning network model according to the present invention for training.

In step S130, after training the global attention-based deep learning network model, a test input is supplied to the model and a predicted segmentation output may be obtained.

The architecture according to the present invention can be discussed in detail as (i) Uniform Patched Input, (ii) Encoder Module, (iii) Decoder Module and (iv) Global Attention Module (GAM).

In terms of uniform patchy input, the local information can be very important than the global information identifying WM, GM and CSF in brain MRI. Patch-based inputs for model training can be used to capture better local details and obtain accurate tissue segmentation.

For example, each subject's brain MRI scan is dimension H W It consists of S. Some of the beginning and ending segments of brain MRI scans don't contain much useful information, as previous studies have investigated, and successive segments can share almost the same information.

Therefore, to reduce multi-learning of consecutive slices, excluding slices without such information, we selected 48 slices with a spacing of 3 slices ensuring the existence of slices with more or less information for model training. Each slice extracted is 256 256 will be resized.

Splitting slices into individual patches can improve localization, as a trained global attention-based deep learning network can focus more on the local details of a patch. Thus, each slice can be divided into 4 uniform patch inputs. These patch inputs can be input to a global attention-based deep learning network model for training, and a predicted segmentation result for test data can be obtained.

2A is a diagram illustrating an encoder fire module according to a conventional embodiment. 2B is a diagram illustrating a decoder fire module according to a conventional embodiment. 2c is a diagram illustrating the encoder fire module 210 according to an embodiment of the present invention. 2d is a diagram illustrating the decoder fire module 220 according to an embodiment of the present invention.

Referring to Figures 2a and 2b, it can be seen that the convolutional layer on the encoder and decoder side of the conventional U-net having each convolution block including the F _i filter, width A feature map of the channel can be used as input.

Referring to FIGS. 2C and 2D , the encore fire module 210 used on the encoder side and the decoder fire model 220 used on the decoder side according to the present invention can be confirmed.

The fire module may include (i) a squeeze layer and (ii) an expand layer.

As shown in Figure 2c, the squeeze module 211 has a kernel size of 1 1 and an output channel of F _i/4 . Here, F _i may represent the number of convolution filters. An output of the squeeze layer 211 may be supplied to the enhancement layer 213 .

The extension layer 213 has a kernel size of 1 1 and 3 3 is composed of two parallel convolutions (the second convolution layer 214 and the third convolution layer 215), and each convolution may be an F _i/2 output channel. The outputs of these parallel convolutions can also be concatenated 216 to form the output of the encoder fire module 210 .

Referring to FIG. 2D , the decoder path may reduce model parameters using a transposed decoder fire module 220 . A major component of the decoder path may include upsampling.

Each upsampling may consist of a transposed decoder fire module 220 as shown in FIG. 2d. In one embodiment, the decoder fire module 220 may be referred to as a transphos fire module or a term having an equivalent technical meaning.

The decoder fire module 220 has a F _{i / 4} output channel included in the squeeze layer 221 1 It may consist of a first transpose convolution layer 222 of 1. The output of the first transpose convolution layer 222 is included in the extension layer 223 connected to form the output of the decoder fire module 220 as in the down sampling unit, and the F _{i / 2} output channel and 3 respectively 3 and 1 It can be fed into two parallel transpose convolution layers (second transpose convolution layer 224 and second transpose convolution layer 225) with 1 kernel size.

The outputs of these parallel convolutions can also be concatenated 226 to form the output of the decoder fire module 220 .

3 is a diagram illustrating an image segmentation process using the global attention-based deep learning network 300 according to an embodiment of the present invention.

Referring to FIG. 3 , the global attention-based deep learning network 300 may be composed of an encoder and a decoder path. The encoder path is composed of a series of first fire modules 310, and the output of each first fire module 310 is applied to a first global attention module (GAM) 311, followed by a max pooling layer. 312 may be configured.

In one embodiment, the first fire module 310 may include the in-core fire module 210 of FIG. 2c.

The first fire module 310 together with the max pooling 312 may constitute a down sampling module. In addition to the fire module input, the first global attention module 311 may receive an input obtained from a multi-scale input feature fusion strategy from the multi-scale layer 301 .

In one embodiment, the input information in the multi-scale layer 301 has a stride of 2 Max pooling by 2 (303) and 1 1, 3 It can be downsampled using the convolutional layer 301 of the 3 filters individually.

The convolution output of the multi-scale layer 301 may form a multi-scale feature map provided as an input to the first global attention module 311 . The output of the first global attention module 3110 can be provided to the maxpooling layer 312 to reduce the dimensionality and focus on the details of the feature map.

In one embodiment, the pooling index is stored during each max pooling 312 and can be used to upsample the feature map at the decoder. The decoder can be integrated with an attention gate that can emphasize features.

The feature maps extracted from the l ^th (lower level) and l+1 ^th (higher level) encoding layers are used as input and gating signals for the global attention module, respectively.

Therefore, the feature map obtained from the encoding layer including the context information is calculated by the first global attention module 311 to remove irrelevant responses, and the output thereof may be connected to the feature map of the corresponding upsampling layer.

Accordingly, an attention-based skip connection introduced between an encoder and a decoder can be created to capture multi-scale information. These skip connections can use all of the high-resolution feature information and focus on the most relevant information while performing the upsampling task.

Output information of the first fire module 310 and output information of the second fire module 320 may be input to the second global attention module 321 .

The output information of the second global attention module 321 and the output information of the second fire module 320 are up-sampled 322 and the resulting value is input to the second fire module 320 to perform decoding.

A final output value may be calculated by inputting a decoded value calculated by performing decoding to a classification layer 330 .

Using the global attention module in each encoder and decoder layer, class localization details can be determined through the global context as a guide for low-level features.

The classification layer 330 is 1 with a softmax activation function that outputs a reconstructed image. It can be composed of 1 convolutional layer. For example, the softmax layer can predict four output classes: GM, WM, CSF and background. The global attention-based deep learning network according to the present invention can generate a learned expression by taking an input image.

An input image can be classified into one of four output classes based on this feature representation. In one embodiment, cross-entropy loss can be used to measure the loss of a global attention-based deep learning network. The softmax layer can learn the representation decoder (l) and interpret it as an output class. In one embodiment, a probability score y' may be assigned to an output class.

In one embodiment, when the number of output classes is defined as c, it can be expressed as in Equation 1 below.

In one embodiment, the cross entropy loss function can be used to calculate the network cost as shown in Equation 2 below.

Here, y and y' denote the measured and predicted distribution scores for each class i, respectively.

4 is a diagram illustrating a global attention module 400 according to an embodiment of the present invention. In one embodiment, the global attention module 400 may include the first global attention module 311 and the second global attention module 321 of FIG. 3 .

Referring to FIG. 4 , x _l for the global attention module 400 may include an output feature map of l ^th encoding layer (lower level feature). Also, x _(l+1) can be collected at a coarser scale and serve as a gating signal vector and applied to all pixels to select a focus area.

α _l may include an attention coefficient that suppresses non-relative feature responses to maintain activation related to the target task. The output of the global attention module 400 can be expressed by adding a feature map for each element in the l ^th encoding layer where the attention coefficient is defined as in Equation 3 below.

In one embodiment, multi-dimensional attention coefficients may be learned in the context of multi-semantic classes.

Global average pooling 401 may provide global context information as a guide to low-level features for integrating local features with the global context.

Set the result of Global average pooling (401) to 1 It can be provided as an input to the first convolution 403 of 1.

1 including the ReLU activation feature for global information generated from higher-level features. 1 may be provided as an input to the sixth convolution layer 403 .

1 to extract weighted low-level features 1 may be multiplied with the convolved low-level feature of the seventh convolutional layer 405 .

An attention coefficient may be generated by performing upsampling 409 on the resultant value of the multiplication operation 407 .

In one embodiment, the formula for the attention coefficient can be expressed as Equation 4 below.

Here, W _x and W _g are weights associated with the input and gating signals, respectively, b is a bias term, and GAP may represent global average pooling.

The upsampling result value, the attention coefficient, is summed (411) with the low-level features, and pixel localization specific to the class category of the high-level feature map can be extracted.

Also, a feature map obtained by obtaining attention module x _lout including context information may be connected to a feature map of a corresponding decoding layer forming a skip connection. These skip connections can use all of the high-resolution feature information and focus on the most relevant information while performing the upsampling task.

5 is a diagram illustrating a classification result based on a first data set according to an embodiment of the present invention. 6 is a diagram illustrating a classification result based on a second data set according to an embodiment of the present invention.

Referring to Figures 5 and 6, the method according to the present invention can be tested with two sets of brain MRI data. 5 and 6, (a) is an original input image, (b) is a ground truth segmentation map, (c) is a predicted segmentation map, (d) is a predicted GM (binary map), (e) is a predicted CSF (binary map), and (f) is a predicted WM (binary map) indicates

For example, the first data set may include T1-weighted brain MRIs of 416 subjects obtained from the Open Access Series of Imaging Studies (OASIS) database.

In one embodiment, as shown in <Table 1>, the experiment was conducted by selecting 50 subjects out of a total of 416 subjects.

Dataset No. of subjects Experiment data Male Female Total training set Testing set OASIS 160 256 416 30 20 IBSR 14 4 18 12 6

Among the selected data, the first 30 subjects were used for model training and the remaining 20 subjects could be used as a test data set. Axial, sagittal and coronal planes of MRI slices can be used for training and testing for brain MRI segmentation in experiments.

For example, the dimension of each input axis scan in the first data set is 208 176 176, and each scan can consist of 176 slices. A distinguishable tissue area can be observed, mostly found near the middle part of the volume.

Also, successive pieces may share nearly identical information. Therefore, in order to reduce the repetitive training of consecutive slices except for slices without such information, 48 slice samples can be selected at 3 slice intervals starting from the 10th slice for the evaluation procedure.

The extracted slice is 256 by adding 24 pixels of zeros to the top and bottom of the image and 40 pixels of zeros to the left and right of the image. 256 It can be resized to a size of 48. Similarly, sagittal and coronal views of MRI slices 256 256 can be resized.

Thus, each input scan is 256 It can consist of 48 slices of 256 size. In the training phase, the slices of each MRI scan and the corresponding ground truth segmentation map can be segmented into uniform patches.

The size of the input slice is 256 256, and each slice can be divided into 4 patches. Therefore, the size of each patch divided in the method according to the present invention is 128 may be 128. These patches are provided as inputs to the model for training and can obtain predicted segmentation results on the test data.

For example, the second data set may include an MRI of an Internet Brain Segmentation Repository (IBSR) data set. A second data set may include 18 T1-weighted MRIs of 14 healthy males and 4 healthy females between the ages of 7 and 71 years.

MRI of the IBSR can be presented after preprocessing such as skull removal, normalization, and bias field correction. The training data set included 12 subjects with manually annotated and verified ground truth labels, and the remaining 6 could test the model.

Original axial scan (256 128 256) is 64 pixels of zero padding above and below the image, resulting in 256 256 256 size, the patch of the proposed method can be used effectively. Similarly, the original crown (256 256 128) and awards (128) 256 256) for the scan diagram experiment 256 256 256 can be resized.

In one embodiment, the global attention-based deep learning network model can be optimized according to a categorical cross-entropy loss function. In one embodiment, a normalization technique may be employed to initialize the weights.

Referring to FIGS. 5 and 6 , it is possible to check the segmentation results of the axial, coronal, and sagittal planes of the first and second data sets.

From the results, it can be observed that the method according to the present invention achieves well-disaggregated results for GM, WM and CSF in both data sets.

In one embodiment, the performance of the method according to the present invention can be evaluated using the quantitative indices detailed in Table 2 below.

Dice similarity
coefficient (DSC) Jaccard Index (JI) Hausdorff
distance (HD) Mean square error
(MSE)

Here, DSC and JI are tuned to compare volume based on overlap, and can be used to compare ground truth with results of automated segmentation methods.

DSC can be defined as twice the number of elements common to both sets divided by the sum of the number of elements in each set. here and The cardinality of the ground truth set and the predicted split set (ie, the number of elements in each set) may be indicated.

JI can be expressed as DSC mentioned in <Table 2>. The DSC and JI metrics can determine the match between the predicted segmentation map and the corresponding ground truth map.

In addition, segmentation performance can be evaluated in terms of the mean squared error (MSE), which is the mean squared difference between the original X value and the predicted Y value.

HD can be used to determine dissimilarity between two sets in metric space. Two sets of small HDs can look almost identical. HD and MSE can be calculated as shown in Table 2. Here, D may mean the Euclidean distance of two pixels, and R and C may mean the image height and width, respectively.

To more intuitively show the segmentation effects of different network architectures, U-net, SegNet and U-SegNet models can be trained on experimental data.

7 is a diagram illustrating comparison of classification results based on a first data set according to an embodiment of the present invention. 8 is a diagram illustrating comparison of classification results based on a second data set according to an embodiment of the present invention.

Referring to FIGS. 7 and 8 , it can be seen that the quality of the segmentation map generated by the method according to the present invention is clearly superior to that of other existing methods.

In one embodiment, (a) of FIGS. 7 and 8 is an original input image, (b) is a ground truth segmentation map, (c) is a segmentation result generated by SegNet, U-net, U-SegNet, and the method according to the present invention, (d) is a GM map generated by SegNet, U-net, U-SegNet and the method according to the present invention, (e) is SegNet, U-net, U-SegNet and the method according to the present invention The generated CSF maps and (f) represent SegNet, U-net, U-SegNet and WM maps generated by the method according to the present invention.

U-net has a shallow network and is not sufficient to capture image space information. In addition, if the video content is complicated, the segmentation accuracy of U-net and SegNet is greatly reduced. In particular, it can be confirmed in the feature maps generated by SegNet and U-net, and it can be confirmed that the highlighted area is concentrated in a specific organization that caused misclassification.

U-SegNet can use skip connections to combine feature maps from encoder to decoder and pull indices to localize these features during upsampling.

Nevertheless, U-SegNet yields better segmentation results with a combination of index and skip concatenation, but may fail to capture fine details. In the highlighted red box, it can be seen that U-SegNet usually subdivides WM and CSF organizations.

Integrating attention can overcome some of these limitations and improve segmentation performance by focusing attention on relevant areas. This improved partitioning can be observed in the results obtained with the method according to the present invention.

Referring to FIG. 8 , similar results can be confirmed in the segmentation obtained from the IBSR image. In particular, it can be seen that the network according to the present invention can obtain finer details than other architectures. These visual results confirm that the method according to the present invention can robustly recover finer segmentation details while bypassing the distraction of ambiguous regions.

In one embodiment, quantitative analysis of the method according to the present invention was performed in comparison with existing SegNet, U-net, and U-SegNet methods, and the results are shown in <Table 3> to <Table 6> below.

OASIS dataset Axial plane parameter WM SegNet U-net U-Segnet Proposed
Method DSC (%) 87.36 92.18 93.36 95.54 JI(%) 77.18 85.50 87.56 90.13 HD 5.09 4.40 4.2 3.6 Coronal plane DSC (%) 82.12 93.14 94.12 96.37 JI(%) 69.23 87.20 89.63 91.17 HD 5.4 4.14 3.9 3.2 Sagittal plane DSC (%) 82.42 92.44 93.25 96.72 JI(%) 69.54 86.34 87.65 91.38 HD 7.2 4.3 4.0 3.35 IBSR dataset Axial plane DSC (%) 72.86 89.45 90.35 91.76 JI(%) 65.34 81.56 82.49 84.23 HD 6.51 5.14 4.6 4.2 Coronal plane DSC (%) 70.15 88.45 89.36 90.47 JI(%) 62.35 79.38 80.14 82.53 HD 6.3 5.45 5.5 4.6 Sagittal plane DSC (%) 71.53 86.84 87.32 89.52 JI(%) 63.41 78.63 79.62 81.42 HD 6.49 5.75 5.4 4.82

OASIS dataset Axial plane parameter GM SegNet U-net U-Segnet Proposed
Method DSC (%) 84.93 90.32 82.05 94.65 JI(%) 72.20 82.35 85.27 88.53 HD 5.7 4.3 4.0 3.8 Coronal plane DSC (%) 78.21 82.25 93.53 95.94 JI(%) 64.16 85.45 87.85 90.46 HD 4.6 4.2 4.12 3.42 Sagittal plane DSC (%) 80.23 91.13 92.56 95.69 JI(%) 67.57 83.26 85.69 90.53 HD 5.9 5.2 4.2 3.49 IBSR dataset Axial plane DSC (%) 75.63 91.53 92.20 93.83 JI(%) 67.32 85.41 86.15 87.52 HD 6.53 4.87 4.2 4.0 Coronal plane DSC (%) 73.65 90.23 91.45 92.85 JI(%) 65.42 83.56 84.16 85.61 HD 6.21 5.17 4.8 4.53 Sagittal plane DSC (%) 74.62 89.46 90.44 91.71 JI(%) 66.85 81.53 82.15 84.45 HD 6.36 5.77 5.3 4.25

OASIS dataset Axial plane parameter CSF SegNet U-net U-Segnet Proposed
Method DSC (%) 80.67 89.82 91.64 93.83 JI(%) 67.85 81.53 84.57 86.53 HD 4.9 4.6 4.1 3.9 Coronal plane DSC (%) 74.03 89.53 91.46 94.18 JI(%) 61.26 82.36 84.25 89.27 HD 4.6 4.1 4.15 3.81 Sagittal plane DSC (%) 77.45 88.63 92.15 94.53 JI(%) 63.51 81.26 85.36 89.44 HD 6.3 4.4 4.15 3.56 IBSR dataset Axial plane DSC (%) 68.42 84.34 84.95 85.64 JI(%) 59.32 75.85 75.98 77.16 HD 6.3 4.4 4.3 4.86 Coronal plane DSC (%) 66.54 83.65 84.15 85.83 JI(%) 57.32 76.86 76.94 77.86 HD 6.84 5.54 5.2 4.9 Sagittal plane DSC (%) 65.49 80.75 81.19 83.56 JI(%) 54.86 73.96 74.10 75.28 HD 6.99 5.83 5.6 5.1

MSE SegNet U-net U-Segnet Proposed
Method OASIS 0.021 0.008 0.006 0.003 IBSR 0.013 0.009 0.007 0.005

It can be seen that the network according to the present invention achieves average improvements of 10%, 3.9% and 2.3% (based on DSC) compared to SegNet, U-net and U-SegNet methods and achieves a lower MSE value of 0.003.

This performance difference can be explained by the fact that SegNet only stores max pooling indices. That is, the position of the maximum feature value in each pooling window can be stored for each encoder map and used for upsampling.

This could improve borderline with 3 million parameters and require 2.9 hours less training time than existing methods. However, when SegNet performs upsampling on low-resolution feature maps, it tends to miss a lot of detail because it loses surrounding information.

On the other hand, U-net can use skip connections as the core of an architecture that mixes deep coarse information with shallow fine semantic information. U-net can use low-level feature maps for upsampling. As a result, translational constancy can often be compromised. Also, U-SegNet tends to be insensitive to fine details and has difficulty discerning boundaries between adjacent organizations such as WM and GM. Subdivision maps generated by these existing models may have relatively low resolution due to the pooling layer of the encoder stage.

Therefore, pooling layers may need to be removed to maintain high spatial resolution. However, since convolution is a local operation, SegNet, U-net, and U-SegNet models cannot learn global features from images without layer pooling.

The method according to the present invention presents a multi-scale feature fusion scheme combined with GAM as a potential solution to the problems of the prior art discussed above and can produce improved segmentation accuracy. 1 linked together 1, 3 A max-full input convolved with a 3-kernel can capture the global context without reducing the resolution of the subdivision map.

In this way, global information can be exchanged between layers without sacrificing resolution and blurring of semantic segmentation maps can be reduced. In addition, the encoder's GAM can provide global context information as a guide to low-level features to extract the original resolution for segmentation.

The GAM of the decoder confirms that the combination of global and local features was important in discriminating brain tissue and is consistent with the reference results. Uniform input patching can also allow the network to focus more on local details.

As a result of selectively integrating spatial information through uniform patches, feature maps followed by multi-scale guided multi-GAM help capture contextual information and efficiently encode complementary information to accurately segment brain MRI.

We can also use the Fire module to identify networks with few learnable parameters while maintaining the same accuracy.

9 is a diagram illustrating comparison between the number of learning parameters and calculation time according to an embodiment of the present invention.

Referring to FIG. 9 , it is possible to check learnable parameters and calculation time consumed by the proposed method compared to the existing method.

One A smaller model can be built by arranging a series of fire modules consisting of squeeze layers with only 1 convolutional filter. this is 1 1 and 3 An extension layer in which three filters are combined may be provided.

The number of filters in the squeeze layer is 1 in the extension layer. 1 and 3 Less than 3 filters can be defined. 1 of the squeeze layer 1 The filter can reduce parameters by downsampling the input channels before being fed as input to the enhancement layer. extension layer is 1 1 and 3 It can consist of 3 filters.

1 of the extension layer 1 filters combine channels and perform cross-channel pooling, but may not be able to recognize spatial structure. 3 of the extension layer 3 Convolutional filters can identify spatial representations. Combining these two size filters allows the model to be more expressive while working with fewer parameters.

Thus, the Fire module can reduce the parameter maps to build smaller deep learning networks that can reduce the computational load and maintain higher accuracy.

The total parameters of the method according to the present invention are 1 million parameters and can be 3.3 times, 4 times, and 5 times smaller than SegNet, U-SegNet, and U-net networks, respectively.

The training time of the first data set method is 73% of U-SegNet and can be 12% faster than U-SegNet network. Reduced memory requirements can significantly reduce energy and computational requirements compared to existing methods.

Tests can be performed on different proposed modules to investigate the impact of each choice on the segmentation performance according to the present invention. (i) squeeze U-SegNet, (ii) squeeze U-SegNet with multi-scale input, (iii) squeeze U-SegNet with multi-global attention, (iv) multi-scale squeeze U-SegNet with multi-global attention (proposed method).

The squeeze U-SegNet of the first network can be obtained by replacing each convolution block with a fire module in the existing U-SegNet.

The encoder of the squeeze U-SegNet of the second network may include a multi-scale input layer.

This max pools the input and returns 1 1, 3 It can be achieved by performing parallel convolution using a 3 kernel and concatenating these multi-scale features. These fused multi-scale features can be connected to corresponding Fire module outputs and fed as inputs for max pooling operations. This process can be repeated for all encoding layers.

The multi-scale feature module can more accurately extract adjacent scale information of global features while filtering irrelevant information. The influence of the attention mechanism can be explored in a third network where GAMs are integrated in both the encoder and decoder to form a multi-attention network.

Finally, the multi-scale squeeze U-SegNet, which is called the method according to the present invention, can integrate a semantic guide by combining all the proposed modules.

In one embodiment, <Table 7> to <Table 10> below may indicate individual results of various components for segmentation performance.

Models GM DSC JI HD Squeeze U-SegNet 92.05 88.06 3.5 Squeeze U-SegNet
with multi-scale input 93.44 89.47 4.8 Squeeze U-SegNet
with multi global
attention 94.32 89.25 5 Proposed Method 94.65 89.51 4.8

Models WM DSC JI HD Squeeze U-SegNet 93.37 90.42 2.8 Squeeze U-SegNet
with multi-scale input 94.78 91.90 4.1 Squeeze U-SegNet
with multi global
attention 95.5 91.40 4.2 Proposed Method 95.86 92.05 4.1

Models CSF DSC JI HD MSE Squeeze U-SegNet 91.65 88.06 2.0 0.006 Squeeze U-SegNet
with multi-scale input 93.32 90.25 3.0 0.005 Squeeze U-SegNet
with multi global
attention 94.30 89.22 3.3 0.004 Proposed Method 94.43 89.74 3.0 0.003

Computation
time (5 epochs) #learnable
parameters Squeeze U-SegNet 1.7 hours 768,788 Squeeze U-SegNet
with multi-scale input 1.9 hours 860,180
Squeeze U-SegNet
with multi global
attention 2 hours 942,164 Proposed Method 2.2 hours 1,030,420

The model exhibits a significant reduction in the requirements of learnable parameters by reducing the computational time for model training while maintaining network accuracy. Compared to the reference squeeze U-SegNet, it can be seen that the performance of the model integrated with the multi-scale feature fusion input scheme and multi-attention module is improved by 2.1% and 3%, respectively.

Multi-scale feature fusion shows a smaller increase in DSC but can provide more network efficiency in combination with GAM.

In addition, combining multi-scale and multi-global attention strategies improves performance and yields the best values across the three metrics, such as 94.78% (DSC), 90.43% (JI), 3.1 (HD), and the lowest MSE of 0.003.

These results represent a 3.5% improvement in DSC over the baseline U-SegNet and demonstrate the efficiency of the proposed multi-scale guided multi-GAM compared to individual components.

In addition, the effect of patch size in terms of training time and segmentation performance can be investigated. Experiments were conducted with three different patch sizes (128 128, 64 64 and 32 32) on the second data set.

In one embodiment, Table 11 below may indicate segmentation performance of DSC for various patch sizes.

Patch
size DSC HI Training time (hours) WM GM CSF WM GM CSF 128x128 96.33 95.44 94.81 92.93 91.28 90.14 2.2 64x64 96.35 95.48 94.89 92.95 91.37 90.28 3.4 32x32 96.35 95.51 95.12 92.98 91.47 90.34 4.6

It can be seen that the smaller the patch size, the better the performance. This could be because smaller patches generate more training data for the network to train on. Also, localization can be created more accurately.

Also, if the patch size is 128 For 128, it takes 2.2 hours to train the model, while for 32 with almost the same accuracy For 32 patches, the learning time can be doubled.

So 128 According to the results in <Table 11>, it can be confirmed that the 128 patch size provides an appropriate balance between the DSC score and the computation time required for model learning.

10 is a diagram illustrating an image segmentation method using global attention according to an embodiment of the present invention.

Referring to FIG. 10 , step S1001 is a step of acquiring input information.

In one embodiment, an input image may be obtained, and input information in the form of a patch may be generated by dividing the input image. Here, the input image may include a slice image of an object. Also, the input information may include a patch image obtained by dividing a slice image into a plurality of patch shapes.

Step S1003 is a step of inputting input information to a first fire module 310 including an expand layer 213 composed of a squeeze layer 211 composed of a first convolution layer 212, a second convolution layer 214, and a third convolution layer 215.

In one embodiment, the first convolution layer 211 and the second convolution layer 214 may be configured with a first kernel size, and the third convolution layer 215 may be configured with a second kernel size. For example, the first kernel size is 1 1 kernel size, and the second kernel size is 3 3 kernel size, but is not limited thereto.

In one embodiment, an output value of the first convolution layer 212 of the squeeze layer 211 is generated, the generated output value is input to each of the second convolution layer 214 and the third convolution layer 215 configured in parallel of the extension layer 213, and the output values of the second convolution layer 214 and the third convolution layer 215 are concatenated (216) Thus, output information of the first fire module 310 may be generated.

In one embodiment, before step S1005, max pooling 312 is performed on the input information for the first layer to generate input information for the second layer, and the input information for the second layer is configured in parallel. The fourth convolution layer and the fifth convolution layer may generate multi-scale input information for the second layer.

In one embodiment, the fourth convolution layer and the fifth convolution layer may be configured in parallel to configure the multi-scale layer 301 .

Step S1005 is a step of inputting the output information of the first fire module 310 and the multi-scale input information to the first global attention module 311 including the global average pooling 401.

In one embodiment, global average pooling 401 is performed on the output information of the first fire module 310, the resultant value of the global average pooling 401 is input to the sixth convolution layer 403, and the resultant value of the sixth convolution layer 403 and the multi-scale input information are input to the seventh convolution layer 405 to perform upsampling 409 based on the generated resultant to perform attention coefficient ( attention coeffeicient), and output information of the first global attention module 311 may be generated using the attention coefficient and the output information of the first fire module 310 .

In one embodiment, a multiplication operation 407 is performed on a result value generated by inputting the result value of the sixth convolution layer 403 and the multi-scale input information to the seventh convolution layer 405, and upsampling 409 may be performed on the result value generated through the multiplication operation 407.

In one embodiment, a sum operation 411 may be performed on the attention coefficient and output information of the first fire module 310, and output information of the first global attention module 311 may be generated through the sum operation 411.

In one embodiment, output information of the first fire module 310 may mean a low-level feature, and multi-scale input information may mean a high-level feature.

Step S1007 is a step of performing encoding by performing max pooling 312 on output information of the first global attention module 311 .

In one embodiment, an encoding value calculated by performing encoding may be input to the first fire module 310 of the next layer.

In one embodiment, after step S1007, the output information of the first fire module 310 for the first layer and the output information of the second fire module 320 for the second layer are input to the second global attention module 321.

For example, the first fire module 310 may mean the encoder fire module 210, and the second fire module 320 may mean the decoder fire module 220.

In one embodiment, the second fire module 320 may include a squeeze layer 221 composed of a first transposed convolution layer 222, a second transposed convolution layer 224, and an extension layer 223 composed of a second transposed convolution layer 225.

In one embodiment, after step S1007, the output information of the second global attention module 321 and the output information of the second fire module 320 for the second layer are upsampled (322). The resulting value may be input to the second fire module 320 for the first layer to perform decoding.

In one embodiment, a decoded value calculated by performing decoding may be input to a classification layer 330 to calculate a final output value.

11 is a diagram showing a functional configuration of an image segmentation apparatus 1100 using global attention according to an embodiment of the present invention.

Referring to FIG. 11 , an image segmentation apparatus 1100 may include an acquisition unit 1110, a control unit 1120, and a storage unit 1130.

The acquisition unit 1110 may obtain input information. In one embodiment, the acquisition unit 1110 may include a communication unit or a photographing unit. For example, the acquisition unit 1110 may include a magnetic resonance imaging (MRI) camera. Also, the acquisition unit 1110 may include a communication unit that receives data from an external electronic device.

In one embodiment, the communication unit may include at least one of a wired communication module and a wireless communication module. All or part of the communication unit may be referred to as a 'transmitter', a 'receiver', or a 'transceiver'.

The control unit 1120 may input input information to a first fire module 310 including an expand layer 213 composed of a squeeze layer 211 composed of a first convolution layer 212, a second convolution layer 214, and a third convolution layer 215.

In one embodiment, the controller 1120 includes the output information and the multi-scale input information of the first fire module 310 and the first global attention module 311 including global average pooling 401. It can be input.

In an embodiment, encoding may be performed by performing max pooling 312 on output information of the first global attention module 311 .

In one embodiment, the controller 1120 may include at least one processor or microprocessor, or may be a part of the processor. Also, the controller 1120 may be referred to as a communication processor (CP). The controller 1120 may control the operation of the image segmentation device 1100 according to various embodiments of the present disclosure.

The storage unit 1130 may store input information. Also, the storage unit 1030 may store a global attention-based deep learning network.

In one embodiment, the storage unit 1130 may include a volatile memory, a non-volatile memory, or a combination of volatile and non-volatile memories. Also, the storage unit 1130 may provide stored data according to a request of the control unit 1120 .

Referring to FIG. 11 , an image segmentation apparatus 1100 may include an acquisition unit 1110, a control unit 1120, and a storage unit 1130. In various embodiments of the present invention, the image segmentation apparatus 1100 may have more or fewer components than those described in FIG. 11 because the components described in FIG. 11 are not essential.

The above description is only illustrative of the technical idea of the present invention, and various changes and modifications may be made by those skilled in the art without departing from the essential characteristics of the present invention.

The various embodiments disclosed herein may be performed out of order, concurrently or separately.

In one embodiment, at least one step may be omitted or added in each figure described herein, may be performed in reverse order, or may be performed concurrently.

The embodiments disclosed herein are not intended to limit the technical spirit of the present invention, but are intended to explain, and the scope of the present invention is not limited by these embodiments.

The protection scope of the present invention should be interpreted according to the claims, and all technical ideas within the equivalent range should be understood to be included in the scope of the present invention.

210: encoder fire module
211: squeeze module
212: first convolution layer
213: extension layer
214: second convolution layer
215: third convolution layer
216: connection
220: decoder fire module
221: squeeze layer
222: first transpose convolution layer
223: extension layer
224: second transpose convolution layer
225: second transpose convolution layer
226: connection
300: Global attention-based deep learning network
301: multi-scale layer
303 Max pooling
310: first fire module
311: first global attention module
312 Max pooling
320: second fire module
321: second global attention module
322: upsampling
330: classification layer
400: global attention module
401: global average pooling
403: 6th convolution layer
405: 7th convolution layer
407: product operation
409: upsampling
411: sum operation
1100: video segmentation device
1110: acquisition unit
1120: control unit
1130: storage unit

Claims (18)

(a) obtaining input information;
(b) a first fire module including a squeeze layer composed of a first convolution layer, an expand layer composed of a second convolution layer and a third convolution layer, inputting the input information to a fire module;
(c) inputting output information and multi-scale input information of the first fire module to a first global attention module including global average pooling; and
(d) performing encoding by max pooling output information of the first global attention module;
including,
Image segmentation method using global attention.
According to claim 1,
In step (a),
acquiring an input image; and
generating the input information in a patch form by dividing the input image;
including,
Image segmentation method using global attention.
According to claim 1,
The first convolution layer and the second convolution layer are composed of a first kernel size,
The third convolution layer is composed of a second kernel size,
Image segmentation method using global attention.
According to claim 1,
In step (b),
generating an output value of a first convolution layer of the squeeze layer;
inputting the generated output value to each of a second convolution layer and a third convolution layer configured in parallel of the enhancement layer; and
generating output information of the first fire module by concatenating output values of the second convolution layer and the third convolution layer;
including,
Image segmentation method using global attention.
According to claim 1,
Before step (c),
Generating input information for a second convolution layer by performing max pooling on the input information for the first convolution layer;
generating multi-scale input information for the second convolution layer by inputting input information for the second convolution layer to each of a fourth convolution layer and a fifth convolution layer configured in parallel;
Including more,
Image segmentation method using global attention.
According to claim 1,
In step (c),
pooling the global average of output information of the first fire module;
inputting a resultant value of the global average pooling to a sixth convolution layer;
Generating an attention coefficient by performing upsampling based on a result value of the sixth convolution layer and a result value generated by inputting the multi-scale input information to a seventh convolution layer; and
generating output information of the first global attention module using the attention coefficient and output information of the first fire module;
including,
Image segmentation method using global attention.
According to claim 1,
After step (d),
inputting output information of a first fire module for the first convolution layer and output information of a second fire module for the second convolution layer to a second global attention module for the first convolution layer;
Including more,
The second fire module includes a squeeze layer composed of a first transposed convolution layer, an extension layer composed of a second transposed convolution layer and a third transposed convolution layer,
Image segmentation method using global attention.
According to claim 7,
After step (d),
Decoding by inputting a result value generated by upsampling output information of the second global attention module and output information of the second fire module for the second convolution layer to a second fire module for the first convolution layer; and
Calculating a final output value by inputting a decoded value calculated by performing the decoding to a classification layer;
Including more,
Image segmentation method using global attention.
According to claim 7,
In the step of inputting to the second global attention module,
generating an output value of a first transpose convolution layer of the squeeze layer;
inputting the generated output value to a second transpose convolution layer and a third transpose convolution layer configured in parallel of the enhancement layer, respectively; and
generating output information of the second fire module by concatenating output values of the second transpose convolution layer and the third transpose convolution layer;
including,
Image segmentation method using global attention.
an acquisition unit that acquires input information; and
Inputting the input information to a first fire module including a squeeze layer composed of a first convolution layer and an expand layer composed of a second convolution layer and a third convolution layer,
Inputting the output information and multi-scale input information of the first fire module to a first global attention module including global average pooling,
a control unit performing encoding by max pooling output information of the first global attention module;
including,
Video segmentation device using global attention.
According to claim 10,
The acquisition unit obtains an input image,
The control unit divides the input image to generate the input information in the form of a patch.
Video segmentation device using global attention.
According to claim 10,
The first convolution layer and the second convolution layer are composed of a first kernel size,
The third convolution layer is composed of a second kernel size,
Video segmentation device using global attention.
According to claim 10,
The control unit,
generating an output value of a first convolution layer of the squeeze layer;
Inputting the generated output value to each of a second convolution layer and a third convolution layer configured in parallel of the extension layer,
Generating output information of the first fire module by concatenating output values of each of the second convolution layer and the third convolution layer,
Video segmentation device using global attention.
According to claim 10,
The control unit,
Generating input information for a second convolution layer by performing max pooling on the input information for the first convolution layer;
Generating multi-scale input information for the second convolution layer by inputting input information for the second convolution layer to each of a fourth convolution layer and a fifth convolution layer configured in parallel,
Video segmentation device using global attention.
According to claim 10,
The control unit,
The global average pooling of the output information of the first fire module,
Inputting the resultant value of the global mean pooling to a sixth convolution layer;
Generating an attention coefficient by performing upsampling based on a result value of the sixth convolution layer and a result value generated by inputting the multi-scale input information to a seventh convolution layer,
Video segmentation device using global attention.
According to claim 10,
The control unit,
Inputting output information of a first fire module for the first convolution layer and output information of a second fire module for the second convolution layer to a second global attention module for the first convolution layer,
The second fire module includes a squeeze layer composed of a first transposed convolution layer, an extension layer composed of a second transposed convolution layer and a third transposed convolution layer,
Video segmentation device using global attention.
According to claim 16,
The control unit,
Decoding is performed by inputting a result value generated by upsampling the output information of the second global attention module and the output information of the second fire module for the second convolution layer to the second fire module for the first convolution layer,
Calculating a final output value by inputting the decoded value calculated by performing the decoding to a classification layer,
Video segmentation device using global attention.
According to claim 16,
The control unit,
generating an output value of a first transpose convolution layer of the squeeze layer;
Inputting the generated output value to each of a second transpose convolution layer and a third transpose convolution layer configured in parallel of the extension layer,
Video segmentation device using global attention.