KR20180021635A - Method and system for analyzing feature representation of lesions with depth directional long-term recurrent learning in 3d medical images - Google Patents

Abstract

According to an embodiment, a lesion feature representation analyzing method using depth direction recurrent study in 3D medical images comprises the following steps of: extracting a lesion space feature representation of each of a plurality of sectional images by using a convolution neural network (CNN) wherein the plurality of sectional images are included in the 3D medical images which photograph a lesion; and extracting a lesion feature representation of each of the plurality of sectional images along a depth direction of the plurality of sectional images by using a recurrent neural network based on the lesion space feature representation of each of the plurality of sectional images.

Description

TECHNICAL FIELD [0001] The present invention relates to a method and system for analyzing a lesion in a three-dimensional medical image using depth direction recursive learning,

The following embodiments relate to a system for analyzing Feature Representation of Lesions using Depth Directional Long-Term Recurrent Learning and a method for analyzing the Feature Representation of Lesions, including a Convolutional Neural Network The present invention relates to a technology for extracting and analyzing spatial and depth feature expressions included in a 3D medical image using a Recurrent Neural Network.

In recent years, Digital Breast Tomosynthesis (DBT) has been attracting attention as a new image technology that solves the tissue overlap problem of conventional two dimensional (2D) Mammography by imaging the 3D structure information of the breast. DBT reconstructs 3D reconstructed slices from multiple low-dose projection views obtained by X-ray imaging over a limited range of angles to provide three-dimensional structure information of the breast. It has been reported that this DBT can increase the detection rate of breast cancer lesion compared with existing Mammogram when used through clinical research. However, since the DBT generally provides about 35 to 80 frames of sectional images, it has a disadvantage that the number of images to be analyzed is very large compared with the case where the existing weargram provides one image. These disadvantages also raise the problem of increasing the diagnosis time and burden of the radiologist when diagnosing breast cancer using DBT.

A variety of computer-aided detection techniques have been studied to overcome the increased diagnostic burden. Computer assisted diagnosis has been reported to help doctors diagnose various clinical studies with the purpose of helping doctors to show suspected areas of breast cancer to the doctor. However, since computer-assisted diagnosis has a relatively high false positive detection rate, it causes unnecessary biopsy, which may increase the diagnostic burden on the patient.

To distinguish these false positives from actual breast cancer lesions, various hand-crafted features have been developed. However, the hand-crafted feature depends on the physician's knowledge and experience, and there are limitations that it is difficult to develop features that are hidden in many cross-sectional images of about 35-80 sheets.

Accordingly, there is a need for a technique capable of efficiently extracting and classifying lesion characteristics in massive image data, and the following embodiments are based on Recurrent Neural Network-based deep learning, We propose a technique to analyze spatial and depth feature expressions.

One embodiment provides a method and system for analyzing spatial and depth feature representations of a lesion in a plurality of cross-sectional images included in a three-dimensional medical image.

Specifically, one embodiment extracts a lesion spatial feature representation of each of a plurality of sectional images using a Convolution Neural Network (CNN) for a lesion suspected region composed of a plurality of sectional images, A method and system for classifying a given lesion suspicious region by extracting a lesion feature expression of each of a plurality of sectional images along a depth direction of a plurality of sectional images using a Recurrent Neural Network.

In addition, one embodiment provides a method and system for learning a recursive neural network used in a process of extracting a lesion feature expression so that a lesion feature expression is efficiently extracted.

According to one embodiment, a method of analyzing a lesion feature representation using depth directional recursion in a three-dimensional medical image comprises using a Convolution Neural Network (CNN), wherein a plurality of cross-sectional images- Extracting each of the lesion space feature expressions included in one taken three-dimensional medical image; And extracting a lesion feature expression of each of the plurality of sectional images along a depth direction of the plurality of sectional images using a recurrent neural network based on a lesion spatial feature representation of each of the plurality of sectional images. .

Wherein the extracting of the lesion feature expressions of each of the plurality of sectional images includes recognizing a lesion feature vector of each of the plurality of sectional images based on the depth direction of the plurality of sectional images; Acquiring lesion feature expressions of each of the plurality of sectional images by fusing lesion feature vectors of each of the plurality of sectional images; And classifying lesions of each of the plurality of sectional images based on a lesion feature representation of each of the plurality of sectional images.

Wherein recognizing a lesion feature vector of each of the plurality of sectional images comprises grouping lesion spatial feature representations of each of the plurality of sectional images based on a depth direction of the plurality of sectional images; And recognizing a lesion feature vector of each of the plurality of sectional images for each group.

Wherein acquiring the lesion feature expressions of each of the plurality of sectional images comprises acquiring lesion feature expressions of each of the plurality of sectional images by fusing lesion feature vectors of each of the sectional images corresponding to each other between the groups . ≪ / RTI >

Wherein acquiring the lesion feature expressions of each of the plurality of sectional images comprises: recognizing depth direction symmetry of lesions of each of the plurality of sectional images by fusing lesion feature vectors of each of the plurality of sectional images; And obtaining a lesion feature representation of each of the plurality of sectional images based on the depth direction symmetry of the lesion of each of the recognized plurality of sectional images.

Wherein the recursive neural network comprises a first objective function for minimizing classification errors that classify lesions of each of the plurality of sectional images, a correspondence between lesion feature vectors of each of the plurality of sectional images to be fused And a third objective function for minimizing the intra-class variance in which the lesion of each of the plurality of sectional images is classified.

Wherein the obtaining of the lesion feature expressions of each of the plurality of sectional images comprises fusing the lesion feature vectors of each of the plurality of sectional images so that the second objective function is maximized, And recognizing the symmetry.

Wherein classifying lesions of each of the plurality of sectional images comprises classifying lesions of each of the plurality of sectional images so that the first objective function is minimized based on a lesion feature representation of each of the plurality of sectional images .

Wherein classifying lesions of each of the plurality of sectional images comprises classifying lesions of each of the plurality of sectional images so that the third objective function is minimized based on a lesion feature representation of each of the plurality of sectional images .

According to one embodiment, a plurality of sectional images using the Convolution Neural Network (CNN), wherein the plurality of sectional images are included in one 3D medical image capturing a lesion, A depth direction recursion learning method for extracting a lesion feature representation of each of the plurality of sectional images based on a representation of a plurality of sectional images comprises calculating a lesion feature of each of the plurality of sectional images based on a lesion feature expression of each of the plurality of sectional images Learning a Recurrent Neural Network to optimize a first objective function to minimize classifying classification errors; To optimize a second objective function to maximize correspondence between lesion feature vectors of each of the plurality of cross-sectional images to be fused, to extract a lesion feature representation of each of the plurality of cross-sectional images, Learning a neural network; And learning the recursive neural network to optimize a third objective function to minimize intra-class variance in which lesions of each of the plurality of sectional images are classified based on a lesion feature representation of each of the plurality of sectional images do.

According to one embodiment, a lesion feature expression analysis system that uses depth direction recursion in a three-dimensional medical image uses a Convolution Neural Network (CNN) to generate a plurality of cross-sectional images- A convolutional neural network application for extracting each lesion space feature representation included in one photographed 3D medical image; And extracting a lesion feature expression of each of the plurality of sectional images along a depth direction of the plurality of sectional images using a recurrent neural network based on a lesion spatial feature representation of each of the plurality of sectional images. And a recursive neural network application unit.

Wherein the recursive neural network application unit recognizes a lesion feature vector of each of the plurality of sectional images based on a depth direction of the plurality of sectional images and fuses the lesion feature vectors of each of the plurality of sectional images, And to classify the lesion of each of the plurality of sectional images based on the lesion feature expression of each of the plurality of sectional images.

Wherein the recursive neural network comprises a first objective function for minimizing classification errors that classify lesions of each of the plurality of sectional images, a correspondence between lesion feature vectors of each of the plurality of sectional images to be fused And a third objective function for minimizing the intra-class variance in which the lesion of each of the plurality of sectional images is classified.

One embodiment may provide a method and system for analyzing spatial and depth feature representations of a lesion in a plurality of cross-sectional images included in a three-dimensional medical image.

More specifically, one embodiment extracts a lesion spatial feature representation of each of a plurality of sectional images using a convolutional neural network, and calculates a lesion characteristic of each of a plurality of sectional images along a depth direction of a plurality of sectional images using a recurrent neural network By extracting the expression, a method and system for classifying lesions of each of a plurality of sectional images can be provided.

In addition, one embodiment can provide a method and system for learning a recursive neural network used in a process of extracting a lesion feature expression so that a lesion feature expression is efficiently extracted.

Accordingly, one embodiment can provide a method and system that can be widely applied in various 3D image analysis techniques such as CT or MRI by effectively analyzing spatial and depth feature expressions of a lesion.

1 is a conceptual diagram for explaining a lesion feature expression analysis system according to an embodiment.
2 is a conceptual diagram for explaining a method of analyzing a lesion feature expression according to an embodiment.
3A to 3C are diagrams for explaining objective functions used for learning a recursive neural network according to an embodiment.
4A and 4B are flowcharts illustrating a method of analyzing a lesion feature according to an exemplary embodiment of the present invention.
5 is a block diagram illustrating a lesion feature expression analysis system according to an embodiment.
6 is a flowchart illustrating a depth direction recursive learning method according to an embodiment.
7 is a block diagram illustrating a depth direction recursion learning system according to an embodiment.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. However, the present invention is not limited to or limited by the embodiments. In addition, the same reference numerals shown in the drawings denote the same members.

Also, terms used in this specification are terms used to appropriately express the preferred embodiment of the present invention, and this may vary depending on the user, the intention of the operator, or the practice of the field to which the present invention belongs. Therefore, the definitions of these terms should be based on the contents throughout this specification.

1 is a conceptual diagram for explaining a lesion feature expression analysis system according to an embodiment.

Referring to FIG. 1, a lesion feature expression analysis system 100 according to an exemplary embodiment of the present invention performs a deep-run based lesion feature expression analysis method to detect a lesion feature expression in a 3D medical image 110, Can be recognized. Here, the lesion refers to a human tissue changed into a disease, which is photographed in the three-dimensional medical image 110.

Specifically, the lesion feature expression analysis system 100 extracts lesion spatial feature expressions of each of a plurality of sectional images 120 included in the 3D medical image 110 using a Convolution Neural Network (CNN) And extracting a lesion feature expression of each of the plurality of sectional images 120 along the depth direction using a recurrent neural network, The lesion can be recognized and classified in the tumor 110.

Hereinafter, the lesion feature expression analysis method is described as being performed by the lesion feature expression analysis system 100 through the two steps described above. However, the present invention is not limited thereto, And recognizing the feature of the lesion along the depth direction using a recursive neural network.

Hereinafter, the lesion feature expression analysis system 100 will be described as being implemented as a hardware module such as a computer that implements an electronic device. However, the present invention is not limited thereto and may be applied to a computer program stored in a non- . ≪ / RTI >

At this time, the plurality of sectional images 120 are images in which one three-dimensional medical image 110 is divided according to the depth direction, have. However, the present invention is not limited thereto, and it may be an image segmented according to the depth direction with respect to a volume-of-interest (VOI) of the three-dimensional medical image 110, as shown in the figure. Hereinafter, it is assumed that a plurality of sectional images 120 are images in which at least a part of the three-dimensional medical images 110 is divided along the depth direction.

The two steps of the method for analyzing the lesion feature expression can be performed by the two components of the lesion feature expression analysis system 100, respectively.

More specifically, the convolution neural network application unit 130 included in the lesion feature expression analysis system 100 extracts lesion space feature expressions of each of the plurality of sectional images 120 using the convolutional neural network.

The convolution neural network is a neural network technique for analyzing an image using spatial information in an image and extracting spatial features in the image. The convolution neural network includes a convolution layer, a max polling layer and a pulley connected layer (Fully Connected Layer). The detailed description thereof will be omitted because it goes beyond the technical idea of the present invention.

For example, the convolutional neural network application unit 130 can extract the lesion spatial feature expressions of each of the plurality of sectional images 220 by applying the convolution neural network having the structure shown in Table 1 below. However, the convolutional neural network is not limited to or limited to the structure shown in Table 1, but may be a parameter, such as a layer, an input shape, a kernel size, a stride, a pad, or an output shape ) Can have various structures.

Layer Input shape Kernel size Stride Pad Output shape Input 64x64x1 - - - 64x64x1 C1 64x64x1 3x3 One One 64x64x64 C2 64x64x64 3x3 One One 64x64x64 M2 64x64x64 2x2 2 0 32x32x64 C3 32x32x64 3x3 One One 32x32x128 C4 32x32x128 3x3 One One 32x32x128 M4 32x32x128 2x2 2 0 16x16x128 C5 16x16x128 3x3 One One 16x16x256 C6 16x16x256 3x3 One One 16x16x256 C7 16x16x256 3x3 One One 16x16x256 M7 16x16x256 2x2 2 0 8x8x256 C8 8x8x256 3x3 One One 8x8x512 C9 8x8x512 3x3 One One 8x8x512 C10 8x8x512 3x3 One One 8x8x512 M10 8x8x512 2x2 2 0 4x4x512 C11 4x4x512 3x3 One One 4x4x512 C12 4x4x512 3x3 One One 4x4x512 C13 4x4x512 3x3 One One 4x4x512 M13 4x4x512 2x2 2 0 2x2x512 F14 2x2x512 - - - 1024x1 F15 1024x1 - - - 1024x1

In Table 1, C, M, and F denote a convolution layer, a max polling layer, and a pulley connected layer, respectively.

In this case, the convolution neural network application unit 130 assigns the same weight to each of the plurality of sectional images 120, so that the recursive neural network application unit 140 included in the later described lesion feature expression analysis system 100 includes a plurality of In recognizing the depth direction symmetry of the lesion along the depth direction of the sectional images 120, it is possible to consider applying the same weight to the lesion feature vectors of each of the plurality of sectional images 120.

In the drawing, the convolution neural network application unit 130 is grouped into two groups along the depth direction (a first group including the up-centered sectional images and a second group including the down-centered sectional images) Sectional images of each of the sectional images 120 of the sectional images 120 in the process of recognizing the feature vectors of each of the sectional images 120 by the recursive neural network application unit 140, The lesion spatial feature representation of each of the plurality of sectional images 120 is grouped to recognize the depth direction symmetry of the lesion, so that it is shown as a drawing for convenience.

That is, instead of grouping the plurality of sectional images 120, the convolution neural network application unit 130 extracts the lesion space feature expressions of each of the plurality of sectional images separately, Lt; RTI ID = 0.0 > lesions. ≪ / RTI >

The recursive neural network application unit 140 uses the recursive neural network to generate a plurality of sectional images based on the lesion spatial feature representation of each of the plurality of sectional images 120 along the depth direction of the plurality of sectional images 120 120), respectively.

In other words, the recursive neural network application unit 140 learns the lesion space feature expressions of each of the plurality of sectional images 120 extracted by the convolution neural network application unit 130 according to the depth direction using the recursive neural network, The lesion feature expressions of each of the plurality of sectional images 120 can be extracted in consideration of the depth. Hereinafter, it is described that a recursive neural network algorithm used by the recursive neural network application unit 140 uses a long short-term memory (LSTM). However, the present invention is not limited to this and is applicable to a simple RNN (Recurrent Neural Network) Unit) can be used.

Here, the reflexive neural network including the LSTM is a neural network technique for performing a current state based on a previous state by cyclically constructing a state for processing data, and a detailed description thereof will be omitted from the technical idea of the present invention Therefore, it is omitted.

In particular, the recursive neural network application unit 140 may perform the process of analyzing the lesion space feature expressions of each of the plurality of sectional images 120 along the depth direction of the plurality of sectional images 120 in two steps .

For example, the recursive neural network application unit 140 may include a first step of recognizing a lesion feature vector of each of the plurality of sectional images 120 based on the depth direction of the plurality of sectional images 120, A second step of acquiring lesion feature expressions of each of the plurality of sectional images 120 by fusing the lesion feature vectors of the images 120 may be performed. Hereinafter, the recognition of the lesion feature vector of each of the plurality of sectional images 120 means that the change of the lesion spatial feature representation of each of the plurality of sectional images 120 is recognized.

Accordingly, the recursive neural network application unit 140 encodes the texture transformations of the lesions of each of the plurality of sectional images 120 in the depth direction, thereby generating a sub-recursion function for recognizing the lesion feature vectors of each of the plurality of sectional images 120 The neural network application unit 141 and the lesion feature vectors of the plurality of sectional images 120 are fused to encode the depth direction symmetry of the plurality of sectional images 120 so that the lesion of each of the plurality of sectional images 120 And an upper recursive neural network application unit 142 for acquiring feature expressions.

At this time, the recursive neural network application unit 140 applies the lesion spatial feature representation of each of the plurality of sectional images 120 to the plurality of groups (the up-center direction A first group including the sectional images and a second group including the lower-center direction sectional images), and then recognize the lesion feature vector of each of the plurality of sectional images 120 for each group. A detailed description thereof will be described with reference to Fig.

The recursive neural network application unit 140 can be learned based on a plurality of objective functions so that the lesion feature expressions of each of the plurality of sectional images 120 can be efficiently extracted. A detailed description thereof will be made with reference to FIGS. 3A to 3C and FIGS. 6 to 7. FIG.

When the lesion feature expressions of each of the plurality of sectional images 120 are extracted as described above, the recurrent neural network application unit 140 applies a plurality of sectional images based on the lesion feature expressions of the plurality of sectional images 120 It is possible to judge whether the lesion represented by the plurality of sectional images 120 is a mass or not by classifying each lesion.

For example, the recursive neural network application unit 140 classifies the lesion feature expressions of each of the plurality of sectional images 120 into Mass class or False Positive class, ) It is possible to judge whether the lesion is a mass or not by judging whether each lesion characteristic expression belongs more to the mass class or the false positive class.

At this time, the recursive neural network application unit 140 optimizes various objective functions and classifies the lesions of each of the plurality of sectional images 120, thereby minimizing misclassification of lesions and improving lesion detection performance. A detailed description thereof will be made with reference to FIGS. 3A to 3C.

2 is a conceptual diagram for explaining a method of analyzing a lesion feature expression according to an embodiment. More specifically, FIG. 2 is a conceptual diagram for explaining the operation of the recursive neural network application unit included in the lesion feature presentation system shown in FIG. 1 in detail.

2, in the convolution neural network application unit 210 included in the lesion feature expression analysis system 200 according to an exemplary embodiment, a lesion space feature expression of each of the plurality of sectional images 220 is calculated using the convolutional neural network The recursive neural network application unit 230 included in the lesion feature expression analysis system 200 generates a plurality of sectional images 220 along the depth direction of the plurality of sectional images 220 using the recursive neural network The lesion feature expressions of each of the plurality of sectional images 220 are extracted based on the lesion spatial feature representation of the lesion space feature.

At this time, the recursive neural network application unit 230 groups the lesion spatial feature expressions of each of the plurality of sectional images 220 based on the depth direction of the plurality of sectional images 220, The lesion feature vectors of each of the plurality of sectional images 220 can be acquired by fusing the lesion feature vectors of each of the plurality of sectional images 220 by recognizing the lesion feature vectors of the respective lesion feature vectors 220.

For example, the first sub recursion neural network application unit 231 and the second sub recursion neural network application unit 232 included in the recursive neural network application unit 230 may calculate the lesion space feature expressions Sectional images of the cross-sectional images belonging to the first group are grouped into a first group including the up-center direction sectional images and a second group including the down-center direction sectional images, respectively, And recognizes the lesion feature vector of each of the sectional images belonging to the second group.

<Formula 1>

<Formula 2>

은 제1 하위 재귀 신경망 적용부(231)가 적용하는 재귀 신경망 함수를 의미하고,

은 제2 하위 재귀 신경망 적용부(232)가 적용하는 재귀 신경망 함수를 의미하며,

는 복수의 단면영상들(220) 중 k번째 단면영상의 병변 공간 특징 표현을 의미하고,

은 제1 그룹에 속하는 단면영상들 각각의 병변 특징 벡터를 의미하며,

은 제2 그룹에 속하는 단면영상들 각각의 병변 특징 벡터를 의미하고,

는 복수의 단면영상들(220)의 개수를 의미한다.In Equations 1 and 2,

Denotes a recursive neural network function applied by the first sub recursive neural network application unit 231,

Refers to a recursive neural network function applied by the second sub recursive neural network application unit 232,

Represents a lesion spatial feature representation of a k-th sectional image of a plurality of sectional images 220,

Is a lesion feature vector of each of the sectional images belonging to the first group,

Denotes a lesion feature vector of each of the sectional images belonging to the second group,

The number of the cross-sectional images 220 is determined.

In the above description, the sub-recursive neural network applying units 231 and 232 are formed of two constituent units, and the lesion spatial feature expressions of the plurality of sectional images 220 are grouped into two groups. However, The number of constituent parts of the sub recursive neural network application units 231 and 232 can be determined according to the number of groups to which the lesion spatial feature representation of each of the plurality of sectional images 220 is to be grouped.

The upper recursive neural network application unit 233 included in the recursive neural network application unit 230 calculates the number of lesions of each of the sectional images belonging to the first group recognized by the first sub recursive neural network application unit 231, The feature vector and the lesion feature vector of each of the sectional images belonging to the second group recognized by the second lower recurrent neural network application unit 232 are fused to recognize the depth direction symmetry of each lesion of the plurality of sectional images 220 And obtain a lesion feature representation of each of the plurality of sectional images 220 based on the depth direction symmetry of the lesion of each of the plurality of sectional images 220.

<Formula 3>

은 상위 재귀 신경망 적용부(233)가 적용하는 재귀 신경망 함수를 의미하고,

은 제1 그룹에 속하는 단면영상들 각각의 병변 특징 벡터를 의미하며,

은 제2 그룹에 속하는 단면영상들 각각의 병변 특징 벡터를 의미하고,

는 복수의 단면영상들(220) 각각의 병변 특징 표현을 의미하며,

는 복수의 단면영상들(220)의 개수를 의미한다.In Equation 3,

Denotes a recursive neural network function applied by the upper recursive neural network application unit 233,

Is a lesion feature vector of each of the sectional images belonging to the first group,

Denotes a lesion feature vector of each of the sectional images belonging to the second group,

Means a lesion feature representation of each of the plurality of sectional images 220,

The number of the cross-sectional images 220 is determined.

In this case, the upper recursion neural network application unit 233 applies the lesion feature vector of each of the sectional images belonging to the first group recognized by the first lower recurrent neural network application unit 231 and the feature vector of the second lower recurrent neural network application unit 232 In the process of fusing the lesion feature vectors of each of the cross-sectional images belonging to the recognized second group, the cross-sectional images corresponding to each other between the groups (the specific cross-sectional images included in the first group and the second cross- Sectional images) can be fused.

For example, the upper recursion neural network application unit 233 fuses the lesion feature vector of the highest-level sectional image included in the first group and the lesion feature vector of the lowest-level sectional image included in the second group, A lesion characteristic expression of each of the plurality of sectional images 220 can be obtained by fusing the lesion characteristic vector of the central sectional image and the lesion characteristic vector of the central sectional image included in the second group.

In this manner, the upper recursion neural network applying unit 233 fuses the lesion feature vectors of each of the plurality of sectional images 220 in consideration of the depth direction of the plurality of sectional images 220, To be extracted.

Here, the lesion characteristic expression obtained by the fusion process described above is limited to half of the number of the plurality of sectional images 220. However, since the lesion feature vector of each of the plurality of sectional images 220 is acquired by fusion, the lesion feature expression can represent the feature expression of each lesion of the plurality of sectional images 220. Therefore, hereinafter, a lesion feature expression obtained by the above-described fusion process is described as a lesion characteristic expression of each of the plurality of sectional images 220.

1, the same weight is given to each of the plurality of sectional images 220, so that the upper recursive neural network applying unit 233 applies a plurality of sectional images 220 having the same weight to each other, By fusing each lesion feature vector, the depth direction symmetry of lesions of each of the plurality of sectional images 220 can be recognized.

The upper recursion neural network applying unit 233 may be configured to apply the recursive neural network 220 to each of the plurality of sectional images 220 so that the objective function related to the correspondence between the lesion feature vectors of each of the plurality of sectional images 220, Can be fused.

The recursive neural network application unit 230 is divided into the lower recursive neural network application units 231 and 232 and the upper recursive neural network application unit 233. However, the present invention is not limited thereto, 231, and 232 and the upper recursive neural network application unit 233, respectively.

In addition, when a lesion feature representation of each of the plurality of sectional images 220 is obtained, the recursive neural network application unit 230 applies a plurality of sectional images 220 based on lesion feature expressions of each of the obtained plurality of sectional images 220 The lesion of each of the lesions 220 can be classified. In this case, the recursive neural network application unit 230 may classify lesions of each of the plurality of sectional images 220 so that the objective function related to classification errors that classify the lesions of each of the plurality of sectional images 220 is minimized .

In addition, the recursive neural network application unit 230 minimizes the objective function related to intra-class variance in which lesions of each of the plurality of sectional images 220 are classified in the process of classifying lesions of each of the plurality of sectional images 220 The lesion of each of the plurality of sectional images 220 may be classified.

A detailed description of such objective functions will be described with reference to FIGS. 3A to 3C.

Operations of the convolutional neural network application unit 210 and the recurrent neural network application unit 230 as described above can be performed by a neural network having the structure shown in Table 2 below. However, the lesion feature expression analysis system 200 is not limited to or limited to the structure of Table 2, and can have various structures through modification of parameters (layer, input type, kernel size, stride, pad, .

Type Layer Input shape Output shape Input Input

x (64x64x1)

x (64x64x1)

x (64x64x1) Slice feature representation C1 ~ F15 in Table 1

x (64x64x1)

x (1024x1) Depth directional feature representation L16 / L16

x (1024x1)

x (1024x1) L17

x (2048x1)

x (1024x1) F18

x (1024x1)

x (

x1)

In Table 2, Slice feature representation refers to a lesion spatial feature representation of each of a plurality of sectional images 220 extracted and performed by the convolutional neural network application 210 as shown in Table 1, Means a lesion feature expression of each of a plurality of sectional images 220 extracted by the application unit 230. [ Therefore, L16 and L16 'mean the first sub recursive neural network applying unit 231 and the second lower recursive neural network applying unit 232, respectively, and L17 means the upper recursive neural network applying unit 233. In addition, F18 is a constituent unit for classifying the lesions of each of the plurality of sectional images 220 based on the lesion characteristic expressions of the plurality of sectional images 220 in the recursive neural network application unit 230 Not shown).

3A to 3C are diagrams for explaining objective functions used for learning a recursive neural network according to an embodiment.

3A to 3C, the lesion feature expression analysis system according to an exemplary embodiment uses an optimized reflexive neural network in the process of extracting lesion feature expressions of each of the plurality of sectional images described above with reference to FIGS. A first objective function for minimizing a classification error for classifying lesions of each of a plurality of sectional images, a second objective function for maximizing correspondence between lesion feature vectors of the center sectional images to be fused among the plurality of sectional images, The objective function or the third objective function for minimizing the intra-class variance in which the lesion of each of the plurality of sectional images is classified can be pre-learned to optimize the reflexive neural network.

Hereinafter, it is described that the recurrent neural network is learned by the lesion feature expression analysis system, but it can be learned by the depth direction recursive learning system separately provided from the lesion feature expression analysis system. Here, the depth direction recursion learning system can be interlocked with the lesion feature expression analysis system to learn the recurrent neural network based on the operation of the lesion feature expression analysis system described above with reference to FIGS.

Specifically, the lesion feature expression analysis system includes a method of analyzing a lesion feature expression described above with reference to FIGS. 1 and 2, in which a plurality of learning sample medical images (each of the plurality of learning sample medical images includes a plurality of sectional images , It is possible to learn the recursive neural network to optimize at least one of the first objective function, the second objective function or the third objective function.

을최소화하도록 재귀 신경망을 학습시킬 수 있다.For example, as shown in FIG. 3A, the lesion characteristic expression analysis system calculates an error of a result value in which a lesion of a k'th sectional image included in an i-th learning sample medical image is classified into a class of a mass class or a false class The first objective function &lt; RTI ID = 0.0 &gt;

It is possible to learn the recursive neural network to minimize.

<Formula 4>

는 i번째 학습 샘플 의료 영상에 포함되는 k'번째 단면영상의 병변이 분류되는 클래스가 참값 클래스(종괴 클래스)일 확률이다.In Equation 4,

Is a probability that the class in which the lesion of the k'th sectional image included in the i-th learning sample medical image is classified is a true value class (mass class).

를 최대화하도록 재귀 신경망을 학습시킬 수 있다.As another example, the lesion feature expression analysis system may include a plurality of lesion feature vectors between each of the plurality of cross-sectional images to be fused (each of the central cross-sectional images grouped into different groups according to the depth direction) The second objective function &lt; RTI ID = 0.0 &gt;

It is possible to learn the recursive neural network to maximize.

&Lt; EMI ID =

및

은 각각 i번째 학습 샘플 의료 영상에 포함되는 k'번째 단면영상(중심 단면영상)에서 위-중심 방향 단면영상들 각각의 병변 특징 벡터 및 아래-중심 방향 단면영상들 각각의 병변 특징 벡터를 의미하고, 함수

는 Rectified Linear unit(

의 근사 함수로서,

로 정의된다.In Equation 5,

And

Represents the lesion feature vector of each of the lesion feature vector and the lower-center direction sectional image of each of the up-center directional sectional images in the k 'th sectional image (central sectional image) included in the i-th learning sample medical image , function

Rectified Linear unit (

As an approximate function of &lt; RTI ID =

.

를 최소화하도록 재귀 신경망을 학습시킬 수 있다.As another example, the lesion feature expression analysis system may calculate the third objective function &lt; RTI ID = 0.0 &gt;

Can be learned by minimizing the recursive neural network.

&Lt; EMI ID =

는 i번째 학습 샘플 의료 영상에 포함되는 k'번째 단면영상의 병변 특징 표현을 의미하고, 함수

는 Rectified Linear unit(

의 근사 함수로서,

로 정의되며,

는i번째 학습 샘플 의료 영상에서 종괴 클래스(참값 클래스)에 분류된 모든 병변 특징 표현의 평균 벡터를 의미하고,

은 종괴 클래스와 위양성 클래스 각각의 평균 특징 표현 벡터간 거리의 절반을 의미한다.In Equation 6,

Represents a lesion characteristic expression of the k 'th sectional image included in the i-th learning sample medical image,

Rectified Linear unit (

As an approximate function of &lt; RTI ID =

Lt; / RTI &gt;

Means the mean vector of all lesion feature expressions classified in the mass class (true value class) in the i-th learning sample medical image,

Means half the distance between the mean feature representation vectors of the mass class and the false positive class, respectively.

Thus, the recursive neural network is learned to optimize the first objective function, the second objective function, and the third objective function so that the lesion feature expression analysis system can optimize the first objective function, the second objective function, Neural networks can be used.

For example, the lesion feature expression analysis system may include a lesion feature vector for each of a plurality of sectional images so that a second objective function related to the correspondence between lesion feature vectors of each of the plurality of cross- And the depth direction symmetry of the lesion of each of the plurality of sectional images can be recognized by fusion.

In another example, the lesion feature expression analysis system may be configured to detect a lesion in a lesion that is associated with a variance within a class where a first objective function associated with a classification error classifying lesions of each of the plurality of sectional views is minimized, Lesions of each of the plurality of sectional images can be classified so that the third objective function is minimized.

4A and 4B are flowcharts illustrating a method of analyzing a lesion feature according to an exemplary embodiment of the present invention.

Referring to FIG. 4A, a method for analyzing a lesion feature expression according to an embodiment is performed by a lesion feature expression analysis system.

The lesion feature expression analysis system uses a Convolution Neural Network (CNN), and a plurality of cross-sectional images-a plurality of cross-sectional images are included in one 3D medical image of a lesion- (410).

Next, the lesion feature expression analysis system calculates a lesion characteristic expression expression of each of the plurality of sectional images along the depth direction of the plurality of sectional images, using a recurrent neural network based on the lesion spatial feature representation of each of the plurality of sectional images A lesion feature expression is extracted (420).

4B, the lesion feature expression analysis system recognizes (421) a lesion feature vector of each of a plurality of sectional images based on the depth direction of a plurality of sectional images at step 420, (422) a lesion feature expression of each of the plurality of sectional images by fusing the lesion feature vectors of each of the sectional images of the plurality of sectional images based on the lesion characteristic expressions of each of the plurality of sectional images, (423).

In step 421, the lesion feature expression analysis system groups the lesion spatial feature expressions of the plurality of sectional images based on the depth directions of the plurality of sectional images, Can be recognized.

Accordingly, in step 422, the lesion feature expression analysis system may acquire a lesion feature expression of each of the plurality of sectional images by fusing lesion feature vectors of the respective sectional images corresponding to each other between the groups.

In addition, in step 422, the lesion feature expression analysis system fuses the lesion feature vectors of each of the plurality of sectional images to recognize the depth direction symmetry of each lesion of the plurality of sectional images, A lesion feature representation of each of the plurality of sectional images can be obtained based on the depth direction symmetry of the lesion.

In particular, the recursive neural network used in step 420 includes a first objective function for minimizing a classification error for classifying lesions of each of a plurality of sectional images, a second objective function for minimizing a classification error between lesion feature vectors of respective center sectional images to be fused May be pre-learned before step 420 so as to optimize at least one of the second objective function for maximizing the correspondence of the plurality of sectional images or the third objective function for minimizing the intra-class variance in which the lesion of each of the plurality of sectional images is classified .

Accordingly, in step 422, the lesion feature expression analysis system can recognize the depth direction symmetry of lesions of each of the plurality of sectional images by fusing the lesion feature vectors of the plurality of sectional images so that the second objective function is maximized.

In addition, in step 432, the lesion feature expression analysis system may classify the lesions of each of the plurality of sectional images so that the first objective function is minimized based on the lesion feature expressions of the plurality of sectional images, The lesion of each of the plurality of sectional images may be classified so that the third objective function is minimized.

5 is a block diagram illustrating a lesion feature expression analysis system according to an embodiment.

Referring to FIG. 5, a lesion feature expression analysis system according to an embodiment includes a convolutional neural network application unit 510 and a recurrent neural network application unit 520. Referring to FIG.

The convolutional neural network application unit 510 includes a plurality of sectional images-a plurality of sectional images, which are included in one 3D medical image of a lesion, using a Convolution Neural Network (CNN) Expression is extracted.

The recursive neural network application unit 520 uses a recurrent neural network based on the lesion spatial feature representation of each of the plurality of sectional images to calculate a lesion of each of the plurality of sectional images along the depth direction of the plurality of sectional images Extract feature expressions.

Specifically, the recursive neural network application unit 520 recognizes the lesion feature vector of each of the plurality of sectional images based on the depth direction of the plurality of sectional images, fuses the lesion feature vectors of each of the plurality of sectional images, By obtaining a lesion feature representation of each of the cross-sectional images, the lesion of each of the plurality of cross-sectional images can be sorted based on the lesion feature representation of each of the plurality of cross-sectional images.

At this time, the recursive neural network application unit 520 groups the lesion spatial feature expressions of the plurality of sectional images based on the depth directions of the plurality of sectional images, thereby recognizing the lesion feature vectors of each of the plurality of sectional images for each group can do.

Accordingly, the recursive neural network application unit 520 may acquire the lesion feature expressions of each of the plurality of sectional images by fusing the lesion feature vectors of the respective sectional images corresponding to each other among the groups.

In addition, the recursive neural network application unit 520 fuses the lesion feature vectors of each of the plurality of sectional images to recognize the depth direction symmetry of the lesion of each of the plurality of sectional images, A lesion feature representation of each of the plurality of sectional images can be obtained based on the depth direction symmetry.

In particular, the recursive neural network used by the recursive neural network application 520 includes a first objective function for minimizing classification errors that classify lesions of each of a plurality of sectional images, a first objective function for minimizing classification errors for classifying lesions of each of the plurality of sectional images, The second objective function for maximizing the correspondence between the lesion feature vectors or the third objective function for minimizing the intra-class variance in which the lesion of each of the plurality of sectional images is classified is optimized prior to step 420 As shown in FIG.

The learning of the recursive neural network can be performed by the recursive neural network application unit 520 as described above. However, the present invention is not limited thereto, and may be performed by a separate component. A detailed description thereof will be described with reference to Figs. 6 to 7. Fig.

Accordingly, the recursive neural network application unit 520 can recognize the depth direction symmetry of the lesion of each of the plurality of sectional images by fusing the lesion feature vectors of the plurality of sectional images so that the second objective function is maximized.

In addition, the recursive neural network application unit 520 classifies the lesions of each of the plurality of sectional images so that the first objective function is minimized based on the lesion feature expression of each of the plurality of sectional images, The lesion of each of the plurality of sectional images can be classified so that the third objective function is minimized based on the feature expression.

6 is a flowchart illustrating a depth direction recursive learning method according to an embodiment.

Referring to FIG. 6, the depth direction recursion learning method according to an exemplary embodiment uses a Convolution Neural Network (CNN) to generate a plurality of sectional images-a plurality of sectional images, - are used to extract the lesion feature expressions of each of the plurality of sectional images based on the lesion spatial feature expressions extracted from each.

Thus, the depth direction recursion learning method can learn the recursive neural network by optimizing the objective functions by repeatedly performing the lesion feature expression analysis method described with reference to FIGS. 1 to 5 on a plurality of learning sample medical images. Accordingly, the depth direction recursion learning method is performed based on the lesion feature expression analysis method described with reference to Figs. 1-5. That is, steps 610 to 630 described below may be performed adaptively after or without performing the steps of the method for analyzing a lesion feature described with reference to FIG.

Hereinafter, the depth direction recursion learning method is described as being performed by the depth direction recursive learning system. However, the depth direction recursion learning method may be performed by a lesion feature expression analysis system that performs a lesion feature expression analysis method.

The depth direction recursion learning system includes a Recurrent Neural Network (hereinafter referred to as &quot; Recurrent Neural Network &quot;) for optimizing a first objective function for minimizing a classification error for classifying lesions of each of a plurality of sectional images based on a lesion characteristic expression of each of a plurality of sectional images, (610).

The depth direction recursion learning system then calculates a depth direction recursion learning system for extracting a lesion feature representation of each of the plurality of sectional images, The reflexive neural network is learned to optimize the objective function (620).

The depth direction recursion learning system then calculates a recursion neural network to optimize a third objective function for minimizing intra-class variance in which the lesion of each of the plurality of sectional images is classified based on the lesion feature representation of each of the plurality of sectional images (630).

7 is a block diagram illustrating a depth direction recursion learning system according to an embodiment.

Referring to FIG. 7, the depth direction recursion learning system according to an embodiment includes a first objective function optimizer 710, a second objective function optimizer 720, and a third objective function optimizer 730. Here, the first objective function optimizing unit 710, the second objective function optimizing unit 720, and the third objective function optimizing unit 730 are interlocked with each component of the lesion feature expression analysis system described with reference to FIG. 5 Can operate.

The first objective function optimizer 710 may be configured to optimize a first objective function for minimizing classification errors that classify lesions of each of a plurality of sectional images based on a lesion feature representation of each of the plurality of sectional images, Recurrent Neural Network.

The second objective function optimizing unit 720 may include a second objective function optimization unit 720 for maximizing correspondence between lesion feature vectors of each of the plurality of cross-sectional images to be fused, 2 Recursive neural networks are studied to optimize objective functions.

The third objective function optimizing unit 730 optimizes the third objective function for minimizing intra-class variance in which the lesion of each of the plurality of sectional images is classified based on the lesion characteristic expression of each of the plurality of sectional images, .

The apparatus described above may be implemented as a hardware component, a software component, and / or a combination of hardware components and software components. For example, the apparatus and components described in the embodiments may be implemented within a computer system, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable array (FPA) A programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to execution of the software. For ease of understanding, the processing apparatus may be described as being used singly, but those skilled in the art will recognize that the processing apparatus may have a plurality of processing elements and / As shown in FIG. For example, the processing unit may comprise a plurality of processors or one processor and one controller. Other processing configurations are also possible, such as a parallel processor.

The software may include a computer program, code, instructions, or a combination of one or more of the foregoing, and may be configured to configure the processing device to operate as desired or to process it collectively or collectively Device can be commanded. The software and / or data may be in the form of any type of machine, component, physical device, virtual equipment, computer storage media, or device , Or may be permanently or temporarily embodied in a transmitted signal wave. The software may be distributed over a networked computer system and stored or executed in a distributed manner. The software and data may be stored on one or more computer readable recording media.

The method according to an embodiment may be implemented in the form of a program command that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, and the like, alone or in combination. The program instructions to be recorded on the medium may be those specially designed and configured for the embodiments or may be available to those skilled in the art of computer software. Examples of computer-readable media include magnetic media such as hard disks, floppy disks and magnetic tape; optical media such as CD-ROMs and DVDs; magnetic media such as floppy disks; Magneto-optical media, and hardware devices specifically configured to store and execute program instructions such as ROM, RAM, flash memory, and the like. Examples of program instructions include machine language code such as those produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is to be understood that the invention is not limited to the disclosed exemplary embodiments. For example, it is to be understood that the techniques described may be performed in a different order than the described methods, and / or that components of the described systems, structures, devices, circuits, Lt; / RTI &gt; or equivalents, even if it is replaced or replaced.

Therefore, other implementations, other embodiments, and equivalents to the claims are also within the scope of the following claims.

Claims (16)

A method for analyzing a lesion feature using depth direction recursive learning in a three-dimensional medical image,
Extracting a plurality of sectional images using the Convolution Neural Network (CNN), wherein each of the plurality of sectional images is included in one 3D medical image of a lesion; And
Extracting a lesion feature expression of each of the plurality of sectional images along a depth direction of the plurality of sectional images using a recurrent neural network based on a lesion spatial feature representation of each of the plurality of sectional images step
The method comprising the steps of: The method according to claim 1,
The step of extracting a lesion feature representation of each of the plurality of sectional images
Recognizing a lesion feature vector of each of the plurality of sectional images based on a depth direction of the plurality of sectional images;
Acquiring lesion feature expressions of each of the plurality of sectional images by fusing lesion feature vectors of each of the plurality of sectional images; And
Classifying a lesion of each of the plurality of sectional images based on a lesion characteristic expression of each of the plurality of sectional images
The method comprising the steps of: 3. The method of claim 2,
The step of recognizing a lesion feature vector of each of the plurality of sectional images
Grouping a lesion spatial feature representation of each of the plurality of sectional images based on a depth direction of the plurality of sectional images; And
Recognizing a lesion feature vector of each of the plurality of sectional images for each group
The method comprising the steps of: The method of claim 3,
Wherein acquiring a lesion feature representation of each of the plurality of cross-
Acquiring lesion feature expressions of each of the plurality of sectional images by fusing lesion feature vectors of each of the sectional images corresponding to each other among the groups
The method comprising the steps of: 3. The method of claim 2,
Wherein acquiring a lesion feature representation of each of the plurality of cross-
Recognizing depth direction symmetry of lesions of each of the plurality of sectional images by fusing lesion feature vectors of each of the plurality of sectional images; And
Obtaining a lesion feature representation of each of the plurality of sectional images based on a depth direction symmetry of the lesion of each of the recognized plurality of sectional views
The method comprising the steps of: 3. The method of claim 2,
The recursive neural network
A first objective function for minimizing a classification error for classifying lesions of each of the plurality of sectional images, a second objective function for minimizing a classification error of each of the plurality of sectional images, 2 objective function or a third objective function for minimizing intra-class variance in which lesions of each of the plurality of sectional images are classified. The method according to claim 6,
Wherein acquiring a lesion feature representation of each of the plurality of cross-
Recognizing the depth direction symmetry of the lesion of each of the plurality of sectional images by fusing lesion feature vectors of each of the plurality of sectional images so that the second objective function is maximized
The method comprising the steps of: The method according to claim 6,
The step of classifying lesions of each of the plurality of sectional images
Classifying lesions of each of the plurality of sectional images so that the first objective function is minimized based on a lesion feature expression of each of the plurality of sectional images
The method comprising the steps of: The method according to claim 6,
The step of classifying lesions of each of the plurality of sectional images
Classifying lesions of each of the plurality of sectional images so that the third objective function is minimized based on a lesion feature expression of each of the plurality of sectional images
The method comprising the steps of: A plurality of cross-sectional images using a Convolution Neural Network (CNN), the plurality of cross-sectional images being included in a single three-dimensional medical image of a lesion, In a depth direction recursive learning method used to extract a lesion feature expression of each of cross-sectional images of a cross-
Learning a Recurrent Neural Network to optimize a first objective function for minimizing classification errors that classify lesions of each of the plurality of sectional images based on a lesion feature expression of each of the plurality of sectional images; ;
To optimize a second objective function to maximize correspondence between lesion feature vectors of each of the plurality of cross-sectional images to be fused, to extract a lesion feature representation of each of the plurality of cross-sectional images, Learning a neural network; And
Learning the recursive neural network to optimize a third objective function for minimizing intra-class variance in which lesions of each of the plurality of sectional images are classified based on a lesion feature expression of each of the plurality of sectional images
The depth direction recursion learning method comprising: A computer program stored in a medium for executing a method of analyzing a lesion feature expression using depth directional recursion in a three-dimensional medical image in combination with a computer implementing an electronic device,
The lesion feature expression analysis method
Extracting a plurality of sectional images using the Convolution Neural Network (CNN), wherein each of the plurality of sectional images is included in one 3D medical image of a lesion; And
Extracting a lesion feature expression of each of the plurality of sectional images along a depth direction of the plurality of sectional images using a recurrent neural network based on a lesion spatial feature representation of each of the plurality of sectional images step
&Lt; / RTI &gt; 12. The method of claim 11,
The step of extracting a lesion feature representation of each of the plurality of sectional images
Recognizing a lesion feature vector of each of the plurality of sectional images based on a depth direction of the plurality of sectional images;
Acquiring lesion feature expressions of each of the plurality of sectional images by fusing lesion feature vectors of each of the plurality of sectional images; And
Classifying a lesion of each of the plurality of sectional images based on a lesion characteristic expression of each of the plurality of sectional images
&Lt; / RTI &gt; 13. The method of claim 12,
The recursive neural network
A first objective function for minimizing a classification error for classifying lesions of each of the plurality of sectional images, a second objective function for minimizing a classification error of each of the plurality of sectional images, 2 objective function or a third objective function for minimizing intra-class variance in which lesions of each of the plurality of sectional images are classified. A lesion feature expression analysis system using depth direction recursion learning in a three-dimensional medical image,
A plurality of sectional images using the Convolution Neural Network (CNN), wherein the plurality of sectional images are included in one 3D medical image of the lesion, and a convolution neural network extracting each lesion spatial feature representation is applied part; And
Extracting a lesion feature expression of each of the plurality of sectional images along a depth direction of the plurality of sectional images using a recurrent neural network based on a lesion spatial feature representation of each of the plurality of sectional images The recurrent neural network application
Wherein the lesion feature expression analysis system comprises: 15. The method of claim 14,
The recurrent neural network application unit
Recognizing a lesion feature vector of each of the plurality of sectional images based on a depth direction of the plurality of sectional images, and fusing lesion feature vectors of the plurality of sectional images to detect lesion characteristics And classifies lesions of each of the plurality of sectional images based on a lesion feature representation of each of the plurality of sectional images. 16. The method of claim 15,
The recursive neural network
A first objective function for minimizing a classification error for classifying lesions of each of the plurality of sectional images, a second objective function for minimizing a classification error of each of the plurality of sectional images, 2 objective function or a third objective function for minimizing intra-class variance in which lesions of each of the plurality of sectional images are classified.

Images