KR102216279B1 - Method for interpretating visual evidence according to breast mass characteristics and the system thereof - Google Patents

Abstract

The present invention relates to a method and a system for interpreting visual evidence for the shape and margin of a breast mass using a deep network, the step of encoding a target mass characteristic, and the encoded target mass characteristic And generating mass images of corresponding shape characteristics and boundary characteristics, and measuring visual evidence for the target mass using a deep network that receives the mass image and mass characteristic samples as inputs.

Description

A method of interpreting visual evidence according to the characteristics of a breast mass and its system {METHOD FOR INTERPRETATING VISUAL EVIDENCE ACCORDING TO BREAST MASS CHARACTERISTICS AND THE SYSTEM THEREOF}

The present invention relates to a method and system for interpreting visual evidence according to characteristics of a breast mass, and more particularly, a method and system for interpreting visual evidence for the shape and margin of a breast mass using a deep network. It is about.

For women, breast tumors are the second tumor type after skin tumors, but the first tumor type when it comes to deaths.

In particular, the anxious phenomenon is that the average age at which tumors occur is constantly decreasing, and the current screening tool is a feature of premenopausal women and is generally a characteristic of women in good health, which is difficult to diagnose high-density breasts. It is not enough. In fact, mammography is not reliable enough for high-density breasts. It can be inefficient if it is not'pre-directed' to the attentional area that needs to be examined. Current systems that can provide functional information or diagnose high-density breasts, such as MRI, CT, and PET, are very expensive, so they are not suitable for screening or mass diagnostics, and are invasive and repeat in a short period of time. Can't be

In a general context, today's diagnostic pathways always involve the use of ultrasound downstream of each primary diagnosis that is not clear whether it is negative or not. This is because today's primary diagnostic methods do not have sufficient sensitivity to diagnose malignant or benign lesions and sufficient specificity to find the correct location.

Recent studies are using deep learning technology to improve the results of medical image analysis, and it is reported that deep learning technology can obtain good performance for medical image analysis.

However, deep learning technology has achieved great success in a wide range of image analysis applications such as image recognition and medical image analysis, but there is a limit to the application of deep learning technology.

The main reason is the black-box nature of deep learning. The black box characteristic of deep networks is a key limitation that must be solved to increase the reliability of deep learning in computer-aided diagnosis (CADx). However, there are very few studies conducted to interpret the clinical decision of the deep network of the computerized automatic diagnosis system, and an embodiment technology to which the existing deep learning technology is applied is limited in interpreting the clinical decision by the black box characteristic. There is a problem.
[Prior technical literature]
[Patent Literature]
Korean Patent Publication No. 10-2014-0093376 (published on 28 July 2014)

An object of the present invention is to provide a method and system capable of providing a visual evidence capable of analyzing a deep network by decomposing a visual analysis method for a target mass characteristic into shape characteristics and boundary characteristics.

In addition, the present invention is a method and system capable of improving the clinical accuracy of a deep network by improving spatial characteristics by using an interpretationlet of shape characteristics and boundary characteristics that display areas related to the shape and boundary of a mass. Want to provide.

A method for analyzing visual evidence according to mass characteristics according to an embodiment of the present invention includes encoding a target mass characteristic, generating a mass image of shape characteristics and boundary characteristics corresponding to the encoded target mass characteristic, and the mass image And measuring visual evidence for the target mass by using a deep network using the mass characteristic sample as inputs.

In the encoding step, mass characteristics of a region-of-interest (ROI) image for the target mass may be encoded.

In the generating step, mass images of each of the shape characteristics and the boundary characteristics corresponding to the target mass characteristics may be generated using a visual interpretation network that receives the encoded target mass characteristics as input.

In the measuring step, a shape relevance score for the target mass is calculated using a shape guide network in which the mass image of the shape characteristic and a shape characteristic sample are input as inputs, and visual evidence visual evidence by calculating a margin relevance score for the target mass using the step of measuring evidence) and margin guide networks that input the mass image of the boundary characteristic and the boundary characteristic sample as inputs. It may include measuring (visual evidence).

The induction network may be configured to output a relevance score by calculating an encoded mass image obtained by encoding the mass image and embedded description information for the shape feature or the boundary feature as inputs.

In addition, the method of interpreting visual evidence according to mass characteristics according to an embodiment of the present invention may further include verifying a clinical decision using the mass image merged with the encoded target mass characteristics.

The verifying step is performed on the target mass using a diagnostic network that calculates and inputs the merged mass image obtained by merging the mass image of the shape feature and the mass image of the boundary feature and the encoded target mass feature. Malignant or benign clinical decisions can be verified.

In the verifying step, in the process of verifying a clinical decision using the diagnostic network, the shape characteristic and the boundary characteristic of the target mass may be acquired based on the shape of the mass and the boundary of the mass learned in advance.

The deep network obtains the encoded target mass characteristic by encoding a target mass characteristic representing a region-of-interest (ROI) according to guided information on a spatial characteristic map, and the target mass characteristic It may be learned through a process of generating the mass image of the shape characteristic and the boundary characteristic corresponding to, and improving spatial characteristics due to visual evidence measured by the generated mass image and the mass characteristic sample.

In another embodiment of the present invention, a method for analyzing visual evidence according to mass characteristics includes encoding target mass characteristics, generating mass images of shape characteristics and boundary characteristics corresponding to the encoded target mass characteristics, and the mass image And measuring visual evidence for a target mass using a deep network that receives a mass characteristic sample as an input, and verifying a clinical decision using the mass image merged with the encoded target mass characteristic. Include.

The visual evidence analysis system according to mass characteristics according to an embodiment of the present invention includes an encoding unit for encoding target mass characteristics, a visual analysis unit for generating mass images of shape characteristics and boundary characteristics corresponding to the encoded target mass characteristics, and And a measuring unit for measuring visual evidence for the target mass using a deep network that receives the mass image and mass characteristic sample as inputs.

The encoding unit may encode a mass characteristic of a region-of-interest (ROI) image for the target mass.

The visual analysis unit may generate a mass image of each of the shape characteristics and the boundary characteristics corresponding to the target mass characteristics by inputting the encoded target mass characteristics.

The measurement unit calculates a shape relevance score for the target mass using guide networks that input the mass image of the shape feature and the shape feature sample as inputs to provide visual evidence. By using a shape measuring unit to measure and a boundary-relevance score for the target mass using guide networks that input the mass image of the boundary characteristic and the boundary characteristic sample as an input, a visual evidence (visual evidence) is obtained. It may include a boundary measuring unit for measuring evidence).

The induction network may be configured to output a relevance score by calculating an encoded mass image obtained by encoding the mass image and embedded description information for the shape feature or the boundary feature as inputs.

In addition, the visual evidence analysis system according to mass characteristics according to an embodiment of the present invention may further include a verification unit for verifying a clinical decision using the mass image merged with the encoded target mass characteristics.

The verification unit uses a diagnostic network (diagnosis network) that calculates the merged mass image obtained by merging the mass image of shape characteristics and the mass image of boundary characteristics, and the encoded target mass characteristics, and uses a diagnostic network for the target mass. Alternatively, a positive clinical decision can be verified.

The verification unit may acquire the shape characteristic and the boundary characteristic for the target mass based on the shape of the mass and the boundary of the mass learned in advance in the process of verifying the clinical decision using the diagnostic network.

The deep network obtains the encoded target mass characteristic by encoding a target mass characteristic representing a region-of-interest (ROI) according to guided information on a spatial characteristic map, and the target mass characteristic It may be learned through a process of generating the mass image of the shape characteristic and the boundary characteristic corresponding to, and improving spatial characteristics due to visual evidence measured by the generated mass image and the mass characteristic sample.

A visual evidence analysis system according to mass characteristics according to another embodiment of the present invention includes an encoding unit for encoding target mass characteristics, a visual analysis unit for generating mass images of shape characteristics and boundary characteristics corresponding to the encoded target mass characteristics, A clinical decision is made using a measurement unit that measures visual evidence for a target mass using a deep network that takes the mass image and mass characteristic sample as inputs, and the mass image merged with the encoded target mass characteristic. It includes a verification unit to verify.

According to an exemplary embodiment of the present invention, visual evidence capable of analyzing a deep network may be provided by decomposing a visual analysis method for a target mass characteristic into shape characteristics and boundary characteristics.

In addition, according to an embodiment of the present invention, the clinical accuracy of the deep network is improved by improving the spatial characteristics by using an interpretationlet network of shape characteristics and boundary characteristics that display areas related to the shape and boundary of the mass. I can make it.

In addition, according to an embodiment of the present invention, when a clinical decision is correct in the proposed interpretationlet network, it can provide appropriate visual evidence for this.

1A and 1B are flowcharts illustrating a method for interpreting visual evidence according to an embodiment of the present invention.
2 is a diagram illustrating an example of visual evidence for a clinical decision of a deep network according to an embodiment of the present invention.
3 shows a conceptual structure of an analysis network according to an embodiment of the present invention.
4 shows a conceptual structure of an induction network according to an embodiment of the present invention.
5 shows a conceptual structure of a verification network according to an embodiment of the present invention.
6 is a block diagram showing a detailed configuration of a visual evidence interpretation system according to an embodiment of the present invention.
7 is a block diagram showing a detailed configuration of a measuring unit according to an embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. However, the present invention is not limited or limited by the embodiments. In addition, the same reference numerals shown in each drawing denote the same member.

In addition, terms used in the present specification are terms used to properly express preferred embodiments of the present invention, which may vary depending on the intention of viewers or operators, or customs in the field to which the present invention belongs. Accordingly, definitions of these terms should be made based on the contents throughout the present specification.

Embodiments of the present invention make use of a deep network to interpret visual evidence for the shape and margin of a breast mass.

The technology according to the present invention proposes a new deep network called an interpretationlet network to visually interpret clinical decisions of a computer-aided diagnosis (CADx). That is, the present invention displays an important region of a region-of-interest (ROI) image, and decomposes the visual interpretation method into boundary characteristics and shape characteristics to provide interpretable visual evidence of a deep network. have. In addition, the interpretationlet network according to an embodiment of the present invention achieves clinical performance and can provide reasonable visual evidence even when a clinical decision is not correct. For this reason, the analysis network according to an embodiment of the present invention can be a promising approach to developing an interpretable computer automatic diagnosis system (CADx).

The present invention will be described with reference to FIGS. 1A to 7.

1A and 1B are flowcharts illustrating an operation of a method for interpreting visual evidence according to an embodiment of the present invention. For example, FIG. 1B illustrates an entire framework of a method for analyzing visual evidence according to an embodiment of the present invention. Is shown.

1A and 1B, a method according to an embodiment of the present invention encodes a target mass characteristic (step 110).

In step 110, the encoding unit may encode a mass characteristic of an input image, for example, a region-of-interest (ROI) image for a target mass suspected of being a breast mass.

For example, the encoding unit is based on guided information on the spatial characteristic map of the Breast Imaging-Reporting and Data System (BIRADS), the mass characteristics of the region of interest image showing the region of interest. Can be encoded.

A mass image of a shape feature and a margin feature corresponding to the target mass feature encoded in step 110 is generated (Step 120).

In step 120, the visual analysis unit separates and generates mass images for each shape feature and boundary feature corresponding to the target mass feature using a visual interpretation network that inputs the target mass feature encoded by the encoding unit. I can. Furthermore, the visual analysis unit may provide improved visual characteristics by emphasizing the spatial characteristics of the shape characteristics and boundary characteristics of the target mass characteristics.

Visual evidence for the target mass is measured using a deep network using the mass image generated in step 120 and mass characteristic sample as inputs (step 130).

In step 130, the measurement unit calculates a shape relevance score for the target mass using guide networks that input the mass image of the shape feature and the shape feature sample as inputs to provide visual evidence. It is possible to measure visual evidence by calculating the margin relevance score for the target mass using guide networks that take the mass image of the boundary characteristic and the boundary characteristic sample as input. I can.

At this time, the induction network may output a relevance score by calculating an encoded mass image that encodes the mass image and embedding description information for shape characteristics or boundary characteristics as inputs. The higher is, the higher the score can be calculated.

As an example, the shape measurement unit may be provided to interpret a clinical decision as visual evidence by using a shape inducing network that receives a mass image of a shape characteristic and a sample of an irregular shape (Irregular) as inputs. In step 130, the measurement unit encodes the mass image of the shape characteristic through the visual characteristic encoder, outputs the embedding characteristic based on the description embedding for the description of the shape characteristic sample, and refines the image characteristic and Related scores can be calculated based on embedding characteristics. In this case, the refined image property may be an image property of a refined image obtained by encoding an input image refined by an encoding unit according to an embodiment of the present invention through a visual feature encoder.

As another example, the boundary measuring unit may be provided to interpret a clinical decision as visual evidence by using a boundary guidance network that receives a boundary characteristic mass image and a spiculate boundary characteristic sample as inputs. In step 130, the measurement unit encodes the mass image of the boundary characteristic through a visual characteristic encoder, outputs the embedding characteristic based on the description embedding for the description of the boundary characteristic sample, and refines the image characteristic and Related scores can be calculated based on embedding characteristics. In this case, the refined image property may be an image property of a refined image obtained by encoding an input image refined by an encoding unit according to an embodiment of the present invention through a visual feature encoder.

In addition, by calculating the relevant score using the guidance network (shape guidance network and the boundary guidance network) based on the mass image and the lesion characteristic sample, and updating the visual evidence measurement process and the spatial characteristic improvement process, the deep network of the present invention By repeating the training process, the deep network can be trained.

The analysis network, which is a deep network that provides a detailed level of visual interpretation for clinical decisions according to the present invention, is the shape of the mass and the boundary of the mass (also referred to as mass lexicon) in order to increase the image interpretation ability. Disassemble the visual evidence according to. At this time, the mass vocabulary is mainly used to record the characteristics of the breast mass.

Referring to FIG. 2, an example of shape characteristics and boundary characteristics of a decomposed mass is shown.

2 is a diagram illustrating an example of visual evidence for a clinical decision of a deep network according to an embodiment of the present invention.

More specifically, FIG. 2(a) shows an image of an original region of interest, and FIG. 2(b) shows an image visualized through a class activation map method, and FIG. 2(c) and 2(d) shows a mass image of shape characteristics and boundary characteristics proposed in the present invention.

Referring to FIG. 2, for visualization of shape characteristics and boundary characteristics, a class activation map and a mass image are normalized to [0, 1], and may be overlapped with a region-of-interest (ROI) image.

Furthermore, it can be seen that the interpretation network, which is the deep network of the present invention, can provide appropriate visual evidence when the clinical decision is correct. On the other hand, when the deep network makes an error decision, the analysis network may indicate an area without mass.

The present invention can improve the clinical accuracy of a deep network by improving spatial characteristics by using a mass image of a shape characteristic and a boundary characteristic representing a region related to a mass vocabulary.

Referring again to FIGS. 1A and 1B, the method according to another embodiment of the present invention verifies a clinical decision using the encoded target mass characteristic and the merged mass image (step 140).

In step 140, the verification unit merges the mass image of the shape feature and the mass image of the boundary feature extracted in step 120, and calculates the encoded target mass feature and calculates the diagnostic network (diagnosis network). ) Can be used to verify a malignant or benign clinical decision (diagnostic decision) for the target mass.

In step 140, in the process of verifying the clinical decision using the diagnostic network, shape characteristics and boundary characteristics of the target mass may be acquired based on the previously learned shape of the mass and the boundary of the mass. In other words, in step 140, the diagnostic network may output three types of a clinical decision, a margin characteristic, and a shape characteristic.

In the method according to an embodiment of the present invention, the deep network is a target encoded by encoding a target mass characteristic representing a region-of-interest (ROI) according to guided information on a spatial characteristic map. It is learned through the process of acquiring mass characteristics, generating mass images of shape characteristics and boundary characteristics corresponding to target mass characteristics, and improving spatial characteristics due to visual evidence measured by the generated mass image and mass characteristic samples. I can.

3 shows a conceptual structure of an analysis network according to an embodiment of the present invention.

As shown in FIG. 3, the interpretationlet network includes a spatial feature encoding unit (or spatial feature encoder), a visual analysis unit of a visual analysis network that separates shape characteristics and boundary characteristics, a verification unit of a diagnostic network, and a derivation. It consists of the measurement unit of the network.

For example, the visual interpretation network is configured to generate a mass image of each shape characteristic and boundary characteristic corresponding to the target mass characteristic by taking the encoded target mass characteristic as an input. In addition, the induction network is configured to measure visual evidence by calculating a relevance score for the target mass by inputting a mass image of a shape characteristic or boundary characteristic and a shape characteristic sample or boundary characteristic sample as inputs. In addition, the diagnostic network is configured to verify the clinical decision of malignant or benign for the target mass by calculating the merged mass image obtained by merging the mass image of the shape feature and the mass image of the boundary feature and the encoded target mass feature as input. .

4 shows a conceptual structure of an induction network according to an embodiment of the present invention.

4 shows a detailed structure of the shape guidance network, but the boundary guidance network is also designed in the same structure as the shape guidance network.

Referring to FIG. 4, the shape measurement unit according to an embodiment of the present invention includes a visual characteristic encoder and a description embedding, and a mass image of the shape characteristic generated by the visual analysis unit is transmitted through a visual feature encoder. Encoding, outputting the embedding characteristics based on the description embedding layer according to the description of the shape characteristic sample (e.g., Irregular), and based on the relevant numerical measurement unit (relevance score estimator) Shape relevance scores based on refined image characteristics and embedding characteristics can be calculated. In this case, the shape relevance score can be calculated as a higher score as the degree of relevance increases.

Here, the refined image characteristic may be an image characteristic of a refined image obtained by encoding an input image refined by an encoding unit according to an embodiment of the present invention through a visual feature encoder.

5 shows a conceptual structure of a verification network according to an embodiment of the present invention.

Referring to FIG. 5, the verification unit according to an embodiment of the present invention merges the target mass characteristic encoded by the encoding unit and the mass image of the shape characteristic generated separated by the visual analysis unit and the mass image of the boundary characteristic. The malignant or benign clinical decision for the target mass can be verified by inputting the resulting mass image as an input.

Furthermore, the verification unit may acquire shape characteristics and boundary characteristics for the target mass based on the shape of the mass and the boundary of the mass learned in advance in the process of verifying the clinical decision using the diagnostic network.

Hereinafter, a visual analysis diagnosis network, a guidance network, and a verification network according to an embodiment of the present invention will be described.

Visual interpretation diagnostic network

The visual interpretation diagnosis network includes an encoder, a visual interpreter, and a diagnosis network.

The visual interpretation diagnostic network encodes a region-of-interest (ROI) image representing an important region according to guided information on a spatial feature map using an encoding unit, for example, an encoder. Here, the encoding unit encodes according to the following [Equation 1].

[Equation 1]

은 공간 특성 맵을 나타내고,

은 원본 영상(seed image)를 나타낸다. 또한,

는 학습 가능 파라미터

를 포함하는 공간 특성 인코더에 대한 함수를 나타낸다. here,

Denotes a spatial characteristic map,

Represents the original image (seed image). Also,

Is the learnable parameter

Represents a function for a spatial feature encoder including.

The target mass characteristic according to the present invention is encoded from the spatial characteristic map as shown in [Equation 2] and [Equation 3] below.

[Equation 2]

[Equation 3]

및

는 각각 경계 특성(margin interpretationlet)과 모양 특성(shape interpretationlet)을 나타낸다. 또한,

및

는 각각 학습 가능 파라미터

및

를 포함하며, 경계 특성과 모양 특성을 인코딩하는 함수를 나타낸다. here,

And

Represents a margin interpretationlet and a shape interpretationlet, respectively. Also,

And

Are each learnable parameter

And

It includes, and represents a function that encodes the boundary characteristics and shape characteristics.

By the visual interpretation network, boundary characteristics and shape characteristics are encoded from a set of spatial characteristics through a sequential convolutional layer representing sigmoid activation. Important areas obtained from boundary characteristics and shape characteristics are used to emphasize spatial characteristics as shown in [Equation 4] below.

[Equation 4]

은 경계 특성 및 모양 특성에 의해 개선된 시각적 특성을 나타내고,

은 원소 단위의 곱셈 연산을 나타낸다.here,

Represents the visual characteristics improved by the boundary characteristics and shape characteristics,

Represents an element-wise multiplication operation.

The clinical decision according to an embodiment of the present invention is determined by processing the improved visual characteristics as shown in [Equation 5] below.

[Equation 5]

는 예측되는 임상 결정(악성 또는 양성)을 나타내며,

는 학습 가능 파라미터

를 포함하는 진단 네트워크를 위한 함수를 나타낸다. here,

Represents the predicted clinical decision (malignant or benign),

Is the learnable parameter

Represents a function for a diagnostic network including.

By using two characteristics of shape characteristics and boundary characteristics, the spatial characteristic map can be improved to help the diagnostic network make more accurate clinical decisions. In addition, according to the present invention, by analyzing shape characteristics and boundary characteristics, it is possible to visualize a region in which a deep network uses more information on a corresponding spatial characteristic map.

Induction network

A detailed structure of a shape guide network comprising a visual feature encoder, a description embedding layer, and a relevance score estimator is shown in FIG. 4.

)는 모양 특성과 모양 설명을 고려하여 하기의 [수학식 6]과 같이 산출된다. Shape characteristics and shape descriptions (shape descriptions or mass lexicons) are used as inputs to the shape inducing network. In shape derivation networks for shape characteristics, shape-related figures

) Is calculated as [Equation 6] below in consideration of shape characteristics and shape description.

[Equation 6]

는 학습 가능 파라미터

를 포함하는 모양 특성을 위한 모양 유도 네트워크를 나타내며,

는 모양 설명(shape description)을 나타낸다. 세부적으로, 시각적 특성은 미리 사전 학습된 VGG 네트워크를 사용하여 하기의 [수학식 7]과 같이 모양 특성에서 추출된다. here,

Is the learnable parameter

Represents a shape inducing network for shape characteristics including,

Represents the shape description. In detail, visual features are extracted from shape features as shown in [Equation 7] below using a pre-learned VGG network.

[Equation 7]

는 시각적 특성 추출 및 파라미터

을 위한 함수를 나타낸다. 출력이 pool5 계층 및 완전 연결 계층 이후의 특성 맵인 사전 학습 VGG-네트워크가 시각적 특성 추출부(visual feature extractor)로 사용될 수 있다. 의미론적으로(semantically) 의미있는 특징을 갖도록 모양 설명을 인코딩하기 위해서, 모양 설명(shape description,

)은 하기의 [수학식 8]과 같이 사용된다. here,

Is the visual feature extraction and parameters

Represents a function for A pre-learning VGG-network whose output is a feature map after the pool5 layer and the fully connected layer may be used as a visual feature extractor. To encode a shape description to have a semantically meaningful feature, the shape description,

) Is used as in [Equation 8] below.

[Equation 8]

는 모양 임베딩 특성(shape embedded features)를 나타낸다. 또한,

는 학습 가능 파라미터

를 포함하는 임베딩 함수를 나타낸다.here,

Denotes shape embedded features. Also,

Is the learnable parameter

Represents an embedding function including.

)는 하기의 [수학식 9]를 통해 산출된다.For embedding, shape descriptions are transformed into word vectors via a pre-trained word embedding model and processed by a fully connected layer. By the difference between the shape embedding characteristics and the visual characteristics of the shape characteristics,

) Is calculated through the following [Equation 9].

[Equation 9]

은 관련 수치를 계산하며 학습 가능 파라미터

를 포함하는 관련 수치 측정부(relevance score estimator)를 나타낸다. 이로 인하여, 모양 관련 수치(

)는 모양 임베딩 특성과 모양 특성의 시각적 특성 간의 관련도를 나타낼 수 있다. here,

Is a learnable parameter that calculates the relevant number

It represents a related numerical measurement unit (relevance score estimator) including. Due to this, shape-related figures (

) Can indicate the degree of relationship between shape embedding characteristics and visual characteristics of shape characteristics.

Sigmoid activation can be performed in the last fully connected layer so that the output has a [0, 1] range.

)는 하기의 [수학식 10]과 같이 경계 특성과 경계 설명(margin description)에 의해 산출된다.The margin guide network is designed with the same structure as the shape guide network described, and similarly, the boundary-related values (

) Is calculated by the boundary characteristics and the margin description as shown in [Equation 10] below.

[Equation 10]

는 학습 가능 파라미터

를 포함하는 경계 특성을 위한 경계 유도 네트워크를 나타내며,

는 경계 설명(margin description)을 나타낸다. here,

Is the learnable parameter

Represents a boundary inducing network for boundary characteristics including,

Denotes a margin description.

Verification network

The core purpose of learning interpretationlets is to provide more accurate clinical decisions and visual evidence. For this purpose, the interpretation network according to an embodiment of the present invention has been devised for diagnosis loss and interpretation loss.

)은 하기의 [수학식 11]과 같이 정의된다. Diagnostic loss(

) Is defined as in [Equation 11] below.

[Equation 11]

는 영상(

)에 대한 실측 자료(ground truth)를 나타내며,

는 개선된 공간 특성(

)에 대한 진단 네트워크(

)의 출력 확률을 나타낸다. here,

Is the image(

Represents the ground truth for ),

Is the improved spatial characteristics (

) For the diagnostic network (

) Represents the output probability.

)을 최소화하여 학습될 수 있다. 나아가, 본 발명의 일 실시예에 따른 시각적 해석부(visual interpreter) 및 인코더(encoder) 또한 의미있는 시각적 증거를 제공하기 위하여 학습될 수 있다. Interpretation network according to an embodiment of the present invention is a diagnostic loss (

) Can be learned by minimizing it. Furthermore, a visual interpreter and an encoder according to an embodiment of the present invention may also be learned to provide meaningful visual evidence.

Due to this, the interpretation loss is defined by the following [Equation 12].

[Equation 12]

및

는 각각 모양 특성 및 경계 특성에 대한 해석 손실을 나타낸다. 이 때, 각 해석 손실은 하기의 [수학식 13] 및 [수학식 14]과 같이 정의된다.here,

And

Represents the analysis loss for shape characteristics and boundary characteristics, respectively. At this time, each analysis loss is defined as in [Equation 13] and [Equation 14] below.

[Equation 13]

[Equation 14]

,

,

및

는 각각 매칭 모양 설명(matching shape description), 미스매칭 모양 설명(mismatching shape description), 매칭 경계 설명(matching margin description) 및 미스매칭 경계 설명(mismatching margin description)을 나타낸다. here,

,

,

And

Denotes a matching shape description, a mismatching shape description, a matching margin description, and a mismatching margin description, respectively.

및

)은 분포(

)에서 무작위로 샘플링될 수 있다. 일 예로, ‘둥근 모양(round shape)’을 가지는 샘플이라면, 타원(oval) 모양, 소엽(lobulated) 모양 또는 불규칙(irregular) 모양으로부터 미스매칭 모양 설명(

)이 샘플링될 수 있다. 다른 예로, ‘경계형(circumscribed) 경계’를 가지는 샘플이라면, 모호한(obscured) 경계, 불분명(ill-defined) 경계, 미세소엽(microlobutated) 경계 또는 침상(spiculated) 경계로부터 미스매칭 경계 설명(

)이 샘플링될 수 있다. At this time, the mismatch description (

And

) Is the distribution (

) Can be randomly sampled. For example, in the case of a sample having a'round shape', a description of a mismatched shape from an oval shape, a lobulated shape, or an irregular shape (

) Can be sampled. As another example, if a sample has a'circumscribed boundary', the mismatch boundary description from an obscured boundary, an ill-defined boundary, a microlobutated boundary, or a spiculated boundary (

) Can be sampled.

)을 최소화함으로써, 모양 설명과 관련된 모양 특성을 인코딩하도록 학습된다. 동일한 방법으로, 딥 네트워크는 경계 특성에 대한 해석 손실(

)을 최소화함으로써, 경계 설명과 관련된 경계 특성을 인코딩하도록 학습된다. Deep network is the analysis loss (

) By minimizing it, it is learned to encode shape features related to the shape description. In the same way, the deep network has an analysis loss (

) By minimizing it, it is learned to encode the boundary characteristics associated with the boundary description.

)가 추가되어 경계 특성이 성기게 되도록(be sparse)할 수 있다. 나아가,

는 희소성 페널티의 중요도를 결정하는 하이퍼 파라미터(hyper-parameter)를 나타낸다. In the case of boundary properties, a sparsity penalty

) Can be added to make the boundary properties be sparse. Furthermore,

Denotes a hyper-parameter that determines the importance of the scarcity penalty.

As a result, the overall analysis network is learned by minimizing the loss function as shown in [Equation 15] below.

[Equation 15]

는 진단 손실과 해석 손실 간의 균형을 유지하기 위한 균형 하이퍼 파라미터(balancing hyper-parameter)를 나타낸다. here,

Represents a balancing hyper-parameter for maintaining a balance between diagnostic loss and interpretation loss.

As described above, the analysis network, which is a deep network according to an embodiment of the present invention, is characterized in that it is learned to optimize the relationship between the boundary description and the boundary characteristics, increase the correlation between the shape description and the shape characteristics, and minimize diagnostic errors. .

6 is a block diagram illustrating a detailed configuration of a visual evidence interpretation system according to an embodiment of the present invention, and FIG. 7 is a block diagram illustrating a detailed configuration of a measurement unit according to an embodiment of the present invention.

Referring to FIG. 6, the system according to an embodiment of the present invention uses a deep network to analyze visual evidence for the shape and margin of a breast mass.

To this end, the visual evidence analysis system 600 according to an embodiment of the present invention includes an encoding unit 610, a visual analysis unit 620, and a measurement unit 630. In addition, the visual evidence interpretation system 600 according to another embodiment of the present invention further includes a verification unit 640.

The encoding unit 610 encodes the target mass characteristic.

The encoding unit 610 may encode a mass characteristic of an input image, for example, a region-of-interest (ROI) image for a target mass suspected as a breast mass.

For example, the encoding unit 610 is a region of interest showing an important region, that is, a region of interest, according to guided information on a spatial characteristic map of a breast imaging-reporting and data system (BIRADS). You can encode the mass characteristics of the image.

The visual analysis unit 620 generates a mass image of shape characteristics and boundary characteristics corresponding to the encoded target mass characteristics.

The visual analysis unit 620 uses a visual interpretation network that inputs the target mass characteristics encoded by the encoding unit 610 as an input to generate mass images of shape characteristics and boundary characteristics corresponding to the target mass characteristics. Can be created separately. Furthermore, the visual analysis unit 620 may provide improved visual characteristics by emphasizing the spatial characteristics of the shape characteristics and boundary characteristics of the target mass characteristics.

The measurement unit 630 measures visual evidence for the target mass using a deep network that receives mass images and mass characteristic samples as inputs.

Referring to FIG. 7, the measurement unit 630 calculates a shape relevance score for the target mass using guide networks that input a mass image of shape characteristics and a shape characteristic sample as inputs. A margin relevance score for the target mass using a shape measuring unit 631 that measures visual evidence, and boundary guide networks that input a mass image of boundary characteristics and a boundary characteristic sample as inputs. It may include a boundary measuring unit 632 that calculates) and measures visual evidence.

At this time, the induction network may output a relevance score by calculating an encoded mass image that encodes the mass image and embedding description information for shape characteristics or boundary characteristics as inputs. The higher is, the higher the score can be calculated.

As an example, the shape measuring unit 631 may provide the clinical decision to be interpreted as visual evidence by using a shape inducing network that receives a mass image of a shape characteristic and a sample of an irregular shape (Irregular) as inputs. More specifically, the shape measuring unit 631 encodes the mass image of the shape characteristic through a visual characteristic encoder, outputs the embedding characteristic based on the description embedding for the description of the shape characteristic sample, and uses the previously learned shape induction network. Based on the refined image characteristics and embedding characteristics, a related score can be calculated. In this case, the refined image characteristic may be an image characteristic of a refined image obtained by encoding an input image refined by an encoding unit according to an embodiment of the present invention through a visual feature encoder.

As another example, the boundary measurement unit 632 may provide the clinical decision to be interpreted as visual evidence by using a boundary guidance network that receives a boundary characteristic mass image and a spiculate boundary characteristic sample as inputs. More specifically, the boundary measurement unit 632 encodes the mass image of the boundary characteristic through a visual characteristic encoder, outputs the embedding characteristic based on the description embedding for the description of the boundary characteristic sample, and uses the boundary induction network learned in advance. Based on the refined image characteristics and embedding characteristics, a related score can be calculated. In this case, the refined image property may be an image property of a refined image obtained by encoding an input image refined by an encoding unit according to an embodiment of the present invention through a visual feature encoder.

In addition, by calculating the relevant score using the guidance network (shape guidance network and the boundary guidance network) based on the mass image and the lesion characteristic sample, and updating the visual evidence measurement process and the spatial characteristic improvement process, the deep network of the present invention By repeating the training process, you can learn a deep network through it.

The analysis network, which is a deep network that provides a detailed level of visual interpretation for clinical decisions according to the present invention, is the shape of the mass and the boundary of the mass (also referred to as mass lexicon) in order to increase the image interpretation ability. According to the visual evidence. At this time, the mass vocabulary is mainly used to record the characteristics of the breast mass.

The verification unit 640 of the system 600 according to another embodiment of the present invention verifies the clinical decision using the encoded target mass characteristic and the merged mass image.

The verification unit 640 uses a merged mass image obtained by merging a mass image of a shape feature and a mass image of a boundary feature, and a diagnostic network that calculates and inputs the encoded target mass feature. Malignant or benign clinical decisions about the mass can be verified.

The verification unit 640 may acquire shape characteristics and boundary characteristics for the target mass based on the shape of the mass and the boundary of the mass learned in advance in the process of verifying the clinical decision using the diagnostic network. In other words, the verification unit 640 may output three types of a clinical decision, a margin characteristic, and a shape characteristic through the diagnosis network.

In the system according to an embodiment of the present invention, the deep network is a target encoded by encoding a target mass characteristic representing a region-of-interest (ROI) according to guided information on a spatial characteristic map. It is learned through the process of acquiring mass characteristics, generating mass images of shape characteristics and boundary characteristics corresponding to target mass characteristics, and improving spatial characteristics due to visual evidence measured by the generated mass image and mass characteristic samples. I can.

Even though the description thereof is omitted in the systems of FIGS. 6 and 7, each of the components constituting FIGS. 6 and 7 may include all the contents described in FIGS. 1A to 5, which are engaged in the technical field. It is obvious to a person skilled in the art.

The apparatus described above may be implemented as a hardware component, a software component, and/or a combination of a hardware component and a software component. For example, the devices and components described in the embodiments include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable array (FPA), It can be implemented using one or more general purpose computers or special purpose computers, such as a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. The processing device may execute an operating system (OS) and one or more software applications executed on the operating system. In addition, the processing device may access, store, manipulate, process, and generate data in response to the execution of software. For the convenience of understanding, although it is sometimes described that one processing device is used, one of ordinary skill in the art, the processing device is a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that it may include. For example, the processing device may include a plurality of processors or one processor and one controller. In addition, other processing configurations are possible, such as a parallel processor.

The software may include a computer program, code, instructions, or a combination of one or more of these, configuring the processing unit to behave as desired or processed independently or collectively. You can command the device. Software and/or data may be interpreted by a processing device or to provide instructions or data to a processing device, of any type of machine, component, physical device, virtual equipment, computer storage medium or device. , Or may be permanently or temporarily embodyed in a transmitted signal wave. The software may be distributed over networked computer systems and stored or executed in a distributed manner. Software and data may be stored on one or more computer-readable recording media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the embodiment, or may be known and usable to those skilled in computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. -A hardware device specially configured to store and execute program instructions such as magneto-optical media, and ROM, RAM, flash memory, and the like. Examples of the program instructions include not only machine language codes such as those produced by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like. The hardware device described above may be configured to operate as one or more software modules to perform the operation of the embodiment, and vice versa.

Although the embodiments have been described by the limited embodiments and drawings as described above, various modifications and variations can be made from the above description to those of ordinary skill in the art. For example, the described techniques are performed in a different order from the described method, and/or components such as a system, structure, device, circuit, etc. described are combined or combined in a form different from the described method, or other components Alternatively, even if substituted or substituted by an equivalent, an appropriate result can be achieved.

Therefore, other implementations, other embodiments, and claims and equivalents fall within the scope of the claims to be described later.

Claims (20)

In the visual evidence interpretation method according to mass properties performed by the visual evidence interpretation system according to mass properties,
Encoding a target mass characteristic by an encoding unit in the visual evidence analysis system according to the mass characteristic;
Generating, by a visual analysis unit, a mass image of shape characteristics and boundary characteristics corresponding to the encoded target mass characteristics; And
Including, by a measuring unit, measuring visual evidence for the target mass using a deep network that receives the mass image and mass characteristic sample as inputs,
The deep network is
The encoded target mass characteristic is obtained by encoding a target mass characteristic representing a region-of-interest (ROI) according to guided information on a spatial characteristic map, and the target mass characteristic corresponding to the target mass characteristic is obtained. It is characterized in that learning through the process of generating the mass image of shape characteristics and the boundary characteristics, and improving spatial characteristics due to visual evidence measured by the generated mass image and the mass characteristic sample, according to mass characteristics. How to interpret visual evidence. The method of claim 1,
The encoding step
A method of interpreting visual evidence according to mass properties for encoding mass characteristics of a region-of-interest (ROI) image for the target mass. The method of claim 1,
The generating step
Using a visual interpretation network that takes the encoded target mass characteristic as an input, the shape characteristic corresponding to the target mass characteristic and the mass image of each of the boundary characteristics are generated. Visual evidence interpretation method according to. The method of claim 3,
The measuring step
Measuring visual evidence by calculating a shape relevance score for the target mass using shape guide networks that input the mass image of the shape feature and the shape feature sample as inputs. step; And
Visual evidence is measured by calculating a margin relevance score for the target mass using margin guide networks that input the mass image of the boundary characteristic and a boundary characteristic sample as inputs. step
Visual evidence interpretation method according to the mass characteristics, including. The method of claim 4,
The induction network is
The mass characteristic, characterized in that configured to output a relevance score by calculating an encoded mass image that encodes the mass image and embedding description information for the shape feature or the boundary feature as an input. Visual evidence interpretation method according to. The method of claim 1,
Verifying a clinical decision by using the mass image merged with the encoded target mass characteristic
Visual evidence interpretation method according to the mass characteristics further comprising a. The method of claim 6,
The verifying step
Malignant or benign diagnosis for the target mass using a diagnostic network that calculates the merged mass image and the encoded target mass feature as input by combining the mass image of the shape feature and the mass image of the boundary feature. A method of interpreting visual evidence according to mass characteristics, characterized in that the clinical decision is verified. The method of claim 7,
The verifying step
In the process of verifying the clinical decision using the diagnostic network, the shape characteristic and the boundary characteristic for the target mass are acquired based on the shape of the mass and the boundary of the mass learned in advance. Visual evidence interpretation method according to. delete In the visual evidence interpretation method according to mass properties performed by the visual evidence interpretation system according to mass properties,
Encoding a target mass characteristic by an encoding unit in the visual evidence analysis system according to the mass characteristic;
Generating, by a visual analysis unit, a mass image of shape characteristics and boundary characteristics corresponding to the encoded target mass characteristics;
Measuring, by a measurement unit, visual evidence for the target mass using a deep network that receives the mass image and mass characteristic sample as inputs; And
Including, by a verification unit, verifying a clinical decision using the mass image merged with the encoded target mass characteristics,
The deep network is
The encoded target mass characteristic is obtained by encoding a target mass characteristic representing a region-of-interest (ROI) according to guided information on a spatial characteristic map, and the target mass characteristic corresponding to the target mass characteristic is obtained. It is characterized in that learning through the process of generating the mass image of shape characteristics and the boundary characteristics, and improving spatial characteristics due to visual evidence measured by the generated mass image and the mass characteristic sample, according to mass characteristics. How to interpret visual evidence. An encoding unit encoding the target mass characteristic;
A visual analysis unit for generating a mass image of shape characteristics and boundary characteristics corresponding to the encoded target mass characteristics; And
Including a measuring unit for measuring visual evidence (visual evidence) for the target mass using a deep network that receives the mass image and mass characteristic sample as inputs,
The deep network is
The encoded target mass characteristic is obtained by encoding a target mass characteristic representing a region-of-interest (ROI) according to guided information on a spatial characteristic map, and the target mass characteristic corresponding to the target mass characteristic is obtained. It is characterized in that learning through the process of generating the mass image of shape characteristics and the boundary characteristics, and improving spatial characteristics due to visual evidence measured by the generated mass image and the mass characteristic sample, according to mass characteristics. Visual evidence interpretation system. The method of claim 11,
The encoding unit
A system for analyzing visual evidence according to mass properties for encoding mass characteristics of a region-of-interest (ROI) image for the target mass. The method of claim 11,
The visual interpretation unit
A visual evidence analysis system according to mass properties, characterized in that generating mass images of each of the shape characteristics and the boundary characteristics corresponding to the target mass characteristics by inputting the encoded target mass characteristics as inputs. The method of claim 13,
The measurement unit
A shape for measuring visual evidence by calculating a shape relevance score for the target mass using guide networks that input the mass image of the shape characteristic and a shape characteristic sample as inputs. Measurement unit; And
Boundary for measuring visual evidence by calculating a margin relevance score for the target mass using guide networks that input the mass image of the boundary characteristic and a boundary characteristic sample as inputs. Measuring part
Visual evidence interpretation system according to the mass characteristics comprising a. The method of claim 14,
The induction network is
The mass characteristic, characterized in that configured to output a relevance score by calculating an encoded mass image that encodes the mass image and embedding description information for the shape feature or the boundary feature as an input. Visual evidence interpretation system according to. The method of claim 11,
Verification unit for verifying clinical decisions using the encoded mass characteristics and the merged mass image
Visual evidence interpretation system according to the mass characteristics further comprising a. The method of claim 16,
The verification unit
Malignant or benign diagnosis for the target mass using a diagnostic network that calculates the merged mass image and the encoded target mass feature as input by combining the mass image of the shape feature and the mass image of the boundary feature. Visual evidence interpretation system according to mass characteristics, characterized in that to verify the clinical decision (diagnostic decision). The method of claim 17,
The verification unit
In the process of verifying the clinical decision using the diagnostic network, the shape characteristic and the boundary characteristic for the target mass are acquired based on the shape of the mass and the boundary of the mass learned in advance. Visual evidence interpretation system according to. delete An encoding unit encoding the target mass characteristic;
A visual analysis unit for generating a mass image of shape characteristics and boundary characteristics corresponding to the encoded target mass characteristics;
A measuring unit for measuring visual evidence for a target mass using a deep network that receives the mass image and mass characteristic sample as inputs; And
Including a verification unit for verifying a clinical decision using the mass image merged with the encoded target mass characteristics,
The deep network is
The encoded target mass characteristic is obtained by encoding a target mass characteristic representing a region-of-interest (ROI) according to guided information on a spatial characteristic map, and the target mass characteristic corresponding to the target mass characteristic is obtained. It is characterized in that learning through the process of generating the mass image of shape characteristics and the boundary characteristics, and improving spatial characteristics due to visual evidence measured by the generated mass image and the mass characteristic sample, according to mass characteristics. Visual evidence interpretation system.

Images