KR102067340B1 - Method for generating breast masses according to the lesion characteristics and the system thereof - Google Patents

Abstract

Disclosed are a method for generating a breast mass according to the characteristics of breast cancer lesion and a system thereof. The method for generating a breast mass according to the characteristics of breast cancer lesion according to an embodiment of the present invention comprises the steps of: encoding target mass characteristics; and generating a mass image corresponding to the target mass characteristics by using a pre-trained deep network that receives encoded target mass description and a seed mass image as inputs, wherein the mass image corresponding to the target mass characteristics can be generated by using a generative adversarial network (GAN) mapped from the seed mass image and the encoded target mass characteristics.

Description

Methods for generating breast masses according to the lesion characteristics and the system approximately

The present invention relates to a technique for generating breast masses according to lesion characteristics defined in the Breast Imaging-Reporting and Data System (BIRADS), and more particularly, to the BIASDS (Breast Imaging-Reporting and Data System). A method and system for generating breast masses that can embed features to produce breast masses with desired characteristics.

For women, breast tumors are the second type of tumor after skin tumors, but the first type with respect to deaths.

In particular, the anxiety is consistently lower in the average age at which tumors develop, and current screening tools are used to diagnose high density breasts, which are characteristic of pre-menopausal women and generally healthy women. It is insufficient. In fact, mammography is not reliable for high density breasts. It can be inefficient if it is not "pre-directed" to the area of attention to be examined. Current systems capable of providing functional information or diagnosing dense breasts, such as MRI, CT and PET, are very expensive and therefore not suitable for examination or mass diagnostics, are invasive and repeat in a short period of time. Can't be.

In the general context, today's diagnostic pathways always involve the use of ultrasound downstream of each primary diagnosis, which is not clear whether it is negative or not. This is because today's primary diagnostic methods do not have sufficient sensitivity and specificity to find the exact location to diagnose malignant or benign lesions.

Optical technology used today is in the analysis of changes in deoxyhemoglobin concentrations in breast tissue. In particular, it is known to analyze the change in the concentration of deoxyhemoglobin in capillaries by detecting the light attenuation values of the video camera that is opposite the light source by shining 640 nm red light on the breast. This makes it possible to detect the presence of "neoangiogenesis" regions, i.e. abnormal angiogenic regions that are created to feed tumor cells, regardless of the density of the tissue to be analyzed. This technology overcomes the significant limitations of the present. Indeed, the preference consists of the ability to detect the presence of "neovascularization" regions, i.e. abnormal angiogenesis regions produced to grow tumor cells, regardless of the density of the tissue to be analyzed.

Lesion generation is an important field for overcoming the limitations of lack of medical data in learning machine learning and deep learning. Lesion production can also be used to train radiologists, residents, and interns to diagnose lesions.

One embodiment technique for generating existing lesions is to create random lesions randomly, and since the existing techniques do not make them have desired characteristics, there are limitations in utilizing the actually generated lesions in machine learning or hospital. .

delete

Publication No. 10-2016-0117807 (published: October 11, 2016)

Embodiments of the present invention provide a method and system for generating breast masses that can embed breast characteristics with desired characteristics by embedding BIRADS characteristics.

In particular, embodiments of the present invention provide a breast mass generation method and system capable of generating breast masses having BIRADS characteristics based on BIRADS characteristics, for example, a description and a pre-learned deep network.

A method of generating a mass according to an embodiment of the present invention includes encoding a target mass characteristic; And generating a mass image corresponding to the target mass characteristic using a pre-learned deep network that receives the encoded target mass characteristic and the seed mass image as inputs.

The generating may generate a mass image corresponding to the target mass characteristic by using a generative adversarial network (GAN) mapped from the seed mass image and the encoded target mass characteristic.

The generating may generate a mass image corresponding to the target mass characteristic based on an encoder-decoder network having a residual block.

The generating may include encoding the seed mass image to extract features of the seed mass image, and concatenating and decoding the features of the extracted seed mass image and the encoded target mass characteristic to correspond to the target mass characteristic. To create a mass image.

The deep network encodes a Breast Imaging-Reporting and Data System (BIRADS) characteristic to obtain BIRADS embedded with the characteristic, and inputs the BIRADS embedded with the characteristic and the actual mass image as the BIRADS. A mass image corresponding to a characteristic may be generated, and the generated mass image and the characteristic may be learned through a process of distinguishing the actual mass and the fake mass using the embedded BIRADS.

According to another embodiment of the present invention, there is provided a method for generating a mass; And generating a mass image corresponding to the mass characteristic by using a pre-learned deep network that takes the embedded mass characteristic and the seed mass image as inputs.

The generating may generate a mass image corresponding to the mass characteristics by using a generative adversarial network (GAN).

The generating may include: mapping a mass having the embedded mass characteristic to a latent space formed by the seed mass image and the embedded mass characteristic, and performing a visual feature processing process in the latent space to correspond to the mass characteristic. Mass images can be created.

The generating may generate a mass image corresponding to the mass property by using at least one of visual feature linear interpolation and visual feature linear interpolation in the latent space.

The deep network embeds margin and shape characteristics of the Breast Imaging-Reporting and Data System (BIRADS) to obtain embedded margin characteristics and embedded shape characteristics, and embeds the embedded margin characteristics and the embedding characteristics. Map the embedded margin characteristic and the mass having the embedded shape characteristic to the latent space formed by the actual shape image and the actual mass image, and perform a visual feature processing process on the latent space and the shape characteristic. It can be learned by generating a corresponding mass image and distinguishing the actual mass from the fake mass using the generated mass image and the actual mass image.

Mass generation system according to an embodiment of the present invention includes an encoding unit for encoding the target mass characteristics; And a generation unit configured to generate a mass image corresponding to the target mass characteristic using a pre-learned deep network that receives the encoded target mass characteristic and the seed mass image as inputs.

A mass generating system according to another embodiment of the present invention includes an embedding part for embedding mass characteristics; And a generation unit configured to generate a mass image corresponding to the mass characteristics using a pre-learned deep network that receives the embedded mass characteristics and seed mass images.

According to embodiments of the present invention, breast masses having BIRADS characteristics may be generated based on BIRADS characteristics, for example, a description and a pre-learned deep network.

According to embodiments of the present invention, the actual breast mass may be generated using a pre-trained deep network and visual feature processing in latent space.

According to embodiments of the present invention, by creating a lesion according to the BIRADS characteristics used to characterize the lesion, it is possible to create a variety of lesions having the desired shape, characteristics, and to actually use the generated lesions in a real environment easily Can be.

1 is a flowchart illustrating an operation of a method for generating a breast mass according to an embodiment of the present invention.
2 is a flowchart illustrating an operation of a method for generating breast mass according to another embodiment of the present invention.
FIG. 3 illustrates an exemplary diagram for describing a process of processing a visual characteristic in a latent space using the method of FIG. 2.
4 illustrates an exemplary diagram for explaining a concept of interpolation and extrapolation between two visual features in a latent space.
5 shows a conceptual configuration of a breast mass generation system according to an embodiment of the present invention.
6 shows a conceptual configuration of a breast mass generation system according to another embodiment of the present invention.

Advantages and features of the present invention and methods for achieving them will be apparent with reference to the embodiments described below in detail with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but will be implemented in various different forms, only the embodiments are to make the disclosure of the present invention complete, and common knowledge in the art to which the present invention belongs It is provided to fully inform the person having the scope of the invention, which is defined only by the scope of the claims.

The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. In this specification, the singular also includes the plural unless specifically stated otherwise in the phrase. As used herein, “comprises” and / or “comprising” refers to a component, step, operation and / or element that is one or more of the other components, steps, operations and / or elements. It does not exclude existence or addition.

Unless otherwise defined, all terms (including technical and scientific terms) used in the present specification may be used in a sense that can be commonly understood by those skilled in the art. In addition, terms that are defined beforehand that are generally used are not to be interpreted ideally or excessively unless they are clearly specifically defined.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings. The same reference numerals are used for the same components in the drawings, and duplicate descriptions of the same components are omitted.

Embodiments of the present invention are directed to generating breast masses having BIRADS characteristics based on BIRADS characteristics, for example, BIRADS description and a pre-learned deep network.

The technique according to the present invention may generate a real breast mass according to BIRADS using a pre-learned deep network, or may generate a real breast mass using visual feature processing in a latent space. . That is, the present invention is to generate lesions according to the BIRADS characteristics used to characterize the lesions, and if the lesions are generated according to BIRADS, various lesions having desired shapes and characteristics can be made, and the generated lesions are actually Easy to use in the environment.

This invention will be described with reference to FIGS. 1 to 6 as follows.

Breast Image Reporting Data System ( BIRADS ; Breast Imaging-Reporting and Data System In accordance Practical  Breast Mass Generation

1 is a flowchart illustrating a method for generating a breast mass according to an embodiment of the present invention. The entire framework for synthesizing lesions according to BIRADS characteristics, for example, BIRADS description, is illustrated in FIG. It is shown.

As shown in FIG. 1, the method according to an embodiment of the present invention is based on conditional adversarial networks (GANs) mapped from seed images (actual masses) and target BIRADS descriptions. The framework of the present invention is based on a GAN comprising an embedding part, a visual interpreter and a discriminator, the embedding part may include a BIRADS description embedding network, the visual interpreter may include a lesion generator network, and the discriminator is a discriminator It may include a network.

The embedding portion encodes the target BIRADS characteristic that is combined with the visual interpreter and the discriminator at the same semantic level. Visual Interpreter For example, the lesion generator network synthesizes images based on image features encoded from embedded BIRADS and seed images. Determination unit For example, the discriminator network performs a distinguishing task on the basis of the embedded BIRADS. That is, the discriminator network uses the embedded BIRADS to determine whether the input image is a real mass (seed image) or a fake mass (generated image). The lesion generator network and discriminator network can be enhanced by competing with each other through hostile learning.

BIRADS  characteristic Embedding  network

)은 각 단어 레벨

로 나누어진다. 여기서

은 BIRADS 설명 문장에서 단어의 수를 의미할 수 있다. 각 단어는 아래 <수학식 1>과 같이 미리 트레이닝된 단어 레벨 임베딩 모델(E)을 통해 단어 벡터로 변환될 수 있다.The BIRADS feature embedding network encodes a target BIRADS feature, for example, a target BIRADS description, to combine the embedded BIRADS and image features at the same semantic level of the lesion generator network and the discriminator network. Visual-Semantic Target BIRADS Description Sentence (

) Is each word level

Divided into. here

May mean the number of words in the BIRADS description sentence. Each word may be converted into a word vector through a pre-trained word level embedding model E as shown in Equation 1 below.

[Equation 1]

In order to extract the embedded BIRADS, a gated cyclic unit (GRU) network-based text encoder may be applied as shown in Equation 2 below.

[Equation 2]

는 GRU 레이어의 구동을 수행하는 함수를 의미하고,

는 n번째 단어 벡터에서 GRU의 히든 상태를 의미할 수 있다.here,

Means a function that performs the driving of the GRU layer,

May mean a hidden state of the GRU in the n-th word vector.

는 두 개의 완전 연결 레이어(fully connected layer)에 의해 처리된다. 그러면, 임베딩된 BIRADS

은 아래 <수학식 3>과 같이 표현될 수 있다.Output of the last stage of the GRU network

Is handled by two fully connected layers. Then embedded BIRADS

May be expressed as Equation 3 below.

[Equation 3]

과

는 완전 연결 레이어의 동작을 수행하는 함수를 의미할 수 있다.here,

and

May mean a function that performs an operation of a fully connected layer.

는 병변 생성기 네트워크(G)와 판별기 네트워크(D)의 이미지 특징과 결합된다.In order to increase the number of BIRADS embeddings, the text embedding augmentation method can be applied to the embedded BIRADS using the mean and standard deviation of the embedded BIRADS, which allows the deep network of the present invention to visually It can help generate various masses from semantic BIRADS properties. Finally, embedded BIRADS

Is combined with the image features of the lesion generator network (G) and the discriminator network (D).

Embedded BIRADS  Having lesion generator network

와 임베딩된 BIRADS

를 입력으로 사용한다. 보다 구체적으로, 인코더는

시드 이미지를

이미지 특징 표현으로 인코딩하는 컨볼루션 뉴럴 네트워크(CNN)일 수 있으며,

는 이미지 특징 표현의 특징 맵 수를 의미할 수 있다.In order to synthesize the mass with respect to the visual semantic BIRADS characteristic, the lesion generator network may be designed based on an encoder-decoder network with residual blocks. The visual interpreter, ie the lesion generator network, inputs the seed image

With embedded BIRADS

Is used as input. More specifically, the encoder

Seed image

May be a convolutional neural network (CNN) that encodes to an image feature representation,

May mean the number of feature maps in the image feature representation.

는 채널 차원을 따라 인코딩된 이미지 특징들과 연쇄되어

를 형성하기 위해 공간적으로 복제될 수 있다. 여기서,

는 임베딩된 BIRADS 벡터의 차원을 의미할 수 있다. 임베딩된 BIRADS와 결합된 인코딩된 이미지 특징들은 몇 개의 잔차 블록으로 피드될 수 있다. 이 잔차 블록은 스트라이드 1(stride 1)을 갖는 3 × 3 크기의 컨볼루션으로 구성될 수 있다. Embedded BIRADS

Is concatenated with image features encoded along the channel dimension

Can be replicated spatially to form here,

May refer to the dimension of the embedded BIRADS vector. Encoded image features combined with embedded BIRADS can be fed into several residual blocks. This residual block may consist of 3 × 3 convolutions with stride 1.

로 변환할 수 있다. 여기서, 병변 생성기 네트워크에 의해 생성된 이미지는 아래 <수학식 4>와 같이 계산될 수 있다.The decoder can consist of several upsampling layers, and the decoder can generate a potential feature representation of the generated image.

Can be converted to Here, the image generated by the lesion generator network may be calculated as in Equation 4 below.

[Equation 4]

는 병변 생성기 네트워크를 의미할 수 있다.here,

May refer to a lesion generator network.

With the exception of the first and last layers of the lesion generator network, rectified linear unit (ReLU) activation and batch normalization can be applied behind all convolutional layers of the encoder and decoder network. The residual block can help the generator network to learn the identity function. Here, the identity function allows the generated image to maintain a similar breast background of the source image. As a result, actual masses are created with the target BIRADS category in the background of the actual breast tissue.

Embedded BIRADS  Branch Discriminator  network

이미지는 차원

의 특징 표현으로 인코딩된다. 인코딩된 특징은 공간적으로 복제된 임베딩된 BIRADS

와 연쇄된다. 연쇄된(concatenated) 특징들은 두 개의 컨볼루션 레이어들에 의해 처리되고, 두 개의 컨볼루션 레이어들은 실제 종괴 또는 가짜 종괴의 최종 확률을 계산한다. 리키(leaky) ReLU 활성화는 출력 레이어를 제외한 모든 레이어에 대해 수행되고, 배치 정규화는 첫 번째 레이어와 마지막 레이어를 제외한 모든 레이어에 적용될 수 있다.The discriminator, that is, the discriminator network, distinguishes between the actual mass and the fake mass using the BIRADS embedded in the generated image and the actual image. Like the encoder in the lesion generator network,

Image is dimension

Is encoded into a feature representation of the. Encoded features are spatially replicated embedded BIRADS

Chained with. Concatenated features are processed by two convolutional layers, and the two convolutional layers calculate the final probability of a real or fake mass. Leaky ReLU activation is performed for all layers except the output layer, and batch normalization can be applied to all layers except the first and last layers.

Learning Strategies of the Breast Mass Development Network

The loss function for learning the lesion generator network G may be defined as in Equation 5 below.

[Equation 5]

의 분포에 적합해지도록 유도한다. 판별기 네트워크를 학습하기 위한 손실 함수를 설명하기 위해 몇 가지 표기법을 정의한다. t와

는 시드 이미지의 오리지널 BIRADS 설명과 잘못 정렬된 BIRADS 설명을 각각 의미한다. 잘못 정렬된 BIRADS 설명은 부정확한 순서를 가지는 설명으로 정의된다. 예를 들어, "spiculated this mass has margin"은 오리지널 BIRADS 설명 "이 종괴는 침상 마진(speculated margin)을 가진다"와 불일치하는(mismatching) BIRADS 설명 중 하나이다. 판별기 네트워크(D)를 학습하기 위한 손실 함수는 아래 <수학식 6>과 같이 정의될 수 있다.Loss function is that the generated image is the actual data

To fit the distribution of. Several notations are defined to describe the loss function for learning the discriminator network. t and

Means the original BIRADS description and the misaligned BIRADS description of the seed image, respectively. Misaligned BIRADS descriptions are defined as descriptions with an incorrect order. For example, "spiculated this mass has margin" is one of the BIRADS descriptions that is inconsistent with the original BIRADS description "This mass has a speculated margin". The loss function for learning the discriminator network D may be defined as in Equation 6 below.

[Equation 6]

The entire network can be trained to alternately minimize the equations (5) and (6). The lesion generator network G and the discriminator network D may be improved during alternate training.

As such, the method according to an embodiment of the present invention may generate a breast mass corresponding to the BIRADS description using a pre-learned deep network and the BIRADS description.

Breast mass generation by visual feature processing in latent space

The method according to another embodiment of the present invention relates to a method for generating breast mass according to the medical characteristics of the BIRADS (Breast Imaging Reporting and Data System). BIRADS is designed to standardize reports to characterize masses in mammograms.

2 is a flowchart illustrating a method for generating a breast mass according to another embodiment of the present invention, which is based on Generative Adversarial Networks (GAN), and includes a generation unit, a determination unit, and a margin embedding unit. And a shape embedding portion. Here, the generation unit includes the breast mass generator module (G), the determination unit includes the discriminator modules (D ₁ and D ₂ ), and each of the margin embedding unit and the shape embedding unit is a BIRADS description embedding module (E ₁ and E _2). ).

The goal of the learning phase is to allow G to learn the various breast masses to be used in the production phase. The following describes the learning and production phase procedures in the method of the present invention.

)과 형상(

) 라벨이 네트워크에 입력된다. 마진과 형상 라벨은 생성기 모듈 G에 입력되기 전에 BIRADS 설명 임베딩 모듈을 통해 E₁(

)와 E₂(

)에 임베딩된다. 그런 다음, G는 입력된 유방 종괴(I) 즉, 시드 유방 종괴와 임베딩된 라벨들을 이용하여 가짜 유방 종괴(I')를 생성한다. 여기서, 가짜 유방 종괴(I')는 아래 <수학식 7>과 같이 생성될 수 있다.As shown in FIG. 2, the margin of breast lump image and the main information in BIRADS in the learning stage (

) And geometry (

) The label is entered into the network. Through BIRADS described embedding module before it is input to the shape margin label is a generator module G E ₁ (

) And E ₂ (

Is embedded in). G then creates a fake breast mass I ' using the entered breast mass I , the seed breast mass and the embedded labels. Here, the fake breast mass I ' may be generated as shown in Equation 7 below.

[Equation 7]

)와 E₂(

)는 BIRADS 특성 예를 들어, BIRADS 설명의 임베딩된 마진(

)과 형상(

) 라벨을 의미할 수 있다.Where E ₁ (

) And E ₂ (

) Is the BIRADS attribute, for example, the embedded margin (

) And geometry (

) May mean a label.

D ₁ outputs predictions for real / fake and margin labels with inputs of real and fake masses. D ₂ inputs the actual mass and the fake mass as inputs and outputs the prediction results for the fake / actual and shape labels. Repeating the above process, G, D ₁ and D ₂ can be learned in a hostile and collaborative manner so that G can learn various breast masses.

As shown in FIG. 3, the G and BIRADS characteristic embedding modules trained to approximate breast manifold in the production phase are used. G includes an encoder and a decoder, each of which consists of a convolutional layer and a transposed convolutional layer. In front of G, seed breast masses and corresponding margins, shape labels are embedded via BIRADS embedding modules similar to the training phase. When a breast mass with embedded labels enters the input of G, the encoder maps the breast mass with embedded labels to the seed mass and the latent space formed by the corresponding BIRADS characteristic. By performing the visual feature processing in the latent space, the processed visual features can be decoded into pixel space to produce the actual breast mass.

Visual feature processing

)은 유방 종괴와 임베딩된 라벨들을 인코더를 통해 잠재 공간으로 매핑함으로써 인코딩된다. 처리되지 않은 시각적 특징이 디코더에 입력되면, 시각적 특징을 만드는데 사용된 유방 종괴와 유사한 유방 종괴가 생성된다. 이를 통해 생성된 유방 종괴를 사용하면, 비선형 다양성의 증가로 인한 회전(rotation), 플립(flip) 등과 같은 아핀 변환(affine transform)을 통해 증강된 데이터와 비교할 때 컴퓨터 보조 진단(CAD; computer-aided diagnosis)을 학습하는데 약간 도움이 될 수도 있다. 본 발명은 유방 종괴의 시각적 특징들이 존재하는 다양성에서 가능한 많은 시각적 특징을 고려하기 위하여 시각적 특징 선형 보간법과 보외법을 사용할 수 있다.As mentioned above, visual features (

) Is encoded by mapping breast masses and embedded labels to latent space through an encoder. When an unprocessed visual feature is input to the decoder, a breast mass similar to the breast mass used to create the visual feature is created. The resulting breast mass allows computer-aided diagnostics (CAD) when compared to data augmented through affine transforms, such as rotation, flip, etc., due to an increase in nonlinear diversity. It may help a little to learn the diagnosis. The present invention may use visual feature linear interpolation and extrapolation to account for as many visual features as possible in the variety in which the visual features of the breast mass are present.

(1) visual feature linear interpolation

Visual Features The easiest case of the linear interpolation method is linear interpolation between two visual features in latent space. Linear interpolation between two different visual features makes it possible to take into account possible visual features in a one-dimensional line between the two visual features. Here, linear interpolation between two different visual features can be expressed as Equation 8 below.

[Equation 8]

과

는 선형 보간에서 시각적 특징들에 곱해지는 가중치들을 의미할 수 있다.here,

and

May mean weights that are multiplied by the visual features in linear interpolation.

Linear interpolation between these two different visual features can be extended by interpolation between the N visual features of the breast mass in the entire training data set, as shown in Equation 9 below.

[Equation 9]

는 선형 보간에서 시각적 특징

에 의해 곱해지는 가중치를 의미할 수 있다.here,

Is a visual feature in linear interpolation

It may mean a weight multiplied by.

가 0인 경우 이러한 N개의 시각적 특징들은 (N-1) 차원의 초평면을 형성한다. 즉, 상기 수학식 9는 아래 <수학식 10>과 같이 나타낼 수 있다.If N visual features are linearly independent, that is, the only condition that makes the linear combination of visual features zero is

If 0, these N visual features form a hyperplane in the (N-1) dimension. That is, Equation 9 may be expressed as Equation 10 below.

[Equation 10]

In other words, if N is smaller than the dimension of the visual feature itself, then all possible visual features present in the hyperplane of the (N-1) dimension can be considered if the N visual features are linearly independent of each other.

When visual feature interpolation is applied to a visual feature encoded from a breast mass having a particular BIRADS description label, the breast mass may be created to have the features of that BIRADS description. Thus, the method according to another embodiment of the present invention does not require additional annotation of the resulting mass.

(2) visual feature linear extrapolation

The concept of linear interpolation is used to take into account the possible visual features that exist between certain visual features in latent space. The present invention contemplates possible visual features that exist outside of the visual features of the training data set in consideration of visual feature linear extrapolation.

Interpolation and extrapolation may consider multiple visual features in the (N-1) dimensional hyperplane when there are N visual features. However, for interpolation, the range of consideration is limited to the visual characteristics of breast masses in the training data set. On the other hand, extrapolation provides additional diversity as a way to consider outside the training data set. Here, the extrapolation between the N visual features of the breast mass in the entire training data set may be represented by Equation 11 below.

[Equation 11]

의 범위를 제한함으로써 결정할 수 있다. 도 4는 잠재 공간에서 두 개의 시각적 특징들 간의 보간법 및 보외법에 대한 개념을 설명하기 위한 일 예시도를 나타낸 것으로, 도 4에 도시된 바와 같이, 시드 종괴 이미지의 시각적 특징들을 이용한 선형 보간법을 통해 시드 종괴 이미지의 시각적 특징들 간의 시각적 특징들을 보간하고, 이렇게 보간된 시각적 특징들 간 또는 보간된 시각적 특징과 시드 종괴 이미지의 시각적 특징 간 보간을 통해 두 시각적 특징들 사이의 시각적 특징을 보간할 수 있으며, 시드 종괴 이미지의 시각적 특징들을 이용한 선형 보외법을 통해 시드 종괴 이미지의 시각적 특징들 간의 시각적 특징들을 보외하고, 이렇게 보외된 시각적 특징들 간 또는 보외된 시각적 특징과 시드 종괴 이미지의 시각적 특징 간 보외을 통해 두 시각적 특징들 외부의 시각적 특징을 보외할 수 있다.As can be seen from Equation 11, how far the mass produced by the extrapolation is from the existing visual features

This can be determined by limiting the range of. FIG. 4 illustrates an exemplary diagram for explaining a concept of interpolation and extrapolation between two visual features in latent space. As shown in FIG. 4, through linear interpolation using visual features of a seed mass image. Interpolate the visual features between the visual features of the seed mass image and interpolate between the visual features of the interpolated visual features or between the interpolated visual features and the visual features of the seed mass image. In addition, through the linear extrapolation using the visual features of the seed mass image, the visual features between the visual features of the seed mass image are excluded, and between the extrapolated visual features or between the extra visual features and the visual features of the seed mass image. You can add visual features outside of the two visual features The.

Breast feature generation through visual feature processing

In the learning process, G learns the variety of breast masses that can be used in the production phase. In the production phase, G and BIRADS characteristic embedding modules are used to generate actual breast masses.

The operation of the BIRADS feature embedding modules in the learning and generation phase is as follows. First, the BIRADS feature embedding modules consist of two fully connected layers. In the learning and generation phase, BIRADS feature embedding modules map the one-hot label form of the BIRADS feature label to the size of the breast mass image. The breast mass image and the embedded label are then concatenated (or combined) and entered into the encoder.

Details of each generation step in the generation step are as follows. 1) The encoder accepts a concatenated input of the breast mass image and the embedded label and maps it to the 1024-dimensional latent feature space. The mapped visual feature includes information about the seed breast mass image and the corresponding BIRADS feature label. Therefore, the visual features can be processed to change the shape and characteristics of the mass produced. 2) The aforementioned treatments for visual features are applied to create visual features of breast masses that have appearances and characteristics that do not appear in the seed breast mass image. In other embodiments of the present invention, relatively simple operations such as linear interpolation and extrapolation may be applied. However, there is room for expansion because visual feature processing can include vector operations that can be applied to visual features in addition to linear interpolation and extrapolation. 3) When the processed visual feature is input to the decoder, the decoder maps it to pixels.

The method according to another embodiment of the present invention may generate breast masses having various appearances and features different from seed breast mass images by performing linear interpolation and extrapolation in a high-dimensional latent feature space. By applying an encoder and decoder approach, actual breast masses can be generated, and the resulting masses cannot be distinguished from actual breast masses. The resulting breast mass may have much more nonlinear diversity than the breast mass enhanced through the classic affine transformation.

Learning Strategy of Proposed Structure

The method according to another embodiment of the present invention uses two BIRADS characteristic labels (margin label and shape label) and uses two discriminators D ₁ and D ₂ to predict them. Here, the loss function of D ₁ may be defined as in Equation 12 below.

[Equation 12]

와

는 D₁의 실제/가짜 및 추정 마진 각각에 대한 예측을 의미하고,

는 전체 손실 함수의 균형을 맞추기 위해 각 손실 항에 곱해지는 가중치를 의미할 수 있다.here,

Wow

Is the prediction for each real / fake and estimated margin of D ₁ ,

May denote a weight multiplied by each loss term to balance the overall loss function.

)을 예측할 때

의 손실 항은 감소한다.The beginning of the loss function, wherein two means a general loss of GAN hostile learning to predict a real / fake of breast masses actual image (I) to produce a breast tumor image (I ') by the generator (G). From the input I and I ' with D _{1 the} actual measurement margin label (

When predicting

The loss term of decreases.

The loss function of D ₂ may be defined as in Equation 13 below.

[Equation 13]

는 D₂의 형상에 대한 예측을 의미할 수 있다.here,

May mean prediction of the shape of D ₂ .

)을 예측하면,

의 손실 항은 감소한다.The first two terms of the loss function of D ₂ represent the general GAN loss of aggressive learning that predicts the real / fake of I and I ' . The actual measured shape label from D I and I 'the _second input (

),

The loss term of decreases.

에 입력으로 M을 취하는 좌측 손실 항은 BIRADS 특성 라벨을 정확하게 예측하기 위해 D를 푸쉬한다. M'을 입력으로 취함으로써 BIRADS 특성 라벨을 예측하는 손실 항은 G가 실제적인 유방 종괴를 생성하지 않는 초기 학습 단계에서 노이즈로 작용한다. 그러나 G가 실제 유방 종괴와 유사한 유방 종괴를 생성할 수 있게 된 후에는, 비선형적으로 다른 측면을 가진 생성된 종괴에 대해 BIRADS 특성 라벨을 더 잘 예측할 수 있게 된다. 따라서 데이터 증강 효과가 있다. 이는 G가 D의 예측 정확도를 증가시키는 방향으로 작동하기 때문에 협동 학습 손실 항으로 볼 수 있다.In Equation 12 and Equation 13,

The left loss term taking M as the input to pushes D to accurately predict the BIRADS characteristic label. The loss term, which predicts BIRADS characteristic labels by taking M 'as input, acts as noise in the early learning phase where G does not produce actual breast masses. However, after G has been able to produce breast masses that are similar to actual breast masses, it is possible to better predict BIRADS signature labels for generated masses with non-linearly different aspects. Therefore, there is a data augmentation effect. This can be viewed as a cooperative learning loss term because G acts to increase the prediction accuracy of D.

Next, the loss function of G may be defined as in Equation 14 below.

[Equation 14]

와

는 전체 손실 함수의 균형을 맞추기 위해 각 손실 항에 곱해지는 가중치를 의미하고, 손실 항 L1(I, I')는 실제 유방 종괴 I와 생성된 유방 종괴 I' 사이의 L1 손실을 의미할 수 있다.here,

Wow

Denotes the weight multiplied by each loss term to balance the overall loss function, and the loss term L1 ( I , I ' ) may mean L1 loss between the actual breast mass I and the resulting breast mass I' . .

,

에 따라 더 강한 특성을 갖는 I '를 생성할 수 있을 때,

의 손실 항은 감소한다. D의 협동 학습 손실 항과 마찬가지로, 이 손실 항은 D가 정확하게 구별되지 않기 때문에 초기 학습 단계에서 노이즈로 작용한다. 그러나 D가 BIRADS 설명 라벨을 적절하게 식별할 수 있게 된 후, 이는 G가 I' 생성에서 사용된 BIRADS 특성 라벨의 특성을 더 잘 나타내도록 한다. 마지막 L1 손실 항은 G가 실제 데이터 분포와 유사한 다양성을 형성하도록 한다.The first two anti-loss such as loss of function of the D ₁ and D ₂ means a general loss of GAN hostile learning the real / false prediction of D ₁ and D ₂ to I '. G is the BIRADS attribute

,

When we can produce I ' with stronger properties,

The loss term of decreases. Like the cooperative learning loss term in D, this loss term acts as noise in the initial learning phase because D is not accurately distinguished. However, after D can properly identify the BIRADS description label, it allows G to better characterize the BIRADS property label used in the I ' generation. The final L1 loss term causes G to form a variety similar to the actual data distribution.

The overall learning phase can proceed in the form of alternately minimizing the loss function of D ₁ , D _{2 and} the loss function of G. As learning progresses, G can approximate the diversity of breast masses.

As such, the method according to another embodiment of the present invention may generate the actual breast mass using a pre-trained deep network and visual feature processing in the latent space.

FIG. 5 illustrates a conceptual configuration of a breast mass generation system according to an embodiment of the present invention, and illustrates a conceptual configuration of a system for performing the method of FIG. 1.

Referring to FIG. 5, a system 500 according to an embodiment of the present invention includes an encoder 510, a generator 520, and a determiner 530.

The encoding unit 510 encodes a target mass characteristic, for example, a target mass description.

Here, the encoder 510 may encode a target mass description, for example, a target BIRADS characteristic by applying a gated cyclic unit (GRU) network-based text encoder.

The generation unit 520 generates a mass image corresponding to the target mass characteristic using a pre-learned deep network that receives the encoded target mass characteristic and the seed mass image.

Here, the generation unit 520 may generate a mass image corresponding to the target mass characteristic by using a generative adversarial network (GAN) mapped from the seed mass image and the encoded target mass characteristic.

Here, the generator 520 may generate a mass image corresponding to the target mass characteristic based on the encoder-decoder network having the residual block.

Here, the generation unit 520 encodes the seed mass image to extract the features of the seed mass image, and decodes by concatenating and extracting the features of the extracted seed mass image and the encoded target mass characteristics, thereby corresponding to the target mass characteristics. You can create an image.

The deep network encodes the Breast Imaging-Reporting and Data System (BIRADS) feature to obtain BIRADS with embedded features, and inputs the BIRADS with embedded features and actual mass images to correspond to BIRADS features. A mass image is generated, and the generated mass image and characteristics can be learned through the process of distinguishing the actual mass from the fake mass using the embedded BIRADS.

The determination unit 530 distinguishes between the mass image generated by the generator 520 and the mass image generated by using the BIRADS characteristic encoded by the encoder 510.

Although the description is omitted in the system of FIG. 5, each constituent means constituting FIG. 5 may include all the contents described in FIG. 1, which is obvious to those skilled in the art.

FIG. 6 illustrates a conceptual configuration of a breast mass generation system according to another embodiment of the present invention, and illustrates a conceptual configuration of a system for performing the method of FIGS. 2 to 4.

Referring to FIG. 6, the system 600 according to an embodiment of the present invention includes an embedding unit 610, a generation unit 620, and a determination unit 630.

The embedding unit 610 embeds the mass characteristics.

Here, the embedding unit 610 may embed the margin label and the shape label using the embedding module with the margin label and the shape label having BIRADS characteristics.

The generation unit 620 generates a mass image corresponding to the mass characteristic, for example, a fake mass image, by using a pre-learned deep network using the embedded mass characteristic and the seed mass image as inputs.

Here, the generation unit 620 may generate a mass image corresponding to the mass characteristics by using a generative adversarial network (GAN).

Here, the generation unit 620 maps the mass having the mass characteristics embedded in the latent space formed by the seed mass image and the embedded mass characteristics, and performs a visual feature processing process in the latent space to correspond to the mass characteristics. Can be generated.

Here, the generator 620 may generate a mass image corresponding to the mass characteristic by using at least one of visual feature linear interpolation and visual feature linear interpolation in the latent space.

The deep network embeds margin and shape characteristics of the Breast Imaging-Reporting and Data System (BIRADS) to obtain embedded and embedded shape characteristics, and describes embedded margin characteristics and embedded shape characteristics. And maps the masses with embedded margin characteristics and embedded shape characteristics in the latent space formed by the actual mass image, and performs a visual feature processing process in the latent space to generate a mass image corresponding to the margin and shape characteristics. It can be learned through the process of distinguishing the actual mass from the fake mass using the generated mass image and the actual mass image.

The determination unit 630 outputs a prediction result for a fake / real, margin label, and a shape label by inputting a real mass and a fake mass.

Although the description is omitted in the system of FIG. 6, each constituent means of FIG. 6 may include all the contents described in FIGS. 2 to 4, which are obvious to those skilled in the art.

The apparatus described above may be implemented as a hardware component, a software component, and / or a combination of hardware components and software components. For example, the devices and components described in the embodiments may be, for example, processors, controllers, arithmetic logic units (ALUs), digital signal processors, microcomputers, field programmable arrays (FPAs), It may be implemented using one or more general purpose or special purpose computers, such as a programmable logic unit (PLU), microprocessor, or any other device capable of executing and responding to instructions. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to the execution of the software. For convenience of explanation, one processing device may be described as being used, but one of ordinary skill in the art will appreciate that the processing device includes a plurality of processing elements and / or a plurality of types of processing elements. It can be seen that it may include. For example, the processing device may include a plurality of processors or one processor and one controller. In addition, other processing configurations are possible, such as parallel processors.

The software may include a computer program, code, instructions, or a combination of one or more of the above, and configure the processing device to operate as desired, or process it independently or collectively. You can command the device. Software and / or data may be any type of machine, component, physical device, virtual equipment, computer storage medium or device in order to be interpreted by or to provide instructions or data to the processing device. It can be embodied in. The software may be distributed over networked computer systems so that they may be stored or executed in a distributed manner. Software and data may be stored on one or more computer readable recording media.

The method according to the embodiment may be embodied in the form of program instructions that can be executed by various computer means and recorded in a computer readable medium. The computer readable medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the media may be those specially designed and constructed for the purposes of the embodiments, or they may be of the kind well-known and available to those having skill in the computer software arts. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tape, optical media such as CD-ROMs, DVDs, and magnetic disks, such as floppy disks. Magneto-optical media, and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine code generated by a compiler, but also high-level language code that can be executed by a computer using an interpreter or the like.

Although the embodiments have been described by the limited embodiments and the drawings as described above, various modifications and variations are possible to those skilled in the art from the above description. For example, the described techniques may be performed in a different order than the described method, and / or components of the described systems, structures, devices, circuits, etc. may be combined or combined in a different form than the described method, or other components. Or even if replaced or substituted by equivalents, an appropriate result can be achieved.

Therefore, other implementations, other embodiments, and equivalents to the claims are within the scope of the claims that follow.

Claims (20)

Encoding a target mass characteristic in an encoding unit; And
Generating a mass image corresponding to the target mass property by using a pre-learned deep network that receives the encoded target mass property and the seed mass image as inputs from a generator;
Mass generation method comprising a.
The method of claim 1,
The generating step
And generating a mass image corresponding to the target mass characteristic by using a generative adversarial network (GAN) mapped from the seed mass image and the encoded target mass characteristic.
The method of claim 1,
The generating step
And generating a mass image corresponding to the target mass characteristic based on an encoder-decoder network having a residual block.
The method of claim 1,
The generating step
Encoding the seed mass image to extract features of the seed mass image, and concatenating and decoding the features of the extracted seed mass image and the encoded target mass characteristic to generate a mass image corresponding to the target mass characteristic Mass generation method characterized in that.
The method of claim 1,
The deep network
Encoding a Breast Imaging-Reporting and Data System (BIRADS) feature to obtain a BIRADS embedded with the feature, and using the embedded BIRADS and the actual mass image as inputs to correspond to the BIRADS feature. Generating a mass image, wherein the generated mass image and the characteristic are learned by using the embedded BIRADS to discriminate between the actual mass and the fake mass.
Embedding the mass characteristics in the embedding portion; And
Generating a mass image corresponding to the mass characteristics by using a pre-learned deep network that receives the embedded mass characteristics and the seed mass image in a generator;
Mass generation method comprising a.
The method of claim 6,
The generating step
A mass generation method comprising generating a mass image corresponding to the mass characteristics by using a generative adversarial network (GAN).
The method of claim 6,
The generating step
Mapping a mass having the embedded mass characteristic to a latent space formed by the seed mass image and the embedded mass characteristic, and performing a visual feature processing process in the latent space to generate a mass image corresponding to the mass characteristic. A mass producing method, characterized in that.
The method of claim 8,
The generating step
And generating a mass image corresponding to the mass characteristics using at least one of visual feature linear interpolation and visual feature linear interpolation in the latent space.
The method of claim 6,
The deep network
By embedding margin characteristics and shape characteristics of the Breast Imaging-Reporting and Data System (BIRADS), the embedded margin characteristics and the embedded shape characteristics are obtained, and the embedded margin characteristics, the embedded shape characteristics and A mass image having the embedded margin characteristic and the embedded shape characteristic is mapped to a latent space formed by an actual mass image, and a visual image processing process is performed in the latent space to correspond to the margin characteristic and the shape characteristic. And generating a mass, and learning through the process of distinguishing the actual mass from the fake mass using the generated mass image and the actual mass image.
An encoding unit for encoding the target mass characteristic; And
A generation unit generating a mass image corresponding to the target mass characteristic by using a pre-learned deep network that receives the encoded target mass characteristic and the seed mass image as inputs
Mass generation system comprising a.
The method of claim 11,
The generation unit
And generate a mass image corresponding to the target mass characteristic using a Generic Adversarial Network (GAN) mapped from the seed mass image and the encoded target mass characteristic.
The method of claim 11,
The generation unit
And generate a mass image corresponding to the target mass characteristic based on the encoder-decoder network having the residual block.
The method of claim 11,
The generation unit
Encoding the seed mass image to extract features of the seed mass image, and concatenating and decoding the features of the extracted seed mass image and the encoded target mass characteristic to generate a mass image corresponding to the target mass characteristic A mass generating system, characterized in that.
The method of claim 11,
The deep network
Encoding a Breast Imaging-Reporting and Data System (BIRADS) feature to obtain a BIRADS embedded with the feature, and using the embedded BIRADS and the actual mass image as inputs to correspond to the BIRADS feature. A mass generation system for generating a mass image, wherein the generated mass image and the characteristic are learned by using the embedded BIRADS to discriminate between the actual mass and the fake mass.
Embedding portion for embedding the mass characteristics; And
A generation unit for generating a mass image corresponding to the mass characteristics using a pre-learned deep network that takes the embedded mass characteristics and seed mass images as inputs.
Mass generation system comprising a.
The method of claim 16,
The generation unit
A mass generation system, characterized by generating a mass image corresponding to the mass characteristics using a Generative Adversarial Network (GAN).
The method of claim 16,
The generation unit
Mapping a mass having the embedded mass characteristic to a latent space formed by the seed mass image and the embedded mass characteristic, and performing a visual feature processing process in the latent space to generate a mass image corresponding to the mass characteristic. A mass generating system, characterized in that.
The method of claim 18,
The generation unit
And generate a mass image corresponding to the mass characteristic using at least one of visual feature linear interpolation and visual feature linear interpolation in the latent space.
The method of claim 16,
The deep network
By embedding margin characteristics and shape characteristics of the Breast Imaging-Reporting and Data System (BIRADS), the embedded margin characteristics and the embedded shape characteristics are obtained, and the embedded margin characteristics, the embedded shape characteristics and A mass image having the embedded margin characteristic and the embedded shape characteristic is mapped to a latent space formed by an actual mass image, and a visual image processing process is performed in the latent space to correspond to the margin characteristic and the shape characteristic. And generating a mass and using the generated mass image and the actual mass image to learn a process of distinguishing a real mass from a fake mass.

Images