KR20220153171A - Device and Method for Translating Breast Image - Google Patents

Abstract

The present invention provides a breast image translation device and method, which can translate a breast calcification original image into a breast cancer image by using a first generator and translate a breast cancer original image into a breast calcification image by using a second generator. The device comprises: a dataset management unit (200) which manages a dataset (210) including a plurality of breast calcification images (S) including a breast calcification original image (SO), a breast calcification similar image (SA), and a breast calcification translation image (ST), and a plurality of breast cancer images (C) including a breast cancer original image (CO), a breast cancer similar image (CA), a breast cancer translation image (CT); a learning model (300) composed of a first generator (310), a second generator (320), a first classifier (330), and a second classifier (340); and a training unit (400) for training the first generator (310), the second generator (320), the first classifier (330), and the second classifier (340). The dataset management unit (200) includes a pre-processing unit (250) which increases resolutions of the breast calcification original image (SO) and the breast cancer original image (CO) and then, for the breast calcification original image (SO) and the breast cancer original image (CO), adjusts ranges of a plurality of pixel values expressed by values from 0 to 255 into values between -1 and +1, respectively.

Description

Breast image conversion device and method {Device and Method for Translating Breast Image}

The present invention relates to a breast image conversion apparatus and method, and more particularly, a breast calcification original image may be converted into a breast cancer image using a first generator, and a breast cancer original image may be converted into a breast calcification image using a second generator. It relates to a breast image conversion device and method.

GAN (Generative Adversarial Network) is a machine learning model that can automatically generate images that are almost similar to real images while generators and classifiers compete with each other. A user can convert one image into an image with a different style using GAN, or create a new image with a similar style.

GAN (Image to Image Translation GAN) for image conversion is a technology that learns style conversion, which is the most widely used in the field of AI. Specifically, a GAN for image conversion can convert an image belonging to one dataset to an image belonging to another dataset in two datasets having different image styles without pair of images being connected to each other.

The GAN for image conversion is composed of a first generator and a second generator. In the GAN for image conversion, a first generator converts an image belonging to a first dataset into an image belonging to a second dataset, and a second generator converts an image belonging to the second dataset converted by the first generator back into first data. You can reconvert to an image that belongs to the set.

As image conversion technology using GAN for image conversion is widely used in the general industrial field, an image generation device capable of converting a real image into an image with a different style is being introduced in the medical field.

KR 10-2067340 B1

The present invention has been made to solve the above problems, and an object of the present invention is to convert a breast calcification original image into a breast cancer image using a first generator, and convert the breast cancer original image into a breast calcification image using a second generator. It is to provide a breast image conversion device and method capable of converting to .

In order to solve the above technical problems, the breast image conversion device 100 according to the present invention includes a breast calcification original image (SO), a breast calcification-like image (SA), and a breast calcification conversion image (ST). A dataset management unit ( 200), the first generator 310, the second generator 320, the first classifier 330 and the second classifier 340, and the learning model 300 and the first generator 310 , and a learning unit 400 for training the second generator 320, the first classifier 330, and the second classifier 340, and the dataset management unit 200 is a breast calcification original image (SO). ) and after increasing the resolution of the breast cancer original image (CO), a range of a plurality of pixel values expressed as values between 0 and 255 for the breast calcification original image (SO) and the breast cancer original image (CO), respectively. It is characterized in that it includes a pre-processing unit 250 that adjusts to a value within the range of -1 or more and +1 or less.

In addition, the breast calcification original image (SO) and breast cancer original image (CO) are images (Unpaired Image) images in which a breast with calcifications and a breast with cancer are each X-rayed and do not correspond to each other. The calcification-like image (SA) and the breast cancer-like image (CA) are images obtained by converting the breast calcification original image (SO) and the breast cancer original image (CO) by the learning model 300, respectively, and the breast calcification converted image The (ST) and the breast cancer conversion image (CT) are characterized in that the breast calcification original image (SO) and the breast cancer original image (CO) are converted by the learning model 300, respectively.

In addition, the first generator 310 converts the breast calcification original image SO processed by the preprocessor 250 into a breast cancer-like image CA, and the second generator 320 converts the first generator 320 into a breast cancer-like image CA. The breast cancer-like image (CA) converted by step 310 is converted into a breast calcification-like image (SA), and the second classifier 340 generates a breast cancer original image (CO) and the first generator (310). The second generator 320 converts the breast cancer original image CO processed by the pre-processing unit 250 into a breast calcification-like image SA. and the first generator 310 converts the breast calcification-like image SA converted by the second generator 320 into a breast cancer-like image CA, and the first classifier 330 It is characterized in that the authenticity of the breast calcification original image (SO) and the breast calcification-like image (SA) converted by the second generator 320 is determined.

In addition, the learning unit 400 converts the breast calcification original image SO so that the second classifier 340 can truly classify the breast cancer-like image CA converted by the first generator 310. The first generator learning unit 410 which trains the first generator 310 to convert the breast cancer similar image CA, the breast cancer original image CO, and the breast cancer similar image converted by the first generator 310 A second classifier learning unit 430 that receives (CA), classifies the breast cancer original image CO as real, and trains the second classifier 340 to classify the breast cancer similar image CA as fake. ), the breast cancer original image CO is converted into the breast calcification-like image SA so that the first classifier 330 can truly classify the breast calcification-like image SA converted by the second generator 320. ), the second generator learning unit 440 that trains the second generator 320 to convert the breast calcification original image SO and the breast calcification-like image SA converted by the second generator 320 to The first classifier learning unit 450 receives the input and trains the first classifier 330 to classify the original breast calcification image SO as real and classify the breast calcification-like image SA as fake. It is characterized by including.

In addition, the learning unit 400 converts the breast calcification original image SO into a breast cancer-like image CA and the first generator 310 converts the breast cancer-like image CA into a breast calcification-like image SA. The breast calcification original image SO is converted into the breast calcification original image SO so that the breast calcification-like image SA sequentially converted by the second generator 320 for conversion is converted to be similar to the breast calcification original image SO. The first generator 310 is trained to convert the breast cancer-like image CA, and the second generator 320 is trained to convert the breast cancer-like image CA into the breast calcification-like image SA. The generator learning unit 420 converts the breast cancer original image (CO) into a breast calcification-like image (SA), and the second generator 320 converts the breast calcification-like image (SA) and the breast calcification-like image The breast cancer similar image (CA) sequentially converted by the first generator 310 that converts (SA) into a breast cancer similar image (CA) is converted to be similar to the breast cancer original image (CO), Train the second generator 320 to convert a breast cancer original image CO into the breast calcification-like image SA, and convert the breast calcification-like image SA into the breast cancer-like image CA Characterized in that the first constructor 310 is learned.

In addition, the first generator 310 converts the breast cancer original image CO into the breast cancer similar image CA so that the overall color of the breast cancer similar image CA is similar to the overall color of the breast cancer original image CO, , The second generator 320 converts the breast calcification original image SO into the breast calcification-like image SA so that the overall color of the breast calcification-like image SA is similar to that of the breast calcification original image SO. , and the first generator learning unit 410

Learning the first constructor 310 to convert the breast cancer original image CO into the breast cancer similar image CA so that the overall color of the breast cancer similar image CA is similar to that of the breast cancer original image CO. and the second generator learning unit 440 converts the breast calcification original image SO into the breast calcification-like image so that the overall color of the breast calcification-like image SA is similar to that of the breast calcification-like image SO. It is characterized in that the second generator 320 is trained to convert to (SA).

In the breast image conversion method according to the present invention, after the preprocessing unit 250 increases the resolution of a plurality of breast calcification original images SO and a breast cancer original image CO, respectively, the plurality of breast calcification original images SO and In the first step, the learning unit 400 adjusts the range of a plurality of pixel values expressed as values between 0 and 255 in the breast cancer original image (CO) to a value within the range of -1 or more and +1 or less, respectively. The first generator 310, the second generator 320, and the first generator 310 convert the plurality of breast calcification original images SO processed by the preprocessor 250 into a plurality of breast calcification-like images SA, respectively. 2 In the second step of learning the classifier 340, the learning unit 400 converts the plurality of breast cancer original images CO processed by the preprocessing unit 250 into a plurality of breast cancer similar images CA. A third step of learning the second generator 320, the first generator 310, and the first classifier 330 to transform;

A fourth step in which the first generator 310 receives a plurality of new breast calcification original images SO and outputs a plurality of breast cancer conversion images CT, respectively. The second generator 320 receives a plurality of new breast cancer transformation images. A fifth step of receiving the original image CO and outputting a plurality of breast calcification conversion images ST, respectively, and a verification unit 260 of the plurality of breast cancer conversion images CT and breast calcification conversion images ST It is characterized in that it includes a sixth step of verifying reliability.

In addition, in the second step, the first generator learning unit 410 enables the second classifier 340 to truly classify the breast cancer-like image CA converted by the first generator 310, In a 2-1st step of learning the first generator 310 to convert the calcification original image SO into the breast cancer-like image CA, the generator learning unit 420 converts the first generator 310 and the first generator 310 2 The breast calcification original image SO is converted into the breast cancer-like image CA so that the breast calcification-like image SA sequentially converted by the generator 320 is similar to the breast calcification original image SO. In a 2-2 step of learning the first generator 310 so that the generator learning unit 420 converts the breast calcification-like image SA sequentially by the first generator and the second generator into the breast Step 2-3 of training the second generator 320 to convert the breast cancer-like image CA into the breast calcification-like image SA, similar to the calcification original image SO, the first generator Steps 2-4 of converting the breast calcification original image (SO) processed by the preprocessor (250) into the breast cancer-like image (CA) in the step 310, wherein the second generator 320 converts the first Steps 2 to 5 of converting the breast cancer-like image CA converted by the generator 310 into the breast calcification-like image SA, wherein the first generator learning unit 410 converts the breast cancer original image CO and Steps 2-6 of learning the first generator 310 to convert the breast cancer original image CO into a separate breast cancer similar image CA so that the overall color of the breast cancer similar image CA is similar; Steps 2-7 in which the first generator 310 converts the breast cancer original image (CO) processed by the pre-processor 250 into the separate breast cancer-like image (CA), the second classifier learning unit 430 ) is the breast cancer original image (C Steps 2-8 of training the second classifier 340 to classify O) as real and to classify the breast cancer-like image CA converted by the first generator 310 as fake; Steps 2 to 9 in which the classifier 340 determines whether the breast cancer original image CO and the breast cancer similar image CA converted by the first generator 310 are genuine or not, and the plurality of breast calcification originals It is characterized in that it includes steps 2-10 in which the steps 2-1 to 2-9 are repeated a predetermined number of times for the image SO and the original breast cancer image CO.

In addition, the third step is to enable the first classifier 330 of the second generator learning unit 440 to truly classify the breast calcification-like image (SA) converted by the second generator 320, such as breast cancer. In a 3-1st step of learning the second generator 320 to convert the original image CO into the breast calcification-like image SA, the generator learning unit 420 converts the second generator 320 and the second generator 320 1 The breast cancer original image CO is converted into the breast calcification-like image SA so that the breast cancer-like image CA sequentially converted by the generator 320 is similar to the breast cancer original image CO. Step 3-2 of learning the second generator 320, the generator learning unit 420 converts the breast cancer-like image (CA) sequentially by the second generator 320 and the first generator 320 A 3-3 step of learning the first generator 310 to convert the breast calcification-like image SA into the breast cancer-like image CA so that is similar to the breast cancer original image CO; 2 Steps 3 and 4 in which the generator 320 converts the breast cancer original image CO processed by the preprocessor 250 into the breast calcification-like image SA; In steps 3 to 5 of converting the breast calcification-like image SA converted by the second generator 320 into the breast cancer-like image CA, the second generator learning unit 440 converts the breast calcification original image ( A third method for training the second generator 320 to convert the breast calcification original image SO into a separate breast calcification-like image SA so that the overall color of the breast calcification-like image SO and the breast calcification-like image SA are similar. -6th step, 3rd to 7th steps in which the second generator 320 converts the breast calcification original image SO processed by the preprocessor 250 into the separate breast calcification-like image SA; The first classifier learning unit 450 A 3-8 step of learning the first classifier 330 so that ) classifies the original breast calcification image SO as real and classifies the breast calcification-like image SA as fake, the first classifier Steps 3-9 of determining whether the breast calcification original image SO and the breast calcification-like image SA converted by the second generator 320 are authentic, and the plurality of breast calcification originals (330) It is characterized in that it includes steps 3-10 in which steps 3-1 to 3-9 are repeated a predetermined number of times for the image SO and the original breast cancer image CO.

The apparatus and method for converting a breast image according to the present invention can convert a breast calcification original image into a breast cancer image using a first generator and convert a breast cancer original image into a breast calcification image using a second generator, so that it can be used in the medical field. The problem of insufficient breast cancer images and breast calcification images can be solved.

1 is a block diagram of a breast image conversion device according to the present invention.
Figure 2 is a configuration diagram of the dataset shown in Figure 1.
FIG. 3 is a diagram for explaining a process of generating a breast cancer-like image by the first generator shown in FIG. 1 .
FIG. 4 is a diagram for explaining a process of generating a breast calcification-like image by the second generator shown in FIG. 1 .
5 is a diagram for explaining a process in which a first generator converts an original breast cancer image into a third breast cancer-like image.
6 is a diagram for explaining a process in which a second creator converts an original breast calcification image into a third breast calcification-like image.
FIG. 7 is a diagram for explaining a process of verifying a breast cancer conversion image by the verification unit shown in FIG. 1 .
8 is a flowchart illustrating a breast image conversion method according to the present invention.
9 is a flowchart for explaining steps S125 in step S121.
10 is a flowchart for explaining steps S129-2 in step S126.
11 is a flowchart for explaining steps S135 in step S131.
12 is a flowchart for explaining steps S139-2 in step S136.

Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings in order to describe in detail enough for those skilled in the art to easily implement the technical idea of the present invention.

However, the following examples are merely examples to aid understanding of the present invention, and the scope of the present invention is not reduced or limited thereby. In addition, the present invention may be embodied in many different forms and is not limited to the embodiments described herein.

First, the breast image conversion device 100 according to the present invention will be described.

1 is a configuration diagram of a breast image conversion device 100 according to the present invention. Referring to FIG. 1 , a breast image conversion device 100 according to the present invention includes a dataset management unit 200, a learning model 300, and a learning unit 400.

FIG. 2 is a configuration diagram of the dataset 210 shown in FIG. 1 . Referring to FIG. 2 , the dataset management unit 200 includes a breast calcification original image (SO), a breast calcification image (S) including a breast calcification-like image (SA) and a breast calcification conversion image (ST), a breast cancer original image ( CO), a breast cancer similar image (CA), and a breast cancer image (C) including a breast cancer transformation image (CT) are managed.

Here, the breast calcification original image SO and the breast cancer original image CO are X-ray images of a breast with calcifications and a breast with cancer, respectively. Calcifications refer to a state in which calcium is deposited in breast tissue observed when a breast is X-rayed. The breast calcification original image (SO) and breast cancer original image (CO) are unpaired images, and can be acquired from public datasets such as INBreast and DDSM.

Also, the breast calcification-like image SA and the breast cancer-like image CA refer to images obtained by transforming the breast calcification original image SO and the breast cancer original image CO by the learning model 300 in training.

Also, the breast calcification conversion image ST and the breast cancer conversion image CT refer to images obtained by converting the breast calcification original image SO and the breast cancer original image CO by the learning model 300 after learning has been completed.

The dataset management unit 200 includes a pre-processing unit 250.

First, the preprocessor 250 increases the resolution of the breast calcification original image SO and the breast cancer original image CO. Thereafter, the pre-processing unit 250 sets a range of a plurality of pixel values represented by values between 0 and 255 for the breast calcification original image SO and the breast cancer original image CO, respectively, from -1 to +1. Adjust to a value within the range.

The learning model 300 includes a first generator 310 , a second generator 320 , a first classifier 330 and a second classifier 340 .

3 is a diagram for explaining a process of generating a breast cancer-like image (CA) and a breast calcification-like image (SA) by the first generator 310 and the second generator 320, respectively.

Referring to FIG. 3 , the first generator 310 converts the breast calcification original image SO processed by the preprocessor 250 into a breast cancer-like image CA.

Then, the second generator 320 converts the breast cancer-like image CA converted by the first generator 310 into a breast calcification-like image SA.

In addition, the second classifier 340 determines whether the breast cancer original image CO and the breast cancer similar image CA converted by the first generator 310 are genuine.

4 is a diagram for explaining a process of generating a breast calcification-like image (SA) and a breast cancer-like image (CA) by the second generator 320 and the first generator 310, respectively.

Referring to FIG. 4 , the second generator 320 converts the breast cancer original image CO processed by the preprocessor 250 into a breast calcification-like image SA.

Also, the first generator 310 converts the breast calcification-like image SA converted by the second generator 320 into a breast cancer-like image CA.

Also, the first classifier 330 determines whether the original breast calcification image SO and the breast calcification-like image SA converted by the second generator 320 are genuine.

5 is a diagram for explaining a process in which the first generator 310 converts a breast cancer original image CO into a breast cancer similar image CA.

Referring to FIG. 5 , the first generator 310 converts the breast cancer original image CO into the breast cancer similar image CA so that the overall color of the breast cancer similar image CA is similar to the overall color of the breast cancer original image CO. convert to

FIG. 6 is a diagram for explaining a process in which the second generator 320 converts a breast calcification original image SO into a breast calcification-like image SA.

Referring to FIG. 6 , the second generator 320 converts the breast calcification original image SO such that the overall color of the breast calcification-like image SA is similar to that of the breast calcification original image SO. It is converted into an image (SA).

Also, the learning unit 400 trains the first generator 310 , the second generator 320 , the second classifier 340 , and the first classifier 330 of the learning model 300 . Referring to FIG. 1 , the learning unit 400 includes a first generator learning unit 410, a generator learning unit 420, a second classifier learning unit 430, a second generator learning unit 440, and a first classification. It is configured to include a self-learning unit 450 .

First, the first generator learning unit 410 converts the breast calcification original image SO so that the second classifier 340 can truly classify the breast cancer-like image CA converted by the first generator 310. The first generator 310 is trained to transform into a breast cancer-like image (CA).

In addition, the first generator learning unit 410 converts the breast cancer original image CO into the breast cancer similar image CA so that the overall color of the breast cancer-like image CA is similar to the overall color of the breast cancer original image CO. The first constructor 310 is trained to

Here, the reason why the first generator learning unit 410 trains the first generator 310 to convert the breast cancer original image CO into a breast cancer similar image CA is that the first generator 310 converts the breast calcification image When converting to a breast cancer image, this is to make the overall color of the converted breast cancer image similar to the overall color of the original breast cancer image.

In addition, the generator learning unit 420 converts the breast calcification original image SO into a breast cancer-like image CA and the first generator 310 converts the breast cancer-like image CA into a breast calcification-like image SA. The breast calcification original image SO is transformed into the breast calcification original image SO so that the breast calcification-like image SA sequentially converted by the second generator 320 is converted to be similar to the breast calcification original image SO. The first generator 310 is trained to transform into an image CA, and the second generator 320 is trained to transform the breast cancer-like image CA into the breast calcification-like image SA.

Here, the reason why the generator learning unit 420 learns the first generator 310 and the second generator 320 as described above is the breast transformed through the first generator 310 and the second generator 320 sequentially. This is to make the calcification-like image (SA) as similar as possible to the initially input breast calcification original image (SO).

Meanwhile, after learning of the first constructor 310 by the first generator learning unit 410 and learning by the first constructor 310 and the second constructor 320 by the generator learning unit 420 have progressed, The first generator 310 reconverts the breast calcification original image SO into a separate breast cancer-like image, and the second generator 320 converts the separate breast cancer-like image converted by the first generator 310 into a breast calcification image. Reconvert to a similar image. Then, the first generator 310 reconverts the breast cancer original image CO into a separate breast cancer-like image.

Then, the second classifier learning unit 430 receives the breast cancer original image CO and the breast cancer similar image CA converted by the first generator 310, and classifies the breast cancer original image CO as real, The second classifier 340 is trained to classify the breast cancer-like image (CA) as fake.

Meanwhile, the second classifier 340 learned by the second classifier learning unit 430 outputs 1 when receiving the breast cancer original image CO. Then, when the second classifier 340 receives a breast cancer similar image (CA), it compares the image with the breast cancer original image (CO), and if the image is close to the breast cancer original image (CO), outputs 1 and, otherwise, output 0.

Also, the second generator learning unit 440 converts the breast cancer original image CO so that the first classifier 330 can classify the breast calcification-like image SA converted by the second generator 320 as real. The second generator 320 is trained to transform into a breast calcification-like image (SA).

In addition, the second generator learning unit 440 converts the breast calcification original image SO into the breast calcification-like image (so that the overall color of the breast calcification-like image SA is similar to the overall color of the breast calcification-like image SO) SA) to train the second generator 320.

Here, the reason why the second generator learning unit 440 trains the second generator 320 to convert the breast calcification original image SO into a breast calcification-like image SA is that the second generator 320 converts the breast cancer image This is to make the overall color of the converted breast calcification image similar to the overall color of the original breast calcification image when converting to a breast calcification image.

In addition, the generator learning unit 420 converts the breast cancer original image CO into a breast calcification-like image SA, and the second generator 320 converts the breast calcification-like image SA into a breast cancer-like image CA. The breast cancer original image CO is converted into the breast calcification-like image (CO) so that the breast cancer-like image (CA) sequentially converted by the first generator 310 to be converted to be similar to the breast cancer original image (CO) SA), and the first generator 310 is trained to convert the breast calcification-like image SA into the breast cancer-like image CA.

Here, the reason why the generator learning unit 420 learns the second generator 320 and the first generator 310 as described above is the breast cancer transformed through the second generator 320 and the first generator 310 sequentially. This is to ensure that the similar image (CA) is as similar as possible to the initially input breast cancer original image (CO).

Meanwhile, after learning of the second constructor 320 by the second generator learning unit 440 and learning by the second generator 320 and the first constructor 310 by the generator learning unit 420 have progressed, 2 The generator 320 reconverts the breast cancer original image CO into a separate breast calcification-like image, and the first generator 310 transforms the separate breast calcification-like image converted by the first generator 310 into breast cancer Reconvert to a similar image. Then, the second generator 320 reconverts the breast calcification original image SO into a separate breast calcification-like image.

Further, the first classifier learning unit 450 receives the breast calcification original image SO and the breast calcification-like image SA converted by the second generator 320 as input, and converts the breast calcification original image SO into a real one. and trains the first classifier 330 to classify the breast calcification-like image SA as fake.

Meanwhile, the first classifier 330 learned by the first classifier learning unit 450 outputs 1 when receiving the breast calcification original image SO, and receives the breast calcification similar image SA , the corresponding image is compared with the breast calcification original image SO, and 1 is output if the corresponding image is close to the breast calcification original image SO, and 0 is output otherwise.

After that, learning of the first constructor 310 by the first constructor learning unit 410, learning of the first constructor 310 and the second constructor 320 by the generator learning unit 420, and the first constructor ( 310) and conversion of the image by the second generator 320, learning of the second classifier 340 by the second classifier learning unit 430, discrimination of the image by the second classifier 340, Learning of the second generator 320 by the two-generator learning unit 440, learning of the second generator 320 and the first constructor 310 by the generator learning unit 420, and the second generator 320 and the second generator 320 Conversion of the image by the first generator 310, learning of the first classifier 330 by the first classifier learning unit 450, and discrimination of images by the first classifier 330 are sequentially performed a certain number of times. iteration proceeds.

As the learning of the first generator 310, the second generator 320, the second classifier 340, and the first classifier 330 is sequentially repeated, the first generator 310 and the second generator ( 320), the similarity of the similar image to the original image and the discrimination accuracy of the second classifier 340 and the first classifier 330 are further improved.

When learning of the learning model 300 by the learning unit 400 is completed, the first generator 310 receives a new breast calcification original image SO, outputs a breast cancer conversion image CT, and generates a second generator 310. The 320 may receive a new breast cancer original image CO and output a breast calcification conversion image ST.

FIG. 7 is a diagram for explaining a process in which the verification unit 260 shown in FIG. 1 verifies a breast cancer conversion image (CT).

Referring to FIGS. 1 and 7 , the dataset management unit 200 further includes a verification unit 260 that verifies reliability of a breast cancer conversion image (CT) and a breast calcification conversion image (ST).

First, the verification unit 260 inputs a breast cancer original image CO and a breast cancer conversion image CT obtained by converting the breast cancer original image CO into a classifier model, and outputs a predetermined number of vectors, respectively.

For example, the verification unit 260 may input a breast cancer transformation image (CT) to a classifier model and output a certain number of vectors from a final layer.

Thereafter, the verification unit 260 receives a certain number of vectors for the breast cancer original image (CO) and breast cancer conversion image (CT) by using a dimensionality reduction algorithm, and expresses them in a two-dimensional graph, respectively. Here, the dimension reduction algorithm is an algorithm that receives a certain number of vectors as input and reduces the dimension to a specific dimension.

After that, the verification unit 260, when the distribution of the 2D graph of the breast cancer conversion image (CT) overlaps the distribution of the 2D graph of the breast cancer original image (CO) by more than a preset range, the corresponding breast cancer conversion image (CT) ) can be classified as a reliable breast cancer image (C). For example, if the distribution of the two-dimensional graph of the breast cancer conversion image (CT) overlaps with the distribution of the two-dimensional graph of the breast cancer original image (CO) by 90% or more, the corresponding breast cancer conversion image (CT) is a reliable breast cancer image ( C) can be classified as

After that, the user randomly ranks the plurality of breast cancer transformation images (CT) and breast cancer original images (CO) and presents them to a medical imaging expert to further verify the reliability of the plurality of breast cancer transformation images (CT). can do.

A process of verifying the breast calcification conversion image ST by the verifier 260 is similar to a process of verifying the breast cancer conversion image CT, so it will be omitted here.

On the other hand, the verification unit 260 determines the number of second classifiers 340 that have learned a lot of breast cancer conversion images (CT) and third classifiers 350 that have learned a lot of breast cancer original images (CO) through the ROC curve and AUC. By comparing performance, reliability of breast cancer transformation images (CT) can be verified.

The ROC (Receiver Operating Characteristic) curve shows the false positive rate (FPR, 1-specificity) on the x-axis and the true positive rate (TPR, sensitivity) according to a specific threshold. ) as the y-axis. At this time, the true positive rate means the probability that the classifier classifies an image containing cancer as an image containing cancer, and the false positive rate means the probability that the classifier classifies an image that does not contain cancer as an image containing cancer. .

It can be said that the ROC curve is most ideal when the true positive rate is 1, that is, when it is formed by a line segment parallel to the horizontal direction.

Further, AUC (Area Under the ROC Curve) means an area under the ROC curve. The verifier 230 may determine that the performance of the first generator 310 and the second generator 320 is excellent as the AUC calculated by the verifier 230 is closer to 0.5.

Because, the closer the AUC is to 0.5, the discriminators 330 and 340 and the expert find it difficult to determine whether the similar image converted by the first generator 310 or the second generator 320 is the original image or the converted similar image. because it loses

The verification unit 260 verifies the reliability of the breast cancer conversion image (CT) through the ROC curve and AUC as follows.

First, a number of breast cancer transformation images (CT) and breast cancer original images (CO) that are randomly mixed are provided to a number of medical professionals.

Thereafter, a plurality of medical professionals write identification data for identifying breast cancer original data 231 among a plurality of breast cancer conversion images (CT) and breast cancer original data 231 on result sheets, respectively.

Thereafter, the verification unit 260 receives identification data written on a plurality of result sheets, generates an ROC curve, and calculates an AUC. In this case, the verifier 230 may determine that the performance of the first generator 310 and the second generator 320 is excellent as the calculated AUC is closer to 0.5.

On the other hand, in the breast image conversion apparatus 100 according to the present invention, the first classifier 330 and the second classifier 340 are provided with an encoder generating an overall feature map for an input image and a region for the input image. U-net, a convolutional network that is composed of a decoder that generates a characteristic map and can quickly and accurately segment medical images, can be applied.

When U-net is applied to the first classifier 330, the first classifier 330 is applied to the breast cancer original image (CO) and the breast calcification-like image (SA) converted by the second generator 320. By detecting an object for each pixel, overall features and regional features of the breast cancer original image (CO) and the breast calcification-like image (SA) can be grasped at the same time.

Further, the first classifier 330 specifically feeds back global features and regional features of the breast cancer original image 221 and the breast calcification-like image SA to the second generator 320 and the first generator 310. can do.

Through this, the second generator 320 can more accurately convert the breast cancer original image CO into a breast calcification-like image SA, and the first generator 310 converts the second breast calcification image 222b into a breast cancer-like image SA. It can be more accurately converted into an image CA.

Similarly, when U-net is applied to the second classifier 340, the second classifier 340 is applied to the breast cancer original image CO and the breast cancer similar image CA converted by the first generator 310. By detecting an object per pixel for each pixel, global and regional features of the breast cancer original image (CO) and the breast cancer-like image (CA) can be grasped at the same time.

Further, the second classifier 340 may specifically feed back overall features and regional features of the breast cancer original image CO and the breast cancer-like image CA to the first generator 310 and the second generator 320. can

Through this, the first generator 310 can more accurately convert the breast calcification original image SO into a breast cancer-like image CA, and the second generator 320 converts the breast cancer-like image CA into a breast calcification-like image. (SA) can be more accurately converted to

Next, a breast image conversion method according to the present invention will be described.

8 is a flowchart illustrating a breast image conversion method according to the present invention. Referring to FIG. 8 , the preprocessor 250 increases the resolution of a plurality of original breast calcification images SO and an original breast cancer image CO, and then sets the plurality of original breast calcification images SO and the original breast cancer image ( CO), the range of a plurality of pixel values expressed as a value between 0 and 255 is respectively adjusted to a value within the range of -1 or more and +1 or less. (S110)

Thereafter, the learning unit 400 converts the plurality of breast calcification original images SO processed by the preprocessing unit 250 into a plurality of breast calcification-like images SA, respectively, using a first generator 310 and a first generator 310 . 2 The generator 320 and the second classifier 340 are trained (S120).

9 is a flowchart for explaining steps S125 in step S121, and FIG. 10 is a flowchart for explaining steps S129-2 in step S126. Referring to FIGS. 9 to 10 , the first generator learning unit 410 allows the second classifier 340 to classify the breast cancer-like image CA converted by the first generator 310 as real. The first generator 310 is trained to convert the calcification original image SO into a breast cancer-like image CA (S121).

Thereafter, the generator learning unit 420 converts the breast calcification-like image SA sequentially converted by the first generator 310 and the second generator 320 to be similar to the original breast calcification image SO. The first generator 310 is trained to convert the original image SO into a breast cancer-like image CA (S122).

Thereafter, the generator learning unit 420 converts the breast cancer similar image CA so that the breast calcification-like image SA sequentially converted by the first generator and the second generator is similar to the breast calcification original image SO. The second generator 320 is trained to convert the calcification-like image SA (S123).

After that, the first generator 310 converts the breast calcification original image SO processed by the preprocessor 250 into a breast cancer-like image CA (S124).

Thereafter, the second generator 320 converts the breast cancer-like image CA converted by the first generator 310 into a breast calcification-like image SA (S125).

Thereafter, the first generator learning unit 410 converts the breast cancer original image CO into a separate breast cancer similar image CA so that the overall color of the breast cancer original image CO and the breast cancer similar image CA is similar. The first constructor 310 is trained. (S126)

After that, the first generator 310 converts the breast cancer original image CO processed by the pre-processing unit 250 into a separate breast cancer-like image CA (S127).

After that, the second classifier learning unit 430 classifies the breast cancer original image CO as real and classifies the breast cancer similar image CA converted by the first generator 310 as fake. 340) is learned. (S128)

After that, the second classifier 340 determines whether the breast cancer original image CO and the breast cancer similar image CA converted by the first generator 310 are genuine or not (S129).

Thereafter, steps 2-1 to 2-9 are repeated a predetermined number of times for a plurality of breast calcification original images SO and breast cancer original images CO. (S129-2)

Thereafter, the learning unit 400 generates a second generator 320 and a first generator to convert the plurality of breast cancer original images CO processed by the preprocessor 250 into a plurality of breast cancer similar images CA, respectively. (310) and the first classifier 330 are trained (S130).

11 is a flowchart for explaining steps S131 to S135, and FIG. 12 is a flowchart for explaining steps S139-2 in step S136. 11 and 12, the second generator learning unit 440 allows the first classifier 330 to classify the breast calcification-like image SA converted by the second generator 320 as real, The second generator 320 is trained to convert the breast cancer original image CO into a breast calcification-like image SA (S131).

Thereafter, the generator learning unit 420 makes the breast cancer original image (CA) sequentially converted by the second generator 320 and the first generator 320 similar to the breast cancer original image (CO). CO) into a breast calcification-like image (SA), the second generator 320 is trained (S132).

Thereafter, the generator learning unit 420 makes the breast cancer-like image CA sequentially converted by the second generator 320 and the first generator 320 similar to the breast cancer original image CO, so that the breast calcification-like image The first generator 310 is trained to convert the (SA) into a breast cancer-like image (CA) (S133).

After that, the second generator 320 converts the breast cancer original image CO processed by the preprocessor 250 into a breast calcification-like image SA (S134).

Thereafter, the first generator 310 converts the breast calcification-like image SA converted by the second generator 320 into a breast cancer-like image CA (S135).

Thereafter, the second generator learning unit 440 converts the original breast calcification image SO into a separate breast calcification-like image SA so that the overall color of the breast calcification-like image SO and the breast calcification-like image SA are similar. The second generator 320 is trained to convert to . (S136)

Thereafter, the second generator 320 converts the breast calcification original image SO processed by the preprocessor 250 into a separate breast calcification-like image SA (S137).

Thereafter, the first classifier learning unit 450 trains the first classifier 330 to classify the original breast calcification image SO as real and classify the breast calcification-like image SA as fake (S138). )

Thereafter, the first classifier 330 determines whether the original breast calcification image SO and the breast calcification-like image SA converted by the second generator 320 are genuine or not (S139).

Thereafter, steps 3-1 to 3-9 are repeated a predetermined number of times for a plurality of breast calcification original images SO and breast cancer original images CO. (S139-2)

Thereafter, the first generator 310 receives a plurality of new breast calcification original images SO and outputs a plurality of breast cancer conversion images CT, respectively. (S140)

Thereafter, the second generator 320 receives a plurality of new breast cancer original images CO, and outputs a plurality of breast calcification conversion images ST, respectively. (S150)

Thereafter, the verification unit 260 verifies the reliability of the plurality of breast cancer conversion images (CT) and breast calcification conversion images (ST) (S160).

A breast image conversion apparatus and method according to the present invention converts a breast calcification original image (SO) into a breast cancer conversion image (CT) almost similar to a real one using a first generator (310), and uses a second generator (320) Therefore, it is possible to convert the breast cancer original image CO into a breast calcification transformation image ST that is almost similar to the actual one, thereby solving the problem of insufficient breast cancer images 230 and breast calcification images S in the medical field.

As described above, the main technical idea of the present invention is to provide a breast image conversion device 100 and a method, and the embodiment described above with reference to the drawings is only one embodiment, and is a true right of the present invention. The scope is based on the claims, but will extend to equivalent embodiments that may exist in various ways.

100: breast image converter
200: dataset management unit
210: dataset
S: Breast calcification image
SO: Breast Calcification Original Image
SA: similar image of breast calcifications
ST: Breast Calcification Transformation Image
C: breast cancer image
CO: breast cancer original image
CA: similar image of breast cancer
CT: Breast Cancer Transformation Image
250: pre-processing unit
260: verification unit
300: learning model
310: first constructor
320: second constructor
330: first classifier
340: second classifier
350: third classifier
400: learning unit
410: first generator learning unit
420: constructor learning unit
430: second classifier learning unit
440: second generator learning unit
450: first classifier learning unit

Claims (9)

Breast calcification original image (SO), breast calcification image (S) with breast calcification-like image (SA) and breast calcification transformed image (ST), breast cancer original image (CO), breast cancer-like image (CA), breast cancer transformed image a dataset management unit 200 that manages a dataset 210 including a plurality of breast cancer images (C) including (CT);
a learning model 300 including a first generator 310, a second generator 320, a first classifier 330, and a second classifier 340; and
A learning unit 400 for learning the first generator 310, the second generator 320, the first classifier 330 and the second classifier 340;
The dataset management unit 200
After increasing the resolution of the breast calcification original image (SO) and the breast cancer original image (CO), a plurality of values expressed as values between 0 and 255 for the breast calcification original image (SO) and the breast cancer original image (CO). A breast image conversion device comprising a; pre-processing unit 250 which adjusts a range of pixel values to a value within a range of -1 to +1, respectively. According to claim 1,
The breast calcification original image (SO) and breast cancer original image (CO) are
Breasts with calcifications and breasts with cancers are each X-rayed and are unpaired images,
The breast calcification-like image (SA) and breast cancer-like image (CA)
A breast calcification original image (SO) and a breast cancer original image (CO) are converted images by the learning model 300 being trained, respectively,
The breast calcification conversion image (ST) and breast cancer conversion image (CT)
The breast image conversion device, characterized in that the breast calcification original image (SO) and the breast cancer original image (CO) are converted images by the learning model (300) for which learning has been completed. According to claim 1,
The first constructor 310 is
The breast calcification original image (SO) processed by the pre-processing unit 250 is converted into a breast cancer-like image (CA),
The second constructor 320 is
Converting the breast cancer-like image CA converted by the first generator 310 into a breast calcification-like image SA;
The second classifier 340 is
Determine whether the breast cancer original image (CO) and the breast cancer similar image (CA) converted by the first generator 310 are authentic;
The second constructor 320 is
The breast cancer original image (CO) processed by the pre-processing unit 250 is converted into a breast calcification-like image (SA),
The first generator 310 converts the breast calcification-like image SA converted by the second generator 320 into a breast cancer-like image CA,
The first classifier 330 is
The breast image conversion device characterized in that it determines whether the original breast calcification image (SO) and the breast calcification-like image (SA) converted by the second generator (320) are authentic. According to claim 1,
The learning unit 400
The breast calcification original image SO is converted into the breast cancer similar image CA so that the second classifier 340 can truly classify the breast cancer similar image CA converted by the first generator 310. a first generator learning unit 410 that trains the first generator 310 to do so;
The breast cancer original image CO and the breast cancer similar image CA converted by the first generator 310 are received, the breast cancer original image CO is classified as real, and the breast cancer similar image CA is fake. a second classifier learning unit 430 for training the second classifier 340 to classify;
The breast cancer original image CO is converted into the breast calcification-like image SA so that the first classifier 330 can truly classify the breast calcification-like image SA converted by the second generator 320. a second generator learning unit 440 that trains the second generator 320 to transform; and
The breast calcification original image SO and the breast calcification-like image SA converted by the second generator 320 are received, the breast calcification original image SO is classified as real, and the breast calcification-like image SA A first classifier learning unit 450 for training the first classifier 330 to classify ) as fake. According to claim 4,
The learning unit 400
The first generator 310 converts a breast calcification original image SO into a breast cancer-like image CA and the second generator 320 converts the breast cancer-like image CA into a breast calcification-like image SA so that the breast calcification-like image (SA) sequentially converted by
The first generator 310 is trained to convert the breast calcification original image SO into the breast cancer-like image CA, and the breast cancer-like image CA is converted into the breast calcification-like image SA. A generator learning unit 420 for learning the second generator 320 to
The generator learning unit 420
The second generator 320 converts a breast cancer original image CO into a breast calcification-like image SA, and the first generator 310 converts the breast calcification-like image SA into a breast cancer-like image CA so that the breast cancer-like image (CA) sequentially converted by
The second generator 320 is trained to convert the breast cancer original image CO into the breast calcification-like image SA, and the breast calcification-like image SA is converted into the breast cancer-like image CA. Breast image conversion device characterized in that for learning the first generator (310) to. According to claim 1,
The first constructor 310 is
Converting the breast cancer original image CO into the breast cancer similar image CA so that the overall color of the breast cancer similar image CA is similar to the overall color of the breast cancer original image CO;
The second constructor 320 is
Converting the breast calcification original image SO into the breast calcification-like image SA so that the overall color of the breast calcification-like image SA is similar to that of the breast calcification original image SO;
The first generator learning unit 410
Learning the first constructor 310 to convert the breast cancer original image CO into the breast cancer similar image CA so that the overall color of the breast cancer similar image CA is similar to that of the breast cancer original image CO. let it,
The second generator learning unit 440
The second generator to convert the breast calcification original image SO into the breast calcification-like image SA so that the overall color of the breast calcification-like image SA is similar to that of the breast calcification original image SO ( 320) for learning. After the preprocessing unit 250 increases the resolution of the plurality of breast calcification original images SO and breast cancer original images CO, respectively, for the plurality of breast calcification original images SO and breast cancer original images CO, a first step of adjusting a range of a plurality of pixel values expressed by a value between 0 and 255 to a value within a range of -1 or more and +1 or less, respectively;
The first generator 310 and the second generator 310 allow the learning unit 400 to convert the plurality of breast calcification original images SO processed by the preprocessor 250 into a plurality of breast calcification-like images SA. A second step of learning the generator 320 and the second classifier 340;
The second generator 320 and the first generator 320 convert the plurality of breast cancer original images CO processed by the pre-processing unit 250 into a plurality of breast cancer-like images CA, respectively, so that the learning unit 400 converts A third step of learning the generator 310 and the first classifier 330;
a fourth step in which the first generator 310 receives a plurality of new breast calcification original images SO and outputs a plurality of breast cancer conversion images CT;
a fifth step in which the second generator 320 receives a plurality of new breast cancer original images CO and outputs a plurality of breast calcification conversion images ST; and
A breast image conversion method comprising a sixth step of verifying reliability of the plurality of breast cancer conversion images (CT) and breast calcification conversion images (ST) by the verifier (260). According to claim 7,
The second step is
The first generator learning unit 410 converts the breast calcification original image SO so that the second classifier 340 can truly classify the breast cancer-like image CA converted by the first generator 310. a 2-1st step of training the first generator 310 to convert the breast cancer-like image CA;
The generator learning unit 420 configures the breast calcification-like image SA, sequentially converted by the first generator 310 and the second generator 320, to be similar to the original breast calcification image SO. a 2-2 step of training the first generator 310 to convert the original image SO into the breast cancer-like image CA;
The generator learning unit 420 converts the breast cancer similar image CA such that the breast calcification-like image SA, sequentially converted by the first generator and the second generator, is similar to the breast calcification original image SO. a 2-3 step of training the second generator 320 to convert ? into the breast calcification-like image SA;
Steps 2-4 of converting, by the first generator 310, the breast calcification original image SO processed by the pre-processor 250 into the breast cancer-like image CA;
Steps 2-5 of converting, by the second generator 320, the breast cancer-like image CA converted by the first generator 310 into the breast calcification-like image SA;
The first generator learning unit 410 converts the breast cancer original image CO into a separate breast cancer similar image CA so that the overall color of the breast cancer original image CO and the breast cancer similar image CA is similar. Steps 2-6 of learning the first generator 310;
Steps 2-7 of converting, by the first generator 310, the breast cancer original image (CO) processed by the pre-processing unit 250 into the separate breast cancer-like image (CA);
The second classifier so that the second classifier learning unit 430 classifies the breast cancer original image CO as real and classifies the breast cancer similar image CA converted by the first generator 310 as fake. Steps 2-8 of learning (340);
Steps 2-9 of determining, by the second classifier 340, whether the breast cancer original image CO and the breast cancer similar image CA converted by the first generator 310 are genuine or not; and
Steps 2-10 in which steps 2-1 to 2-9 are repeated a predetermined number of times for the plurality of breast calcification original images SO and breast cancer original images CO; Breast image conversion method characterized by. According to claim 7,
The third step is
The second generator learning unit 440 converts the breast cancer original image CO so that the first classifier 330 can truly classify the breast calcification-like image SA converted by the second generator 320. a 3-1st step of training the second generator 320 to transform into a breast calcification-like image SA;
The generator learning unit 420 configures the breast cancer original image (CA) sequentially converted by the second generator 320 and the first generator 320 to be similar to the breast cancer original image (CO). CO) into the breast calcification-like image SA;
The generator learning unit 420 calculates the breast calcification so that the breast cancer similar image CA sequentially converted by the second generator 320 and the first generator 320 is similar to the original breast cancer image CO. a 3-3 step of training the first generator 310 to convert the similar image SA into the breast cancer similar image CA;
Steps 3-4 of converting, by the second generator 320, the breast cancer original image CO processed by the pre-processing unit 250 into the breast calcification-like image SA;
Steps 3-5 of converting, by the first generator 310, the breast calcification-like image SA converted by the second generator 320 into the breast cancer-like image CA;
The second generator learning unit 440 converts the original breast calcification image SO into a separate breast calcification-like image SA so that the overall color of the breast calcification original image SO and the breast calcification-like image SA are similar. Step 3-6 of learning the second generator 320 to transform;
a step 3-7 in which the second generator 320 converts the breast calcification original image SO processed by the pre-processor 250 into the separate breast calcification-like image SA;
A third classifier for learning the first classifier 330 so that the first classifier learning unit 450 classifies the original breast calcification image SO as real and classifies the breast calcification-like image SA as fake. Step 8;
a step 3-9 in which the first classifier 330 determines whether the breast calcification original image SO and the breast calcification-like image SA converted by the second generator 320 are genuine; and
Steps 3-10 in which steps 3-1 through 3-9 are repeated a predetermined number of times for the plurality of breast calcification original images (SO) and breast cancer original images (CO); Breast image conversion method characterized by.

Images