KR102095684B1 - Method for domain adaptation based on adversarial learning and apparatus thereof - Google Patents

Abstract

Provided are a domain adaptation method based on adversarial learning, and an apparatus thereof. According to some embodiments of the present invention, the domain adaptation method based on adversarial learning comprises the following steps: extracting feature data from a plurality of data sets; learning, by using first feature data extracted from a first data set corresponding to a first class of a first domain among the plurality of data sets, a first discriminator which distinguishes a domain of data corresponding to the first class; learning, by using second feature data extracted from a second data set corresponding to the first class of a second domain among the plurality of data sets, the first discriminator; learning, by using third feature data extracted from a third data set corresponding to a second class of the first domain among the plurality of data sets, a second discriminator which distinguishes a domain of data corresponding to the second class; and learning, by using fourth feature data extracted from a fourth data set corresponding to the second class of the second domain among the plurality of data sets, the second discriminator. According to the present invention, the domain adaptation method based on adversarial learning is capable of reducing costs required for constructing a model.

Description

METHOD FOR DOMAIN ADAPTATION BASED ON ADVERSARIAL LEARNING AND APPARATUS THEREOF}

The present disclosure relates to a domain adaptation method and apparatus based on hostile learning. More specifically, in performing domain adaptation between a source domain and a target domain, while improving model performance in a target domain using hostile learning, a method for reducing model construction cost and an apparatus for supporting the method will be.

In the field of machine learning, domain adaptation refers to a method of training a model so as not to distinguish between a source domain and a target domain.

Domain adaptation can be used to reduce the cost of building a model in the target domain. Alternatively, it can be used to build a model showing satisfactory performance in the target domain by using a source domain that can easily secure a large data set.

However, even if domain adaptation is performed between similar domains, it is not easy to build a model showing satisfactory performance in the target domain.

Korean Open Patent No. 2018-0120478 (published Nov. 6, 2018)

A technical problem to be solved through some embodiments of the present disclosure is a method of performing domain adaptation based on hostile learning using a neural network including a class-specific discriminator and a method of performing the method Is to provide a device.

Another technical problem to be solved through some embodiments of the present disclosure, in performing hostile learning on a neural network including a plurality of discriminators, by utilizing a discriminator corresponding to each of a plurality of classes included in a domain, the neural network It is to provide a method and an apparatus for performing the method so as to learn a better representation for the target task to be performed.

Another technical problem to be solved through some embodiments of the present disclosure, in performing domain adaptation based on hostile learning on a neural network including a plurality of discriminators, discriminate according to the learning accuracy of a layer performing a target task It is to provide a method for obtaining a performance improvement effect according to domain adaptation based on hostile learning and an apparatus for performing the method by adjusting learning based on an inverted label of a group.

The technical problems of the present disclosure are not limited to the technical problems mentioned above, and other technical problems not mentioned will be clearly understood by those skilled in the art from the following description.

In order to solve the above technical problem, a domain adaptation method based on hostile learning according to some embodiments of the present disclosure includes: extracting feature data from a plurality of datasets in a method performed by a computing device; A first discriminator that classifies domains of data corresponding to the first class by using the first characteristic data extracted from the first dataset corresponding to the first class of the first domain among the datasets (discriminator) learning, using the second feature data extracted from the second dataset corresponding to the first class of the second domain among the plurality of datasets, learning the first discriminator, the The data corresponding to the second class is obtained by using the third feature data extracted from the third data set corresponding to the second class of the first domain among the plurality of data sets. Learning a second discriminator that classifies the domain of the data center and using the fourth feature data extracted from the fourth dataset corresponding to the second class of the second domain among the plurality of datasets, the second And training the discriminator.

In order to solve the above technical problem, a domain adaptation apparatus based on hostile learning according to some other embodiments of the present disclosure includes a plurality of datasets by executing one or more instructions and a memory storing one or more instructions. Feature data is extracted from, and the first feature data extracted from the first dataset corresponding to the first class of the first domain (domain) among the plurality of datasets is used to correspond to the first class. Learning a first discriminator that distinguishes a domain of data, and using the second feature data extracted from the second dataset corresponding to the first class of the second domain among the plurality of datasets, the A first discriminator is trained, and extracted from a third dataset corresponding to the second class of the first domain among the plurality of datasets Using a third feature data, a second discriminator for classifying a domain of data corresponding to the second class is trained, and in a fourth dataset corresponding to the second class of the second domain among the plurality of datasets And a processor that trains the second discriminator using the extracted fourth feature data.

In order to solve the above technical problem, a computer program according to another exemplary embodiment of the present disclosure is combined with a computing device to extract feature data from a plurality of datasets, and a first domain of the plurality of datasets ( learning a first discriminator that classifies a domain of data corresponding to the first class by using the first characteristic data extracted from the first data set corresponding to the first class of the domain) , Learning the first discriminator using the second feature data extracted from the second dataset corresponding to the first class of the second domain among the plurality of datasets, and the first of the plurality of datasets. Learning a second discriminator to classify domains of data corresponding to the second class by using the third characteristic data extracted from the third data set corresponding to the second class of the 1 domain The key is a computer for executing the step of training the second discriminator using the fourth feature data extracted from the fourth dataset corresponding to the second class of the second domain among the plurality of datasets. It can be stored in a readable recording medium.

1 and 2 are diagrams for explaining a machine learning apparatus and a learning environment according to some embodiments of the present disclosure.
3 is a flow diagram of a domain adaptation method based on hostile learning according to some embodiments of the present disclosure.
4 is a flowchart illustrating a detailed process of the data set acquisition step S100 shown in FIG. 3.
5 to 6 are conceptual diagrams for explaining a domain adaptation method based on hostile learning according to some embodiments of the present disclosure.
7 is a flowchart illustrating a detailed process of the output layer learning step S500 illustrated in FIG. 3.
8 to 9 are conceptual diagrams for explaining a domain adaptation method based on hostile learning according to some other embodiments of the present disclosure.
10 is a flowchart of a domain adaptation method based on hostile learning according to some other embodiments of the present disclosure.
11 to 12 are conceptual diagrams for explaining a domain adaptation method based on hostile learning according to some other embodiments of the present disclosure.
13 is a hardware configuration diagram illustrating an example computing device capable of implementing an apparatus according to various embodiments of the present disclosure.
14 is a configuration diagram of a medical image analysis system according to some embodiments of the present disclosure.

Hereinafter, preferred embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. Advantages and features of the present disclosure, and a method of achieving them will be apparent with reference to embodiments described below in detail together with the accompanying drawings. However, the technical spirit of the present disclosure is not limited to the following embodiments, but may be implemented in various different forms, and only the following embodiments make the technical spirit of the present disclosure complete, and in the technical field to which the present disclosure belongs. It is provided to fully inform the person of ordinary skill in the scope of the present disclosure, and the technical spirit of the present disclosure is only defined by the scope of the claims.

It should be noted that in adding reference numerals to the components of each drawing, the same components have the same reference numerals as possible even though they are displayed on different drawings. In addition, in describing the present disclosure, when it is determined that detailed descriptions of related known configurations or functions may obscure the subject matter of the present disclosure, detailed descriptions thereof will be omitted.

Unless otherwise defined, all terms (including technical and scientific terms) used in this specification may be used as meanings commonly understood by those skilled in the art to which the present disclosure pertains. In addition, terms defined in the commonly used dictionary are not ideally or excessively interpreted unless specifically defined. The terminology used herein is for describing the embodiments and is not intended to limit the present disclosure. In this specification, the singular form also includes the plural form unless otherwise specified in the phrase.

In addition, in describing the components of the present disclosure, terms such as first and second may be used. These terms are only for distinguishing the component from other components, and the nature, order, or order of the component is not limited by the term. When a component is described as being "connected", "coupled" or "connected" to another component, that component may be directly connected to or connected to the other component, but another component between each component It should be understood that elements may be "connected", "coupled" or "connected".

As used herein, “comprises” and / or “comprising” refers to the components, steps, operations, and / or elements mentioned above of one or more other components, steps, operations, and / or elements. Presence or addition is not excluded.

Prior to the description of the present specification, some terms used in the specification will be clarified.

In the present specification, a task refers to a task to be solved through machine learning or a task to be performed through machine learning. For example, when performing face recognition, facial expression recognition, gender classification, pose classification, etc. from face data, each of face recognition, facial expression recognition, gender classification, and pose classification may correspond to an individual domain. As another example, when it is said that abnormality is recognized, classified, and predicted from medical image data, each of the abnormality recognition, the abnormality classification, and the abnormality prediction may correspond to an individual task. And the task can also be called a target task.

In this specification, a neural network is a term encompassing all kinds of machine learning models designed to mimic a neural structure. For example, the neural network may include all types of neural network based models such as an artificial neural network (ANN), a convolutional neural network (CNN), and the like.

In this specification, an instruction is a series of computer readable instructions grouped by function, and refers to a component of a computer program and executed by a processor.

In the present specification, a domain discriminator is a term encompassing a model trained to determine a domain to which specific data belongs. For example, since the domain discriminator may be implemented based on various types of machine learning models, the technical scope of the present disclosure is not limited by the implementation method of the domain discriminator. The domain discriminator may be abbreviated as a discriminator.

Hereinafter, some embodiments of the present disclosure will be described in detail according to the accompanying drawings.

1 is a diagram for explaining a machine learning apparatus 10 and a learning environment according to some embodiments of the present disclosure.

Referring to FIG. 1, the machine learning apparatus 10 is a computing apparatus that performs machine learning for a neural network. The computing device may be a laptop, a desktop, a laptop, a server, etc., but is not limited thereto and may include all types of devices equipped with computing functions. An example of the computing device will be described with reference to FIG. 13. Hereinafter, for convenience of description, the machine learning device 10 will be abbreviated as the learning device 10.

1 illustrates an example in which the learning device 10 is implemented as a single computing device, the function of the learning device 10 in an actual physical environment may be implemented through a plurality of computing devices. For example, the first function of the learning device 10 may be implemented in the first computing device, and the second function of the learning device 10 may be implemented in the second computing device. Also, a plurality of computing devices may be implemented by dividing the first function and the second function.

The datasets 12 and 13 illustrated in FIG. 1 are training datasets given a correct answer label and may belong to a plurality of domains. For example, the first data set 12 is a data set composed of a plurality of training samples (eg Data1) belonging to the first domain, and the second data set 13 is a plurality belonging to a second domain different from the first domain. It may be a data set consisting of a training sample (eg Data2). Here, the training sample may mean a unit of data for learning, and may be various data. For example, the training sample may be one image, and may further include various data other than the image according to a learning object or task.

According to various embodiments of the present disclosure, the learning device 10 may train a neural network using domain adaptation based on hostile learning. For example, the learning device 10 may construct a neural network that can be used together in the first domain and the second domain by using the domain adaptation. The learning can be performed using datasets 12 and 13 belonging to each domain. Only in this embodiment, the learning device 10 may be referred to as a domain adaptation device 10.

The neural network may be configured, for example, as illustrated in FIG. 2. 2 illustrates a neural network that can be used for domain adaptation to two different domains.

As illustrated in FIG. 2, the first dataset 12 belonging to the first domain includes a dataset 12-1 classified as a first class and a dataset 12-2 classified as a second class. ). The second dataset 13 belonging to the second domain may also include a dataset 13-1 classified as a first class and a dataset 12-2 classified as a second class. Hereinafter, in order to provide convenience of understanding, it is described that domain adaptation is performed on two domains, but the number of domains may vary according to embodiments.

As shown in FIG. 2, the neural network may include an output layer 15, two discriminators 16 and 17 and a shared feature extraction layer 14.

The first discriminator 16 may correspond to the first class, and the second discriminator 17 may correspond to the second class. That is, each discriminator 16 and 17 may be a class-specific discriminator. Accordingly, the first discriminator 16 is based on the data set 12-1 corresponding to the first class of the first domain and the data set 13-1 corresponding to the first class of the second domain. Can be learned. In addition, the second discriminator 17 is based on the data set 12-2 corresponding to the second class of the first domain and the data set 13-2 corresponding to the second class of the second domain. Can be learned.

The output layer 15 may be trained to perform target tasks such as classification using the entire datasets 12 and 13 belonging to the first domain and the second domain.

Since the feature extraction layer 14 needs to extract common features of both domains, it can be learned based on both the first domain and the second domain datasets 12 and 13. At this time, hostile learning may be performed between the feature extraction layer 14 and each of the discriminators 16 and 17. That is, the discriminators 16 and 17 are trained to distinguish domains well, and the feature extraction layer 14 can be trained to not distinguish domains well. The hostile learning will be described in detail with reference to FIGS. 3 to 12.

2 illustrates that there are two target classes of the neural network, but the number of classes may be variously defined and designed according to a target task of the neural network.

For example, when the target task is a task for determining the presence or absence of malignantness, a class indicating positive and a class indicating negative may be defined as target classes of the neural network. Further, in this case, the neural network may include two discriminators corresponding to each of the two classes.

For another example, if the target task is a task for diagnosing cancer, a class pointing to cancer, a class pointing to benign, and a class pointing to normal are defined as target classes of the neural network. You can. In addition, in this case, the neural network may include three discriminators corresponding to each of the three classes.

For another example, when the target task is a task for determining the type of disease or the type of tumor, three or more classes indicating the type of each disease or the type of tumor may be defined as target classes of the neural network.

So far, the learning apparatus 10 and the learning environment according to some embodiments of the present disclosure have been described with reference to FIGS. 1 and 2. Hereinafter, methods according to various embodiments of the present disclosure will be described.

Each step of the above methods can be performed by a computing device. In other words, each step of the methods may be implemented with one or more instructions executed by a processor of a computing device. All of the steps included in the above methods may be performed by one physical computing device, but the above methods may also be performed by a plurality of computing devices. For example, the first steps of the methods may be performed by a first computing device, and the second steps of the methods may be performed by a second computing device. Hereinafter, it is assumed that each step of the above methods is performed by the learning apparatus 10 illustrated in FIG. 1 to continue the description. Accordingly, it may be understood that when the subject of each operation is omitted in the description of the above methods, it may be performed by the illustrated device 10. In addition, the methods to be described hereinafter will be of course capable of changing the execution order of each operation within a range in which the execution order can be logically changed as necessary.

3 is a flow diagram of a domain adaptation method based on hostile learning according to some embodiments of the present disclosure. However, this is only a preferred embodiment for achieving the object of the present disclosure, and of course, some steps may be added or deleted as necessary.

In the following description, the number of target domains to which domain adaptation is performed is two, and the structure of the learning target neural network is assumed to be configured as illustrated in FIG. 2. However, this is only for convenience of understanding, and the number of target domains and the structure of the neural network may be variously defined and designed according to embodiments.

In step S100, a data set for training a neural network is obtained. For example, the dataset belongs to a first domain and is a first dataset associated with a first class, a second dataset belongs to a second domain, and is associated with the first class, a first dataset belongs to the first domain, and a second A third dataset associated with a class and a fourth dataset belonging to the second domain and associated with the second class may be included. Hereinafter, unless otherwise stated, the first to fourth datasets will be used in the same sense as described above.

In some embodiments, the dataset of the first domain (ie, the first dataset and the third dataset) is composed of images generated by the first photographing method, and the dataset of the second domain (ie, the first The second dataset and the fourth dataset) may be composed of images generated by the second photographing method. That is, domains may be classified based on a photographing method. For example, the first imaging method may be a full-field digital mammography (FFDM) method, and the second imaging method may be a digital breast tomosynthesis (DBT) method. In this case, the neural network may be trained to perform a specific task (e.g. abnormal diagnosis, lesion location identification) on the FFDM image and the DBT image.

In some embodiments, the number of datasets of the first domain (ie, the first or third dataset) is greater than the number of datasets of the second domain (ie, the second or fourth dataset). Data (ie, training samples). In this case, a process of increasing the number of samples of the second domain through over-sampling may be further performed.

In some embodiments, the dataset of the first domain (ie, the first dataset and the third dataset) is different from the dataset of the second domain (ie, the second dataset and the fourth dataset). (Or format) data. For example, the first dataset may consist of a 2D image (e.g. FFDM image), and the second dataset may consist of a 3D image (e.g. DBT image). As another example, the first data set may be composed of a single channel or single layer image (e.g. FFDM image), and the second data set may be composed of a multi channel or multi layer image (e.g. DBT image). In this case, before inputting data to the neural network, a process such as adjusting (or transforming) the form of the input data according to the input format of the neural network may be further performed. In this regard, it will be described in detail with reference to FIG. 4.

4 assumes that the input form of the neural network is implemented according to the form of the first domain dataset (ie, the first dataset or the third dataset).

Referring to FIG. 4, in step S101, it is determined whether the first data set (or the third data set) and the second data set (or the fourth data set) contain different types of data. When the data types are different, in step S102, for each data included in the second data set (or fourth data set), the first data set (or third data set) has the same input form. Can be adjusted (or converted). The specific adjustment process may vary depending on the embodiment.

In some embodiments, the first dataset (or third dataset) is an FFDM image, and the second dataset (or fourth dataset) can be a DBT image. The DBT image may be multi-channel or 3D input, but the neural network may be implemented to receive a single channel image as an input as in the FFDM image. In this case, a single channel image is extracted (or sampled) from a multi-channel image, and the extracted single channel image can be input to a neural network.

In some embodiments, the first dataset (or third dataset) may include a single layer image, and the second dataset (or fourth dataset) may include a multi-layer image. Also, the neural network may be implemented to receive a single layer image as an input. In this case, a single layer image may be extracted (or sampled) from the multi-layer image, and the extracted single channel image may be input to a neural network.

In step S103, it is determined whether the adjusted data satisfies the condition. The condition refers to a criterion for determining whether the adjusted data is suitable as sample data for learning. For example, the sharpness, the ratio between the first data set (or third data set) and the second data set (or fourth data set), whether a specific color is included, or the size of the data may be a condition. The condition may be set through user input or may be set automatically according to the type of task. Further, the above-described conditions may be learned and reflected in the above-described adjustment process.

So far, an embodiment related to the detailed process of the data set acquisition step S100 has been described.

It will be described again with reference to FIG. 3.

The following steps S200 to S500 are related to a process of performing domain adaptation using a plurality of discriminators specialized for each class. Prior to the full-scale description of the steps S200 to S500, in order to provide convenience of understanding, when one discriminator is used, the reason why the accuracy of the neural network (that is, the task) is poor is briefly described with reference to FIGS. 5 and 6. Do it.

The neural network illustrated in FIG. 5 includes a feature extraction layer 31, an output layer 32 performing a task, and a first discriminator 33 for determining a domain. That is, the first discriminator 33 performs an operation of discriminating domains for data sets of all classes.

When domain adaptation based on hostile learning is performed in the neural network illustrated in FIG. 5, the first discriminator 33 is trained to distinguish domains well, and the feature extraction layer 31 distinguishes domains regardless of the class of the dataset You will be taught not to.

FIG. 6 conceptually shows the learning result of the neural network shown in FIG. 5. In particular, FIG. 6 conceptually illustrates the distribution of each data set 41, 42, 43, 44 on a feature space.

See Table 1 below for the meaning of each data set (41, 42, 43, 44).

First domain Second domain Class 1 (C1) First data set (41) Second data set (43) Class 2 (C2) 3rd data set (42) 4th data set (44)

As illustrated in FIG. 6, when one discriminator is used, datasets 41 to 44 belonging to two domains may be densely packed in a feature space regardless of class. That is, as a result of learning the feature extraction layer 31 so that differences between different domains are minimized regardless of the class, the distance between different classes is also reduced, so that datasets (eg 41 and 42) corresponding to different classes are obtained. It may be mixed in the dense region (46). In this case, since the reference line 45 for classifying the datasets (eg 41 and 42) corresponding to different classes cannot be clearly distinguished, the accuracy of the task is inevitably deteriorated. To solve, in some embodiments of the present disclosure, a plurality of discriminators specialized for each class may be included in the neural network. Each discriminator may correspond to one class and perform a domain discrimination function, but according to an embodiment, a specific discriminator may correspond to one or more classes.

Hereinafter, a process of performing domain adaptation using a plurality of discriminators will be described in detail with reference to FIG. 3 again.

In step S200, the feature data of the obtained dataset is extracted through the feature extraction layer of the neural network.

In step S300, the first discriminator and the feature extraction layer are learned using feature data corresponding to the first class. The feature data corresponding to the first class refers to feature data extracted by receiving a data set (ie, a first data set and a third data set) corresponding to the first class by the feature extraction layer. The first discriminator may mean a domain discriminator in charge of the first class.

In addition, a feature extraction layer is also learned by using feature data corresponding to the first class. Hostile learning may be performed between the feature extraction layer and the first discriminator.

The specific manner in which the hostile learning is performed may vary depending on the embodiment.

In some embodiments, the feature extraction layer can be learned based on an error based on an inverted label. The reverse label may mean a label in which a ground truth domain label is reversed. More specifically, a domain prediction value for feature data corresponding to the first class may be obtained through the first discriminator. The domain prediction value may mean a probability value for each domain (e.g., a confidence score for each domain) indicating which domain the dataset from which feature data is extracted belongs to. In addition, an error is calculated based on a difference between the domain prediction value and an inverted label, and the error is back propagated to update the weight of the feature extraction layer. At this time, the weight of the first discriminator is not updated through the back propagation of the error. This is because the first discriminator must be trained to distinguish domains well.

In some other embodiments, the domain predictor of the first discriminator is inverted, and an error may be calculated based on the difference between the inverted predictor and the correct answer domain label. For example, when the first domain probability and the second domain probability are 8/10 and 2/10 in the domain predicted value, the inverted domain predicted value has a first domain probability of 2/10 and a second domain probability of 8 It can be understood to be / 10. In addition, the calculated error may be back propagated to update the weight of the feature extraction layer. Even in such a case, the feature extraction layer may be learned such that it cannot distinguish domains of the input dataset.

In some other embodiments, an error is calculated between the domain predicted value of the first discriminator and the correct answer domain label, and the gradient of the calculated error may be inverted. That is, the weight of the feature extraction layer may be updated based on the inverted gradient.

In step S400, the second discriminator and the feature extraction layer are learned using feature data corresponding to the second class. The feature data corresponding to the second class refers to feature data extracted by receiving a data set corresponding to the second class by the feature extraction layer. The second discriminator may mean a domain discriminator in charge of the second class.

In addition, a feature extraction layer is also learned using feature data corresponding to the second class, and hostile learning may be performed between the feature extraction layer and the second discriminator. In this regard, reference is made to the description of step S300 described above.

In step S500, the output layer is learned. The output layer is a layer that is learned to perform a target task (ie, a task-specific layer), and outputs a probability (e.g. class-specific confidence score) of the input dataset belonging to each class. The detailed learning process of this step is illustrated in FIG. 7.

As illustrated in FIG. 7, an error (that is, a difference between the predicted value and the correct answer label) for the predicted value output from the output layer is calculated, and the weight of the output layer can be updated by back propagating the calculated error (S501 to S501). S503). At this time, the weight of the feature extraction layer may also be updated.

It will be described again with reference to FIG. 3.

FIG. 3 shows that step S500 is performed after steps S300 and S400 as an example. However, this is only for convenience of understanding, and some processes in step S500 (ie, the learning process associated with the first domain) are performed together with step S300, and some other processes (ie, learning process associated with the second domain) ) May be performed together with step S400. In addition, the learning process associated with the first domain and the learning process associated with the second domain may be performed simultaneously.

So far, the domain adaptation method based on hostile learning according to some embodiments of the present disclosure has been described with reference to FIGS. 3 to 7. Hereinafter, a learning result that can be obtained through the domain adaptation method described above with reference to FIGS. 8 and 9 will be briefly introduced.

8 illustrates a configuration of a neural network to which a domain adaptation method based on hostile learning is applied. As illustrated in FIG. 8, the neural network includes a feature extraction layer 51, an output layer 52, a first discriminator 53 specialized for the first class, and a second discriminator 54 specialized for the second class. It can contain.

FIG. 9 conceptually shows the learning result of the neural network shown in FIG. 8. In particular, FIG. 9 conceptually shows the distribution of each data set 61, 62, 63, and 64 on the feature space.

See Table 2 below for the meaning of each data set (61, 62, 63, 64).

First domain Second domain Class 1 (C1) First data set (61) Second data set (63) Class 2 (C2) Third Data Set (62) 4th data set (64)

As illustrated in FIG. 9, when domain adaptation based on hostile learning is performed using a class-specific discriminator, the distance between datasets 61/63 or 62/64 corresponding to the same class in the feature space becomes closer, and each other The distance between datasets 61/62 or 63/64 corresponding to different classes may be further increased. This is because the difference between domains for each class can be minimized by performing hostile learning independently for each class using a class-specific discriminator. In this case, since the datasets of different classes are not mixed and appear in the dense region, the first class and the second class can be clearly distinguished by the reference line 65 that classifies the classes. Therefore, the accuracy of the task can be improved. That is, the target neural network can learn an optimal representation capable of performing a task with high accuracy on datasets of different domains. [0059] So far, referring to FIGS. 3 to 9, the present disclosure A domain adaptation method based on hostile learning according to some embodiments has been described. According to the above-described method, by performing hostile learning for each class using a class-specific discriminator, a neural network performing a task with high accuracy in a target domain as well as a source domain can be constructed. Therefore, the cost of building a model in the target domain can be greatly reduced.

In particular, the above-described method can be utilized to improve the prediction performance of a neural network in a domain (eg DBT domain) where it is difficult to secure data easily, and the higher the similarity between the two domains, the more the prediction performance can be improved. .

Hereinafter, a domain adaptation method based on hostile learning according to some other embodiments of the present disclosure will be described with reference to FIGS. 10 to 12.

10 is a flowchart of a domain adaptation method based on hostile learning according to some other embodiments of the present disclosure. For clarity of the present disclosure, descriptions of contents overlapping with the previous embodiment will be omitted.

As shown in Fig. 10, in steps S1000 and S2000, a data set is obtained, and characteristic data of the obtained data set is extracted. For detailed descriptions of steps S1000 and S2000, the descriptions of steps S100 and S200 shown in FIG. 3 will be referred to.

In step S3000, each discriminator is learned using feature data corresponding to each class. That is, the feature extraction layer is fixed, and learning about the discriminator can be performed.

In step S4000, if the accuracy of learning of each discriminator exceeds a threshold, a feature extraction layer and an output layer are learned. If the accuracy of the learning exceeds a threshold, it may mean that the correct answer rate of the domain prediction result of each discriminator exceeds the threshold.

In step S4000, hostile learning may be performed between the feature extraction layer and each of the discriminators. That is, unlike the discriminator, the feature extraction layer can be learned so that it cannot distinguish domains. The specific method of the hostile learning has been described in detail in the previous embodiment, and further description will be omitted.

Meanwhile, in some embodiments of the present disclosure, hostile learning on the feature extraction layer may be controlled based on learning accuracy (ie, task accuracy) of the output layer. For example, when the accuracy of learning of the output layer exceeds (or exceeds) a threshold, hostile learning for the feature extraction layer may be controlled to continue (or resume). For another example, when the accuracy of learning of the output layer is less than (or less than) a threshold, it may be controlled to stop learning of the feature extraction layer. The learning accuracy of the output layer is low because it means that the distance between datasets of different classes in the feature space is getting closer. In this case, in order to increase the distance between datasets of different classes, hostile learning is stopped, and learning based on prediction errors of the output layer may be performed on the feature extraction layer. In order to provide a more convenient understanding, the present embodiment will be described in detail with reference to FIGS. 11 and 12.

11 conceptually shows the learning result of the neural network. In particular, FIG. 11 conceptually shows the distribution of each data set 71, 72, 73, and 74 on the feature space. Refer to Table 3 below for the meaning of each data set (71, 72, 73, 74).

First domain Second domain Class 1 (C1) First data set (71) Second data set (73) Class 2 (C2) Third Data Set (72) 4th data set (74)

When domain adaptation is performed for each class, datasets 72 and 73 of different classes in the feature space can be prevented from being mixed. Accordingly, as illustrated in FIG. 11, the first class and the second class may be distinguished through the reference line 75. However, the distance d1 between the datasets 72 and 73 of different classes is further increased. If the distance d3 between the datasets of the same class (eg 72, 74) is closer, the performance improvement effect according to the domain adaptation can be further improved. This is because the first class and the second class can be more clearly distinguished as the distance d1 becomes farther, and the distinction between the first domain and the second domain becomes more difficult as the distance d3 gets closer.

Here, in order to further narrow the distance d3, when hostile learning using the discriminator is performed intensively, a problem may arise where the distance d1 also approaches in the feature space in some cases. Therefore, it is necessary to monitor the distance d1 and to control the overall learning so that the distance d1 can be further distant again if necessary. It can be understood that the above-described embodiment is intended to solve this problem.

More specifically, learning accuracy of the output layer (ie, performance evaluation result) may be used as a monitoring index for the distance d1. The accuracy of the output layer is low because it can mean that the distance d1 is close.

Therefore, when the learning accuracy of the output layer is less than the threshold, hostile learning of the feature extraction layer using the discriminator may be stopped. Further, in order to increase the distance d1, learning about the output layer may be performed. Learning about the output layer may include updating the weights of the output layer and the feature extraction layer using prediction errors of the output layer.

Conversely, when the learning accuracy of the output layer is greater than or equal to a threshold, hostile learning of the feature extraction layer using the discriminator is resumed, and learning can be controlled such that the distance d3 between datasets of the same class is close.

 In some embodiments, when the accuracy of learning of the output layer is less than a threshold value, the importance of the output layer is increased, and learning of the output layer may be performed by reflecting the increased importance. For example, learning of the output layer may be performed in a form of amplifying a prediction error of the output layer based on the importance. In this case, the learning accuracy of the output layer can be increased again.

12 conceptually illustrates a learning result of a neural network according to the above-described embodiment.

As illustrated in FIG. 12, it can be seen that the distance d4 between the datasets 72 and 74 of the same class is close, and the distance d2 between the datasets 73 and 74 of different classes is significantly increased. . As described above, according to the above-described embodiment, the learning is controlled such that the distance between datasets belonging to the same class is closer and the distance between different classes is farther away, so that the effect of improving the performance of the neural network according to domain adaptation can be maximized.

So far, the domain adaptation method based on hostile learning according to some other embodiments of the present disclosure has been described with reference to FIGS. 10 to 12.

The technical idea of the present disclosure described so far with reference to FIGS. 1 to 12 may be embodied as computer readable codes on a computer readable medium. The computer-readable recording medium may be, for example, a removable recording medium (CD, DVD, Blu-ray Disc, USB storage device, removable hard disk), or a fixed recording medium (ROM, RAM, hard disk equipped with a computer). You can. The computer program recorded on the computer-readable recording medium may be transmitted to another computing device through a network such as the Internet and installed on the other computing device, thereby being used in the other computing device.

Hereinafter, an exemplary computing device 100 capable of implementing a device (e.g. learning device 10) according to various embodiments of the present disclosure will be described.

13 is a hardware configuration diagram illustrating the exemplary computing device 100.

As shown in FIG. 13, the computing device 100 loads one or more processors 110, buses 150, communication interfaces 170, and computer programs 191 performed by the processors 110. The memory 130 may include a storage 190 storing the computer program 191. However, only components related to the exemplary embodiment of the present disclosure are illustrated in FIG. 13. Accordingly, a person skilled in the art to which the present disclosure belongs may recognize that other general-purpose components may be further included in addition to the components illustrated in FIG. 13.

The processor 110 controls the overall operation of each component of the computing device 100. The processor 110 may include at least one of a central processing unit (CPU), a micro processor unit (MPU), a micro controller unit (MCU), a graphics processing unit (GPU), or any type of processor well known in the art of the present disclosure. It can be configured to include. Also, the processor 110 may perform operations on at least one application or program for executing a method / operation according to embodiments of the present disclosure. Computing device 100 may include one or more processors.

The memory 130 stores various data, commands and / or information. The memory 130 may load one or more programs 191 from the storage 190 to execute a method / operation according to various embodiments of the present disclosure. The memory 130 may be implemented as a volatile memory such as RAM, but the technical scope of the present disclosure is not limited thereto.

The bus 150 provides communication functions between components of the computing device 100. The bus 150 may be implemented as various types of buses, such as an address bus, a data bus, and a control bus.

The communication interface 170 supports wired and wireless Internet communication of the computing device 100. Also, the communication interface 170 may support various communication methods other than Internet communication. To this end, the communication interface 170 may include a communication module well known in the technical field of the present disclosure. In some cases, the communication interface 170 may be omitted.

The storage 190 may store the one or more computer programs 191 and various data (e.g. learning dataset), machine learning models, and the like temporarily. The storage 190 may include a non-volatile memory such as a flash memory, a hard disk, a removable disk, or any type of computer-readable recording medium well known in the art.

Computer program 191 may include one or more instructions that, when loaded into memory 130, cause processor 110 to perform a method / operation in accordance with various embodiments of the present disclosure. That is, the processor 110 may perform the methods / operations according to various embodiments of the present disclosure by executing the one or more instructions.

For example, the computer program 191 extracts feature data from a plurality of datasets, and a first extracted from a first dataset corresponding to a first class of a first domain among the plurality of datasets Learning a first discriminator to classify domains of data corresponding to a first class using feature data, extracted from a second dataset corresponding to the first class of the second domain among the plurality of datasets Using the second feature data, learning the first discriminator, using the third feature data extracted from the third data set corresponding to the second class of the first domain among the plurality of data sets, An operation of learning a second discriminator that classifies domains of data corresponding to 2 classes, and a fourth dataset corresponding to the second class of the second domain among the plurality of datasets Using the fourth feature data is extracted, it may include instructions to cause the operation of the second study group determination. In such a case, a domain adaptation device (e.g. 10) according to some embodiments of the present disclosure may be implemented through the computing device 100.

So far, with reference to FIG. 13, an exemplary computing device 100 capable of implementing devices according to various embodiments of the present disclosure has been described.

Next, the configuration and operation of the medical image analysis system according to some embodiments of the present disclosure will be described with reference to FIG. 14.

As shown in FIG. 14, the medical image analysis system according to the present embodiment includes a medical image capturing apparatus 200 and a machine learning apparatus 100. According to an embodiment, the medical image analysis result display device 300 may be further included in the medical image analysis system according to the present embodiment.

The medical image photographing apparatus 200 is an apparatus for photographing a medical image of the body, and may be, for example, an apparatus for photographing images such as X-ray, CT, and MRI. The medical image capturing apparatus 200 provides image data captured through a network to the machine learning apparatus 100. Since the medical image is sensitive personal information, the network may be a network from which external access is blocked. That is, the machine learning apparatus 100 and the medical imaging apparatus 200 may be devices located in the same hospital.

The machine learning apparatus 100 of FIG. 14 may be understood to be the same as that shown in FIG. 14. That is, the machine learning apparatus 100 accumulates image data provided from the medical imaging apparatus 200 and, when the machine learning performance criterion is satisfied, outputs output data suitable for the machine learning purpose using the newly accumulated image data. You will be able to train the model more advanced. In this process, a domain adaptation method based on hostile learning described with reference to FIGS. 1 to 12 is performed.

The definition data of the model trained by the machine learning device 100 may be transmitted to the medical image analysis result display device 300. Unlike the medical image photographing apparatus 200 and the machine learning apparatus 100, the medical image analysis result display apparatus 300 may be a computing device located outside the hospital where the medical image photographing apparatus 200 is installed. The medical image analysis result display device 300 receives and stores the definition data of the model from the machine learning apparatus 100, and inputs the medical image to be analyzed into the model to obtain analysis result data and to obtain the analysis result data. By rendering and displaying the result on the screen, it is possible to display the reasoning result for the medical image.

So far, various embodiments of the present disclosure and effects according to the embodiments have been described with reference to FIGS. 1 to 14. Effects according to the technical spirit of the present disclosure are not limited to the above-mentioned effects, and other effects not mentioned will be clearly understood by those skilled in the art from the following description.

In the above, even if all components constituting the embodiments of the present disclosure are described as being combined or operated as one, the technical spirit of the present disclosure is not necessarily limited to these embodiments. That is, within the scope of the present disclosure, all of the components may be selectively combined and operated.

Although the operations in the drawings are shown in a specific order, it should not be understood that the operations must be executed in a specific order or in a sequential order, or that all the illustrated actions must be executed to obtain a desired result. In certain situations, multitasking and parallel processing may be advantageous. Moreover, the separation of the various configurations in the above-described embodiments should not be understood as such separation is necessary, and the described program components and systems may generally be integrated together into a single software product or packaged into multiple software products. It should be understood that there is.

Although the embodiments of the present disclosure have been described with reference to the accompanying drawings, a person of ordinary skill in the art to which the present disclosure pertains may implement the present disclosure in other specific forms without changing the technical spirit or essential characteristics. You can understand that there is. Therefore, it should be understood that the above-described embodiments are illustrative in all respects and not restrictive. The scope of protection of the present disclosure should be interpreted by the claims below, and all technical spirits within the scope equivalent thereto should be interpreted as being included in the scope of the technical spirits defined by the present disclosure.

Claims (18)

In the method performed by the computing device,
Extracting a plurality of feature data from the data set by the feature extraction layer;
The first characteristic data corresponding to the first class of the first domain among the extracted plurality of characteristic data and the second characteristic data corresponding to the first class of the second domain among the plurality of characteristic data Learning a first discriminator that discriminates a domain of data corresponding to the first class by using;
A second feature data is generated by using a third feature data corresponding to the second class of the first domain and a fourth feature data corresponding to the second class of the second domain among the plurality of feature data. Learning a second discriminator to classify domains of data corresponding to the class;
Learning at least one of the feature extraction layer and the output layer according to learning of at least one of the first discriminator and the second discriminator; And
The output layer comprises the step of classifying a class by receiving at least one feature data, and outputting a class classification result,
Learning at least one of the feature extraction layer and the output layer,
And performing hostile learning between at least one of the first discriminator and the second discriminator and the feature extraction layer,
Machine learning method.
delete According to claim 1,
Learning at least one of the feature extraction layer and the output layer is
And if the learning accuracy of the first discriminator and the second discriminator exceeds a first threshold, learning at least one of the feature extraction layer and the output layer.
Machine learning method.
According to claim 1,
The step of performing the hostile learning is
If the learning accuracy of the output layer exceeds or exceeds a second threshold, learning the feature extraction layer using the first discriminator and the second discriminator; And
And when the learning accuracy of the output layer is less than or equal to or less than the second threshold, learning at least one of the output layer and the feature extraction layer.
Machine learning method.
According to claim 1,
The step of performing the hostile learning is
And using the error between the correct answer label and the predictive label or the error between the correct answer label and the reverse predictive label, training the feature extraction layer,
The correct answer label is correct answer domain information for at least one characteristic data among the plurality of characteristic data,
The inverted correct answer label is a reverse of the correct answer label,
The predicted label is a label predicted by the first discriminator or the second discriminator,
The inverted prediction label is an inversion of the predicted label.
Machine learning method.
According to claim 1,
The first domain and the second domain include domains corresponding to medical data,
The class includes a plurality of classes corresponding to at least one of the presence or absence and type of disease
Machine learning method.
According to claim 1,
The first domain corresponds to a 2D image,
The second domain corresponds to a 3D image
Machine learning method.
The method of claim 7,
The 2D image includes a full-field digital mammography (FFDM) image,
The 3D image includes a digital breast tomosynthesis (DBT) image
Machine learning method.
According to claim 1,
The first domain corresponds to a single layer image,
The second domain corresponds to a multi-layer image
Machine learning method.
A memory that stores one or more instructions; And
By executing the stored one or more instructions,
Extracting a plurality of feature data from the data set by the feature extraction layer,
The first characteristic data corresponding to the first class of the first domain among the extracted plurality of characteristic data and the second characteristic data corresponding to the first class of the second domain among the plurality of characteristic data By using, to train a first discriminator (discriminator) to distinguish the domain of the data corresponding to the first class,
A second feature data is generated by using a third feature data corresponding to the second class of the first domain and a fourth feature data corresponding to the second class of the second domain among the plurality of feature data. Train a second discriminator to classify the domain of data corresponding to the class,
Learning at least one of the feature extraction layer and the output layer according to learning of at least one of the first discriminator and the second discriminator,
The output layer includes a processor to classify a class by receiving at least one feature data, and to output a class classification result,
The processor
Performing hostile learning between at least one of the first discriminator and the second discriminator and the feature extraction layer,
Machine learning device.
delete The method of claim 10,
The processor
When the learning accuracy of the first discriminator and the second discriminator exceeds a first threshold, learning at least one of the feature extraction layer and the output layer
Machine learning device.
The method of claim 10,
The processor
When the learning accuracy of the output layer exceeds or exceeds a second threshold, the feature extraction layer is trained using the first discriminator and the second discriminator,
If the learning accuracy of the output layer is less than or equal to or less than the second threshold, learning at least one of the output layer and the feature extraction layer
Machine learning device.
The method of claim 10,
The processor
Using the error between the correct answer label and the predictive label or the error between the correct answer label and the reverse predictive label, the feature extraction layer is trained,
The correct answer label is correct answer domain information for at least one characteristic data among the plurality of characteristic data,
The inverted correct answer label is a reverse of the correct answer label,
The predicted label is a label predicted by the first discriminator or the second discriminator,
The inverted prediction label is an inversion of the predicted label.
Machine learning device.
The method of claim 10,
The first domain and the second domain include domains corresponding to medical data,
The class includes a plurality of classes corresponding to at least one of the presence or absence and type of disease
Machine learning device.
The method of claim 10,
The first domain corresponds to a 2D image,
The second domain corresponds to a 3D image
Machine learning device.
The method of claim 16,
The 2D image includes a full-field digital mammography (FFDM) image,
The 3D image includes a digital breast tomosynthesis (DBT) image
Machine learning device.
The method of claim 10,
The first domain corresponds to a single layer image,
The second domain corresponds to a multi-layer image
Machine learning device.

Images