JP2020198041A - Training device, training method, estimation device, and program - Google Patents

Abstract

To improve accuracy in task of a target domain.SOLUTION: A training device includes a third training part for training a third model, in which second data belonging to a second domain different from first data belonging to a first domain, pseudo first data belonging to the first domain obtained on the basis of the second data, and annotation data in the first domain given to the pseudo first data are inputted to output the annotation in a second domain corresponding to the second data.SELECTED DRAWING: Figure 1

Description

The present disclosure relates to training equipment, training methods, estimation equipment and programs.

In the machine learning method, there is a method called transfer learning that constructs a highly accurate prediction model while using a dataset with a small amount of annotations. Transfer learning, especially domain adaptation, is a method of adapting a model trained in one domain to another domain, and the original domain (source domain) trained with the trained model is transferred to the destination domain (target domain). It is adapted with less data than before. For example, it is a method of adapting a trained model to domains with different observation data, domains with different teacher labels, and the like.

Such transfer learning has made it possible to adapt different datasets to models trained on one dataset. However, there is a problem that it is difficult to contribute to improving the accuracy of the model with respect to the target domain because the data set is only adapted to the trained model of other domains.

JP-A-2016-203276

Provide training equipment and the like to improve the accuracy of tasks in the target domain.

According to one embodiment, the training device has a second data belonging to a second domain different from the first data belonging to the first domain, and a pseudo belonging to the first domain obtained based on the second data. A third model is trained in which the first data and the annotation data in the first domain given to the pseudo first data are input and the annotation in the second domain corresponding to the second data is output. It has 3 training departments.

The block diagram which shows the function of the training apparatus which concerns on one Embodiment. The figure which shows the input / output of the 1st model which concerns on one Embodiment. The figure which shows the input / output of the 2nd model which concerns on one Embodiment. The figure which shows the input / output of the 3rd model which concerns on one Embodiment. The block diagram which shows the function of the estimation apparatus which concerns on one Embodiment. A flowchart showing a training process according to an embodiment. A flowchart showing an estimation process according to an embodiment. The figure which shows the hardware implementation example which concerns on one Embodiment.

Hereinafter, embodiments of the present invention will be described with reference to the drawings. The drawings and the description of the embodiments are shown as an example, and do not limit the present invention.

(First Embodiment)
FIG. 1 is a block diagram showing the functions of the training device according to the present embodiment. The training device 1 of the present embodiment includes an input unit 10, a storage unit 11, a first training unit 12, a second training unit 13, a third training unit 14, and an output unit 15. In this training device 1, the data (first data) belonging to the annotated source domain (first domain) and the data belonging to the target domain (second domain) whose domain is different from the first data (second data). ) And, by executing the training based on, a trained model that outputs an annotation when the second data is input is generated.

The input unit 10 receives input of training data and the like including the first data and the second data. The data received by the input unit 10 may be stored in the storage unit 11 or output to each training unit.

The storage unit 11 stores data such as training data required for processing. As described above, for example, the data received by the input unit 10 may be stored. Further, when the training device 1 specifically executes software processing using hardware, it may store programs, binary data, and the like necessary for execution. In this case, the process may be executed by the processor reading the executable file or the like stored in the storage unit 11.

When the first data is input, the first training unit 12 trains (hereinafter, also referred to as generation) the first model that outputs the annotation based on the first data and the annotation. The first model may be, for example, a model provided in a classifier, a classifier, or the like. As the first model, one generated by another training device may be used. In this case, the training device 1 does not have to include the first training unit 12, and the parameters necessary for forming the pre-trained first model are input or stored via the input unit 10. It may be stored in advance in the unit 11.

The second training unit 13 generates pseudo first data (hereinafter referred to as pseudo first data) when the second data is input based on the first data, the second data, and the first model. Train (generate) the second model. In the present specification, the pseudo-first data means data belonging to the first domain or a domain close to the first domain, or data that can be regarded as belonging to the first domain. More specifically, the pseudo-first data is data in which the second data is converted by the second model or data generated based on the second data, and when input to the first model, an annotation close to the correct answer is acquired. It is data that can be done. The second model may be, for example, a model provided in a data generator or a data converter.

The third training unit 14 obtains the pseudo first data and the second data based on the annotation that can be obtained by inputting the pseudo first data into the first model, the second data, and the annotation of the second data. Upon input, a third model that outputs annotations in the second domain corresponding to the second data is trained (generated). Similar to the first model, the third model may be, for example, a model provided in a classifier, a classifier, or the like.

The parameters and the like constituting at least one of the first model, the second model, and the third model may be stored in the storage unit 11. In this case, at least one of the first training unit 12, the second training unit 13, and the third training unit forms a model based on the parameters and the like stored in the storage unit 11, and the formed model is trained. You may update it.

The output unit 15 outputs the parameters and the like of each generated model to the outside via the interface. In addition, the parameters being trained may be output to the user via the user interface. In this embodiment, the term “output” includes the concept of storing in the storage unit 11. When outputting parameters and the like to the storage unit 11, output from the output unit 15 to the outside is not essential.

FIG. 2 is a diagram showing an example of input / output of the first model among the above three models. The solid line shows the data transition in the training phase, and the broken line shows the data transition in the estimation phase.

In the learning phase, the first training unit 12 includes a data set (from the first data storage unit 30 in which the data of the first domain is stored, the first data and the annotation associated with the first data). 1st data set) is acquired. Based on the acquired data set, when the first data is input, the first model 100 in which the annotation corresponding to the first data is output is trained.

The first model 100 is, for example, a model based on a CNN (Convolutional Neural Network), an MLP (Multilayer Perceptron), or another neural network model. Further, the first model 100 is formed as, for example, a classifier that uses a softmax function or the like in the final layer. As the parameters forming the network such as hyperparameters and activation functions, any parameters can be used as long as the model can be appropriately generated.

The first training unit 12 trains the first model 100 by using a machine learning method. The training method is not particularly limited as long as the training method is appropriately optimized so that the first model 100 operates as, for example, a classifier. Any mini-batch process, back-propagation method, etc. may be used as long as they are appropriately optimized. The first training unit 12 trains the first model 100 based on the first data and the annotation associated with the first data used as the teacher label.

The first data storage unit 30 stores the first data. When the first data is used for training, annotations corresponding to each first data are added. Normally, these data are stored in the first data storage unit 30 as a first data set associated with the first data as annotations corresponding to the first data. The first data storage unit 30 is provided inside or outside the training device 1. For example, the first data storage unit 30 may be provided in the storage unit 11. As another example, the first data storage unit 30 may be provided outside the training device 1 and may be connected via a network. In this case, the training device 1 acquires the data stored in the first data storage unit 30 via the input unit 10 and transmits the data to the first training unit 12. Further, when the first data storage unit 30 is provided externally, the training device 1 that has acquired the first data or the like via the input unit 10 may temporarily store the first data or the like in the storage unit 11. .. Similarly, the second data storage unit 31, which will be described later, may be provided inside or outside the training device 1 and may transmit and receive data as described above.

As described above, when the first model 100 is trained externally in advance, the first training unit 11 is not an indispensable configuration. In this case, parameters and the like related to the pre-trained model may be stored in the storage unit 11. Then, at the time of training the third model described later, the first model 100 may be formed at a necessary timing.

In the estimation phase, the trained first model 100 trained by the first training unit 12 outputs the annotation D102 corresponding to the first data when the first data D100 is input, as shown by the broken line. ..

FIG. 3 is a diagram showing an example of input / output of the second model. The solid line shows the data transition in the training phase, and the broken line shows the data transition in the estimation phase.

In the training phase, the second training unit 13 stores, for example, the first data from the first data storage unit 30 and the second data storage unit 31 to the second data set in which the second domain data set (second data set) is stored. Get each data. Then, when the second data is input based on the first data and the second data, the second model 200 that outputs the data obtained by converting the second data into the pseudo first data is trained.

The second model 200 is, for example, a data generation model, and is formed by various models such as CNN and MLP like the first model 100. The second training unit 13 trains the second model 200 by using, for example, the method of GAN (Generative Adversarial Network). In particular, the CycleGAN method may be used to train a model that generates pseudo-first data from the second data. In addition, any method for training the generative model may be used.

When using CycleGAN, for example, a model (second model 200) that outputs data obtained by converting the second data into the first data when the second data is input, and a model (second model 200) that outputs the first data when the first data is input. A model that outputs data converted into two data is provided as a generator. Then, a model for determining whether the input data is legitimate data belonging to each domain or output from the generator is provided as a discriminator. The second training unit 13 executes training of these generators and discriminators.

In the estimation phase, the second model 200 trained by the second training unit 13 has data belonging to the first domain corresponding to the second data (pseudo-first) when the second data D200 is input, as shown by the broken line. 1 data) Outputs D202.

The second training unit 13 may use the first model 100 as an auxiliary training as shown by the dotted line in FIG. In this case, for example, by inputting the pseudo first data converted (output) by the second model 200 into the first model 100, the annotation corresponding to the pseudo first data is acquired. By confirming the acquired annotation, it is possible to acquire an index of the certainty of the conversion from the second data to the pseudo first data, and this certainty can be used for the training of the second model 200.

Further, the second data may be annotated in the second domain of the second data. The annotation of the second data in the second domain can be used for training the third model 300 shown below.

FIG. 4 is a diagram showing an example of input / output of the third model. The solid line shows the data transition in the training phase, and the broken line shows the data transition in the estimation phase.

In the training phase, first, the third training unit 14 prepares the data necessary for training the third model 300. The third training unit 14 uses the second data, the annotation in the second domain of the second data, and the annotation in the first domain of the pseudo first data corresponding to the second data as the training data. The annotation in the first domain of the pseudo first data means an annotation in the first domain that is pseudo-assigned to the corresponding second data. In other words, the training of the third model 300 by the third training unit 14 is performed by using the second data and annotations in the first domain and the second domain of the second data as training data. Annotation

Such annotations in the first domain of the second data can be obtained, for example, by inputting the second data into the model in the order of the second model 200 and the first model 100.

Specifically, first, in order to obtain the annotation in the first domain of the pseudo first data corresponding to the second data, the annotation in the first domain of the pseudo first data is acquired. Therefore, for example, first, the second data is acquired from the second data storage unit 31, and the second data is input to the second model 200 that can be converted into the first data to acquire the pseudo first data. Next, the acquired pseudo-first data is input to the first model 100 capable of adding annotations in the first domain corresponding to the first data, and the annotations in the first domain of the pseudo-first data are acquired. By this process, it is possible to acquire a pseudo annotation in the first domain corresponding to the second data. In addition, the annotation in the second domain corresponding to the second data is acquired from the second data storage unit 31.

The third training unit 14 trains the third model 300 using these second data, the annotations in the corresponding second domain, and the annotations in the first domain of the corresponding pseudo-first data.

The third model 300 is, for example, a model based on a CNN, MLP, or other neural network model, like the first model. Further, the third model 300 is formed as a classifier that uses a softmax function or the like in the final layer, for example. As the parameters forming the network such as hyperparameters and activation functions, any parameters can be used as long as the model can be appropriately generated. When the second data and the annotation in the first domain corresponding to the second data are input, the third model 300 outputs the annotation in the second domain of the second data.

In the third model 300, the second data and the annotation in the corresponding first domain may be input in the input layer, or the second data in the input layer and the annotation in the corresponding first domain in the intermediate layer may be input. Good. As described above, the network formation and the model may be any as long as the input / output is appropriately performed.

For example, the third training unit 14 inputs the second data and the annotation in the corresponding first domain to the third model 300, and teaches the annotation in the second domain corresponding to the input second data to the output. The third model 300 is trained by using it as data.

The third training unit 14 may execute the process from the second data to the acquisition of the annotation of the pseudo first data. For example, in this case, the third training unit 14 refers to the data of the second data storage unit 31 and acquires the parameters of the first model 100 and the second model 200 from the storage unit 11 and the like. The third training unit 14 forms each model from the acquired parameters, and inputs the data stored in the second data storage unit 31 into the formed model to acquire the data required for processing. In this way, the third training unit 14 may form the first model 100 and the second model 200 and execute the process. Further, the third training unit 14 may acquire necessary data by inputting the second data to the first model 100 and the second model 200 that have been reconstructed in advance. Further, as described above, it is also possible to acquire necessary data from the outside of the third training unit 14, and the third training unit 14 can execute the training of the third model 300. As described above, the third training unit 14 may acquire the training data in any way as long as the training of the third model 300 is feasible.

When the estimation is performed using the training device of the present embodiment, the third model 300 trained by the third training unit 14 has the second data D200 and the corresponding first model 300 as shown by the broken line in FIG. When the annotation D204 in the domain is input, the annotation D300 in the second domain corresponding to the second data is output. As shown in the figure, the annotation D204 in the corresponding first domain is acquired by inputting the pseudo first data D202 obtained by inputting the second data D200 into the second model 200 into the first model 100. To.

FIG. 5 is a block diagram showing the functions of the estimation device 2 according to the present embodiment. The estimation device 2 includes an input unit 20, a storage unit 21, a first estimation unit 22, a second estimation unit 23, a third estimation unit 24, and an output unit 25.

The input unit 20 accepts the input of the second data for which the annotation is to be estimated.

The storage unit 21 stores data necessary for processing of the estimation device 2, data necessary for control if necessary, and the like. Further, the data received by the input unit 20 may be temporarily stored. Further, when the software processing is specifically executed by the hardware, the estimation device 2 may store a program, binary data, or the like necessary for the execution. In this case, the process may be executed by the processor reading the executable file or the like stored in the storage unit 21. In addition, parameters and the like necessary for forming various models may be stored.

The first estimation unit 22 includes, for example, a first model, or forms a first model at a required timing, and when data belonging to the first domain (including pseudo-first data) is input, the data Estimate and output the annotation corresponding to.

The second estimation unit 23 is considered to belong to the first domain when, for example, the second model is provided or the second model is formed at a required timing and the second data belonging to the second domain is input. Pseudo first data is estimated (generated) and output.

The third estimation unit 24 includes, for example, a third model, or forms a third model at a required timing, and when the second data and the pseudo first data corresponding to the second data are input, the third estimation unit 24 The annotation in the second domain of the second data is estimated and output.

The output unit 25 outputs the estimated annotation to the outside via the interface. Further, the annotation may be output to the user via the user interface. In this embodiment, the term "output" includes the concept of storing in the storage unit 21. When outputting the annotation to the storage unit 21, the output from the output unit 25 to the outside is not indispensable.

The estimation device 2 may be provided in the same hardware as the training device 1. In this case, the training device 1 can also function as the estimation device 2. For example, the input unit 10 and the input unit 20, the storage unit 11 and the storage unit 21, and the output unit 15 and the output unit 25 have the same interface. Or it may be a device. Further, in this case, in the training of the third model in the training device 1, the first estimation unit 22 and the second estimation unit 23 are respectively based on the parameters already trained at the timing of using the first model and the second model. It may work.

The flow of data in each estimation unit in the estimation device 2 is the same as the flow of processing in the estimation phase with respect to FIG. 4 described above.

The above processing flow is briefly summarized using a flowchart. FIG. 6 is a flowchart showing a processing flow of the training device 1.

First, the first training unit 12 trains the first model using the input from the input unit 10 or the first data belonging to the first domain stored in the storage unit 11 and its annotation (S100). As described above, the trained first model already outside or inside the training device 1 can be used. In this case, this S100 can be omitted.

Next, the second training unit 13 trains a second model that generates data belonging to the first domain from the second data belonging to the second domain (S102). When this S102 is independent of S100, the order may be changed. As described above, when the first model is required for training by the second training unit 13, S102 and S100 cannot usually be interchanged.

Next, the training of the third model in the third training unit 14 is performed. Since various data are required for training the third model, data is generated. Specifically, first, pseudo-first data is generated by inputting the second data into the second model trained in S102 (S104). Subsequently, by inputting this pseudo-first data into the first model trained in S100, the annotation in the first domain corresponding to the input second data is estimated (S106).

Next, the third training unit 14 trains the third model using the second data, the annotation in the second domain corresponding to the second data, and the annotation in the corresponding first domain acquired in S106. (S108).

The above is the training process. Although data input / output is omitted in the flowchart except for parts that are particularly difficult to understand, for example, the second data and the like in S102 are appropriately acquired from the input unit 10 or the storage unit 11. Further, the parameters of each model trained in each step are appropriately output to the outside or the storage unit 11. This also applies to the estimation phase described below.

Next, the processing flow of the estimation phase will be described. FIG. 7 is a flowchart showing an example of processing by the estimation device 2.

First, the pseudo first data corresponding to the second data is acquired from the input second data (S200). Specifically, the second estimation unit 23 acquires the pseudo first data by inputting the second data into the trained second model.

Next, the pseudo first data acquired in S200 is input to the trained first model provided in the first estimation unit 22, and the first domain for the pseudo first data, that is, for the corresponding second data. Estimate the annotation in (S202).

Next, the second data and the annotation in the first domain of the second data are input to the third model provided in the third estimation unit 24, and the annotation in the second domain is estimated (S204).

As described above, when the second data is input to the estimation device 2 as a whole, it is possible to acquire the annotation of the second data in the second domain.

The third model makes it possible to make highly accurate estimates using a smaller number of datasets in the second domain than simply performing training using annotations in the second domain that correspond to the second data. Become. This is possible by adding annotations in the first domain, which are not in the second domain, to the data in the second domain.

As described above, according to the present embodiment, the data of the second domain is transcriptionally learned based on the data of the first model that already exists or the data of the first domain for which a large amount of data can be acquired. Generates an annotation model in the second domain. Further, by adding the annotation in the first domain to the second data, it is possible to improve the accuracy thereof.

(Modification example 1)
The first model, the second model, and the third model are assumed to be generated by different models in the above description, but are not limited thereto. For example, one model may have the functionality of at least two of these three models. In this case, the corresponding training unit may also be composed of one training unit.

(Modification 2)
In the above-described embodiment, for example, the first data and the second data may be data acquired by the same type of sensor. In particular, the first data and the second data may be images. In the case of an image, the sensor may include various technologies such as CT (Computed Tomography), optical CT, MRI (Magnetic Resonance Imaging), X-ray, RGB camera, RGBD camera, and infrared camera. In this case, the annotation in the first domain and the annotation in the second domain may be attached to each data for different purposes or different viewpoints.

When the data is acquired by the same type of sensor, for example, it is possible to convert the annotation information between the domains to which the data including the annotations given for different purposes belong. For example, even if the sensors are of the same type, the parameters and the like in data acquisition may be different. In addition, when more detailed annotation is required, for example, medical annotation tends to be costly. However, according to the above-described embodiment, it is possible to appropriately adapt annotations in different domains in the same type of sensor.

As a more specific example, the first data and the second data may be cross-sectional images of the brain acquired by MRI, respectively. For example, the first data may be a data set of brain tissue, and the second data may be a data set of brain tumor. In this case, since the purpose of interest is different, it is assumed that the image is acquired using different parameters. Then, for example, the first model may be a model for classifying the region of the brain tissue in the cross-sectional image.

In this case, for example, when the first data and the second data have acquired images of different channels in MRI, even if the second data is input to the first model as it is, a valid annotation estimate can be obtained. It is difficult. However, by training the second model and the third model according to the above-described embodiment, whether or not a tumor is present in the input second data using the region discrimination model already generated as the first model. It is possible to easily build a model that outputs an annotation for determining whether or not.

(Modification 3)
In another example, the first data and the second data may be data acquired by different types of sensors. Similarly, the first data and the second data may be images acquired by the sensor shown in the above example. For example, by using the first model that estimates the annotation for the data acquired by CT, it is possible to improve the accuracy of the model that estimates the annotation for the data acquired by MRI.

(Modification example 4)
Further, although the two-dimensional image has been described above, the present invention is not limited to this. For example, a transformation between a two-dimensional pixel and a three-dimensional voxel may be performed. For example, either the image of the first domain or the image of the second domain is a three-dimensional image such as a CT image of a helical scan, and may be a two-dimensional image acquired by MRI.

(Modification 5)
In the above-described embodiment, one model is used as the first model, but a plurality of first models for data belonging to different domains may be used. For example, if there are already models that estimate annotations for data belonging to a plurality of different domains, the results of these models may be used to build a third model. In this case, for each first model, a plurality of second models that output data matching the inputs of each first model are trained.

For example, if there is a first model for annotating an MRI image and another first model for annotating a CT image as the first model, X-rays belonging to the second domain are used using these models. It is possible to improve the accuracy of annotating an image. In this case, as the second model, a second model that generates a pseudo MRI image from the X-ray image and a second model that generates a pseudo CT image from the X-ray image are trained. At this time, the number of channels output from the second model may be adjusted as data that can be input to each first model. Of course, as described above, the plurality of first domains and the second domain may not all be data acquired by different sensors, and at least two may be data acquired by the same sensor.

(Modification 6)
In the above-described embodiment, the number of channels may be arbitrarily set in each domain. For example, the second model appropriately converts the data belonging to the second domain into the data belonging to the first domain, and the third training unit 14 performs the converted pseudo first data and the corresponding second data. The third model may be trained by concatenating (for example, as a separate channel). Alternatively, it may be input as separate data to the third model, which is the same network. In this way, it is possible to arbitrarily change the number of channels, and it is also possible to arbitrarily set the input layer and the like of the data set to generate an appropriate model.

(Modification 7)
In the above-described embodiment, the second training unit 13 may perform supervised learning or unsupervised learning. For example, as supervised learning, a model may be formed in which data of different domains are acquired under the same environment and the corresponding first data is used as a teacher label to generate pseudo-first data from the second data. On the other hand, as unsupervised learning, a model that acquires a sufficient amount of data in different domains under different environments and generates pseudo-first data from the second data by a method such as CycleGAN as described above without associating. May be formed. In general, it is often difficult to obtain data from different domains under the same environment, especially for medical images, but such unsupervised learning makes it easy to train. A third model can be obtained.

Although various modification examples have been described above, these have been described as examples. In principle, the training and estimation according to the present embodiment are, for example, the data and processing flow shown in FIG. However, the present invention is not limited to this, and the flow of data belonging to the first domain and data belonging to the second domain can be appropriately and appropriately changed within the scope of the purpose of model generation, as in each of the above modifications. Please note that.

In the above description, medical data is used as an example of data, and medical images are used in more detail, but the present invention is not limited to this.

Also, the classification of domains does not have to be definitely different as described above. For example, even if it is an MRI image, if the type and model number of the device to be captured are different, the captured image may be different even if the data is acquired using the same parameters. Even in such a case, it is possible to unify these results, and when different parameters are used, it is possible to convert annotations between different parameters. Further, for example, different annotation expressions may be used for each data consumer such as each hospital. Also, by generating a model that considers the data of other hospitals (data consumers), it is possible to output the annotation expression for each hospital while using the data of other hospitals.

Further, in the above-described embodiment, the first model is used to add annotations in the first domain of the pseudo first data, and the second model is used to acquire the second data from the pseudo first data. , Not limited to this. For example, the annotation in the first domain may be obtained from the pseudo first data by using another method such as a rule base for outputting the annotation in the first domain of the first data as an alternative means of the first model. Further, it may be used as an alternative means of the second model by using an image processing method or the like for acquiring pseudo data of the first domain from the data of the second domain.

In the training device 1 and the estimation device 2 in the above-described embodiment, each function may be a circuit composed of an analog circuit, a digital circuit, or an analog / digital mixed circuit. Further, a control circuit for controlling each function may be provided. The mounting of each circuit may be by ASIC (Application Specific Integrated Circuit), FPGA (Field Programmable Gate Array) or the like.

In all the above descriptions, at least a part of the training device 1 and the estimation device 2 may be configured by hardware, or may be configured by software and executed by a CPU or the like by information processing of the software. When composed of software, the training device 1, the estimation device 2, and a program that realizes at least a part of the functions are stored in a storage medium such as a flexible disk or a CD-ROM, and read by a computer for execution. It may be. The storage medium is not limited to a removable one such as a magnetic disk or an optical disk, and may be a fixed storage medium such as a hard disk device or a memory. That is, information processing by software may be concretely implemented using hardware resources. Further, the processing by software may be implemented in a circuit such as FPGA and executed by hardware. The generation of the model and the processing after inputting to the model may be performed by using an accelerator such as a GPU, for example.

For example, the computer can be made into the device of the above-described embodiment by reading the dedicated software stored in the storage medium readable by the computer. The type of storage medium is not particularly limited. Further, by installing the dedicated software downloaded via the communication network on the computer, the computer can be used as the device of the above embodiment. In this way, information processing by software is concretely implemented using hardware resources.

FIG. 8 is a block diagram showing an example of a hardware configuration according to an embodiment of the present invention. The training device 1 and the estimation device 2 include a processor 71, a main storage device 72, an auxiliary storage device 73, a network interface 74, and a device interface 75, and these are computer devices connected via a bus 76. It can be realized as 7.

Although the computer device 7 of FIG. 8 includes one component, the computer device 7 may include a plurality of the same components. Further, although one computer device 7 is shown in FIG. 8, software may be installed on a plurality of computer devices, and each of the plurality of computer devices may execute a part of processing different from the software. ..

The processor 71 is an electronic circuit (processing circuit, processing circuitry) including a control device and an arithmetic unit of a computer. The processor 71 performs arithmetic processing based on data and programs input from each apparatus of the internal configuration of the computer apparatus 7, and outputs the arithmetic result and the control signal to each apparatus and the like. Specifically, the processor 71 controls each component constituting the computer device 7 by executing an OS (operating system) of the computer device 7, an application, or the like. The processor 71 is not particularly limited as long as it can perform the above processing. Each training unit, each estimation unit, and each component thereof are realized by the processor 71.

The main storage device 72 is a storage device that stores instructions executed by the processor 71, various data, and the like, and the information stored in the main storage device 72 is directly read by the processor 71. The auxiliary storage device 73 is a storage device other than the main storage device 72. Note that these storage devices mean arbitrary electronic components capable of storing electronic information, and may be memory or storage. Further, the memory includes a volatile memory and a non-volatile memory, but either of them may be used. The memory for storing various data in the training device 1 and the estimation device 2 may be realized by the main storage device 72 or the auxiliary storage device 73. For example, the storage units 11 and 21 may be mounted on the main storage device 72 or the auxiliary storage device 73. As another example, when an accelerator is provided, the storage units 11 and 21 may be mounted in the memory provided in the accelerator. Further, various data may exist on the cloud connected via the network interface 74. In this case, the various data include various data such as data used for training, data to be estimated, and parameters used in the trained model.

The network interface 74 is an interface for connecting to the communication network 8 wirelessly or by wire. As the network interface 74, one conforming to the existing communication standard may be used. Information may be exchanged by the network interface 74 with the external device 9A which is communicated and connected via the communication network 8.

The external device 9A includes, for example, a camera, motion capture, an output destination device, an external sensor, an input source device, and the like. Further, the external device 9A may be a device having some functions of the components of the training device 1 and the estimation device 2. Then, the computer device 7 may receive a part of the processing results of the training device 1 and the estimation device 2 via the communication network 8 like a cloud service.

The device interface 75 is an interface such as a USB (Universal Serial Bus) that directly connects to the external device 9B. The external device 9B may be an external storage medium or a storage device. The storage units 11 and 21 may be realized by the external device 9B.

The external device 9B may be an output device. The output device may be, for example, a display device for displaying an image, a device for outputting audio, or the like. For example, there are LCD (Liquid Crystal Display), CRT (Cathode Ray Tube), PDP (Plasma Display Panel), speaker and the like, but the present invention is not limited thereto.

The external device 9B may be an input device. The input device includes devices such as a keyboard, a mouse, and a touch panel, and gives the information input by these devices to the computer device 7. The signal from the input device is output to the processor 71.

Based on all the above descriptions, those skilled in the art may be able to conceive of additions, effects or various modifications of the present invention, but aspects of the present invention are not limited to the individual embodiments described above. Absent. Various additions, changes and partial deletions can be made without departing from the conceptual idea and purpose of the present invention derived from the contents defined in the claims and their equivalents. For example, in all the above-described embodiments, the numerical values used in the explanation are shown as an example, and are not limited thereto.

Various operations of training and inference may be executed by parallel processing using, for example, an accelerator such as a GPU or using a plurality of computers via a network. For example, processing such as batch processing in training and generation of operation information of each object in inference may be performed by allocating operations to a plurality of arithmetic cores and executing them at the same timing.

1: Training device, 10: Input unit, 11: Storage unit, 12: 1st training unit, 13: 2nd training unit, 14: 3rd training unit, 15: Output unit,
2: Estimator, 20: Input unit, 21: Storage unit, 22: 1st estimation unit, 23: 2nd estimation unit, 24: 3rd estimation unit, 25: Output unit,
30: 1st data storage unit, 31: 2nd data storage unit

Claims (20)

Second data belonging to the second domain, which is different from the first data belonging to the first domain,
as well as,
Pseudo-first data belonging to the first domain obtained based on the second data, and annotation data in the first domain given to the pseudo-first data.
To train a third model that outputs annotations in the second domain corresponding to the second data, the third training unit,
Training device equipped with. When the first data is input, the annotation in the first domain is output, the first model is trained, the first training unit,
With more
The training device according to claim 1, wherein the annotation data in the first domain given to the pseudo first data is acquired by using the first model. The training device according to claim 2, wherein the first training unit trains based on the first data and the annotation of the first data. The second training unit, which trains the second model, outputs the pseudo first data by inputting the second data.
With more
The training device according to any one of claims 1 to 3, wherein the pseudo first data based on the second data is acquired by using the second model. The training device according to claim 4, wherein the second training unit trains based on the first data and the second data. The training device according to claim 5, wherein the second training unit further trains the second model based on the first model that outputs an annotation in the first domain when the first data is input. The third training unit trains based on the annotation, the second data, and the annotation of the second data output by inputting the pseudo first data into the first model, claim 2 or claim. Item 6. The training device according to item 6. The training device according to claim 7, wherein the third model is an input layer in which the annotation and the second data are input by inputting the pseudo first data into the first model and outputting the data. The third model is claimed in claim 7, wherein the second data is input in the input layer, and the annotation output by inputting the pseudo first data into the first model is input in the intermediate layer. Described training device. The training device according to any one of claims 1 to 9, wherein the first data and the second data are images acquired by the same type of sensor. The training device according to any one of claims 1 to 9, wherein the first data and the second data are images acquired by different types of sensors. It has at least two of the first models that process data belonging to different domains and the first models that output different annotations for data belonging to the same domain. The training device according to claim 2 or 6. At least two of each model to be trained consist of one neural network,
The training device according to any one of claims 2 to 12. The third training department
Second data belonging to the second domain, which is different from the first data belonging to the first domain,
as well as,
Pseudo-first data belonging to the first domain obtained based on the second data, and annotation data in the first domain given to the pseudo-first data.
To train a third model that outputs annotations in the second domain corresponding to the second data,
Training method with. A step of training a first model, in which the first training unit outputs an annotation in the first domain when the first data is input.
With more
The training method according to claim 14, wherein the annotation data in the first domain given to the pseudo first data is acquired by using the first model. The step of training the second model, in which the second training unit outputs the pseudo first data by inputting the second data.
With more
The training method according to claim 14 or 15, wherein the pseudo first data based on the second data is acquired by using the second model. Computer,
Second data belonging to the second domain, which is different from the first data belonging to the first domain,
as well as,
Pseudo-first data belonging to the first domain obtained based on the second data, and annotation data in the first domain given to the pseudo-first data.
A third training means, which trains a third model that outputs annotations in the second domain corresponding to the second data by inputting
A program that functions as. When the first data is input, the annotation in the first domain is output, the first model is trained, the first training means,
With more
The program according to claim 17, wherein the annotation data in the first domain added to the pseudo first data is acquired by using the first model. A second training means for training a second model, which takes the second data as an input and outputs the pseudo first data.
With more
The program according to claim 17 or 18, wherein the pseudo first data based on the second data is acquired by using the second model. Second data belonging to the second domain, which is different from the first data belonging to the first domain,
as well as,
Pseudo-first data belonging to the first domain obtained based on the second data, and annotation data in the first domain given to the pseudo-first data.
Is input, and the annotation in the second domain corresponding to the second data is output, the third estimation unit,
Estimator equipped with.

Images