JP6804009B2 - Learning devices, learning methods, and learning programs - Google Patents

Description

The present disclosure relates to learning devices, learning methods, and learning programs.

An evaluation item for changing the data having the second result value into the data having the first result value based on the relationship between the data having the first result value and the data having the second result value. And a data analyzer for extracting the value thereof have been proposed (see JP-A-2000-305941). When the value of the extracted evaluation item is changed, this data analyzer investigates the influence on the result value and calculates the effect of changing the value of the extracted evaluation item.

Further, a data set selection device has been proposed in which the correspondence relationship of output values corresponding to a plurality of attribute values of training data is learned by a plurality of prediction algorithms by using an active learning device including a plurality of different prediction algorithms. (See JP-A-2007-304782). This data set selection device predicts the output value corresponding to the prediction data by using a plurality of correspondences learned by each of the plurality of prediction algorithms, and acquires a plurality of output values for each of the plurality of prediction algorithms as prediction result values. Further, this data set selection device selects a data set in which the variation of a plurality of prediction result values by the acquired plurality of prediction algorithms is large in the corresponding prediction data data set.

Further, a technique for calculating a model relating to an output amount of a technical system that is non-linearly dependent on a plurality of input amounts in the form of an input amount vector has been proposed (see JP-A-2016-530585). ).

In material research and development, in order to obtain a material with better performance, a material with better performance is searched for by repeating experiments. In this case, it is preferable for the efficiency of research and development that an appropriate experimental condition can be newly searched from the combination of the experimental condition for producing the material performed in the past and the performance value of the experimental result. ..

Since the technique described in JP-A-2000-305941 can only search for data similar to the data existing in the database, it is not always appropriate even if it is applied to a method for searching for experimental conditions for producing a material. There is a problem that it may not be possible to search for various experimental conditions. Further, the techniques described in JP-A-2007-304782 and JP-A-2016-530585 do not take into consideration the search for new experimental conditions in the first place. It should be noted that this problem is a problem that can occur not only in the research and development of materials but also in the research and development of drugs.

The present disclosure has been made in view of the above circumstances, and an object of the present disclosure is to make it possible to search for appropriate experimental conditions for producing a material or a drug.

The learning device of the present disclosure inputs a plurality of combinations of experimental conditions for producing a material or a drug and performance values of experimental results, and inputs a plurality of combinations to an output model using the experimental conditions as an output. An out-licensing unit that derives the evaluation value of the output model using the performance value of the experimental result obtained by inputting the output experimental conditions into the experimental model for performing a virtual experiment, and an evaluation derived by the out-licensing unit. It is equipped with a learning unit that trains the output model by machine learning that reflects the value.

This makes it possible to search for suitable experimental conditions for producing a material or drug.

In the learning device of the present disclosure, the higher the ratio of the evaluation values to the values satisfying the target performance in the plurality of performance values, the better the value may be, and the performance value satisfying the target performance can be obtained. The smaller the number of virtual experiments up to, the better the value, or the closer the performance value is to the target performance, the better the value.

As a result, an appropriate output model is determined by evaluating the search behavior using an appropriate value as a performance value, and as a result, an appropriate experimental condition for producing a material or a drug can be searched.

Further, in the learning device of the present disclosure, the derivation unit may correct the evaluation value to a low value when the output model outputs experimental conditions that do not satisfy the predetermined rules.

As a result, it is possible to increase the possibility that experimental conditions satisfying predetermined rules based on past empirical rules and the like can be obtained, and as a result, it is possible to search for appropriate experimental conditions for producing a material or a drug. ..

Further, in the learning device of the present disclosure, the derivation unit may correct the experimental conditions output from the output model to experimental conditions that can be used in an actual experiment.

This makes it possible to increase the possibility of obtaining experimental conditions that can be used in actual experiments, and as a result, it is possible to search for appropriate experimental conditions for producing a material or a drug.

Further, the learning device of the present disclosure may be a model in which the output model is learned by using a genetic algorithm.

This makes it possible to search for more suitable experimental conditions for producing a material or drug.

The learning device of the present disclosure is an output model in which a plurality of combinations of experimental conditions for producing a material or a drug and performance values of experimental results, and candidates for experimental conditions are input, and the action value in reinforcement learning is output. , A virtual experiment is performed on the candidates for the experimental conditions corresponding to the action values above a predetermined value among the multiple action values output by inputting each of a plurality of combinations and candidates for a plurality of different experimental conditions. It is equipped with a learning unit that trains the output model using a value derived based on the performance value of the experimental result obtained by inputting it into the experimental model to be performed as a reward.

This makes it possible to search for suitable experimental conditions for producing a material or drug.

In the learning device of the present disclosure, the higher the ratio of the value satisfying the target performance in the plurality of performance values, the better the reward, or the virtual value until the performance value satisfying the target performance is obtained. The smaller the number of experiments, the better the value, or the closer the performance value is to the target performance, the better the value.

As a result of using an appropriate value as the performance value, it is possible to search for an appropriate experimental condition for producing a material or a drug.

Further, in the learning device of the present disclosure, reinforcement learning may be Q-learning, and the action value may be a Q-value.

This makes it possible to search for more suitable experimental conditions for producing a material or drug.

Further, in the learning device of the present disclosure, when the output model learned by the learning unit is used, the cumulative action value output by sequentially inputting the candidates of a plurality of experimental conditions into the output model a plurality of times is the maximum. An output unit may be further provided to output the candidate of the experimental condition to be the candidate of the experimental condition to be the next experimental condition.

This makes it possible to search for more suitable experimental conditions for producing a material or drug.

Further, in the learning device of the present disclosure, the experimental model may be a model obtained by machine learning.

This makes it possible to generate an output model that is specific to a specific problem.

Further, the learning device of the present disclosure may have a plurality of experimental models, and the creation conditions of the plurality of experimental models may be different.

Thereby, by using a plurality of virtual experimental results obtained by experimental models having different preparation conditions, it is possible to search for more appropriate experimental conditions for producing a material or a drug. Further, the experimental model may be a mathematical formula constructed by including a function such as sin or exp. This makes it possible to generate an output model even in an experimental system for which no experimental data has been obtained.

Further, in the learning device of the present disclosure, a plurality of output models may exist, and the creation conditions of the plurality of output models may be different.

This makes it possible to search for more appropriate experimental conditions for producing a material or drug by learning by evaluating multiple performance values obtained from multiple experimental conditions obtained by output models with different creation conditions. it can.

In the learning method of the present disclosure, a plurality of combinations of experimental conditions for producing a material or a drug and performance values of experimental results are input, and a plurality of combinations are input to an output model using the experimental conditions as an output. By inputting the output experimental conditions into an experimental model for performing a virtual experiment, the evaluation values of the output model are derived using the performance values of the experimental results obtained, and the derived evaluation values are reflected by machine learning. This is a method in which a computer executes a process of training an output model.

In the learning program of the present disclosure, a plurality of combinations of experimental conditions for producing a material or a drug and performance values of experimental results are input, and a plurality of combinations are input to an output model using the experimental conditions as an output. By inputting the output experimental conditions into an experimental model for performing a virtual experiment, the evaluation values of the output model are derived using the performance values of the experimental results obtained, and the derived evaluation values are reflected by machine learning. This is to make the computer execute the process of training the output model.

The learning method of the present disclosure is an output model in which a plurality of combinations of experimental conditions for producing a material or a drug and performance values of experimental results, and candidates for experimental conditions are input, and the action value in reinforcement learning is output. , A virtual experiment is performed on the candidates for the experimental conditions corresponding to the action values above a predetermined value among the multiple action values output by inputting each of the plurality of combinations and the candidates for the plurality of different experimental conditions. This is a method in which a computer executes a process of training an output model using a value derived based on the performance value of the experimental result obtained by inputting the experimental model to be performed as a reward.

The learning program of the present disclosure is an output model in which a plurality of combinations of experimental conditions for producing a material or a drug and performance values of experimental results, and candidates for experimental conditions are input, and the action value in reinforcement learning is output. , A virtual experiment is performed on the candidates for the experimental conditions corresponding to the action values above a predetermined value among the multiple action values output by inputting each of the plurality of combinations and the candidates for the plurality of different experimental conditions. This is for causing a computer to execute a process of training an output model by using a value derived based on the performance value of the experimental result obtained by inputting the experimental model to be performed as a reward.

Further, the learning device of the present disclosure inputs a plurality of combinations of experimental conditions for producing a material or a drug and performance values of experimental results, and inputs a plurality of combinations to an output model using the experimental conditions as an output. A machine that derives the evaluation value of the output model using the performance value of the experimental result obtained by inputting the experimental conditions output by the above into the experimental model for performing a virtual experiment, and reflects the derived evaluation value. It has a processor that trains an output model by training.

Further, the learning device of the present disclosure inputs a plurality of combinations of experimental conditions for producing a material or a drug and performance values of experimental results, and candidates for experimental conditions, and outputs an action value in reinforcement learning as an output. Of the multiple action values output by inputting a plurality of combinations and each of a plurality of different experimental condition candidates into the model, the experimental condition candidates corresponding to the action values equal to or higher than a predetermined value are virtually created. It has a processor that trains an output model by using a value derived based on the performance value of the experimental result obtained by inputting the experimental model to perform the experiment as a reward.

According to the present disclosure, suitable experimental conditions for producing a material or drug can be searched for.

It is a block diagram which shows an example of the functional structure of the learning apparatus in the learning phase which concerns on 1st Embodiment. It is a figure which shows an example of the learning data which concerns on each embodiment. It is a figure which shows an example of the output model which concerns on 1st Embodiment. It is a figure which shows an example of the data structure of the data output from the output model which concerns on 1st Embodiment. It is a figure which shows an example of the experimental model which concerns on each embodiment. It is a figure which shows an example of the data structure of the data output from the experimental model which concerns on each embodiment. It is a figure for demonstrating the derivation process of the evaluation value of the output model which concerns on 1st Embodiment. It is a figure for demonstrating the derivation process of the evaluation value of the output model which concerns on a modification. It is a block diagram which shows an example of the functional structure of the learning apparatus in the operation phase which concerns on 1st Embodiment. It is a block diagram which shows an example of the hardware composition of the learning apparatus which concerns on each embodiment. It is a flowchart which shows an example of the experimental model learning process which concerns on each embodiment. It is a flowchart which shows an example of the output model learning process which concerns on 1st Embodiment. It is a flowchart which shows an example of the experimental condition output processing which concerns on 1st Embodiment. It is a block diagram which shows an example of the functional structure of the learning apparatus in the learning phase which concerns on 2nd Embodiment. It is a figure which shows an example of the output model which concerns on 2nd Embodiment. It is a figure for demonstrating the derivation process of the evaluation value of the output model which concerns on 2nd Embodiment. It is a block diagram which shows an example of the functional structure of the learning apparatus in the operation phase which concerns on 2nd Embodiment. It is a flowchart which shows an example of the output model learning process which concerns on 2nd Embodiment. It is a flowchart which shows an example of the experimental condition output processing which concerns on 2nd Embodiment.

Hereinafter, examples of embodiments for carrying out the technique of the present disclosure will be described in detail with reference to the drawings.

[First Embodiment]
First, with reference to FIG. 1, the functional configuration of the learning device 10 in the learning phase according to the present embodiment will be described. As shown in FIG. 1, the learning device 10 includes a derivation unit 12 and a learning unit 14. Further, the storage unit 42 (see FIG. 10) of the learning device 10 stores the learning data 20, the plurality of output models 22, and the plurality of experimental models 24.

FIG. 2 shows an example of learning data 20. As shown in FIG. 2, the learning data 20 according to the present embodiment includes a combination of experimental conditions for producing a material and the performance value of the material as an experimental result when the experiment is performed under the experimental conditions. .. The experimental conditions are conditions for producing a material such as a semiconductor resist material, and include a main component composition, an amount of additives, and process conditions. In the example of FIG. 2, the main component composition indicates the ratio of the main components of the material, the amount of additive indicates the concentration of the additive, and the process condition indicates the temperature at which the material is produced.

Further, the performance value of the learning data 20 indicates the performance value of the material when the material is produced under the corresponding experimental conditions. The performance value according to the present embodiment is a scale indicating the quality of the material, and examples thereof include the degree of unevenness on the surface of the material and the degree of indicating whether a hole of a desired size has been formed. Further, in the present embodiment, the smaller the performance value, the better the quality of the material. Further, the learning data 20 of the present embodiment includes a combination of a plurality of different experimental conditions and performance values. The experimental conditions may include a plurality of the same ones.

FIG. 3 shows an example of the output model 22. As shown in FIG. 3, the output model 22 according to the present embodiment is a neural network including an input layer, a plurality of intermediate layers, and an output layer. A plurality of combinations of experimental conditions and performance values are input to the input layer of the output model 22. The output layer of the output model 22 outputs one experimental condition. FIG. 4 shows an example of the data structure of the experimental conditions output from the output layer of the output model 22. As shown in FIG. 4, the output layer of the output model 22 outputs experimental conditions including, for example, the main component composition, the amount of additives, and the process conditions.

In detail, the output model 22 is configured as shown in (1) to (3) below, for example.
(1) Number of nodes in the input layer: N × M
In addition, N represents the number of items of an experiment condition, and M represents the number of experiments.
(2) Structure of intermediate layer: The kernel has 3 × 3, the number of filters is 32, the stride is 2, and the activation function has 10 convolution layers of Relu.
(3) Number of nodes in the output layer: N × 1

Further, the plurality of output models 22 according to the present embodiment have different model creation conditions. Specifically, the plurality of output models 22 differ in model creation conditions because at least one of the number of layers in the intermediate layer, the number of nodes in each layer of the intermediate layer, and the initial value of the weight is different.

FIG. 5 shows an example of the experimental model 24. As shown in FIG. 5, the experimental model 24 according to the present embodiment is a neural network including an input layer, a plurality of intermediate layers, and an output layer. The experimental model 24 is a model for performing a virtual experiment, and one experimental condition is input to the input layer of the experimental model 24. The output layer of the experimental model 24 outputs the performance value of the experimental result corresponding to one experimental condition input to the input layer. FIG. 6 shows an example of the data structure of the performance value of the experimental result output from the output layer of the experimental model 24. The experimental model 24 may output a plurality of types of performance values. In this case, for example, the experimental model 24 outputs both the degree of unevenness on the surface of the material and the light sensitivity of the material as the performance value of the material.

In detail, the experimental model 24 is configured as shown in (4) to (6) below, for example.
(4) Number of nodes in the input layer: N × 1
In addition, N represents the number of items of the experimental condition.
(5) Structure of intermediate layer: The kernel has 3 × 3, the number of filters is 32, the stride is 2, and the activation function has 4 convolution layers of Relu.
(6) Number of nodes in the output layer: 1 x J
In addition, J represents the number of types of performance values.

Further, the plurality of experimental models 24 according to the present embodiment have different model creation conditions. Specifically, the plurality of experimental models 24 differ in model creation conditions because at least one of the number of layers in the intermediate layer, the number of nodes in each layer of the intermediate layer, and the initial value of the weight is different.

The derivation unit 12 inputs a plurality of combinations of the experimental conditions for producing the material and the performance values of the experimental results into the output model 22, and acquires the experimental conditions output from the output model 22. Specifically, the derivation unit 12 first inputs the combinations of all the experimental conditions and the performance values included in the learning data 20 into the output model 22, and acquires the experimental conditions output from the output model 22. The derivation unit 12 may input a combination of some plurality of experimental conditions and performance values included in the learning data 20 into the output model 22, or a plurality of experimental conditions different from the learning data 20. The combination of and the performance value may be input to the output model 22.

Further, the derivation unit 12 corrects the experimental conditions output from the output model 22 to experimental conditions that can be used in an actual experiment. In the present embodiment, the out-licensing unit 12 corrects the experimental conditions output from the output model 22 to the closest experimental conditions that satisfy the restrictions of the experimental device actually used. For example, due to the specifications of the experimental equipment, the temperature that can be set in the process conditions is in units of 5 ° C, and the temperature of the process conditions included in the experimental conditions output from the output model 22 is not in units of 5 ° C (for example, 92. In the case of 3 ° C.), the derivation unit 12 corrects the temperature output from the output model 22 to the nearest multiple of 5 (for example, 90 ° C.).

Next, the derivation unit 12 inputs the corrected experimental conditions to each experimental model 24, and acquires the performance values output from each experimental model 24, respectively.

Further, the derivation unit 12 adds a combination of the experimental conditions input to the corresponding experimental model 24 and the derived performance values to a plurality of combinations of the plurality of sets of experimental conditions and performance values input to the output model 22. Obtain multiple combinations of multiple sets of experimental conditions and performance values. Then, the derivation unit 12 again inputs the plurality of combinations of the obtained plurality of sets of experimental conditions and performance values to the output model 22, and the plurality of experimental conditions obtained by inputting the plurality of experimental conditions to the corresponding experimental models 24. input. As a result, the out-licensing unit 12 again obtains performance values corresponding to the experimental conditions input to the corresponding experimental models 24. The derivation unit 12 adds a combination of the experimental conditions input to the corresponding experimental model 24 and the obtained performance values to the plurality of combinations of the plurality of sets of experimental conditions and performance values input to the above output model 22. Then, the process of obtaining the performance value using the corresponding experimental model 24 is repeated a predetermined number of times (for example, 100 times).

Further, the derivation unit 12 performs the above processing for each output model 22. That is, the derivation unit 12 obtains a plurality of combinations of the experimental conditions output from the output model 22 for a predetermined number of times and the performance values corresponding to the experimental conditions for each output model 22.

The derivation unit 12 derives the evaluation value of the output model 22 for each output model 22 by using the obtained performance values for a predetermined number of times. In the present embodiment, as shown in FIG. 7 as an example, the out-licensing unit 12 is a virtual experiment until a performance value satisfying the target performance (in the present embodiment, a performance value equal to or less than the target value) is obtained. As the number of times (N shown in FIG. 7) decreases, the evaluation value of the output model 22 is derived as a better value. The vertical axis of FIG. 7 shows the performance value, and the horizontal axis shows the number of virtual experiments indicating how many times the performance value is the value obtained in the virtual experiment. In the example of FIG. 7, it is shown that the performance value satisfying the target performance was obtained for the first time in the Nth virtual experiment.

As shown in FIG. 8 as an example, the out-licensing unit 12 is a one-dot chain line with respect to the ratio of the performance values satisfying the target performance in the obtained performance values for a predetermined number of times (the number of all the performance values shown in FIG. 8). The higher the ratio of the number of performance values surrounded by the rectangle, the better the evaluation value of the output model 22 may be derived. In addition, "good" in FIG. 8 means that the target performance is satisfied. Further, the derivation unit 12 may derive the evaluation value of the output model 22 as a better value as each performance value is closer to the target value.

The out-licensing unit 12 may correct the evaluation value to be low when the output model 22 outputs experimental conditions that do not satisfy the predetermined rules. Examples of the predetermined rules include rules according to the user's rule of thumb, such as not mixing material A and material B, or mixing five or more kinds of materials.

The learning unit 14 trains the experimental model 24 by using the error backpropagation method as an example of machine learning. Specifically, the learning unit 14 inputs the experimental conditions included in the learning data 20 into the experimental model 24, and acquires the performance value output from the experimental model 24. Then, the learning unit 14 trains the experimental model 24 so that the difference between the acquired performance value and the performance value corresponding to the experimental conditions included in the learning data 20 is minimized. The learning unit 14 performs the process of training the experimental model 24 by using a combination of all the experimental conditions and the performance values included in the learning data 20. The learning unit 14 may train the experimental model 24 by using a plurality of combinations of some experimental conditions and performance values included in the learning data 20. Further, the data input to each experimental model 24 when the learning unit 14 trains each experimental model 24 may be the same data or different data among the experimental models 24.

Further, the learning unit 14 trains each output model 22 by machine learning using a genetic algorithm as an example of the optimization algorithm, using the evaluation values derived by the derivation unit 12 for each output model 22. Parameters such as an individual selection method (for example, roulette selection), a crossover method (for example, two-point crossover), and a mutation probability used in this genetic algorithm are preset by the user.

Specifically, for example, the learning unit 14 generates a new output model 22 by mating the two output models 22 having the highest evaluation among the output models 22. This mating is performed, for example, with the input layer of one output model 22 and the middle layer of the middle layer on the input layer side, and the middle layer of the other output model 22 on the output layer side. This is done by combining the output layers. The mating method is not limited to this example. For example, the upper half of the input layer, the intermediate layer, and the output layer shown in FIG. 3 of one output model 22, and the lower half of the input layer, the intermediate layer, and the output layer shown in FIG. 3 of the other output model 22. Mating may be carried out by binding. Further, in the present embodiment, the learning unit 14 generates the next-generation output model 22 by a genetic algorithm so that the number of output models 22 does not change between generations. That is, the output model 22 is learned by updating the weight value of the output model 22 by using the genetic algorithm. Further, by learning the output model 22, the evaluation value derived by the out-licensing unit 12 is reflected.

The derivation process of the evaluation value of each output model 22 by the derivation unit 12 and the learning process of the output model 22 group by the learning unit 14 are performed for a predetermined number of generations (for example, 10,000 generations). Then, the learning unit 14 stores in the storage unit 42 one output model 22 having the best evaluation indicated by the evaluation value in the final generation as an output model 22A used in the operation phase described later. The derivation process of the evaluation value of each output model 22 by the derivation unit 12 and the learning process of the output model 22 group by the learning unit 14 may be performed until the evaluation values converge.

Next, with reference to FIG. 9, the functional configuration of the learning device 10 in the operation phase according to the present embodiment will be described. As shown in FIG. 9, the learning device 10 includes a reception unit 30 and an output unit 32. Further, the storage unit 42 of the learning device 10 stores the output model 22A obtained in the learning phase described above.

The reception unit 30 receives a plurality of combinations of the experimental conditions for generating the material input by the user via the input unit 44 (see FIG. 10) and the performance value of the material of the experimental result.

The output unit 32 inputs a plurality of combinations of the experimental conditions and the performance values received by the reception unit 30 into the output model 22A, and acquires the experimental conditions output from the output model 22A. Further, the output unit 32 corrects the experimental conditions output from the output model 22A to the experimental conditions that can be used in the actual experiment, similarly to the derivation unit 12 in the learning phase. Then, the output unit 32 outputs the corrected experimental conditions to the display unit 43 (see FIG. 10). The user visually observes the experimental conditions displayed on the display unit 43, and if necessary, conducts an experiment under the experimental conditions. The output unit 32 may output (store) the experimental conditions obtained by correction to the storage unit 42.

Next, the hardware configuration of the learning device 10 will be described with reference to FIG. The learning device 10 is realized by the computer shown in FIG. As shown in FIG. 10, the learning device 10 includes a CPU (Central Processing Unit) 40, a memory 41 as a temporary storage area, and a non-volatile storage unit 42. Further, the learning device 10 includes a display unit 43 such as a liquid crystal display and an input unit 44 such as a keyboard and a mouse. The CPU 40, the memory 41, the storage unit 42, the display unit 43, and the input unit 44 are connected via the bus 45.

The storage unit 42 is realized by an HDD (Hard Disk Drive), an SSD (Solid State Drive), a flash memory, or the like. The learning program 50 is stored in the storage unit 42 as a storage medium. The CPU 40 reads the learning program 50 from the storage unit 42, expands the read learning program 50 into the memory 41, and then executes the learning program 50. When the CPU 40 executes the learning program 50, it functions as a derivation unit 12, a learning unit 14, a reception unit 30, and an output unit 32.

Next, the operation of the learning device 10 according to the present embodiment will be described with reference to FIGS. 11 to 13. When the learning device 10 executes the learning program 50, the experimental model learning process shown in FIG. 11, the output model learning process shown in FIG. 12, and the experimental condition output process shown in FIG. 13 are executed. The experimental model learning process shown in FIG. 11 is executed, for example, when an execution instruction for the experimental model learning process is input by the user via the input unit 44 in the learning phase. Further, the output model learning process shown in FIG. 12 is executed, for example, when an execution instruction for the output model learning process is input by the user via the input unit 44 in the learning phase. Further, the experimental condition output process shown in FIG. 13 is executed, for example, when an execution instruction of the experimental condition output process is input by the user via the input unit 44 in the operation phase.

In step S10 of FIG. 11, the learning unit 14 reads the learning data 20 from the storage unit 42. In step S12, the learning unit 14 generates a plurality of experimental models 24 having different model creation conditions. In step S14, the learning unit 14 selects one experimental model 24 to be trained from the plurality of experimental models 24 generated by the process of step S12. When the process of step S14 is repeatedly executed, the learning unit 14 selects the experimental model 24 that has not been selected by then.

In step S16, as described above, the learning unit 14 trains the experimental model 24 selected by the process of step S14 by the error backpropagation method using the learning data 20 read by the process of step S10. .. In step S18, the learning unit 14 stores the experimental model 24 learned by the process of step S16 in the storage unit 42. In step S20, the learning unit 14 determines whether or not the processes of steps S14 to S18 have been completed for all the experimental models 24 generated by the process of step S12. If this determination is a negative determination, the process returns to step S14, and if the determination is affirmative, the experimental model learning process ends.

In step S30 of FIG. 12, the learning unit 14 generates a plurality of output models 22 having different model creation conditions. In step S32, the derivation unit 12 inputs a plurality of combinations of the experimental conditions for generating the material and the performance values of the experimental results into each output model 22, and acquires the experimental conditions output from each output model 22. To do.

The plurality of combinations of the experimental conditions and the performance values are used when step S32 is executed for the first time of each generation of the output model 22 (that is, when step S32 is executed for the first time, or in step S46 described later). When step S32 is executed for the first time after the determination is a negative determination), all the experimental conditions and performance values included in the learning data 20 are combined. Further, the plurality of combinations of the experimental conditions and the performance values are obtained when step S32 is executed after the second time of each generation of the output model 22 (that is, after the determination in step S40 is a negative determination, step S32 is performed. (When executed), the combination of the experimental condition and the performance value is added to the plurality of combinations of the experimental condition and the performance value input to the output model 22 in the previous step S32 in the step S38 described later. It becomes.

In step S34, as described above, the derivation unit 12 corrects the experimental conditions output from each output model 22 by the process of step S32 to the experimental conditions that can be used in the actual experiment. In step S36, the derivation unit 12 inputs each experimental condition obtained by being corrected by the process of step S34 into each experimental model 24, and acquires the performance value output from each experimental model 24, respectively. Further, the derivation unit 12 holds a plurality of combinations of the experimental conditions and the performance values for each output model 22 corresponding to the experimental conditions output from the output model 22.

In step S38, the derivation unit 12 adds the following combinations of experimental conditions and performance values to a plurality of combinations of experimental conditions and performance values input to the output model 22 by the processing of step S32 this time (immediately before). to add. That is, in this case, the derivation unit 12 adds a combination of the experimental conditions input to the experimental model 24 and the performance value by the process of step S36 this time. The plurality of combinations of the experimental conditions and the performance values obtained by performing this addition are used in the next step S32 to be executed after the determination in the step S40 described later becomes a negative determination.

In step S40, the out-licensing unit 12 determines whether or not the processes of steps S32 to S38 have been repeated a predetermined number of times (for example, 100 times). If this determination is a negative determination, the process returns to step S32, and if the determination is affirmative, the process proceeds to step S42.

In step S42, as described above, the derivation unit 12 derives the evaluation value of the output model 22 for each output model 22 by using the performance values for the predetermined number of times obtained by the iterative processing of steps S32 to S38. To do. In step S44, as described above, the learning unit 14 generates the next-generation output model 22 by the genetic algorithm using the evaluation values derived by the processing of step S42 for each output model 22. This next-generation output model 22 is used in step S32, which is executed next after the determination in step S46, which will be described later, becomes a negative determination.

In step S46, the learning unit 14 determines whether or not the number of generations of the output model 22 has reached a predetermined number of generations (for example, 10,000 generations). If this determination is a negative determination, the process returns to step S32, and if the determination is affirmative, the process proceeds to step S48. In step S48, as described above, the learning unit 14 stores in the storage unit 42 as the output model 22A one output model 22 having the best evaluation value indicated by the evaluation value in the final generation. When the process of step S48 is completed, the output model learning process is completed.

In step S50 of FIG. 13, the reception unit 30 receives a plurality of combinations of the experimental conditions for generating the material input by the user via the input unit 44 and the performance value of the material of the experimental result. In step S52, the output unit 32 reads the output model 22A from the storage unit 42. In step S54, the output unit 32 inputs a plurality of combinations of the experimental conditions and the performance values received by the process of step S50 into the output model 22A read by the process of step S52, and outputs from the output model 22A. Obtain the experimental conditions.

In step S56, as described above, the output unit 32 corrects the experimental conditions output from the output model 22A by the process of step S54 to the experimental conditions that can be used in the actual experiment. In step S58, the output unit 32 outputs the experimental conditions corrected by the process of step S56 to the display unit 43 as described above. By the process of step S58, the experimental conditions are displayed on the display unit 43. When the process of step S58 is completed, the experimental condition output process is completed.

As described above, according to the present embodiment, the experiment obtained by the output model 22 in which a plurality of combinations of the experimental conditions for producing the material and the performance values of the experimental results are input and the experimental conditions are output. The conditions are input to the experimental model 24 for performing a virtual experiment. Further, the evaluation value of the output model 22 is derived using the performance value of the experimental result obtained by this input. Then, the output model 22 is trained by machine learning using the derived evaluation values of the output model 22. Therefore, by using the output model 22 learned in this way, it is possible to search for appropriate experimental conditions for the material.

[Second Embodiment]
A second embodiment of the disclosed technique will be described. The same components as those in the first embodiment are designated by the same reference numerals, and the description thereof will be omitted. First, with reference to FIG. 14, the functional configuration of the learning device 10 in the learning phase according to the present embodiment will be described. As shown in FIG. 14, the learning device 10 includes a derivation unit 12A, a learning unit 14A, and a generation unit 16. The storage unit 42 stores the learning data 20, the plurality of output models 22B, and the plurality of experimental models 24.

FIG. 15 shows an example of the output model 22B. As shown in FIG. 15, the output model 22B according to the present embodiment is a neural network including an input layer, a plurality of intermediate layers, and an output layer. In the input layer of the output model 22B, a plurality of combinations of experimental conditions for producing a material and performance values of experimental results, and a candidate for one experimental condition are input. The output layer of the output model 22B outputs a Q value as an example of action value in reinforcement learning. That is, the learning device 10 according to the present embodiment sets the output model 22B according to Q-learning as an example of reinforcement learning, with a plurality of combinations of experimental conditions and performance values as the current state s and candidates for experimental conditions as action a. Let them learn. The plurality of output models 22B according to the present embodiment also have different model creation conditions, as with the output model 22 according to the first embodiment.

The generation unit 16 generates candidates for a plurality of different experimental conditions. In the present embodiment, the generation unit 16 generates candidates for experimental conditions that satisfy predetermined rules and can be used in an actual experiment. Since this rule and the experimental conditions that can be used in the actual experiment are the same as those in the first embodiment, the description thereof will be omitted. Specifically, the generation unit 16 randomly generates candidates for experimental conditions that satisfy predetermined rules and can be used in actual experiments each time a plurality of candidates for different experimental conditions are generated.

The derivation unit 12A derives a value (hereinafter, referred to as “reward value”) used as a reward when the learning unit 14A, which will be described later, trains each output model 22B according to Q-learning. Hereinafter, the details of the process in which the out-licensing unit 12A derives the reward value will be described.

First, the derivation unit 12A inputs a plurality of combinations of the experimental conditions for generating the material and the performance values of the experimental results, and the candidates for the experimental conditions generated by the generation unit 16 into the output model 22B, and the output model 22B. The Q value output from is acquired. In detail, as shown in FIG. 16, the derivation unit 12A is a plurality of combinations of experimental conditions for producing a material and performance values of experimental results, and candidates for a plurality of experimental conditions generated by the generation unit 16. Any one of the above is individually input to the output model 22B for all the generated experimental condition candidates. That is, the derivation unit 12A acquires the Q value output from the output model 22B corresponding to each of the candidates for the plurality of experimental conditions generated by the generation unit 16.

Next, the derivation unit 12A inputs to the experimental model 24 a candidate for the experimental condition corresponding to any of the acquired Q values equal to or higher than the predetermined value. In the present embodiment, the derivation unit 12A inputs the candidate of the experimental condition corresponding to the maximum Q value among the acquired plurality of Q values into each experimental model 24, and inputs the performance value output from each experimental model 24. get. Further, the out-licensing unit 12A holds a plurality of combinations of experimental conditions and performance values of experimental results, respectively, as in the out-licensing unit 12 according to the first embodiment.

Further, the derivation unit 12A adds the combination of the experimental condition input to the experimental model 24 and the derived performance value to the plurality of combinations of the experimental condition and the performance value input to the output model 22B, and the experimental condition and the performance value. Get multiple combinations of. In addition, the out-licensing unit 12A again used a plurality of combinations of the obtained experimental conditions and performance values, and any one of the candidates for the plurality of experimental conditions generated by the generation unit 16 in all the experiments generated. The condition candidates are individually input to the output model 22B. Then, the derivation unit 12A again uses the Q value output from the output model 22B and the experimental model 24 corresponding to each of the candidates for the plurality of experimental conditions, as in the above-described processing, to be candidates for the experimental conditions. Acquire the performance value corresponding to. The derivation unit 12A repeats the process for acquiring the performance value corresponding to the candidate of the experimental condition a predetermined number of times (for example, 100 times).

Further, the out-licensing unit 12A performs the above processing for each output model 22B. That is, the out-licensing unit 12A acquires the performance values for the predetermined number of times for each of the output models 22B. Similar to the derivation unit 12 according to the first embodiment, the derivation unit 12A derives the evaluation value of the output model 22B for each output model 22B using the obtained performance values for a predetermined number of times (see FIG. 7). ..

Further, the derivation unit 12A derives the reward value so that the higher the output model 22B, the higher the reward is obtained. For example, the derivation unit 12A derives the reward values of the upper three output models 22B as "1" in descending order of the evaluation value, derives the reward values of the lower three output models 22B as "-1", and derives the reward values of the other three output models 22B. The reward value of the output model 22B is derived as "0".

The learning unit 14A trains each output model 22B by using the reward value derived by the derivation unit 12A as the reward r in Q-learning.

The process for deriving the reward value of each output model 22B by the derivation unit 12A and the learning process of each output model 22B by the learning unit 14A are performed a predetermined number of times (for example, 10,000 times). Then, in the final round, the learning unit 14A stores one output model 22B having the best evaluation indicated by the evaluation value in the storage unit 42 as an output model 22C used in the operation phase described later. The process for deriving the reward value of each output model 22B by the derivation unit 12A and the learning process of each output model 22B by the learning unit 14A may be performed until the evaluation values converge.

Further, the learning unit 14A trains the experimental model 24 according to the error back propagation method using the learning data 20 as in the learning unit 14 according to the first embodiment.

Next, with reference to FIG. 17, the functional configuration of the learning device 10 in the operation phase according to the present embodiment will be described. As shown in FIG. 17, the learning device 10 includes a generation unit 16, a reception unit 30, and an output unit 32A. Further, the storage unit 42 of the learning device 10 stores the output model 22C obtained in the learning phase described above.

The output unit 32A is used to generate all the experiments in which any one of a plurality of combinations of the experimental conditions and the performance values received by the reception unit 30 and a plurality of experimental condition candidates generated by the generation unit 16 is generated. The candidate conditions are individually input to the output model 22C. The output unit 32A acquires the Q value output from the output model 22C corresponding to each of the inputs. Then, the output unit 32A outputs the candidate of the experimental condition corresponding to the maximum Q value among the acquired plurality of Q values to the display unit 43 as a candidate of the experimental condition to be the next experiment target. It should be noted that the output unit 32A selects candidates for experimental conditions corresponding to any of the acquired Q values of the predetermined value or more (for example, the Q value of the predetermined value or more and the second largest Q value). Next, it may be output to the display unit 43 as a candidate for the experimental conditions to be tested. Further, the output unit 32A outputs (stores) a candidate for the experimental condition corresponding to the maximum Q value among the acquired plurality of Q values to the storage unit 42 as a candidate for the experimental condition to be the next experiment target. May be good.

Since the hardware configuration of the learning device 10 according to the present embodiment is the same as that of the learning device 10 according to the first embodiment (see FIG. 10), the description thereof will be omitted. When the CPU 40 executes the learning program 50, it functions as a derivation unit 12A, a learning unit 14A, a generation unit 16, a reception unit 30, and an output unit 32A.

Next, the operation of the learning device 10 according to the present embodiment will be described with reference to FIGS. 18 and 19. Since the experimental model learning process is the same as that of the first embodiment (see FIG. 11), the description thereof will be omitted. The output model learning process shown in FIG. 18 is executed, for example, when an execution instruction for the output model learning process is input by the user via the input unit 44 in the learning phase. Further, the experimental condition output process shown in FIG. 19 is executed, for example, when an execution instruction of the experimental condition output process is input by the user via the input unit 44 in the operation phase.

In step S60 of FIG. 18, the learning unit 14 generates a plurality of output models 22B having different model creation conditions. The following processes of steps S62 to S70 are similarly executed for each output model 22B generated by the process of step S60. In step S62, the generation unit 16 generates a plurality of candidates for different experimental conditions as described above.

In step S64, as described above, the derivation unit 12A outputs a plurality of combinations of the experimental conditions for producing the material and the performance values of the experimental results, and the candidates of the experimental conditions generated by the process of step S62 as an output model. Input to 22B and acquire the Q value output from the output model 22B.

It should be noted that the plurality of combinations of the experimental conditions and the performance values are used when step S64 is executed for the first time in the learning process of the output model 22B (that is, when step S64 is executed for the first time, or in step S78 described later). When step S62 is executed for the first time after the determination is a negative determination), all the experimental conditions and performance values included in the learning data 20 are combined. Further, the plurality of combinations of the experimental conditions and the performance values are obtained when the step S64 is executed after the second time in the learning process of the output model 22B (that is, after the determination in the step S70 is a negative determination, the step S64 is performed. When executed), the combination of the experimental condition and the performance value is added to the plurality of combinations of the experimental condition and the performance value input to the output model 22B in the previous step S64 in the step S68 described later. It becomes.

In step S66, the derivation unit 12A inputs a candidate for the experimental condition corresponding to the maximum Q value among the plurality of Q values acquired by the process of step S64 into each experimental model 24, and outputs the candidate from each experimental model 24. Get the performance value. Further, the derivation unit 12A holds a plurality of combinations of the experimental conditions and the performance values of the experimental results, respectively, corresponding to the candidates of the experimental conditions corresponding to the maximum Q value.

In step S68, the derivation unit 12A adds the following combinations of experimental conditions and performance values to a plurality of combinations of experimental conditions and performance values input to the output model 22B by the processing of step S64 this time (immediately before). to add. That is, in this case, the derivation unit 12A adds a combination of the experimental conditions input to the experimental model 24 by the process of step S66 this time and the acquired performance values. The plurality of combinations of the experimental conditions and the performance values obtained by performing this addition are used in the next step S64 to be executed after the determination in the step S70 described later becomes a negative determination.

In step S70, the out-licensing unit 12A determines whether or not the processes of steps S62 to S68 have been repeated a predetermined number of times (for example, 100 times). If this determination is a negative determination, the process returns to step S62, and if the determination is affirmative, the process proceeds to step S72.

In step S72, as described above, the derivation unit 12A derives the evaluation value of the output model 22B for each output model 22B by using the performance values for the predetermined number of times obtained by the iterative processing of steps S62 to S68. To do. In step S74, as described above, the derivation unit 12A derives the reward value so that the higher the output model 22B derived by the process of step S72, the higher the reward.

In step S76, the learning unit 14A trains each output model 22B by using the reward value derived by the process of step S74 as the reward r in Q-learning. In step S78, the learning unit 14 determines whether or not the processes of steps S62 to S76 have been repeated a predetermined number of times (for example, 10,000 times). If this determination is a negative determination, the process returns to step S62, and if the determination is affirmative, the process proceeds to step S80. In step S80, as described above, the learning unit 14A stores in the storage unit 42 as the output model 22C one output model 22B having the best evaluation indicated by the evaluation value derived by the processing of the last executed step S72. To do. When the process of step S80 is completed, the output model learning process is completed.

In step S90 of FIG. 19, the reception unit 30 receives a plurality of combinations of the experimental conditions for generating the material input by the user via the input unit 44 and the performance value of the material of the experimental result. In step S92, the output unit 32A reads the output model 22C from the storage unit 42. In step S94, the generation unit 16 generates a plurality of candidates for different experimental conditions, as described above.

In step S96, the output unit 32A selects any one of a plurality of combinations of the experimental conditions and the performance values received by the process of step S90 and a plurality of candidates for the experimental conditions generated by the process of step S92. All the generated experimental condition candidates are individually input to the output model 22C. The output unit 32A acquires the Q value output from the output model 22C corresponding to each of the inputs.

In step S98, the output unit 32A displays a candidate for the experimental condition corresponding to the maximum Q value among the plurality of Q values acquired by the process of step S96 as a candidate for the experimental condition to be the next experiment target. Output to 43. When the process of step S98 is completed, the experimental condition output process is completed.

As described above, according to the present embodiment, an output model in which a plurality of combinations of experimental conditions for producing a material and performance values of experimental results and candidates for experimental conditions are input and a Q value is output. The candidate for the experimental condition that maximizes the Q value obtained by 22B is input to the experimental model 24. Further, the evaluation value of the output model 22B is derived using the performance value of the experimental result obtained by this input, and the reward given to the output model 22B is derived according to the derived evaluation value. Then, the output model 22B is trained by Q-learning using the derived reward. Therefore, by using the output model 22B learned in this way, it is possible to search for appropriate experimental conditions for the material.

In each of the above embodiments, the case of deriving the experimental conditions for producing the material has been described, but the present invention is not limited to this. For example, it may be in the form of deriving experimental conditions for producing a drug.

Further, in each of the above embodiments, the case where the trained model obtained by machine learning is applied as the experimental model 24 has been described, but the model is not limited to this as long as it is a model capable of virtual experiments. For example, as the experimental model 24, an arbitrary function may be applied in which one experimental condition is input and the performance value of the experimental result corresponding to the input experimental condition is output. Even when such a model is applied, it is optimized by learning the output models 22 and 22B. Further, for example, the experimental model 24 may be a simulator that simulates an experiment.

Further, in the operation phase of the second embodiment, the output unit 32A sequentially inputs candidates for a plurality of experimental conditions into the output model 22C a plurality of times, so that the cumulative Q value obtained is the maximum experimental condition. Can be output as a candidate for the experimental conditions to be tested next. In this case, the output unit 32A first obtains Q values corresponding to each of the candidates for the first plurality of experimental conditions from the output model 22C, as in the second embodiment. Next, the output unit 32A uses, for example, a combination of a plurality of combinations of the experimental conditions and performance values input to the output model 22C for the first time, and a combination of the experimental condition candidates and performance values input to the output model 22C for the first time. To add. This performance value may be estimated by, for example, a known method such as SVM (Support Vector Machine). Then, as in the first time, the output unit 32A inputs a plurality of combinations of the additionally obtained experimental conditions and the performance values, and each of the candidates for the second plurality of experimental conditions into the output model 22C. From the output model 22C, Q values corresponding to each of the candidates for the second plurality of experimental conditions are obtained. In this case, the output unit 32A outputs the candidate of the experimental condition in which the cumulative value of the first Q value and the second Q value is the maximum as the candidate of the experimental condition to be the next experiment target. Although the case where the cumulative value of the Q value is used twice is described here, the case where the cumulative value of the Q value three times or more is used is also possible.

Further, various processors other than the CPU may execute various processes executed by the CPU executing software (program) in each of the above embodiments. In this case, the processor is a PLD (Programmable Logic Device) whose circuit configuration can be changed after manufacturing an FPGA (Field-Programmable Gate Array) or the like, and an ASIC (Application Specific Integrated Circuit) or the like in order to execute a specific process. An example is a dedicated electric circuit or the like, which is a processor having a circuit configuration designed exclusively for the purpose. Further, the above-mentioned various processes may be executed by one of these various processors, or a combination of two or more processors of the same type or different types (for example, a plurality of FPGAs and a combination of a CPU and an FPGA). Etc.). Further, the hardware structure of these various processors is, more specifically, an electric circuit in which circuit elements such as semiconductor elements are combined.

Further, in each of the above embodiments, the mode in which the learning program 50 is stored (installed) in the storage unit 42 in advance has been described, but the present invention is not limited to this. The learning program 50 is provided in a form recorded on a non-temporary recording medium such as a CD-ROM (Compact Disk Read Only Memory), a DVD-ROM (Digital Versatile Disk Read Only Memory), and a USB (Universal Serial Bus) memory. May be done. Further, the learning program 50 may be downloaded from an external device via a network.

This application claims the priority of Japanese Application No. 2018-076001 filed on April 11, 2018, the full text of which is incorporated herein by reference.

Claims (16)

The experimental conditions output by inputting the plurality of combinations to the output model in which a plurality of combinations of the experimental conditions for producing a material or a drug and the performance values of the experimental results are input and the experimental conditions are output are input. , A derivation unit that derives the evaluation value of the output model using the performance value of the experimental result obtained by inputting it to the experimental model for performing a virtual experiment.
A learning unit that trains the output model by machine learning that reflects the evaluation value derived by the derivation unit.
A learning device equipped with. The higher the ratio of the values satisfying the target performance in the plurality of performance values, the better the evaluation value, or the number of virtual experiments until a performance value satisfying the target performance is obtained is small. The learning device according to claim 1, wherein the learning device has a moderate value, or the performance value is closer to the target performance. The learning device according to claim 1 or 2, wherein the derivation unit corrects the evaluation value to be low when experimental conditions that do not satisfy a predetermined rule are output from the output model. The learning device according to any one of claims 1 to 3, wherein the derivation unit corrects the experimental conditions output from the output model to experimental conditions that can be used in an actual experiment. The learning device according to any one of claims 1 to 4, wherein the output model is a model learned by using a genetic algorithm. A plurality of combinations and a plurality of combinations of the above-mentioned multiple combinations and a plurality of combinations of the experimental conditions for generating a material or a drug and the performance values of the experimental results, and an output model in which the candidates of the experimental conditions are input and the action value in reinforcement learning is output. Of the plurality of action values output by inputting each of the different candidates for the experimental condition, the candidate for the experimental condition corresponding to the action value equal to or higher than the predetermined value is used as an experimental model for performing a virtual experiment. A learning device provided with a learning unit that trains the output model using a value derived based on the performance value of the experimental result obtained by inputting as a reward. The higher the ratio of the values satisfying the target performance in the plurality of the performance values, the better the reward, or the smaller the number of virtual experiments until the performance value satisfying the target performance is obtained. The learning device according to claim 6, wherein the better the value is, or the closer the performance value is to the target performance, the better the value. The reinforcement learning is Q-learning,
The learning device according to claim 6 or 7, wherein the action value is a Q value. When the output model learned by the learning unit is used, the cumulative action value output by sequentially inputting a plurality of candidates for the experimental conditions into the output model a plurality of times is maximized. The learning device according to any one of claims 6 to 8, further comprising an output unit that outputs a candidate as a candidate for an experimental condition to be tested next. The learning device according to any one of claims 1 to 9, wherein the experimental model is a model obtained by machine learning. There are multiple experimental models,
The learning device according to any one of claims 1 to 10, wherein the plurality of experimental models have different model creation conditions. There are multiple output models,
The learning device according to any one of claims 1 to 11, wherein the plurality of output models have different model creation conditions. The experimental conditions output by inputting the plurality of combinations to the output model in which a plurality of combinations of the experimental conditions for producing a material or a drug and the performance values of the experimental results are input and the experimental conditions are output are input. , The evaluation value of the output model is derived using the performance value of the experimental result obtained by inputting to the experimental model for performing the virtual experiment.
A learning method in which a computer executes a process of learning the output model by machine learning that reflects the derived evaluation value. The experimental conditions output by inputting the plurality of combinations to the output model in which a plurality of combinations of the experimental conditions for producing a material or a drug and the performance values of the experimental results are input and the experimental conditions are output are input. , The evaluation value of the output model is derived using the performance value of the experimental result obtained by inputting to the experimental model for performing the virtual experiment.
A learning program for causing a computer to execute a process of learning the output model by machine learning that reflects the derived evaluation value. A plurality of combinations and a plurality of the above-mentioned combinations and a plurality of combinations of the experimental conditions for producing a material or a drug and the performance values of the experimental results, and an output model in which the candidates of the experimental conditions are input and the action value in the reinforcement learning is output. Of the plurality of action values output by inputting each of the different candidates for the experimental condition, the candidate for the experimental condition corresponding to the action value equal to or higher than the predetermined value is used as an experimental model for performing a virtual experiment. A learning method in which a computer executes a process of training the output model using a value derived based on the performance value of the experimental result obtained by input as a reward. A plurality of combinations and a plurality of the above-mentioned combinations and a plurality of combinations of the experimental conditions for producing a material or a drug and the performance values of the experimental results, and an output model in which the candidates of the experimental conditions are input and the action value in the reinforcement learning is output. Of the plurality of action values output by inputting each of the different candidates for the experimental condition, the candidate for the experimental condition corresponding to the action value equal to or higher than the predetermined value is used as an experimental model for performing a virtual experiment. A learning program for causing a computer to execute a process of training the output model using a value derived based on the performance value of the experimental result obtained by inputting as a reward.