JP2021082014A - Estimation device, training device, estimation method, training method, program, and non-transitory computer readable medium - Google Patents

Abstract

To make adaptation from a simulation environment to an actual environment.SOLUTION: An estimation device comprises: one or more memories; and one or more processors. The memory stores a trained model which can estimate a dynamics parameter. The one or more processors encode a state, encodes an environmental dynamics parameter on the basis of a dynamics parameter estimated by the trained model, and estimates an action on the basis of the encoded state and the encoded dynamics parameter.SELECTED DRAWING: Figure 1

Description

The present disclosure relates to estimators, training devices, estimation methods, training methods, programs and non-transitory computer-readable media.

Conventionally, attempts have been widely made today to optimize the control for automatically executing a specific task by a control device or the like by machine learning, particularly reinforcement learning, or to optimize the method of allocating tasks.

Bernd Waschneck, et. Al., "Optimization of global production scheduling with deep reinforcement learning," Procedia CIRP, pp. 1264-1269, Vol. 72, 2018.

Therefore, in the present disclosure, an estimation device that can be appropriately estimated and a training device that easily trains such an estimation device are realized.

The estimator according to an embodiment comprises one or more memories and one or more processors, the memory storing a trained model capable of estimating dynamics parameters, the one or more processors. , The state is encoded, the dynamics parameters of the environment are encoded based on the dynamics parameters estimated by the trained model, and the behavior is estimated based on the encoded state and the encoded dynamics parameters.

The block diagram of the estimation apparatus which concerns on one Embodiment. The flowchart which shows the processing of the estimation apparatus which concerns on one Embodiment. The block diagram of the training apparatus which concerns on one Embodiment. The flowchart which shows the processing of the training apparatus which concerns on one Embodiment. The flowchart which shows the processing of the training apparatus which concerns on one Embodiment. The figure which shows the result which concerns on one Embodiment. The figure which shows the result which concerns on one Embodiment. The figure which shows the result which concerns on one Embodiment. The figure which shows the hardware implementation example which concerns on one Embodiment.

Hereinafter, embodiments of the present invention will be described with reference to the drawings. The drawings and the description of the embodiments are shown as an example, and do not limit the present invention. Hereinafter, the model (trained model) provided for each configuration may be a neural network such as MLP (Multi-Layer Perceptron) or CNN (Convolutional Neural Network), but is not limited thereto and is appropriately used. Any model may be used as long as the output data can be acquired for the input data. Further, a model other than the neural network model may be used.

In this embodiment, an estimator that estimates the dynamics in the simulation environment and an estimator that estimates the behavior in the simulation environment according to the dynamics are used. As a result, we are trying to make it possible to better apply reinforcement learning, which requires a large number of trials for learning, in a real environment. For example, trying reinforcement learning in a real environment, which requires a considerable number of trials for training, takes too much time in reality, and it is less safe to select an action with a randomly initialized strategy. There is a problem. In this embodiment, training and estimation are performed in order to solve this problem.

(Estimator)
FIG. 1 is a block diagram showing the functions of the estimation device according to the embodiment. The estimation device 1 in the present disclosure includes a first estimation unit 10 and a second estimation unit 12. The first estimation unit 10 estimates the behavior based on the mechanical conditions. The second estimation unit 12 estimates the mechanical situation.

The first estimation unit 10 includes a first encoder 100, a second encoder 102, and a decoder 104. In addition, the training device 2 may include an interface for inputting / outputting data and the like, and a storage unit for storing data, parameters, and the like.

For example, the first estimation unit 10 inputs the state o and the dynamics parameter η to an object, and outputs the action a. These quantities are represented, for example, as vectors, but are not limited to, and may be represented by tensors having arbitrary dimensions.

The state refers to, for example, the observed state of an object and a controlled object in a certain environment. This state is, for example, information acquired by various sensors, and may include information such as a state of a controlled object and a state of an object on which the controlled object exerts some action.

The dynamics is, for example, the dynamics in the environment of the space where the controlled object and the object exist, and the parameters that characterize the dynamics are described as the dynamics parameters. This dynamics parameter is, for example, the friction coefficient between the object and the region where the object is placed, what kind of inertial system the object is placed in, or under what circumstances the sensor detects the object and the controlled object. It may include information such as whether or not it is present. In addition, the dynamics parameters may include parameters indicating how the environment affects the controlled object and the object.

The action indicates, for example, a desirable action taken by the controlled object or a desirable control given to the controlled object in the input state in the space described by the input dynamics parameter. That is, the behavior indicates, for example, the behavior conditioned on the dynamics parameter.

Further, the policy is, for example, a function that outputs a desired action taken by the controlled object or a desired control given to the controlled object in the input state. Strategies are generally expressed in the form of conditional probability distributions for behavior or control, conditioned on state. Here, for example, the policy is expressed not only as a state but also as a conditional probability distribution conditioned by dynamics parameters.

The first encoder 100 takes the state o _t ^(e) at the environment e and the time t as an input, encodes this state, and outputs the feature amount.

The second encoder 102 is input with dynamics parameters eta _e in the environment e, and encodes the dynamics parameters, and outputs the feature quantity.

If the conversion of the first encoder 100 _{is expressed as f φ} () and the conversion of the second encoder 102 _{is expressed as M ζ} (), the latent state Z _t = [f _φ (o _t ^(e) )) at the environment e and time t. It can be expressed as M _ζ (η _e)]. As described above, the first encoder 100 and the second encoder 102 encode the state and dynamics parameters, respectively.

By encoding the state separately from the dynamics parameter, it distinguishes between the state that changes from moment to moment and the dynamics parameter that changes only slowly even if it changes, compared to the case where the state and the dynamics parameter are encoded together. Therefore, in the same environment where the dynamics parameters do not change, even if the lighting or camera changes, the encoding of the same dynamics parameters can be reused, and it can be handled efficiently. In addition, the state changes from moment to moment based on the data sensed for each unit time, for example, but the dynamics parameter changes less than the state, for example, does not change for each unit time, and does not change for a certain period of time. Since it may be virtually fixed inside, the state and the dynamics parameter are separated and encoded in this way.

That is, the state and dynamics parameters do not have to be re-encoded at the same time. For example, the encoded value may continue to use the same value for a while. The frequency of re-encoding may vary depending on the state and dynamics. If the encoded value is present, for example, the state encoding may be done in a different span than the dynamics parameter encoding, eg, a shorter span. Therefore, in this embodiment, the first encoder 100 and the second encoder 102 are used to encode the state and dynamics parameters.

The decoder 104 takes the latent state Z _t as an input and outputs the _{action a t} ^(e). The behavior is output, for example, as a distribution of predicted behaviors or controls. As an example, the decoder 104 probabilistically outputs the _{action a t} ^(e) from the conditioned probability distribution with the _{latent state Z t as a condition.} This probability distribution may be a distribution that deterministically generates a sample of behavior such as a delta function, or may be a normal distribution. When the decoder 104 outputs an action as a normal distribution, for example, the decoder 104 may determine and output the average value m _t of the action and the variance Σ _{t of} the action based on the latent state Z _t. Using these values, behaviors, for example, can be expressed as ^{_{a t (e) ~ N (}} m t, Σ t). Here, N (m, Σ) indicates a normal distribution of the mean m and the variance vector Σ. The decoder 104 outputs, for example, these m _t and Σ t, and for example, _{generates an action a t} ^(e) as a sample from this distribution.

That is, the decoder 104 outputs, for example, information such as what kind of action the controlled object should take. More specifically, the decoder 104 outputs stochastic behavior in the controlled object. Expressed conversion decoder 104 g _theta (), for example, the distribution behavior in the case of a normal _{distribution, (m t, Σ t)} = g θ ([f φ (o t (e)), M ζ ( It _{is expressed as η e} )]).

The distribution of behavior is not limited to the normal distribution. It may be a distribution that can appropriately represent the probability distribution of behavior, and if it is another distribution, the decoder 104 probabilistically represents the behavior based on parameters and the like suitable for expressing the distribution. Output to.

Further, in the above description, the first encoder 100, the second encoder 102, and the decoder 104 are provided with separate models, but the present invention is not limited to this, and all or at least two configurations may be provided in the same model. For example, when it is desired to estimate the next action when the state is updated, the first encoder 100 and the decoder 104 may be one continuous model. In this case, the output of the second encoder 102 may be received in the middle of processing. Moreover, as yet another example, all the above configurations may be provided in one model. In this case, the input of the dynamics parameter and the input of the state may be executed at different timings.

The first encoder 100, the second encoder 102, and the decoder 104 may be, for example, a trained model having an arbitrary configuration trained by an arbitrary machine learning method.

In this way, the first estimation unit 10 estimates the behavior based on the state and the dynamics parameters.

The second estimation unit 12 of the present embodiment includes an estimator 120 and an aggregator 122. The second estimation unit 12 estimates the dynamics parameters corresponding to each of the various environments for input to the second encoder 102. The second estimation unit 12 outputs the dynamics parameter when the time series of the state and the time series of the action are input. The dynamics parameter to be output may be represented by a probability distribution.

_{The relationship between the state o t} and the action a _t at a certain time is that when an action is taken for a situation, the next situation is established. That is, _{o t + 1} should be generated _{from o t} and a _{t at} time t based on the dynamics parameters. Therefore, the processing of the set of these three values by a function of the _{(o t, a t, o} t + 1) receiving the second estimation unit 12 is performed.

Here, if there are inputs from t = 1 to kT {(o _t , a _t , o _{t + 1} ) _n } _{n = 1} ^kT , this input is divided into chunks of, for example, T pieces. That is, the input data of this time series is {(o _t , a _t , o _{t + 1} ) _t } _{t = 1} ^T , {(o _t , a _t , o _{t + 1} ) _t } _{t = T + 1} Divide into ^2T , ···, {(o _t , a _t , o _{t + 1} ) _t } _{t = (k-1) T + 1} ^kT. Processing is executed for each of the divided data. If the time series is not a multiple of T, for example, the last chunk may be up to the last data, and the last chunk may be complemented by appropriate data without limitation.

The estimator 120 obtains an estimator of dynamics parameters for the divided chunks based on the input for each divided chunk. The estimator may be expressed in the form of a probability distribution, for example, an ^{estimator such as mean μ and variance σ 2} can be obtained. That is, the estimator of the dynamics parameter is obtained from _{T (o t} , a _t , o _{t + 1) data.} Do this for k chunks and get an estimator for each. The estimator 120 is, for example, an estimator of dynamics parameters μ ⁽¹⁾ , σ ⁽¹⁾ , μ ⁽²⁾ , σ ⁽²⁾ , ···, μ ^(k) , σ from the data of k chunks. ^{Get (k)} .

Note that this calculation may be performed in parallel, and for example, a plurality of estimators 120 may be provided for each chunk. Further, in the present embodiment, as an example of the estimated amount, the average value and the variance value are acquired, but the present invention is not limited to this, and any estimate can be obtained as long as it can be obtained.

The aggregator 122 synthesizes the estimator output by the estimator 120 using, for example, a trained model, and estimates and outputs the dynamics parameters in the environment in which the time series data is observed.

The estimator 120 and the aggregator 122 may be, for example, a trained model having an arbitrary configuration trained by an arbitrary machine learning method.

In this way, the second estimation unit 12 estimates the dynamics parameters based on the state and the behavior.

The estimation device 1 estimates the dynamics parameters based on the state and behavior data already acquired by, for example, the second estimation unit 12. Then, the first estimation unit 10 estimates the behavior based on the state at the current time and the estimated dynamics parameters.

FIG. 2 is a flowchart showing the processing of the estimation device 1 according to the present embodiment.

First, the acquired state and action are input to the second estimation unit 12 to estimate the dynamics parameter (S100).

Next, the estimated dynamics parameters are encoded by the second encoder 102 of the first estimation unit 10 (S102).

Next, the observed state is encoded by the first encoder 100 of the first estimation unit 10 (S100).

Next, the decoder 104 of the first estimation unit 10 estimates the behavior based on the encoded estimated dynamics parameters and the encoded state (S106).

Next, it is determined whether or not to end the process (S108).

If the process is not completed (S108: NO), the process from S104 is repeated. In this case, for example, the state may be a new state transitioned by the estimated behavior.

When the process is terminated (S108: YES), the required termination process is executed and the process is terminated.

Although the case where the dynamics parameter is not updated has been described above, the dynamics parameter may be updated at a predetermined timing. For example, when the process is not completed (S108: NO), it is determined whether or not to update the dynamics parameter (S110).

If the dynamics parameters are not updated (S110: NO), the processing from S104 is repeated.

When updating the dynamics parameters (S110: YES), the process from S100 is repeated. In this case, as the state and action used for estimating the dynamics parameter, the state observed in the previous time and the action executed in the previous time may be used.

In this way, the dynamics parameters may be updated at an appropriate time.

As described above, by estimating the dynamics parameters based on the already acquired state and behavior data, and further estimating the behavior based on the acquired state and estimated dynamics parameters, the actual environment and It is possible to quickly acquire measures to obtain desired behavior without interaction. As a result, for example, it is possible to improve the accuracy of processing automation at a production site or the like. In the above, the first estimation unit 10 and the second estimation unit 12 have separate configurations, but the configuration is not limited to this. These estimation units may be implemented as one estimation unit.

According to the estimation device 1 implemented in this way, it is possible to ensure safety rather than starting from a random initial value and trying and training in a real environment. Further, according to the above-described embodiment, it is possible to acquire a policy that can adapt the difference that occurs from the actual environment in the policy trained in the simulation environment. This result can be used, for example, for automation by a robot.

For example, the dynamics parameters may change due to different environments, and a second encoder 102 capable of outputting appropriate results for the various dynamics parameters may be required. Therefore, the first estimation unit 10 uses, for example, a second encoder 102 trained in a simulator environment. By using the second encoder 102 trained in the simulator environment, it is possible to estimate the behavior online in various environments.

(Training device)
The training device according to the present embodiment trains the first encoder 100, the second encoder 102, the decoder 104, the estimator 120, and the aggregator 122 of the estimation device 1. Training is performed, for example, by machine learning techniques. The training of this embodiment is based on the fact that the dynamics parameters can be acquired in advance when the simulator is used. That is, in the simulator environment, training is performed so that actions can be acquired from the state using known dynamics parameters. Then, by adapting this simulation environment to the actual space, training is performed to improve the estimation accuracy of the behavior in the actual space.

FIG. 3 is a block diagram of the training device 2 according to the present embodiment. The training device 2 includes a forward propagation unit 20, an error calculation unit 22, and an update unit 24. In addition, the training device 2 may include an interface for inputting / outputting data and the like, and a storage unit for storing data, parameters, and the like. The training device 2 trains the model in each part of the estimation device 1 described above.

The training device 2 may perform training of the first encoder 100, the second encoder 102, the decoder 104, the estimator 120, and the aggregator 122 shown on the right side of the figure in one device. As another example, one training device 2 may be provided for each of the first estimation unit 10 and the second estimation unit 12. As yet another example, a training device 2 may be provided to train a part of all models. When one training device 2 trains a plurality of models, each process such as a loss calculation method and an update method, that is, the training method may be switched for each model.

As described above, it is preferable that the model of each configuration of the first estimation unit 10 is trained in the simulation environment. It is generally difficult to obtain accurate information on the dynamics parameters in the space where the device to be controlled or the like is arranged. By preparing a plurality of simulation environments in which each element of the dynamics parameter is given and performing training in this simulation environment, it is possible to easily obtain a model capable of estimating the behavior of the device or the like to be actually controlled.

The second estimation unit 12 estimates the dynamics parameters from the state and the behavior, and the dynamics parameters estimated by the second estimation unit 12 are used to construct a simulation environment for executing the training of the first estimation unit 10. Can be used as a parameter. Further, the second estimation unit 12 is used not only for training the first estimation unit 10, but also for estimating dynamics parameters for estimating behavior in an actual environment, as described above. In this way, the training device 2 trains the first estimation unit 10 or the model for estimating the dynamics parameters for executing the estimation.

The forward propagation unit 20 executes the forward propagation process of the model to be trained. For example, predetermined data is input to each model and output data is acquired.

The error calculation unit 22 calculates the error described by the data acquired by the forward propagation unit 20 and the error based on the teacher data or the like, or the cumulative cost acquired from the system by executing the policy. The method of calculating the loss or the like may differ depending on the model. Further, as will be described later, the error calculation unit 22 executes the calculation of the loss described by the cost of the reward (cumulative reward) in the reinforcement learning, which is not necessarily used for the error calculation of supervised learning. May be good. In this way, the error calculation unit 22 calculates, in a broad sense, the values required for updating the loss, reward, and other parameters based on the objective function.

The update unit 24 updates the network based on the loss calculated by the error calculation unit 22. Network updates are performed, for example, by backpropagating errors. When the error is back-propagated, as shown by the dotted arrow in the figure, for example, the processing of the error calculation unit 22 and the update unit 24 may be executed for each layer of the neural network.

The training of the model provided in each part will be described in detail below.

First, a model provided in the first estimation unit 10 will be described. The models provided in the first encoder 100, the second encoder 102, and the decoder 104 of the first estimation unit 10 may be trained at the same timing.

FIG. 4 is a flowchart showing a training process of the first estimation unit 10 according to the present embodiment. Based on FIG. 4, the training process related to the first estimation unit 10 of the training device 2 will be described.

First, the training device 2 initializes the parameters (S200). A parameter is a first encoder 100, second encoder 102, model f _phi provided to the decoder 104, M _eta, g _theta, and a first encoder 100, the model f _rec to train decoder 104 relates g _inv It is a parameter.

Next, n environments are randomly generated (S202). This environment is formed on a simulator, for example. Based on the state and behavior acquired from this simulator, the second estimation unit 12 can estimate and acquire n dynamic parameters corresponding to each environment described later.

Next, the parameter is updated by loop processing. First, one environment is randomly selected from the generated n environments (S204). The process from the selection of this environment may be regarded as an episode, and the parameter update may be repeated in the episode so as to satisfy a predetermined condition. In addition, any algorithm can be used as the training algorithm for each model.

Then get the dynamics parameters for the selected environment (S206). The dynamics parameter may be, for example, the one estimated by the second estimation unit 12 in S202.

Next, the initial state is randomly sampled (S208). For example, a state that becomes an initial state is randomly acquired from various data given as test data.

Next, update the parameters (S210). More specifically, the initial state acquired in S208 is input to the first encoder 100, and the dynamics parameter acquired in S206 is input to the second encoder 102 to acquire the _{latent state Z t.} Then, _{by inputting this Z t} to the decoder 104, the action a _t ^(e) is acquired. This may be performed, for example, by the forward propagation unit 20 progressively propagating the input data to each model. Then, based on the acquired result, the error calculation unit 22 calculates an appropriate error or the like for each model and propagates it back, and the update unit 24 updates the parameters. The training of each configuration will be described later.

Next, it is determined whether or not to end the process (S212). The end of the process may be determined, for example, by repeatedly calculating a predetermined number of episodes by a predetermined number of epochs, having an accuracy of a predetermined value or more, satisfying a predetermined condition by validation, and the like. Here, the predetermined numbers for episodes and epochs are not the same, but may be set for each.

If it is determined that the processing is not completed (S212: NO), the processing from S204 is repeated. In this case, if the number of episodes repeated is within a predetermined number, the episodes may be repeated. Also, when the repetition of the episode is completed, it is judged whether the number of epochs has reached the predetermined number, and if the number of epochs has not reached the predetermined number, the next epoch process can be executed. Good.

On the other hand, when it is determined to end the process (S212: YES), the training device 2 ends the process after executing necessary processes such as output of optimized parameters and storage.

The first estimation unit 10 may be trained in the simulation environment in this way. This simulation environment may be optimized by imparting randomness and learning strategies that can adapt to it. Since the simulation environment is used, the training device 2 can acquire dynamics parameters for various environments, for example. The training device 2 trains the first estimation unit 10 using this dynamics parameter.

The training of the first estimation unit 10 described above can be performed by reinforcement learning. Reinforcement learning generally requires a very large number of trials, so it is practically difficult to perform reinforcement learning in multiple different environments so that appropriate estimation can be performed in different environments. Is. In addition, training in a real environment may be dangerous. However, in the present embodiment, since reinforcement learning can be easily performed by training the first estimation unit in the simulation environment, it is possible to obtain the first estimation unit that can be appropriately estimated even in different environments. Can be done.

ここで、γは、γ∈[0, 1]を満たす定数とし、E_πは、方策の確率分布πと初期状態o₀、システムの状態遷移確率と報酬確率に関する期待値を意味する。このような目的関数の確率分布πのパラメータに関する最大化として、強化学習を行ってもよい。また、Tは、エピソードの長さを表す定数であり、正の整数である。 When performing reinforcement learning, for example, the probability distribution for generating the action a is set to π (a | o, η), the reward r _k is set, and the reinforcement learning can be performed using the following objective function.

Here, γ is a constant satisfying γ ∈ [0, 1], and E _π means the probability distribution π of the policy, the initial state o ₀ , and the expected value regarding the state transition probability and the reward probability of the system. Reinforcement learning may be performed as a maximization of the parameters of the probability distribution π of such an objective function. In addition, T is a constant representing the length of the episode and is a positive integer.

Even when performing reinforcement learning, the parameters in the simulation environment may be acquired in advance in the simulation environment as described above. When reinforcement learning is performed as in the present embodiment, it is possible to apply the training results in the simulation environment to the actual environment while suppressing the decrease in accuracy.

ここで、f_rec()は、第１エンコーダ１００により変換された状態o_tを元に戻すモデルである。例えば、第１エンコーダ１００における状態o_tの圧縮率が高すぎると、圧縮された状態f_φ(o_t)から元の状態o_tへと逆変換できない場合がある。このような場合には、デコーダ１０４において、状態o_t又はその一部が行動へと反映されない場合がある。そこで、このような問題を回避するために、圧縮された状態がある程度状態へと逆変換できるように、第１エンコーダ１００を、［数２］の損失を用いて更新する。 Training of the first encoder 100 may be performed, for example, with the additional loss L _rec ^(t) of:

Here, f _rec () is a model that restores _{the state o t} converted by the first encoder 100. For example, if the compression ratio _{of the state o t} in the first encoder 100 is too high, it may not be possible to reversely convert _{the compressed state f φ} (o _t ) to the original state o _t. In such a case, in the decoder 104, the state o _t or a part thereof may not be reflected in the action. Therefore, in order to avoid such a problem, the first encoder 100 is updated with the loss of [Equation 2] so that the compressed state can be inversely transformed into a state to some extent.

The objective function is not limited to this, and may be any objective function that can compare an appropriately encoded state with the original state.

ここで、g_inv()は、第１潜在状態Z_tと、第２潜在状態Z_t+1の間にどのような行動が実行されたかを推定するモデルである。Z_tは、上述したように、状態o_t ^(e)とダイナミクスパラメータη_eを用いて、Z_t=[f_φ(o_t ^(e)), M_ζ(η_e)]と表される。これに対して、Z_t+1は、デコーダ１０４により推定された時刻tにおける行動a_t（第１行動と呼ぶ）が与えられた場合の時刻t+1の状態o_t+1 ^(e)とダイナミクスパラメータη_eを用いて、Z_t+1=[f_φ(o_t+1 ^(e)), M_ζ(η_e)]と表される。g_inv(Z_t+1, Z_t)は、Z_t+1とZ_tが与えられた場合に、対象となる環境においてどのような行動a_t'（第２行動と呼ぶ）がZ_tに対して実行されたかを推定するモデルである。 Further, the training of the second encoder 102 and the decoder 104 may be performed, for example, by additionally using the following loss L _inv ^(t) .

Here, g _inv () is a model that estimates what kind of action was executed between the first latent state Z _t and the second latent state Z _{t + 1.} Z _t is expressed as Z _t = [f _φ (o _t ^(e) ), M _ζ (η _{e )] using the state o t} ^(e) and the dynamics parameter η _e , as described above. On the other hand, Z _{t + 1 is the state o t + 1} ^(e) at time t + 1 when the _{action a t} (called the first action) at time t estimated by the decoder 104 is given. Using the dynamics parameter η _e , it is expressed as _{Z t + 1} = [f _φ (o _{t + 1} ^(e) ), M _ζ (η _e)]. g _inv (Z _{t + 1} , Z _t ) gives what action a _t '(called the second action) to Z _t in the _{target environment given Z t + 1} and Z _t. It is a model that estimates whether or not it was executed.

By defining Lrec (t) and Linv (t) in this way and training the model, information that is not necessary for estimating behavior, such as changes in light intensity, lighting direction, and shadow position, can be obtained. It is possible to perform training to estimate the impact by reducing it from the condition and environment.

L _inv ^(t) is the difference between the first action a _t actually given, the state generated by the first action, and what kind of second action a _t'was given when that state occurred. calculate. Using this loss function, the parameters of _{the model M ζ} of the second encoder 102, the model g _{θ of the decoder 104, and the model g inv} that acquires the action from the latent state are updated.

Next, the training of the second estimation unit 12 for estimating the dynamics parameters input to the first estimation unit 10 will be described.

In the training of the first estimation unit 10, it is necessary to acquire the dynamics parameters in n random environments. Therefore, first, the behavior may be randomly sampled within the range of the permitted behavior, and a measure may be set so that several episodes or the like can be executed in each environment. Here, the permitted action means, for example, an action permitted in the environment or an arbitrary action in a pre-trained safe measure.

ここで、R_tは、ノイズ項であり、簡単のため、平均0、分散v²とする正規分布と仮定してもよい。これは、状態遷移がQ(o_t+1 ^(e)|o_t ^(e), a_t ^(e), η_e)の尤度モデルを定義することと同義である。このため、D_e={(o_t ^(e), a_t ^(e), o_t+1 ^(e))_t}_t=1 ^Neとした場合に、事後分布p(η_e|D_e)を介してη_eを推定することに、問題は帰着される。そこで、ダイナミクスパラメータを正しく推定するために、エピソード内の相関状態遷移データを仮定してもよい。 Therefore, the state transition data, as described ^{_{above, {(o t (e)}} , a t (e), o t + 1 (e))} were collected in the form of, perform the training of the second estimating unit 12 To do. Here, F () is defined as a forward-propagating dynamics parameter model of the simulator. The true next state and the next state by the simulator are expressed as follows when the true value of _{the dynamics parameter η e can be obtained.}

Here, R _t is a noise term, and for the sake of simplicity, it may be assumed that it is a normal distribution with ^{mean 0 and variance v 2.} This state transition ^{_{Q (o t + 1 (e}} ) | o t (e), a t (e), η e) is synonymous with defining a likelihood model. Therefore, when _De = {(o _t ^(e) , a _t ^(e) , o _{t + 1} ^(e) ) _t } _{t = 1} ^Ne , the posterior distribution p (η _e | De _e ) The problem is reduced to estimating _{η e through.} Therefore, in order to correctly estimate the dynamics parameters, the correlation state transition data in the episode may be assumed.

ここで、チャンクの長さは、適切に選択されていてもよい。例えば、Tは、大きすぎない範囲で十分に大きく取ってもよい。例えば、Tが十分大きい場合には、中心極限定理が成り立つような場合において、事後分布が正規分布に近づくため近似が正確になる。一方で、Tが大きすぎる場合、データ数がTと比較して少なくなるとそのままの状態で適用することが困難となる。また、時系列のデータをTで割った場合のあまりが生じる場合に、残りのデータを有効に利用することができない等の無駄が生じる可能性があるためである。 As described above, divided into k chunks of D _e T number of tuples ^{_{(o t (e), a}} t (e), o t + 1 (e)) for each respective chunk i ∈ { Train the model of the estimator 120 so ^{that μ (i)} and σ ⁽ⁱ⁾ can be estimated at 1, ..., k}. If the time series of actions in chunk i in the environment e is x _i ^(e) and the time series of states is y _i ^(e) , it may be executed by estimating the following posterior distribution p.

Here, the chunk length may be appropriately selected. For example, T may be large enough so that it is not too large. For example, when T is sufficiently large, the approximation becomes accurate because the posterior distribution approaches the normal distribution when the central limit theorem holds. On the other hand, if T is too large and the number of data is smaller than that of T, it becomes difficult to apply the data as it is. In addition, if too much time-series data is divided by T, waste such as not being able to effectively use the remaining data may occur.

これは、以下のように書き換えられる。

ここで、事後分布p(η_e|x_i ^(e), y_i ^(e))と事後分布p(η_e)とが独立した正規分布であると仮定すると、事後分布p(η_e|D_e)は、以下のように書き換えることもできる。

ここで、添え字のjは、ベクタのj番目の要素を表す。ニューラルネットワーク（ディープニューラルネットワークを含む）において実現されたμ_j ⁽ⁱ⁾()、σ_j ⁽ⁱ⁾()を用いると、p(η_e|D_e)は、パラメータθにより、事後分布p_θ(η_e|D_e)としてパラメタライズすることができる。パラメータθは、スカラー、例えば、f_0,j、g_0,j（j=1, ... ,d）（dは、ηの次元）を含んでいてもよい。パラメータθは、真の事後分布を近似するように最適化されてもよい。 _{To aggregate the estimates of k η e} calculated from each chunk i, the posterior distribution p (η _e | x _i ^(e) , of _{the dynamics parameter η e} conditioned on a single data point pair. You may use the relationship that should hold between y _i ^(e) _{) and the posterior distribution of the dynamics parameter η e} conditioned on the entire dataset. _{The posterior distribution of η e} conditioned on the entire data set is expressed as follows.

This can be rewritten as follows:

Assuming that the posterior distribution p (η _e | x _i ^(e) , y _i ^(e) ) and the posterior distribution p (η _e ) are independent normal distributions, the posterior distribution p (η _e | D) _e ) can also be rewritten as follows.

Here, the subscript j represents the j-th element of the vector. _{Using μ j} ⁽ⁱ⁾ () and σ _j ⁽ⁱ⁾ () realized in neural networks (including deep neural networks), p (η _e | _De ) has a posterior distribution p _{θ due to the parameter θ.} It can be parameterized as (η _e | _De). The parameter θ may include a scalar, for example, f _{0, j} , g _{0, j} (j = 1, ..., d) (where d is the dimension of η). The parameter θ may be optimized to approximate the true posterior distribution.

KL (Kullback-Leibler) divergence may be used as a method for approximating the posterior distribution. For example, the KL divergence between the _true posterior distribution p true (η _e | De _e ) and the posterior distribution p _θ (η _e | _{De ) expressed by parameters KL [p θ} (η _e | _De ) | p _true (η _e | De _e )] can be approximated by taking the minimum value for the parameter θ. According to lower bound optimization, it is possible to approximate the posterior distribution for each environment e without explicitly evaluating the _true posterior distribution p true (η _e | _De). For example, optimization may be performed using the following equation.

Also, the true posterior distribution p _true | for obtaining the (η _e D _e) and KL divergence normal distribution close, may be used re parametric Tri's trick often used in VAE (Variational Auto Encoder). That, ε _j ~ N | a (ε _j 0, 1), that expressed according to the following equation eta _e using the obtained epsilon _j, the posterior distribution p θ _{(η e} expressed by the parameter | D _e ) May be replaced with the expected value of the standard normal distribution ε ~ N (ε | 0, 1).

As mentioned above, standalone tuple ^{_{(o t (e), a}} t (e), o t + 1 (e)) instead of by time considering the chunk sequence, friction, complex, such as gravity It is possible to effectively approximate the posterior distribution of dynamics parameters. In general, the posterior distribution of dynamics parameters can be a complex multimodal distribution, but in cases such as according to the central limit theorem, as the number of samples in a time-series chunk increases, the η conditioned on that sample The posterior distribution can be approximated to the normal distribution.

FIG. 5 is a flowchart showing the training process of the second estimation unit 12 by the training device 2. Since the details have been described above, they will be briefly described. First, the parameters of the estimator 120 and the aggregator 122 are initialized (S300). Next, the dynamics parameters of the environment e are randomly set, samples of states and behaviors are extracted, and the entire time series is generated (S302). Next, the time series is divided and a plurality of chunks used for parameter generation are generated (S304). The parameters of the estimator 120 and the aggregator 122 are updated according to the above [Equation 4] to [Equation 13] (S308). When the process is completed (S308: YES), the training device 2 ends the training process of the second estimation unit 12. The end of the training may be according to the conditions described in the training of the first estimation unit 10 described above, or may be different. If the process is not completed (S308: NO), the process may be repeated from the state and behavior sample extraction process (S302) in different environments. Further, for example, when changing the training environment, the process may be repeated from the state / behavior sample extraction process (S302) as needed.

Given the state transition data in this way, the trained model can be used to estimate the dynamics parameters of a given environment, including a real environment such as a test environment. According to the above, the estimation of the dynamics parameters may be performed using only the data outside the policy. Data outside this policy can be collected simply by executing a random policy or an existing policy created based on a rule base that is known to be safe to implement. Therefore, data for training models that acquire dynamics parameters in various environments can be collected at low cost.

As described above, according to the present embodiment, it is possible to train the model of the second estimation unit 12 that acquires the dynamics parameters and generate a simulator using the dynamics parameters obtained by this model. Then, by executing training in various environments using the simulator for the observed state, application of the model in a highly accurate real environment and training of the model using the model trained in the simulator environment are performed. It becomes possible to execute.

The training of the second estimation unit 12 can be trained by supervised learning as described above by utilizing the fact that the dynamics parameters are known in the simulation environment. Furthermore, even if the dynamics parameters are unknown, it is possible to learn the dynamics parameters and the state transition parameters representing the dynamics so that the future state can be appropriately predicted.

By the second estimation unit 12 trained in advance in such a simulation environment, in the actual environment, the behavior is actually selected and the transition is performed by the existing method without realizing the interaction with the environment that works on the environment. The dynamics parameters of the environment can be estimated simply by observing the state and behavioral sequence. By using the dynamics parameters estimated by the second estimation unit 12 and using the first estimation unit 10 as a policy, it is possible to obtain a good policy in the actual environment.

Using the first estimation unit 10 trained using the estimated values of the dynamics parameters in n environments estimated by the second estimation unit 12 thus trained, the first estimation unit 10 has an unknown dynamics parameter η _{n + 1} . The behavior in the n + 1th environment, for example, the environment of the actual machine may be estimated according to the flowchart shown in FIG.

For example, first _{generate models of f φ} , M _η , g _θ , respectively (and may include _{f rec} , g _{inv) from the trained parameters.} Suppose the environment is the n + 1th environment, for example a test environment. Here, we get _{the time series (D n + 1} ) of the state and action outside the policy. This acquisition may be performed, for example, by operating according to the action acquired by the first estimation unit 10 in the actual environment. Using this D _{n + 1} , the second estimation unit 12 estimates the dynamics parameter η _{n + 1} (S100). Then, the behavior is estimated by executing the measures based on the dynamics parameters from the models f _φ , M _η , and g _{θ (S106).} As described above, according to the present embodiment, it is possible to estimate appropriate behavior from the state based on the measures trained in the simulator environment without making a trial for the environment.

Further, in the present embodiment, fine tuning may be further performed. For example, using the dynamics parameters generated in the real environment, the real machine is operated by the estimated behavior. Then, when a mistake occurs, for example, a human takes a more desirable action to acquire a time series of states and actions. Fine tuning of the first estimation unit 10 may be executed using this time series. In this way, the first estimation unit 10 can be tuned to have higher accuracy.

For example, in a real environment, a step of evaluating whether or not good performance can be obtained by the acquired measure may be included before actually applying the acquired measure. This evaluation may include whether the estimated simulation environment mimics the real environment well and whether the measures perform well in the mimicking simulation environment. That is, it may be applied to the actual environment when it is determined that a better measure than the existing method is obtained.

Then, the A / B test may be executed using the result A using the actually estimated policy and the result B using the existing rule-based policy. You may switch to a well-estimated strategy as a result of running A / B testing. Further, if the result A is inferior to the result B, the result may be directly fed back to improve the policy. Further, if necessary, the simulated environment may be improved, and the measures may be further improved in the improved simulation environment. By fine-tuning in this way, it is possible to improve the performance of measures in the actual environment.

There are two main causes when the estimation by the estimation device 1 is inferior. One is when the simulation environment deviates from the actual environment, and the other is when the simulation environment can reproduce the actual environment with high accuracy, but the performance of the policy itself in the simulation environment is not high. Is.

Whether or not the simulation environment deviates from the actual environment can be judged by whether or not the state transition is reproduced well. In this case, measures may be trained to reduce this divergence. Regarding the performance of the policy in the simulation environment, the policy can be actually tested in the simulation environment. In this case, further training of measures in the simulation environment may be performed. Then, after the performance is improved, the estimation device 1 may be tuned by executing the A / B test again.

For example, during training in a real environment (during fine tuning), if the estimation of the estimation device is not appropriate and a problem occurs, for example, the policy is not appropriate in an automated factory line, and the device is controlled. A mistake may occur, but a human or other device may back it up. That is, a human worker or other device may correct or respond to a device error. For example, when a person and a robot (control device) being trained work together, if the robot's work (for example, picking) does not go well, the human can cover it and train to increase the success rate of work picking. it can.

When learning on a factory line or the like, the speed of the line may be set so that the robot can perform the work at a speed that allows humans to cover the mistakes of the robot at the initial stage of training.

You may also increase the speed of the line as the training progresses.

In the above case, the speed adjustment may be performed based on the success rate or may be performed by a human, and the success rate is calculated based on the information acquired by a sensor such as an image pickup device provided in the environment. It may be based on or done.

This makes it possible to train the model while maintaining the productivity of the line.

In addition to changing the speed of the line, changing the ratio of current measures and newly tested measures that are known to be appropriate as a way to balance the training of such a model with the productivity of the line. Can also be given. Here, the new test strategy may be the one that seems to be the best at the moment, such as the one that was appropriate in the simulation environment, for example, a random strategy to test if there is a better strategy. May be.

For example, the ε-greedy method may be used to change this ratio. The ε-greedy search in reinforcement learning is a probabilistic trial of both a random strategy and what seems to be the best policy at present, at a rate of probability ε and 1-ε.

(Modification example)
The estimation device 1 trained by the training device 2 as described above can be made more robust. The training device 2 performs the training in an environment having different dynamics. This dynamics determines the current state and the state transition probability of what state the selected action transitions to. In this case, not only can different dynamics be generated by changing the parameters included in the predetermined state transition probability, but even more diverse dynamics can be expressed by making the behavior vary in this state transition probability. Good.

The training device 2 may apply other information added to the environment or state, such as changes in motor noise. For example, a behavior may be interpreted as domain randomization, and deviations that depend on a particular state may be added to the behavior. In this case, the deviation added to the action may be implemented by adding disturbance to the action output of the policy model.

To estimate the disturbance ε _t using a model, for example, if the disturbance ε _t is represented by the inner product of the vector function of the current state o _t ^(e) weighted by the environment-dependent parameter vector ω _e. Assume. For example, Φ can be represented as a non-linear mapping, specifically, as a feedforward neural network with known or randomly assigned parameters τ, as disturbance ε _t = ω _e Φ _τ (o _t ^(e) ). Good. Incidentally, for example, represent the omega _e in the horizontal vector represents [Phi _tau and ^(o _t ^(e)) in the vertical _{vector, ω e Φ τ (o t} (e)) represents the inner product of vectors. In this case, it is executed by identifying the perturbation caused by the motor noise in the environment e through the estimation of _{ω e.}

ここで、Kは、ノイズに対するスカラーの係数である。ω_eは、ダイナミクスパラメータη_eと同様に環境に依存するパラメータであるので、拡張されたダイナミクスパラメータη_e'=(η_e, ω_e)として上述のダイナミクスパラメータη_eの推定モデルと同様に処理することでω_eの推定に関しても訓練することが可能である。 _{Let a ^ t} ^(e) be the original predictive behavior at time step t. The behavior that is input to the simulator due to the variation in behavior is expressed as follows, for example.

Here, K is a scalar coefficient with respect to noise. omega _e is because it is a parameter that depends on the environment as well as the dynamics parameter eta _e, enhanced dynamics parameters _{η e '= (η e,} ω e) as similar to the estimation model of the above-mentioned dynamics parameters eta _e processing By doing so, it is possible to train on the estimation _{of ω e.}

_{As described above, by extending the dynamics parameter η e} with respect to the environment e, the information added to the environment and the state, for example, the influence of motor noise can also be estimated by the estimation device 1. This estimation device 1 can of course be realized by the _{training device 2 by transforming the dynamics parameter η e based on the above equation.}

Hereinafter, some results using the method in the present disclosure will be described.

FIG. 6 is a graph showing the reward of the estimation device 1 trained by the method described above. In the figure, (1) is f _rec, results trained without using a model of g _inv, (2) a result of trained using a model of g _inv, (3) is f _rec Model results, which is trained using (4), which is a result of the trained using both models f _rec, g _inv. As a comparative example, (5) is the result of training using the correct dynamics parameters in the environment, and it can be considered that it is theoretically impossible to obtain a better reward. (6) is the result of training using MAML (Model Agnostic Meta-Learning), which is a method of meta-learning, as a comparative example.

From this graph, _{it can be seen that those trained using both f rec} and g _inv receive better rewards than the results of training using either or neither. In addition, it can be seen that the results can be compared with MAML.

FIG. 7 is a graph showing the transition of reward during training of the above-mentioned method. The horizontal axis is proportional to the number of episodes in the training, and the vertical axis shows the reward. (4) and (5) are the same as those described in FIG. As a comparative example, the result using only domain randomization is shown. As shown in this graph, it can be seen that according to this embodiment, better results can be obtained from the early stage of training than in other trainings.

FIG. 8 is a graph showing the effect of the disturbance shown above. The horizontal axis shows the coefficient K related to the disturbance, and the vertical axis shows the reward. τ is randomly selected. The result with disturbance shown by the solid line is the result of estimating the behavior _{by setting the disturbance at the time of training and estimating ω e by the above-mentioned method.} The result without disturbance shown by the broken line is the result of estimating the behavior without encoding the _{disturbance ω e in the latent state Z and estimating the ω e, although the disturbance was taken into consideration in the training.} As shown in this graph, according to the present embodiment, it can be seen that better results can be obtained by considering the disturbance and further reflecting it in the latent state Z.

A part or all of each device (estimation device 1 or training device 2) in the above-described embodiment may be composed of hardware, a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or the like. It may consist of information processing of software (program) to be executed. When it is composed of information processing of software, the software that realizes at least a part of the functions of each device in the above-described embodiment is a flexible disk, a CD-ROM (Compact Disc-Read Only Memory), or a USB (Universal Serial). Bus) Information processing of software may be executed by storing it in a non-temporary storage medium (non-temporary computer-readable medium) such as a memory and loading it into a computer. In addition, the software may be downloaded via a communication network. Further, information processing may be executed by hardware by implementing software in a circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

The type of storage medium that stores the software is not limited. The storage medium is not limited to a removable one such as a magnetic disk or an optical disk, and may be a fixed storage medium such as a hard disk or a memory. Further, the storage medium may be provided inside the computer or may be provided outside the computer.

FIG. 9 is a block diagram showing an example of the hardware configuration of each device (estimating device 1 or training device 2) in the above-described embodiment. As an example, each device includes a processor 71, a main storage device 72 (memory), an auxiliary storage device 73 (memory), a network interface 74, and a device interface 75, which are connected via a bus 76. It may be realized as a computer 7.

The computer 7 of FIG. 9 includes one component for each component, but may include a plurality of the same components. Further, although one computer 7 is shown in FIG. 9, software is installed on a plurality of computers, and each of the plurality of computers executes the same or different part of the software. May be good. In this case, it may be a form of distributed computing in which each computer communicates via a network interface 74 or the like to execute processing. That is, each device (estimation device 1 or training device 2) in the above-described embodiment is a system that realizes a function by executing instructions stored in one or a plurality of storage devices by one or a plurality of computers. It may be configured. Further, the information transmitted from the terminal may be processed by one or a plurality of computers provided on the cloud, and the processing result may be transmitted to the terminal.

Various operations of each device (estimation device 1 or training device 2) in the above-described embodiment are executed in parallel processing by using one or more processors or by using a plurality of computers via a network. May be good. Further, various operations may be distributed to a plurality of arithmetic cores in the processor and executed in parallel processing. In addition, some or all of the processes, means, etc. of the present disclosure may be executed by at least one of a processor and a storage device provided on the cloud capable of communicating with the computer 7 via a network. As described above, each device in the above-described embodiment may be in the form of parallel computing by one or a plurality of computers.

The processor 71 may be an electronic circuit (processing circuit, Processing circuit, Processing circuitry, CPU, GPU, FPGA, ASIC, etc.) including a control device and an arithmetic unit of a computer. Further, the processor 71 may be a semiconductor device or the like including a dedicated processing circuit. The processor 71 is not limited to an electronic circuit using an electronic logic element, and may be realized by an optical circuit using an optical logic element. Further, the processor 71 may include an arithmetic function based on quantum computing.

The processor 71 can perform arithmetic processing based on data and software (programs) input from each device or the like having an internal configuration of the computer 7, and output the arithmetic result or control signal to each device or the like. The processor 71 may control each component constituting the computer 7 by executing an OS (Operating System) of the computer 7, an application, or the like.

Each device (estimation device 1 and / or training device 2) in the above-described embodiment may be realized by one or more processors 71. Here, the processor 71 may refer to one or more electronic circuits arranged on one chip, or may refer to one or more electronic circuits arranged on two or more chips or two or more devices. You may point. When a plurality of electronic circuits are used, each electronic circuit may communicate by wire or wirelessly.

The main storage device 72 is a storage device that stores instructions executed by the processor 71, various data, and the like, and the information stored in the main storage device 72 is read out by the processor 71. The auxiliary storage device 73 is a storage device other than the main storage device 72. Note that these storage devices mean arbitrary electronic components capable of storing electronic information, and may be semiconductor memories. The semiconductor memory may be either a volatile memory or a non-volatile memory. The storage device for storing various data in each device (estimation device 1 or training device 2) in the above-described embodiment may be realized by the main storage device 72 or the auxiliary storage device 73, and is built in the processor 71. It may be realized by the built-in memory. For example, the storage unit in the above-described embodiment may be realized by the main storage device 72 or the auxiliary storage device 73.

A plurality of processors may be connected (combined) or a single processor may be connected to one storage device (memory). A plurality of storage devices (memory) may be connected (combined) to one processor. Each device (estimation device 1 or training device 2) in the above-described embodiment is composed of at least one storage device (memory) and a plurality of processors connected (combined) to the at least one storage device (memory). In the case, a configuration in which at least one of a plurality of processors is connected (combined) to at least one storage device (memory) may be included. Further, this configuration may be realized by a storage device (memory) and a processor included in a plurality of computers. Further, a configuration in which the storage device (memory) is integrated with the processor (for example, a cache memory including an L1 cache and an L2 cache) may be included.

The network interface 74 is an interface for connecting to the communication network 8 wirelessly or by wire. As the network interface 74, an appropriate interface such as one conforming to an existing communication standard may be used. The network interface 74 may exchange information with the external device 9A connected via the communication network 8. The communication network 8 may be any one of WAN (Wide Area Network), LAN (Local Area Network), PAN (Personal Area Network), or a combination thereof, and the computer 7 and the external device 9A may be used. Any information can be exchanged between them. An example of a WAN is the Internet, an example of a LAN is IEEE802.11 or Ethernet (registered trademark), and an example of a PAN is Bluetooth (registered trademark or NFC (Near Field Communication)).

The device interface 75 is an interface such as USB that directly connects to the external device 9B.

The external device 9A is a device connected to the computer 7 via a network. The external device 9B is a device that is directly connected to the computer 7.

The external device 9A or the external device 9B may be an input device as an example. The input device is, for example, a device such as a camera, a microphone, a motion capture, various sensors, a keyboard, a mouse, or a touch panel, and gives the acquired information to the computer 7. Further, it may be a personal computer, a tablet terminal, or a device having an input unit such as a smartphone, a memory, and a processor.

Further, the external device 9A or the external device 9B may be an output device as an example. The output device may be, for example, a display device such as an LCD (Liquid Crystal Display), a CRT (Cathode Ray Tube), a PDP (Plasma Display Panel), or an organic EL (Electro Luminescence) panel, or may output audio or the like. It may be an output speaker or the like. Further, it may be a personal computer, a tablet terminal, or a device having an output unit such as a smartphone, a memory, and a processor.

Further, the external device 9A or the external device 9B may be a storage device (memory). For example, the external device 9A may be a network storage or the like, and the external device 9B may be a storage such as an HDD.

Further, the external device 9A or the external device 9B may be a device having some functions of the components of each device (estimating device 1 or training device 2) in the above-described embodiment. That is, the computer 7 may transmit or receive a part or all of the processing result of the external device 9A or the external device 9B.

In the present specification (including claims), the expression (including similar expressions) of "at least one (one) of a, b and c" or "at least one (one) of a, b or c" is used. When used, it includes any of a, b, c, ab, ac, bc, or abc. It may also include multiple instances of any element, such as a-a, a-b-b, a-a-b-b-c-c, and the like. It also includes adding elements other than the listed elements (a, b and c), such as having d, such as a-b-c-d.

In the present specification (including claims), when expressions such as "with data as input / based on / according to / according to" (including similar expressions) are used, unless otherwise specified. This includes the case where various data itself is used as an input, and the case where various data are processed in some way (for example, noise-added data, normalized data, intermediate representation of various data, etc.) are used as input. In addition, when it is stated that some result can be obtained "based on / according to / according to the data", it includes the case where the result can be obtained based only on the data, and other data other than the data. It may also include cases where the result is obtained under the influence of factors, conditions, and / or conditions. In addition, when it is stated that "data is output", unless otherwise specified, various data itself is used as output, or various data is processed in some way (for example, noise is added, normal). It also includes the case where the output is output (intermediate representation of various data, etc.).

In the present specification (including claims), when the terms "connected" and "coupled" are used, direct connection / coupling and indirect connection / coupling are used. , Electrically (electrically) connection / coupling, communication (communicatively) connection / coupling, functionally (operatively) connection / coupling, physical connection / coupling, etc. Intended as a term. The term should be interpreted as appropriate according to the context in which the term is used, but any connection / combination form that is not intentionally or naturally excluded is not included in the term. It should be interpreted in a limited way.

When the expression "A configured to B" is used in the present specification (including claims), the physical structure of the element A can perform the operation B. Including that the element A has a configuration and the permanent or temporary setting (setting / configuration) of the element A is set (configured / set) to actually execute the operation B. Good. For example, when the element A is a general-purpose processor, the processor has a hardware configuration capable of executing the operation B, and the operation B is set by setting a permanent or temporary program (instruction). It suffices if it is configured to actually execute. Further, when the element A is a dedicated processor, a dedicated arithmetic circuit, or the like, the circuit structure of the processor actually executes the operation B regardless of whether or not the control instruction and data are actually attached. It only needs to be implemented.

In the present specification (including claims), when a term meaning inclusion or possession (for example, "comprising / including" and "having", etc.) is used, the object of the term is used. It is intended as an open-ended term, including the case of containing or owning an object other than the indicated object. If the object of these terms that mean inclusion or possession is an expression that does not specify a quantity or suggests a singular (an expression with a or an as an article), the expression is interpreted as not being limited to a specific number. It should be.

In the present specification (including claims), expressions such as "one or more" or "at least one" are used in some places, and the quantity is specified in other places. Even if expressions that do not or suggest the singular (expressions with a or an as an article) are used, the latter expression is not intended to mean "one". In general, expressions that do not specify a quantity or suggest a singular (expressions with a or an as an article) should be interpreted as not necessarily limited to a particular number.

In the present specification, when it is stated that a specific effect (advantage / result) can be obtained for a specific configuration of an embodiment, unless there is a specific reason, one or more of the other configurations having the configuration. It should be understood that the effect can also be obtained in the examples of. However, it should be understood that the presence or absence of the effect generally depends on various factors, conditions, and / or states, etc., and that the effect cannot always be obtained by the configuration. The effect is merely obtained by the configuration described in the examples when various factors, conditions, and / or conditions are satisfied, and in the invention relating to the claim that defines the configuration or a similar configuration. , The effect is not always obtained.

In the present specification (including claims), when terms such as "maximize" are used, the global maximum value is obtained, the approximate value of the global maximum value is obtained, and the local maximum value is obtained. Should be interpreted as appropriate according to the context in which the term was used, including finding an approximation of the local maximum. It also includes probabilistically or heuristically finding approximate values of these maximum values. Similarly, when terms such as "minimize" are used, find the global minimum, find the approximation of the global minimum, find the local minimum, and find the local minimum. It should be interpreted as appropriate according to the context in which the term was used, including finding an approximation of the value. It also includes probabilistically or heuristically finding approximate values of these minimum values. Similarly, when terms such as "optimize" are used, finding a global optimal value, finding an approximation of a global optimal value, finding a local optimal value, and local optimization It should be interpreted as appropriate according to the context in which the term was used, including finding an approximation of the value. It also includes probabilistically or heuristically finding approximate values of these optimal values.

In the present specification (including claims), when a plurality of hardware performs a predetermined process, the respective hardware may cooperate to perform the predetermined process, or some hardware may perform the predetermined process. You may do all of the above. Further, some hardware may perform a part of a predetermined process, and another hardware may perform the rest of the predetermined process. In the present specification (including claims), when expressions such as "one or more hardware performs the first process and the one or more hardware performs the second process" are used. , The hardware that performs the first process and the hardware that performs the second process may be the same or different. That is, the hardware that performs the first process and the hardware that performs the second process may be included in the one or more hardware. The hardware may include an electronic circuit, a device including the electronic circuit, or the like.

Although the embodiments of the present disclosure have been described in detail above, the present disclosure is not limited to the individual embodiments described above. Various additions, changes, replacements, partial deletions, etc. are possible without departing from the conceptual idea and purpose of the present invention derived from the contents defined in the claims and their equivalents. For example, in all the above-described embodiments, when numerical values or mathematical formulas are used for explanation, they are shown as examples, and the present invention is not limited thereto. Further, the order of each operation in the embodiment is shown as an example, and is not limited to these.

1: Estimator,
10: First estimation part,
100: 1st encoder,
102: Second encoder,
104: Decoder,
12: Second estimation part,
120: Estimator,
122: Aggregator,
2: Training device,
20: Forward propagation part,
22: Error calculation unit,
24: Update department

Claims (27)

With one or more memories
With one or more processors,
The memory stores a trained model capable of estimating dynamics parameters.
The one or more processors
Encode the state and
Based on the dynamics parameters estimated by the trained model, the dynamics parameters of the environment are encoded.
Estimate behavior based on the encoded state and the encoded dynamics parameters.
Estimator. The one or more processors
Encoding the state and encoding the dynamics parameters are performed using different trained models.
The estimation device according to claim 1. By the one or more processors
The frequency of encoding the state is different from the frequency of encoding the dynamics parameter.
The estimation device according to claim 1 or 2. The one or more processors
Output the probability distribution for the behavior,
The estimation device according to any one of claims 1 to 3. The one or more processors
Output the probability distribution for the behavior using the trained model,
The estimation device according to claim 4. The one or more processors
The dynamics parameters of the environment are estimated based on the time series data of the state and the state as a result of acting on the state in the environment.
The estimation device according to any one of claims 1 to 5. The one or more processors
Divide the time series data into multiple chunks and divide it into multiple chunks.
Evaluate the distribution of the dynamics parameters corresponding to the chunks
Estimate the dynamics parameters based on the distribution of the plurality of evaluated dynamics parameters.
The estimation device according to claim 6. The one or more processors
Based on the time-series state and the time-series behavior in the chunk, an estimator of the dynamics parameter is obtained.
The dynamics parameters are estimated by synthesizing the estimated amounts of the dynamics parameters obtained from the plurality of chunks.
The estimation device according to claim 7. The one or more processors
Obtaining an estimator of the dynamics parameters using a trained model,
The estimation device according to claim 8. The one or more processors
Synthesize estimates of the dynamics parameters using a trained model,
The estimation device according to claim 8 or 9. With one or more memories
With one or more processors,
The one or more processors
Encode the state and
Encode dynamics parameters in a randomly selected environment
The first action is output based on the encoded state and the encoded dynamics parameters.
Based on the above-mentioned state and the state in which the first action is performed, what kind of second action is performed is estimated.
By comparing the first action with the second action, a model for estimating the first action is trained.
Training equipment. The one or more processors
Along with the model that estimates the first behavior, the model that estimates the second behavior is trained.
The training device according to claim 11. The one or more processors
Restore the encoded state and
Training a model that encodes the state by comparing the state with the restored state.
The training device according to claim 11 or 12. The one or more processors
Along with the model that encodes the state, train the model that restores the encoded state.
The training device according to claim 13. The one or more processors
Training a model that adds noise to the first action and encodes the state.
The training device according to claim 13 or 14. The one or more processors
Produces the noise based on the selected environment,
The training device according to claim 15. The one or more processors
Furthermore, a model for estimating the first behavior is trained by reinforcement learning.
The training device according to any one of claims 11 to 16. The one or more processors
Generate the dynamics parameters in multiple environments
Select the dynamics parameters that correspond to a randomly selected environment,
The training device according to any one of claims 11 to 17. The one or more processors
Fine tuning using the plurality of dynamics parameters.
The training device according to claim 18. The one or more processors
The dynamics parameters are estimated based on the time-series data of the state, the first action performed on the state, and the state after the first action is performed.
The training device according to claim 18 or 19. The one or more processors
Train a model that estimates the dynamics parameters based on the hypothesized prior distribution and the hypothesized posterior distribution.
The training device according to claim 20. One or more processors encode the state and
The one or more processors encode the dynamics parameters of the environment based on the dynamics parameters estimated by the trained model stored in one or more memories.
The one or more processors estimate behavior based on the encoded state and the encoded dynamics parameters.
Estimating method. One or more processors encode the state and
The one or more processors encode the dynamics parameters in a randomly selected environment.
The one or more processors output the first action based on the encoded state and the encoded dynamics parameters.
The one or more processors estimate what kind of second action is performed based on the state and the state in which the first action is performed.
The one or more processors train a model that estimates the first action by comparing the first action with the second action.
Training method. When executed by one or more processors,
Encode the state and
Encode the dynamics parameters of the environment based on the dynamics parameters estimated by the trained model stored in one or more memories.
Estimate behavior based on the encoded state and the encoded dynamics parameters.
program. When executed by one or more processors,
Encode dynamics parameters in a randomly selected environment
The first action is output based on the encoded state and the encoded dynamics parameters.
Based on the above-mentioned state and the state in which the first action is performed, what kind of second action is performed is estimated.
By comparing the first action with the second action, a model for estimating the first action is trained.
program. When executed by one or more processors,
Encode the state and
Encode the dynamics parameters of the environment based on the dynamics parameters estimated by the trained model stored in one or more memories.
Estimate behavior based on the encoded state and the encoded dynamics parameters.
A non-transitory computer-readable medium containing a program. When executed by one or more processors,
Encode dynamics parameters in a randomly selected environment
The first action is output based on the encoded state and the encoded dynamics parameters.
Based on the above-mentioned state and the state in which the first action is performed, what kind of second action is performed is estimated.
By comparing the first action with the second action, a model for estimating the first action is trained.
A non-transitory computer-readable medium containing a program.

Images