KR102147017B1 - Multi-agent PPO Guided By The Best Local Policy - Google Patents

Abstract

번의 훈련이 수행됨에 따라 각 에이전트의 성능 정보를 해당 에이전트로부터 수신하는 단계, 및 수신된 상기 각 에이전트의 성능 정보에 기초하여 결정된 베스트 에이전트의 정책 매개 변수를 상기 복수의 에이전트들이 공유하도록 제어하는 단계를 포함할 수 있다.The present invention relates to a PPO algorithm using an efficient policy parameter search method guided from the policy of the best agent in a multi-agent system. A method of controlling training of a policy parameter of each of a plurality of agents, the policy training step of controlling each of the plurality of agents to independently train based on a shared guidance policy, the following for each training: Receiving information of each of the plurality of agents from a corresponding agent in order to obtain variables to be used in training, and transmitting variables to be used for training to the plurality of agents after being determined based on the information of each agent for each training , Predefined

Receiving performance information of each agent from the corresponding agent as the training is performed, and controlling the plurality of agents to share the policy parameter of the best agent determined based on the received performance information of each agent. Can include.

Description

Multi-agent PPO Guided By The Best Local Policy}

Embodiments of the present invention are an exploration method of a policy parameter of a multi-agent system, and more particularly, a multi-agent PPO using an efficient policy parameter search method. It's about the algorithm.

의 매개변수

를 훈련하는 알고리즘으로, 업데이트 전 매개변수

에서 성능을 증가시키는 방향으로 조금씩

를 변화하여 업데이트하는 방법이다. 에이전트(agent)는 정책

를 이용하여 먼저 H번의 행동을 한 후, 그 동안의 상태 (state)

, 행동 (action)

, 보상 (reward)

을 모두 모은 경험

을 저장한다. Proximal Policy Optimization (PPO) was proposed as one of the reinforcement learning algorithms in the context of only one agent. PPO is a policy that has the best performance in a given environment

Parameters of

It is an algorithm that trains the parameters before update

Gradually in the direction of increasing the performance

It is a way to update by changing. Agent is a policy

First, H times, and then the state in the meantime

, Action

, Reward

Experience

Save it.

개의 관측, 행동, 보상, 그리고 다음 관측으로 이루어진 순서쌍

을 뽑아 수학식 1과 같은 정책의 목적 함수와 수학식 2와 같은 매개변수

로 구성된 가치 함수(value function)의 목적 함수를 계산한다.Randomly from accumulated experience

An ordered pair of observations, actions, rewards, and next observations

By pulling out, the objective function of the policy as in Equation 1 and the parameters as in Equation 2

Compute the objective function of a value function consisting of

[Equation 1]

[Equation 2]

는 순서쌍

에 대한 샘플 평균이며,

는 예정 정책 매개변수

로 구성된 정책이다. 또한

는 경험, 가치 함수

와, 변수

, _λ로 계산된 Generalized Advantage Estimator(GAE)를 나타내는 것으로서, Generalized Advantage Estimator(GAE)는 아래의 비특허 문헌 [1] Schulman, John, et al. "High-dimensional continuous control using generalized advantage estimation."　arXiv preprint arXiv:1506 .02438　(2015).에 자세히 나와있다.

는 현재 에피소드가 끝날 때까지의 discounted cumulated return으로

에 에피소드가 끝날 경우,

로 계산된다. 이후 에이전트는 목적 함수들의 합

를 증가시키는 방향으로 매개변수

와

를 Adam optimizer를 이용해 아래의 수학식 3과 같이 업데이트 한다. here

Is an ordered pair

Is the sample mean for

Is the expected policy parameter

It is a policy composed of. Also

Is an experience, value function

Wow, variable

As representing the Generalized Advantage Estimator (GAE) calculated as, _λ , the Generalized Advantage Estimator (GAE) is a non-patent document [1] Schulman, John, et al. "High-dimensional continuous control using generalized advantage estimation." arXiv preprint arXiv:1506 .02438 (2015). Detailed in

Is a discounted cumulated return until the end of the current episode.

If the episode ends on,

Is calculated as Then the agent is the sum of the objective functions

In the direction of increasing the parameter

Wow

Is updated as in Equation 3 below using the Adam optimizer.

[Equation 3]

개의 순서쌍을 임의로 뽑고, 뽑은 순서쌍을 이용해 목적 함수들을 계산하고 이들의 합을 증가시키는 방향으로 각 에이전트의 현재 정책 매개변수

와 가치 함수 매개변수

를 업데이트하는 과정을

진행하며 이것을 한번의 훈련이라고 부른다.From saved experiences

The current policy parameters of each agent in the direction of randomly picking two ordinal pairs, calculating the objective functions using the picked ordered pairs, and increasing the sum of them.

And value function parameters

The process of updating

It proceeds and is called a one-time training.

을 이용하여 평균값이 원하는 값

보다 1.5배 보다 크거나 1/1.5배 보다 작으면 변수

를 아래의 수학식 4와 같이 조정하여 다음 업데이트에서 KL divergence의 평균값을

과 비슷하도록 만든다. Sample mean value of KL divergence, which is the distance between the probability distribution function after one training

The average value is the desired value using

If greater than 1.5 times or less than 1/1.5 times the variable

By adjusting as in Equation 4 below, the average value of the KL divergence in the next update

Make it similar to

[Equation 4]

로

를 가까워지도록 하는 목적을 가지며, 이를

과 비슷하게 유지함으로써 학습의 안정성을 증가시키는 효과가 있다.The KL divergence term in Equation 1 is the policy before the update

in

Has the purpose of getting closer to

It has the effect of increasing the stability of learning by keeping it similar to.

를 조정하는 것을

번 반복한 후, 학습을 종료한다.Variable to use in the next training after one training

To adjust

After repeating once, the learning is ended.

만을 이용해서 자신의 행동

을 취하며, 자신의 경험만을 이용해 자신의 정책 매개변수

를 훈련한다. 예를 들어 PPO를 사용한 완전 분산적 다중 에이전트 알고리즘은 각 에이전트가 자신의 경험

을 모은 뒤, 이를 이용하여 아래의 수학식 5와 수학식 6의 목적 함수를 계산하고, 목적 함수의 합

을 증가시키는 방향으로 아담 최적화(Adam optimizer)를 이용하여

와

를 업데이트한다. Currently, in the field of reinforcement learning, in addition to research on reinforcement learning algorithms in an environment with only one agent, like the previous PPO algorithm, research on reinforcement learning algorithms in a multi-agent situation in which several agents cooperate with each other is actively being conducted. Unlike the reinforcement learning algorithm in a system with only one agent, the multi-agent reinforcement learning algorithm cannot use the assumption that each agent can see the state of all situations, so each agent has its own observation and behavior. However, it is assumed that only rewards can be obtained from the environment. Then, the multi-agent reinforcement learning algorithm can be classified by the presence or absence of information exchange with other agents in the training or execution stage, and each is classified into centralized and decentralized. The case where each agent trains and executes each agent's policy using only its own information is called fully decentralized. In this case, each agent has its own observation.

Your own actions using only

And use only your own experience to set your policy parameters

To train. For example, a fully distributed multi-agent algorithm using PPO allows each agent to

After collecting, using this, the objective function of Equations 5 and 6 below is calculated, and the sum of the objective functions

In the direction of increasing the value, using the Adam optimizer

Wow

Update.

[Equation 5]

[Equation 6]

는 각 에이전트의 정책 매개변수

로 구성된 정책이고,

는 업데이트 전 정책 매개변수

로 구성된 정책,

는 가치 함수 매개변수

로 구성된 가치 함수이다. 또한

는 자신의 경험의 샘플 평균이며,

는 자신의 경험과 자신의 가치 함수를 이용하여 계산된 GAE이다.In Equations 5 and 6,

Is the policy parameter of each agent

Is a policy consisting of

Is the pre-update policy parameter

A policy consisting of,

Is the value function parameter

It is a value function composed of Also

Is the sample mean of your own experience,

Is the GAE calculated using your own experience and your own value function.

However, this method does not exchange information for other agents, and considers it as the randomness of the environment, thus increasing the number of experiences required for convergence, and slowing the convergence speed. Also, since the convergence speed is slow, the performance after convergence is also low. In addition, there is a problem of a lazy agent that slows learning by an agent that is not able to learn if there is an agent that does well among the agents.

를 공유하여 하나의 정책

을 모든 에이전트가 사용하는 중앙집권적 훈련과 분산적 실행 방법이 있다. 기존에 많은 연구에서 사용하는 정책 매개변수 공유 방법은 모든 에이전트들이 같은 정책,

으로 행동을 한 후 얻은 경험을 중앙 시스템에 전송을 하며, 중앙 시스템에서는 모든 에이전트들로부터 전송된 경험을 이용하여 중앙 시스템의 가치 함수 매개변수

와 공유된 정책 매개변수

를 학습한다. 이 경우 중앙 시스템에서 학습된 정책 매개변수

는 다시 모든 에이전트들에게 전송이 되고, 다시 에이전트들은 공유된 정책을 이용해 경험을 얻는다. 이 방법은 한번의 훈련에 필요한 경험을 모든 에이전트들로부터 얻기 때문에 에이전트 수

배만큼 빠르게 경험을 모아 학습을 하지만, 경험과 정책 매개변수 모두 공유하는 방법은 상당한 통신량을 요구하며 통신량에 제한이 있는 실제 환경에서는 적용하기 힘들다. To solve the lazy agent problem, the policy parameters of all agents

Share one policy

There are centralized training and distributed execution methods used by all agents. The policy parameter sharing method used in many studies is that all agents have the same policy,

The experience gained after performing the action is transmitted to the central system, and the central system uses the experience transmitted from all agents to the value function parameter of the central system.

Policy parameters shared with

To learn. In this case, the policy parameters learned from the central system

Is sent back to all agents, again the agents gain experience using the shared policy. This method provides the necessary experience for one training session from all agents, so the number of agents

Experience is collected and learned twice as fast, but the method of sharing both experience and policy parameters requires a significant amount of communication and is difficult to apply in a real environment where the amount of communication is limited.

For an environment that requires cooperation, there is a problem of a lazy agent in which an agent continues to learn slowly, which cannot be a fully distributed method in which each agent trains and executes its own policy. Such a lazy agent slows down the learning rate, and ultimately lowers the overall performance. As a way to solve this problem, there is a method where all agents share the same policy and use it to take action, but in this case, the speed of searching for high-performance policy parameters is slow, and the final performance is low. In addition, the method of sharing all of the experiences and policies that are well known before requires a considerable amount of communication during training.

Therefore, there is a need for a technique that solves the problem of lazy agents and efficiently searches for policy parameters by using a small amount of communication.

[1] Schulman, John, et al. "High-dimensional continuous control using generalized advantage estimation." arXiv preprint arXiv: 1506.02438 (2015).

An embodiment of the present invention is to solve the lazy agent problem in the multi-agent system described above, and uses a small amount of communication at the same time, and is a centralized system that increases the learning speed and performance by efficiently searching for policy parameters. It is to provide a training and distributed execution PPO algorithm, a multi-agent training control method and system.

A method of controlling training of a policy parameter of each of a plurality of agents, the policy training step of controlling each of the plurality of agents to independently train based on a shared guidance policy, the following for each training: In order to obtain variables to be used in training, receiving information of each of the plurality of agents from a corresponding agent, transmitting variables to be used for training next to the plurality of agents determined based on the information of each agent for each training , Receiving the performance information of each agent from the corresponding agent as the training of the predefined R times is performed, and the plurality of agents share the policy parameter of the best agent determined based on the received performance information of each agent. It may include controlling to do so.

)의 매개변수(

)는 상기 복수의 에이전트들이 공유하는 상기 다음 훈련에 사용할 변수(

,

), 안내 정책(

), 및 각 에이전트의 경험 정보

에 기초하여 업데이트될 수 있다.According to one aspect, the current policy corresponding to each of the plurality of agents (

) Of the parameter (

) Is a variable to be used for the next training shared by the plurality of agents (

,

), information policy (

), and experience information of each agent

It can be updated based on.

)의 매개변수(

)는, 미리 정의된 해당 에이전트의 목표 함수를 상대적으로 증가시키는 방향으로 업데이트될 수 있다.According to another aspect, the current policy corresponding to each of the plurality of agents (

) Of the parameter (

) May be updated in a direction to relatively increase the target function of the corresponding agent.

)를 계산하는 단계를 포함할 수 있다.According to another aspect, the transmitting of the variable to be used for the next training to the plurality of agents may be used for the next training based on the KL divergence between the guidance policy and the current policy of each agent. variable(

) May include the step of calculating.

번의 훈련이 수행되면, 새로운 안내 정책을 선택하기 위해, 해당 에이전트의 성능을 나타내는 일정 개수(

)의 에피소드 보상의 평균(

)을 수신하는 단계를 포함할 수 있다.According to another aspect, the controlling of the plurality of agents to share the policy parameter includes, in each of the plurality of agents, the

Once training is performed, a certain number representing the performance of the agent in order to select a new guidance policy (

) Of the episode reward average (

) May include receiving.

)에 기초하여 상기 복수의 에이전트 중 평균이 가장 높은 에이전트를 상기 베스트 에이전트로 결정하는 단계, 및 결정된 상기 베스트 에이전트로 상기 정책 매개 변수(

)를 요청하는 단계를 더 포함할 수 있다.According to another aspect, the controlling of the policy parameter to be shared by the plurality of agents includes an average of the received episode rewards (

Determining an agent having the highest average among the plurality of agents as the best agent, and the policy parameter as the determined best agent (

) May further include requesting.

)를 상기 베스트 에이전트로부터 수신하는 단계, 안내 정책의 매개변수(

)를 수신된 상기 베스트 에이전트의 현재 정책에 해당하는 매개변수(

)로 설정하는 단계, 및 설정된 상기 안내 정책의 매개변수(

)를 상기 복수의 에이전트들로 전송하는 단계를 더 포함할 수 있다.According to another aspect, the controlling of the plurality of agents to share the policy parameter includes, in response to a request for the policy parameter, a parameter corresponding to the current policy of the best agent (

) From the best agent, the parameters of the guidance policy (

Parameter corresponding to the current policy of the best agent received (

), and a parameter of the set guide policy (

) May further include transmitting to the plurality of agents.

번의 훈련이 수행됨에 따라 각 에이전트의 성능 정보를 해당 에이전트로부터 수신하는 수신 제어부, 매 훈련 마다 각 에이전트의 정보를 기반으로 결정된 다음 훈련에 사용할 변수를 상기 복수의 에이전트에게 전송하는 전송 제어부, 및 수신된 상기 각 에이전트의 성능 정보에 기초하여 결정된 베스트 에이전트의 정책 매개 변수를 상기 복수의 에이전트들이 공유하도록 제어하는 공유 제어부를 포함할 수 있다.A system for controlling training of a policy parameter of each of a plurality of agents, comprising: a training control unit that controls each of the plurality of agents to independently train based on a previously shared guide policy; the next training for each training In order to obtain variables to be used in, the information of each of the plurality of agents is received from the agent, and

A reception control unit that receives performance information of each agent from the corresponding agent as training is performed, a transmission control unit that transmits variables to be used for training after being determined based on the information of each agent for each training to the plurality of agents, and received It may include a sharing control unit for controlling the plurality of agents to share the policy parameter of the best agent determined based on the performance information of each agent.

)의 매개변수(

)는 상기 복수의 에이전트들이 공유하는 상기 다음 훈련에 사용할 변수(

,

), 안내 정책(

), 및 각 에이전트의 경험 정보

에 기초하여 업데이트될 수 있다.According to one aspect, the current policy corresponding to each of the plurality of agents (

) Of the parameter (

) Is a variable to be used for the next training shared by the plurality of agents (

,

), information policy (

), and experience information of each agent

It can be updated based on.

)의 매개변수(

)는, 미리 정의된 해당 에이전트의 목표 함수를 상대적으로 증가시키는 방향으로 업데이트될 수 있다.According to another aspect, the current policy corresponding to each of the plurality of agents (

) Of the parameter (

) May be updated in a direction to relatively increase the target function of the corresponding agent.

)를 계산할 수 있다.According to another aspect, the transmission control unit, based on the KL divergence between the guidance policy and the current policy of each agent, a variable to be used for the next training (

) Can be calculated.

번의 훈련이 수행되면, 새로운 안내 정책을 선택하기 위해, 해당 에이전트의 성능을 나타내는 일정 개수(

)의 에피소드 보상의 평균(

)을 수신할 수 있다.According to another aspect, the shared control unit, in each of the plurality of agents

Once training is performed, a certain number representing the performance of the agent in order to select a new guidance policy (

) Of the episode reward average (

) Can be received.

)에 기초하여 상기 복수의 에이전트 중 평균이 가장 높은 에이전트를 상기 베스트 에이전트로 결정하고, 결정된 상기 베스트 에이전트로 상기 정책 매개 변수(

)를 요청할 수 있다.According to another aspect, the sharing control unit, the average of the received episode compensation (

), the agent having the highest average among the plurality of agents is determined as the best agent, and the policy parameter (

) Can be requested.

)를 상기 베스트 에이전트로부터 수신하고, 안내 정책의 매개변수(

)를 수신된 상기 베스트 에이전트의 현재 정책에 해당하는 매개변수(

)로 설정하고, 설정된 상기 안내 정책의 매개변수(

)를 상기 복수의 에이전트들로 전송할 수 있다.According to another aspect, in response to the request for the policy parameter, the sharing control unit includes a parameter corresponding to the current policy of the best agent (

) From the best agent, and the parameters of the guidance policy (

Parameter corresponding to the current policy of the best agent received (

), and the parameters of the set guide policy (

) Can be transmitted to the plurality of agents.

By sharing the policy of the best agent, an agent with poor performance can train its policy parameters, and the performance can be increased more quickly by being guided by the parameters of other agents with the best performance. Agent problems can be solved.

In addition, by learning the policy of each agent from the shared parameter without updating the policy parameter of each agent to the same shared parameter, it has the effect of simultaneously searching several parts in the policy parameter space. By searching around good parameters, you can quickly find good parameters.

In addition, a significantly smaller amount of communication is required compared to a method of sharing both experiences and policies known in the past, and communication costs can be greatly reduced.

일 때의 Water-World 환경의 한 프레임을 나타낼 수 있다.
도 4는 본 발명의 일실시예에 있어서,

일 때의 Multi-Walker 환경의 한 프레임을 나타낼 수 있다.
도 5는 본 발명의 일실시예에 있어서, Water-World 환경에서의 시뮬레이션으로 얻은 시간에 따른 정책의 최근 에피소드들의 평균 에피소드 보상을 나타내는 그래프이다.
도 6은 본 발명의 일실시예에 있어서, Multi-Walker 환경에서의 시뮬레이션으로 얻은 시간에 따른 정책의 최근 에피소드들의 평균 에피소드 보상을 나타내는 그래프를 나타낼 수 있다.
도 7은 본 발명의 실시 예에 있어서, Water-World 환경의 에이전트 수

로 변화하였을 때의 시뮬레이션으로 얻은 훈련을 마친 후의 최근 에피소드들의 평균 에피소드 보상을 나타내는 그래프에 해당할 수 있다.
도 8은 Water-World 환경에 완전 분산 PPO 알고리즘을 이용한 시뮬레이션으로 얻은 각 에이전트의 시간에 따른 베스트 에이전트로 뽑힌 비율 나타내는 그래프이다.
도 9는 Water-World 환경에 본 발명에서 제안하는 알고리즘을 이용한 시뮬레이션으로 얻은 각 에이전트의 시간에 따른 베스트 에이전트로 뽑힌 비율을 나타내는 그래프이다.
도 10은 경험과 정책을 모두 공유하는 방법과 본 발명에서 제안하는 방법의 한번의 훈련에 필요한 통신량의 비율을 나타내는 그래프이다.1 is a flowchart illustrating an operation of a method for controlling multi-agent training according to an embodiment of the present invention.
2 is a block diagram showing the internal configuration of a multi-agent training control system according to an embodiment of the present invention.
3 is an embodiment of the present invention,

It can represent one frame of the Water-World environment when
4 is an embodiment of the present invention,

It can represent one frame of the Multi-Walker environment at the time of.
5 is a graph showing average episode compensation of recent episodes of a policy over time obtained by simulation in a water-world environment according to an embodiment of the present invention.
6 is a graph showing average episode compensation of recent episodes of a policy over time obtained by simulation in a multi-walker environment according to an embodiment of the present invention.
7 is the number of agents in the Water-World environment in an embodiment of the present invention

It may correspond to a graph showing the average episode compensation of recent episodes after training obtained by the simulation when changed to.
8 is a graph showing the ratio of each agent selected as the best agent over time obtained through a simulation using a fully distributed PPO algorithm in a Water-World environment.
9 is a graph showing the ratio of each agent selected as the best agent over time obtained by simulation using the algorithm proposed by the present invention in a water-world environment.
10 is a graph showing the ratio of the amount of communication required for one training of the method of sharing both experience and policy and the method proposed in the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

만 공유하면, 에이전트의 정책 매개변수를 학습하는 방법은 완전 분산적 학습 알고리즘과 동일하게 학습하지만, 매

번의 훈련이 끝난 후, 그 중 한 에이전트의 정책 매개변수

를 공유하여 모든 에이전트들이

번의 훈련이 지날 때마다 같은 정책

을 가질 수 있다. 이 경우 매

번의 학습이 끝난 후, 중앙시스템은 모든 에이전트들에게 공유될 에이전트를 선택하고, 선택된 에이전트에게 정책 매개변수를 요청할 수 있다. 그러면, 요청 받은 에이전트는 현재 정책의 매개변수를 중앙시스템에 전송하고, 중앙시스템은 수신한 정책 매개변수를 모든 에이전트들에게 전송할 수 있다. 그러면, 모든 에이전트들이 공유된 정책을 직접적으로 사용하여, 에이전트들이 행동을 취할 때 사용할 수 있다. 이처럼, 모든 에이전트들이 경험을 공유하는 것보다 적은 통신량이 필요하며, 미리 정의된 기준 횟수인

번의 훈련마다 하나의 매개변수로 초기화되어 정책을 학습하며, 결과적으로 정책 매개변수 공간 상에서 하나의 매개변수로부터만 탐색을 진행하게 될 수 있다. 이런 제한적인 매개변수의 탐색은 정책을 느리게 학습시키므로, 효율적인 탐색 방법을 적용하여, 학습 속도를 증가시키고, 결과적으로 기존 방법보다 높은 성능을 제공하고자 한다.The present invention relates to a multi-agent PPO algorithm guided by a policy of a best agent, a method and a system for controlling multi-agent training, and relates to a technique of selecting a best agent and controlling the policy parameters of the best agent to be shared by all agents. For example, without sharing the experience of each agent, the parameters of the policy

If only shared, the method of learning the agent's policy parameters is the same as the fully distributed learning algorithm, but

After training, one of the agent's policy parameters

And all agents

Same policy every time training passes

Can have In this case every

After each learning is completed, the central system can select an agent to be shared with all agents and request policy parameters from the selected agent. Then, the requested agent transmits the parameters of the current policy to the central system, and the central system can transmit the received policy parameters to all agents. Then, all agents can use the shared policy directly, which can be used by agents to take action. In this way, less communication is required than all agents share their experiences, and

Each training is initialized with one parameter to learn the policy, and as a result, the search can proceed from only one parameter in the policy parameter space. Since the search for such a limited parameter learns the policy slowly, an efficient search method is applied to increase the learning speed, and as a result, it is intended to provide higher performance than the existing method.

In the present embodiments, "agent" refers to a user terminal and may refer to, for example, an autonomous vehicle, an electronic device forming a smart city, and the like.

In the present embodiments, the "central system" is an electronic device for integrated control and management of a plurality of agents, and may be expressed as a multi-agent training control system.

명의 에이전트가 각각 관측의 차원

, 행동의 차원

을 가지고 있다고 가정한다. 또한 각 에이전트는 자신의 정책 매개변수

, 가치 함수 매개변수

를 가지고 있으며, 모든 에이전트는 자신의 정책의 훈련에 필요한 변수

,

와 안내 정책의 매개변수

를 공유하고 있다고 가정한다. The present invention

2 agents each observation dimension

, The dimension of action

Suppose you have In addition, each agent has its own policy parameters

, Value function parameter

And all agents need variables to train their policies

,

And guiding policy parameters

Suppose you are sharing

들은 입력 차원이

, 출력 차원

이며, 뉴런의 개수가 64개인 은닉 층

개를 가지는 심층 신경망(multi-layer perceptron)의 매개변수로, 정책 매개변수를 이용하여 확률적 정책(stochastic policy)이 구성될 수 있다. 관측

에 대한 구성된 심층 신경망의 출력을

차원의 벡터

,

로 하여, 행동

에 대한 확률을 아래의 수학식 9와 같이 정규 분포 함수로 정의할 수 있다. Policy parameter

The input dimension

, Output dimension

Is a hidden layer with 64 neurons

As a parameter of a multi-layer perceptron with dogs, a stochastic policy can be constructed using policy parameters. observation

The output of the constructed deep neural network for

Vector of dimensions

,

By action

Probability for can be defined as a normal distribution function as shown in Equation 9 below.

[Equation 9]

로 자신의 정책

를 구성하고, 자신의 관측

에 대한 행동

를 자신의 정책

로부터 샘플링(sampling)할 수 있다. 또한 공유된 매개변수

를 이용하여 안내 정책

를 구성하여 학습에 사용할 수 있다. Policy parameters each agent has

As your own policy

Construct and observe your own

Action on

Own policy

It can be sampled from Also shared parameters

Using the guide policy

Can be used for learning.

들은 입력 차원이

, 출력 차원이 1이며, 뉴런의 개수가 64개인 은닉 층 2개를 가지는 심층 신경망의 매개변수로, 관측

에 대한 가치를

로 예측할 수 있다.Value function parameter

The input dimension

, As a parameter of a deep neural network with two hidden layers with an output dimension of 1 and 64 neurons.

Value for

Can be predicted.

1 is a flowchart showing the operation of a method for controlling multi-agent training according to an embodiment of the present invention, and FIG. 2 is a block diagram showing an internal configuration of a multi-agent training control system in an embodiment of the present invention. .

That is, FIG. 1 may correspond to a flow chart showing each step of a method for independently training agents using a guide policy in a multi-agent training control system, which is a central system. Further, according to FIG. 2, the multi-agent training control system 200 may include a training control unit 210, a reception control unit 220, a transmission control unit 230, and a shared control unit 240. Each of the steps of FIG. 1 (that is, steps 110 to 190) is a training control unit 210, a reception control unit 220, a transmission control unit 230, and a shared control unit, which are components of the multi-agent training control system 200 of Fig. 2 It can be done by 240.

,

와 안내 정책의 매개변수

가 초기화될 수 있다. 그러면, 초기화된 안내 정책의 매개변수는 중앙시스템인 다중 에이전트 훈련 제어 시스템(200)에 속하는 모든 에이전트들이 공유할 수 있다.In step 110, before starting training in a plurality of agents managed by the multi-agent training control system 200, which is a central system, the policy parameters of all agents and the parameters of the guide policy shared with the same arbitrary parameter Can be initialized. For example, variables needed for training of one's policy

,

And guiding policy parameters

Can be initialized. Then, the parameters of the initialized guide policy can be shared by all agents belonging to the multi-agent training control system 200, which is a central system.

를 이용하여 _{미리 지정된}

번의 행동을 취하며, 행동을 취하는 동안의 경험을 나타내는 경험 정보

를 각자 자신의 저장 장치(예컨대, memory)에 저장할 수 있다. 이때,

번의 행동을 취하는 도중, 에피소드(episode)가 종료되고 다시 처음 상황으로 돌아가면 그 동안의 보상의 합이 각 에이전트 별로 저장될 수 있으며, 저장된 각 에이전트 별 보상의 합은 에피소드 보상으로 표현될 수 있다. In step 120, each agent

_{Pre-specified} using

Taking an action, and experiential information representing the experience during the action

Each of them can be stored in their own storage device (eg, memory). At this time,

In the middle of taking an action, when the episode ends and returns to the initial situation, the sum of the rewards during that time can be stored for each agent, and the sum of the saved rewards for each agent can be expressed as an episode reward.

를 예전 정책 매개변수

로 설정할 수 있다. 이처럼, 훈련을 시작하기 전에 요구되는 초기화, 안내 정책 공유 등의 전처리가 수행되면, 훈련을 위한 프로세스가 시작될 수 있다.Before proceeding with training, each agent will have the current policy parameters

Old policy parameters

Can be set to In this way, if the required pre-processing such as initialization and guide policy sharing is performed before starting training, a process for training can be started.

In step 130, the training control unit 210 may control each of the plurality of agents to independently perform training (ie, policy training) based on a shared guidance policy.

를 예전 정책 매개변수

로 설정한 이후, 각 에이전트들은 자신의 경험 중 임의로

개의 관측, 행동, 보상, 그리고 다음 관측으로 이루어진 순서쌍

을 추출하여 아래의 수학식 10, 수학식 11에 기초하여 정책의 목적 함수

, 가치 함수의 목적 함수

를 계산할 수 있다. 이때, 복수의 에이전트들 각각에 해당하는 현재 정책

의 매개변수

는 복수의 에이전트들이 현재 공유하고 있는 변수(

,

), 안내 정책

, 및 각 에이전트의 경험 정보

에 기초하여 업데이트될 수 있다. 예컨대, 상기 현재 정책

의 매개변수

는 미리 정의된 해당 에이전트의 목적 함수를 상대적으로 증가시키는 방향으로 업데이트될 수 있다. 즉, 현재 정책 매개변수

와 가치 함수 매개변수

는 아담 최적화(Adam optimizer)에 기초하여 목적 함수의 합

을 상대적으로 증가시키는 방향으로 업데이트될 수 있다.As an example, each agent has the current policy parameters

Old policy parameters

After setting to, each agent randomly

An ordered pair of observations, actions, rewards, and next observations

Is extracted and the objective function of the policy based on Equations 10 and 11 below

, The objective function of the value function

Can be calculated. At this time, the current policy for each of the plurality of agents

Parameters of

Is a variable currently shared by multiple agents (

,

), information policy

, And experience information of each agent

It can be updated based on. For example, the above current policy

Parameters of

May be updated in a direction to relatively increase the predefined objective function of the corresponding agent. That is, the current policy parameter

And value function parameters

Is the sum of the objective functions based on the Adam optimizer

It can be updated in a direction of relatively increasing.

[Equation 10]

[Equation 11]

는 순서쌍

에 대한 샘플 평균을 나타내고,

는 예정 정책 매개변수

로 구성된 예전 정책

을 나타내고,

는 가치 함수 매개변수

로 구성된 가치 함수를 나타낼 수 있다.

는 현재 에피소드가 끝날 때까지의 discounted cumulated return으로

에 에피소드가 끝날 경우,

로 계산될 수 있다. 그리고,

는 해당 에이전트 자신의 경험을 나타내는 경험 정보와 해당 에이전트 자신의 가치 함수로 계산된 GAE(Generalized Advantage Estimator)를 나타낼 수 있다. 이처럼, 에이전트 자신의 경험으로부터

개의 순서쌍을 임의로 추출하고, 추출한 순서쌍을 기반으로 목적 함수를 계산 및 증가하는 방향으로 각 에이전트의 현재 정책 매개변수

를 업데이트하는 과정이

수행될 수 있으며, 이러한 과정을 한 번의 훈련(training)이라고 정의할 수 있다. here

Is an ordered pair

Represents the sample mean for,

Is the expected policy parameter

Old policy consisting of

Represents,

Is the value function parameter

We can represent a value function composed of

Is a discounted cumulated return until the end of the current episode.

If the episode ends on,

Can be calculated as And,

May represent experience information representing the agent's own experience and a GAE (Generalized Advantage Estimator) calculated as a function of the agent's own value. Like this, from the agent's own experience

Randomly extracts the ordered pairs, calculates the objective function based on the extracted ordered pairs, and increases the current policy parameters of each agent

The process of updating

It can be performed, and this process can be defined as a single training.

과 해당 에이전트의 현재 정책

사이의 거리를 나타내는 쿨백 라이블러 발산(KL divergence)에 기초하여 다음 훈련에 사용할 변수

와

를 계산할 수 있다.In step 140, the transmission control unit 230 may transmit a variable to be used for training to a plurality of agents after it is determined based on the information of each agent received for each training. At this time, the reception control unit 220 transmits information of each of the plurality of agents (e.g., KL divergence information) of each agent in order to obtain variables to be used in the next training for each training. Can be received from Then, the transmission control unit 230 is a guide policy of each of the plurality of agents

And the current policy of that agent

Variable to use for the next training based on the KL divergence, which represents the distance between

Wow

Can be calculated.

과 현재 정책

사이의 거리를 나타내는 쿨백 라이블러 발산(KL divergence)의 샘플 평균

과 모든 에이전트 간에 공유된 안내 정책

을 중앙시스템인 다중 에이전트 훈련 제어 시스템(200)으로 전송할 수 있다. 그러면, 전송 제어부(230)는 수신된 상기 쿨백 라이블러 발산(KL divergence)의 샘플 평균(즉, 복수의 에이전트들 각각에 해당하는 KL divergence의 평균) 및 아래의 수학식 12에 기초하여 다음 훈련에 사용할 변수

와

를 계산할 수 있다. 그러면, 계산된 변수

와

가 복수의 에이전트들로 전송되어 공유될 수 있다.For example, after one training is conducted on each of a plurality of agents, all agents have their previous guidance policy (i.e., the previous policy)

And current policy

Sample mean of KL divergence representing the distance between

Guidance policy shared between users and all agents

Can be transmitted to the multi-agent training control system 200, which is a central system. Then, the transmission control unit 230 performs the next training based on the received sample average of the KL divergence (that is, the average of the KL divergence corresponding to each of the plurality of agents) and Equation 12 below. Variable to use

Wow

Can be calculated. Then, the calculated variable

Wow

May be transmitted to multiple agents and shared.

[Equation 12]

와

가 중앙시스템인 다중 에이전트 훈련 제어 시스템(200)에 속하는 모든 에이전트들에게 전송됨에 따라, 상기 변수

와

가 각 에이전트의 다음 훈련에 이용될 수 있다. As such, the newly determined variable to be used for the next training

Wow

As is transmitted to all agents belonging to the multi-agent training control system 200, which is a central system, the variable

Wow

Can be used for the next training of each agent.

, 150, 예), 160 단계에서, 수신 제어부(220)는 각 에이전트의 성능 정보를 해당 에이전트로부터 수신할 수 있다. 즉, 수신 제어부(220)는 새로운 안내 정책을 선택하기 위해, 해당 에이전트의 성능을 나타내는 일정 개수(

)의 에피소드 보상의 평균(

)을 각 에이전트로부터 수신할 수 있다. In step 150, if the number of training is a multiple of R, which is a predefined reference number (

In operation 150, yes) and 160, the reception control unit 220 may receive performance information of each agent from the corresponding agent. That is, in order to select a new guidance policy, the reception control unit 220 determines a certain number (

) Of the episode reward average (

) Can be received from each agent.

번의 훈련이 수행되면(즉, 훈련 횟수가 기준 횟수인 R의 배수가 되면), 모든 에이전트들은 각자 저장해놓은 미리 지정된 일정 개수

의 에피소드 보상의 평균

을 계산하여, 중앙시스템인 다중 에이전트 훈련 제어 시스템(200)으로 전송할 수 있다. 여기서, 상기 일정 개수

의 에피소드 보상의 평균

은 각 에이전트의 메모리(memory)에 저장된 복수의 에피소드들 중 현재 시점을 기준으로 가장 가까운 순서로 미리 지정된 일정 개수의 에피소드 보상을 기반으로 계산된 평균을 나타낼 수 있다. 즉, 최근 에피소드 보상들의 평균에 해당할 수 있다.For example, in each of the plurality of agents,

When one training is performed (i.e., when the number of training is a multiple of R, which is the reference number), all agents each store a predetermined number of presets.

Average of episode rewards

Can be calculated and transmitted to the multi-agent training control system 200, which is a central system. Here, the predetermined number

Average of episode rewards

May represent an average calculated based on compensation for a predetermined number of episodes preset in the nearest order based on a current time point among a plurality of episodes stored in the memory of each agent. That is, it may correspond to the average of recent episode rewards.

On the other hand, in step 150 again, if the number of training is not a multiple of R, which is a predefined reference number (150, No), it goes back to step 120 and each agent performs H times based on the agent's policy. , You can store your own experience information that occurred as you performed the action.

In step 170, the sharing control unit 240 may control a plurality of agents to share the policy parameter of the best agent determined based on the received performance information of each agent.

에 기초하여 최대 성능을 가지는 에이전트를 베스트 에이전트로 선택/결정하고, 베스트 에이전트에 정책 매개 변수를 요청할 수 있다. In step 171, the sharing control unit 240 is the average of the received episode compensation

Based on, the agent with the maximum performance may be selected/determined as the best agent, and a policy parameter may be requested from the best agent.

에 기초하여 복수의 에이전트들 중 평균이 가장 높은 에이전트를 베스트 에이전트로 결정할 수 있다. 그리고, 결정된 베스트 에이전트로 해당 에이전트의 현재 정책의 매개 변수를 정책 매개변수로서 요청할 수 있다.For example, the sharing control unit 240 is the average of the received episode compensation

Based on, an agent having the highest average among a plurality of agents may be determined as the best agent. And, the determined best agent may request a parameter of the current policy of the corresponding agent as a policy parameter.

를 수신할 수 있다.In step 172, the sharing control unit 240 is a policy parameter from the best agent

Can receive.

를 전송할 수 있다. 그러면, 공유 제어부(240)는 베스트 에이전트로부터 수신한 베스트 에이전트의 현재 정책의 매개변수

를 안내 정책의 매개변수

로 설정할 수 있다(

). 그리고, 공유 제어부(240)는 새로이 설정된 안내 정책의 매개변수(

)를 복수의 에이전트들로 전송할 수 있다.For example, the best agent that has received a request for policy parameters from the central system, in response to the request, sends the parameters of its current policy to the central system.

Can be transmitted. Then, the sharing control unit 240 is a parameter of the current policy of the best agent received from the best agent.

Guide to the parameters of the policy

Can be set to (

). In addition, the sharing control unit 240 is a parameter of the newly set guide policy (

) Can be transmitted to multiple agents.

In step 180, if the number of training is greater than the predetermined maximum number T (180, YES), training may be terminated.

에 따라 각 에이전트는 H번의 행동을 취하고, 행동 수행으로 인해 발생한 그 동안의 경험을 나타내는 경험 정보를 자신의 메모리에 저장할 수 있다. 다시 각 에이전트의 예전 정책 매개변수

를 현재 정책 매개변수

로 업데이트하고, 각 에이전트 별로 저장된 경험 정보를 기반으로 각자의 정책 매개변수

가 업데이트될 수 있다. 이어, 에이전트와 중앙시스템 간의 통신을 통해(140, 160, 170, 180) 다음 훈련에 사용할 변수

와

및 안내 정책의 매개변수

가 업데이트될 수 있다. 이러한 과정은 총 T번의 훈련이 마무리 될 때까지 반복 수행될 수 있다.And, if it is less than the maximum number of times T (180, no), it goes back to step 120, and after each agent performs H actions based on the agent's policy, it stores their own experience information that occurred as the action was performed. I can. That is, when communication between the agent and the central system is completed, the current policy of each newly created agent

As a result, each agent can take H actions and store experience information representing the past experiences that occurred due to the action performance in its own memory. Again each agent's old policy parameters

The current policy parameter

And each agent's own policy parameters based on the stored experience information

Can be updated. Next, through communication between the agent and the central system (140, 160, 170, 180), variables to be used for the next training

Wow

And parameters of the guidance policy

Can be updated. This process can be repeated until a total of T trainings are completed.

를 조정하여 베스트 에이전트의 정책의 매개변수

에서 약

만큼 떨어진 매개변수 공간에서

를 학습하도록 제어할 수 있다. 이에 따라, 업데이트 전 정책(즉, 예전 정책)

과 업데이트 된 현재 정책

사이의 KL divergence와 모든 에이전트들에게 공유된 안내 정책

와 에이전트의 현재 정책

사이의 KL divergence는 정책 매개변수 학습에 서로 다른 영향을 줄 수 있다. 이를 통해,

와

사이의 KL divergence는 에이전트의 정책 변화 정도를

수준으로 조정하기 위해 이용될 수 있으며, 결국, 학습을 안정적이게 수행되도록 할 수 있다. 그리고,

와

사이의 KL divergence는 가장 잘하는 정책과 비슷한 정책이 되도록 에이전트의 정책

을 학습하기 위해 이용될 수 있으며,

만큼 다른 위치에서 좋은 정책을 찾도록 할 수 있다.As described above, the present invention adds a KL divergence term to be similar to the policy of the best agent in addition to the target function of the fully distributed multi-agent PPO algorithm, and the KL of the policy of the best agent and the current policy of another agent. Variables in the divergence term

The parameters of the best agent's policy

From about

In the parameter space away by

Can be controlled to learn. Accordingly, the pre-update policy (i.e. the old policy)

And updated current policy

KL divergence between and guidance policy shared with all agents

And agent's current policy

The KL divergence between can have different effects on the learning of policy parameters. because of this,

Wow

The KL divergence between the agent's policy changes

It can be used to adjust to the level, and in the end, it can ensure that learning is performed stably. And,

Wow

The KL divergence between the agent's policy so that the policy is similar to the

Can be used to learn

As long as you can find good policies in different locations.

Hereinafter, with reference to FIGS. 3 to 10, it will be described that the multi-agent PPO algorithm guided to the policy of the best agent through simulation has superior performance compared to the method of sharing both experience and policy.

As the simulation environment, an environment called Water-World and an environment called Multi-Walker were used. The Water-World environment is an environment where the agent avoids moving poisons during 500 actions (e.g., during 500 timesteps) and catches the invaders, -1 reward for the agent hitting the poison, +0.01 reward for hitting the invader, and If two or more agents collide with the invader and catch the invader, a reward equal to 20/the number of agents that caught the invader can be provided to the caught agents.

일 때의 Water-World 환경의 한 프레임을 나타내고, 도 4는 본 발명의 일실시예에 있어서,

일 때의 Multi-Walker 환경의 한 프레임을 나타낼 수 있다.3 is an embodiment of the present invention,

It shows one frame of the Water-World environment at the time of, and FIG. 4 is in an embodiment of the present invention,

It can represent one frame of the Multi-Walker environment at the time of.

In FIG. 3, blue may correspond to an agent, red may correspond to poison, light green may correspond to an invader, and dark green may correspond to obstacles. Multi-Walker environment is an environment in which the agent does not fall and moves the bar above the head forward. When the agent falls, the agent is rewarded with -100, and one episode ends, and when the bar falls to the ground, -100 to all agents. It can give rewards and indicate an environment in which an episode ends. At this time, one episode may end after 500 timesteps previously specified after the environment starts. When the bar moves forward or backward, 130 times the bar's movement speed can be rewarded to all agents.

Multi-Walker의 경우

이 모든 알고리즘에 공통으로 사용되었다. 그리고,

가 모든 알고리즘의 시뮬레이션에 사용되었다.In Figure 4, the remaining variables are in the case of Water-World

For Multi-Walker

It was used in common with all of these algorithms. And,

Was used for simulation of all algorithms.

FIG. 5 is a graph showing average episode compensation of recent episodes of a policy over time obtained by simulation in a water-world environment, according to an embodiment of the present invention. FIG. -A graph showing the average episode compensation of recent episodes of the policy over time obtained by simulation in the Walker environment can be displayed.

를 이용한 학습방법(proposed), 완전 분산 학습방법(Dec. PPO), 경험은 공유하지 않고 매 R번의 훈련 후 베스트 에이전트의 정책 π만을 모든 에이전트가 공유하여 직접적으로 사용하는 방법(Dir. Best Pol. Sharing PPO), 그리고 경험과 정책을 모두 공유하는 방법(Exp. Sharing PPO)의 훈련에 따른 평균 에피소드 보상을 나타낼 수 있다.The graphs shown in FIGS. 5 and 6 are guide policies proposed by the present invention.

Method (proposed), fully distributed learning method (Dec. PPO), using only the best agent's policy π after each R training session without sharing experiences (Dir. Best Pol. Sharing PPO), and how to share both experiences and policies (Exp.Sharing PPO).

축으로 N배 늘인 그래프에 해당할 수 있다. 제안하는 안내 정책

를 이용한 학습방법(proposed)을 이용하는 경우, 정책을 모든 에이전트들이 공유하여 직접적으로 사용하는 경우보다 매개변수 탐색을 효율적으로 하며 결과적으로 높은 성능을 가짐을 확인할 수 있다.According to FIGS. 5 and 6, in the case of a method of sharing both experience and policy, training speed is fast because N times experience is gathered quickly, but in order to compare the efficiency of search for policy parameters

It can correspond to a graph stretched by N times as an axis. Proposed Guidance Policy

In the case of using the proposed method using, it can be seen that the parameter search is more efficient than when the policy is shared by all agents and used directly, and as a result, it has high performance.

로 변화하였을 때의 시뮬레이션으로 얻은 훈련을 마친 후의 최근 에피소드들의 평균 에피소드 보상을 나타내는 그래프에 해당할 수 있다.7 is the number of agents in the Water-World environment in an embodiment of the present invention

It may correspond to a graph showing the average episode compensation of recent episodes after training obtained by the simulation when changed to.

일 때 학습 완료 후, 최근 에피소드들의 평균 에피소드 보상을 나타낼 수 있다.That is, the graph shown in FIG. 7 is a water-world environment.

When is, after completion of learning, the average episode compensation of recent episodes may be indicated.

8 is a graph showing the ratio of each agent selected as the best agent over time obtained by simulation using a fully distributed PPO algorithm in a water-world environment, and FIG. 9 is a simulation using the algorithm proposed by the present invention in a water-world environment. This is a graph showing the percentage of each obtained agent selected as the best agent over time.

일 때, 주어진 timestep 동안 가장 높은 평균 에피소드 보상을 가지고 있는 에이전트로 뽑힌 비율을 도시하고 있으며, 각각 완전 분산 학습을 이용한 PPO 방법과 안내 정책을 이용한 PPO 방법에 대해 도시하고 있다. 도 8 및 도 9를 참고하면, 완전 분산 학습 방법은 잘하는 에이전트가 계속 잘하는 현상이 발생하여 게으른 에이전트 문제를 겪고 있지만, 본 발명에서 제안하는 방법(안내 정책

를 이용한 학습방법)은 잘하는 에이전트(즉, 베스트 에이전트)가 계속 바뀌며, 최종적으로는 모든 에이전트가 비슷한 비율로 베스트 에이전트로 뽑혀, 게으른 에이전트 문제를 해결함을 확인할 수 있다. 그리고, 도 10을 참고하면, 경험과 정책을 모두 공유하는 방법은 본 발명에서 제안하는 방법(안내 정책

를 이용한 학습방법)에 비해 상당한 통신량을 요구함을 확인할 수 있다. 이때, 도 10에 도시된 그래프는 한번의 훈련에 필요한 통신량의 비율을 나타낸 것으로, 경험과 정책을 모두 공유하는 방법의 통신량과 본 발명에서 제안하는 방법의 통신량의 Water-World와 Multi-Walker 환경에 대한 비율을 나타낼 수 있다. The graphs of FIGS. 8 and 9 are in the Water-World environment.

When is, the ratio of the agent selected as the agent with the highest average episode reward during a given timestep is shown, respectively, showing the PPO method using fully distributed learning and the PPO method using guidance policy. Referring to FIGS. 8 and 9, in the fully distributed learning method, a good agent continues to do well, and thus a lazy agent problem occurs, but the method proposed by the present invention (guide policy

As for the learning method using ), it can be confirmed that the agent that does well (that is, the best agent) is constantly changing, and finally all agents are selected as the best agent at a similar rate, solving the lazy agent problem. And, referring to FIG. 10, a method of sharing both experiences and policies is a method proposed by the present invention (guide policy

It can be seen that it requires a considerable amount of communication compared to the learning method using In this case, the graph shown in FIG. 10 shows the ratio of the amount of communication required for one training, and in the Water-World and Multi-Walker environment of the communication amount of the method of sharing both experience and policy and the amount of communication of the method proposed in the present invention. It can represent the ratio for.

The method of the embodiment may be recorded in a computer-readable medium in the form of program instructions executed through computer means. The computer-readable medium includes program instructions, data files, data structures, and the like. The program instructions recorded in the medium may be designed for an exemplary embodiment or may be known to and usable by those skilled in computer software. Examples of computer-readable recording media include hard disks, floppy disks, magnetic media, optical media such as CD-ROM and DVD, and magnetic-optical media such as floptical disks. ), ROM, RAM, and a hardware device configured to store and execute program instructions such as flash memory. Examples of program instructions include high-level language codes of a computer using an interpreter or the like.

As described above, the embodiments have been described with reference to limited embodiments and drawings, but various modifications and variations are possible from the above description. For example, the embodiment may be modified by changing the order of configurations or combinations.

Therefore, other implementations, other embodiments, and those equivalent to the claims also fall within the scope of the claims to be described later.

Claims (14)

In the method of controlling the training of each policy parameter of a plurality of agents performed using a system for controlling the training of each policy parameter of a plurality of agents,
A policy training step of controlling the training of the plurality of agents to independently train each of the plurality of agents based on a guide policy shared in advance for each of the plurality of agents;
The receiving control unit is the variable to be used in the next training for each training (

,

Receiving information of each of the plurality of agents from a corresponding agent to obtain );
The transmission control unit is determined based on the information of each agent for each training, and then the variable to be used for training (

,

Transmitting) to the plurality of agents;
The reception control unit is a predefined

Receiving performance information of each agent from the corresponding agent as the training is performed once; And
The shared control unit is the policy parameter of the best agent determined based on the received performance information of each agent (

) To be shared by the plurality of agents
Including,
Each agent has its own policy parameter (a parameter of a multi-layer perceptron).

), and the policy parameter (

), and the variables required for training of the policies (

,

) And the parameters of the guidance policy (

), and the parameters of the shared guidance policy (

) To construct a guide policy and use it for training
Multi-agent training control method, characterized in that. The method of claim 1,
Current policy corresponding to each of the plurality of agents (

) Of the parameter (

) Is a variable to be used for the next training shared by the plurality of agents (

,

), information policy (

), and experience information of each agent

Is updated based on
The variable to be used for the next training (

,

) To the plurality of agents,
Based on the KL divergence between the guidance policy and the current policy of each agent, the variable to be used for the next training (

Including the step of calculating ),
The variables of the guidance policy of the best agent and the KL divergence clause of the current policy of each agent (

) To adjust the parameters of the best agent's guidance policy (

)in

The policy parameter (

Controlling to learn)
Multi-agent training control method, characterized in that. The method of claim 2,
Current policy corresponding to each of the plurality of agents (

) Of the parameter (

) Is to be updated in the direction of relatively increasing the target function of the corresponding agent.
Multi-agent training control method, characterized in that. delete The method of claim 1,
The above policy parameters (

Controlling the) to be shared by the plurality of agents,
In each of the plurality of agents

Once training is performed, a certain number representing the performance of the agent in order to select a new guidance policy (

) Of the episode reward average (

) Receiving steps
Multi-agent training control method comprising a. The method of claim 5,
The above policy parameters (

Controlling the) to be shared by the plurality of agents,
Average of the episode rewards received (

Determining an agent having the highest average among the plurality of agents as the best agent based on ); And
The policy parameters (

Steps to request)
Multi-agent training control method further comprising a. The method of claim 6,
The above policy parameters (

Controlling the) to be shared by the plurality of agents,
The above policy parameters (

In response to the request of ), the parameter corresponding to the current policy of the best agent (

Receiving) from the best agent;
Guidance policy parameters (

Parameter corresponding to the current policy of the best agent received (

Setting to ); And
The parameters of the set guide policy (

) To the plurality of agents
Multi-agent training control method further comprising a. In a system for controlling the training of policy parameters of each of a plurality of agents,
A training control unit configured to independently train each of the plurality of agents based on a previously shared guidance policy;
In order to obtain variables to be used in the next training for each training, information of each of the plurality of agents is received from the agent, and a predefined

A receiving control unit for receiving performance information of each agent from a corresponding agent as the training is performed once;
A transmission control unit configured to transmit a variable to be used for training after each training is determined based on the information of each agent to the plurality of agents; And
Policy parameters of the best agent determined based on the received performance information of each agent (

) To the plurality of agents to share
Including,
Each agent has its own policy parameter (a parameter of a multi-layer perceptron).

), and the policy parameter (

), and the variables required for training of the policies (

,

) And the parameters of the guidance policy (

), and the parameters of the shared guidance policy (

) To construct a guide policy and use it for training
Multi-agent training control system, characterized in that. The method of claim 8,
Current policy corresponding to each of the plurality of agents (

) Of the parameter (

) Is a variable to be used for the next training shared by the plurality of agents (

,

), information policy (

), and experience information of each agent

Is updated based on
The transmission control unit,
Based on the KL divergence between the guidance policy and the current policy of each agent, the variable to be used for the next training (

) Is calculated, and the variable of the guide policy of the best agent and the KL divergence term of the current policy of each agent

) To adjust the parameters of the best agent's guidance policy (

)in

The policy parameter (

Controlling to learn)
Multi-agent training control system, characterized in that. The method of claim 9,
Current policy corresponding to each of the plurality of agents (

) Of the parameter (

) Is to be updated in the direction of relatively increasing the target function of the corresponding agent.
Multi-agent training control system, characterized in that. delete The method of claim 9,
The sharing control unit,
In each of the plurality of agents

Once training is performed, a certain number representing the performance of the agent in order to select a new guidance policy (

) Of the episode reward average (

Receiving)
Multi-agent training control system, characterized in that. The method of claim 12,
The sharing control unit,
Average of the episode rewards received (

), the agent having the highest average among the plurality of agents is determined as the best agent, and the policy parameter (

Requesting)
Multi-agent training control system, characterized in that. The method of claim 13,
The sharing control unit,
The above policy parameters (

In response to the request of ), the parameter corresponding to the current policy of the best agent (

) From the best agent, and the parameters of the guidance policy (

Parameter corresponding to the current policy of the best agent received (

), and the parameters of the set guide policy (

) To the plurality of agents
Multi-agent training control system, characterized in that.

Images