CN113095498A - Divergence-based multi-agent cooperative learning method, divergence-based multi-agent cooperative learning device, divergence-based multi-agent cooperative learning equipment and divergence-based multi-agent cooperative learning medium - Google Patents

Abstract

The invention discloses a divergence-based multi-agent cooperative learning method, a divergence-based multi-agent cooperative learning device, divergence-based multi-agent cooperative learning equipment and a divergence-based storage medium, wherein the divergence-based multi-agent cooperative learning method comprises the following steps of: initializing a value network, a policy network and a target policy network; changing the updating modes of the value network, the strategy network and the target strategy network according to a preset regular term based on divergence to obtain a latest updating mode; training the plurality of agents according to the value network, the strategy network and the target strategy network to obtain experience data, and updating the value network, the strategy network and the target strategy network according to the experience data and the latest updating mode; and a plurality of agents acquire observation data from the environment, and make a decision by combining the experience data and the updated strategy network to obtain action data. The multi-agent cooperative learning method provided by the embodiment of the disclosure utilizes the regular term based on divergence to enhance the exploration capability of the agents and solve the cooperative problem of the multi-agents.

Description

Divergence-based multi-agent cooperative learning method, device, equipment and medium

technical field

The present invention relates to the technical field of machine learning, in particular to a multi-agent cooperative learning method, device, device and medium based on divergence.

Background technique

Reinforcement learning agents can complete autonomous learning of behavioral policies by interacting with the environment, so they have been successfully applied in tasks such as robotic arm control, chess and card games, and games in single-agent domains. However, many tasks in real life often require multiple agents to cooperate to complete tasks, such as logistics robots, unmanned driving, large-scale real-time strategy games and other tasks. Therefore, multi-agent cooperative learning has attracted more and more attention in recent years.

In cooperative multi-agent tasks, due to communication constraints, each agent usually only perceives local information within its own visible range. If each agent learns according to its own local information, it is difficult to form effective cooperation between agents. In the prior art, the exploration ability of the agent is improved by adding the entropy regularization term, but the addition of the entropy regularization term also modifies the original Markov decision process, which leads to the convergence strategy obtained by the reinforcement learning based on the entropy regularization. is not the optimal strategy for the original problem. will bring bias to the convergence strategy.

SUMMARY OF THE INVENTION

Embodiments of the present disclosure provide a divergence-based multi-agent cooperative learning method, apparatus, device, and medium. In order to provide a basic understanding of some aspects of the disclosed embodiments, a brief summary is given below. This summary is not intended to be an extensive review, nor is it intended to identify key/critical elements or delineate the scope of protection of these embodiments. Its sole purpose is to present some concepts in a simplified form as a prelude to the detailed description that follows.

In a first aspect, an embodiment of the present disclosure provides a multi-agent cooperative learning method based on divergence, including:

Initialize the value network, policy network and target policy network;

According to the preset divergence-based regular term, change the update method of the value network, the strategy network, and the target strategy network, and obtain the latest update method;

Multiple agents are trained according to the value network, strategy network and target strategy network to obtain experience data, and update the value network, strategy network and target strategy network according to the experience data and the latest update method;

Multiple agents obtain observation data from the environment, and combine the empirical data and the updated policy network to make decisions to obtain action data.

In an optional embodiment, before changing the update modes of the value network, the strategy network, and the target strategy network according to a preset divergence-based regular term, the method further includes:

Construct a multi-agent maximization objective function based on a divergence-based regularization term.

In an optional embodiment, the regularization term based on divergence is:

Among them, π represents the policy network, a _t represents the action, s _t represents the state, and ρ represents the target policy network.

In an optional embodiment, the update method of the value network is changed according to a preset divergence-based regular term, including:

The value network is updated according to the maximizing objective function:

是目标值网络的参数，s表示环境中的全部信息，a表示动作，y表示需要拟合的目标值，r表示全局奖励，E表示数学期望，γ表示折扣因子，s′表示智能体决策后环境转移到的新状态，a′表示智能体在新的状态下做出的动作，

表示值网络的损失函数，

表示目标值网络，Q_φ表示值网络，τ表示目标值网络进行滑动平均更新的参数。where λ is the regularization term coefficient, π is the policy network, ρ is the target policy network, φ is the parameter of the value network,

is the parameter of the target value network, s represents all the information in the environment, a represents the action, y represents the target value to be fitted, r represents the global reward, E represents the mathematical expectation, γ represents the discount factor, and s' represents the agent after the decision. The new state to which the environment is transferred, a' represents the action taken by the agent in the new state,

represents the loss function of the value network,

represents the target value network, Q _φ represents the value network, and τ represents the parameters of the target value network for moving average update.

In an optional embodiment, changing the update method of the policy network according to a preset divergence-based regular term, including:

The policy network is updated according to the minimized objective function:

表示策略网络的损失函数，E表示数学期望，a_i表示动作，s表示状态，

表示在给定策略π和目标策略ρ的情况下的值函数，λ表示正则项系数，ρ_i表示目标策略网络，D_KL表示KL散度。where θ _i represents the parameters of the policy network π _i ,

represents the loss function of the policy network, E represents the mathematical expectation, a _i represents the action, s represents the state,

denote the value function given a policy π and a target policy ρ, λ denotes the regularization term coefficient, ρ _i denotes the target policy network, and D _KL denotes the KL divergence.

In an optional embodiment, according to a preset divergence-based regular term, changing the update method of the target policy network, including:

The target policy network is updated according to the moving average of the policy network:

表示智能体i的目标策略网络的参数，τ表示目标策略网络用于滑动平均更新的参数，θ_i表示策略网络的参数。in,

represents the parameters of the target policy network of agent i, τ represents the parameters of the target policy network for moving average update, and θ _i represents the parameters of the policy network.

In an optional embodiment, multiple agents are trained according to the value network, the strategy network and the target strategy network to obtain experience data, and update the value network, the strategy network and the target strategy network according to the experience data and the latest update method, include:

The agent obtains observation data from the environment, and uses the empirical data to make decisions according to the policy network to obtain action data;

The environment gives rewards based on the current state and joint action, and moves to the next state;

The tuple is stored in the cache database as a piece of experience data, and the tuple includes the current environment state, current action, next environment state, observation data, and global reward;

When a period of experience is over, several pieces of experience data are obtained from the cache database for training, and the value network, policy network, and target policy network are updated.

In a second aspect, an embodiment of the present disclosure provides a multi-agent cooperative learning device based on divergence, including:

The initialization module is used to initialize the value network, the strategy network and the target strategy network;

a change module, used for changing the update mode of the value network, the strategy network and the target strategy network according to the preset divergence-based regular term, so as to obtain the latest update mode;

The training module is used for multiple agents to train according to the value network, the strategy network and the target strategy network, obtain the experience data, and update the value network, the strategy network and the target strategy network according to the experience data and the latest update method;

The execution module is used for multiple agents to obtain observation data from the environment, and combine the experience data and the updated policy network to make decisions to obtain action data.

In a third aspect, an embodiment of the present disclosure provides a multi-agent cooperative learning device based on divergence, including a processor and a memory storing program instructions. Divergence-based multi-agent cooperative learning method.

In a fourth aspect, embodiments of the present disclosure provide a computer-readable medium on which computer-readable instructions are stored, and the computer-readable instructions can be executed by a processor to implement a divergence-based multi-agent provided by the foregoing embodiments Cooperative learning methods.

The technical solutions provided by the embodiments of the present disclosure may include the following beneficial effects:

The embodiments of the present disclosure provide a multi-agent cooperative learning method, which is used to solve the multi-agent cooperation problem. This method adds a target policy to the agent, and uses the target policy to add a divergence-based regularization term to the reward function of the general Markov decision process, and proposes a divergence-based Markov decision process. The new policy update method, the embodiment of the present disclosure enhances the exploration ability of the agent by using the regular term based on divergence, and at the same time, when the policy of the agent converges, the regular term will disappear, avoiding adding regular terms to the convergence policy. come deviation. And after adding this regular term, off-policy training can be realized, which solves the problem of sampling efficiency in a multi-agent environment. At the same time, the regular term also controls the step size of policy update, which enhances the stability of policy promotion. The method proposed by the embodiments of the present disclosure has certain flexibility, and has important application value in the fields of intelligent transportation and robot control.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention.

Description of drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and together with the description serve to explain the principles of the invention.

1 is a schematic flowchart of a divergence-based multi-agent cooperative learning method according to an exemplary embodiment;

2 is a schematic flowchart of a multi-agent training method based on divergence according to an exemplary embodiment;

3 is a schematic structural diagram of a divergence-based multi-agent cooperative learning apparatus according to an exemplary embodiment;

4 is a schematic structural diagram of a divergence-based multi-agent cooperative learning device according to an exemplary embodiment;

Fig. 5 is a schematic diagram of a computer storage medium according to an exemplary embodiment.

Detailed ways

The following description and drawings sufficiently illustrate specific embodiments of the invention to enable those skilled in the art to practice them.

It should be understood that the described embodiments are only some, but not all, embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

Where the following description refers to the drawings, the same numerals in different drawings refer to the same or similar elements unless otherwise indicated. The implementations described in the illustrative examples below are not intended to represent all implementations consistent with the present invention. Rather, they are merely examples of systems and methods consistent with some aspects of the invention, as recited in the appended claims.

In the description of the present invention, it should be understood that the terms "first", "second" and the like are used for descriptive purposes only, and should not be construed as indicating or implying relative importance. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood in specific situations. Furthermore, in the description of the present invention, unless otherwise specified, "a plurality" means two or more. "And/or", which describes the association relationship of the associated objects, means that there can be three kinds of relationships, for example, A and/or B, which can mean that A exists alone, A and B exist at the same time, and B exists alone. The character "/" generally indicates that the associated objects are an "or" relationship.

In the multi-agent cooperation problem, the state of the environment, that is, the total information in the environment is s. Each agent i can obtain partial information o _i in the environment, obtain and execute its own action a _i according to the historical information τ _i ={(o _i , a _i )} and its own strategy π _i , according to the current state s and the _joint action a=(a ₁ , a ₂ , . a). Note that the joint strategy of all agents is π=π ₁ ×π ₂ ×…×π _n , then the common objective function that each agent maximizes is:

where E is the mathematical expectation, π is the policy, at is the action, s _{t is} the state, and γ is the discount factor.

The reinforcement learning based on entropy regularization mainly subtracts logπ(a _t |s _t ) from the reward function, that is, the optimization objective of the agent is modified as:

Among them, E is the mathematical expectation, π is the policy, at is the action, s _{t is} the state, γ is the discount factor, and λ is the coefficient of the regular term. The addition of the entropy regularization term can improve the exploration ability of the agent, but the entropy regularization term also modifies the original Markov decision process, which leads to the convergence strategy obtained by the reinforcement learning based on the entropy regularization is not the optimal of the original problem strategy, which will bias the convergence strategy.

The embodiment of the present disclosure proposes a new divergence-based regular term, and proposes a new policy update method for divergence-based Markov decision process, using the divergence-based regular term to enhance the exploration ability of the agent , and when the agent's policy converges, the regular term will disappear, avoiding the deviation of adding regular term to the convergence policy. And after adding this regular term, off-policy training can be realized, which solves the problem of sampling efficiency in a multi-agent environment. At the same time, the regular term also controls the step size of policy update, which enhances the stability of policy promotion.

The divergence-based multi-agent cooperative learning method provided by the embodiments of the present application will be described in detail below with reference to FIG. 1 to FIG. 2 .

The applicable scenario of this method is that there are multiple cooperative agents, and each agent interacts with the environment during the training process. In real life, many tasks often require the cooperation of multiple agents to complete, such as logistics robots, wireless Human driving, large-scale real-time strategy games, etc. This kind of system that requires the cooperation of multiple agents to complete tasks is called a multi-agent system, that is, multi-agent. For example, the warehouse logistics system is a multi-agent, and each logistics robot is an agent.

Referring to Figure 1, the method specifically includes the following steps.

S101 initializes the value network Q, the strategy network π _i and the target strategy network ρ _i .

S102 changes the update modes of the value network, the policy network, and the target policy network according to the preset divergence-based regular term, and obtains the latest update mode.

In a possible implementation manner, before step S102 is performed, the method further includes constructing the maximization objective function of the multi-agent according to the regularization term based on the divergence.

作为正则项，称之为基于散度的正则项。其中，π表示策略网络，a_t表示动作，s_t表示状态，ρ表示目标策略网络。Specifically, add a target policy ρ _i to each agent, and use the newly added target policy to subtract a target policy from the reward function

As a regular term, it is called a divergence-based regular term. Among them, π represents the policy network, a _t represents the action, s _t represents the state, and ρ represents the target policy network.

Further, the maximization objective function of multi-agent is obtained by using the regularization term based on divergence as:

Among them, E is the mathematical expectation, π is the policy, at is the action, s _{t is} the state, γ is the discount factor, λ is the coefficient of the regular term, and ρ is the target policy network.

In the embodiment of the present disclosure, the entropy regularization term is changed to a divergence-based regularization term, and a new optimization objective function is obtained by using the divergence-based regularization term. When the target strategy is taken as the past strategy, when the probability of a certain action increases, the regular term will reduce the reward, otherwise it will increase the reward, that is, the regular term encourages the agent to take actions with a decreasing probability, and discourages the agent to take Actions with increasing probability so that they can be explored more fully during training. At the same time, the addition of the regular term actually controls the step size of the policy update, that is, the new policy cannot deviate too much from the past policy, which is also conducive to the stability of policy improvement. Moreover, after adding the regular term based on divergence, the latest update method can be obtained according to the derivation, which can realize off-policy training, thus solving the sampling problem faced by the policy-based reinforcement learning method in the multi-agent situation. question of efficiency.

Specifically, according to the preset divergence-based regular term, change the update method of the value network, including:

In general reinforcement learning, the value function is usually defined as:

where Q is the value function, π is the policy, at is the action, s _{t is} the state, γ is the discount factor, E is the mathematical expectation, and r is the global reward.

In the training process, it is often updated according to the Bellman optimal equation, namely:

Among them, Q ^* represents the updated value function, π represents the policy, a represents the action, s represents the state, γ represents the discount factor, E represents the mathematical expectation, r represents the global reward, s' represents the state at the next moment, and a' represents the next a moment of action.

After adding the regularization term based on divergence, a new iterative formula for the optimal value function is obtained, namely:

表示最优的值函数，a表示动作，s表示状态，γ表示折扣因子，E表示数学期望，r表示全局奖励，s′表示下一时刻的状态，a′表示下一时刻的动作，λ表示正则项的系数，ρ表示目标策略网络，

表示最优策略。该公式除了加入基于散度的正则项之外，相比于贝尔曼最优方程，其并不限制a′为最优动作，可以任取，这样就增加了训练时的灵活性，同时可以在离策略训练中采用TD(λ)的目标函数设计。in,

Represents the optimal value function, a represents the action, s represents the state, γ represents the discount factor, E represents the mathematical expectation, r represents the global reward, s' represents the state at the next moment, a' represents the action at the next moment, and λ represents the The coefficient of the regular term, ρ represents the target policy network,

represents the optimal strategy. In addition to adding a regular term based on divergence, compared to the Bellman optimal equation, this formula does not restrict a' to be the optimal action, which can be taken arbitrarily, which increases the flexibility during training, and can be used in The objective function design of TD(λ) is adopted in off-policy training.

According to the above iterative formula, the designed value network is updated according to the maximization objective function:

是目标值网络的参数，s表示环境中的全部信息，a表示动作，y表示需要拟合的目标值，r表示全局奖励，E表示数学期望，γ表示折扣因子，s′表示智能体决策后环境转移到的新状态，a′表示智能体在新的状态下做出的动作，

表示值网络的损失函数，

定示目标值网络，Q_φ表示值网络，τ表示目标值网络进行滑动平均更新的参数。where λ is the regularization term coefficient, π is the policy network, ρ is the target policy network, φ is the parameter of the value network,

is the parameter of the target value network, s represents all the information in the environment, a represents the action, y represents the target value to be fitted, r represents the global reward, E represents the mathematical expectation, γ represents the discount factor, and s' represents the agent after the decision. The new state to which the environment is transferred, a' represents the action taken by the agent in the new state,

represents the loss function of the value network,

The target value network is fixed, Q _φ represents the value network, and τ represents the parameters of the moving average update of the target value network.

Further, according to the preset divergence-based regular term, change the update method of the policy network, including:

The policy-based reinforcement learning method usually obtains the update method of the policy network according to the policy gradient theorem, but the policy gradient theorem restricts that the update must be on-policy. After adding the regularization term based on divergence, a new strategy promotion theorem is obtained, that is, given an old strategy π _old , only the new strategy π _new needs to satisfy

D_KL代表KL散度，那么π_new就比π_old更优。根据上面的定理，经过推导可以根据下面的公式来计算策略的梯度：in

D _KL stands for KL divergence, then π _new is better than π _old . According to the above theorem, after derivation, the gradient of the policy can be calculated according to the following formula:

表示用于存放训练数据的缓存数据库，

表示策略网络的损失函数，E表示数学期望，a_i表示动作，s表示状态，

表示在给定策略π和目标策略ρ的情况下的值函数，λ表示正则项系数，ρ表示目标策略网络。where θ _i represents the parameters of the policy network π _i ,

Represents a cache database for storing training data,

represents the loss function of the policy network, E represents the mathematical expectation, a _i represents the action, s represents the state,

represents the value function given a policy π and a target policy ρ, λ represents the regularization term coefficient, and ρ represents the target policy network.

则梯度公式可进一步改写为下述，策略网络根据最小化目标函数的更新方法也如下述公式所示：The above gradient formula, in addition to adding regular terms, is different from the policy gradient theorem in that it no longer restricts the distribution of state s, so the above update method can be used to update off-policy. In the setting of multi-agent cooperation, a counterfactual baseline function can be subtracted from the gradient to solve the problem of credit assignment and reduce the variance of the gradient update. In addition, the counterfactual baseline function can also Reduce calculations. Take the baseline function as

Then the gradient formula can be further rewritten as follows, and the update method of the policy network according to the minimized objective function is also shown in the following formula:

表示策略网络的损失函数，E表示数学期望，a_i表示动作，s表示状态，

表示在给定策略π和目标策略ρ的情况下的值函数，λ表示正则项系数，ρ_i表示目标策略网络，D_KL表示KL散度。where θ _i represents the parameters of the policy network π _i ,

represents the loss function of the policy network, E represents the mathematical expectation, a _i represents the action, s represents the state,

denote the value function given a policy π and a target policy ρ, λ denotes the regularization term coefficient, ρ _i denotes the target policy network, and D _KL denotes the KL divergence.

Further, according to the preset divergence-based regular term, change the update method of the target policy network, including:

For the target strategy, in the most ideal case, the following iterations should be used:

从而收敛的策略就是原马尔科夫决策过程的最优策略，也就避免了加入正则项给收敛策略带来的偏差。然而，这样的理想情形每一步迭代都需要通过强化学习的训练直到收敛，代价过大，因此可以采用下面的近似方法：That is, the optimal strategy π ^t+1 is obtained under the condition of ρ=π ^t . When such an iterative process converges, π ^t+1 =π ^t , and then there are

Therefore, the convergence strategy is the optimal strategy of the original Markov decision process, which avoids the deviation caused by adding regular terms to the convergence strategy. However, in such an ideal situation, each iteration needs to be trained by reinforcement learning until convergence, and the cost is too high, so the following approximation method can be used:

表示智能体i的目标策略网络的参数，τ表示目标策略网络用于滑动平均更新的参数，θ_i表示策略网络的参数。即用一步梯度更新来代替最大化，同时让目标策略取为策略的滑动平均。in,

represents the parameters of the target policy network of agent i, τ represents the parameters of the target policy network for moving average update, and θ _i represents the parameters of the policy network. That is, a step gradient update is used to replace the maximization, and the target policy is taken as the sliding average of the policy.

The target policy network is updated according to the moving average of the policy network:

表示智能体i的目标策略网络的参数，τ表示目标策略网络用于滑动平均更新的参数，θ_i表示策略网络的参数。in,

represents the parameters of the target policy network of agent i, τ represents the parameters of the target policy network for moving average update, and θ _i represents the parameters of the policy network.

According to this step, the latest update method of the value network, policy network, and target policy network can be obtained according to the regular term based on divergence. Through the latest update method, when the agent's policy converges, the regular term will disappear, avoiding adding The deviation brought by the regular term to the convergence strategy can also realize off-policy training, which solves the problem of sampling efficiency in a multi-agent environment, and also controls the step size of the policy update, which enhances the stability of the policy improvement.

S103 Multiple agents are trained according to the value network, the strategy network and the target strategy network to obtain experience data, and update the value network, the strategy network and the target strategy network according to the experience data and the latest update method.

Fig. 2 is a schematic flowchart of a divergence-based multi-agent training method according to an exemplary embodiment. As shown in Fig. 2, the multi-agent training process includes: S201, the agent obtains observation data from the environment, And use the experience data to make decisions according to the policy network to obtain action data; S202 the environment gives rewards according to the current state and joint action, and transfers to the next state; S203 stores the tuple as a piece of experience data in the cache database, and the multivariate The group includes the current environment state, the current action, the next environment state, the observation data, and the global reward; S204, when a period of experience ends, obtain several pieces of experience data from the cache database for training, and update the value network, policy network, and target policy network. .

Specifically, the training data is obtained through the interaction between the agent and the environment, that is, the agent i obtains the observation data o _i from the environment, and uses the empirical data τ _i to make decisions according to the strategy π _i to obtain the action a _i , and the environment is based on the current The state s and the joint action a give the reward r and move to the next state s', and store the tuple (s, a, s', o, r) as a piece of empirical data in the cache database D. When a period of experience ends, take several experiences from D for training, and update the value network Q, the strategy network π _i and the target strategy network ρ _i according to the latest update method.

In an exemplary scenario, in the game StarCraft II, the player will control several units to fight the enemy, with the goal of destroying all of the enemy's units. In training, we regard each unit as an agent, each agent has a field of view, and the observed data obtained are related indicators of the units within its field of view (such as health, shield, unit The agent can take actions such as moving, attacking, releasing skills, etc. The agent's strategy is generated by the policy network, which is a probability distribution of the actions that the agent can take in the current state. When you damage or destroy enemy units, you will get rewards. The status of each step (data corresponding to all the information of the current game, only available during training), the observation of all agents, the actions of all agents, the action of The reward obtained after taking the action and the new state transferred to it are stored as a piece of experience. After performing several steps or after a game is over, we take out the experience data obtained before for training, update the strategy of the agent, and then use the new The strategy gets new experience, and the process is repeated until the agent achieves better performance.

S104 Multiple agents obtain observation data from the environment, and combine the experience data and the updated policy network to make decisions to obtain action data.

Further, in the execution process, for the agent i: each time the observation data o _i is obtained from the environment, and combined with the empirical data τ _i to make a decision according to the updated policy network π _i to obtain the action a _i .

According to the cooperative learning method of multi-agent provided by the embodiment of the present disclosure, the cooperative problem of multi-agent is solved, and the regularity based on divergence is used to enhance the exploration ability of the agent, and when the strategy of the agent converges, the regular The term will disappear, avoiding the bias brought by adding a regular term to the convergence strategy. And after adding this regular term, off-policy training can be realized, which solves the problem of sampling efficiency in a multi-agent environment. At the same time, the regular term also controls the step size of policy update, which enhances the stability of policy promotion. The method proposed by the embodiments of the present disclosure has certain flexibility. For example, in the field of robot control, it can solve the problem of cooperative control of multiple robots, and in the field of games, it can solve the cooperative control of multiple game characters. artificial intelligence system.

An embodiment of the present disclosure further provides a divergence-based multi-agent cooperative learning device, which is used to execute the divergence-based multi-agent cooperative learning method of the above embodiment. As shown in FIG. 3 , the device includes:

The initialization module 301 is used to initialize the value network, the strategy network and the target strategy network;

The changing module 302 is configured to change the update mode of the value network, the strategy network, and the target strategy network according to the preset divergence-based regular term, so as to obtain the latest update mode;

The training module 303 is used for a plurality of agents to train according to the value network, the strategy network and the target strategy network to obtain experience data, and to update the value network, the strategy network and the target strategy network according to the experience data and the latest update method;

The execution module 304 is used for a plurality of agents to obtain observation data from the environment, and combine the experience data and the updated policy network to make decisions to obtain action data.

It should be noted that, when the divergence-based multi-agent cooperative learning device provided in the above embodiment executes the divergence-based multi-agent cooperative learning method, only the division of the above functional modules is used as an example for illustration. In practical applications, The above-mentioned function allocation can be completed by different function modules according to requirements, that is, the internal structure of the device is divided into different function modules, so as to complete all or part of the functions described above. In addition, the divergence-based multi-agent cooperative learning device provided by the above embodiments and the divergence-based multi-agent cooperative learning method embodiments belong to the same concept, and the implementation process is detailed in the method embodiments, which will not be repeated here.

Embodiments of the present disclosure further provide an electronic device corresponding to the divergence-based multi-agent cooperative learning method provided by the foregoing embodiments, so as to execute the above-mentioned divergence-based multi-agent cooperative learning method.

Please refer to FIG. 4 , which shows a schematic diagram of an electronic device provided by some embodiments of the present application. As shown in FIG. 4 , the electronic device includes: a processor 400 , a memory 401 , a bus 402 and a communication interface 403 , the processor 400 , the communication interface 403 and the memory 401 are connected through the bus 402 ; The running computer program, when the processor 400 runs the computer program, executes the divergence-based multi-agent cooperative learning method provided by any of the foregoing embodiments of the present application.

The memory 401 may include a high-speed random access memory (RAM: Random Access Memory), and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory. The communication connection between the network element of the system and at least one other network element is realized through at least one communication interface 403 (which may be wired or wireless), and the Internet, a wide area network, a local network, a metropolitan area network, etc. may be used.

The bus 402 may be an ISA bus, a PCI bus, an EISA bus, or the like. The bus can be divided into address bus, data bus, control bus and so on. The memory 401 is used to store a program, and the processor 400 executes the program after receiving the execution instruction. The divergence-based multi-agent cooperative learning method disclosed in any of the foregoing embodiments of the present application can be applied to the processor 400 , or implemented by the processor 400 .

The processor 400 may be an integrated circuit chip with signal processing capability. In the implementation process, each step of the above-mentioned method can be completed by an integrated logic circuit of hardware in the processor 400 or an instruction in the form of software. The above-mentioned processor 400 may be a general-purpose processor, including a central processing unit (CPU for short), a network processor (NP for short), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), off-the-shelf programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic device, discrete hardware component. The methods, steps, and logic block diagrams disclosed in the embodiments of this application can be implemented or executed. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. The steps of the method disclosed in conjunction with the embodiments of the present application may be directly embodied as executed by a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules may be located in random access memory, flash memory, read-only memory, programmable read-only memory or electrically erasable programmable memory, registers and other storage media mature in the art. The storage medium is located in the memory 401, and the processor 400 reads the information in the memory 401, and completes the steps of the above method in combination with its hardware.

The electronic device provided by the embodiments of the present application and the divergence-based multi-agent cooperative learning method provided by the embodiments of the present application are based on the same inventive concept, and have the same beneficial effects as the methods adopted, operated or implemented.

Embodiments of the present application also provide a computer-readable storage medium corresponding to the divergence-based multi-agent cooperative learning method provided by the foregoing embodiments, please refer to FIG. 5 , the computer-readable storage medium shown is an optical disc 500 , a computer program (ie, a program product) is stored thereon, and when the computer program is run by the processor, it will execute the divergence-based multi-agent cooperative learning method provided in any of the foregoing embodiments.

It should be noted that examples of computer-readable storage media may also include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory Memory (RAM), Read-Only Memory (ROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Flash Memory or other optical and magnetic storage media will not be repeated here.

The computer-readable storage medium provided by the above-mentioned embodiments of the present application and the divergence-based multi-agent cooperative learning method provided by the embodiments of the present application are based on the same inventive concept, and have the same inventive concept as the application program they store in, run or implement. method with the same beneficial effect.

The technical features of the above embodiments can be combined arbitrarily. In order to make the description simple, all possible combinations of the technical features in the above embodiments are not described. However, as long as there is no contradiction in the combination of these technical features It is considered to be the range described in this specification.

The above embodiments only represent several embodiments of the present invention, and the descriptions thereof are specific and detailed, but should not be construed as a limitation on the scope of the patent of the present invention. It should be pointed out that for those of ordinary skill in the art, without departing from the concept of the present invention, several modifications and improvements can also be made, which all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention should be subject to the appended claims.

Claims (10)

1. A divergence-based multi-agent cooperative learning method is characterized by comprising the following steps:

initializing a value network, a policy network and a target policy network;

changing the updating modes of the value network, the strategy network and the target strategy network according to a preset regular term based on divergence to obtain a latest updating mode;

training the plurality of agents according to the value network, the strategy network and the target strategy network to obtain experience data, and updating the value network, the strategy network and the target strategy network according to the experience data and the latest updating mode;

and a plurality of agents acquire observation data from the environment, and make a decision by combining the experience data and the updated strategy network to obtain action data.

2. The method of claim 1, wherein before changing the updating modes of the value network, the policy network and the target policy network according to a preset regular term based on divergence, the method further comprises:

and constructing a maximized objective function of the multi-agent according to the divergence-based regularization item.

3. The method of claim 2, based onThe regular term for divergence is:

wherein, pi represents a policy network, a_tRepresents the motion, s_tRepresenting the state and p representing the target policy network.

4. The method of claim 1, wherein changing the update mode of the value network according to a preset regularization term based on divergence comprises:

the value network is updated according to the maximized objective function:

where λ is the regular term coefficient, π represents the policy network, ρ represents the target policy network, φ is a parameter of the value network,

is a parameter of the network of target values, s represents all information in the environment, a represents actions, y represents target values to be fitted, r represents global rewards, E represents mathematical expectations, γ represents a discount factor, s 'represents a new state to which the environment is transferred after the agent makes a decision, a' represents actions made by the agent in the new state,

a loss function representing a network of values,

representing a network of target values, Q_φAnd is used for representing a value network, and tau is used for representing a parameter for performing moving average updating of the target value network.

5. The method of claim 1, wherein changing the update mode of the policy network according to a preset divergence-based regularization term comprises:

the policy network updates according to the minimization objective function:

wherein, theta_iRepresenting a policy network pi_iIs determined by the parameters of (a) and (b),

representing the loss function of the policy network, E representing the mathematical expectation, a_iRepresenting an action, s representing a state,

representing a function of values given a strategy pi and a target strategy p, lambda representing the regular term coefficient, p_iRepresenting a target policy network, D_KLIndicating KL divergence.

6. The method of claim 1, wherein changing the update mode of the target policy network according to a preset divergence-based regularization term comprises:

the target policy network updates according to the running average of the policy network:

wherein,

a parameter representing a target policy network of agent i, τ a parameter of the target policy network for moving average updating, θ_iRepresenting parameters of the policy network.

7. The method of claim 1, wherein training agents according to the value network, the policy network, and the target policy network to obtain experience data, and updating the value network, the policy network, and the target policy network according to the experience data and the latest update method comprises:

the intelligent agent obtains observation data from the environment, and makes a decision according to a policy network by using experience data to obtain action data;

the environment gives a reward according to the current state and the joint action, and moves to the next state;

storing a multi-element group as experience data into a cache database, wherein the multi-element group comprises a current environment state, a current action, a next environment state, observation data and global rewards;

and after a period of experience is finished, acquiring a plurality of pieces of experience data from the cache database for training, and updating the value network, the strategy network and the target strategy network.

8. A divergence-based multi-agent cooperative learning apparatus, comprising:

the initialization module is used for initializing a value network, a strategy network and a target strategy network;

the change module is used for changing the update modes of the value network, the strategy network and the target strategy network according to a preset regular term based on divergence to obtain a latest update mode;

the training module is used for training the plurality of agents according to the value network, the strategy network and the target strategy network to obtain experience data, and updating the value network, the strategy network and the target strategy network according to the experience data and the latest updating mode;

and the execution module is used for acquiring observation data from the environment by a plurality of agents, and making a decision by combining the experience data and the updated strategy network to obtain action data.

9. A divergence-based multi-agent cooperative learning apparatus, comprising a processor and a memory storing program instructions, the processor being configured to, upon execution of the program instructions, perform a divergence-based multi-agent cooperative learning method as claimed in any one of claims 1 to 7.

10. A computer readable medium having computer readable instructions stored thereon which are executable by a processor to implement a divergence-based multi-agent cooperative learning method as claimed in any one of claims 1 to 7.

Images