CN111309880A - Multi-agent action strategy learning method, device, medium and computing device - Google Patents

Abstract

The embodiment of the invention provides a multi-agent action strategy learning method, which comprises the following steps: the multi-agent respectively samples corresponding actions according to respective initial action strategies; respectively estimating advantages obtained after the multi-agent executes corresponding actions; and updating the action strategy of each agent based on the advantages obtained after the multi-agent executes the corresponding action, so that the updated action strategies can enable the corresponding agents to obtain higher return. In a machine learning scene oriented to task processing, the method of the invention trains a plurality of intelligent agents which are cooperated with each other (namely simultaneously trains a plurality of action strategies) at the same time instead of adopting a pre-constructed simulator and the intelligent agents for interaction without manual supervision, thereby greatly saving time cost and resources.

Description

Multi-agent action strategy learning method, device, medium and computing device

technical field

The embodiments of the present invention relate to the field of reinforcement learning, and more particularly, the embodiments of the present invention relate to a multi-agent action strategy learning method, apparatus, medium and computing device.

Background technique

This section is intended to provide a background or context for the embodiments of the invention that are recited in the claims. The descriptions herein are not admitted to be prior art by inclusion in this section.

The action policy determines the next action the agent should take and plays a crucial role in task-oriented systems. In recent years, policy learning has been widely regarded as a reinforcement learning (RL) problem. Since RL requires a lot of interactions for policy training, interacting directly with real users is time-consuming and labor-intensive. The most common approach is to develop a user simulator to help train the target agent to learn action policies.

However, designing a reliable user simulator is not an easy task and is often challenging because it is equivalent to building a good agent. With the ever-increasing demand for agents to handle more complex tasks, building a fully rule-based user simulator will be a daunting and arduous job and require a lot of domain expertise.

SUMMARY OF THE INVENTION

In this context, embodiments of the present invention are expected to provide a multi-agent action policy learning method, apparatus, medium and computing device.

In a first aspect of the embodiments of the present invention, a multi-agent action strategy learning method is provided, including:

The multi-agents sample corresponding actions according to their respective initial action strategies;

separately estimating the advantages obtained by the multi-agents after performing the corresponding actions;

The action strategy of each agent is updated based on the advantages obtained after the multi-agent performs the corresponding action, so that each updated action strategy can enable the corresponding agent to obtain a higher reward.

In an embodiment of this implementation manner, before the multi-agents perform corresponding actions according to their respective initial action strategies, the method further includes:

Pre-training is performed using real action data to obtain the respective initial action policies of the multi-agents.

In an example of this implementation manner, a weighted logistic regression method is used to perform pre-training based on real action data to obtain respective initial action strategies of the multi-agents.

In an example of this implementation manner, the advantages obtained after the multi-agents perform corresponding actions are estimated respectively, including:

Obtain the new state achieved by each agent after performing the corresponding action;

Calculate the current state of each agent and the return after reaching the new state;

According to the current state of each agent and the reward after reaching the new state, the advantages obtained by each agent after performing the corresponding action are calculated.

In one example of this implementation, the multi-agent includes a user agent configured to aim at completing a task and a system agent configured to assist the user agent mission accomplished.

In an example of this implementation manner, the rewards after each agent reaches a certain state include at least its own rewards and global rewards.

In an example of this implementation manner, the reward of the user agent itself includes at least a penalty for inaction, a penalty for executing a new subtask when there are uncompleted subtasks, and a user action reward.

In one example of this embodiment, the user action reward is determined based on whether the user agent performs all actions that are beneficial for completing the task.

In an example of this implementation manner, the reward of the system agent itself is accumulated by at least one of the following: a penalty for inaction, a penalty for not immediately providing corresponding assistance when the user agent requests assistance, and a subtask completion reward .

In an example of this implementation manner, the global reward is accumulated by at least one of the following: an efficiency loss penalty, a subtask completion reward, and a total task completion reward.

In an example of this implementation manner, the user action reward or all task completion rewards are calculated after all actions are terminated.

In an example of this implementation manner, the reward of each agent in a certain state is calculated by the corresponding value network.

In one example of this embodiment, different types of rewards are calculated by different value networks respectively.

In one embodiment of this embodiment, during the multi-agent policy learning process, the value network is configured to minimize the difference between the reward predicted by the preset method and the reward calculated by the value network The variance is optimized for the target.

In one embodiment of this implementation, the preset method is configured to predict rewards based on the state of a certain agent.

In an embodiment of this implementation manner, the preset method is a sequential difference algorithm.

In an example of this implementation manner, a target network is introduced to optimize and update the value network to make the training process more stable.

In a second aspect of the embodiments of the present invention, there is provided a multi-agent action strategy learning device, including a plurality of multi-agents and a plurality of value networks, wherein the plurality of agents are based on their respective initial action strategies Sampling the corresponding action;

The plurality of value networks respectively estimate the advantages obtained by the multi-agents after performing the corresponding actions;

The multiple agents update their respective action strategies based on the advantages obtained after performing the corresponding actions, so that the updated respective action strategies can enable the corresponding agents to obtain higher returns.

In an example of this implementation manner, the apparatus further includes:

The pre-training module is configured to perform pre-training using real action data to obtain respective initial action strategies of the multi-agents.

In an embodiment of this implementation manner, the pre-training module is configured to perform pre-training based on real action data by using a weighted logistic regression method to obtain respective initial action strategies of the multi-agents.

In an example of this implementation manner, the device further includes a plurality of state encoding modules, configured to separately acquire new states reached after each agent performs a corresponding action;

The multiple value networks are also configured to calculate the current state of each agent and the reward after reaching the new state, respectively; Advantage.

In one example of this implementation, the multi-agent includes a user agent configured to aim at completing a task and a system agent configured to assist the user agent mission accomplished.

In an example of this implementation manner, the rewards after each agent reaches a certain state include at least its own rewards and global rewards.

In an example of this implementation manner, the reward of the user agent itself includes at least a penalty for inaction, a penalty for executing a new subtask when there are uncompleted subtasks, and a user action reward.

In one example of this embodiment, the user action reward is determined based on whether the user agent performs all actions that are beneficial for completing the task.

In an example of this implementation manner, the reward of the system agent itself is accumulated by at least one of the following: a penalty for inaction, a penalty for not immediately providing corresponding assistance when the user agent requests assistance, and a subtask completion reward .

In an example of this implementation manner, the global reward is accumulated by at least one of the following: an efficiency loss penalty, a subtask completion reward, and a total task completion reward.

In an example of this implementation manner, the user action reward or all task completion rewards are calculated after the action is terminated.

In an example of this implementation manner, the reward of each agent in a certain state is calculated by the corresponding value network.

In one example of this embodiment, different types of rewards are calculated by different value networks respectively.

In one embodiment of this embodiment, during the multi-agent policy learning process, the value network is configured to minimize the difference between the reward predicted by the preset method and the reward calculated by the value network The variance is optimized for the target.

In one embodiment of this implementation, the preset method is configured to predict rewards based on the state of a certain agent.

In an embodiment of this implementation manner, the preset method is a sequential difference algorithm.

In an example of this implementation manner, a target network is introduced to optimize and update the value network to make the training process more stable.

In a third aspect of an embodiment of the present invention, a computer-readable storage medium is provided, and the storage medium stores a computer program, and the computer program is used to execute the method described in any one of the first aspect of the embodiment. method.

In a fourth aspect of an embodiment of the present invention, a computing device is provided, the computer device includes a processor, and the processor is configured to implement any one of the first aspects of the embodiment when executing a computer program stored in a memory method described in item.

This embodiment proposes a multi-agent action strategy learning method. In a task-oriented machine learning scenario, multiple agents that cooperate with each other are simultaneously trained (that is, each agent learns its own action strategy), instead of using The pre-built simulator interacts with the agent without manual supervision, which greatly saves time, cost and resources. In addition, in this embodiment, in order to enable each agent to learn an excellent action strategy, each The agents assign different rewards, and the differentiated reward distribution enables each agent to learn an action strategy that makes its cumulative reward higher.

Description of drawings

The above and other objects, features and advantages of exemplary embodiments of the present invention will become readily understood by reading the following detailed description with reference to the accompanying drawings. In the accompanying drawings, several embodiments of the present invention are shown by way of example and not limitation, wherein:

FIG. 1 schematically shows a schematic structural scene diagram of a multi-agent action strategy learning method according to an embodiment of the present invention;

FIG. 2 schematically shows a flow scenario diagram of user interaction with the system according to an embodiment of the present invention;

3 is a schematic flowchart of a method for learning a multi-agent action strategy provided by an embodiment of the present invention;

4 is a schematic structural scene diagram of a multi-agent action strategy learning method provided by an embodiment of the present invention;

5 is a schematic block diagram of a multi-agent action strategy learning device according to an embodiment of the present invention;

6 is a schematic diagram of a computer-readable storage medium according to an embodiment of the present invention;

FIG. 7 is a schematic diagram of a computing device according to an embodiment of the present invention;

In the drawings, the same or corresponding reference numerals denote the same or corresponding parts.

Detailed ways

The principles and spirit of the present invention will now be described with reference to several exemplary embodiments. It should be understood that these embodiments are only given for those skilled in the art to better understand and implement the present invention, but not to limit the scope of the present invention in any way. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

As will be appreciated by those skilled in the art, embodiments of the present invention may be implemented as a system, apparatus, device, method or computer program product. Accordingly, the present disclosure may be embodied in entirely hardware, entirely software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.

According to an embodiment of the present invention, a multi-agent action strategy learning method, apparatus, medium and computing device are provided.

Furthermore, any number of elements in the drawings is for illustration and not limitation, and any designation is for distinction only and does not have any limiting meaning.

The principles and spirit of the present invention are explained in detail below with reference to several representative embodiments of the present invention.

Scenario overview

In task-oriented scenarios, users often come with a clear purpose, hoping to get information or services that meet specific constraints, such as: ordering meals, booking tickets, online shopping, booking taxis, booking hotels, looking for music, movies or some kind of product. In the above-mentioned scenarios, the system (agent) is often required to provide corresponding information according to the user's goal. The current method often interacts with the system (agent) through a pre-built user simulator to train the system (agent) to have the ability to serve users.

The inventors have found that dialogue agents (Agents) with different roles can be trained simultaneously based on the Actor-Critic framework, so that they can learn strategies for interacting and collaborating to achieve goals. For example, in the above framework, the agent is regarded as an actor (Actor), which selects actions according to the current state, the critic (Critic) module scores the performance of the actor (the agent performs a certain action), and the actor is based on the critic (Critic) module. The score adjusts his actions in order to get a higher score.

However, since the roles of different agents are actually different, that is, agents are divided into user agents and system agents. Based on this, it is inappropriate to use the same set of evaluation criteria to judge the performance of agents, and it is necessary to evaluate the performance of different agents. The corresponding standards are used to score, that is, the critics (Critic) who need to use different evaluation standards to score the performance of the agent (Actor) respectively, as shown in Figure 1, so that the agent can adjust its own action strategy, so as to improve the performance of the agent. Complete tasks efficiently.

Mission overview

The embodiment of the present invention provides a multi-agent action strategy learning method, the multi-agent interacts and cooperates to complete a task goal, and the task goal may be a common task goal of the multi-agents, or may be one of the intelligent The multi-agent can be realized as various virtual/entities capable of implementing behaviors or actions, such as a robot capable of dialogue/grabbing/moving. The following takes the scenario of multi-agent dialogue as an example To illustrate, the dialogue includes multiple rounds of dialogue between a user agent (hereinafter referred to as a user) and a system agent (hereinafter referred to as a system), and the user agent has a task goal G=(C, R) , where C is the constraint condition (eg, Japanese restaurant in the city center), R is the demand type (eg, query the hotel's address, phone number), and the information requested by the user agent is stored in an external database that the system agent can access In DB, the user agent and the system agent interact and cooperate in the dialogue process to achieve user goals. There can be multiple domains in G, and two agents must complete all subtasks in each domain. Both agents are limited to only perceive part of the environmental information, as shown in Figure 2, that is, only the user agent knows the task goal G, and only the system agent can access DB. The only way for the two to know each other's information is through Conversational interaction. In the present invention, the two agents in the dialogue process communicate asynchronously. That is, the user agent and the system agent communicate alternately.

Exemplary method

The following describes a multi-agent action policy learning method according to an exemplary embodiment of the present invention with reference to FIG. 3 in conjunction with the application scenario of FIG. 1 . It should be noted that the above application scenarios are only shown for the convenience of understanding the spirit and principle of the present invention, and the embodiments of the present invention are not limited in this respect. Rather, embodiments of the present invention can be applied to any scenario where applicable.

FIG. 3 schematically shows an exemplary processing flow 300 of a multi-agent action policy learning method according to an embodiment of the present application. After the process flow 300 starts, step S310 is executed.

Step S310, the multi-agent samples corresponding actions according to their respective initial action strategies;

和动作

状态s＝(s₁，...，s_N)→s′＝(s′₁，...s′_N)，其中的状态转换取决于所有智能体依据各自的行动策略π_i(a_i|s_i)所采取的动作(a₁，...，a_N)，其中

在本实施例中，由用户首先行动，开启任务，即用户首先根据行动策略(μ(a^U|s^U))基于当前的状态选择待执行的动作，进而由系统根据行动策略(π(a^S|s^S))基于当前的状态选择待执行的动作，基于此，每一轮对话都可以采用状态-动作对表示：where each agent has its own state

and action

State _{s = ( s 1} , _. |s _i ) the action taken (a ₁ , . . . , a _N ), where

In this embodiment, the user first acts to start the task, that is, the user first selects the action to be performed based on the current state according to the action strategy (μ(a ^U |s ^U )), and then the system selects the action to be performed according to the action strategy (π(a U |s U )). ^S | s ^S )) selects the action to be performed based on the current state. Based on this, each round of dialogue can be represented by a state-action pair:

Among them, the superscript U represents the user, the superscript S represents the system, and the subscript represents a certain round of actions.

In an example of this implementation manner, the initial action strategy of each agent may be a random strategy, that is, each agent randomly samples actions to be performed from a set of optional actions based on the current state. Considering that when dealing with multi-domain, multi-objective complex (dialogue) tasks, the action space of each agent's action strategy may be very large. In this case, using a random strategy and training from scratch will consume a lot of resources and resources. time. Therefore, in an example of this implementation manner, the training process can be divided into two stages: first, the action strategy of each agent is pre-trained by using real action data (such as a dialogue corpus), and then step S310 is executed, using RL Improve pre-trained strategies. Since each agent only generates a few dialogue actions in a round of dialogue, in an example of this implementation, β-weighted logistic regression is used to perform policy pre-training to reduce the data sample bias:

L(X, Y; β)=-[β·Y ^T logσ(X)+(IY) ^T log(I-σ(X))],

where X is the state and Y is the action of the corpus in that task.

In addition, there may be multiple rounds of dialogue between the user and the system, where each round of dialogue refers to the dialogue content of the system and the dialogue content of the user in each round.

For example, use S(t) to represent the system dialogue content of the tth round, and U(t) to represent the user dialogue content of the tth round, where t represents the round sequence number, t=1, 2, 3, . . . For example, S(1) represents the system dialog content of the first round, U(1) represents the system dialog content of the first round, and so on.

It should be noted that in each round of dialogue, the speaking time of the system dialogue content is after the speaking time of the user dialogue content. That is, in a single round of dialogue, the user first issues a query, and the system gives a corresponding response.

由以下组成：(I)本轮对话的用户操作

(II)上一轮对话的系统动作

(III)跟踪用户提供的约束槽和请求槽的置信状态b_t；以及(IV)来自DB的查询结果数量的嵌入向量qt。In this embodiment, the action strategy π of the system determines the action a ^S to be performed by the system according to the state s ^S of the system, so as to give the user an appropriate response. Each system action a ^S is a subset of the action set A. In this embodiment, the system state of the t-th round of dialogue

It consists of the following: (I) User operation of this round of dialogue

(II) System actions of the previous round of dialogue

(III) Confidence states _bt that track user-supplied constraint and request slots; and (IV) an embedding vector qt of the number of query results from the DB.

由以下组成：(I)上一轮对话的系统动作

(II)上一轮对话的用户动作

(III)目标状态g_t，表示需要发送的剩余约束和需求类型；(IV)差异向量ct，指示系统响应和约束条件C之间的差异信息。The user policy μ decides the action a ^U will perform by the user according to the user state s ^U to communicate its constraints and requirements to the system. In this embodiment, the user state

It consists of the following: (I) System actions of the previous round of dialogue

(II) User action of the previous round of dialogue

(III) Target state g _t , indicating the remaining constraints and demand types that need to be sent; (IV) Difference vector ct , indicating the difference information between the system response and the constraint condition C.

A user's action is an abstract representation of an intent (constraint and requirement type), which can be represented by a quadruple of domain, requirement type, slot type, and slot value (eg [restaurant, notification requirement, food, italian]). The above example indicates that the user wants the system to inform about restaurants with Italian dishes. To be clear, there may be multiple intents in a round of conversation.

In addition to predicting user actions, the user policy is also configured to output an action termination signal T, ie μ=μ(a ^U , T|s ^U ). That is, when the user has no remaining tasks to perform, the termination signal T is output.

After obtaining the current state of each agent and the action to be performed, the new state after the action can be estimated, and then the advantage obtained after the action is calculated to update the action strategy of each agent.

As an example, in the dialog process, the user and the system have started a dialog. In order to update the action strategy of each agent, step S320 is executed next.

In step S320, the advantages obtained by the multi-agents after performing the corresponding actions are estimated respectively;

Specifically, first obtain the new states reached by each agent after performing the corresponding action;

Then, calculate the current state of each agent and the return after reaching the new state;

Finally, according to the current state of each agent and the reward after reaching the new state, the advantages obtained by each agent after performing the corresponding action are calculated.

其中r为环境在所述智能体选择某一动作后反馈的奖励，γ为折扣因子。Among them, the reward of the current state of an agent is the accumulation of the reward obtained. Specifically, in the process of reinforcement learning, after each action of the agent, the environment will feed back a reward value to the agent. As an example, The expected return of an agent's t-round dialogue

where r is the reward that the environment feeds back after the agent chooses an action, and γ is the discount factor.

至少由以下部分组成：In traditional reinforcement learning, there is often only one agent that needs to learn a strategy, that is to say, the environment only needs to give feedback to the one agent. Consider, on the one hand, that the roles of users and systems are not the same. The user will actively initiate the task and may change the task during the dialogue, while the system will only passively respond to the user and return appropriate information, that is, the performance of the user and the system cannot be completely judged by a unified standard, so each agent should be Rewards are calculated separately. In one example of this implementation, the system rewards

It consists of at least the following parts:

的惩罚；(I) Penalties for inaction; for example, in a round of dialogue, the system does not give any response

punishment;

(II) Penalty for not immediately providing corresponding assistance when the user agent requests assistance; for example, if the user sends a request in a round of dialogue, but the system does not respond to the corresponding information in time, it will be punished;

(III) Subtask completion reward; that is, the reward for helping the user complete the subtask according to the request sent by the user. For example, in a round of dialogue, if the user requests to inform a certain information, the system gives the user an appropriate response, and the corresponding reward is fed back to the system.

至少由以下部分组成：User reward

It consists of at least the following parts:

(I) Penalties for inaction; similar to system rewards, i.e.

(II) There is a penalty for executing a new subtask when the subtask is not completed; that is, when there is still a constraint to inform the system, if the user requests new information, a penalty is given;

(III) The user action reward is determined based on whether the user agent performs all actions that are beneficial to completing the task. For example, in this implementation, it is determined based on whether the user informs the system of all constraints C and requirement types R.

Based on the above methods, in a round of dialogue, system rewards and user rewards can be clearly obtained based on the state of each agent.

However, considering that on the other hand, two agents communicate and collaborate to accomplish the same task, the reward should also involve the (common) global goal of both agents.

至少由以下部分组成：Based on this, in one example of this implementation, the global reward

It consists of at least the following parts:

(I) Efficiency loss (penalty); for example, each round of dialogue will give a penalty within a preset range (such as a small negative value);

(II) Subtask completion reward; for example, after completing a subtask in a certain field in the user's overall goal G, a reward is given;

(III) All task completion rewards; for example, based on the user's overall goal G, the rewards for completing all tasks.

The above detailed decomposition of the reward is only for this embodiment. When using the method of the present application for strategy learning in other fields, it is not necessary to completely follow the above method for setting and calculation. In general, the above embodiments of the present invention It is only to explain how to calculate the reward separately for each agent (that is, independently calculate the reward and accumulate it), and for multi-agent policy learning that requires cooperation, on the basis of calculating the reward separately for each agent, Additional global rewards (accumulated global rewards) are calculated and distributed to corresponding agents in a preset manner. For example, the global rewards can be distributed to each agent according to a preset ratio/weight.

(上标G表示全局)。需要注意的是，全部任务成功和用户行动奖励仅全部任务结束后计算，并且系统奖励中计算的子任务完成奖励与全局奖励中的子任务完成奖励不同。In this embodiment, the reward of a certain agent is its own reward and global reward. For example, for the system, its total reward is the accumulated system reward and global reward:

(Superscript G indicates global). It should be noted that all task success and user action rewards are only calculated after all tasks are completed, and the subtask completion reward calculated in the system reward is different from the subtask completion reward in the global reward.

The action policy of each agent is designed to maximize the reward obtained by its corresponding agent. In order to make the action strategy learned by each agent to be the optimal action strategy, step S330 is executed next. In step S330, the action strategy of each agent is updated based on the advantages obtained after the multi-agent performs the corresponding action, so that the update Each subsequent action strategy can make the corresponding agent obtain higher rewards. Specifically, in an example of this implementation, the action strategy of the agent is updated based on the advantage after the agent performs a certain action, and the advantage is calculated based on the reward, that is, A(s)=r+γV(s')- V(s), after obtaining the advantages of different parts, for an agent, its action strategy can be updated based on the relative advantages of each part. As an example, the system strategy and user strategy can be based on Update the following way:

由

参数化，而用户策略μ_w由w参数化，即基于以上策略梯度分别更新参数

和w。Through the above methods, the strategy gradients of the user and the system can be obtained respectively. Based on this, the action strategy can be updated in a corresponding manner to obtain the optimal action strategy of each agent. In this embodiment, the system strategy

Depend on

parameterized, and the user policy μw is parameterized by _w , that is, the parameters are updated respectively based on the above policy gradients

and w.

The above embodiment illustrates how the action strategy of each agent is updated, that is, the update is based on the advantage of the agent performing a certain action in a certain state. Based on this, we can see that the calculation of the advantage is equally important, that is, the appropriate calculation of the advantage The method helps the agent to learn a more excellent action strategy. Specifically, in an example of this implementation manner, the value network (that is, the above-mentioned Critic) is set corresponding to different agents. In addition, in order to complete the Accumulate the rewards of different agents, and additionally set up a separate value network (Critic) to estimate the global return. To sum up, as shown in Figure 4, in this embodiment, a mixture of three independent value networks (Critic) is set up. The value network HVN (Hybrid ValueNetwork) is based on the corresponding part of the reward, the cumulative return, and the calculation advantage. On the basis of the above embodiment, the value network (Critic) of this embodiment is based on the cumulative rewards of the states of each agent. Specifically, in this embodiment, the dialogue state of each agent is first encoded to Learning state means:

where f( ) can be any neural network (for example, a multilayer perceptron, a convolutional neural network, etc., and the above two neural networks can be the same neural network, or the same or different. Not limited), tanh is a hyperbolic tangent function (activation function). Next, the reward can be calculated according to the cumulative reward of each agent's dialogue state:

where f _S , f _U and f _G are three arbitrary neural networks.

It should be noted that, the neural networks described in the above embodiments may be the same or different, which are not limited in this embodiment.

In an embodiment of the present application, the calculation method of the advantage is also continuously updated and optimized in the training process. Specifically, in the multi-agent strategy learning process, the value network is configured to minimize the use of pre- Optimization updates are made targeting the variance between the return predicted by the method and the return calculated by the value network. The preset method may be any known method for predicting rewards based on the state of a certain agent, such as a time series difference algorithm. Considering that the update frequency of the value network is high, this will cause the estimated value to change too much, especially for the multi-agent policy learning in this embodiment. Therefore, in an example of this implementation, the target The network optimizes and updates the value network to make the training process more stable. Specifically, the following loss functions can be used for optimization respectively to update the value network:

where HVNV _θ is parameterized by θ, θ ^- is the weight of the target network, and the total loss _LV is the sum of the estimated return losses for each component reward.

This embodiment proposes a multi-agent action strategy learning method. In a task-oriented machine learning scenario, multiple agents that cooperate with each other are simultaneously trained (that is, each agent learns its own action strategy), instead of using The pre-built simulator interacts with the agent without manual supervision, which greatly saves time, cost and resources. In addition, in this embodiment, in order to enable each agent to learn an excellent action strategy, a mixed value is constructed at the same time. The network divides the total reward corresponding to each agent and task, so that the performance of each agent is judged by different value networks, and the differentiated reward distribution enables each agent to learn action strategies that make their cumulative returns higher.

Exemplary device

After introducing the method of the exemplary embodiment of the present invention, next, referring to FIG. 5, a multi-agent action strategy learning apparatus is provided for the exemplary embodiment of the present invention, including multiple multi-agents and multiple value networks, Wherein, the multiple agents sample corresponding actions according to their respective initial action strategies;

The plurality of value networks respectively estimate the advantages obtained by the multi-agents after performing the corresponding actions;

The multiple agents update their respective action strategies based on the advantages obtained after performing the corresponding actions, so that the updated respective action strategies can enable the corresponding agents to obtain higher returns.

It should be noted that the number of agents and the number of value networks are not fixed, and can be set according to actual scene needs. For example, if there are several agents with the same roles (that is, the same value network can be used), no A value network is set corresponding to each agent, and in addition to different value networks for agents with different roles, an additional value network for calculating global returns can be set. The above value networks can be mixed to form a mixed value network.

In an example of this implementation manner, the apparatus further includes:

The pre-training module is configured to perform pre-training using real action data to obtain respective initial action strategies of the multi-agents.

In an embodiment of this implementation manner, the pre-training module is configured to perform pre-training based on real action data by using a weighted logistic regression method to obtain respective initial action strategies of the multi-agents.

In an example of this implementation manner, the device further includes a plurality of state encoding modules, configured to separately acquire new states reached after each agent performs a corresponding action;

The multiple value networks are also configured to calculate the current state of each agent and the reward after reaching the new state, respectively; Advantage.

In one example of this implementation, the multi-agent includes a user agent configured to aim at completing a task and a system agent configured to assist the user agent mission accomplished.

In an example of this implementation manner, the rewards after each agent reaches a certain state include at least its own rewards and global rewards.

In an example of this implementation manner, the reward of the user agent itself includes at least a penalty for inaction, a penalty for executing a new subtask when there are uncompleted subtasks, and a user action reward.

In one example of this embodiment, the user action reward is determined based on whether the user agent performs all actions that are beneficial for completing the task.

In an example of this implementation manner, the reward of the system agent itself is accumulated by at least one of the following: a penalty for inaction, a penalty for not immediately providing corresponding assistance when the user agent requests assistance, and a subtask completion reward .

In an example of this implementation manner, the global reward is accumulated by at least one of the following: an efficiency loss penalty, a subtask completion reward, and a total task completion reward.

In an example of this implementation manner, the user action reward or all task completion rewards are calculated after the action is terminated.

In an example of this implementation manner, the reward of each agent in a certain state is calculated by the corresponding value network.

In one example of this embodiment, different types of rewards are calculated by different value networks respectively.

In one embodiment of this embodiment, during the multi-agent policy learning process, the value network is configured to minimize the difference between the reward predicted by the preset method and the reward calculated by the value network The variance is optimized for the target.

In one embodiment of this implementation, the preset method is configured to predict rewards based on the state of a certain agent.

In an embodiment of this implementation manner, the preset method is a sequential difference algorithm.

In an example of this implementation manner, a target network is introduced to optimize and update the value network to make the training process more stable.

Exemplary Media

After introducing the method and apparatus of the exemplary embodiment of the present invention, next, the computer-readable storage medium of the exemplary embodiment of the present invention will be described with reference to FIG. 6 .

Please refer to FIG. 6 , the computer-readable storage medium shown is an optical disc 60, on which a computer program (ie, a program product) is stored, and when the computer program is run by the processor, the computer program described in the above-mentioned method embodiments will be implemented. For example: the multi-agent samples corresponding actions according to their respective initial action strategies; respectively estimates the advantages obtained by the multi-agents after performing the corresponding actions; based on the multi-agents performing the corresponding actions The advantage updates the action strategy of each agent, so that the updated action strategy can make the corresponding agent obtain higher returns. The specific implementation manner of each step is not repeated here.

It should be noted that examples of the computer-readable storage medium may also include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random Access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other optical and magnetic storage media will not be repeated here.

Exemplary Computing Device

After introducing the method, apparatus, and medium of the exemplary embodiment of the present invention, next, the computing device of the exemplary embodiment of the present invention will be described with reference to FIG. A block diagram of an exemplary computing device 70, which may be a computer system or server. The computing device 70 shown in FIG. 7 is only an example, and should not impose any limitations on the functions and scope of use of the embodiments of the present invention.

As shown in FIG. 7, components of computing device 70 may include, but are not limited to, one or more processors or processing units 701, system memory 702, and a bus 703 connecting different system components (including system memory 702 and processing unit 701).

Computing device 70 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by computing device 70, including both volatile and nonvolatile media, removable and non-removable media.

System memory 702 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 7021 and/or cache memory 7022 . Computing device 70 may further include other removable/non-removable, volatile/non-volatile computer system storage media. For example only, the ROM 7023 may be used to read and write to a non-removable, non-volatile magnetic medium (not shown in Figure 7, commonly referred to as a "hard disk drive"). Although not shown in Figure 7, a disk drive may be provided for reading and writing to removable non-volatile magnetic disks (eg "floppy disks"), as well as removable non-volatile optical disks (eg CD-ROM, DVD- ROM or other optical media) to read and write optical drives. In these cases, each drive may be connected to bus 703 through one or more data media interfaces. System memory 702 may include at least one program product having a set (eg, at least one) of program modules configured to perform the functions of various embodiments of the present invention.

A program/utility 7025 having a set (at least one) of program modules 7024, which may be stored, for example, in system memory 702, and such program modules 7024 include, but are not limited to: an operating system, one or more application programs, other program modules As well as program data, each or some combination of these examples may include an implementation of a network environment. Program modules 7024 generally perform the functions and/or methods in the described embodiments of the present invention.

Computing device 70 may also communicate with one or more external devices 704 (eg, keyboards, pointing devices, displays, etc.). Such communication may take place through input/output (I/O) interface 705 . Also, the computing device 70 may communicate with one or more networks (eg, a local area network (LAN), a wide area network (WAN), and/or a public network such as the Internet) through a network adapter 706 . As shown in FIG. 7 , network adapter 706 communicates with other modules of computing device 70 (eg, processing unit 701 , etc.) through bus 703 . It should be appreciated that, although not shown in FIG. 7 , other hardware and/or software modules may be used in conjunction with computing device 70 .

The processing unit 701 executes various functional applications and data processing by running the programs stored in the system memory 702, for example, acquiring at least one monitoring data of the first monitoring quantity and monitoring data of the second monitoring quantity, wherein the multiple The agents sample corresponding actions according to their respective initial action strategies; respectively estimate the advantages obtained by the multi-agents after performing the corresponding actions; update the action strategies of the agents based on the advantages obtained after the multi-agents perform the corresponding actions, So that the updated action strategies can make the corresponding agents obtain higher rewards.

It should be noted that although several units/modules or sub-units/modules of the time-series based abnormal data detection apparatus are mentioned in the above detailed description, this division is merely exemplary and not mandatory. Indeed, the features and functions of two or more units/modules described above may be embodied in one unit/module according to embodiments of the present invention. Conversely, the features and functions of one unit/module described above may be further subdivided to be embodied by multiple units/modules.

Furthermore, although the operations of the methods of the present invention are depicted in the figures in a particular order, this does not require or imply that the operations must be performed in the particular order, or that all illustrated operations must be performed to achieve desirable results. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined to be performed as one step, and/or one step may be decomposed into multiple steps to be performed.

While the spirit and principles of the present invention have been described with reference to a number of specific embodiments, it should be understood that the invention is not limited to the specific embodiments disclosed, nor does the division of aspects imply that features of these aspects cannot be combined to perform Benefit, this division is only for convenience of presentation. The invention is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims.

Through the above description, the embodiments of the present invention provide the following technical solutions, but are not limited thereto:

1. A multi-agent action strategy learning method, comprising:

The multi-agents sample corresponding actions according to their respective initial action strategies;

separately estimating the advantages obtained by the multi-agents after performing the corresponding actions;

The action strategy of each agent is updated based on the advantages obtained after the multi-agent performs the corresponding action, so that each updated action strategy can enable the corresponding agent to obtain a higher reward.

2. The method according to solution 1, wherein before the multi-agents perform corresponding actions according to their respective initial action strategies, the method further comprises:

Pre-training is performed using real action data to obtain the respective initial action policies of the multi-agents.

3. The method according to solution 2, wherein a weighted logistic regression method is used to perform pre-training based on real action data to obtain respective initial action strategies of the multi-agents.

4. The method of scheme 1, wherein the advantages obtained after the multi-agents perform corresponding actions are estimated separately, including:

Obtain the new state achieved by each agent after performing the corresponding action;

Calculate the current state of each agent and the return after reaching the new state;

According to the current state of each agent and the reward after reaching the new state, the advantages obtained by each agent after performing the corresponding action are calculated.

5. The method of claim 1, wherein the multi-agent includes a user agent configured to aim at completing a task and a system agent configured to assist the user The agent completes the task.

6. The method according to solution 5, wherein the rewards of each agent after reaching a certain state include at least its own rewards and global rewards.

7. The method of claim 6, wherein the reward of the user agent itself includes at least a penalty for inaction, a penalty for executing a new subtask when there is an uncompleted subtask, and a reward for user action.

8. The method of claim 7, wherein the user action reward is determined based on whether the user agent performs all actions conducive to completing the task.

9. The method of claim 6, wherein the reward of the system agent itself is accumulated by at least one of the following: a penalty for inaction, a penalty for not immediately providing corresponding assistance when the user agent requests assistance, and subtasks Completion reward.

10. The method of claim 6, wherein the global reward is accumulated by at least one of the following: an efficiency loss penalty, a subtask completion reward, and a full task completion reward.

11. The method of claim 8 or 10, wherein the user action reward or all task completion rewards are calculated after all actions are terminated.

12. The method according to solution 3, wherein the reward of each agent in a certain state is calculated by the corresponding value network.

13. The method of claim 12, wherein the rewards for different classes are calculated by different value networks, respectively.

14. The method of claim 13, wherein, during the multi-agent policy learning process, the value network is configured to minimize the difference between the return predicted using a preset method and the return calculated by the value network. The variance between them is optimized to update the target.

15. The method of claim 14, wherein the preset method is configured to predict rewards based on the state of an agent.

16. The method of claim 15, wherein the preset method is a sequential difference algorithm.

17. The method of claim 14, wherein a target network is introduced to optimize and update the value network to make the training process more stable.

18. A multi-agent action strategy learning device, comprising a plurality of multi-agents and a plurality of value networks, wherein the plurality of agents respectively sample corresponding actions according to their respective initial action strategies;

The plurality of value networks respectively estimate the advantages obtained by the multi-agents after performing the corresponding actions;

The multiple agents update their respective action strategies based on the advantages obtained after performing the corresponding actions, so that the updated respective action strategies can enable the corresponding agents to obtain higher returns.

19. The apparatus of claim 18, wherein the apparatus further comprises:

The pre-training module is configured to perform pre-training using real action data to obtain respective initial action strategies of the multi-agents.

20. The apparatus of claim 19, wherein the pre-training module is configured to perform pre-training based on real action data using a weighted logistic regression method to obtain respective initial action strategies of the multi-agents.

21. The apparatus according to claim 18, wherein the apparatus further comprises a plurality of state encoding modules, configured to respectively acquire new states reached after each agent performs a corresponding action;

The multiple value networks are also configured to calculate the current state of each agent and the reward after reaching the new state, respectively; Advantage.

22. The apparatus of clause 18, wherein the multi-agent comprises a user agent and a system agent, the user agent being configured to target completion of a task, the system agent being configured to assist the user The agent completes the task.

23. The apparatus according to claim 22, wherein the rewards of each agent after reaching a certain state include at least its own rewards and global rewards.

24. The apparatus of claim 23, wherein the reward of the user agent itself includes at least a penalty for inaction, a penalty for performing a new subtask when there are uncompleted subtasks, and a user action reward.

25. The apparatus of clause 24, wherein the user action reward is determined based on whether the user agent performs all actions conducive to completing the task.

26. The apparatus of claim 23, wherein the reward of the system agent itself is accumulated by at least one of the following: a penalty for inaction, a penalty for not immediately providing corresponding assistance when the user agent requests assistance, and subtasks Completion reward.

27. The apparatus of claim 23, wherein the global reward is accumulated by at least one of an efficiency loss penalty, a subtask completion reward, and an overall task completion reward.

28. The apparatus of clause 25 or 27, wherein the user action reward or all task completion reward is calculated after the action is terminated.

29. The apparatus of claim 20, wherein the reward for a certain state of each agent is calculated by the corresponding value network.

30. The apparatus of clause 29, wherein different classes of rewards are calculated by different value networks, respectively.

31. The apparatus of claim 30, wherein, during the multi-agent policy learning process, the value network is configured to minimize the difference between the return predicted using a preset method and the return calculated by the value network. The variance between them is optimized to update the target.

32. The apparatus of claim 31, wherein the predetermined method is configured to predict rewards based on the state of an agent.

33. The apparatus of claim 32, wherein the preset method is a sequential difference algorithm.

34. The apparatus of claim 31, wherein a target network is introduced to optimize and update the value network to make the training process more stable.

35. A medium on which a computer program is stored, characterized in that: when the computer program is executed by a processor, the method according to any one of the solutions 1-17 is implemented.

36. A computing device, characterized in that: the computer device comprises a processor, and the processor is configured to implement the method according to any one of the solutions 1-17 when executing a computer program stored in a memory.

Claims (10)

1. A multi-agent action strategy learning method, comprising: The multi-agents sample corresponding actions according to their respective initial action strategies; separately estimating the advantages obtained by the multi-agents after performing the corresponding actions; The action strategy of each agent is updated based on the advantages obtained after the multi-agent performs the corresponding action, so that each updated action strategy can enable the corresponding agent to obtain a higher reward. 2. The method according to claim 1, wherein before the multi-agents perform corresponding actions according to their respective initial action strategies, the method further comprises: Pre-training is performed using real action data to obtain the respective initial action policies of the multi-agents. 3. The method of claim 2, wherein a weighted logistic regression method is used to perform pre-training based on real action data to obtain respective initial action strategies of the multi-agents. 4. The method of claim 1, wherein separately estimating the advantages obtained by the multi-agents after performing the corresponding actions, comprising: Obtain the new state achieved by each agent after performing the corresponding action; Calculate the current state of each agent and the return after reaching the new state; According to the current state of each agent and the reward after reaching the new state, the advantages obtained by each agent after performing the corresponding action are calculated. 5. A multi-agent action strategy learning device, comprising a plurality of multi-agents and a plurality of value networks, wherein the plurality of agents respectively sample corresponding actions according to their respective initial action strategies; The plurality of value networks respectively estimate the advantages obtained by the multi-agents after performing the corresponding actions; The multiple agents update their respective action strategies based on the advantages obtained after performing the corresponding actions, so that the updated respective action strategies can enable the corresponding agents to obtain higher returns. 6. The apparatus of claim 5, wherein the apparatus further comprises: The pre-training module is configured to perform pre-training using real action data to obtain respective initial action strategies of the multi-agents. 7. The apparatus of claim 6, wherein the pre-training module is configured to perform pre-training based on real action data using a weighted logistic regression method to obtain respective initial action strategies of the multi-agents. 8. The apparatus according to claim 5, wherein the apparatus further comprises a plurality of state encoding modules, configured to respectively acquire new states reached after each agent performs a corresponding action; The multiple value networks are also configured to calculate the current state of each agent and the reward after reaching the new state, respectively; and calculate the return obtained after each agent performs the corresponding action according to the current state of each agent and the return after reaching the new state. Advantage. 9. A medium on which a computer program is stored, characterized in that: when the computer program is executed by a processor, the method according to any one of claims 1-4 is implemented. 10. A computing device, characterized in that: the computer device comprises a processor, and the processor is configured to implement the method according to any one of claims 1-4 when executing a computer program stored in a memory.

Images