CN115648204A - Training method, device, equipment and storage medium of intelligent decision model - Google Patents

Abstract

The application discloses a training method, device, equipment and storage medium of an intelligent decision-making model, belonging to the field of computer technology. Through the technical solution provided by the embodiment of this application, the external information collected by the robot in the target environment is obtained, the external information is input into the intelligent decision-making model, and the distributed executor model of the intelligent decision-making model outputs multiple action branches. The action branches are actions that the robot may perform in the target environment when the external information is acquired. Based on the external information and the multiple action branches, the distribution of the reward value of each action branch is determined, that is, the multiple action branches are all evaluated. Based on the distribution of reward values of multiple action branches, reward aggregation is performed to determine mixed rewards and integrated rewards. Training the intelligent decision-making model based on the mixed reward, the integrated reward and external information can achieve a relatively stable training effect.

Description

Training method, device, equipment and storage medium of intelligent decision-making model

technical field

The present application relates to the field of computer technology, in particular to a training method, device, equipment and storage medium for an intelligent decision-making model.

Background technique

With the development of computer technology, Multi-Agent Reinforcement Learning (MARL) has made great progress in the fields of large-scale real-time battle games, robot control, automatic driving and stock trading.

In a multi-agent environment, affected by factors such as the potential interaction between agents and the uncertainty of the environment itself, the reward value obtained by the agent will be uncertain, and the reward value with uncertainty will affect Multi-Agent Learning. For example, in an environment including multiple robots, the interaction between robots and the uncertainty of the environment itself will affect the training effect of the robot's intelligent decision-making model, which is also a reinforcement learning model. It becomes very difficult and tricky to realize the stable learning of the multi-agent reinforcement learning model. Therefore, there is an urgent need for a method that can realize the stable learning of the multi-agent reinforcement learning model.

Contents of the invention

Embodiments of the present application provide a training method, device, device, and storage medium for an intelligent decision-making model, which can improve the training effect of a robot's decision-making model. Described technical scheme is as follows:

On the one hand, a kind of training method of intelligent decision-making model is provided, and described method comprises:

Obtain external information collected by the robot in the target environment, the external information includes external environment information and interaction information, the external environment information is information obtained by the robot observing the target environment, and the interaction information is information from interactions with other robots in the target environment;

The external information is input into the intelligent decision-making model of the robot, and the distributed executor model of the intelligent decision-making model performs prediction based on the external information, and outputs multiple action branches of the robot, and the action branches are all Actions that the robot may perform in the target environment;

determining a distribution of reward values for each of the action branches based on the external information and the plurality of action branches;

Reward aggregation is performed on the sampling reward values sampled in the reward value distribution of each of the action branches to obtain a mixed reward and an integrated reward;

An intelligent decision-making model of the robot is trained based on the mixed reward, the integrated reward and the external information.

In a possible implementation manner, the prediction is performed by the distributed executor model of the intelligent decision-making model based on the external information, and outputting multiple action branches of the robot includes:

Performing at least one of full connection, convolution, and attention encoding on the external information by the distributed executor model of the intelligent decision-making model to obtain the external information characteristics of the external information;

The distributed executor model of the intelligent decision-making model performs full connection and normalization on the external information features, and outputs multiple action branches of the robot.

In a possible implementation manner, the determining the reward value distribution of each of the action branches based on the external information and the plurality of action branches includes:

Inputting the external information and the plurality of action branches into a reward value estimation model, performing reward value distribution estimation based on the external information and the plurality of action branches through the reward value estimation model, and outputting each of the action branches Distribution of reward values.

In a possible implementation manner, performing reward aggregation on the sampled reward values sampled in the reward value distribution of each of the action branches to obtain a mixed reward and an integrated reward includes:

Sampling the reward value distribution of each of the action branches to obtain the sampling reward value of each of the action branches;

performing strategy weighted fusion on the sampling reward values of the various action branches to obtain the mixed reward;

Perform any one of global average, local average and direct selection on the sampling reward values of each of the action branches to obtain the integrated reward.

In a possible implementation manner, the training of the distributed executor model of the robot's intelligent decision-making model based on the mixed reward, the integrated reward and the external information includes:

training a reward value estimation model of the intelligent decision-making model based on the external information;

Based on the mixed reward, the external information, and the critic model of the intelligent decision-making model, the action advantage values of the multiple action branches are obtained, and the critic model is used to evaluate the multiple action branches based on the external information. The action branch is evaluated;

Based on the action advantage value and the integration reward, the distributed executor model of the intelligent decision-making model is trained.

In a possible implementation manner, the acquiring action advantage values of the plurality of action branches based on the mixed reward, the external information, and the critic model of the intelligent decision-making model includes:

Train the critic model of the intelligent decision-making model based on the mixed reward and the external information; input the external information and the multiple action branches into the critic model, and output the multiple action branches from the critic model The action advantage value of each action branch.

In a possible implementation manner, the training the intelligent decision-making model based on the action advantage value and the integrated reward includes:

Constructing a loss function of the distributed executor model based on the action advantage value and the integrated reward;

Based on the loss function, the distributed executor model is trained using a gradient descent method.

In one aspect, a training device for an intelligent decision-making model is provided, the device comprising:

An external information acquisition module, configured to acquire external information collected by the robot in the target environment, the external information includes external environment information and interaction information, the external environment information is information obtained by the robot observing the target environment, and the The interaction information is information obtained by the interaction between the robot and other robots in the target environment;

an action prediction module, configured to input the external information into the intelligent decision-making model of the robot, and the distributed executor model of the intelligent decision-making model performs prediction based on the external information, and outputs multiple action branches of the robot, The action branch is an action that the robot may perform in the target environment;

A reward value prediction module, configured to determine the reward value distribution of each of the action branches based on the external information and the plurality of action branches;

A reward value aggregation module, configured to perform reward aggregation on sampled reward values sampled in the reward value distribution of each of the action branches to obtain mixed rewards and integrated rewards;

A training module, configured to train an intelligent decision-making model of the robot based on the mixed reward, the integrated reward and the external information.

In a possible implementation manner, the action prediction module is configured to perform at least one of full connection, convolution and attention coding on the external information by the distributed executor model of the intelligent decision-making model, The external information features of the external information are obtained; the distributed executor model of the intelligent decision-making model performs full connection and normalization on the external information features, and outputs multiple action branches of the robot.

In a possible implementation manner, the reward value prediction module is configured to input the external information and the plurality of action branches into a reward value estimation model, and the reward value estimation model is based on the external information and the The multiple action branches are used to estimate the reward value distribution, and the reward value distribution of each of the action branches is output.

In a possible implementation manner, the reward value aggregation module is configured to sample the reward value distribution of each of the action branches to obtain the sampling reward value of each of the action branches; the sampling of each of the action branches The reward value is fused by policy weighting to obtain the mixed reward; the sampling reward value of each of the action branches is subjected to any one of global average, local average and direct selection to obtain the integrated reward.

In a possible implementation manner, the training module is configured to acquire action advantage values of the plurality of action branches based on the mixed reward, the external information, and the critic model of the intelligent decision-making model, so The critic model is used to evaluate the multiple action branches based on the external information; based on the action advantage value and the integrated reward, the distributed executor model of the intelligent decision-making model is trained.

In a possible implementation manner, the training module is configured to train a reward value estimation model of the intelligent decision-making model based on the external information;

In a possible implementation manner, the training module is configured to train the critic model based on the mixed reward and the external information; input the external information and the plurality of action branches into the critic A model, wherein the critic model outputs the action advantage values of the plurality of action branches.

In a possible implementation manner, the training module is configured to form a loss function of the distributed actor model based on the action advantage value and the integrated reward; based on the loss function, a gradient descent method is used The distributed executor model is trained.

In one aspect, a computer device is provided, the computer device includes one or more processors and one or more memories, at least one computer program is stored in the one or more memories, and the computer program is executed by the One or more processors are loaded and executed to realize the training method of the intelligent decision-making model.

In one aspect, a computer-readable storage medium is provided, wherein at least one computer program is stored in the computer-readable storage medium, and the computer program is loaded and executed by a processor to realize the training method of the intelligent decision-making model.

In one aspect, a computer program product or computer program is provided, the computer program product or computer program includes program code, the program code is stored in a computer-readable storage medium, and a processor of a computer device reads the program code from the computer-readable storage medium The program code, the processor executes the program code, so that the computer device executes the above-mentioned intelligent decision-making model training method.

Through the technical solution provided by the embodiment of this application, the external information collected by the robot in the target environment is obtained, the external information is input into the intelligent decision-making model, and the distributed executor model of the intelligent decision-making model outputs multiple action branches. The action branches are actions that the robot may perform in the target environment when the external information is acquired. Based on the external information and the multiple action branches, the distribution of the reward value of each action branch is determined, that is, the multiple action branches are all evaluated. Based on the distribution of reward values of multiple action branches, reward aggregation is performed to determine mixed rewards and integrated rewards. Training the intelligent decision-making model based on the mixed reward, the integrated reward and external information can achieve a relatively stable training effect.

Description of drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings that need to be used in the description of the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application. For those skilled in the art, other drawings can also be obtained based on these drawings without creative effort.

FIG. 1 is a schematic diagram of an implementation environment of a training method for an intelligent decision-making model provided in an embodiment of the present application;

Fig. 2 is a flow chart of a training method for an intelligent decision-making model provided by an embodiment of the present application;

Fig. 3 is a flow chart of a training method for an intelligent decision-making model provided by an embodiment of the present application;

FIG. 4 is a framework diagram of a training method for an intelligent decision-making model provided in an embodiment of the present application;

Fig. 5 is a schematic diagram of an experimental result provided in an embodiment of the present application;

FIG. 6 is a schematic structural diagram of a training device for an intelligent decision-making model provided in an embodiment of the present application;

FIG. 7 is a schematic structural diagram of a terminal provided in an embodiment of the present application;

FIG. 8 is a schematic structural diagram of a server provided by an embodiment of the present application.

Detailed ways

In order to make the purpose, technical solution and advantages of the present application clearer, the implementation manners of the present application will be further described in detail below in conjunction with the accompanying drawings.

In this application, the terms "first" and "second" are used to distinguish the same or similar items with basically the same function and function. It should be understood that "first", "second" and "nth" There are no logical or timing dependencies, nor are there restrictions on quantity or order of execution.

In order to describe the technical solutions provided by the embodiments of the present application, terms involved in the embodiments of the present application are firstly introduced below.

Artificial Intelligence (AI) is a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

Machine Learning (ML) is a multi-field interdisciplinary subject, involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. Specialize in the study of how computers simulate or implement human learning behaviors to acquire new knowledge or skills, and reorganize existing knowledge sub-models to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent, and its application pervades all fields of artificial intelligence. Machine learning and deep learning usually include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and teaching and learning.

Deep Reinforcement Learning (DRL): Deep reinforcement learning combines the perception ability of deep learning and the decision-making ability of reinforcement learning, and can be directly controlled according to the input image. It is an artificial intelligence method closer to the way of human thinking. . Deep reinforcement learning is a branch of deep learning. Deep reinforcement learning can be attributed to a five-tuple representation of Markov decision process (Markov Decision Processes, MDP), that is, <S, A, R, P, γ>, respectively Environment state, action, reward, state transition matrix, cumulative discount factor. The agent obtains state and rewards from the environment, and generates actions to act on the environment, and the environment obtains the actions of the agent and generates the next state to the agent according to the current state. The purpose of the agent is to obtain long-term profit maximization.

Normalization: Map the sequence of values with different value ranges to the (0, 1) interval to facilitate data processing. In some cases, normalized values can be implemented directly as probabilities.

Learning Rate: It is used to control the learning progress of the model. The learning rate can instruct the model how to use the gradient of the loss function to adjust the network weights in the gradient descent method. If the learning rate is too large, the loss function may directly cross the global optimal point, and the loss is too large at this time; if the learning rate is too small, the change speed of the loss function is very slow, which will greatly increase the convergence complexity of the network, and quickly It is easy to get trapped in local minima or saddle points.

Embedded coding (Embedded Coding): Embedded coding represents a corresponding relationship mathematically, that is, the data on the X space is mapped to the Y space through a function F, where the function F is an injective function, and the result of the mapping is a structure preservation. The injective function indicates that the data after mapping is uniquely corresponding to the data before mapping, and the structure preservation indicates that the size relationship of the data before mapping is the same after mapping. For example, there are data X ₁ and X ₂ before mapping, and X ₁ is obtained after mapping Y 1 corresponds to X ₂ and Y ₂ corresponds to X ₂ . If the data before mapping X ₁ >X ₂ , correspondingly, the data Y ₁ after mapping is greater than Y ₂ . For words, it is to map words to another space for subsequent machine learning and processing.

Attention Weight: It can indicate the importance of a certain data in the training or prediction process, and the importance indicates the influence of the input data on the output data. Data with high importance has a higher value of attention weight, and data with low importance has a lower value of attention weight. In different scenarios, the importance of data is not the same, and the process of training the attention weight of the model is also the process of determining the importance of data.

After introducing the nouns involved in the embodiment of the present application, the implementation environment of the embodiment of the present application is introduced below.

FIG. 1 is a schematic diagram of an implementation environment of a method for training an intelligent decision-making model provided by an embodiment of the present application. Referring to FIG. 1 , the implementation environment may include a terminal 110 and a server 140 .

The terminal 110 is connected to the server 140 through a wireless network or a wired network. Optionally, the terminal 110 is a smart phone, a tablet computer, a notebook computer, a desktop computer, etc., but not limited thereto. The terminal 110 is installed and runs an application program supporting intelligent decision-making model training.

The server 140 is an independent physical server, or a server cluster or a distributed system composed of multiple physical servers, or provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, and middleware services , domain name service, security service, distribution network (Content Delivery Network, CDN), and cloud servers for basic cloud computing services such as big data and artificial intelligence platforms. The server 140 provides background services for applications running on the terminal.

Those skilled in the art may know that the number of the foregoing terminals may be more or less. For example, there is only one terminal above, or there are dozens or hundreds of terminals, or more, and at this time, the above implementation environment also includes other terminals. The embodiment of the present application does not limit the number of terminals and device types.

After introducing the implementation environment of the embodiment of the present application, the application scenario of the embodiment of the present application is introduced below.

The technical solution provided by the embodiment of this application can be applied in the scenario of multi-robot navigation, and the intelligent decision-making model of the robot can be trained by using the technical solution provided by the embodiment of the application. The intelligent decision-making model can output actions according to the external information observed by the robot. Actions enable navigation of the robot. Configuring the intelligent decision-making model for multiple robots in an environment can realize the purpose of multi-robot navigation. In the above multi-robot navigation scenario, each robot can be regarded as an agent.

Alternatively, the technical solutions provided by the embodiments of the present application can also be applied to other scenarios that include multiple agents. The embodiment of the present application does not limit the scene where the objects are engaged in battle in the game scene. In the following description process, the application of the technical solutions provided by the embodiments of the present application in the scenario of multi-robot navigation is taken as an example for description.

After introducing the implementation environment and application scenarios of the embodiments of the present application, the technical solutions provided by the embodiments of the present application will be introduced below. It should be noted that, in the following description of the technical solution provided by the present application, the server is used as an execution subject as an example. In other possible implementation manners, the terminal may also be used as an execution subject to execute the technical solution provided in the present application, and the embodiment of the present application does not limit the type of the execution subject.

Fig. 2 is a flow chart of a method for training an intelligent decision-making model provided by an embodiment of the present application. Referring to Fig. 2 , taking the execution subject as a server as an example, the method includes the following steps.

201. The server acquires the external information collected by the robot in the target environment. The external information includes external environment information and interaction information. The external environment information is the information obtained by the robot observing the target environment, and the interaction information is the Information obtained from interactions with other robots in the environment.

Wherein, the target environment includes a plurality of robots, and the robot in step 201 is any one of the plurality of robots. The external information includes external environment information and interaction information. The external environment information is the information obtained by the robot observing the target environment, that is, the information obtained by the robot observing the target environment through multiple sensors. For example, the external information includes the robot The environmental image acquired by the image sensor of the robot also includes the position acquired by the position sensor of the robot, and the attitude information acquired by the gyroscope of the robot. Sensors vary. The interaction information is information obtained by the robot interacting with other robots in the target environment. For example, the interaction information includes collision information between the robot and other robots.

202. The server inputs the external information into the robot's intelligent decision-making model, and the distributed executor model of the intelligent decision-making model makes predictions based on the external information, and outputs multiple action branches of the robot. Actions that may be performed in the target environment.

Among them, the distributed executor model of the intelligent decision-making model is used to predict actions based on external information. The external information is the observation of the target environment by the robot. The function of the distributed executor model is to make execution based on the observation of the robot. The decision of which action to take. The multiple action branches are actions that the robot may perform in the target environment when the external information is acquired.

203. Based on the external information and the multiple action branches, the server determines the reward value distribution of each of the action branches.

Among them, the reward value distribution of each action branch is the potential reward distribution that can be obtained by executing each action branch in the case of the external information, and the reward distribution is related to the actions taken by the agent, the interaction between the agents and the external information. This agent is also a robot.

204. The server rewards and aggregates the sampled reward values sampled in the reward value distribution of each action branch to obtain a mixed reward and an integrated reward.

Among them, Mixed reward Aggregation is used to train the critic model; Lumped reward Aggregation is used to train the distributed executor model.

205. The server trains an intelligent decision-making model of the robot based on the mixed reward, the integrated reward, and the external information.

Through the technical solution provided by the embodiment of this application, the external information collected by the robot in the target environment is obtained, the external information is input into the intelligent decision-making model, and the distributed executor model of the intelligent decision-making model outputs multiple action branches. The action branches are actions that the robot may perform in the target environment when the external information is acquired. Based on the external information and the multiple action branches, the distribution of the reward value of each action branch is determined, that is, the multiple action branches are all evaluated. Based on the distribution of reward values of multiple action branches, reward aggregation is performed to determine mixed rewards and integrated rewards. Training the intelligent decision-making model based on the mixed reward, the integrated reward and external information can achieve a relatively stable training effect.

The above-mentioned steps 201-205 are a brief introduction to the technical solution provided by the embodiment of the present application. The technical solution provided by the embodiment of the present application will be explained more clearly below with some examples. See Figure 3 and Figure 4, and the execution subject is the server As an example, the method includes the following steps.

301. The server obtains the external information collected by the robot in the target environment. The external information includes external environment information and interaction information. The external environment information is the information obtained by the robot observing the target environment, and the interaction information is the Information obtained from interactions with other robots in the environment.

Wherein, the target environment includes a plurality of robots, and the robot in step 301 is any one of the plurality of robots. The external information includes external environment information and interaction information. The external environment information is the information obtained by the robot observing the target environment, that is, the information obtained by the robot observing the target environment through multiple sensors. For example, the external information includes the robot The environmental image acquired by the image sensor of the robot also includes the position acquired by the position sensor of the robot, and the attitude information acquired by the gyroscope of the robot. Sensors vary. The interaction information is information obtained by the robot interacting with other robots in the target environment. For example, the interaction information includes collision information between the robot and other robots. In some embodiments, the target environment is a simulated virtual environment.

In some embodiments, multiple robots in the target environment interact with the target environment in a distributed interactive manner. During the interaction process, each robot can collect external information at different times, and multiple robots can collect external information at different times. The external information is stored in the experience buffer pool (Replay Buffer), the server can obtain the external information collected by the robot at different times from the experience buffer pool, the experience buffer pool is equivalent to a database, and the external information stored in the experience buffer pool The information can be used as a training sample when training the robot's intelligent decision-making model. The form of the experience buffer pool is shown in Figure 4. Since the robot is any robot among the plurality of robots, the server can acquire the external information collected by the robot in the target environment from the experience buffer pool based on the robot's identifier. In some embodiments, the experience buffer pool also stores the actions performed by the robot in the target environment and the reward values of the actions performed by the robot in the target environment.

Under the framework of deep reinforcement learning, the external information is also the state of the environment, and the robot is also the agent. The reward uncertainty caused by the interaction between the agents increases exponentially with the increase of the number of agents, and There may be multiple rewards corresponding to the state-action pair, so different action branches are selected for reward distribution modeling in the subsequent processing. A straightforward approach is to directly choose to model the reward distribution over the joint state-action space. But as the number of agents increases, the method suffers from huge uncertainties due to the interactions among agents. In order to cope with the above challenges, the embodiment of the present application proposes another method to achieve the goal. From the perspective of an agent, treating other agents as part of the environment simplifies the problem, so that we only need to focus on the estimation of the action branch reward distribution of each agent, rather than all agents. In the above steps, that is, the interaction information is regarded as a part of the environment, and the external information used for action prediction includes external environment information and interaction information, thereby simplifying the influence of other robots on the robot. The idea of multi-action branch reward distribution estimation is intuitively inspired by the fact that the human brain can always speculate on the possible consequences of related decision-making behaviors based on existing facts.

302. The server inputs the external information into the robot's intelligent decision-making model, and the distributed executor model of the intelligent decision-making model makes predictions based on the external information, and outputs multiple action branches of the robot. Actions that may be performed in the target environment.

Among them, the distributed executor model is used to predict actions based on external information. The external information is the observation of the target environment by the robot. The function of the distributed executor model is to make a decision on which action to execute based on the observation of the robot. . The multiple action branches are actions that the robot may perform in the target environment when the external information is obtained. In some embodiments, the distributed actor model is also called a distributed actor network (Decentralized Actor).

In a possible implementation, the server inputs external information into the robot's distributed actor model, and the distributed actor model makes predictions based on the external information and decision-making strategies, and outputs multiple action branches of the robot.

Among them, the decision policy (Policy) is also the model parameter of the distributed executor model, and training the distributed executor model is to optimize the decision policy (adjust the model parameters), so that the distributed executor model can More accurate action prediction based on external information.

For example, the server inputs the external information into the distributed executor model of the robot, and the distributed executor model performs feature extraction on the external information to obtain the external information features of the external information, for example, the distributed executor The former model performs at least one of full connection, convolution and attention encoding on the external information to obtain the external information features of the external information. The distributed executor model performs full connection and normalization on the external information features, and outputs the multiple action branches. Wherein, after the distributed executor model is fully connected and normalized to the external information features, a probability set can be obtained, and the probability set includes multiple probabilities, and each probability corresponds to an action. The server determines the action corresponding to the highest number of probabilities of the previous target in the probability set as the action branch. For example, the number of actions that the robot can perform in the target environment is N, the probability set output by the distributed executor model includes the probability of the N actions, and the server selects M optional actions (M It can also be equal to N) to be determined as an action branch. Referring to Figure 4, the distributed executor model of the robot is the Actor in Figure 4.

303. The server determines the reward value distribution of each action branch based on the external information and the plurality of action branches.

Among them, the reward value distribution of each action branch is the potential reward distribution that can be obtained by executing each action branch in the case of the external information, and the reward distribution is related to the actions taken by the agent, the interaction between the agents and the external information .

In a possible implementation manner, the server inputs the external information and the multiple action branches into the reward value estimation model, uses the reward value estimation model to estimate the reward value distribution based on the external information and the multiple action branches, and outputs each The reward value distribution of the action branch.

Wherein, referring to FIG. 4, the reward value estimation model is also called a multi-action-branch reward estimation (Multi-action-branch Reward Estimation) device, which is used to output a reward value based on the environment state (external information) and action (action branch) .

去建模智能体i在观测到

的情况下选择第k个动作分支上的动作

可能存在的奖励的分布。其中，

表示奖励分布空间，

表示智能体i在第k个动作分支上的奖励分布，

表示智能体i的有关奖励分布的参数，

表示机器人i观测到的该外部信息。使用

表示在观测

下智能体i所有动作分支上估计的分布。进一步来说，可以通过优化目标函数实现多动作分支分布估计，进而实现捕捉奖励函数的不确定性，为接下来减小奖励不确定性带来的影响提供前置条件，目标函数的形式为下述公式(1)和公式(2)。For example, at each time step t, for each agent (robot) i, the server uses the multi-action branch reward estimator

to model agent i observing

In the case of choosing the action on the kth action branch

Distribution of possible rewards. in,

represents the reward distribution space,

Represents the reward distribution of agent i on the kth action branch,

Represents the parameters of agent i's reward distribution,

Indicates the external information observed by robot i. use

means observing

The estimated distribution over all action branches of the lower agent i. Furthermore, it is possible to estimate the distribution of multi-action branches by optimizing the objective function, thereby capturing the uncertainty of the reward function and providing preconditions for reducing the impact of reward uncertainty. The form of the objective function is as follows Formula (1) and Formula (2) above.

是关于奖励分布的正则化项。在一些实施例中，

可以采用高斯分布，但是优化目标其实并没有限制必须使用高斯分布，其他的分布形式也是可以的。当采用高斯分布的时候，

其中var(μ)表示关于高斯均值向量的方差。α和β是超参数。Among them, formula (1) is the optimization function of the multi-action branch reward estimator for all agents, _Jr is the function value of the objective function, in formula (2), the superscript of the agent number is omitted, and D is the experience buffer pool, -log P[.] for negative log-likelihood loss,

is a regularization term on the reward distribution. In some embodiments,

Gaussian distribution can be used, but the optimization objective does not limit the use of Gaussian distribution, and other distribution forms are also possible. When a Gaussian distribution is used,

where var(μ) represents the variance about the Gaussian mean vector. α and β are hyperparameters.

The multi-action branch reward estimation can predict the potential rewards of all action branches, just like human beings will consider every possible consequence, and usually consider the consequences of all possible actions before making a decision. Therefore, in order to better evaluate historical experience and use historical experience to achieve stable training, policy-weighted reward aggregation will be introduced later to weaken the impact of reward uncertainty during training.

304. The server rewards and aggregates the sampled reward values sampled in the reward value distribution of each action branch to obtain a mixed reward and an integrated reward.

中采样出来的奖励值。然后对嵌入奖励向量进行奖励聚合，获得两类奖励值用于更新集中式评论家网络(Centralized Critic)和分布式执行者网络(分布式执行者模型)。这两类奖励值分别是混合奖励(Mixed reward Aggregation)

它被用于训练集中式评论家V_γ，ψ；集成奖励(Lumped reward Aggregation)

它被用于训练分布式执行者

集中式评论家网络用于对分布式执行者网络输出的动作进行评价，该集中式评论家网络也被称为评论家模型。Among them, at each time step, the agent i can only take a certain action a _k to obtain the reward r _k on this action branch, and the action branch referred to here is the action a _k . However, after the above multi-action branch reward estimation, r _k can be enhanced into an embedding reward vector, the reward value at the kth position of the embedding reward vector is r _k , and the values at other positions are from

The reward value sampled from . Then reward aggregation is performed on the embedded reward vector, and two types of reward values are obtained for updating the centralized critic network (Centralized Critic) and the distributed executor network (distributed executor model). These two types of reward values are Mixed reward Aggregation

It is used to train the centralized critic V _{γ, ψ} ; integrated reward (Lumped reward Aggregation)

It is used to train distributed executors

The centralized critic network is used to evaluate the actions output by the distributed executor network, and the centralized critic network is also called the critic model.

First, the method for the server to obtain the sampling reward value will be described.

In a possible implementation manner, the server samples the reward value distribution of each action branch to obtain the sampled reward value of each action branch.

The method for the server to obtain the mixed reward and the integrated reward in step 304 above will be introduced respectively below.

First, the method for the server to obtain mixed rewards is introduced.

In a possible implementation manner, the server performs policy-weighted fusion on the sampled reward values of each action branch to obtain the mixed reward.

其中函数

表示一种替换操作，这个替换操作是将某个动作分支上的奖励替换为从环境获得的奖励，获得的最终结果就是

这个向量是通过将r_k放置在嵌入向量的第k个位置，其他的位置放上

对应的值。Among them, the reward values of the multiple action branches form the embedding reward vector

which function

Represents a replacement operation, this replacement operation is to replace the reward on an action branch with the reward obtained from the environment, and the final result obtained is

This vector is obtained by placing r _k in the kth position of the embedding vector, and the other positions are placed

corresponding value.

获得经过策略加权之后的混合奖励

其中，函数g(.)表示两种操作，它们分别是：平均操作gMO：即直接对输入的量进行平均操作；选择操作g_SS：即直接选择和智能体i对应那个mⁱ作为g(.)的输出结果，

表示策略加权的权重，o表示外部信息。总结来说，服务器能够通过下述公式(3)来对该多个动作分支的奖励值进行策略加权融合，得到该混合奖励。For example, the server formula

Get mixed rewards after policy weighting

Among them, the function g(.) represents two operations, which are: average operation gMO: directly average the input amount; selection operation _gSS : directly select the mi corresponding to agent ⁱ as g(. ) output result,

Indicates the weight of policy weighting, and o indicates external information. In summary, the server can use the following formula (3) to carry out strategic weighted fusion of the reward values of the multiple action branches to obtain the mixed reward.

表示智能体i对应的混合奖励，m¹，…，m^N为智能体i的多个动作分支的奖励值，N为动作分支的数量，N为正整数。in,

Indicates the mixed rewards corresponding to agent i, m ¹ ,..., m ^N are the reward values of multiple action branches of agent i, N is the number of action branches, and N is a positive integer.

It should be noted that the mixed reward can be determined by any one of the above average operation g _MO and selection operation g _SS , which is not limited in this embodiment of the present application.

The following is an introduction to the method for the server to obtain integration rewards.

In a possible implementation manner, the server performs any one of global average, local average and direct selection on the sampling reward values of each action branch to obtain the integrated reward.

For example, the server obtains the integration reward through the following formula (4).

表示智能体i对应的集成奖励，函数l(.)有三种常见的表达形式：平均操作(全局平均)l_MO，直接对输入的所有嵌入奖励向量

求平均，获得的输出作为集成奖励

简化平均(局部平均)l_SMO，只选择和智能体i所对应的嵌入奖励向量mⁱ求平均作为集成奖励

选择操作(直接选取)l_SS，直接选择环境奖励反馈

作为输出。需要说明的是，可以采用上述任一种方式来获取该集成奖励，本申请实施例对此不作限定。in,

Indicates the integrated reward corresponding to the agent i, the function l(.) has three common expressions: the average operation (global average) l _MO , directly for all input embedded reward vectors

Averaging, the obtained output is used as the ensemble reward

Simplified average (local average) l _SMO , only select the embedding reward vector m ⁱ corresponding to agent i to average as the integrated reward

Select operation (direct selection) l _SS , directly select environment reward feedback

as output. It should be noted that the integration reward can be acquired in any of the above ways, which is not limited in this embodiment of the present application.

In some embodiments, referring to FIG. 4 , the server obtains the mixed reward and integrated reward through a reward aggregator (Reward Aggregation) in a distributed reward estimation (Distributional RewardEstimation) network. Wherein, the multi-action-branch reward estimation (Multi-action-branch Reward Estimation) device in the above step 303 also belongs to the distributed reward estimation network.

305. The server obtains the action advantage values of the multiple action branches based on the mixed reward, the external information, and the critic model of the intelligent decision-making model, and the critic model is used to evaluate the multiple action branches based on the external information .

Wherein, the critic model is also the above-mentioned centralized critic network.

In a possible implementation manner, the server trains the critic model based on the mixed reward. The server inputs the external information and the plurality of action branches into the critic model, and the critic model outputs action advantage values of the plurality of action branches.

In order to describe the foregoing implementation manner more clearly, the description of the foregoing implementation manner will be described below in two parts.

In the first part, the server trains the critic model based on the mixed reward and the external information.

In a possible implementation manner, the server trains the critic model based on the mixed reward, the multiple action branches, and the external information of the target environment collected by the robot in the next time step.

For example, the server inputs the mixed reward, the multiple action branches, the external information, and the external information of the target environment collected by the robot at the next time step into the critic model, and the critic model gives the mixed reward , the plurality of action branches, the external information, and the external information of the target environment collected by the robot in the next time step are encoded to obtain encoded features. Through the critic model, the server aggregates the coding features and at least one full connection, outputs the reference evaluation value and target evaluation value of the multiple action branches, and the reference evaluation value is used for the multiple action branches and the external information Evaluation is performed, and the target evaluation value is used to evaluate the plurality of action branches and the external information of the target environment collected by the robot in the next time step. The server trains the critic model based on the difference information between the target evaluation value and the reference evaluation value.

For example, the server optimizes the critic model V _γ,ψ by minimizing the Bellman residual, that is, trains the critic model through the following formula (5).

分别表示当前状态值网络的参数和目标状态值网络的参数。V_γ，ψ为模型参数，γ表示动作分支，o′表示该机器人在下一个时间步采集到的该目标环境的外部信息。Among them, J _c (ψ) is the function value of the optimization function, ψ and

Respectively represent the parameters of the current state value network and the parameters of the target state value network. V _{γ, ψ} are model parameters, γ represents the action branch, and o' represents the external information of the target environment collected by the robot in the next time step.

In the second part, the server inputs the external information and the plurality of action branches into the critic model, and the critic model outputs action advantage values of the plurality of action branches.

也即是动作优势值

但是我们不是使用混合奖励

计算动作优势值，而是使用集成奖励

即

Among them, see the above formula (5), in the square term

action advantage

But instead of using mixed rewards

Compute action advantage, use ensemble rewards instead

Right now

In some embodiments, the server trains the reward value estimation model of the intelligent decision-making model based on the external information.

306. The server trains the intelligent decision-making model based on the action advantage value and the integrated reward.

In a possible implementation manner, the server forms a loss function of the distributed executor model based on the action advantage value and the integrated reward. Based on the loss function, the server uses the gradient descent method to train the distributed executor model.

In some embodiments, the loss function is a loss function similar to Proximal Policy Optimization (PPO).

For example, the server uses the following formula (6) to train the distributed actor model, and the following formula (6) omits the superscript of the agent index.

表示重要性权重，∈是裁剪超参数，在实际使用的过程中，一般选择

表示智能体i的策略熵，控制着熵在最终损失J_a(θ)中的重要性程度，动作优势值

与集成奖励

相关。in,

Indicates the importance weight, ∈ is the clipping hyperparameter, in the process of actual use, generally choose

Indicates the strategy entropy of agent i, which controls the importance of entropy in the final loss J _a (θ), and the action advantage value

with integrated rewards

relevant.

It should be noted that the above-mentioned steps 301-306 are described as an iterative training of the distributed executor model. In other iterative training, the training method is the same as the above-mentioned steps 301-306, and will not be repeated here. repeat.

In addition, the above-mentioned steps 301-306 are described by taking the server as the execution subject to train the intelligent decision-making model as an example. In other possible implementation manners, the above-mentioned steps 301-306 can also be executed by the terminal. Not limited.

After the training of the intelligent decision-making model is completed, the distributed executor model can be deployed to multiple robots, and multiple robots can be placed in a specified environment to realize the navigation of multiple robots. The robot is given a task to move from position A to position B in a specified environment, which includes various terrains and obstacles. The multiple robots can observe the specified environment, obtain external information of the specified environment, and input the external information into the trained intelligent decision-making model to obtain corresponding actions. For example, the actions include going straight, turning, raising legs, etc. By performing these actions, multiple robots are able to accomplish the task of moving from location A to location B.

All the above optional technical solutions may be combined in any way to form optional embodiments of the present application, which will not be repeated here.

During the experiment, a multi-agent environment containing three collaborations was selected for testing. The three scenarios are CN, REF, and TREA. The first two scenes are scenes in the particle system, and the third scene is a more difficult version of the Cooperative Treasure Collection developed based on the particle system. Compared methods include MADDPG, MAAC, QMIX, IQL, and MAPPO, among which MAPPO is the latest algorithm that uses a centralized training decentralized execution framework to extend PPO to multi-agent scenarios. Fig. 5 shows the experimental results. In Fig. 5, the method provided by the embodiment of the present application achieves the best results in three scenarios, indicating that the method provided by the embodiment of the present application can indeed capture the different rewards in the environment. It can be seen from the performance curve that the method provided by the embodiment of the present application successfully realizes a stable training process in the learning process.

Through the technical solution provided by the embodiment of this application, the external information collected by the robot in the target environment is obtained, the external information is input into the intelligent decision-making model, and the distributed executor model of the intelligent decision-making model outputs multiple action branches. The action branches are actions that the robot may perform in the target environment when the external information is obtained. Based on the external information and the multiple action branches, the distribution of the reward value of each action branch is determined, that is, the multiple action branches are all evaluated. Based on the distribution of reward values of multiple action branches, reward aggregation is performed to determine mixed rewards and integrated rewards. Training the intelligent decision-making model based on the mixed reward, the integrated reward and external information can achieve a relatively stable training effect.

The embodiment of the present application provides a multi-agent distributed reward estimation and policy weighted aggregation framework (ie, DRE-MARL in Figure 5), which is used to capture and reduce the problem of unstable learning performance caused by uncertain reward signals, and realize It improves the effectiveness and robustness of model training. In this framework, the possible reward signals on all action branches in a certain state are considered as a whole, and a more stable reward signal is provided for the critic (Critic) network to update the model parameters. First, a distributed multi-action branch distribution estimation is proposed, and a separate reward distribution function is constructed for each action branch. Secondly, the reward signal is sampled from these different action distributions to form a sampled reward vector, and then the reward on one of the action branches is replaced with the reward obtained from the environment to form an embedded reward vector. A reward aggregation operation to obtain a mixed reward signal as the reward signal of the critic network and the distributed executor (DecentralizedActor) network. Experimental results show that the method provided in the embodiment of the present application shows excellent performance, which proves the effectiveness of the method provided in the embodiment of the present application.

Fig. 6 is a schematic structural diagram of a training device for an intelligent decision-making model provided by an embodiment of the present application. Referring to Fig. 6, the device includes: an external information acquisition module 601, an action prediction module 602, a reward value prediction module 603, and a reward value aggregation module 604 and a training module 605 .

The external information acquisition module 601 is used to acquire the external information collected by the robot in the target environment, the external information includes external environment information and interaction information, the external environment information is the information obtained by the robot observing the target environment, and the interaction information is The robot interacts with other robots in the target environment.

The action prediction module 602 is used to input the external information into the intelligent decision-making model of the robot, and the distributed executor model of the intelligent decision-making model performs prediction based on the external information, and outputs multiple action branches of the robot, and the action branches are Actions that the robot may perform in the target environment.

The reward value prediction module 603 is configured to determine the reward value distribution of each of the action branches based on the external information and the plurality of action branches.

The reward value aggregation module 604 is configured to perform reward aggregation on the sampled reward values sampled in the reward value distribution of each action branch to obtain mixed rewards and integrated rewards.

A training module 605, configured to train the robot's intelligent decision-making model based on the mixed reward, the integrated reward and the external information.

In a possible implementation, the action prediction module 602 is configured to perform at least one of full connection, convolution, and attention encoding on the external information by the distributed actor model of the intelligent decision-making model to obtain the External information characteristics of external information. The distributed executor model of the intelligent decision-making model performs full connection and normalization on the external information features, and outputs multiple action branches of the robot.

In a possible implementation manner, the reward value prediction module 603 is configured to input the external information and the multiple action branches into the reward value estimation model, and the reward value estimation model is based on the external information and the multiple action branches Estimate the reward value distribution, and output the reward value distribution of each action branch.

In a possible implementation manner, the reward value aggregation module 604 is configured to sample the reward value distribution of each action branch to obtain the sampled reward value of each action branch. The strategy weighted fusion is performed on the sampling reward values of each action branch to obtain the mixed reward. Perform any one of global average, local average, and direct selection on the sampling reward values of each action branch to obtain the integrated reward.

In a possible implementation manner, the training module 605 is configured to obtain action advantage values of the plurality of action branches based on the mixed reward, the external information and the critic model of the intelligent decision model, and the critic model uses The multiple action branches are evaluated based on the external information. A reward value estimation model of the intelligent decision-making model is trained based on the external information. Based on the action advantage value and the integrated reward, the distributed executor model of the intelligent decision-making model is trained.

In a possible implementation manner, the training module 605 trains the critic model of the intelligent decision-making model based on the mixed reward and the external information. The external information and the plurality of action branches are input into the critic model, and the critic model outputs action advantage values of the plurality of action branches.

In a possible implementation manner, the training module 605 is configured to form a loss function of the distributed actor model based on the action advantage value and the integrated reward. Based on the loss function, the distributed executor model is trained using the gradient descent method.

It should be noted that: when the training device for the intelligent decision-making model provided by the above-mentioned embodiment trains the intelligent decision-making model, the division of the above-mentioned functional modules is used as an example for illustration. In practical applications, the above-mentioned functions can be assigned by different The functional modules are completed, that is, the internal structure of the computer device is divided into different functional modules to complete all or part of the functions described above. In addition, the intelligent decision-making model training device and the intelligent decision-making model training method embodiment provided by the above-mentioned embodiments belong to the same concept, and the specific implementation process thereof can be found in the method embodiment, and will not be repeated here.

Through the technical solution provided by the embodiment of this application, the external information collected by the robot in the target environment is obtained, the external information is input into the intelligent decision-making model, and the distributed executor model of the intelligent decision-making model outputs multiple action branches. The action branches are actions that the robot may perform in the target environment when the external information is acquired. Based on the external information and the multiple action branches, the distribution of the reward value of each action branch is determined, that is, the multiple action branches are all evaluated. Based on the distribution of reward values of multiple action branches, reward aggregation is performed to determine mixed rewards and integrated rewards. Training the intelligent decision-making model based on the mixed reward, the integrated reward and external information can achieve a relatively stable training effect.

An embodiment of the present application provides a computer device for performing the above method. The computer device can be implemented as a terminal or a server. The structure of the terminal is firstly introduced below:

FIG. 7 is a schematic structural diagram of a terminal provided by an embodiment of the present application. The terminal 700 may be: a smart phone, a tablet computer, a notebook computer or a desktop computer. The terminal 700 may also be called user equipment, portable terminal, laptop terminal, desktop terminal and other names.

Generally, the terminal 700 includes: one or more processors 701 and one or more memories 702 .

The processor 701 may include one or more processing cores, such as a 4-core processor, an 8-core processor, and the like. The processor 701 can adopt at least one hardware form in DSP (Digital Signal Processing, digital signal processing), FPGA (Field-Programmable Gate Array, field programmable gate array), PLA (Programmable Logic Array, programmable logic array) accomplish. The processor 701 may also include a main processor and a coprocessor, the main processor is a processor for processing data in the wake-up state, and is also called a CPU (Central Processing Unit, central processing unit); the coprocessor is used to Low-power processor for processing data in standby state. In some embodiments, the processor 701 may be integrated with a GPU (Graphics Processing Unit, image processor), and the GPU is used for rendering and drawing content that needs to be displayed on the display screen. In some embodiments, the processor 701 may further include an AI (Artificial Intelligence, artificial intelligence) processor, where the AI processor is configured to process computing operations related to machine learning.

Memory 702 may include one or more computer-readable storage media, which may be non-transitory. The memory 702 may also include high-speed random access memory, and non-volatile memory, such as one or more magnetic disk storage devices and flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 702 is used to store at least one computer program, and the at least one computer program is used to be executed by the processor 701 to implement the methods provided by the method embodiments in this application. Training methods for intelligent decision-making models.

In some embodiments, the terminal 700 may optionally further include: a peripheral device interface 703 and at least one peripheral device. The processor 701, the memory 702, and the peripheral device interface 703 may be connected through buses or signal lines. Each peripheral device can be connected to the peripheral device interface 703 through a bus, a signal line or a circuit board. Specifically, the peripheral device includes: at least one of a radio frequency circuit 704 , a display screen 705 , a camera assembly 706 , an audio circuit 707 and a power supply 708 .

The peripheral device interface 703 may be used to connect at least one peripheral device related to I/O (Input/Output, input/output) to the processor 701 and the memory 702 . In some embodiments, the processor 701, memory 702 and peripheral device interface 703 are integrated on the same chip or circuit board; in some other embodiments, any one of the processor 701, memory 702 and peripheral device interface 703 or The two can be implemented on a separate chip or circuit board, which is not limited in this embodiment.

The radio frequency circuit 704 is configured to receive and transmit RF (Radio Frequency, radio frequency) signals, also called electromagnetic signals. The radio frequency circuit 704 communicates with the communication network and other communication devices through electromagnetic signals. The radio frequency circuit 704 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals into electrical signals. Optionally, the radio frequency circuit 704 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a subscriber identity module card, and the like.

The display screen 705 is used to display a UI (User Interface, user interface). The UI can include graphics, text, icons, video, and any combination thereof. When the display screen 705 is a touch display screen, the display screen 705 also has the ability to collect touch signals on or above the surface of the display screen 705 . The touch signal can be input to the processor 701 as a control signal for processing. At this time, the display screen 705 can also be used to provide virtual buttons and/or virtual keyboards, also called soft buttons and/or soft keyboards.

The camera assembly 706 is used to capture images or videos. Optionally, the camera component 706 includes a front camera and a rear camera. Usually, the front camera is set on the front panel of the terminal, and the rear camera is set on the back of the terminal.

Audio circuitry 707 may include a microphone and speakers. The microphone is used to collect sound waves of the user and the environment, and convert the sound waves into electrical signals and input them to the processor 701 for processing, or input them to the radio frequency circuit 704 to realize voice communication.

The power supply 708 is used to supply power to various components in the terminal 700 . Power source 708 may be alternating current, direct current, disposable batteries, or rechargeable batteries.

In some embodiments, the terminal 700 also includes one or more sensors 709 . The one or more sensors 709 include, but are not limited to: an acceleration sensor 710 , a gyro sensor 711 , a pressure sensor 712 , an optical sensor 713 and a proximity sensor 714 .

The acceleration sensor 710 can detect the acceleration on the three coordinate axes of the coordinate system established by the terminal 700 .

The gyro sensor 711 can measure the body direction and rotation angle of the terminal 700 , and the gyro sensor 711 can cooperate with the acceleration sensor 710 to collect 3D actions of the user on the terminal 700 .

The pressure sensor 712 may be disposed on a side frame of the terminal 700 and/or a lower layer of the display screen 705 . When the pressure sensor 712 is installed on the side frame of the terminal 700 , it can detect the user's grip signal on the terminal 700 , and the processor 701 performs left and right hand recognition or shortcut operation according to the grip signal collected by the pressure sensor 712 . When the pressure sensor 712 is disposed on the lower layer of the display screen 705, the processor 701 controls operable controls on the UI interface according to the user's pressure operation on the display screen 705.

The optical sensor 713 is used to collect ambient light intensity. In one embodiment, the processor 701 may control the display brightness of the display screen 705 according to the ambient light intensity collected by the optical sensor 713 .

The proximity sensor 714 is used to collect the distance between the user and the front of the terminal 700 .

Those skilled in the art can understand that the structure shown in FIG. 7 does not constitute a limitation on the terminal 700, and may include more or less components than shown in the figure, or combine certain components, or adopt different component arrangements.

The above-mentioned computer equipment can also be realized as a server, and the structure of the server is introduced below:

FIG. 8 is a schematic structural diagram of a server provided by an embodiment of the present application. The server 800 may have relatively large differences due to different configurations or performances, and may include one or more processors (Central Processing Units, CPU) 801 and a or more memories 802, wherein at least one computer program is stored in the one or more memories 802, and the at least one computer program is loaded and executed by the one or more processors 801 to implement the methods provided by the above method embodiments method. Of course, the server 800 may also have components such as wired or wireless network interfaces, keyboards, and input and output interfaces for input and output, and the server 800 may also include other components for implementing device functions, which will not be repeated here.

In an exemplary embodiment, there is also provided a computer-readable storage medium, such as a memory including a computer program, and the computer program can be executed by a processor to complete the intelligent decision-making model training method in the above-mentioned embodiment. For example, the computer-readable storage medium may be a read-only memory (Read-Only Memory, ROM), a random access memory (Random Access Memory, RAM), a read-only optical disc (Compact Disc Read-Only Memory, CD-ROM), Magnetic tapes, floppy disks, and optical data storage devices, etc.

In an exemplary embodiment, a computer program product or computer program is also provided, the computer program product or computer program includes program code, the program code is stored in a computer-readable storage medium, and the processor of the computer device reads from the computer The program code is read by reading the storage medium, and the processor executes the program code, so that the computer device executes the above-mentioned training method for the intelligent decision-making model.

In some embodiments, the computer programs involved in the embodiments of the present application can be deployed and executed on one computer device, or executed on multiple computer devices at one location, or distributed in multiple locations and communicated Executed on multiple computer devices interconnected by the network, multiple computer devices distributed in multiple locations and interconnected through a communication network can form a blockchain system.

Those of ordinary skill in the art can understand that all or part of the steps for implementing the above-mentioned embodiments can be completed by hardware, and can also be completed by instructing related hardware through a program. The program can be stored in a computer-readable storage medium. The above-mentioned The storage medium can be read-only memory, magnetic disk or optical disk and so on.

The above are only optional embodiments of the application, and are not intended to limit the application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the application shall be included in the protection scope of the application. within.

Claims (10)

1. A method for training an intelligent decision model, the method comprising:

acquiring external information acquired by a robot in a target environment, wherein the external information comprises external environment information and interaction information, the external environment information is information obtained by observing the target environment by the robot, and the interaction information is information obtained by interacting the robot with other robots in the target environment;

inputting the external information into an intelligent decision model of the robot, performing prediction by a distributed executor model of the intelligent decision model based on the external information, and outputting a plurality of action branches of the robot, wherein the action branches are actions which can be executed by the robot in the target environment;

determining a reward value distribution of each action branch based on the external information and the plurality of action branches;

carrying out reward aggregation on the sampled reward values obtained by sampling in the reward value distribution of each action branch to obtain mixed reward and integrated reward;

training an intelligent decision model of the robot based on the hybrid reward, the integrated reward, and the external information.

2. The method of claim 1, wherein the predicting by the distributed actor model of the intelligent decision model based on the external information, outputting a plurality of action branches for the robot comprises:

performing at least one of full connection, convolution and attention coding on the external information by a distributed executor model of the intelligent decision model to obtain external information characteristics of the external information;

and fully connecting and normalizing the external information features by a distributed executor model of the intelligent decision model, and outputting a plurality of action branches of the robot.

3. The method of claim 1, wherein determining a reward value distribution for each of the action branches based on the external information and the plurality of action branches comprises:

inputting the external information and the action branches into a reward value estimation model, performing reward value distribution estimation through the reward value estimation model based on the external information and the action branches, and outputting reward value distribution of each action branch.

4. The method of claim 1, wherein the aggregating rewards of the sampled reward values sampled in the reward value distribution of each of the action branches, and wherein the obtaining of the hybrid reward and the integrated reward comprises:

sampling the reward value distribution of each action branch to obtain the sampling reward value of each action branch;

carrying out strategy weighted fusion on the sampling reward values of all the action branches to obtain the mixed reward;

and carrying out any one of global averaging, local averaging and direct selection on the sampled reward values of the action branches to obtain the integrated reward.

5. The method of claim 1, wherein training a smart decision model of the robot based on the hybrid reward, the integrated reward, and the external information comprises:

training a reward value estimation model of the intelligent decision model based on the external information;

obtaining action advantage values of the action branches based on the hybrid reward, the external information and a critic model of the intelligent decision model, wherein the critic model is used for evaluating the action branches based on the external information;

training a distributed actor model of the intelligent decision model based on the action advantage value and the integrated reward.

6. The method of claim 5, wherein obtaining the action advantage values for the plurality of action branches based on the hybrid reward, the external information, and a critic model of the intelligent decision model comprises:

training a critic model of the intelligent decision model based on the hybrid reward and the external information;

inputting the external information and the plurality of action branches into the critic model, and outputting action dominance values of the plurality of action branches by the critic model.

7. The method of claim 5, wherein training the intelligent decision model based on the action advantage value and the integrated reward comprises:

constructing a loss function for the distributed actor model based on the action dominance value and the integrated reward;

and training the distributed executor model by adopting a gradient descent method based on the loss function.

8. An apparatus for training an intelligent decision model, the apparatus comprising:

the external information acquisition module is used for acquiring external information acquired by the robot in a target environment, wherein the external information comprises external environment information and interaction information, the external environment information is information obtained by observing the target environment by the robot, and the interaction information is information obtained by interacting the robot with other robots in the target environment;

the action prediction module is used for inputting the external information into an intelligent decision-making model of the robot, performing prediction by a distributed executor model of the intelligent decision-making model based on the external information, and outputting a plurality of action branches of the robot, wherein the action branches are actions which are possibly executed by the robot in the target environment;

the reward value prediction module is used for determining reward value distribution of each action branch based on the external information and the action branches;

the reward value aggregation module is used for carrying out reward aggregation on the sampled reward values sampled in the reward value distribution of each action branch to obtain mixed rewards and integrated rewards;

and the training module is used for training the intelligent decision model of the robot based on the mixed reward, the integrated reward and the external information.

9. A computer device, characterized in that the computer device comprises one or more processors and one or more memories, in which at least one computer program is stored, which is loaded and executed by the one or more processors to implement the method of training of an intelligent decision model according to any one of claims 1 to 7.

10. A computer-readable storage medium, in which at least one computer program is stored, which is loaded and executed by a processor to implement a method of training an intelligent decision model as claimed in any one of claims 1 to 7.

Images