CN109934753B - Multi-Agent emergency action decision method based on JADE platform and reinforcement learning - Google Patents

Abstract

The invention provides a multi-Agent emergency action decision method based on a JADE platform and reinforcement learning, which comprises the following steps: starting a JADE platform and establishing a monitoring Agent, and judging whether an emergency public event occurs or not in real time by using the monitoring Agent; registering emergency resource guarantee service behaviors of each emergency resource warehouse Agent on the monitoring Agent, executing reinforcement learning of each emergency resource warehouse Agent, and obtaining reinforcement learning feedback values corresponding to each emergency resource warehouse Agent from the monitoring Agent; one or more emergency resource warehouse agents are selected from the reinforcement learning feedback values and added into the emergency resource allocation sequence. The multi-Agent emergency action decision method combines a multi-Agent technology with a reinforcement learning algorithm, and the supply of an emergency resource warehouse is allocated from the global start of the whole emergency action system, and the reinforcement learning algorithm fully utilizes the autonomy of agents to promote the intelligent level and the self-adaption capability of the multi-Agent system.

Description

Multi-agent emergency action decision-making method based on JADE platform and reinforcement learning

technical field

The invention belongs to the technical field of artificial intelligence, in particular to a multi-Agent emergency action strategy method based on a JADE platform and reinforcement learning.

Background technique

With the rapid development of my country's economy and society, various public emergencies emerge in endlessly. According to official data, in 2018, natural disasters alone have caused 130 million people to be affected, with direct economic losses exceeding 260 billion yuan. Effective emergency actions can not only prevent and reduce the occurrence of public emergencies, but also ensure the safety of people's personal and property when public emergencies occur, control the situation as soon as possible and minimize losses. Therefore, how to use artificial intelligence technologies such as multi-agent and reinforcement learning to systematically and effectively monitor, manage and assist decision-making during the entire emergency operation process is a work that should be further developed and is of great significance.

Agent is a kind of computing entity or program that can perceive the environment in a specific environment and can operate autonomously to achieve a series of goals on behalf of its designer or user. Multi-Agent system, that is, MAS (Multi-Agent System), its essence is the thinking of "divide and conquer". The characteristics of multi-agent system determine its unique advantages in many distributed application fields. The e-commerce field, transportation field, emergency rescue field, debate system, and telecommunication system all have the characteristics of distributed interaction. The use of multi-Agent system can significantly improve the interaction mode of different entities, optimize the execution plan, and provide better, faster, More reliable service. In addition, in the construction of some information decision support systems, the multi-agent system is also an extremely effective solution. As a multi-Agent system simulation implementation platform based on FIPA specification, JADE has complete functions, a sound system, and strong portability, which greatly simplifies the development of multi-Agent systems. Reinforcement learning, as a typical unsupervised learning method, has been widely used in many fields such as unmanned driving, intelligent control, and auxiliary decision-making. Using the autonomy of Agents in multi-Agent systems to implement reinforcement learning algorithms is conducive to improving the performance of multi-Agent systems. overall intelligence.

Contents of the invention

The purpose of the present invention is to provide a multi-Agent emergency action decision-making method based on the JADE platform and reinforcement learning, by comprehensively considering the transportation cost, distance, time, effectiveness, etc. to determine how to use each emergency resource warehouse to provide emergency resources collaboratively, Provide emergency resource protection in a timely and effective manner at a relatively low economic cost.

In order to achieve the above-mentioned purpose of the invention, the present invention provides a multi-Agent emergency action decision-making method based on the JADE platform and reinforcement learning, comprising the following steps:

Step 1. Start the JADE platform and establish a monitoring agent. Use the monitoring agent to judge whether there is a public emergency in real time. If there is a public emergency, go directly to step 2. If there is no public emergency, repeat this step. continue to judge;

Step 2, register the emergency resource guarantee service behavior of each emergency resource warehouse Agent on the monitoring agent, and execute the reinforcement learning of each emergency resource warehouse Agent, and obtain the reinforcement learning feedback value corresponding to each emergency resource warehouse Agent from the monitoring agent;

Step 3: Select one or more emergency resource warehouse Agents from each reinforcement learning feedback value to join in the emergency resource allocation sequence.

Further, in step 2, the monitoring agent searches for all possible emergency resource warehouse agents in real time through the yellow page service of JADE.

Further, in step 2, the specific steps of reinforcement learning of the emergency resource warehouse Agent are:

Step a, initialize learning rate Λ _t , discount factor γ and Q value;

Step b, each emergency resource warehouse agent obtains the current state s _t through the interaction between the initiation class of the JADE interaction protocol and the response class of the environment, and selects the optimal action a _t under the current state s _t according to the state transition function P, and executes the action a _t transfers to a new state s _t+1 ;

Step c, the emergency resource warehouse Agent uses the initiation class of the JADE interaction protocol to obtain the return value r _t+1 from the external environment, and update the Q value;

Step d, quit reinforcement learning after the Q value converges.

Further, in step b, the JADE content language is used to store the interaction information of the initiator class of the JADE interaction protocol and the response class of the environment in the form of ontology.

Further, in step b, the state transition function P selects an action strategy based on the softmax function, so that the action strategy with a larger average feedback value has a higher probability of being adopted.

Further, in step b, the probability normalization formula of the state transition function P is:

表示归一化前相应动作选择造成状态转移的概率，

表示归一化前动作集合中全部动作造成状态转移的概率。In the formula, τ represents the annealing temperature, which is used to control the search rate. When τ is smaller, the difference in average reward is greater, and the optimal strategy may be selected larger.

Indicates the probability of state transition caused by the corresponding action selection before normalization,

Indicates the probability of state transition caused by all actions in the action set before normalization.

Further, in step c, the formula for calculating the Q value is:

In the formula, γ∈[0,1) is the discount factor, Λ _t is the learning rate, A is the action set, S is the state set, Q _{t (} s _t , a _{t )} represents the q value, Q _t+1 (s _t , a _t ) represents the update value at time t+1, and max _a∈A Q(s ^′ , a ^′ ) represents the maximum value in these Q value tables.

Further, the action set A={a1, a2}, the state set S={C1, C2, D, F1, F2}, C1 represents the inventory capacity that the emergency resource warehouse Agent can effectively provide, and C2 represents that the emergency resource warehouse Agent can effectively The types of emergency supplies provided, D represents the distance between the emergency resource warehouse Agent and the place where public emergencies occur, F1 represents the transportation cost of emergency resources per unit distance, F2 represents the transportation cost of emergency resources per unit quality, a1 represents the emergency resource warehouse Agent chooses Join the ranks of emergency resource allocation, a2 means that the emergency resource warehouse Agent chooses not to join.

The beneficial effect of the present invention is: (1) multi-Agent technology and reinforcement learning algorithm are combined, set out from the overall situation of whole emergency action system to deploy the supply of emergency resource warehouse, reinforcement learning algorithm has fully utilized the autonomy of Agent, to promote The intelligent level and self-adaptive ability of the multi-Agent system; (2) It has strong scalability and applicability, and can be combined with the digital emergency plan system, using the existing monitoring data information and case database for computer-aided decision-making, Command emergency operations more scientifically and effectively; (3) Use the JADE platform to build Agents and realize multi-Agent system development. The multi-Agent system based on the JADE platform can use the communication interaction protocol, yellow pages service, ontology support, and Agent migration provided by JADE. The simulation of the emergency rescue and handling process and action details is combined with the auxiliary decision-making application of the actual emergency response to public emergencies to build a system application framework system that optimizes drills in peacetime and provides auxiliary decision-making in wartime.

Description of drawings

Fig. 1 is the overall flowchart of the emergency action decision-making of the finite state machine model of the present invention;

Fig. 2 is a structural diagram of the reinforcement learning of the JADE platform of the present invention.

Detailed ways

As shown in Figure 1, the present invention provides a kind of multi-Agent emergency action decision-making method based on JADE platform and reinforcement learning, comprises the following steps;

Step 1, start the JADE platform and establish a monitoring agent, use the finite state machine model (FSM) to schedule sub-behaviors to manage emergency actions for public emergencies, the finite state machine starts from the initial state 1, and execute behavior 1: use the monitoring agent to judge in real time Whether there is a sudden public event, if there is a public emergency, go directly to step 2 and enter the intermediate state 3, if there is no public emergency, enter the intermediate state 2 (warning behavior) and then migrate to the initial state 1 Continue to judge by looping this step;

Step 2, execute behavior 3, register the emergency resource guarantee service behavior of each emergency resource warehouse Agent on the monitoring agent, and then enter the intermediate state 4, execute behavior 4: reinforcement learning of each emergency resource warehouse Agent, and obtain each emergency resource guarantee service behavior from the monitoring agent The reinforcement learning feedback value corresponding to the resource warehouse Agent; the task of reinforcement learning corresponds to a quaternion: E=<S,A,P,R>, where S is the current state, A is the action set, and P is the state transition function , R is the feedback function, and the state transition P is based on the softmax function to select the action strategy, which ensures that the action strategy with a relatively large average feedback value is more likely to be adopted, and at the same time ensures that the action strategy with a low average feedback value is still adopted. Opportunity; the emergency resource warehouse Agent obtains feedback values based on effectiveness, economic benefits, time distance, etc. from the environment (monitoring agent). According to the basic principle of reinforcement learning, if a certain behavior strategy of the emergency resource warehouse Agent changes the environment, the If the reward value is positive, then the agent’s tendency to produce this behavior strategy will be strengthened; otherwise, it will be weakened. In the multi-agent system, the reinforcement learning goal is still to reward the largest reward value. The long-term cumulative reward value is calculated by the γ-discount cumulative reward value method. That is, each time the discount factor decreases at a rate of γ;

Step 3, execute behavior 5, select one or more emergency resource warehouse agents from each reinforcement learning feedback value to join the emergency resource allocation sequence, at this time, the finite state machine behavior of the monitoring agent is terminated, and the emergency action is over.

Further, in step 2, the monitoring agent searches for all possible emergency resource warehouse agents in real time through the yellow page service of the JADE platform.

Further, in step 2, the specific steps of reinforcement learning of the emergency resource warehouse Agent are:

Step a, initialize learning rate Λ _t , discount factor γ and Q value, discount factor γ∈[0,1), γ being 0 means only looking at r in front of you;

Step b, each emergency resource warehouse agent obtains the current state s _t through the interaction between the initiation class of the JADE interaction protocol and the response class of the environment, and selects the optimal action a _t under the current state s _t according to the state transition function P, and executes the action a _t transfers to a new state s _t+1 ;

Step c, the emergency resource warehouse Agent uses the initiation class of the JADE interaction protocol to obtain the return value r _t+1 from the external environment, and update the Q value;

Step d, exit reinforcement learning after the Q value converges, enter termination state 5, and then execute behavior 5.

Further, in step b, the JADE content language is used to store the interaction information of the initiator class of the JADE interaction protocol and the response class of the environment in the form of ontology.

Further, in step b, the state transition function P selects an action strategy based on the softmax function, so that the action strategy with a larger average feedback value has a higher probability of being adopted.

Further, in step b, the probability normalization formula of the state transition function P is:

表示归一化前相应动作选择造成状态转移的概率，

表示归一化前动作集合中全部动作造成状态转移的概率。In the formula, τ represents the annealing temperature, which is used to control the search rate. When τ is smaller, the difference in average reward is greater, and the optimal strategy may be selected larger.

Indicates the probability of state transition caused by the corresponding action selection before normalization,

Indicates the probability of state transition caused by all actions in the action set before normalization.

Further, in step c, the formula for calculating the Q value is:

γ∈[0,1) is the discount factor, Λ _t is the learning rate, r _t+1 represents the feedback value at time t+1, A is the action set, S is the state set, Q(s,a) represents s and a The determined q value, Q _t (st _t , a _t ) represents the q value determined by st _t and a _t at time t, Q _t+1 (st _t , a _t ) represents the updated value at time t+1, max _{a ∈A} Q(s ^′ , a ^′ ) represents the maximum value in these Q value tables. The learning rate α _t controls the learning speed, and its value is proportional to the convergence speed, but it cannot be too large, otherwise the convergence will be immature. γ∈[0,1) is used as a discount factor. The larger it is, the more it is inclined to the long-term reward value, and the smaller it is, the more it is inclined to the short-term current reward value. The calculation formula of the Q value is to use the updated iterative calculation to continuously approach the optimal solution after being processed by the Bellman equation. Therefore, the reinforcement learning behavior is set to CyclicBehaviour (cyclic class), and the learning process can be repeated continuously.

Further, the action set A={a1, a2}, the state set S={C1, C2, D, F1, F2}, C1 represents the inventory capacity that the emergency resource warehouse Agent can effectively provide, and C2 represents that the emergency resource warehouse Agent can effectively The types of emergency supplies provided, D represents the distance between the emergency resource warehouse Agent and the place where public emergencies occur, F1 represents the transportation cost of emergency resources per unit distance, F2 represents the transportation cost of emergency resources per unit quality, a1 represents the emergency resource warehouse Agent chooses Join the ranks of emergency resource allocation, a2 means that the emergency resource warehouse Agent chooses not to join.

As shown in FIG. 2 , it is a structural diagram of a modeled reinforcement learning algorithm combined with the multi-agent emergency action decision-making method based on the JADE platform and reinforcement learning of the present invention. Manage emergency actions for public emergencies by monitoring Agents. The specific interaction methods between the emergency resource warehouse Agent and the environment (monitoring Agent) are as follows: the emergency resource warehouse Agent establishes the initiation class of the JADE interaction protocol, the monitoring agent establishes the JADE interaction protocol sending and response class, and uses the JADE ontology class to define the feedback value of the Concept structure r _t+1 , the state s _t of the Predicate structure, the action a _t of the Action structure, and communicate and exchange these information.

Based on multi-agent technology, JADE platform and reinforcement learning method, the present invention combines multi-agent technology and reinforcement learning algorithm to establish a more intelligent decision-making support method from the perspective of the overall situation of emergency response actions, and comprehensively considers time, cost, effective The global allocation of each emergency resource warehouse can be fully and effectively used for emergency support work, and the multi-agent idea is applied to the decision support system, which greatly enhances the self-adaptive ability of the system; and then uses reinforcement learning Improve the coordination among various Agents and promote the intelligence of the system. It can be combined with the digital emergency plan system, and use the existing monitoring data information and case database for computer-aided decision-making, and command emergency actions more scientifically and effectively. The system can effectively Coordinating the relationship between economic cost and time efficiency, it is more intelligent and adaptable, has higher scalability and more important practical application value.

Claims (5)

1. A multi-Agent emergency action decision-making method based on JADE platform and reinforcement learning, is characterized in that, comprises the steps: Step 1. Start the JADE platform and establish a monitoring agent. Use the monitoring agent to judge whether there is a public emergency in real time. If there is a public emergency, go directly to step 2. If there is no public emergency, repeat this step. continue to judge; Step 2, register the emergency resource guarantee service behavior of each emergency resource warehouse Agent on the monitoring agent, and execute the reinforcement learning of each emergency resource warehouse Agent, and obtain the reinforcement learning feedback value corresponding to each emergency resource warehouse Agent from the monitoring agent; Step 3, select one or more emergency resource warehouse Agents from each reinforcement learning feedback value to join the emergency resource deployment sequence; In step 2, the specific steps of the reinforcement learning of the emergency resource warehouse Agent are as follows: Step a, initialize learning rate Λ _t , discount factor γ and Q value; Step b, each emergency resource warehouse agent obtains the current state s _t through the interaction between the initiation class of the JADE interaction protocol and the response class of the environment, and selects the optimal action a _t under the current state s _t according to the state transition function P, and executes the action a _t transfers to a new state s _t+1 ; Step c, the emergency resource warehouse Agent uses the initiation class of the JADE interaction protocol to obtain the return value r _t+1 from the external environment, and update the Q value; Step d, quit reinforcement learning after the Q value converges. 2. The multi-Agent emergency action decision-making method based on the JADE platform and reinforcement learning according to claim 1, wherein in step 2, the monitoring Agent searches for all possible emergency resource warehouse Agents in real time through the Yellow Pages service of the JADE platform. 3. The multi-Agent emergency action decision-making method based on JADE platform and reinforcement learning according to claim 1, characterized in that, in step b, utilize the content language of JADE to store the response of the initiation class and environment of JADE interactive protocol in ontology form Class interaction information. 4. the multi-Agent emergency action decision-making method based on JADE platform and reinforcement learning according to claim 3, is characterized in that, in step c, the computing formula of Q value is:

In the formula, γ∈[0,1) is the discount factor, Λ _t is the learning rate, A is the action set, S is the state set, Q _{t (} s _t , a _{t )} represents the q value, Q _t+1 (s _t , a _t ) represents the update value at time t+1, and max _a∈A Q(s ^′ , a ^′ ) represents the maximum value in these Q value tables. 5. the multi-Agent emergency action decision-making method based on JADE platform and reinforcement learning according to claim 4, is characterized in that, action set A={a1, a2}, state set S={C1, C2, D, F1, F2}, C1 represents the inventory capacity that the emergency resource warehouse Agent can effectively provide, C2 represents the type of emergency supplies that the emergency resource warehouse Agent can effectively provide, D represents the distance between the emergency resource warehouse Agent and the place where public emergencies occur, and F1 represents the unit distance The transportation cost of emergency resources, F2 means the transportation cost of emergency resources per unit quality, a1 means that the agent of the emergency resource warehouse chooses to join the ranks of emergency resource allocation, and a2 means that the agent of the emergency resource warehouse chooses not to join.

Images