EP4705852A2 - Methods and systems for autonomous agent control - Google Patents

Abstract

Embodiments operate a plurality of autonomous agents in a real-world environment. In an embodiment, each of the plurality of autonomous agents determines if a communication condition is above or below a threshold. Responsive to determining the communication condition is above the threshold, each agent determines a real-world action in the environment based on a respective indication of properties of the environment and a coordination communication. Responsive to determining the communication condition is below the threshold, each agent determines a real-world action in the environment based on the respective indication of properties of the environment. In turn, each agent performs its determined real-world action in the environment.

Description

METHODS AND SYSTEMS FOR AUTONOMOUS AGENT CONTROL

RELATED APPLICATION

[0001] This application claims the benefit of U.S. Provisional Application No. 63/498,600, filed on April 27, 2023. The entire teachings of the above application are incorporated herein by reference.

GOVERNMENT SUPPORT

[0002] This invention was made with government support under grant number 1937403 awarded by the National Science Foundation. The government has certain rights in the invention.

BACKGROUND

[0003] The use of autonomous agents, such as Unmanned Aerial Vehicles (UAVs), has become prevalent for multiple different objectives, e.g., Search-And-Rescue (SAR) missions.

SUMMARY

[0004] While the use of autonomous agents has become prevalent, the existing solutions to control and coordinate autonomous agents are generally limited to specific environments and are not robust to handle unreliable and unstable communications. Thus, improved functionality is needed to coordinate and control autonomous agents in environments. Embodiments provide such functionality.

[0005] One such embodiment is directed to a computer-implemented method for operating autonomous agents in a real-world environment. The method begins with a control agent receiving respective indications of properties of a real-world environment from one or more agents of a plurality of autonomous agents. Based on the respective indications received and a global control policy, the control agent determines a coordination action for each agent of the plurality of autonomous agents and sends the coordination action to each agent of the plurality of autonomous agents. In turn, each autonomous agent determines a real-world action to perform in the environment. In an embodiment, if the determined coordination action is received, the real- world action is determined based on a respective indication of properties of the environment and the received coordination action and, if the determined coordination action is not received, the real-world action is determined based on the respective indication of properties of the environment and a prior coordination action. After determining the real-world action, each agent performs the determined real-world action in the environment.

[0006] An embodiment may also update the global control policy. In one such embodiment, updating the global control policy includes, by a given agent of the plurality of autonomous agents, determining quality of the determined real-world action. In turn, the control agent updates the global control policy using an indication of the determined quality. In such an embodiment, the given agent may include a critic neural network. Such a critic neural network may be configured to determine the quality of the determined real-world action responsive to one or more inputs. According to an embodiment, the one or more inputs include at least one of: a location of the given agent in the environment and an image of the environment, e g., an image captured using a camera mounted to the given agent.

[0007] In another embodiment, a given agent of the plurality of autonomous agents includes an actor neural network. In such an embodiment, the real-world action is determined using the actor neural network. For instance, if the determined coordination action is received, the real- world action in the environment is determined by providing the respective indication of properties of the environment and the received coordination action as inputs to the actor neural network. Similarly, if the determined coordination action is not received, the real-world action in the environment is determined by providing the respective indication of properties of the environment and the prior coordination action as inputs to the actor neural network. The actor neural network is configured to determine the real-world action responsive to the inputs.

[0008] Embodiments may also be performed iteratively so as to perform/complete an objective in the real -world world environment. For example, such an embodiment may iteratively perform the receiving, the determining a coordination action, the sending, the determining a real -world action, and the performing until an objective is achieved. Amongst other examples, an objective may be a search operation of the environment.

[0009] In embodiments, the plurality of autonomous agents may include any such agents known to those of skill in the art. For instance, in embodiments, the autonomous agents may include unmanned aerial vehicles (UAVs) and/or autonomous underwater vehicles (AUVs). Embodiments can be used to control heterogeneous or homogeneous groupings of vehicles, including terrestrial vehicles, UAVs, and AUVs.

[0010] Another embodiment is directed to a system for operating autonomous agents in a real-world environment. The system includes a plurality of autonomous agents and a control agent. According to an embodiment, the control agent is configured to (i) receive respective indications of properties of a real-world environment from one or more agents of the plurality of autonomous agents, (ii) based on the respective indications received and a global control policy, determine a coordination action for each agent of the plurality of autonomous agents, and (iii) send the determined coordination action to each agent of the plurality of autonomous agents. Further, each agent is configured to determine and perform a real-world action in the environment. According to an embodiment of the system, if the determined coordination action is received, each agent determines the real-world action based on a respective indication of properties of the environment and the received coordination action and, if the determined coordination action is not received, each agent determines the real-world action based on the respective indication of properties of the environment and a prior coordination action. It is noted that embodiments of the system may be configured to implement any embodiments or combination of embodiments described herein.

[0011] Yet another embodiment is directed to a computer program product for operating autonomous agents in a real-world environment. According to an embodiment, the computer program product comprises a computer-readable medium with computer code instructions stored thereon, where the computer code instructions are configured, when executed by a processor, to cause an apparatus associated with the processor to implement any embodiment or combination of embodiments described herein.

[0012] Another embodiment is directed to a computer-implemented method for operating a plurality of autonomous agents in a real-world environment. In this embodiment, each agent of the plurality of autonomous agents determines if a communication condition is above or below a threshold. Responsive to determining the communication condition is above the threshold, each agent determines a real-world action in the environment based on a respective indication of properties of the environment and a coordination communication. Responsive to determining the communication condition is below the threshold, each agent determines a real-world action in the environment based on the respective indication of properties of the environment. Then, each agent performs its determined real-world action in the environment.

[0013] In an embodiment, determining if a communication condition is above or below a threshold includes, by a given agent of the plurality of autonomous agents, analyzing communication quality between the given agent and a subset of the plurality of autonomous agents and determining the communication condition is above or below the threshold based on the analyzing.

[0014] In another embodiment, responsive to determining the communication condition is above the threshold, the method further includes receiving local observations from agents of the subset of the plurality of autonomous agents. In such an embodiment, the received local observations comprise, i.e., make-up, the coordination communication.

[0015] According to an embodiment, each agent of the plurality of autonomous agents comprises a transformer trained in accordance with a global control policy and, responsive to determining the communication condition is above the threshold, each agent is configured to determine the real-world action by processing the respective indication of properties and the coordination communication with the transformer.

[0016] In yet another embodiment, each agent of the plurality of autonomous agents includes a neural network trained in accordance with a global control policy and, responsive to determining the communication condition is below the threshold, each agent is configured to determine the real-world action by processing the respective indication of properties with the neural network.

[0017] In another embodiment, the respective indication of properties of the environment includes a local observation by the agent.

[0018] In an embodiment, determining if the communication condition is above or below the threshold includes (i) receiving the coordination communication from a control agent and responsively determining the communication condition is above the threshold and (ii) identifying a failure to receive the coordination communication from the control agent and responsively determining the communication condition is below the threshold.

[0019] An embodiment includes, by a control agent, receiving respective indications of properties of the real-world environment from one or more agents of the plurality of autonomous agents. Based on the respective indications received and a global control policy, the control agent determines the coordination communication for each agent of the plurality of autonomous agents and sends the determined coordination communication to each agent of the plurality of autonomous agents.

[0020] Another embodiment includes updating the global control policy.

[0021] In yet another embodiment, updating the global control policy includes (i) a given agent of the plurality of autonomous agents determining quality of the determined real-world action and the control agent updating the global control policy using an indication of the determined quality.

[0022] In a further embodiment, the given agent includes a critic neural network configured to determine the quality of the determined real-world action responsive to one or more inputs. In such an embodiment, the one or more inputs may include at least one of a location of the given agent and an image of the environment.

[0023] In another embodiment, a given agent of the plurality of autonomous agents includes an actor neural network. In this embodiment, determining the real-world action includes: (i) responsive to determining the communication condition is above the threshold, determining the real-world action in the environment by providing the respective indication of properties of the environment and the coordination communication as inputs to the actor neural network, where the actor neural network is configured to determine the real-world action responsive to the inputs and (ii) responsive to determining the communication condition is below the threshold, determining the real-world action in the environment by providing the respective indication of properties of the environment and a prior coordination communication as inputs to the actor neural network, where the actor neural network is configured to determine the real-world action responsive to the inputs.

[0024] Another embodiment includes iteratively performing the determining if a communication condition is above or below a threshold, the determining a real-world action, and the performing until an objective is achieved.

[0025] In an embodiment, the objective is a search operation of the environment.

[0026] In another embodiment, the plurality of autonomous agents includes at least one of: an unmanned aerial vehicle (UAV) and an autonomous underwater vehicle (AUV). [0027] Another embodiment is directed to a system for operating autonomous agents in a real-world environment. The system includes a plurality of autonomous agents. Each agent of the plurality of autonomous agents is configured to determine if a communication condition is above or below a threshold. Responsive to determining the communication condition is above the threshold, each agent determines a real-world action in the environment based on a respective indication of properties of the environment and a coordination communication. Responsive to determining the communication condition is below the threshold, each agent determines a real- world action in the environment based on the respective indication of properties of the environment. Further, each agent performs its determined real-world action in the environment. [0028] Another embodiment is directed to a non-transitory computer program product including a computer-readable medium with computer code instructions stored thereon. The computer code instructions are configured, when executed by a processor, to cause an autonomous agent associated with the processor to (i) determine if a communication condition is above or below a threshold, (ii) responsive to determining the communication condition is above the threshold, determine a real-world action in a real-world environment based on a respective indication of properties of the environment and a coordination communication, (iii) responsive to determining the communication condition is below the threshold, determine a real-world action in the real-world environment based on the respective indication of properties of the environment, and (iv) perform the determined real-world action in the environment.

[0029] It is noted that embodiments of the method, system, and computer program product may be configured to implement any embodiments, or combination of embodiments, described herein.

BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The foregoing will be apparent from the following more particular description of example embodiments, as illustrated in the accompanying drawings in which like reference characters refer to the same parts throughout the different views. The drawings are not necessarily to scale, emphasis instead being placed upon illustrating embodiments.

[0031] FIG. 1 is an illustration of an environment in which embodiments may be implemented. [0032] FIG. 2 is a flowchart of a method for controlling autonomous agents according to an embodiment.

[0033] FIG. 3A is a graphical model of an existing agent control method.

[0034] FIG. 3B is a graphical model of an agent control method according to an embodiment.

[0035] FIG. 4 is simplified diagram of a workflow for controlling agents according to an embodiment.

[0036] FIGs. 5A-C are plots of reward per agent versus the number of training episodes for different control methods, using different numbers of agents.

[0037] FIGs. 6A-C are plots of explored spaces versus number of steps taken for different control methods subject to different communication failure probabilities.

[0038] FIGs. 7A-C are plots of explored spaces versus number of agents for different control methods subject to different communication failure probabilities.

[0039] FIGs. 8A-C are plots of number of collisions versus numbers of agents for different control methods subject to different communication failure probabilities.

[0040] FIG. 9 is an illustration of an environment in which embodiments may be implemented.

[0041] FIG. 10 is a plot of explored grids versus number of agents for different agent control methods implemented in the environment of FIG. 9.

[0042] FIG. 11 is a plot of number of collisions versus number of agents for different control methods implemented in the environment of FIG. 9.

[0043] FIG. 12 is a block diagram showing an agent control method according to an embodiment.

[0044] FIG. 13 is a workflow diagram for a method for operating a plurality of agents in a real-world environment, according to an embodiment.

[0045] FIG. 14 is a diagram depicting an architecture of an embodiment.

[0046] FIG. 15A is a simplified picture of an implementation of an embodiment.

[0047] FIG. 15B is a bird’s eye view of an environment in which embodiments may be implemented. [0048] FIG. 15C is a depiction of an environment in which embodiments may be implemented.

[0049] FIGs. 16A - 16C are plots of rewards per agent versus number of training episodes for different control methods, using different numbers of agents.

[0050] FIGs. 17A - 17C are plots of results (i.e., total rewards versus number of agents, number of occupied landmarks versus number of agents, and number of collisions versus number of agents) for different control methods implementing a mission.

[0051] FIGs. 18A - 18C are plots of training rewards versus number of training episodes for different control methods for different numbers of agents operating in an example environment.

[0052] FIGs. 19A - 19C are plots of results (i.e., total rewards versus number of agents, area coverage versus number of agents, and area coverage versus number of steps) for different control methods operating in an example environment.

[0053] FIGs. 20A - 20C are agent trajectory results for different numbers of agents being controlled in an environment utilizing an embodiment.

[0054] FIGs. 21 A is a plot showing area of coverage versus communication failure probability for different numbers of agents controlled using an embodiment.

[0055] FIG. 2 IB is a plot showing total rewards per step versus communication failure probability for different numbers of agents controlled using an embodiment.

[0056] FIG. 22A is a picture illustrating hardware that may be utilized to implement embodiments.

[0057] FIG. 22B is an example graphical interface showing power consumption of the hardware shown in FIG. 22A implementing an embodiment.

[0058] FIG. 22C is an example graphical interface showing inference time of the hardware shown in FIG. 22A implementing an embodiment.

[0059] FIG. 23 depicts a computer network or similar digital processing environment in which embodiments of the present disclosure may be implemented.

[0060] FIG. 24 is a diagram of an example internal structure of a computer in the environment of FIG. 23, according to an embodiment.

DETAILED DESCRIPTION [0061] A description of example embodiments follows.

[0062] As noted above, the use of autonomous agents, such as Unmanned Aerial Vehicles (UAVs), for performing different tasks and objectives has become prevalent. One such task is Search-And-Rescue (SAR). For SAR, it is typically impractical and risky to deploy rescue personnel directly into the field because the environment is unknown and generally dynamic [1]- [3] (bracketed numbers in this document refer to the enumerated list of references hereinbelow). A promising and achievable solution is to deploy unmanned robots to perform multi-robot SAR operations instead of relying on human beings. Remotely-controlled robots operated by professionals can guarantee good performance, but such an approach is not scalable because it requires significant manpower when the number of robots gets large. Therefore, intelligent robots, such as UAVs, have become a good choice because of their low cost, scalability, and flexibility. To that end, embodiments can be used for the search portion of multi-robot SAR [4], [5], and can be further extended to the search for more generalized targets such as specific persons, vehicles, or valuable equipment. Further, it is noted that while embodiments are described herein as being utilized for controlling agents during search tasks, embodiments are not limited to such functionality and, instead, embodiments can be utilized for any task or objective where multi-agent control is desired.

[0063] A challenge for multi-agent control, particularly in multi-agent SAR missions, is the communication and coordination of the multiple agents. On the one hand, generally, multiple agents can accelerate the process of reaching the goal of the mission, but, on the other hand, a bad coordination and communication strategy can lead to outcomes that are even worse than single-agent deployments (e.g., collisions between agents). Furthermore, from a single agent’s point of view, the environment is non- stationary (i.e., partially observable) because of the actions the other agents take. This makes the environment unpredictable for any individual agent. If each agent is able to observe the states and actions of the other agents, the multi-agent problem boils down to a single-agent problem. However, such full observability is difficult to achieve in real- world SAR missions, and the only way to approach full observability is through communication. In reality, however, communication conditions are not stable and reliable, especially in disaster scenarios such as earthquakes and floods where communication access points (APs) can be damaged or oversubscribed. In such scenarios, communication among the UAVs cannot be guaranteed and it is possible that, at any time during the mission, any agent can become isolated from the others. Thus, determining how to act when a subset of agents is isolated is a significant challenge to the coordination of multiple agents under unstable communication conditions.

[0064] Although communication issues have been discussed in multi-agent machine learning, such as Multi-Agent Reinforcement Learning (MARL) [6], [7] and federated learning [8], [9], most of the existing methods do not consider disconnection issues, but instead, focus on reducing bandwidth requirements. Many existing SAR methods use model-based methods [10], [11] and only focus on a single-robot scenario [12], [13], The existing methods that involve multiple robots either do not consider coordination aspects among agents [14] or do not take communication problems into account [15], [16],

[0065] Existing aerial SAR frameworks can be categorized into two types, optimization based and Reinforcement Learning (RL) based. Delmerico et al. [20] proposed an active search framework in which drones scan and classify the terrain to find feasible paths for ground robots to reach the targets. The objective function in this problem is defined as the time for the ground robots to cover the waypoints proposed by drones. In terms of learning-based frameworks, Sadhu et al. [3] proposed a distributed Multi-Agent Actor Critic (MAAC) framework for SAR missions, where each agent is deployed with an actor and critic pair. In such a method, the critics take the agent’s state and the actions of all agents as input to calculate the action value. This framework can handle the disconnection issues but, as the number of UAVs increases, the requirement for communication bandwidth grows significantly. Sampedro et al. [21] designed a fully- autonomous UAV equipped with a Convolutional Neural Network (CNN) learning model and a Deep Deterministic Policy Gradients (DDPG)-based algorithm to deal with Image-Based Visual Servoing (IB VS) tasks in rescue missions with an acceptable computational cost. However, the performance is only evaluated in an indoor environment. Igoe et al. [5] proposed a RL framework using perfect communication during training and imperfect communication during testing. This RL framework uses a latent state distribution to model the system state and the reward is defined based on the Maximum a Posterior (MAP) of the state distribution. For continuous actions, Igoe et al. [5] used the Poisson Point Process formulation to model the generation of targets. This framework, however, assumes perfect communication during training, which is unrealistic because the offline training environment often deviates from the real environment.

[0066] In the area of MARL, only a few works take unreliable communication into consideration. Lowe et al. [17] proposed MAAC, which extends the actor-critic framework to the multi-agent scenario. In MAAC, the critic receives the states and actions from all agents during training to guide the agents’ coordination while each actor only depends on its own state. Although this framework can perform well, it only works for frameworks that have reliable communication during training. Furthermore, Lowe [17] assumes that there is no communication during deployment/testing. Das et al. [22] proposed TarMAC, which is an actor-critic framework with decentralized actors and a centralized critic. Targeted communication is achieved in Das [22] by integrating the attention mechanism and, at each time step, the final message is generated by running the attention calculation process multiple rounds. Han et al. [23] proposed a hierarchical RL framework based on options, which can be regarded as a set of sub-policies, for multi-agent systems. In Han et al. [23], each agent selects an option and executes the option until it terminates. To ensure full observability, the agent needs to broadcast the option it selects whenever the last option terminates; and, to reduce the communication overhead, an on-demand communication mechanism is implemented. Foerster et al. [7] proposed two communication frameworks, namely Differentiable Inter-Agent Learning (DIAL) and Reinforced Inter- Agent Learning (RIAL). RIAL regards the communication message as an action and trains a Q network to structure the message. In DIAL, the message coming out from one agent’s neural network (NN) goes into another agent’s NN as input and, once the gradient is calculated, the gradient is transmitted back to the former agent. DIAL and RIAL can perform well if the communication is stable, but DIAL and RIAL are not robust when operating using unreliable communications where temporary disconnections can occur.

[0067] As such, improved multi-agent control functionality is needed. To provide such functionality embodiments implement a distributed Hierarchical Multi-Agent Actor-Critic (HMAAC) method for multi-agent control, e.g., for aerial SAR missions. Consider a disaster scene with obstacles and victims scattered in the area. In such an example scenario, the goal of the framework is to control the UAVs in such a way so as to explore the entire unknown environment (i.e., with zero prior knowledge about the environment and the targets) while avoiding (i) overlaps between each UAV’s observations and (ii) collisions with surrounding obstacles and other UAVs. Embodiments provide such functionality.

[0068] Thus, in the proposed HMAAC framework, a high-level policy on top of each agent is introduced, which can generate coordination commands. An embodiment assumes that the low-level agents are conditionally independent of each other given the coordination action generated by the high-level policy, and the low-level agent can make decisions and updates locally even when the communication link is temporarily broken. Further, such an embodiment assumes that the agents connect to a device that can monitor the agents’ status in real time. According to an embodiment, this device (i.e., the device the agents connect to) is operated by a rescue operator and such a device can be regarded as a network Coordination Point (CP) since the multiple agents communicate with it. Unlike most MARL frameworks [5], [7], [17], embodiments do not adopt a Centralized Training Decentralized Execution (CTDE) pipeline, because (i) sometimes embodiments do not have perfect communication during training and (ii) multi-agent systems can benefit from communication among agents and a fully decentralized deployment loses this benefit.

[0069] Embodiments are robust under unreliable communication conditions, even during training. Instead of talking to each other at each time step, each agent only talks to the CP to obtain the coordination action. During disconnection (i.e., when an agent cannot communicate with the CP) for an individual agent, the individual agent uses the coordination action from the last successful communication. Further, embodiments can also work for a multi-team scenario where multiple robots form multiple teams and these robots can have different types and configurations or even come from different parties (e.g., a joint SAR mission conducted by multiple countries, each with its own robots). In such scenarios, a centralized communication structure can greatly help the coordination between the teams/robots toward better performance. Although a centralized structure might suffer from single-point failure (i.e., the CP suddenly breaks down), there are generally sufficient resources on the ground to be used as a backup (e.g., a backup vehicle) CP.

[0070] FIG. 1 illustrates an example environment 100 in which embodiments may be implemented. In the environment 100, embodiments can be used to control the UAVs lOla-b to find cars 102a-b, of a certain color. FIG. 2 is a flowchart of a method 220 that be employed in the environment 100 to implement such functionality.

[0071] In an embodiment, the method 220 is implemented using one or more computing devices. For example, in an example implementation, portions of the method are performed by a control point, i.e., a control agent, and a plurality of autonomous agents through use of one or more associated processors and storage devices, e.g., memory. In such an implementation, the control point and agents implement the method 220 by executing (using the one or more associated processors) computer code instructions stored on storage devices.

[0072] The method 220 begins at step 221 by receiving, at a control agent, respective indications of properties of a real-world environment from one or more agents of a plurality of autonomous agents. At step 222, the control agent determines a coordination action for each agent of the plurality of autonomous agents based on the respective indications received and a global control policy. According to an embodiment, the coordination action includes a respective action for each agent, i.e., the coordination action may include multiple pieces and each piece is for a respective agent. As such, at step 222, a respective action may be determined for each agent. Next, at step 223, the coordination action (determined at step 222) is sent by the control agent to each agent of the plurality of autonomous agents. As previously noted, a respective action may be determined for each agent at step 222 and, as such, at step 223, each respective action may be sent to the appropriate agent of the plurality of autonomous agents. In an embodiment, at step 222 the control agent determines an action for each agent and at step 223 the control agent sends the determined actions to the appropriate agents. To continue, at step 224, each agent determines a real-world action to perform in the environment. In an embodiment, at step 224, if the determined coordination action is received, the real-world action is determined based on a respective indication of properties of the environment and the received coordination action and, if the determined coordination action is not received, the real-world action is determined based on the respective indication of properties of the environment and a prior coordination action (e.g., the last successfully received coordination action). After determining the real-world action (at step 224) each agent performs the determined real-world action in the environment at step 225. [0073] The method 220 may further include updating the global control policy that is used at step 222 to determine the coordination action. In one such embodiment, updating the global control policy includes, by a given agent of the plurality of autonomous agents, determining quality of the determined real-world action and, in turn, the control agent updating the global control policy using an indication of the determined quality. In such an embodiment, the given agent may include a critic neural network. Such a critic neural network may be configured to determine the quality of the determined real-world action responsive to one or more inputs. According to an embodiment, the one or more inputs include at least one of: a location of the given agent in the environment and an image of the environment, e.g., an image captured using a camera mounted to the given agent. Such functionality may be performed by any number of agents. As such, each agent may determine the quality of their respective real-world action and send an indication of the determined quality to the control point. Further, an indication of this determined quality may be communicated to the control point and the control point may, in turn, update the global control policy.

[0074] In another embodiment of the method 220, a given agent of the plurality of autonomous agents includes an actor neural network. In such an embodiment, the real-world action is determined using the actor neural network. For instance, if the determined coordination action is received, the real-world action in the environment is determined by providing the respective indication of properties of the environment and the received coordination action as inputs to the actor neural network. Similarly, if the determined coordination action is not received, the real-world action in the environment is determined by providing the respective indication of properties of the environment and the prior coordination action as inputs to the actor neural network. In such an embodiment, the actor neural network is configured to determine the real-world action responsive to the inputs (i.e., the latest available coordination action and the indication of properties).

[0075] An embodiment of the method 220 may also be performed iteratively so as to perform/complete an objective in the real -world world environment. For example, such an embodiment may iteratively perform the receiving (221), the determining a coordination action (222), the sending (223), the determining a real-world action (224), and the performing (225) until an objective is achieved. Amongst other examples, an objective may be a search operation of an environment.

[0076] In embodiments of the method 220, the plurality of autonomous agents may include any such agents known to those of skill in the art. For instance, in embodiments, the autonomous agents may include unmanned aerial vehicle (UAVs) and/or autonomous underwater vehicle (AUVs).

[0077] Further, it is noted that while the method 220 is implemented in relation to a plurality of autonomous agents, the method 220 does not require stable communication between a control point (i.e., control agent) and the plurality of autonomous agents. For instance, at step 221, indications of properties are received from one or more agents of the plurality, rather than from each agent. In an embodiment, the indications are received from each agent that can successfully communicate with the control agent. Likewise, at step 223 the coordination action is sent to each agent (e.g., the control point attempts to send the coordination action to each agent), but at step 224 each agent implements particular functionality depending on whether the communication at step 223 was successful. In this way, the method 220 can operate even where communication is unstable and unreliable. Further, in an embodiment, the control point determines agents that the control point can successfully communicate with, e.g., by sending a ping, and, in turn, at step 223 the control point only sends the actions to agents that can currently communicate with the control point, e.g., where a response to the ping was received by the control point.

[0078] Notation

[0079] Below, are notations for MARL and a HMAAC framework according to an embodiment. HMAAC, in an embodiment, introduces a high-level policy to model coordination explicitly and low-level policies are assumed to be conditionally independent given the coordination action. Unlike MAAC, HMAAC does not require the states and actions of other agents to perform an update and, instead, HMAAC uses the coordination action and its own state.

[0080] MARL can be modeled with a Partially Observable Markov Decision Process (POMDP), which is an extension to a multi-agent manner of the Markov Decision Process (MDP). A MDP for N agents includes a set of states , which describe the configurations for the participating agents, a set of actions and a set of observations Each agent i possesses a policy parameterized by 6^. The environment generates the next state based on the state transition function T\S X ... x <A_N>-+S. Each agent receives a reward from the environment as a function of state and action r^.S X c/Zi^IR as well as an individual observation that is correlated with the state, o_l\S'->O_L. Each agent tries to maximize its total expected return where y is a discount factor and T is the total time length.

[0081] Hierarchical Multi-Agent Actor Critic

[0082] In MAAC, due to the partial observability, each agent utilizes the states and actions information from the other agents to make a decision, implying that the agents are interdependent on each other. In reality, this dependency is realized via the continuous communication among the agents. However, this requirement cannot be guaranteed, especially in disaster scenarios where a communication point can be damaged or oversubscribed. The main reason for this continuous communication requirement is that the critic in MAAC needs global information to perform an update towards better coordination. The graphical model 330 in FIG. 3A shows that each agent 33 la-n in MAAC depends on every other agent.

[0083] To relax this inter-dependency, embodiments add a high-level policy to the MAAC framework, e.g., 330. As such, in embodiments, low-level policies take local observations and the coordination action produced by the high-level policy as input to generate the local action. The high-level policy can also be regarded as a latent variable, and the low-level agents are conditionally independent given the latent variable. This is shown by graph 332 of FIG. 3B where the agents 334a-n take a coordination action 335a-n from the high-level agent 333.

Consider that each agent, e.g., 334a-n, has a deterministic policy parameterized by then the joint policy of low-level agents, [ij_t, can be represented as jj.j_t are the observations from the agents and a_c is the coordination action. Adding the high-level policy as a latent variable, as in FIG. 3B, removes interdependency among the low-level policies. By querying the high-level policy agent, e.g., 333, to obtain the coordination action, the agents, e.g., 334a-n, are able to make decisions without directly talking with other agents and perform local optimizations on their own.

Furthermore, the same equation holds for stochastic policies with discrete action spaces. Unlike MAAC, where the critic needs the states and actions from all agents, the local critic in HMAAC only takes the states and actions information from its corresponding actor for the prediction of the action-value function. The global information for coordination is provided by the high-level policy, which can observe the states of all agents.

[0084] To formally describe the proposed framework, according to an embodiment, let fi_h denote the high-level policy parameterized by 6_h as a function of all agents’ observations. Actornetwork of agent i takes the local state, Oj, as input and outputs a deterministic action while the high-level policy [i_h takes the observations of all agents o as input and outputs the coordination action a_c 6 Jl_c, where <A_C is the coordination action space. In such an embodiment, there are two sets of parameters to be optimized: (1) the local agents’ parameters {6^,.,.,0/v} and (2) the high-level policy’s parameters 0_h. If an agent can communicate with the CP, the agent will then upload its local observation to the CP to get the latest coordination action a_c and if the agent cannot communicate with the CP, the agent will use the coordination action from the last successful communication.

[0085] What follows is a derivation of the policy gradient for these two sets of parameters, the local agents’ parameters {0_b...,0/v} and the high-level policy’s parameters 6_h. The overall objective of MARL is to optimize the accumulated rewards of the agents, which can be represented by J(0_b9h) = E [7?i] . The gradient of this objective function with reference to the local policy model, V_e/(0(,0/_l), can be written as:

As for the high-level policy, it will be updated along with each agent. Specifically, for agent i, if it can communicate with the CP, then the gradient for the high-level policy on agent i’s side, V_8h J(O_b0h), can b^e written as: where Dj represents the local Experience Replay (ER) buffer for agent i, which contains tuples < o_ba_ba_c,r_bo_b > . Note that the ER buffer stores the local information without the need for any global information. The critic for agent i, Qf, is updated by the loss function defined as: where .j is the target policy with delayed parameters 6j' for agent j. When the communication link works, the local agents will follow equations (2) and (5) to update their actors and critics. It is noted that, unlike in (5), where a_c is generated from the high-level policy, in (2) such an embodiment samples the coordination actions directly from the ER memory, i.e., ER buffer, because such an embodiment does not need the gradients to backpropagate through the high- level policy during the local update.

[0086] FIG. 4 is simplified diagram of a workflow 440 for controlling agents according to an embodiment, e.g., HMAAC. In the workflow 440, during real-time decision-making, the high- level policy 441 first generates its action a_c 442 based on observations 448 of the environment 445. In turn, the high-level policy, i.e., agent 441, transmits the determined action 442 to the agents 443a-n. Then, individual agents 443a-n make decisions, i.e., determine an action a, 446, based on their local observation o, 444 of environment 445 and the coordination action 442, if the coordination action 442 is received. If the coordination action 442 is not received, the agents 443a-n determine the action 446 based on their local observation o_t 444a and the last coordination action successfully received. In addition the critic 451 passes action value 444b to the actor 450, and the actor 450 may be updated according to the action value 444b. To continue, each agent 443a-n executes their determined action 446 and, after executing the action 446, the agents 443a-n receive a reward 453 and the next observation 447 from the environment 445. To update the models, mini-batches are sampled from the ER memory 449 and passed to actors 450 and critics 451 for loss calculation. If the communication between one UAV (i.e., agents 443a-n) and the CP (i.e., high-level policy agent 441) is good, then the agent will upload the loss 452 for the agent 441 to update the high-level policy. It is noted that in FIG. 4 solid lines indicate the forward inference and dashed lines represent update/learning information flows.

[0087] Below is a description of a method that may be implemented in the framework 440:

1. Initialize random noise JV for exploration;

2. Receive initial observations x;

3. for episode = 1,2, ...do

4. for step = 1,2, ...do

5. Upload agents’ observations o to the CP; 6. Calculate coordination action a_c = | o);

8. if communication is available then

9. Receive a_c from CP;

10. else

11. Use a_c from last communication;

12. Select action a_t = gi(ai|o₍-,a_c) + TV);

13. Execute a_t to get and o ;

14. Update and Q_L via (2) and (6);

15. if communication is available then

16. Compute gradients for [i_h via (5);

17. Upload gradients to CP to update

[0088] Performance and Evaluation

[0089] Below is an explanation of performance of embodiments along with an evaluation of embodiments in comparison to existing methods. First, the setup for the experiments used to obtain the discussed results is described. Second, performance of embodiments under two scenarios is evaluated: (i) unreliable communications, where packets drop following a Bernoulli process and, as a result, communication between UAVs and the CP fails following a failure probability p and (ii) shadow zones, where there are certain zones (e.g., an area blocked by a building) in which the UAVs lose communication.

[0090] Comparison Plan

[0091] The description below compares embodiments against two existing algorithms: (i) MAAC [3] and (ii) REINFORCE [19], In MAAC [3] each UAV observes/receives the states and actions of all other UAVs to update its critic and independently updates its actor based on its own experiences. Further, in MAAC [3], when one UAV is isolated (i.e., cannot communicate with others), the isolated UAV cannot receive the states and actions from others and, therefore, cannot perform the update for both its actor and critic. In REINFORCE [19] each agent has a policy network and updates the individual policy network based only on its own states and actions. In REINFORCE [19] no communication is needed because it is a decentralized framework. [0092] Environment Setup And Definition

[0093] The experiments were conducted in AirSim [24], a simulator developed using Unreal Engine for realistic visual simulation. To do experiments, the environment for the problem is formally defined. In the environment, it is assumed that each UAV can translate and rotate freely. The state of agent i consists of the agent’s position, angle, and an ego-centric grid containing obstacle information. To obtain this ego-centric grid, the map is discretized with a certain granularity. This experiment used a 7 x 7 ego-centric grid that is able to define 49 area units surrounding the agent. The action consists of the translation and rotation of the UAV. The reward is defined as a combination of multiple components including exploration rewards (+1), the target found reward (+5), and collision reward (-100). The rewards were designed in such a way that the agents could learn to explore more areas and avoid collisions. A well-trained model will learn to explore more areas while avoiding collisions with other agents as well as obstacles. At each time step, the agent takes an action and the environment checks if the new state reached after taking the action satisfies these conditions. For example, if the agent discovered a new area, but collided with a teammate, then the agent will get both exploration and collision rewards, which sum up to -99. In the experiment, collision-avoidance algorithms were disabled to count the number of collisions to asses coordination performance.

[0094] Training And Testing Setup

[0095] In the embodiment implemented to determine the experimental results, both the low- level and high-level policy had a three-layer Multi-Layer Perceptron (MLP) structure with 256 units each. For the critic, the state and action first went through a fully-connected layer with 64 units and, then, are concatenated and fed to a fully-connected layer with 256 units. The learning rates for actors and critics were 0.001 and 0.005, respectively. The two frameworks (i.e., the actor and critic) were both implemented in Pytorch [25], a deep-learning toolkit for Python. To add random exploration in the actors’ behaviors, Ornstein-Uhlenbeck noise [26] was added to the output of the actor-network. A pre-trained Yolov4 [27] model was used as the detector because of its good performance and high speed. As mentioned above, in contrast to existing methods, the experimental implementation did not utilize centralized training and decentralized testing mechanisms. As a result, the models were trained with unreliable communication conditions and saved and, then, the trained model was used for inference to generate the testing results. In the experiments, continuous space and action space were used, however, embodiments also work for discrete action spaces with the help of the Gumbel-Softmax method [28], which transforms the continuous output from the actor-network to a differentiable categorical distribution.

[0096] Bernoulli Communication Model

[0097] Performance of embodiments was evaluated under unstable communication and partial observability settings. The evaluation relied upon modeling packet loss as a Bernoulli process [5], [29], with a defined packet drop probability p. That is, with p — 0, the communication is ideal while with p = 1 there will be no communication at all among the agents. It is assumed that, whenever UAV (A) wants to send a message to UAV (B), the packet containing the message has probability p of being dropped. For embodiments, such a message could contain the states and actions of the UAV (A), which is needed when updating UAV (B). The experiments were conducted in the AirSim simulator, using a collection of 3D people models as victims and utilizing the detection functionality provided by AirSim for victim detection. The obstacles’ and the victims’ locations were different for training and testing to ensure that the method generalizes.

[0098] Reward Versus Number Of Episodes

[0099] FIGs. 5A-C are plots 550a-c that evaluate how embodiments and existing methods learn during training. The plots 550a-c indicate total reward average per agent 551a-c versus number of episodes 552a-c for experiments with three agents (FIG. 5 A), six agents (FIG. 5B), and 9 agents (FIG. 5C). Each plot 550a-c includes a series for embodiments (i.e., HMAAC) 553a-c, MAAC 554a-c, and REINFORCE 555a-c. The results in FIGs. 5A-C were obtained using a communication failure probability of p = 0.4. FIGs. 5A-C show that as the number of UAVs increases, the values of the rewards decrease because the number of victims and the search space is the same in these three scenarios, and a higher number of agents leads to a decrease in the average reward. Furthermore, it can be observed that HMAAC (553a-c) and MAAC (554a-c) converge to a good reward level in just a few episodes, while REINFORCE (555a-c) does not converge at all. This is because both HMAAC (553a-c) and MAAC (554a-c) can do a good job in a multi-agent scenario. Conversely, for decentralized methods like REINFORCE (555a-c), there is neither communication nor coordination and, therefore, performance degrades rapidly. [00100] Exploration Speed

[00101] To see how embodiments and the MAAC and REINFORCE methods perform in terms of team exploration in different communication scenarios, the search space was discretized into a 20 x 20 grid and the cumulative number of explored grids was evaluated. Results are shown in the plots 660a-c of explored spaces 661a-c versus number of steps 662a-c for experiments with six agents and communication failure probablity p equal to zero (FIG. 6A), 0.3 (FIG. 6B), and 0.6 (FIG. 6C). Each plot 660a-c includes a series for embodiments (i.e., HMAAC) 663a-c, MAAC 664a-c, and REINFORCE 665a-c. Since the agents do not have any prior knowledge about the environment or the targets, the number of explored grids, i.e., spaces, 661a-c can reflect the efficiency of the target search system. It can be observed that, as the communication failure probability increases, the performance of HMAAC (663 a-c) improves. This is because the agents can still maintain a decent global knowledge of the network from previous steps during disconnection.

[00102] Exploration Performance Versus Number of Agents

[00103] To show the scalability of embodiments versus existing methods, the search space was discretized into a 20 x 20 grid and the number of UAVs deployed in the SAR mission was varied. The results are shown in FIGs. 7A-C in the plots plots 770a-c of explored grids, i.e., spaces, 771a-c versus number of agents 772a-c for experiments with communication failure probablity p equal to zero (FIG. 7A), 0.3 (FIG. 7B), and 0.6 (FIG. 7C). Each plot 770a-c includes a series for embodiments (i.e., HMAAC) 773a-c, MAAC 774a-c, and REINFORCE 775a-c. Because the agents do not have prior knowledge about the targets, explored spaces reflects the efficiency of finding targets. It can be observed that, as the number of agents increases, MAAC (774a-c) experiences a diminishing return. This is due to the fact that MAAC (774a-c) is fl at- structured and, thus, can only handle a low number of agents (8 and 9 for p equal to 0.3 (FIG. 7B) and 0.6 (FIG. 7C), respectively), implying that once the number of agents “saturates”, the performance of MAAC (774a-c) starts dropping.

[00104] Coordination Performance

[00105] Another metric to measure how well coordination is achieved in multi-agent missions is the number of inter-agent collisions. FIGs. 8A-C show the results for different communication conditions in the plots 880a-c of number of inter-agents collisions per episode 881 a-c versus number of agents 882a-c for experiments with communication failure probability p equal to zero (FIG. 8A), 0.3 (FIG. 8B), and 0.6 (FIG. 8C). Each plot 880a-c includes a series for different methodologies, embodiments (i.e., HMAAC) 883a-c, MAAC 884a-c, and REINFORCE 885a-c. The plots 880a-c show that, as the number of agents increases, the number of collisions also increases, which is expected since more agents leads to a greater chance of collisions due to increased coordination challenges. Moreover, it can also be observed that, as the communication conditions deteriorate, the inter-collisions in the HMAAC framework (883a-c) do not increase like in MAAC (884a-c) and REINFORCE (885a-c). MAAC (884a-c), in fact, shows less robustness to communication loss, while REINFORCE (885a-c) cannot use coordination to avoid collisions.

[00106] Shadow Zones

[00107] In addition to modeling packet dropping as a Bernoulli process, embodiments were also evaluated against existing methods in a shadow zone scenario, i.e., where there are obstacles, e.g., multiple buildings, in the search area that block communications. In this experiment, it was assumed that buildings can block communications and create shadow zones, in which agents are not be able to communicate with other agents and the control point. FIG. 9 illustrates such an example environment 990. The environment 990 includes four buildings 991a- d and the coordination point 992. Due to building blockage, shaded areas 993a-d cannot be reached by the coordination point 992 and, therefore, the zones 993 a-d are shadow zones. When UAVs enter these zones 993a-d, the UAVs lose communication with the CP 992 and can only perform local updates without being able to acquire global information. This scenario was evaluated in AirSim with buildings and targets scattered around.

[00108] Exploration Performance Versus Number Of Agents

[00109] The plot 1000 in FIG. 10 shows the performance of embodiments and existing methods in terms of explored areas while operating in a shadow zone environment. In particular, the plot 1000 shows the number of explored grids 1001 during testing against the number of agents 1002. The plot 1000 includes a series for different methodologies, including embodiments (i.e., HMAAC) 1003, MAAC 1004, and REINFORCE 1005. It can be observed that HMAAC 1003 achieves the best performance among these three models as the number of agents 1002 increases. Moreover, at one point, the performance of MAAC 1004 drops and becomes worse than decentralized REINFORCE 1005 due to the saturation of the number of agents 1002. When the number of agents 1002 reaches 8, REINFORCE 1005 outperforms MAAC 1004 because the bad coordination due to disconnection in MAAC 1004 causes negative effects on the group performance.

[00110] Collision Performance

[00111] FIG. 11 is a plot 1100 of the number of inter-agent collisions 1101 against the number of agents 1102 for an embodiment HMAAC 1103 and existing methods, MAAC 1004 and REINFORCE 1105, operating in a shadow zone environment. The plot 1100 shows that REINFORCE 1105 achieves the best performance in terms of number of collisions 1101. However, as shown by the plot 1000 described hereinabove in relation to FIG. 10, REINFORCE is not the best model in terms of exploration, i.e., number of explored grids 1001. As such, REINFORCE has learned not to collide, but REINFORCE is not good at exploring the environment due to the lack of coordination.

[00112] Embodiments implement a hierarchical multi-agent actor-critic framework that can be used for a plurality of objectives, e.g., distributed aerial search and rescue. To relax interdependency among agents, embodiments utilize a high-level policy, and low-level agents are assumed to be conditionally independent given a coordination action. According to an embodiment, the high-level policy is deployed on a network coordination point and the actors and critics are deployed on UAVs. The above described evaluations examined the performance of embodiments in two scenarios: (a) unstable communication and (b) shadow zones. Compared to MAAC and decentralized REINFORCE, embodiments are more scalable and robust to communication loss since embodiments outperform MAAC and REINFORCE when the number of agents is large and when the communication conditions are bad.

[00113] Embodiments can be extended to decentralized versions, where the communications are restricted to n-hop links, and communication to the CP is not performed/attempted during each iteration. Embodiments can also lower communication overhead by employing an on- demand communication mechanism. Embodiments can also implement a federated update protocol, where each agent copies the high-level policy to their local storage and performs joint optimization for the actor, critic, and the copy of the high-level policy. Then, after a predefined interval, the local agents can upload their copies to the CP for merging in order to aggregate the knowledge.

[00114] As described above, UAVs have become prevalent in SAR missions. However, existing solutions to the control and coordination of UAVs are mostly limited to specific environments and are not robust to handle unreliable/unstable communications. To deal with these challenges, embodiments place a high-level policy on top of individual low-level actorcritic policies to relax the inter-dependency among the agents. The low-level policies are considered conditionally independent given the coordination action, which is generated by the high-level policy. A Centralized Training Decentralized Execution (CTDE) would not work because it cannot be assumed that communication is always perfect during training and that the whole system can rely on stable communications during deployment. Results show that embodiments are scalable and robust to unreliable communication and outperform existing methods in terms of exploration and coordination when the number of agents is large and communications are not stable.

[00115] Embodiments, which may be referred to herein as HMAAC, may implement a hierarchical MARL framework which introduces another hierarchy on top of the multi-agent actor-critic framework. In an embodiment, this additional level is implemented by adding a high- level policy that observes the states of all low-level policies to achieve better coordination during disconnection. Embodiments may also relax the inter-dependency among the agents and allow the actors and critics to perform updates with their own states and actions even during disconnection.

[00116] As described herein, embodiments, implemented in an aerial SAR mission have been evaluated in AirSim [18] under two conditions, (i) unreliable communications, where communication fails with a probability and (ii) shadow zones created by buildings. These evaluations compare HMAAC (embodiments) with MAAC [3] and REINFORCE [19], The results show that HMAAC performs better in terms of scalability and robustness to unreliable communication as it outperforms other frameworks when the number of agents gets large and the communication conditions are unreliable.

[00117] Advantageously, embodiments implement multi-agent reinforcement learning in environments with unreliable and/or discontinuous communication conditions. Embodiments achieve better coordination during disconnection by adding a high-level policy that observes the states of low-level policies. Further, embodiments relax the inter-dependency among the agents and allow the actors and critics to perform updates with their own states and actions even during disconnection.

[00118] Transformer-based Communication-Aware Multi-Agent Actor-Critic (TMAAC) for Cooperative Missions

[00119] Although embodiments presented above, e.g., HMAAC, offer superior performance compared to existing methods, the performance of the above presented embodiments may suffer in the event of a coordination point failure. The embodiments presented below, e.g., TMAAC, provide a solution to this vulnerability and implement functionality that does not rely upon a coordination point.

[00120] Multi-agent controls have become prevalent as more and more multi-robot systems are put into use. Among the many controlling frameworks, MARL -based frameworks can achieve decent performance and possess flexibility due to their data-driven nature. However, most of these MARL frameworks focus on decentralized execution (i.e., no communication) or structured communication, without considering unreliable and unpredictable communication, which is common in scenarios like disaster scenes. To mitigate this problem and use communication as much as possible, embodiments propose a localized MAAC -based framework (which may be referred to herein as TMAAC), where each agent will communicate (e g., only communicate) with its reachable neighbors. Since the set of reachable neighbors might vary, an embodiment utilizes a transformer neural network to handle the consequently varied input dimension. Furthermore, during execution, to provide robustness to disconnections, an embodiment implements a dual-modality execution strategy. Specifically, when communication is better than a certain condition for an agent, the agent will enter the networked modality to communicate with other agents before making decisions; otherwise, the agent will enter the decentralized modality where the agent will only use its local observation to make decisions. Evaluations of embodiments, e.g., TMAAC, on a benchmark simulator, as well as a SAR mission scenario with the Airsim simulator are provided below. The results show that dualmodality TMAAC can achieve superior performance compared to other frameworks such as Multi-Agent Deep Deterministic Policy Gradient (MADDPG) and Multi-Agent Proximal Policy Optimization (MAPPO).

[00121] MARL has been widely applied to various applications, including SAR [A2]-[A4], autonomous vehicles [A5]-[A7], smart traffic control [A8]-[A10], and other applications that involve coordination. Since multiple agents are involved in MARL, communication and coordination have become crucial to the success of a MARL framework, especially for applications where communication is sparse, such as disaster scenes. Previous work investigating simply the use of individual RL algorithms, such as Multi-Agent Q Learning (MADQN), failed because the coordination aspect was not considered. There has also been work studying intelligent communication methods (e.g., who to communicate with, what message to send, etc.) in MARL frameworks [Al 1 ]— [Al 3] . However, most of these works aim at reducing the required bandwidth, without considering unreliable communications.

[00122] In MARL, CTDE [A14]-[A16] is a popular approach for training and deploying intelligent agents. In the training phase of CTDE, each agent receives global information, including information from every other agent through ideal communication, as inputs to make decisions. Then, during deployment, communication is completely removed, assuming that the agents have learned to coordinate with each other even without communication. Although studies have shown that CTDE can achieve satisfactory performance if agents are properly trained, full decentralization may be a waste of resources and communication can further improve the performance of MARL frameworks.

[00123] The main reason for adopting CTDE is unreliable and possibly sparse communication during the execution (i.e., deployment) phase, and decentralized execution can avoid the disruption caused by problems such as packet drop and temporary disconnection. If the communication is perfect, that is, each agent can have global information whenever the agent wants, the MARL problem boils down to a single-agent RL problem, with the action space being the joint action space of the agents, assuming real-time synchronization among agents. However, perfect or ideal communication is hardly achievable in real applications, especially in multi-robot applications. If communication is deprived, the agents are completely independent, even if they are equipped with communication hardware. Furthermore, the idea that decentralized agents can outperform centralized agents is counterintuitive because even the most well -trained team needs to communicate inside the team to coordinate moves and timing.

[00124] The description below further details existing methodologies and their shortcomings.

[00125] Multi-Agent Reinforcement Learning (MARL)

[00126] MARL has been an active research topic for a long time. Lowe et al. [A15] proposed MADDPG, which extends the actor-critic framework to the multi-agent scenario. In MADDPG, the critic receives the states and actions from all agents during training to guide the agents’ coordination, while each actor only makes decisions based on its local observation. Das et al. [A21] proposed TarMAC, which is an actor-critic framework with decentralized actors and a centralized critic; Targeted communication is achieved in TarMAC by integrating the attention mechanism, and, at each time step, the final message is generated by running the attention calculation process multiple rounds. Singh et al. [A22] devised a gating mechanism that restrains agents so that they communicate only when necessary. Singh also claimed that using individual rewards instead of integral rewards for the whole group is more efficient. Jiang et al. [ A23 ] proposed to incorporate the Graph Neural Network (GNN) into the multi-agent reinforcement learning environment. In Jiang a graph of k nearest neighbors is built for all agents in which each agent logically connects to its nearest neighbor. Convolutional layers are leveraged in the Jiang framework to extract latent features among agents.

[00127] The methods discussed above, e.g., MADDPG, can handle MARL problems under the assumption that communication is perfect. However, communication is not typically ideal, and thus these methods suffer from performance drops when communication is unreliable. Foerster et al. [A24] proposed two MARL frameworks, namely Differentiable Inter-Agent Learning (DIAL) and Reinforced Inter-Agent Learning (RIAL); the latter regards the communication message as an action and trains a Q network to structure the message, while in the former, the message coming out from one agent’s neural networks goes into another agent’s neural networks as input and, once the gradient is calculated, it is transmitted back to the former agent. In this way, gradients can flow through the communication link and the agents can learn what to communicate by the process of backpropagation. Sheng et al. [A25], proposes a framework for agents to learn the topological structure of the communication network. Sheng leverages the intra- and intergroup communication to coordinate agents to change their positions in order to form a specific network structure. Agents form groups with their neighbors according to the algorithm and the desired structure is computed using a GNN. These approaches can perform well if the communication is stable, but the existing approaches are not robust in operating in environments with unreliable communications where temporary disconnections can happen. The aforementioned approaches adopt the CTDE approach and cannot incorporate communication during execution. Karabag et al. [A26] proposed an imaginary play approach to deal with possible temporary disconnect. In the Karabag framework, each agent maintains imaginary copies of their teammates so that, in case of disconnection, the agent can estimate other team members’ states and actions locally. Further, Karabag proposed reducing correlation among agents’ policies to guarantee convergence of the algorithm. Nevertheless, the Karabag framework is not scalable since each agent has to store all the other agents’ policies. Zhang et al. [A27] proposed a fully decentralized framework, where agents coordinate only with their neighbors at each time step. In Zhang, to combine the knowledge of all the agents, each agent updates its parameters with a weighted sum of its neighbors’ parameters and, as time passes by, a global consensus is reached. Although convergence is reached, the Zhang method requires much more time to train compared to a centralized structure.

[00128] Hierarchical Reinforcement Learning

[00129] Hierarchical Reinforcement Learning (HRL) adds another hierarchy on top of the RL framework. Bacon et al. [A28] proposed an option-critic framework, where an option selector is implemented as a Deep Q Network (DQN) to choose options, which can be regarded as a policy for a short time. In Bacon, a termination function is also trained to determine whether the current option should terminate. Levy et al. [A29] proposed a hierarchical actor-critic algorithm, where multiple hierarchies are built on the actor. The Levy method treats the coordination actions as goals for the actor in the next hierarchy to reach. Han et al. [A30] proposed a hierarchical RL framework based on options, which can be regarded as a set of sub-policies, for multi-agent systems. Each agent selects an option and executes the option until it terminates. To ensure full observability, the agent needs to broadcast the option it selects whenever the last option terminates and, to reduce the communication overhead, an on-demand communication mechanism is designed.

[00130] Communication under Predefined Protocol [00131] Melo et al. [Al l] proposed Query Partially Observable Markov Decision Process (QueryPOMDP). Melo defined various variables, including communication observations, primitive observations, communication rewards, and domain-specific rewards. According to the Melo method, an agent can query another agent’s primitive observation as its communication observation at the current time step. With all these variables, a dynamic Bayesian network is adopted to model the variables. Zhang et al. [A12] applied the Distributed Constraint Optimization (DCO) algorithm to coordinate among agents. In Zhang Q learning is deployed on each agent in their framework and the sum of several local Q functions of the agents is used to approximate the global Q function. In this framework (Zhang), within each timestep, agents can learn which agents to coordinate with and balance the trade-off between the benefits of coordination and the communication cost. Kasai et al. [A13] designed several environmentspecific strategies to achieve multiagent coordination. A code-learning approach is proposed to learn binary signals for communication. These protocol-based learning methods achieve good results in certain settings, but do not generalize when the environment changes.

[00132] Embodiments described in previous sections, referred to as HMAAC [Al], aims to mitigate the problems described above. To relax the interdependence (that is, the need for intercommunication) between agents, a high-level policy deployed on a coordination point is utilized. In HMAAC, each agent tries to communicate with this coordination point to acquire information about other agents. However, HMAAC still requires communication between the agents and the centralized coordination point.

[00133] To solve the aforementioned shortcomings, another embodiment implements Centralized Training, Dual-Modality Execution (CTDME) where execution has two modalities, a networked mode and a decentralized mode. In the networked mode, an agent will receive observations from other agents as input to generate/determined an action, while in the decentralized mode, an agent will use its local observation as input even if it can receive information from a small fraction of agents. According to an embodiment, the switch between these two modalities depends on the communication condition. For instance, in an embodiment, when the communication condition for an agent is better than a certain condition, the agent enters the networked modality and, otherwise, the agent will enter the decentralized modality. Further details are described below in relation to FIG. 12. [00134] According to an embodiment, in network mode, each agent will make decisions based on the observations of its neighbors, rather than all agents. A key problem is that the set of neighbors of an agent can vary over time and, therefore, integrating the varying information of other agents becomes a key challenge. Embodiments remedy this through utilizing a transformer neural network architecture, leveraging the transformer neural network’s ability to take various input length input data. Furthermore, evaluation results show that when communication is worse than a threshold, network execution becomes worse than decentralized execution. Based on this finding, TMAAC contains two modalities: i) when communication condition is better than a threshold condition, agents adopt a networked execution protocol, where agents can communicate with each other; and ii) when the communication condition is worse than the threshold, the agent will not communicate, but rely on itself.

[00135] Embodiments provide a transformer-based multi-agent reinforcement learning framework that adopts a CTDME scheme. In an embodiment, agents can communicate with each other during execution. The messages (e.g., observations) received from other agents can be the input to a transformer actor, so that the action (i.e., determined action to perform) is conditioned on the observations of neighboring agents. Embodiments also utilize a communication-aware dual-modality execution approach to mitigate the problems caused by bad connectivity. To be specific, embodiments can employ a threshold representing a number of agents with which one agent is connected. In such an embodiment, if the number of connected agents is greater than the threshold, the agent will operate in accordance with the networked modality and, otherwise, the agent will operate in accordance with the decentralized modality.

[00136] Herein, embodiments, e.g., TMAAC, are evaluated in both benchmark environments and a realistic SAR simulator. Embodiments are evaluated in a modified version of the benchmark environment, Multi-Particle Environment (MPE) [A15], [A17], and an aerial SAR simulator in AirSim [Al 8], Embodiments are compared to Multi -Agent Deep Deterministic Policy Gradient (MADDPG) [Al 5], Multi-Agent Proximal Policy Optimization (MAPPO) [A19], Off-Policy Multi-Agent Decomposed Policy Gradients (DOP) [A20], and HMAAC.

[00137] FIG. 12 is a diagram illustrating a method 1200 of centralized training, dual-modality execution, according to an embodiment. The method 1200 includes the two modalities of operations, a networked modality 1202 and a decentralized modality 1203 for controlling operation of the plurality of agents 1201 a-d. During training, the agents 1201a-d are trained with global observations, in which ideal communication is assumed.

[00138] During execution, each agent 1201a-d will enter one of the two modalities 1202 and 1203 based on their specific communication conditions. If the communication condition for an agent is acceptable, e.g., above a threshold, the agent will enter the networked mode of operation 1202. If the communication condition for an agent is unacceptable, e.g., below a threshold, the agent will enter the decentralized mode of operation.

[00139] An agent 1201a-d in networked mode 1202, determines an action 1208a by processing (i) a local observation 1204a (i.e., an observation made by the agent itself) and (ii) observations 1204b from other agents (i.e., observations obtained from agents communicatively coupled to the agent determining the action) with the networked actor 1206 (which may be a trained transformer). An agent 1201a-d in decentralized mode 1203 determines an action 1208b by processing a local observation 1205 with a decentralized actor 1207 (which may be a neural network).

[00140] FIG. 13 is a flow diagram of a method 1300 for operating a plurality of autonomous agents in a real-world environment according to an embodiment. According to an embodiment, the method 1300 is implemented by each agent of the plurality of autonomous agents. The method 1300 may be implemented using any computing devices or combination of computing devices known to those of skill in the art. Further, the method 1300 may be implemented on board UAV devices and loT devices, amongst other examples.

[00141] The method 1300 begins by each agent of the plurality of autonomous agents determining 1301 if a communication condition is above or below a threshold. If an agent (of the plurality) determines 1301 the communication condition is above the threshold, the agent determines 1302 a real -world action in the environment based on a respective indication of properties of the environment and a coordination communication. Conversely, responsive to determining 1301 the communication condition is below the threshold, the agent determines

1303 a real-world action in the environment based on the respective indication of properties of the environment. After determining (1302 or 1303) the real-world action, each agent performs

1304 its determined real-world action in the environment. [00142] According to an embodiment, determining! 301, if a communication condition is above or below a threshold, includes a given agent of the plurality of agents analyzing communication quality between the given agent and a subset of the plurality of autonomous agents and determining (1301) the communication condition is above or below the threshold based on the analyzing. Further, in an embodiment of the method 1300, responsive to determining the communication condition is above the threshold, the method 1300 further includes receiving local observations from agents of the subset of the plurality of autonomous agents, where the received local observations comprise the coordination communication. [00143] Further, in an embodiment of the method 1300, each agent of the plurality of autonomous agents comprises a transformer trained in accordance with a global control policy and, responsive to determining the communication condition is above the threshold, each agent is configured to determine (1302) the real-world action by processing the respective indication of properties and the coordination communication with the transformer. In another embodiment, each agent of the plurality of autonomous agents comprises a neural network trained in accordance with a global control policy and, responsive to determining the communication condition is below the threshold, each agent is configured to determine (1303) the real-world action by processing the respective indication of properties with the neural network.

[00144] In embodiments of the method 1300, each respective indication of properties of the environment may include respective local observations by each agent.

[00145] Determining (1301) if the communication condition is above or below the threshold may include, according to an embodiment, receiving the coordination communication from a control agent and responsively determining the communication condition is above the threshold and identifying a failure to receive the coordination communication from the control agent and responsively determining the communication condition is below the threshold.

[00146] An embodiment of the method 1300 may also include, by a control agent, (1) receiving respective indications of properties of the real-world environment from one or more agents of the plurality of autonomous agents, (2) based on the respective indications received and a global control policy, determining the coordination communication for each agent of the plurality of autonomous agents, and (3) sending the determined coordination communication to each agent of the plurality of autonomous agents. Additionally, an embodiment may update the global control policy. In an embodiment, updating the control policy includes: (1) by a given agent of the plurality of autonomous agents, determining quality of the determined real-world action and (2) by the control agent, updating the global control policy using an indication of the determined quality. Further, the given agent may include, according to an embodiment, a critic neural network configured to determine the quality of the determined real-world action responsive to one or more inputs. In some embodiments, the one or more inputs may include at least one of (i) a location of the given agent and (ii) an image of the environment.

[00147] Further, according to an embodiment of the method 1300, a given agent of the plurality of autonomous agents includes an actor neural network. In such an embodiment, determining the real -world action includes: (i) responsive to determining the communication condition is above the threshold, determining (1302) the real-world action in the environment by providing the respective indication of properties of the environment and the coordination communication as inputs to the actor neural network, where the actor neural network is configured to determine the real-world action responsive to the inputs, and (ii) responsive to determining the communication condition is below the threshold, determining (1303) the real- world action in the environment by providing the respective indication of properties of the environment and a prior coordination communication as inputs to the actor neural network, where the actor neural network is configured to determine the real-world action responsive to the inputs.

[00148] Embodiments of method 1300 may iteratively perform the determining (1301) if a communication condition is above or below a threshold, the determining a real-world action (1302/1303), and the performing (1304) until an objective is achieved. In embodiments, the objective may be any objective known to those of skill in the art. For instance, the objective may be a search operation of the environment. Additionally, in some embodiments, the plurality of autonomous agents may include at least one of: (i) a UAV and (ii) an AUV.

[00149] Notations

[00150] Below, are notations for MARL and a TMAAC framework according to an embodiment. TMAAC, in an embodiment, adopts a transformer neural network as the policy for each agent. In such an embodiment, the transformer architecture enables the CTDME scheme because transformers can take inputs of varying lengths. [00151] Problem Definition

[00152] MARL can be modeled with the Decentralized Partially Observable Markov Decision Process (Dec-POMDP) [A31], an extension to a multi-agent manner of the Markov Decision Process (MDP). Specifically, Dec-POMDP-Com is considered, where communication is involved, and the information is exchanged among agents, which serves as additional observations. A MDP for N agents consists of a set of states .S', which describes all the configurations for the participating agents, a set of actions and a set of observations O_±

,...,O_N. Each agent i has a policy TzyO; X c/Z;i->[0,l] parameterized by The environment will generate the next state based on the state transition function T\S x c/Zj X Each agent will receive a reward from the environment as a function of state and action rpS X as well as an individual observation that is correlated with the state, opS^Ot. Each agent tries to maximize its total expected return R_L = T>t=o Y^tr^ where y is a discount factor and T is the total duration of time.

[00153] Policy Gradient (PG) Algorithms

[00154] Policy gradient methods are also popular for several RL tasks, for example, continuous control [A32], playing Atari games [A33], Instead of learning the optimal Q-function and then extracting the optimal policy as in Q-leaming, PG algorithms try to directly learn the optimal parameters 6 of the policy — that maximize the expected long-term reward, J (0) = ^s-p^.a-TLe [A] — using V₀/(0). Here the distribution of state visitation (occupancy measure), is the discounted sum of probabilities of visiting a given state when the policy n₀, is followed indefinitely. The gradient of the policy [34] can be written using the Q-function above as,

[00155] Several algorithms have been proposed based on this policy-gradient theorem, differing in how Qⁿ is computed.

[00156] Deep O Learning

[00157] A popular algorithm to solve MDP is deep Q learning [A35], which learns a stateaction value function Q(s,d), which generates scores to evaluate state-action pairs. The goal of the Deep Q Network (DQN) is to learn an optimal Q function, denoted as Q*(s,a), by iteratively minimizing the Temporal Difference (TD) error [36], which can be represented as, where y = r Q is the target Q network and is also the network that is being optimized, y is called the Q target, and s' and a' represent the next state and next action, respectively. Typically, the target Q network, Q, is a copy of the Q network, and is updated at a predefined frequency while the Q network is updated at every step. The purpose of using the target Q network is to stabilize the training process and reduce the variance of the model [A35], [00158] Deterministic Policy Gradient Algorithms (PPG) [00159] When the policy gradient is extended to deterministic policies the gradient can be written of the long-term expected reward objective as

[00160] Deep Deterministic Policy Gradient (DDPG) [A32] is the neural network version of the DPG algorithm where the actor and critic (Q^¹) functions are realized using deep neural networks. DDPG also maintains an ER buffer and target networks just like DQN [A37] and is an off-policy algorithm. DDPG samples trajectories from a replay buffer of experiences (which are stored throughout training) to train the actor and critic networks. DDPG also makes use of a target network, as in DQN. While the policy network can be trained using the gradient obtained above, the Q network is trained by minimizing the loss £(0) — y)²], where y = r s,d) is the Q target. It should be noted that there is no ‘max’ operation in the Q-target, unlike DQN, which uses TD-leaming, DDPG uses SARSA [A38] to calculate the Q-target. As mentioned above, 0' refers to target networks. The parameters 6' of the target Q function Q^ti3 s',a'') are frozen, according to an embodiment, for multiple time steps and periodically updated with the parameters 6 to ensure the stability of convergence.

[00161] Transformer Multi-Agent Actor-Critic

[00162] As discussed above, in HMAAC, according to an embodiment the interdependency between agents is relaxed by a high-level policy and it is assumed that the agents are conditionally independent given a high-level action generated by the high-level policy. However, agents in HMAAC still need to communicate with the CP to obtain high-level action, and communication links between agents and the CP are unreliable. When an agent cannot communicate with the CP for a certain time, the coordination performance of the framework will drop. Moreover, this star structure is vulnerable to single-point failure, making the coordination framework potentially unreliable. To mitigate this problem, TMAAC embodiments disclose CTDME, allowing localized communications among the agents during the execution phase, where the agents will be able to communicate with their peers. This decentralized structure brings flexibility and robustness and is not at risk of single-point failures.

[00163] One key challenge to decentralization is that the set of reachable peers might change from time to time, indicating that the proposed framework should be able to handle varied input dimensions. To achieve this, a transformer neural network architecture [39] is adopted by embodiments, which has achieved superior performance in computer vision [A40, A41], natural language processing [A42], and many other fields. Transformers can take various lengths of input, which suits the requirement for the actor model in embodiments, e.g., TMAAC. For the training of the TMAAC framework, in an embodiment, an adapted version of the DOP training mechanism [20] is adopted, to reduce gradient variance when updating policy neural networks. The difference between DOP and MADDPG is that each critic in DOP is only conditioned on local action as opposed to global actions from all agents. Denote the actor (i.e., the policy) of the ith agent as /J._L and the policy gradient for the ith actor in TMAAC during training can be written as, represents the local Experience Replay (ER) buffer for agent i, which contains tuples , and o represents the global observation. As for the critic, embodiments adopt a linear mixer to mix the local Q values, Q_i(o,a_i'), into a joint Q value, Q(o,a), because, most of the time, the environment gives a joint (i.e., scalar) feedback instead of individual ones. Decomposing/factorizing the joint Q value into individual ones is known to be the credit assignment problem, which has been widely studied in value-function-based MARL[A16, A43- A45], The mixing can be represented as, is the ith element in an observation-conditioned weight vector A(o) = (Ai.- ./v) for each individual Q value, which satisfies Z ' ^₌₁ Aj = 1 and c(o) is the bias.

[00164] To mitigate the issue of overestimation of Q values [46], embodiments combine DOP with Twin Delayed DDPG (TD3) [A47], Denote two Q neural networks with the same configurations (that is, neural network structures) but different sets of parameters such as Q( ■ ; ipy) and Q( • ;ip2), where ipj is the set of parameters. The joint Q target for calculating the

Temporal Difference (TD) error can be written as,

7 = 1,2

[00165] An embodiment uses the same set of weights and bias when mixing individual Q target values for the twin Q target networks. Regarding the loss, embodiments adopt a Huber loss instead of the generally used Mean Squared Error (MSE) loss to match the Q prediction with the Q target because the Huber loss does not over-penalize outliers. The loss function for the critic can be written as, where 8 is a predefined hyperparameter.

[00166] Below is an algorithm for TMACC Centralized Training according to an embodiment:

1. Initialize random noise TV for exploration;

2. Initialize target network update interval I;

3. Receive initial observations s;

4.

5.

6.

7.

8.

9.

10.

11.

12. Sample a batch from the replay buffer;

13. Calculate individual Q values Qt(o,cii);

14. Mix Q values to obtain joint Q value according to (A5);

15. Update the critic parameters according to (A6) and (A7);

16. Update the actor parameters according to (A4); 17. if step mode 7= 0 then

18. Update target network parameters;

[00167] During execution, each individual agent will communicate with its peers to exchange messages. Upon receiving these messages, the agent will concatenate them as input to their transformer policy. Denote the concatenated observations for the ith agent as bj The action as the result of its local actor’s inference can be represented as

[00168] Dual-Modality Execution of TMAAC

[00169] Although the training and execution process seems to be reasonable, an interesting observation was discovered. It was observed that when the communication conditions deteriorate (e g., due to the spreading of agents, obstacle blockage, etc.), the overall performance of the agents also drops and, when the communication conditions become worse than a threshold, networked execution becomes worse than decentralized execution. The reason behind this is that, given partial information, the agent might not be able to make global optimal decisions.

[00170] Given this fact, embodiments disclose a dual-modality execution approach to improve the performance of the agents when the communication condition is bad. In TMAAC, in addition to the networked actor, embodiments also train a decentralized actor with a simple neural network structure. Such an embodiment can utilize a simple neural network structure because the simple neural network will only take its local observation as input and the transformer is no longer needed. For each agent, during execution, when the communication conditions are better than the threshold, the agent adopts networked execution and enables message exchange among the agents, and when the communication conditions become worse than the threshold, the agent will adopt the decentralized execution without communication with other agents. To provide more insight, a Bernoulli communication model was considered, where two agents are connected by a probability p. By varying this probability, the overall communication condition can be controlled. According an embodiment, at each step, each agent will send a ping message to discover its neighbors. The probability p can be estimated by dividing the number of neighbors discovered by the total number of agents. If the estimated p is less than the threshold, the TMAAC framework will operate in the decentralized mode.

[00171] Below is an algorithm for TMACC Dual-Modality Networked Execution according to an embodiment: 1 . Initialize trained networked-modality models

2. Initialize trained decentralized-modality mod

3. Initialize the connected agent threshold H;

4. while not done do

5. Discover neighbors by sending ping messages;

6. Calculate the proportion of connected agents h;

7. for agent

8.

9. Broadcast observation Oj to peers;

10. Receive observations, bj, from peers;

11. Sample action from local networked policy

12. else

13. Sample action from decentralized policy aj~/z_d(aj|Oi);

14. Execute actions;

[00172] FIG. 14 is a diagram depicting an architecture 1400 of TMAAC according to an embodiment.

[00173] In operation, each agents’ 1401a-c observations 1402a-c are collected (e.g., from replay buffer(s)) and embedded by an embedding layer 1402. In turn, self-attention 1404 is performed on the inputs 1410a-c (i.e., the embedded observations). The output of performing self-attention 1404 is embeddings of the input observations 1402a-c from connected agents 1401a-c. In other words, self-attention 1404 finds the correlation among these observations 1402a-c and generates new embeddings for each observation 1402a-c. Then, the embedding corresponding to the current agent will be taken by the actor to generate the action.

[00174] To continue, the decentralized actor output layer 1411 will process its corresponding slice 1405 from the self-attention 1404 output to generate the action 1406. Meanwhile, the critic 1413 processes the whole output 1412 and the action sampled from the replay buffer to generate the individual action value 1407. The dashed lines 1408a-d indicate that the data are exchanged by communication. The dotted lines 1409a-b indicate that these steps will happen in training.

[00175] Performance Evaluation [00176] Below is an explanation of performance of embodiments along with an evaluation of embodiments in comparison to other methods.

[00177] Experiment Setup

[00178] Below, the setup for the experiments is first introduced and, then, performance is evaluated for embodiments under two scenarios: (i) a modified benchmark multi-agent simulator, Multi-Particle Environment (MPE) and (ii) a Microsoft Airsim-based SAR environment. The dual-modality execution approach is evaluated in the Airsim SAR scenario because the benchmark MPE environment is used to demonstrate the performance of single-modality model performance.

[00179] Comparison Plan

[00180] Embodiments, e.g., TMAAC, are compared against four algorithms: (i) HMAAC [Al], where each agent communicates with a high-level policy to obtain a coordination action as input to its policy, (ii) MADDPG [Al 5], where a centralized critic is trained for each agent that takes the observations and actions of all agents as input, (iii) MAPPO [Al 9], where a centralized state value function is used as the critic with PPO policies as actors, and (iv) DOP [A20], where a critic that takes global observation and local action as input is used for each agent to reduce variance in the policy gradient.

[00181] Environment Setup and Definition

[00182] The first MPE-based scenario consists of multiple particles that can be controllable agents or uncontrollable obstacles or landmarks. The frameworks are evaluated only on the spread mission, where N agents are supposed to cover N landmarks without knowing the agent- to-landmark correspondence. Only the spread scenario is used because it fits the cooperative goal better. Several modifications were performed to the original spread mission in MPE: (i) the number of agents and landmarkes were changed as a variable to evaluate the performance of the frameworks for different numbers of agents, (ii) the information about other agents was removed from the observation space, now the agents can only get that information through communication, and (iii) the size of the agents was changed to 0.05 for more precise coverage of landmarks.

[00183] In this simulator, the observation for each agent consists of the agent’s position and velocity and the landmarks’ positions. The action space is the velocity of the agent. [00184] For the second evaluation scenario, the experiments were carried out in AirSim [A48], a simulator developed using Unreal Engine for realistic visual simulation. To do experiments, the environment for the problem is formally defined. For a UAV, it was assumed that the UAV can translate and rotate freely. Each UAV was randomly spawned within the radius

10 circle centered at the center of the map. The observation of agent i consisted of its position, the distance from the four borders, and an ego-centric occupancy map. The map was discretized with a certain granularity to obtain this ego-centric grid. In this experiment, an ego-centered grid 5 x 5 was used that can define 24 area units surrounding the agent (excluding the cell in the center). The occupancy map was initialized as empty and only records the agent’s local trajectory. The action of each agent is the translation on the x and y axes. The reward is defined as a combination of multiple components, which can be written as, where A is the number of new cells explored in the occupancy map and l{_out} is ^a binary indicator function that has the value of 1 if the agent goes out of bound and 0 otherwise. Note that the reward is given jointly to the agents. A well-trained model will learn to explore more areas while avoiding overlapping explorations with other agents. At each time step, the agent had taken an action, and the environment checked if the new state reached after taking the action satisfies the conditions to calculate the reward. In the testing in the Airsim scenario, the Bernoulli communication model was adopted, where a probability p was defined to determine whether communication between two agents is successful. The value of p was varried to simulate different communication conditions.

[00185] FIGs. 15A - 15C further illustrate the evaluation setups described herein. FIG. 15A shows a simple spread implementation in an environment 1500 where particles 1501a-c represent agents and particles 1502a-c represent landmarks. FIG. 15B is a bird’s eye view map 1503 of a neighborhood environment 1505 where operation of embodiments and existing methods are simulated and evaluated, with the goal of covering the whole area with fewer steps. FIG. 15C is a visual depiction of an environment 1507 in which operation of UAVs 1509a-c is simulated and evaluated. [00186] Training and Testing Setup

[00187] For training and testing the actors were realized with a 3 -layer transformer neural network with layer normalization. The input of the transformer during training was of shape (batchsize, IV, observationdimension), and the output of transformer during training was of shape (batchsize, N, featuredimension). The second dimension was reduced of size N taking the dimension corresponding to the current agent. As a result, the final output layer received a feature tensor of size (batchsize, featuredimension) and output the actions. The critics were realized as Multi-Layer Perceptrons (MLPs) with 256 units in each layer. The linear mixer was a hypernetwork [49], where weights and bias for the individual Q values were generated from a two-layer neural network with an absolute function of activation.

[00188] The learning rates for actors and critics were 0.0001 and 0.001, respectively. Both frameworks, i.e., centralized modality and decentralized modality of TMAAC, were implemented in Pytorch [A50], a deep learning toolkit for Python. To add random exploration to actor behavior, Ornstein-Uhlenbeck noise [A32] was added to the output of the actor network. In the experiments, the continuous action space was used; however, TMAAC also works for discrete action spaces with the help of the Gumbel-Softmax trick [A23], which transforms the continuous output of the actor network to a differentiable categorical distribution. The results presented are averaged over 20 runs with different random seeds.

[00189] Rewards during Training vs. Number of Episodes

[00190] To evaluate how the different methodologies learn during training, the total reward averaged across the agents was plotted versus the number of episodes for different numbers of agents in FIGs. 16A - 16C. Specifically, FIGs. 16A - 16C are plots 1601 - 1603 respectively, which show reward results during training for two (FIG. 16A), four (FIG. 16B), and eight (FIG. 16C) agents. Each plot 1601 - 1603 plots the Rewards per Agent 1604a-c against the Number of Episodes 1605a-c for TMAAC 1606a-c, MADDPG 1607a-c, DOP 1608a-c, MAPPO 1609a-c, and HMACC 1610a-c. Plot 1601 shows training results for two agents, plot 1602 shows training results for four agents, and plot 1603 shows training results for eight agents.

[00191] It can be observed from plots 1601-1603 that embodiments, e.g., TMAAC 1606a-c, achieved superior performance in these three scenarios because the agents’ actors directly observe the other agents’ observations. It can also be seen that after adopting the twin critic, TMAAC can outperform DOP in terms of convergence rate and convergence level because the overestimation of the action values is mitigated.

[00192] Performance during Execution vs. Number of agents

[00193] To observe the performance of the agents during evaluation, the evaluation rewards, number of occupied landmarks, and the number of intra-collisions among agents were plotted against the number of deployed agents as shown in FIGs. 17A - 17C. During the evaluation, the critics were disabled, and the agents only perform inference.

[00194] Specifically, FIGs. 17A - 17C are plots 1701 - 1703 respectively, which show evaluation results obtained from the simple spread mission. Plot 1701 (FIG. 17A) plots the Total Rewards 1704a against the Number of Agents 1705b for C-TMAAC 1706a, D-TMAAC 1707a, MADDPG 1708a, HMAAC 1709a, MAPPO 1710a, and DOP 1711a. Plot 1702 (FIG. 17B) plots the Number of Occupied Landmarks 1704b against the Number of Agents 1705b for C-TMAAC 1706b, D-TMAAC 1707b, MADDPG 1708b, HMAAC 1709b, MAPPO 1710b, and DOP 1711b. Plot 1703 (FIG. 17C) plots the Number of Collisions 1704c against the Number of Agents 1705c for C-TMAAC 1706c, D-TMAAC 1707c, MADDPG 1708c, HMAAC 1709c, MAPPO 1710c, and DOP 1711c. The number of evaluation steps within an episode is set to 1000 for each of plots 1701 - 1703.

[00195] It can be observed from the plots 1701-1703 that centralized TMAAC (C-TMAAC) (1706a-c) can outperform decentralized TMAAC (D-TMAAC) (1707a-c) when there are only 2 and 4 agents. When the number of agents reaches 8, the performance of C-TMAAC is slightly worse than that of D-TMAAC because when the number of agents increases, the credit assignment problem becomes more complex and the framework easily converges to local minimums. Another observation is that the performance of MADDPG 1708a-c deteriorates the most when the number of agents increases. This is because MADDPG does not address the credit assignment problem at all. Moreover, it can be seen that DOP 171 la-c is not learning much. The main difference between DOP and TMAAC is that TMAAC adopts the transformer policy to incorporate observations from others, while the policy in DOP only conditions local observations.

[00196] Results on Airsim Simulator [00197] Described herein are the results obtained by running the different methodologies in the Airsim simulator as described above. Furthermore, the performance of the proposed TMAAC methodology in resource-constrained scenarios is also presented. To be specific, inference was ran (i.e., only forward pass of the neural network without backpropagation) on NVIDIA Jetson TX2, an energy-efficient computational board.

[00198] Exploration Speed

[00199] To see how the methods perform in terms of team exploration in different team sizes, the training search space was discretized into a 45 x 46 global occupancy map and the cumulative number of cells explored was plotted in FIGs. 18A - 18C discussed below. Since agents do not have prior knowledge about the environment or the targets, the number of grids explored can reflect the efficiency of the target search system.

[00200] FIGs. 18A - 18C are plots 1801 - 1803 respectively, which show training rewards for different numbers of agents in the Airsim Neighborhood environment. For the results in FIGs. 18A - 18C, the episode length is set to 1000. Each plot 1801 - 1803 plots the Rewards per Agent 1804a-c against the Number of Episodes 1805a-c for two, four, and eight agents, respectively, for for TMAAC 1806a-c, MADDPG 1807a-c, DOP 1808a-c, MAPPO 1809a-c, and HMAAC 1810a-c.

[00201] Results from Evaluation

[00202] FIGs. 19A - 19C are plots 1901 - 1903 respectively, which show evaluation results in an Airsim Neighborhood environment. For the results in FIGs. 19A-C, the number of steps within one episode is 1000. Plot 1901 (FIG. 19A) plots the Total Rewards 1904a against the Number of Agents 1905a for C-TMAAC 1906a, D-TMAAC 1907a, MADDPG 1908a, HMAAC 1909a, MAPPO 1910a, and DOP 1911a. Plot 1902 (FIG. 19B) plots the Area Covered (%) 1904b against the Number of Agents 1905b for C-TMAAC 1906b, D-TMAAC 1907b, MADDPG 1908b, HMAAC 1909b, MAPPO 1910b, and DOP 1911b. Plot 1903 (FIG. 19C) plots the Area Covered (%) 1904c against the Number of Steps 1905c for two 1906c, four 1908c, and eight 1910c agents centralized as well as two 1907c, four 1909c, and eight 1911c agents decentralized. Plots 1902 and 1903 use area coverage as metrics because the agents do not have any prior knowledge about the targets. The explored/covered areas can be indicative of the efficiency of finding targets. [00203] In plot 1901 (FIG. 19A), it can be observed that as the number of agents increases, the overall performance drops. This is because of the non-linearly increasing difficulty in training the multi-agent system when the number of agents increases. It can also be observed that when communication is completely deprived (i.e., D-TMAAC 1907a), for 4 and 8 agents, the performance is much worse than the full-communication scenario (i.e., C-TMAAC 1906a). This happens because agents are trained in a centralized manner, but are evaluated in a decentralized manner. In plot 1902 (FIG. 19B), the team area coverage in 1000 steps is presented. It can be observed that as the number of agents increases, the efficiency of the search mission also improves because more agents are deployed. A more detailed exploration speed is shown in plot 1903. In plot 1903 (FIG. 19C), the progress of exploration within 1000 steps, averaged after 20 runs is plotted. It can be observed that connected execution brings large benefits when the team size grows bigger, but decentralized performance also becomes worse. This is also a motivation for the dual-modality solutions described herein.

[00204] FIGs. 20A - 20C show agent trajectory samples in an Airsim Neighborhood Environment 2001-2003, respectively, for agents implementing a SAR mission. The agents’ trajectories 2004a-b, 2005a-d, and 2006a-h are overlay ed with the bird’s eye view map of the environment in FIGs. 20A-C. FIG. 20A shows the trajectories 2004a-b of two agents, FIG. 20B shows the trajectories 2005a-d of four agents, and FIG. 20C shows the trajectories 2006a-h of eight agents.

[00205] Performance under Different Communication Conditions

[00206] An example framework was evaluated in two scenarios to show how embodiments operate under different communication conditions and to identify an example threshold. In the first scenario, the probability of communication failure p from was varried 0 to 100 and the corresponding results were obtained. In the second scenario, it was assumed that each agent can only communicate with its n-hop neighbors. In particular, the direct neighbors of an agent are called its one-hop neighbors, and the neighbors’ direct neighbors, excluding the agent itself, are called the two-hop neighbors. In this way, the definition of n-hop neighbors for an agent can be inferred.

[00207] FIGs. 21A and 21B are plots 2101 and 2102, respectively, showing the performance of an example TMAAC framework when the probability of communication failure varies. Plot 2101 (FIG. 21 A) plots the Area Coverage (%) 2103a against the Communication Failure Probability (%) 2104a for two 2105a, four 2106a, and eight 2107a agents, as well as eight agents 2108a implementing a dual modality method (such as the methodology illustrated in FIG. 12. Plot 2102 (FIG. 21B) plots the Total Rewards per Step 2103b against the Communication Failure Probability (%) 2104b for two 2105b, four 2106b, and eight 2107b agents, as well as eight agents 2108b implementing a dual modality method. It can be observed from FIGs. 21A-B that a larger team size has a higher upper bound but also a lower lower bound. Further, it can be observed that 40% can be a threshold to switch between the two modalities. The eight agents with dual modality curves 2108a-b represents the performance of dual -modal execution, where if the proportion of connected agents is less than 40%, the agent will adopt the decentralized-modal execution strategy. From FIGs. 21A-B, it can be observed that the dual-modality execution brings a significant improvement to the framework, judging by the difference between results with eight agents 2107a-b and eight agents with dual modality 2108a-b.

[00208] Hardware Performance

[00209] Embodiments, e.g., TMAAC, may be deployed on drones and, as such, hardware performance may be important. To that end, performance, while implementing an embodiment, of example hardware 2201 (NVIDIA Jetson TX2 2201) shown in FIG. 22 A was tested in terms of power consumption and execution time. Results are shown in FIG. 22B and FIG. 22C. FIG. 22B is a graphical interface 2202 showing the power consumption results and FIG. 22C is a graphical interface 2203 showing inference time on the hardware. It can be observed from FIG. 22B that the GPU power consumption when running an example framework is 47mW, which is very low. FIG. 22C shows that the average inference time averaged over 10,000 rounds is 0.03 seconds, which is fast. In other words, hardware such as NVIDIA Jetson TX2 can handle the computational task of embodiments as external hardware mounted on a drone.

[00210] Computer Support

[00211] FIG. 23 illustrates a computer network or similar digital processing environment in which embodiments of the present disclosure may be implemented.

[00212] Client computer(s)/devices 50 and server computer(s) 60 provide processing, storage, and input/output devices executing application programs and the like. The client computer(s)/devices 50 can also be linked through communications network 70 to other computing devices, including other client devices/processes 50 and server computer(s) 60. The communications network 70 can be part of a remote access network, a global network (e.g., the Internet), a worldwide collection of computers, local area or wide area networks, and gateways that currently use respective protocols (TCP/IP, Bluetooth®, etc.) to communicate with one another. Other electronic device/computer network architectures are suitable.

[00213] FIG. 24 is a diagram of an example internal structure of a computer (e.g., client processor/device 50 or server computers 60) in the computer system of FIG. 23. The structure of FIG. 24 may, likewise, be included in the agents (e.g., UAVs) and control points described herein so as to allow the agents and control points to implement the embodiments described herein, e.g., method 220 and workflow 1300, amongst others. Each computer 50, 60 contains a system bus 79, where a bus is a set of hardware lines used for data transfer among the components of a computer or processing system. The system bus 79 is essentially a shared conduit that connects different elements of a computer system (e.g., processor, disk storage, memory, input/output ports, network ports, etc.) that enables the transfer of information between the elements. Attached to the system bus 79 is an VO device interface 82 for connecting various input and output devices (e.g., keyboard, mouse, displays, printers, speakers, etc.) to the computer 50, 60. A network interface 86 allows the computer to connect to various other devices attached to a network (e.g., network 70 of FIG. 23). Memory 90 provides volatile storage for computer software instructions 92A and data 94 used to implement an embodiment of the present disclosure. The computer software instructions follow the methods and operations of 220, 440, and 1300 detailed above. Disk storage 95 provides non-volatile storage for computer software instructions 92B and data 94 used to implement an embodiment of the present disclosure. The computer software instructions follow the methods and operations of 220, 440, 1200, and 1300 detailed above. A central processor unit 84 is also attached to the system bus 79 and provides for the execution of computer instructions.

[00214] In one embodiment, the processor routines 92A-B and data 94 are a computer program product (generally referenced 92), including a non-transitory computer-readable medium (e.g., a removable storage medium such as one or more DVD-ROM’s, CD-ROM’s, diskettes, tapes, etc.) that provides at least a portion of the software instructions for an embodiment. The computer program product 92 can be installed by any suitable software installation procedure, as is well known in the art. In another embodiment, at least a portion of the software instructions may also be downloaded over a cable communication and/or wireless connection. In other embodiments, the invention programs are a computer program propagated signal product embodied on a propagated signal on a propagation medium (e.g., a radio wave, an infrared wave, a laser wave, a sound wave, or an electrical wave propagated over a global network such as the Internet, or other network(s)). Such carrier medium or signals may be employed to provide at least a portion of the software instructions for the present invention routine s/ program 92 A-B .

[00215] Embodiments or aspects thereof may be implemented in the form of hardware, firmware, or software. If implemented in software, the software may be stored on any nontransient computer readable medium that is configured to enable a processor to load the software or subsets of instructions thereof. The processor then executes the instructions and is configured to operate or cause an apparatus to operate in a manner as described herein.

[00216] Further, firmware, software, routines, or instructions may be described herein as performing certain actions and/or functions of the data processors. However, it should be appreciated that such descriptions contained herein are merely for convenience and that such actions in fact result from computing devices, processors, controllers, or other devices executing the firmware, software, routines, instructions, etc.

[00217] It should be understood that the flow diagrams, block diagrams, and network diagrams may include more or fewer elements, be arranged differently, or be represented differently. But it further should be understood that certain implementations may dictate the block and network diagrams and the number of block and network diagrams illustrating the execution of the embodiments be implemented in a particular way.

[00218] Accordingly, further embodiments may also be implemented in a variety of computer architectures, physical, virtual, cloud computers, and/or some combination thereof, and thus, the data processors described herein are intended for purposes of illustration only and not as a limitation of the embodiments.

[00219] The teachings of all patents, published applications and references cited herein are incorporated by reference in their entirety. [00220] While example embodiments have been particularly shown and described, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the scope of the embodiments encompassed by the appended claims.

[00221] References

[00222] [1] V. Sadhu, T. Misu, and D. Pompili, “Deep multi-task learning for anomalous driving detection using can bus scalar sensor data,” in 2019 IEEE/ RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 2038-2043, IEEE, 2019.

[00223] [2] V. Sadhu, S. Zonouz, and D. Pompili, “On-board deep-leaming-based unmanned aerial vehicle fault cause detection and identification,” in 2020 IEEE International Conference on Robotics and Automation (ICRAf pp. 5255-5261, IEEE, 2020.

[00224] [3] V. Sadhu, C. Sun, A. Karimian, R. Tron, and D. Pompili, “Aerial- DeepSearch:

Distributed Multi-Agent Deep Reinforcement Learning for Search Missions,” in IEEE International Conference on Mobile Ad Hoc and Sensor Systems (MASS), (Delhi NCR), pp. 1- 9, dec 2020.

[00225] [4] W. Li, H. Chen, B. Jin, W. Tan, H. Zha, and X. Wang, “Multi-agent path finding with prioritized communication learning,” in 2022 Inter- national Conference on Robotics and Automation (ICRAf pp. 10695- 10701, 2022.

[00226] [5] C. Igoe, R. Ghods, and J. Schneider, “Multi-agent active search: A reinforcement learning approach,” IEEE Robotics and Automation Letters, vol. 7, no. 2, pp. 754-761, 2021.

[00227] [6] C. Zhang and V. Lesser, “Coordinating multi-agent reinforcement learning with limited communication,” in Proceedings of the 2013 international conference on Autonomous agents and multi-agent systems, pp. 1101-1108, 2013.

[00228] [7] J. Foerster, I. A. Assael, N. De Freitas, and S. Whiteson, “Learning to communicate with deep multi-agent reinforcement learning,” m Advances in neural information processing systems, pp. 2137-2145, 2016.

[00229] [8] B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y Areas,

“Communication-efficient learning of deep networks from decentralized data,” in Artificial intelligence and statistics, pp. 1273-1282, PMLR, 2017. [00230] [9] C. Sun, T. Jiang, S. Zonouz, and D. Pompili, “Fed2kd: Heterogeneous federated learning for pandemic risk assessment via two-way knowledge distillation,” in 2022 17th Wireless On-Demand Network Systems and Services Conference (WONS), pp. 1-8, IEEE, 2022.

[00231] [10] N. Basilico and F. Amigoni, “Exploration strategies based on multi- criteria decision making for searching environments in rescue operations,” Autonomous Robots, vol. 31, no. 4, pp. 401-417, 2011.

[00232] [11] Y. Mei, Y.-H. Lu, C. G. Lee, and Y. C. Hu, “Energy-efficient mobile robot exploration,” in Proceedings 2006 IEEE International Conference on Robotics and Automation, 2006. ICRA 2006., pp. 505- 511, IEEE, 2006.

[00233] [12] D. C. Schedl, I. Kurmi, and O. Bimber, “An autonomous drone for search and rescue in forests using airborne optical sectioning,” Science Robotics, vol. 6, no. 55, p. eabgl l88, 2021.

[00234] [13] E. Lygouras, N. Santavas, A. Taitzoglou, K. Tarchanidis, A. Mitropoulos, and A.

Gasteratos, “Unsupervised human detection with an embedded vision system on a fully autonomous uav for search and rescue operations,” Sensors, vol. 19, no. 16, p. 3542, 2019.

[00235] [14] E. T. Alotaibi, S. S. Alqefari, and A. Koubaa, “Lsar: Multi -uav collaboration for search and rescue missions,” IEEE Access, vol. 7, pp. 55817-55832, 2019.

[00236] [15] G. A. Cardona and J. M. Calderon, “Robot swarm navigation and victim detection using rendezvous consensus in search and rescue operations,” Applied Sciences, vol.

9, no. 8, p. 1702, 2019.

[00237] [16] J. P. Queralta, J. Taipalmaa, B. C. Pullinen, V. K. Sarker, T. N. Gia, H.

Tenhunen, M. Gabbouj, J. Raitoharju, and T. Westerlund, “Collaborative multi-robot systems for search and rescue: Coordination and perception,” arXiv preprint arXiv:2008.12610, 2020.

[00238] [17] R. Lowe, Y. Wu, A. Tamar, J. Harb, P. Abbeel, and I. Mordatch, “Multi-Agent

Actor-Critic for Mixed Cooperative-Competitive Environments,” in Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS’ 17, (Red Hook, NY, USA), pp. 6382-6393, Curran Associates Inc., 2017.

[00239] [18] S. Shah, D. Dey, C. Lovett, and A. Kapoor, “Airsim: High-fidelity visual and physical simulation for autonomous vehicles,” in FSR, 2017. [00240] [19] R. J. Williams, “Simple statistical gradient-following algorithms for connectionist reinforcement learning,” Machine learning, vol. 8, no. 3, pp. 229-256, 1992.

[00241] [20] J. Delmerico, E. Mueggler, J. Nitsch, and D. Scaramuzza, “Active autonomous aerial exploration for ground robot path planning,” IEEE Robotics and Automation Letters, vol. 2, no. 2, pp. 664-671, 2017.

[00242] [21] C. Sampedro, A. Rodriguez-Ramos, H. Bavle, A. Carrio, P. de la Puente, and P.

Campoy, “A fully-autonomous aerial robot for search and rescue applications in indoor environments using learning-based techniques,” Journal of Intelligent & Robotic Systems, vol. 95, no. 2, pp. 601-627, 2019.

[00243] [22] A. Das, T. Gervet, J. Romoff, D. Batra, D. Parikh, M. Rabbat, and J. Pineau,

“Tarmac: Targeted multi-agent communication,” in International Conference on Machine Learning, pp. 1538-1546, 2019.

[00244] [23] D. Han, W. Boehmer, M. Wooldridge, and A. Rogers, “Multi-agent hierarchical reinforcement learning with dynamic termination,” in Pacific Rim International Conference on Artificial Intelligence , pp. 80- 92, Springer, 2019.

[00245] [24] S. Shah, D. Dey, C. Lovett, and A. Kapoor, “Airsim: High-fidelity visual and physical simulation for autonomous vehicles,” in Pie Id and service robotics, pp. 621-635, Springer, 2018.

[00246] [25] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z.

Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala, “Pytorch: An imperative style, high- performance deep learning library,” in Advances in Neural Information Processing Systems 32, pp. 8024-8035, Curran Associates, Inc., 2019.

[00247] [26] T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D.

Wierstra, “Continuous control with deep reinforcement learning,” in International Conference on Learning Representations (I CLR) , 2015.

[00248] [27] A. Bochkovskiy, C.-Y. Wang, and H.-Y. M. Liao, “Yolov4: Optimal speed and accuracy of object detection,” arXiv preprint arXiv:2004.10934, 2020.

[00249] [28] E. Jang, S. Gu, and B. Poole, “Categorical reparameterization with gumbel- softmax,” arXiv preprint arXiv: 1611.01144, 2016. [00250] [29] Y. Zhang, V. Paxson, S. Shenker, and L. Breslau, “The stationarity of internet path properties: Routing, loss, and throughput,” tech, rep., Citeseer, 2000.

[00251] [1 A] C. Sun, S. Huang, and D. Pompili, “Hmaac: Hierarchical multi-agent actor-critic for aerial search with explicit coordination modeling,” in 2023 IEEE International Conference on Robotics and Automation (ICRA), pp. 7728-7734, IEEE, 2023.

[00252] [2A] V. Sadhu, T. Misu, and D. Pompili, “Deep multi-task learning for anomalous driving detection using can bus scalar sensor data,” in 2019 IEEE RS J International Conference on Intelligent Robots and Systems (IROS), pp. 2038-2043, IEEE, 2019.

[00253] [3 A] V. Sadhu, G. Salles-Loustau, D. Pompili, S. Zonouz, and V. Sritapan, “Argus:

Smartphone-enabled human cooperation via multi-agent reinforcement learning for disaster situational awareness,” in 2016 IEEE International Conference on Autonomic Computing (ICAC), pp. 251-256, IEEE, 2016.

[00254] [4A] V. Sadhu, C. Sun, A. Karimian, R. Tron, and D. Pompili, “Aerial-deepsearch:

Distributed multi-agent deep reinforcement learning for search missions,” in 2020 IEEE 17th International Conference on Mobile Ad Hoc and Sensor Systems (MASS), pp. 165-173, IEEE, 2020.

[00255] [5 A] S. Shalev-Shwartz, S. Shammah, and A. Shashua, “Safe, multi-agent, reinforcement learning for autonomous driving,” arXiv preprint arXiv: 1610.03295, 2016.

[00256] [6A] P. Palanisamy, “Multi-agent connected autonomous driving using deep reinforcement learning,” in 2020 International Joint Conference on Neural Netw orks (I.JC.NN), pp. 1-7, IEEE, 2020.

[00257] [7A] I. Wang, T. Shi, Y. Wu, L. Miranda-Moreno, and L. Sun, “Multi-agent graph reinforcement learning for connected automated driving,” in Proceedings of the 37 th International Conference on Machine Learning (ICML), pp. 1-6, 2020.

[00258] [8A] J. A. Calvo and I. Dusparic, “Heterogeneous multi-agent deep reinforcement learning for traffic lights control.,” in A ICS, pp. 2-13, 2018.

[00259] [9A] H. Zhang, S. Feng, C. Liu, Y. Ding, Y. Zhu, Z. Zhou, W. Zhang, Y. Yu, H. Jin, and Z. Li, “Cityflow: A multi-agent reinforcement learning environment for large scale city traffic scenario,” in The worldwide web conference, pp. 3620-3624, 2019. [00260] [10A] T. Wu, P. Zhou, K. Liu, Y. Yuan, X. Wang, H. Huang, and D O. Wu, “Multiagent deep reinforcement learning for urban traffic light control in vehicular networks,” IEEE Transactions on Vehicular Technology, vol. 69, no. 8, pp. 8243-8256, 2020.

[00261] [11 A] F. S. Melo, M. T. Spaan, and S. J. Witwicki, “Querypomdp: Pomdp-based communication in multiagent systems,” in European Workshop on Multi-Agent Systems, pp. 189-204, Springer, 2011.

[00262] [12A] C. Zhang and V. Lesser, “Coordinating multi-agent reinforcement learning with limited communication,” in Proceedings of the 2013 international conference on Autonomous agents and multi-agent systems, pp. 1101-1108, 2013.

[00263] [13A] T. Kasai, H. Tenmoto, and A. Kamiya, “Learning of communication codes in multi-agent reinforcement learning problem,” in 2008 IEEE Conference on Soft Computing in Industrial Applications, pp. 1-6, IEEE, 2008.

[00264] [14A] R. Azzam, I. Boiko, and Y. Zweiri, “Swarm cooperative navigation using centralized training and decentralized execution,” Drones, vol. 7, no. 3, p. 193, 2023.

[00265] [15A] R. Lowe, Y. Wu, A. Tamar, J. Harb, P. Abbeel, and I. Mordatch, “Multi-Agent

Actor-Critic for Mixed Cooperative-Competitive Environments,” in Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS’ 17, (Red Hook, NY, USA), pp. 6382-6393, Curran Associates Inc., 2017.

[00266] [16A] K. Son, D. Kim, W. J. Kang, D. E. Hostallero, and Y. Yi, “Qtran: Learning to factorize with transformation for cooperative multi -agent reinforcement learning,” in International conference on machine learning, pp. 5887-5896, PMLR, 2019.

[00267] [17A] I. Mordatch and P. Abbeel, “Emergence of grounded compositional language in multi-agent populations,” in Proceedings of the AAAI conference on artificial intelligence, vol. 32, 2018.

[00268] [18A] S. Shah, D. Dey, C. Lovett, and A. Kapoor, “Airsim: High-fidelity visual and physical simulation for autonomous vehicles,” in FSR, 2017.

[00269] [19A] C. Yu, A. Velu, E. Vinitsky, J. Gao, Y. Wang, A. Bayen, and Y. Wu, “The surprising effectiveness of ppo in cooperative multi-agent games,” Advances in Neural Information Processing Systems, vol. 35, pp. 24611-24624, 2022. [00270] [20A] Y. Wang, B. Han, T. Wang, H. Dong, and C. Zhang, “Dop: Off-policy multiagent decomposed policy gradients,” in International conference on learning representations, 2020.

[00271] [21A] A. Das, T. Gervet, J. Romoff, D. Batra, D. Parikh, M. Rabbat, and J. Pineau,

“Tarmac: Targeted multi-agent communication,” in International Conference on Machine Learning, pp. 1538-1546, 2019.

[00272] [22A] A. Singh, T. Jain, and S. Sukhbaatar, “Individualized controlled continuous communication model for multiagent cooperative and competitive tasks,” in International Conference on Learning Representations, 2019.

[00273] [23 A] J. Jiang, C. Dun, T. Huang, and Z. Lu, “Graph convolutional reinforcement learning,” in International Conference on Learning Representations, 2020

[00274] [24A] J. Foerster, I. A. Assael, N. De Freitas, and S. Whiteson, “Learning to communicate with deep multi-agent reinforcement learning,” in Advances in neural information processing systems, pp. 2137-2145, 2016.

[00275] [25 A] J. Sheng, X. Wang, B. Jin, J. Yan, W. Li, T.-H. Chang, J. Wang, and H. Zha,

“Learning structured communication for multi-agent reinforcement learning,” arXiv preprint arXiv: 2002.04235, 2020.

[00276] [26A] M. O. Karabag, C. Neary, and U. Topcu, “Planning not to talk: Multiagent systems that are robust to communication loss,” arXiv preprint arXiv: 2201.06619, 2022.

[00277] [27A] K. Zhang, Z. Yang, H. Liu, T. Zhang, and T. Ba§ar, “Fully decentralized multiagent reinforcement learning with networked agents,” arXiv preprint arXiv: 1802.08757, 2018.

[00278] [28A] P.-L. Bacon, J. Harb, and D. Precup, “The option-critic architecture,” in

Proceedings of the AAAI Conference on Artificial Intelligence, vol. 31, 2017.

[00279] [29A] A. Levy, G. D. Konidaris, R. W. Platt, and K. Saenko, “Learning multi-level hierarchies with hindsight,” in ICLR, 2019.

[00280] [30A] D. Han, W. Boehmer, M. Wooldridge, and A. Rogers, “Multi-agent hierarchical reinforcement learning with dynamic termination,” in Pacific Rim International Conference on Artificial Intelligence, pp. 80-92, Springer, 2019.

[00281] [31A] F. A. Oliehoek, C. Amato, et al. , A concise introduction to decentralized

POMDPs, vol. 1. Springer, 2016. [00282] [32A] T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and

D. Wierstra, “Continuous control with deep reinforcement learning,” in International Conference on Learning Representations (ICLR), 2015.

[00283] [33A] V. Mnih, A. P. Badia, M. Mirza, A. Graves, T. Harley, T. P. Lillicrap, D.

Silver, and K. Kavukcuoglu, “Asynchronous Methods for Deep Reinforcement Learning,” in Proceedings of the 33rd International Conference on International Conference on Machine Learning - Volume 48, ICML’ 16, pp. 1928-1937, JMLR.org, 2016.

[00284] [34A] R. S. Sutton, D. McAllester, S. Singh, and Y. Mansour, “Policy Gradient

Methods for Reinforcement Learning with Function Approximation,” in Proceedings of the 12th International Conference on Neural Information Processing Systems, NIPS’99, (Cambridge, MA, USA), pp. 1057-1063, MIT Press, 1999.

[00285] [35A] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare,

A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, et al., “Human-level control through deep reinforcement learning,” nature, vol. 518, no. 7540, pp. 529-533, 2015.

[00286] [36A] R. S. Sutton and A. G. Barto, Reinforcement learning: An introduction. MIT press, 2018.

[00287] [37A] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare,

A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, S. Petersen, C. Beattie, A. Sadik, I.

Antonoglou, H. King, D. Kumaran, D. Wierstra, S. Legg, and D. Hassabis, “Human-level control through deep reinforcement learning,” Nature, vol. 518, no. 7540, pp. 529-533, 2015.

[00288] [38 A] R. S. Sutton and A. G. Barto, Reinforcement learning : an introduction. MIT

Press, 1998.

[00289] [39A] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, t.

Kaiser, and I. Polosukhin, “Attention is all you need,” Advances in neural information processing systems, vol. 30, 2017.

[00290] [40A] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenbom, X. Zhai, T.

Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Geliy, et al., “An image is worth 16x16 words: Transformers for image recognition at scale,” arXiv preprint arXiv:2010.I1929, 2020. [00291] [41 A] Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo, “Swin transformer: Hierarchical vision transformer using shifted windows,” in Proceedings of the IEEE/CVF international conference on computer vision, pp. 10012-10022, 2021.

[00292] [42A] H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N.

Bashlykov, S. Batra, P. Bhargava, S. Bhosale, et al., “Llama 2: Open foundation and fine-tuned chat models,” arXiv preprint arXiv:2307.09288, 2023.

[00293] [43 A] T. Rashid, M. Samvelyan, C. S. De Witt, G. Farquhar, J. Foerster, and S.

Whiteson, “Monotonic value function factorisation for deep multi-agent reinforcement learning,” The Journal of Machine Learning Research, vol. 21, no. 1, pp. 7234-7284, 2020.

[00294] [44A] T. Rashid, G. Farquhar, B. Peng, and S. Whiteson, “Weighted qmix:

Expanding monotonic value function factorisation for deep multi-agent reinforcement learning,” Advances in neural information processing systems, vol. 33, pp. 10199-10210, 2020.

[00295] [45A] Y. Yang, J. Hao, B. Liao, K. Shao, G. Chen, W. Liu, and H. Tang, “Qatten: A general framework for cooperative multiagent reinforcement learning,” arXiv preprint arXiv: 2002.03939, 2020.

[00296] [46A] S. Fujimoto, H. Hoof, and D. Meger, “Addressing function approximation error in actor-critic methods,” in International conference on machine learning, pp. 1587-1596, PMLR, 2018.

[00297] [47A] J. Ackermann, V. Gabler, T. Osa, and M. Sugiyama, “Reducing overestimation bias in multi-agent domains using double centralized critics,” arXiv preprint arXiv: 1910.01465, 2019.

[00298] [48A] S. Shah, D. Dey, C. Lovett, and A. Kapoor, “Airsim: High-fidelity visual and physical simulation for autonomous vehicles,” in Field and service robotics, pp. 621-635, Springer, 2018.

[00299] [49A] D. Ha, A. M. Dai, and Q. V. Le, “Hypernetworks,” in International Conference on Learning Representations, 2017.

[00300] [50A] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z.

Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. DeVito, M. Raison, A.

Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala, “Pytorch: An imperative style, high-performance deep learning library,” in Advances in Neural Information Processing Systems 32, pp. 8024-8035, Curran Associates, Inc., 2019.

Claims

CLAIMS What is claimed is:

1. A computer-implemented method for operating a plurality of autonomous agents in a real-world environment, the method comprising: by each agent of the plurality of autonomous agents: determining if a communication condition is above or below a threshold; responsive to determining the communication condition is above the threshold, determining a real-world action in the environment based on a respective indication of properties of the environment and a coordination communication; responsive to determining the communication condition is below the threshold, determining a real-world action in the environment based on the respective indication of properties of the environment; and performing the determined real-world action in the environment.

2. The method of Claim 1, wherein determining if a communication condition is above or below a threshold comprises, by a given agent of the plurality of autonomous agents: analyzing communication quality between the given agent and a subset of the plurality of autonomous agents and determining the communication condition is above or below the threshold based on the analyzing.

3. The method of Claim 2, wherein responsive to determining the communication condition is above the threshold, the method further comprises: receiving local observations from agents of the subset of the plurality of autonomous agents, where the received local observations comprise the coordination communication.

4. The method of Claim 1, wherein each agent of the plurality of autonomous agents comprises a transformer trained in accordance with a global control policy and, responsive to determining the communication condition is above the threshold, each agent is configured to determine the real-world action by processing the respective indication of properties and the coordination communication with the transformer.

5. The method of Claim 1, wherein each agent of the plurality of autonomous agents comprises a neural network trained in accordance with a global control policy and, responsive to determining the communication condition is below the threshold each agent is configured to determine the real-world action by processing the respective indication of properties with the neural network.

6. The method of Claim 1, wherein: the respective indication of properties of the environment comprises a local observation by the agent.

7. The method of Claim 1, wherein determining if the communication condition is above or below the threshold comprises: receiving the coordination communication from a control agent and responsively determining the communication condition is above the threshold; and identifying a failure to receive the coordination communication from the control agent and responsively determining the communication condition is below the threshold.

8. The method of Claim 1, further comprising: by a control agent: receiving respective indications of properties of the real-world environment from one or more agents of the plurality of autonomous agents; based on the respective indications received and a global control policy, determining the coordination communication for each agent of the plurality of autonomous agents; and sending the determined coordination communication to each agent of the plurality of autonomous agents.

9. The method of Claim 8, further comprising: updating the global control policy.

10. The method of Claim 9, wherein updating the global control policy comprises: by a given agent of the plurality of autonomous agents, determining quality of the determined real-world action; and by the control agent, updating the global control policy using an indication of the determined quality.

11. The method of Claim 10, wherein the given agent includes a critic neural network configured to determine the quality of the determined real-world action responsive to one or more inputs.

12. The method of Claim 11, wherein the one or more inputs include at least one of: a location of the given agent and an image of the environment.

13. The method of Claim 1, wherein a given agent of the plurality of autonomous agents includes an actor neural network and where, determining the real-world action includes: responsive to determining the communication condition is above the threshold, determining the real-world action in the environment by providing the respective indication of properties of the environment and the coordination communication as inputs to the actor neural network, where the actor neural network is configured to determine the real-world action responsive to the inputs; and responsive to determining the communication condition is below the threshold, determining the real-world action in the environment by providing the respective indication of properties of the environment and a prior coordination communication as inputs to the actor neural network, where the actor neural network is configured to determine the real-world action responsive to the inputs.

14. The method of Claim 1, further comprising: iteratively performing the determining if a communication condition is above or below a threshold, the determining a real-world action, and the performing until an objective is achieved.

15. The method of Claim 14, wherein the objective is a search operation of the environment.

16. The method of Claim 1, wherein the plurality of autonomous agents includes at least one of: an unmanned aerial vehicle (UAV) and an autonomous underwater vehicle (AUV).

17. A system for operating autonomous agents in a real-world environment, the system comprising: a plurality of autonomous agents, wherein each agent of the plurality of autonomous agents is configured to: determine if a communication condition is above or below a threshold; responsive to determining the communication condition is above the threshold, determine a real-world action in the environment based on a respective indication of properties of the environment and a coordination communication; responsive to determining the communication condition is below the threshold, determine a real-world action in the environment based on the respective indication of properties of the environment; and perform the determined real-world action in the environment.

18. The system of Claim 17 wherein, in determining if the communication condition is above or below a threshold, a given agent of the plurality of autonomous agents is configured to: analyze communication quality between the given agent and a subset of the plurality of autonomous agents and determine the communication condition is above or below the threshold based on the analyzing.

19. The system of Claim 18, wherein responsive to determining the communication condition is above the threshold, the given agent is further configured to: receive local observations from agents of the subset of the plurality of autonomous agents, where the received local observations comprise the coordination communication.

20. A non-transitory computer program product comprising a computer-readable medium with computer code instructions stored thereon, the computer code instructions being configured, when executed by a processor, to cause an autonomous agent associated with the processor to: determine if a communication condition is above or below a threshold; responsive to determining the communication condition is above the threshold, determine a real-world action in a real-world environment based on an indication of properties of the environment and a coordination communication; responsive to determining the communication condition is below the threshold, determine a real-world action in the real-world environment based on the indication of properties of the environment; and perform the determined real-world action in the environment.