CN117236416A - A large language model interaction method and device - Google Patents

Abstract

The invention discloses a large language model interaction method and device. The method proposes a new planner-coordinator-executor large language model interaction framework, in which the large language model serves as the planner and the intelligent agent serves as the executor. , the new coordinator can determine when to request communication with the planner, and convert the current observation data of the executor into a text string in the form of natural language that the planner can understand. The coordinator can use reinforcement learning based on invalid communication penalties Pre-training to implement optimal communication strategies. By implementing the optimal communication strategy, the present invention can significantly reduce the number of communications with the planner after being formally deployed to the test environment. At the same time, the coordinator can reduce dependence on the planner in scenarios where the planner is prone to errors, and in the face of emergencies Turning to the planner for help when the situation arises improves the safety of the executor and the success rate of the task.

Description

A large language model interaction method and device

Technical field

The invention relates to the fields of reinforcement learning and natural language processing, and in particular to a large language model interaction method and device.

Background technique

A large language model is an artificial intelligence model designed to understand and generate human language. They are trained on large amounts of text data, containing billions of parameters, allowing them to learn complex patterns in language data. Large language models have achieved great success in recent years, and the ChatGPT released by OpenAI has attracted widespread attention from all walks of life.

Some studies use the general knowledge and reasoning capabilities of large language models to assist the decision-making and planning of agents. However, how to reasonably and efficiently communicate with large language models during the completion of tasks by agents is still an unsolved open topic. For example, although the SayCan method proposed by the Google team uses a large language model to solve the problem of robotic arm motion control better than traditional methods, this method requires the agent to communicate with the large language model at every moment, because the large language model contains With billions of parameters, each time the agent communicates with the large language model, it will spend a lot of time and computing resources. If the agent communicates with the large language model at every step in the execution of the task, the overhead will be very high. In addition, when the agent encounters an unexpected situation, if it does not turn to the large language model in time, it may lead to security problems; for example, when the agent performs the task of "go to the next room to get a glass of water and return", a gust of wind blows , the door is closed accidentally. If the robot continues to perform forward motion, it will cause damage to itself and the door. When a large language model makes an error, if there is no good error correction mechanism, it will also cause the task to be unable to be completed, and even security issues will arise.

Common intelligent agent systems based on large language model guidance divide the entire control process into a planner based on a large language model that specifically provides high-level instructions at the logical level and an executor based on pre-training or presets that specifically handles low-level motion control. Two parts. On the basis of the original framework, the present invention adds a coordinator as an intermediary between the planner and the executor to determine whether communication with the planner is needed. The coordinator uses reinforcement learning to maximize the cumulative communication reward—that is, to enable the agent to complete the task through the minimum number of communications to solve the communication problem between the agent (executor) and the large language model (planner) mentioned above.

Contents of the invention

The purpose of the present invention is to provide a large language model interaction method and device to address the deficiencies in the existing technology.

The object of the present invention is achieved through the following technical solutions: The first aspect of the embodiment of the present invention provides a large language model interaction method, which includes the following steps:

(1) After the executor interacts with the environment, it sends the currently collected observation data to the coordinator;

(2) The coordinator uses the optimal communication strategy based on the received observation data to determine whether it needs to communicate with the planner. If the coordinator needs to communicate with the planner, the coordinator converts the observation data into standard form data and converts the standard form into The data is sent to the planner; if the coordinator does not need to communicate with the planner, the coordinator resends the current high-level instructions to the executor and jumps to step (4);

(3) After receiving the standard form data, the planner generates new high-level instructions corresponding to the execution actions based on the standard form data and sends them to the executor;

(4) After the executor receives the high-level instruction, it calls the underlying control logic corresponding to the high-level instruction based on the current observation data to perform the corresponding execution action.

Further, the observation data includes sensor data, text data and image data.

Further, the optimal communication strategy specifically includes: the coordinator decides whether to communicate with the planner at each moment to enable the executor to complete the task through the minimum number of communications. This process is defined as a reinforcement learning process, and the reinforcement learning process specifically Including: corresponding to the state at each moment, the coordinator has two different actions-insisting on executing the current plan or requesting a new plan from the planner. Based on the rewards given by the environment, an invalid communication penalty is introduced as a cumulative communication reward; where the state Observation data collected for the executor; the coordinator's optimal communication strategy is obtained by maximizing the cumulative communication reward.

Further, the expression of the cumulative communication reward is:

;

in, is the accumulated communication reward at time t,/> is the reward given by the environment at time t,/> is an indicator function, is the action of the coordinator at time t, ask means that the coordinator needs to communicate with the planner, not ask means that the coordinator does not need to communicate with the planner,/> High-level instructions returned by the planner at time t,/> is the reward discount coefficient,/> is the penalty coefficient for invalid communication.

Further, the training methods for maximizing cumulative communication rewards include proximal policy optimization methods, maximum entropy actor-critic methods, deep Q networks and dominant actor-critic methods.

Further, the executor includes multiple intelligent agents; the planner includes a large language model and users collaborate with the large speech model.

Further, the high-level instructions have a one-to-one correspondence with the underlying control logic, the underlying control logic has a one-to-one correspondence with the execution actions, and the high-level instructions have a one-to-one correspondence with the execution actions.

The second aspect of the embodiment of the present invention provides a large language model interaction device for implementing the above large language model interaction method, including:

The planner module is used to generate new high-level instructions corresponding to execution actions based on the received standard form data;

The coordinator module is used to use the optimal communication strategy based on the observation data to determine whether communication with the planner is needed. If the coordinator needs to communicate with the planner, the coordinator converts the observation data into standard form data and sends the standard form data. to the planner; if the coordinator does not need to communicate with the planner, the coordinator resends the current high-level instructions to the executor; and

The executor module is used to collect observation data, and after receiving high-level instructions, call the underlying control logic corresponding to the high-level instructions to perform the corresponding execution actions.

A third aspect of the embodiment of the present invention provides an electronic device, including a memory and a processor, the memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the Program data to implement the large language model interaction method described above.

The fourth aspect of the embodiment of the present invention provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the above-mentioned large language model interaction method is implemented.

The beneficial effect of the present invention is that by proposing a new interaction framework and adding a new coordinator, the new coordinator is used as an intermediary to connect the large language model and the intelligent agent, thereby effectively reducing the time caused by communication. cost and computing resource cost; at the same time, the introduction of the coordinator will help the agent to turn to large language models in a timely manner when facing emergencies, and reduce the dependence on large language models in scenarios where large language models are prone to errors. , improving the safety and mission success rate of the agent.

Description of drawings

Figure 1 is a schematic flow chart of reinforcement learning;

Figure 2 is a schematic diagram of the overall flow of the large language model interaction method;

Figure 3 is a communication schematic diagram of an embodiment of applying a large language model interaction method in a single-agent system;

Figure 4 is a communication schematic diagram of an embodiment of applying a large language model interaction method in a multi-agent system;

Figure 5 is a communication schematic diagram of another embodiment of applying a large language model interaction method in a multi-agent system;

Figure 6 is a communication schematic diagram of another embodiment of applying a large language model interaction method in a multi-agent system;

Figure 7 is a schematic diagram of the experimental verification results of the effectiveness of the large language model interaction method in the MiniGrid environment that simulates a smart factory; (a) in Figure 7 is the task success rate of the large language model interaction method in the MiniGrid environment that simulates a smart factory. A schematic diagram of the experimental verification results, Figure 7 (b) A schematic diagram of the experimental verification results of the communication cost of the large language model interaction method in the MiniGrid environment simulating a smart factory;

Figure 8 is a structural schematic diagram of a large language model interaction device.

Detailed ways

Exemplary embodiments will be described in detail herein, examples of which are illustrated in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings refer to the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the invention. Rather, they are merely examples of apparatus and methods consistent with aspects of the invention as detailed in the appended claims.

The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. As used in this disclosure and the appended claims, the singular forms "a," "the" and "the" are intended to include the plural forms as well, unless the context clearly dictates otherwise. It will also be understood that the term "and/or" as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

It should be understood that although the terms first, second, third, etc. may be used in the present invention to describe various information, the information should not be limited to these terms. These terms are only used to distinguish information of the same type from each other. For example, without departing from the scope of the present invention, the first information may also be called second information, and similarly, the second information may also be called first information. Depending on the context, the word "if" as used herein may be interpreted as "when" or "when" or "in response to determining."

The present invention will be described in detail below with reference to the accompanying drawings. Features in the following embodiments and implementations may be combined with each other without conflict.

Referring to Figure 2, the large language model interaction method of the present invention specifically includes the following steps:

(1) After the executor interacts with the environment, it sends the currently collected observation data to the coordinator.

Further, observation data includes but is not limited to: sensor data, text data, image data, etc.

It should be noted that the executors include multiple intelligent agents, such as robots, traffic lights, etc., which can perform certain actions based on pre-trained or preset underlying motion control to complete the required tasks. It should be understood that when performing tasks in a certain system, the executor can be a single agent, such as a robot or a traffic light; the executor can also be multiple agents, such as multiple traffic lights, roadside lights, etc. Surveillance and self-driving cars.

Specifically, when the performer is performing a task, for example, when the robot performs the task of passing through a door, the robot needs to collect the current image through various sensors and the image acquisition device carried by the robot during the movement, that is, after interacting with the environment. Observation data, the robot will send the collected observation data to the coordinator, and then the coordinator can judge based on these observation data whether the robot will directly pass through the door or open the door and then pass.

(2) The coordinator uses the optimal communication strategy based on the received observation data to determine whether it needs to communicate with the planner. If the coordinator needs to communicate with the planner, the coordinator converts the observation data into standard form data and converts the standard form into The data is sent to the planner; if the coordinator does not need to communicate with the planner, the coordinator resends the current high-level instructions to the executor and jumps to step (4).

It should be noted that standard form data is a natural language that planners can understand. Observation data can be converted into standard form data, that is, a natural language that planners can understand, through the YOLO target detection algorithm or CLIP multi-modal algorithm. Among them, the standard form data can be a text string or a picture. For example, the large language model-GPT4 can support both picture input and text input.

In this embodiment, the optimal communication strategy specifically includes: the coordinator decides whether to communicate with the planner at each moment to enable the executor to complete the task through the minimum number of communications. This process is defined as a reinforcement learning process. The reinforcement learning process specifically Including: corresponding to the state at each moment, the coordinator has two different actions-insisting on executing the current plan or requesting a new plan from the planner. Based on the rewards given by the environment, an invalid communication penalty is introduced as a cumulative communication reward; where the state Observation data collected for the executor; the coordinator's optimal communication strategy is obtained by maximizing the cumulative communication reward.

It should be understood that reinforcement learning is a subfield of machine learning, and the problem discussed is how an agent acts in the environment to maximize the cumulative reward. Reinforcement learning mainly consists of an agent, environment, state, action, and reward. As shown in Figure 1, after the agent performs an action, the environment will transition to a new state, and the new state environment will give Reward, and then the agent performs new actions according to the preset strategy based on the new state and the reward fed back by the environment.

Furthermore, the reward given by the environment represents the completion of the task. On this basis, the invalid communication penalty is introduced. The invalid communication penalty specifically includes: when the coordinator decides to communicate with the planner, and the high-level communication returned by the planner at the current moment If the instruction is the same as the high-level instruction returned by the communication at the previous moment, it means that there is no need to communicate with the planner at this time, and the coordinator will receive a negative feedback.

Further, the expression of cumulative communication reward is:

in, is the accumulated communication reward at time t,/> is the reward given by the environment at time t,/> is an indicator function, is the action of the coordinator at time t, ask means that the coordinator needs to communicate with the planner, not ask means that the coordinator does not need to communicate with the planner,/> High-level instructions returned by the planner at time t,/> is the reward discount coefficient,/> is the penalty coefficient for invalid communication.

It should be understood that when the coordinator's action is "communication", it means that the coordinator requests a new plan from the planner at this time; when the coordinator's action is "no communication", it means that the coordinator insists on executing the current plan at this time.

Furthermore, the cumulative communication is maximized through classic reinforcement learning methods such as proximal policy optimization method (PPO), maximum entropy actor-critic method (SAC), deep Q network (DQN) or dominant actor-critic method (A2C) As a reward, you can get the coordinator's optimal communication strategy. After being formally deployed to the test environment, the coordinator accepts the status (i.e., observation data) sent by the executor, and by calling the optimal communication strategy, outputs the action of whether to communicate with the planner. On the one hand, it significantly reduces the number of communications with the large language model. , reducing the time cost and computing resource cost caused by communication. On the other hand, it helps the agent to turn to large language models in a timely manner when facing emergencies, and reduces the need for large language models in scenarios where large language models are prone to errors. The dependence of the model improves the safety and task success rate of the agent.

It should be understood that the proximal policy optimization method is a classic reinforcement learning method, which can achieve small-batch updates through multiple training steps, can easily determine the step size, and select a behavior for backpropagation through observation data. Use rewards to directly enhance and weaken the possibility of choosing a behavior. Good behavior will increase the probability of being selected next time, and bad behavior will weaken the probability of being selected next time. The maximum entropy actor-critic method is a classic reinforcement learning method. Through the collaboration of actors and critics, it ultimately guides the agent to select the optimal behavior. The actor simultaneously maximizes the entropy of the expectation and strategy distribution, and selects the optimal behavior. It ensures the randomness of the behavior strategy while acting, is efficient and stable, and is more robust to different environments.

(3) After receiving the standard form data, the planner generates new high-level instructions corresponding to the execution actions based on the standard form data and sends them to the executor.

Planners include but are not limited to large language models and user collaboration with large speech models. It should be understood that the planner can be a large language model that encodes and decodes natural language through a neural network to achieve the output of high-level instructions; it can also be manually intervened by technical experts, such as traffic police in smart transportation systems; or by a large language model Alternatives are given and chosen by the user based on preference.

Specifically, when the standard form data received by the planner is "observed milk being spilled on the table", based on the standard form data it can be determined that the action that needs to be performed by the executor is "control the robotic arm to wipe the table clean", the planner Generate the high-level instruction corresponding to the action, and then send the high-level instruction to the executor.

(4) After the executor receives the high-level instruction, it calls the underlying control logic corresponding to the high-level instruction based on the current observation data to perform the corresponding execution action.

Specifically, the executor has its own corresponding underlying control logic. The underlying control logic corresponds to the high-level instructions one-to-one. The underlying control logic corresponds to the execution actions one-to-one. The high-level instructions correspond to the execution actions one-to-one. In this way, the executor receives the high-level instructions. After the instruction, the underlying control logic corresponding to the high-level instruction will be called to perform the corresponding execution action. The underlying control logic can be pre-trained or preset. After the executor receives the new high-level instruction sent from the planner, it will execute the new execution action corresponding to the new high-level instruction; after the executor receives the current high-level instruction sent from the coordinator, it will continue to execute the current high-level instruction. Corresponding pre-trained or preset execution actions.

The large language model interaction method of the present invention is described in detail below based on the embodiments, and the purpose and effect of the present invention will become more obvious.

Example 1

As shown in Figure 3, a communication schematic diagram of an embodiment of applying a large language model interaction method in a single-agent system is shown.

Specifically, take a certain working state of a handling robot in a smart factory as an example (actual application scenarios include but are not limited to handling robots). At this time, the handling robot is the executor and the large language model is the planner. The large language model interaction method applied in a single-agent system includes the following steps:

(1) On its way to the next object that needs to be transported, the handling robot sends observation data such as image data collected by the camera to the coordinator.

(2) The coordinator receives the observation data sent by the handling robot and uses the optimal communication strategy to determine whether it needs to communicate with the planner. If there is no object that needs to be moved in the image collected by the handling robot's camera or it is far away from the object that needs to be moved, it means that the coordinator does not need to communicate with the large language model, and the coordinator will resend the current high-level instructions to the handling robot to allow the handling The robot continues to perform the original preset "move" action. If the handling robot is close enough to the object that needs to be moved, it means that the coordinator needs to communicate with the planner, and the observation data such as image data collected by the handling robot's camera will be converted into standard form data, that is, a natural language "object" that can be understood by the large language model The height XX, width XX, and the relative position of the handling robot are XX”, and are output to the large language model.

(3) After receiving the standard form data, the large language model generates new high-level instructions corresponding to the execution of the action based on the standard form data, that is, the high-level instruction in the form of natural language "After turning XX angle to align the object, both hands move forward at XX distance Stretch out XX distance and hug the object, raise the object to XX height and then put your hands back to your chest", and then give this new high-level command to the handling robot.

(4) After receiving high-level instructions, the handling robot will call the pre-trained or preset underlying control logic corresponding to the high-level instructions in the form of natural language, such as "rotation direction", "stretch out hands", "raise objects", " "Retract your hands" and so on, to complete these instructions from the underlying mechanical control level and complete the corresponding execution actions.

Example 2

As shown in Figure 4, a communication schematic diagram of an embodiment of applying a large language model interaction method to a multi-agent system is shown.

Specifically, taking a certain working state of the smart transportation system as an example (actual application scenarios include but are not limited to smart transportation systems), in this system, the executors include multiple traffic lights, roadside monitoring and autonomous vehicles, and the planner is Collaboration between large language models and traffic police. The large language model interaction method applied in multi-agent systems includes the following steps:

(1) Multiple roadside surveillance and on-board cameras of autonomous vehicles show that an accident occurred on a certain road section, and the currently collected observation data is sent to the coordinator.

(2) The coordinator receives and summarizes the observation data sent by multiple executors, and determines whether it needs to communicate with the planner. If the coordinator analyzes the observation data collected by multiple roadside monitoring and autonomous vehicles and determines that the accident has a small impact on the traffic conditions of the road section, indicating that the coordinator does not need to communicate with the planner, all executions will be allowed The operator continues to perform actions as originally planned. If the coordinator determines from the observation data that the accident will cause congestion on the road section, and it is necessary to clear the blocked traffic flow or guide the traffic flow to the surrounding idle road sections, indicating that the coordinator needs to communicate with the planner, the observation data will be collected and synthesized from multiple executors. It is converted into standard form data, that is, natural language that planners can understand, "An accident occurred in XX lane on XX section, and XX motor vehicles were blocked." and output to the large language model and traffic police.

(3) After receiving the standard form data, the large language model and the traffic police generate new high-level instructions corresponding to the execution actions based on the standard form data, corresponding to the optimal actions that each executor in the multi-agent system needs to perform. Send high-level instructions to the executor, such as issuing an advanced instruction to "turn in advance" to a self-driving car that has not entered the road section, and issuing an advanced command to "avoid the accident lane in advance" to a self-driving car in the road section. The traffic light on this road section issues high-level instructions such as "keep the green light longer" and so on. Through such joint optimal actions, the entire multi-agent system reaches a Nash equilibrium.

(4) After each executor receives the corresponding high-level instruction, it will call the pre-trained or preset underlying control logic "steering", "deceleration", "lane change", " "Switch lighting scheme" and so on, to complete these instructions from the underlying mechanical control level and complete the corresponding execution actions.

Example 3

As shown in Figure 5, a communication schematic diagram of another embodiment of applying a large language model interaction method to a multi-agent system is shown. In this system, it includes multiple executors, a coordinator and a planner. This embodiment , apply the large language model interaction method, which specifically includes the following steps:

(1) There are multiple executors in the current system. Each executor sends the observation data it collects to the coordinator. The coordinator merges all the observation data received to obtain the merged observation data.

(2) The coordinator uses the optimal communication strategy based on the merged observation data to determine whether it needs to communicate with the planner. If the coordinator does not need to communicate with the planner, the coordinator resends the current high-level instructions to the corresponding executors, allowing each Each executor continues to execute the original high-level instructions; if the coordinator needs to communicate with the planner, the coordinator converts the merged observation data into standard form data, such as observation data in the form of text strings or pictures, and converts the standard form Data is sent to planners.

(3) After receiving the standard form data, the planner generates new high-level instructions corresponding to the execution actions based on the standard form data and sends them to the executor.

(4) After the executor receives the high-level instruction, it calls the underlying control logic corresponding to the high-level instruction based on the current observation data to perform the corresponding execution action.

Example 4

As shown in Figure 6, a communication schematic diagram of another embodiment of applying a large language model interaction method to a multi-agent system is shown. In this system, it includes multiple executors, multiple coordinators, multiple sub-planners and a The master planner, in this embodiment, applies the large language model interaction method, which specifically includes the following steps:

(1) Each executor sends the observation data it collects to the corresponding coordinator.

(2) The coordinator uses the optimal communication strategy based on the received observation data to determine whether it needs to communicate with the sub-planner. If the coordinator does not need to communicate with the sub-planner, the coordinator resends the current high-level instructions to the corresponding executor. , allowing the executor to continue executing the original high-level instructions; if the coordinator needs to communicate with the sub-planner, the coordinator converts the observation data into standard form data, such as text string or picture form observation data, and converts the standard form The data is sent to the sub-planner.

(3) After receiving the standard form data, each sub-planner generates a new high-level instruction corresponding to the execution action based on the standard form data. After that, each sub-planner sends the new high-level instruction generated by each sub-planner to the master planner. After integrating the high-level instructions sent by each sub-planner, the master planner makes global instructions, that is, generates corresponding high-level instructions for each executor and sends them to the corresponding executor.

(4) After each executor receives the high-level instruction, it calls the underlying control logic corresponding to the high-level instruction to perform the corresponding execution action based on the current observation data.

In summary, the task in the MiniGrid environment is used to simulate the agent in the smart factory scenario. Under this task, the agent needs to explore the entire room with a limited field of view, find the key corresponding to the color of the door and use the key. Key opens the door. As shown in Figure 7, four methods are compared through experiments: the method adopted by the present invention, the method of artificially setting when the agent communicates in advance, the method of always communicating with a large language model, and the method of never communicating with a large language model. The results verify the beneficial effects of the present invention. Among them, the ordinate of (a) in Figure 7 represents the task success rate, the ordinate of (b) in Figure 7 represents the number of communications between the agent and the large language model, and the abscissa of both figures represents the number of experimental cycles. It can be seen that with continuous training, the method adopted in the present invention continues to reduce communication costs and continuously improves mission success rate, and ultimately is superior to other methods in both communication cost and mission success rate.

It is worth mentioning that the present invention also provides a large language model interaction device.

Specifically, as shown in Figure 2, the device includes a planner module, a coordinator module and an executor module. Among them, the planner module is used to generate new high-level instructions corresponding to the execution actions based on the received standard form data. The coordinator module is used to use the optimal communication strategy based on the observation data to determine whether communication with the planner is needed. If the coordinator needs to communicate with the planner, the coordinator converts the observation data into standard form data and sends the standard form data to Planner; if the coordinator does not need to communicate with the planner, the coordinator resends the current high-level instructions to the executor. The executor module is used to collect observation data, and after receiving high-level instructions, call the underlying control logic corresponding to the high-level instructions to perform the corresponding execution actions.

The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smartphone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or A combination of any of these devices.

For the convenience of description, when describing the above device, the functions are divided into various units and described separately. Of course, when implementing the present invention, the functions of each unit can be implemented in the same or multiple software and/or hardware.

Referring to Figure 8, an electronic device provided by an embodiment of the present invention includes a memory and a processor. The memory is coupled to the processor. The memory is used to store program data, and the processor is used to execute the program data in a large format in the above embodiment. Language model interaction methods.

Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk memory, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein. The computer program is stored on the readable storage medium. When the program is executed by the processor, the large language model interaction method in the above embodiment is implemented.

The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each process and/or block in the flowchart illustrations and/or block diagrams, and combinations of processes and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing device to produce a machine, such that the instructions executed by the processor of the computer or other programmable data processing device produce a use A device for realizing the functions specified in a process or processes in a flowchart and/or a block or blocks in a block diagram.

These computer program instructions may also be stored in a computer-readable memory that causes a computer or other programmable data processing apparatus to operate in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including the instruction means, the instructions The device implements the functions specified in a process or processes in the flowchart and/or in a block or blocks in the block diagram.

These computer program instructions may also be loaded onto a computer or other programmable data processing device, causing a series of operating steps to be performed on the computer or other programmable device to produce computer-implemented processing, thereby executing on the computer or other programmable device. Instructions provide steps for implementing the functions specified in a process or processes of a flowchart diagram and/or a block or blocks of a block diagram.

It should also be noted that the terms "comprises," "comprises," or any other variation thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that includes a list of elements not only includes those elements, but also includes Other elements are not expressly listed or are inherent to the process, method, article or equipment. Without further limitation, an element defined by the statement "comprises a..." does not exclude the presence of additional identical elements in a process, method, article, or device that includes the stated element.

It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems or computer program products. Thus, the invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein.

The invention may be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present invention may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media including storage devices.

Each embodiment of the present invention is described in a progressive manner. The same and similar parts between the various embodiments can be referred to each other. Each embodiment focuses on its differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple. For relevant details, please refer to the partial description of the method embodiment.

The above embodiments are only used to illustrate the design ideas and features of the present invention, and their purpose is to enable those skilled in the art to understand the content of the present invention and implement it accordingly. The protection scope of the present invention is not limited to the above embodiments. Therefore, all equivalent changes or modifications made based on the principles and design ideas disclosed in the present invention are within the protection scope of the present invention.

Claims (10)

1. A large language model interaction method, comprising the steps of:

(1) After the executor interacts with the environment, the current acquired observation data is sent to a coordinator;

(2) The coordinator judges whether the coordinator needs to communicate with the planner or not by adopting an optimal communication strategy according to the received observation data, and if the coordinator needs to communicate with the planner, the coordinator converts the observation data into standard form data and sends the standard form data to the planner; if the coordinator does not need to communicate with the planner, the coordinator resends the current high-level instruction to the executor and jumps to step (4);

(3) After receiving the standard form data, the planner generates a new high-level instruction corresponding to the execution action based on the standard form data and sends the new high-level instruction to the executor;

(4) After receiving the high-level instruction, the executor calls the bottom control logic corresponding to the high-level instruction according to the current observation data to execute the corresponding execution action.

2. The large language model interaction method of claim 1, wherein the observation data includes sensor data, text data, and image data.

3. The large language model interaction method according to claim 1, wherein the optimal communication strategy specifically comprises: the coordinator decides whether to communicate with the planner at each moment so that the planner can complete the task through the minimum number of communication times, and defines the process as a reinforcement learning process, wherein the reinforcement learning process specifically comprises: corresponding to the state at each moment, the coordinator has two different actions-insisting on executing the current plan or requesting a new plan from the planner, introducing invalid communication penalty as a cumulative communication reward based on the rewards given by the environment; wherein the state is the observed data collected by the executor; the best communication strategy for the coordinator is achieved by maximizing the cumulative communication rewards.

4. A large language model interaction method according to claim 3, wherein the expression of the cumulative communication prize is:

；

wherein,for a cumulative communication reward at time t +.>Awards given by the environment for time t, +.>As a function of the characteristics of the display,for the action of the coordinator at the time t, ask represents the requirement of the coordinatorIn communication with the planner, not ask means that the coordinator does not need to communicate with the planner, +.>High-level instruction returned for time t planner, < >>For rewarding discount coefficient, < >>Penalty coefficients for invalid communications.

5. The large language model interaction method of claim 3, wherein the training method for maximizing cumulative communication rewards comprises a near-end policy optimization method, a maximum entropy actor-critique method, a depth Q network, and a dominant actor-critique method.

6. The large language model interaction method of claim 1, wherein the executor comprises a plurality of agents; the planner includes a large language model and the user cooperates with the large language model.

7. The large language model interaction method of claim 1, wherein the high-level instructions are in one-to-one correspondence with the underlying control logic, the underlying control logic is in one-to-one correspondence with the execution actions, and the high-level instructions are in one-to-one correspondence with the execution actions.

8. A large language model interaction device for implementing the large language model interaction method of any one of claims 1 to 7, comprising:

the planner module is used for generating a new high-level instruction corresponding to the execution action according to the received standard form data;

the coordinator module is used for judging whether the coordinator needs to communicate with the planner or not by adopting an optimal communication strategy according to the observation data, and if the coordinator needs to communicate with the planner, the coordinator converts the observation data into standard form data and sends the standard form data to the planner; if the coordinator does not need to communicate with the planner, the coordinator resends the current high-level instruction to the executor; and

and the executor module is used for collecting the observation data and calling the bottom control logic corresponding to the high-level instruction to execute the corresponding execution action after receiving the high-level instruction.

9. An electronic device comprising a memory and a processor, wherein the memory is coupled to the processor; wherein the memory is for storing program data and the processor is for executing the program data to implement the large language model interaction method of any of claims 1-7.

10. A computer readable storage medium having stored thereon a computer program, wherein the program when executed by a processor implements the large language model interaction method of any of claims 1-7.