EP4630961A1

EP4630961A1 - Multi-agent debate

Info

Publication number: EP4630961A1
Application number: EP24837523.0A
Authority: EP
Inventors: Andrew Estornell; Yang Liu
Original assignee: Lemon Inc Cayman Island
Current assignee: Lemon Inc Cayman Island
Priority date: 2024-03-01
Filing date: 2024-03-01
Publication date: 2025-10-15
Also published as: EP4630961A4; CN119678162A; WO2025183627A1

Abstract

A method is proposed for multi-agent debate. In the method, a set of responses to a task expressed in a natural language are generated by a plurality of agents in at least one round of a debate for the task. The set of responses are updated by selecting responses from the set of responses, and/or adjusting at least one response in the set of responses. The updated set of responses are provided to the plurality of agents as an input for a subsequent round of the debate, to generate a result for the task.

Description

MULTI-AGENT DEBATE

FIELD

[0001] The present disclosure generally relates to the field of computer, and more specifically, to methods, devices, and computer program products for multi-agent debate.

BACKGROUND

[0002] A conversational platform allows users to give their tasks and then provide responses to the tasks. The tasks are usually expressed in a natural language. For example, the users usually input natural language questions, and the conversational platform then provides answers to the questions. Nowadays, the conversational platform can rely on models, such as machine learning models, to perform reasoning on the tasks and provide responses in the natural language. There are a variety of works that strengthen the reasoning ability of the models, so as to improve the accuracy of the generated responses.

SUMMARY

[0003] In a first aspect of the present disclosure, there is provided a method of multi-agent debate. The method includes: obtaining a set of responses to a task expressed in a natural language, the set of responses generated by a plurality of agents in at least one round of a debate for the task, updating the set of responses by at least one of selecting responses from the set of responses, or adjusting at least one response in the set of responses; and generating a result for the task by providing the updated set of responses to the plurality of agents as an input for a subsequent round of the debate.

[0004] In a second aspect of the present disclosure, there is provided an electronic device. The electronic device comprises: a computer processor coupled to a computer-readable memory unit, the memory unit comprising instructions that when executed by the computer processor implements a method according to the first aspect of the present disclosure.

[0005] In a third aspect of the present disclosure, there is provided a computer program product, the computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by an electronic device to cause the electronic device to perform a method according to the first aspect of the present disclosure.

[0006] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.

BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Through the more detailed description of some embodiments of the present disclosure in the accompanying drawings, the above and other objects, features, and advantages of the present disclosure will become more apparent, wherein the same reference generally refers to the same components in the embodiments of the present disclosure.

|0008| FIG. 1 illustrates an example environment in which example embodiments of the present disclosure can be implemented;

[0009] FIG. 2 illustrates a diagram of a debate process with as a hidden Markov model with observable variables and unobservable variables according to some embodiments of the present disclosure;

[0010] FIG. 3 illustrates an example of a misconception between models, and a refutation of that misconception according to some embodiments of the present disclosure;

[0011] FIG. 4 illustrates an example procedure of conducting interventions on previous responses generated by agents according to some embodiments of the present disclosure;

[0012] FIG. 5 illustrates an example flowchart of a method of multi-agent debate according to some embodiments of the present disclosure; and

[0013] FIG. 6 illustrates a block diagram of an electronic device in which various embodiments of the present disclosure can be implemented.

DETAILED DESCRIPTION

[0014] Principle of the present disclosure will now be described with reference to some embodiments. It is to be understood that these embodiments are described only for the purpose of illustration and help those skilled in the art to understand and implement the present disclosure, without suggesting any limitation as to the scope of the disclosure. The disclosure described herein can be implemented in various manners other than the ones described below.

[0015] In the following description and claims, unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skills in the art to which this disclosure belongs

[0016] References in the present disclosure to “one embodiment,” “an embodiment,” “an example embodiment,” and the like indicate that the embodiment described may include a particular feature, structure, or characteristic, but it is not necessary that every embodiment includes the particular feature, structure, or characteristic. Moreover, such phrases are not necessarily referring to the same embodiment. Further, when a particular feature, structure, or characteristic is described in connection with an example embodiment, it is submitted that it is within the knowledge of one skilled in the art to affect such feature, structure, or characteristic in connection with other embodiments whether or not explicitly described.

[0017] It shall be understood that although the terms “first” and “second” etc. may be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another. For example, a first element could be termed a second element, and similarly, a second element could be termed a first element, without departing from the scope of example embodiments. As used herein, the term “and/or” includes any and all combinations of one or more of the listed terms.

[0018] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of example embodiments. As used herein, the singular forms “a”, “an” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms “comprises”, “comprising”, “has”, “having”, “includes” and/or “including”, when used herein, specify the presence of stated features, elements, and/or components etc., but do not preclude the presence or addition of one or more other features, elements, components and/ or combinations thereof.

[0019] Principle of the present disclosure will now be described with reference to some embodiments. It is to be understood that these embodiments are described only for the purpose of illustration and help those skilled in the art to understand and implement the present disclosure, without suggesting any limitation as to the scope of the disclosure. The disclosure described herein can be implemented in various manners other than the ones described below. In the following description and claims, unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skills in the art to which this disclosure belongs.

[0020] It may be understood that data involved in the present technical solution (including but not limited to the data itself, the acquisition or use of the data) should comply with requirements of corresponding laws and regulations and relevant rules.

[0021] It may be understood that, before using the technical solutions disclosed in various embodiment of the present disclosure, the user should be informed of the type, scope of use, and use scenario of the information involved in the present disclosure in an appropriate manner in accordance with relevant laws and regulations, and the user’s authorization should be obtained [0022] For example, in response to receiving an active request from the user, prompt information is sent to the user to explicitly inform the user that the requested operation will need to acquire and use the user’s information. Therefore, the user may independently choose, according to the prompt information, whether to provide the information to software or hardware such as electronic devices, applications, servers, or storage media that perform operations of the technical solutions of the present disclosure.

[0023] As an optional but non-limiting implementation, in response to receiving an active request from the user, the way of sending prompt information to the user, for example, may include a pop-up window, and the prompt information may be presented in the form of text in the pop-up window. In addition, the pop-up window may also carry a selection control for the user to choose “agree” or “disagree” to provide the information to the electronic device.

[0024] It may be understood that the above process of notifying and obtaining the user authorization is only illustrative and does not limit the implementation of the present disclosure. Other methods that satisfy relevant laws and regulations are also applicable to the implementation of the present disclosure.

[0025] As used herein, the term “model” is referred to as an association between an input and an output learned from training data, and thus a corresponding output may be generated for a given input after the training. The generation of the model may be based on a machine learning technique. In general, a machine learning model may be built, which receives input information and makes predictions based on the input information. For example, a classification model may predict a class of the input information among a predetermined set of classes. As used herein, “model” may also be referred to as “machine learning model”, “learning model”, “machine learning network”, or “learning network,” which are used interchangeably herein.

Example environment

[0026] FIG. 1 illustrates a block diagram of an example environment 100 in which various embodiments of the present disclosure may be implemented. In the environment 100 of FIG. 1, a conversational platform 1 10 may receive a task 135 from a user 1 0. The task may be provided or expressed in a natural language (for example, in English, Chinese, or another natural language). The conversational platform 110 includes one or more agents 120-1, 120-2, ..., 120-N (collectively or individually referred to as agents 120, where N indicates an integer larger than or equal to one) to process the task from the user 130 and provides a final result 140 to the task 135. In an example, the conversational platform 110 may receive a question in a natural language from the user 130 and provide a final answer to the question. In the following, for purpose of illustration with any limitation, some embodiments are described by taking the question as an example of the task 135 and taking the answer as an example of the result 140.

[0027] The agents 120-1, 120-2, . . . , 120-N may be configured to use machine learning models 125-1, 125-2, ..., 125-N (collectively or individually referred to as machine learning models 125 or models 125 for short), respectively, to process the natural language task. Alternatively, the agents 120-1, 120-2, ..., 120-N may be implemented by the machine learning models 125-1, 125- 2, ..., 125-N, respectively. In the following, the terms “model” or “agent” may be used interchangeably.

[0028] In some embodiments, the models 125 may be language models (LMs), which are trained on large-scale corpus to learn the capability of understanding one or more types of natural languages. In some embodiments, the models 125 may be generative models for content generation. The models 125 may be constructed based on various machine learning technologies. [0029] LMs trained on large-scale corpus have demonstrated a remarkable ability to perform unseen tasks with high efficacy. This behavior, often referred to as emergent, allows LMs to serve as general purpose tools for a wide array of language based functions. One such behavior of particular interest is the ability of LMs to intake and process opinions from other models (or humans). This ability allows LMs to collaboratively solve tasks by engaging in debate. For a given task, multi-agent debate operates by eliciting responses from each model, distributing those responses among the models, and then eliciting updated responses from each model.

[0030] The multi-agent debate focuses on iterative collaboration between agents in order to make a decision. The works regarding the multi-agent debate often focus on multi-agent debate in the context of question-answering tasks and aim to provide higher quality answers (compared to those of a single model) by engaging multiple models in discussion. The basic debate format facilitates debate by first asking each model the same question, and then iteratively re-asking agents that same question contextualized by the responses of all models in the previous round. Different solutions of this procedure have also been proposed. For example, a solution focuses on debate when each model in the debate is assigned a different role, and another solution focuses on round-robin style debate. Some other solutions for iteratively improving the quality of answers have also been proposed.

[0031] Despite the above solutions, current works on multi-agent debate lack theoretical investigation, and the debate efficacy still needs to be improved.

[0032] Embodiments of the present disclosure propose solutions for multi-agent debate. According to embodiments of the present disclosure, a set of responses to a task expressed in a natural language are generated by a plurality of agents in at least one round of a debate for the task. The set of responses are updated by selecting responses from the set of responses, and/or adjusting at least one response in the set of responses. The updated set of responses are provided to the plurality of agents as an input for a subsequent round of the debate, to generate a result for the task.

[0033] In embodiments of the present disclosure, during the multi-agent debate procedure, previous responses from the agents are not directed used for the subsequent round of debate. Rather, these previous responses are intervened, for example, by selecting a subset of responses and/or by correcting some of them. These intervened responses are observed by the agents for generating response in a next round of debate. In this way, the accuracy of the final result and the efficacy of the debate can be improved.

[0034] Theoretical foundation of the present disclosure and example embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings

Theoretical Formulation of Multi- Agent Debate

[0035] To better understand the embodiments of the present disclosure, theoretical formulation of the multi-agent debate which draws on connections from in-context learning and Bayesian inference is first provided. Let x be a given question, with associated answer z, e.g., x -‘What color is the sky?” and z - ‘Blue” . Tasks x, and their associated answer z, are generated by first selecting a vector of latent concept 0, and then sampling (x,y) ~ For example, in the above task, it may be the case that 0 represents concepts such as “nature”, “general knowledge”, and “refraction”. For a given task x, a collection of n LMs (or agents), collaborates to infer the correct answer z by iteratively engaging in discussion over T rounds. For example, at round t = 0, each agent i observes x and then provides response . At rounds t > 0, each agent i observes x and i.e., the outputs of the n agents at the previous timesetep, and outputs response y® . The process ends if t = T or if the agents reach consensus.

[0036] To measure if consensus is reached, a function a extracts an answer from a given response y. Suppose y =“During the day, the sky is blue”, then a (y) = “Blue” . At round t, the probability that the agent i provides the response y_i ^(t+1\ is given by: where captures model hyperparameters (such as training data, architecture, etc.). Both the input (Y^ , x) as well as the hyperparameters </>,-, ultimately influence the output Note that on each round all agents observe the same input, namely (f , x). Thus, differences in agent outputs are determine by 1) the stochastic nature of output generation, and 2) the unique parameters (pi of each model

[0037] The theoretical formulation of multi agent debate can provide key insights into the inner-workings of the debate procedure, which then is used to develop more effective debate according to some embodiments of the present disclosure. Agent debate can be viewed as a multivariate hidden Markov model with observed variables Y, x, and </>, as well as and hidden variable 0 which correspond to latent concepts, as shown in FIG. 2. An assumption is shown in Table 1.

Table 1

[0038] The assumption in Table 1 can be interpreted as saying that a model’s generation y_L, is uniquely determined by the model parameters <pt and the concepts 0 identified by the model. In the case of encoder-decoder based models, the joint between (f> and 0 may be conceptualized as corresponding to the embedding produced by the encoder. With this embedding in hand, the original input no longer influences the model’s output, rather the embedding and models parameters will uniquely determine the model’s output.

[0039] Now some descriptions are given to derive a lemma shown in Table 2, which will be useful in examining the way that model responses evolve over time.

Table 2 [0040] The significance of this lemma is that the probability of generating a given response with the other responses F® can be expressed in terms of the probability of generating yi^t+1^ without the other responses and a skew term caused by those responses. Note that, g g y .

[0041] As a proof, this result holds via marginalization of the posterior predictive distribution over the latent concepts 0, namely:

[0042] Multi-agent debate can be viewed a special type of in-context learning where each model response is an example (also called a piece of context). The most consequential difference between in-context learning and debate is the generation of each exampl From the perspective of in-context learning, each example corresponds to an Independent Identically Distribution (i i d ) draw from a fixed distribution D(0), i.e. each example is intended to faithfully convey a similar idea. However, in the case of debate each y[^t+1^ is distributed with respect to Y^, x, and more importantly <pp, the latter of which to no longer be i.i.d, unless all models have the same parameters </>. As such, they can be competing factors within the set of responses \ which would not be present with in-context learning. [0043] There may be natural connections between in-context learning and Bayesian inference. In particular, the probability that a model provides a given response, when seeing a prompt and n pieces o fr cont .ext., i.s When the cont .ext . i •s made up of an increasing number of examples sourced from the same concept 0, the model places a higher weight on responses which it associates with that latent concept. That is, for concepts which have larger values of _moc|_e] j_{s more} likely to generate responses corresponding to 0, i.e., adding more examples increases the density placed over 0 during generation. This phenomenon also extends to debate, the responses of the other agents Y^ now takes the place of the context. As noted before, each response may have different distributions over the latent concepts, which ultimately influence the response of each mod ,e ,l at ti .me t+ ,1, as i .s captured , , by t ,he s ,kew term . , . m the lemma shown in Table 2. This added influence can be both a boon and determent to debate. Some embodiments to be described below may center around harnessing this skew term such that it improves debate efficacy.

[0044] Through this perspective of debate, more effective debate procedures can be designed by leveraging the concept space 0. To this end, the multi-agent debate may be formulated as an optimization problem where the skew term, described in the lemma shown in Table 2, correspondingtothe optimization variables. Foragiven task x with correct answer z*, each round of debate can be formulated as the following optimization problem. where that is, time t aims to craft responses Y^ such that they maximize the probability that the n models select the correct answer at the next time step. Next, this objective may be expanded over the latent concept space 0, which becomes. [0045] There are three key challenges with directly optimizing this objective. First the underlying concepts 0, as well as the true concept 0* from which x and z* originate, are unknown. Second, the relationship between a given response and the underlying concepts is also unknown. Third, the optimization variables (the responses in F®) are natural language and thus difficult to directly optimize. However, this latent concept based expansion of the objective allows for a design of several approaches, which work within the concept space, to better optimize debate. In order to motivate these approaches, the present disclosure makes several observations about the debate procedure as whole.

Debate principals

[0046] What is first examined is the way that diversity of model abilities and model opinions impacts the collaboration procedure. As can be seen from the following description, it is essential to have both diversity of information in the responses F^(t) as well as diverse opinions and capabilities across the collection of models.

[0047] Suppose the collaboration process is conducted with only one type of model (in effect n copies of the same model). That is, for all Then, as the number of agents increases, the debate procedure is more greatly impacted by the echo-chamber-effect, i.e., the probability that a round of debate results in a change to the most likely concept, perceived by agents, approaches 0. That is, a greater number of similar agents results in static debate dynamics, in essence defeating the purpose of debate.

Table 3

[0048] The theorem in Table 3 implies that when debate is conducted with multiple copies of the same model (or very similar models), increasing the number of models results in debate centering on a single (unchanging) concept rather than a balanced distribution over multiple concepts. P(y' ^: J P ^{z !} _k x. (b) [0049] As a proof sketch, by the lemma shown in Table 2, the term " ^{1 1} n can be expressed as , wherethe term _{a constan}t _with respect to the number of agents n . Since each y^ is an i.i.d. draw according to x, , and <p , the term _wm vanish f_{or a}n Q except for Thus, each term in the expanded form of will like wise vanish. As n^><x>, all models predict the same concept, namely at timestep t with probability 1.

[0050] Next, the way in which similar responses impacts the collaboration process is examined. At time Z, suppose there are m duplicate responses and k non-duplicate responses, i.e.,yi=y', V i < m. As m increases, the answers of model i at time t would be biased towards concepts related toy'. Table 4 shows such a theorem

Table 4

[0051] As a proof sketch, by the lemma shown in Table 2, the conditional probability of model i giving response y/^t+1\t time t upon observing responses Y can be written as:

[0052] Now consider the ratio between the likelihood of generating any two responses y^)¹^ as below:

[0053] Using the decomposition, in Equation 6, for each of these terms, let & be the concept which model i believes ismost likely to have produced response y', i.e.,

[0054] For each Q _> the equation applies. Therefore, lim - 0 nz— >oc . Thus, when examining the limit of the ratio in Equation 7, with respect to the number of repeated signals m, the following equation is derived:

[0055] Thus, the relationship between any two generation probabilities can be uniquely defined by 6 . Since the final response of the model is sampled according to these ratios, the final response will be uniquely determined by the concept &

[0056] The theorem in Table 4 indicates the susceptibility that muti-LMs debate has towards tyranny of the majority. If a large number of models provides the same answer to a task x, then those repeated answers will drown out the single provided by the other models’ responses, as well as the task x itself.

[0057] Next, a particular type of homogeneity in model capabilities and responses, namely shared misconceptions, is described. When a common misconception is shared among the models, debate is less effective and is likely to converge to erroneous concepts associated with the shared misconception. Given that, a more carefully designed debate procedure is required.

[0058] In FIG 3, an example is provided to elucidate what is meant by a shared misconception. FIG. 3 shows a collection of model responses to a question 301 regarding the animal “red panda”, for which the correct answer 302 is “No”. These models erroneously equate red panda and panda, which ultimately leads to each model providing the wrong answer 311, 312, 313. In situations such as the one shown in FIG. 3, debate would converge to an incorrect answer due to that fact each model shares the same false belief about the question.

[0059] Next, the notion of misconceptions is formalized as below. For a given concept <7*, a model with parameters is said to have a misconception regarding 0* if there exists another concept O' s.t., where that is, for tasks generated from the concept 0*, the model believes that the erroneous concept O' explains more than half of the tasks better than the true concept 0*.

[0060] When m of agents have a misconception and share the same erroneous concept O', those m agents may be considered to have a shared misconception. When the models share a common misconception, the responses produced by those models are biased towards the erroneous concept O'. Table 5 shows such a theorem.

Table 5

[0061] As a proof sketch, for task x ~ D(0*), and agent i possessing the misconception,

[0062] For agents without the shared misconception, the opposite is true. As such, any agent with the shared misconception is more likely to provide a response which has higher values , compared to agents without this misconception. Thus, when seeing x, agents with the shared misconception are more likely to provide responses which convey the erroneous concept 0'. Agents see x at every round of debate; this compounding effect results in the debate centering around O' rather than the true concept 0*. [0063] It should be noted that the phenomenon of converging to erroneous concepts cannot be mitigated by adding more models. When the misconceptions of one model are formed through training data, it is likely that other models might possess the same misconception, unless specifically trained to avoid such errors, due to the high correlation in training data between models.

Interventions to the debate procedure

[0064] In the above, a theoretical framework for multi-LLM debate which draws on connections from in-context learning and Bayesian inference is proposed, and theoretical insights on several key principals of multi-LLM debate are provided. Given the theoretical framework and principals, several modifications to the debate procedure, referred to as interventions, are proposed in some embodiments of the present disclosure.

[0065] Reference is now made to FIG. 4, which illustrate an example debate procedure 400 with interventions. The debate procedure 400 is used to generate a result for a task expressed in a natural language, for example, to generate an answer to a natural language question. At current round t of the debate for the task, the plurality of agents 120 generates respective responses for the task. After round t is completed, a set 401 of responses are obtained. The set 401 of responses may include responses generated by the plurality of agents 120 in at least one round of the debate. For example, as shown in FIG. 4, the responses generated at round t by the agents 120 are determined as at least a portion of the set of responses. In the following, the set 401 of responses may be also referred to as a previous response set 401.

[0066] In some embodiments, the previous response set 401 may consist of responses generated at the last round by the plurality of agents, for example, the response generated by each agent in the current round t.

[0067] In some embodiments, the previous response set 401 may include responses generated at a plurality of rounds of the debate. In other words, in such embodiments, the previous response set 401 may include previous responses resulting from a plurality of historical rounds.

[0068] In some embodiments, the previous response set 401 may include responses all previous responses generated up to the current round t. For example, the previous response set 401 may be denoted as . By using all the previous responses, a better performance may be achieved.

[0069] Continuing with the procedure 400, at block 410, one or more interventions may be applied to the previous response set 401, resulting in an updated set 402 of responses. The interventions may include two categories: pruning intervention and modification intervention.

[00701 The pruning intervention may be used to select a subset of responses from the previous response set 401. In other words, the pruning intervention focuses on choosing which responses to keep in Y^\ The selection may be based on mutual information for two responses, and/or a relevance of a response to the task. As such, the pruning intervention may include diversity pruning 411 and quality pruning 412, which will be detailed below.

[0071] The modification intervention may be used to adjust at least one response in the previous response set 401. In other words, the modification may focus on changing or editing the responses Y^ . The modification intervention may be used to correct a misconception in a response. A response may be corrected by an agent which is different from the agent generating the response. The modification intervention may include the misconception refutation 413.

[0072] In some embodiments, at block 410, two or more interventions may be combined to update the previous response set 401. Any suitable manner for combining different interventions is possible.

[0073] In some embodiments, the pruning intervention may be combined with the modification intervention. For example, the pruning intervention (the diversity pruning and/or the quality pruning) may be first applied to select a subset of responses from the previous response set 401. Then, the modification intervention is applied to adjust responses in the subset of responses.

[0074] In some embodiments, the diversity pruning 41 1 , the quality pruning 412 and the misconception refutation 410 may be combined. For example, the quality pruning 412 is first applied to select a first subset of responses from the previous response set 401 based on a relevance of each response to the task. Then, the diversity pruning 412 may be applied to select a second subset of responses from the first subset of responses based on based on mutual information for each pair of responses in the first subset of responses. Finally, the misconception refutation 410 may be applied to correct misconceptions in responses in the second subset of responses.

[0075] By applying one or more interventions, the updated set 402 of responses are obtained. Accordingly, the updated set 402 of responses may be provided to the plurality of agents 120 as an input for the subsequent round t+1, as shown in FIG. 4. For example, the updated set 402 of responses is included in a prompt to each agent 120. Each of the plurality of agents 120 observes the updated set 402 of responses and the task x, and generates a response in the subsequent round t+1. The procedure 400 may be iteratively performed until a total number of rounds reaches a threshold or a consensus is achieved between the plurality of agents 120.

[0076] Some more embodiments regarding the three interventions are now described

[0077] According to quality intervention, responses may be selected based on a relevance of each response to the task. In some embodiments, a quality metric for each response of a first plurality of responses may be determined based on the task and the response. The quality metric for a response may represent or characterize a relevance of the response to the task. Then, a first number of responses may be selected from the first plurality of responses based on the respective quality metrics for the first plurality of responses.

[0078] In such embodiments, the first plurality of responses is included in the previous response set 401. For example, if the quality intervention is applied as the first intervention (specifically, before the diversity intervention), the first plurality of responses may be the previous response set 401 of responses. If the quality intervention is applied as after the diversity intervention, the first plurality of responses may include responses selected according to the diversity intervention.

[0079] The quality metric may be any suitable metric suitable for reflecting the relevance of a response to the task To this end, in some embodiments, an embedding mode for the natural language, for example, a sentence embedding model, may be used. For example, the embedding model may be used to generate a first representation for the task and a second representation for the response. A similarity between the first and second representations may be determined as the quality metric for the response. [0080] As an example, the quality pruning 412 selects k responses of the n responses at time t. Rather than selecting for diversity, the quality pruning 412 aims to select the k highest question responses. This is done by selecting the k responses which maximize the following equation: where that is, quality pruning aims to select the k responses in T^with the most similar distribution over concepts as the original task x.

[0081] In practice, each distribution cannot be directly accessed, but can be approximated by a sentence embedding model. The embedding space of such a model, say g, serves as a proxy for the latent concept space Thus, in practice, the quality pruning can be operationalized as: where is a representation similarity used as the quality metric.

[0082] Table 6 shows an example theorem for the quality pruning.

Table 6

[0083] According to diversity intervention, responses may be selected based on mutual information for each pair of responses in the previous response set 401. In some embodiments, a diversity metric for each pair of responses among a second plurality of responses may be determined based on the pair of responses. The diversity metric for each pair of responses may represent or characterize mutual information for the pair of responses. Then, a second number of responses may be selected from the second plurality of responses based on the diversity metric for each pair of responses.

[0084] In such embodiments, the second plurality of responses is included in the previous response set 401. For example, if the diversity intervention is applied as the first intervention (specifically, before the quality intervention), the second plurality of responses may be the previous response set 401 of responses. If the diversity intervention is applied as after the quality intervention, the second plurality of responses may include responses selected according to the quality intervention.

[0085] The diversity metric may be any suitable metric suitable for reflecting the mutual information. To this end, in some embodiments, similar to the quality metric, an embedding mode for the natural language, for example, a sentence embedding model, may be used. For example, the embedding model may be used to generate a pair of representations for the pair of responses. The diversity metric for the pair of responses may be determined based on a similarity between the pair of representations, for example, as one minus the similarity.

[0086] As an example, the diversity pruning intervention selects k responses of the n responses in K® which maximizes mutual information. That is, Y'^ is passed at time t where yt), -D(G\ y-/)) s.t |y| = k . (14)

[0087] In practice, the distribution D(0| y) is unknown, but can be approximated by the sentence embedding model. Similar to the quality pruning, the embedding space of such a model, say g, serves as a proxy for the latent concept space. Let d(u, v) = 1-cosine similarity(u, v), then in practice Y'^ may be computed as: y'^=

[0088] Next, we show that when models have a shared misconception, diversity pruning decreases the likelihood that the debate procedure will converge to the erroneous concept corresponding to the shared misconception. Table 6 shows an example theorem for the diversity pruning

Table 7 is [0089] As shown by the theorems in Table 6 and Table 7, the diversity pruning decreases the probability that debate converges to incorrect answers sourced from a particular concept, while quality pruning increases the probability that debate converges to a correct answer sourced from the true concept. In some embodiments, both interventions can be used simultaneously to more effectively guide the debate procedure away from wrong answers, and towards correct answers.

|0090| The effectiveness of the diversity pruning is highly dependent on the quality of the models used in debate, while the quality pruning is highly dependent on the richness of the task (a very detailed task is likely to convey more concepts which are related to the answer, than a terse task which lacks such details) As such, the quality pruning may be prioritized for detailed tasks, and the diversity pruning may be prioritized if all models in the debate are believed to be of very high quality.

[0091] According to the misconception refutation, a given response in the previous response set 401 may be adjusted by a target agent to correct one or more misconceptions in the given response. The given response may not be generated by the target agent. The response may be adjusted by prompting the target agent of the plurality of agents to correct the response. The adjusted response generated by the target agent may be kept in the updated set 402 for the subsequent round t+1.

[0092] In some embodiments, a target agent of the plurality of agents may be assigned for the given response. Prompt information indicating to correct the given response (for example, to correct the misconceptions) may be generated and provided to the target agent. Then, an adjusted response for the given response may be derived from an output of the target agent.

[0093] As an example, in addition to selecting which responses in Y^ will be used in the next round of debate, the responses in Y^ may be modified. Misconception refutation aims to do precisely this by having models first identify misconceptions found in a given response

G Y^ and then correcting those misconceptions. At each round t of the debate procedure, each model i is randomly assigned a response y®. Model i is first prompted to identify a list of misconceptions in the responseyj . Given the list of misconceptions, the model is then asked for both a refutation of the misconception and a corrected version of FIG. 3 shows an example refutation 320 merely for purpose of illustration.

[0094] As compared to directly providing answers, the agents based on LMs are more skilled at evaluating answers. Therefore, by correct responses using the agents, a more accurate response may be achieved. Table 8 shows an example theorem for the misconception refutation.

Table 8

[0095] As can be seen from the above descriptions, results from the theoretical framework are leveraged to improve the efficacy of the debate procedure. In particular, three interventions (modifications to the debate procedure) are proposed. Namely, diversity pruning aims to maximize the mutual information in the responses shared between rounds. This intervention has the added benefit of reducing the severity of the echo chamber effect. Additionally, qualitypruning which aims to maximize the relevance of each shared response is proposed. This intervention improves the likelihood that the debate procedure converges to correct answers.

[0096] Lastly, misconception-refutation directly identifies, and attempts to refute, misconceptions in model responses. This intervention takes inspiration from the fact that LMs are often more skilled at evaluating answers, compared with directly providing answers.

[0097] For each of the interventions, theoretical results are described above to outline the way in which each they improve the debate. Conducted experiments on four common benchmarks demonstrating that these interventions improve debate efficacy in practice.

Example process and device

[0098] FIG. 5 illustrates a flowchart of a method 500 for multi-agent debate in accordance with some example implementations of the present disclosure. The method 500 may be implemented at the conversational platform 1 10 as illustrated in FIG. 1 . At a block 510, a set of responses to a task expressed in a natural language are obtained. The set of responses are generated by a plurality of agents in at least one round of a debate for the task. At a block 520, the set of responses are updated by at least one of selecting responses from the set of responses, or adjusting at least one response in the set of responses. At a block 530, a result for the task is generated by providing the updated set of responses to the plurality of agents as an input for a subsequent round of the debate.

[0099] In some embodiments, selecting responses from the set of responses comprises: determining a quality metric for each response of a first plurality of responses based on the task and the response, the first plurality of responses comprised in the set of responses, the quality metric representing a relevance of the response to the task; and selecting a first number of responses from the first plurality of responses based on the quality metric for each response of the first plurality of responses.

[00100] In some embodiments, determining a quality metric for each response of a plurality of responses based on the task and the response comprises: generating a first representation for the task and a second representation for the response by an embedding model for the natural language; and determining a similarity between the first and second representations as the quality metric for the response.

[00101] In some embodiments, selecting responses from the set of responses comprises: determining a diversity metric for each pair of responses among a second plurality of responses based on the pair of responses, the second plurality of responses comprised in the set of responses, the diversity metric representing mutual information for the pair of responses; and selecting a second number of responses from the second plurality of responses based on the diversity metric for each pair of responses.

[00102] In some embodiments, determining a diversity metric for each pair of responses among a second plurality of responses based on the pair of responses comprises: generating a pair of representations for the pair of responses by an embedding model for the natural language; and determining the diversity metric for the pair of responses based on a similarity between the pair of representations.

[00103] In some embodiments, adjusting at least one response in the set of responses comprises: assigning a target agent of the plurality of agents for a given response in the set of response, the given response not generated by the target agent; providing, to the target agent, prompt information indicating to correct the given response; and deriving an adjusted response for the given response from an output of the target agent. [00104] In some embodiments, updating the set of responses comprises: selecting, from the set of responses, a first subset of responses based on a relevance of each response in the set of responses to the task, selecting, from the first subset of responses, a second subset of responses based on mutual information for each pair of responses in the first subset of responses; and determining the updated set of responses based on the second subset of responses

|00105| In some embodiments, determining the updated set of responses based on the second subset of responses comprises: for a response in the second subset of responses, adjusting the response by prompting a target agent of the plurality of agents to correct the response, the response not generated by the target agent; and determining the adjusted response as one of the updated set of responses.

|00106| In some embodiments, the set of responses comprise responses generated by the plurality of agents in a plurality of previous round of debate.

[00107] In some implementations of the present disclosure, there is provided an electronic device, comprising a computer processor coupled to a computer-readable memory unit, the memory unit comprising instructions that when executed by the computer processor implements a method of multi-agent debate. The method comprises: obtaining a set of responses to a task expressed in a natural language, the set of responses generated by a plurality of agents in at least one round of a debate for the task; updating the set of responses by at least one of selecting responses from the set of responses, or adjusting at least one response in the set of responses; and generating a result for the task by providing the updated set of responses to the plurality of agents as an input for a subsequent round of the debate.

[00108] In some embodiments, selecting responses from the set of responses comprises: determining a quality metric for each response of a first plurality of responses based on the task and the response, the first plurality of responses comprised in the set of responses, the quality metric representing a relevance of the response to the task; and selecting a first number of responses from the first plurality of responses based on the quality metric for each response of the first plurality of responses.

[00109] In some embodiments, determining a quality metric for each response of a plurality of responses based on the task and the response comprises: generating a first representation for the task and a second representation for the response by an embedding model for the natural language;

T1 and determining a similarity between the first and second representations as the quality metric for the response.

[00110] In some embodiments, selecting responses from the set of responses comprises: determining a diversity metric for each pair of responses among a second plurality of responses based on the pair of responses, the second plurality of responses comprised in the set of responses, the diversity metric representing mutual information for the pair of responses; and selecting a second number of responses from the second plurality of responses based on the diversity metric for each pair of responses.

[00111] In some embodiments, determining a diversity metric for each pair of responses among a second plurality of responses based on the pair of responses comprises: generating a pair of representations for the pair of responses by an embedding model for the natural language; and determining the diversity metric for the pair of responses based on a similarity between the pair of representations.

[00112] In some embodiments, adjusting at least one response in the set of responses comprises: assigning a target agent of the plurality of agents for a given response in the set of response, the given response not generated by the target agent; providing, to the target agent, prompt information indicating to correct the given response; and deriving an adjusted response for the given response from an output of the target agent.

[00113] In some embodiments, updating the set of responses comprises: selecting, from the set of responses, a first subset of responses based on a relevance of each response in the set of responses to the task; selecting, from the first subset of responses, a second subset of responses based on mutual information for each pair of responses in the first subset of responses; and determining the updated set of responses based on the second subset of responses.

[00114] In some embodiments, determining the updated set of responses based on the second subset of responses comprises: for a response in the second subset of responses, adjusting the response by prompting a target agent of the plurality of agents to correct the response, the response not generated by the target agent; and determining the adjusted response as one of the updated set of responses.

[00115] In some embodiments, the set of responses comprise responses generated by the plurality of agents in a plurality of previous round of debate. [00116] In some embodiments of the present disclosure, there is provided a non-transitory computer program product, the non-transitory computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by an electronic device to cause the electronic device to perform a method of multi-agent debate. The method comprising: obtaining a set of responses to a task expressed in a natural language, the set of responses generated by a plurality of agents in at least one round of a debate for the task; updating the set of responses by at least one of selecting responses from the set of responses, or adjusting at least one response in the set of responses; and generating a result for the task by providing the updated set of responses to the plurality of agents as an input for a subsequent round of the debate. In some implementations of the present disclosure, the method further comprises other steps as described in the present disclosure.

[00117] FIG. 6 illustrates a block diagram of an electronic device 600 in which various embodiments of the present disclosure can be implemented. It would be appreciated that the electronic device 600 shown in FIG. 6 is merely for purpose of illustration, without suggesting any limitation to the functions and scopes of the present disclosure in any manner. The electronic device 600 may be used to implement the above method 600. As shown in FIG. 6, the electronic device 600 may be a general-purpose electronic device. The electronic device 600 may at least comprise one or more processors or processing units 610, a memory 620, a storage unit 630, one or more communication units 640, one or more input devices 650, and one or more output devices 660.

[00118] The processing unit 610 may be a physical or virtual processor and can implement various processes based on programs 625 stored in the memory 620. In a multi-processor system, multiple processing units execute computer executable instructions in parallel so as to improve the parallel processing capability of the electronic device 600. The processing unit 610 may also be referred to as a central processing unit (CPU), a microprocessor, a controller, or a microcontroller.

[00119] The electronic device 600 typically includes various computer storage medium. Such medium can be any medium accessible by the electronic device 600, including, but not limited to, volatile and non-volatile medium, or detachable and non -detach able medium. The memory 620 can be a volatile memory (for example, a register, cache, Random Access Memory (RAM)), a non-volatile memory (such as a Read-Only Memory (ROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), or a flash memory), or any combination thereof. The storage unit 630 may be any detachable or non-detachable medium and may include a machine-readable medium such as a memory, flash memory drive, magnetic disk, or another other media, which can be used for storing information and/or data and can be accessed in the electronic device 600.

|00120| The electronic device 600 may further include additional detachable/ non-detachable, volatile/non-volatile memory medium. Although not shown in FIG. 6, it is possible to provide a magnetic disk drive for reading from and/or writing into a detachable and non-volatile magnetic disk and an optical disk drive for reading from and/or writing into a detachable non-volatile optical disk. In such cases, each drive may be connected to a bus (not shown) via one or more data medium interfaces.

[00121] The communication unit 640 communicates with a further electronic device via the communication medium. In addition, the functions of the components in the electronic device 600 can be implemented by a single computing cluster or multiple computing machines that can communicate via communication connections. Therefore, the electronic device 600 can operate in a networked environment using a logical connection with one or more other servers, networked personal computers (PCs) or further general network nodes.

[00122] The input device 650 may be one or more of a variety of input devices, such as a mouse, keyboard, tracking ball, voice-input device, and the like The output device 660 may be one or more of a variety of output devices, such as a display, loudspeaker, printer, and the like. By means of the communication unit 640, the electronic device 600 can further communicate with one or more external devices (not shown) such as the storage devices and display device, with one or more devices enabling the user to interact with the electronic device 600, or any devices (such as a network card, a modem, and the like) enabling the electronic device 600 to communicate with one or more other electronic devices, if required. Such communication can be performed via input/output (I/O) interfaces (not shown).

[00123] In some embodiments, instead of being integrated in a single device, some, or all components of the electronic device 600 may also be arranged in cloud computing architecture. In the cloud computing architecture, the components may be provided remotely and work together to implement the functionalities described in the present disclosure. In some embodiments, cloud computing provides computing, software, data access and storage service, which will not require end users to be aware of the physical locations or configurations of the systems or hardware providing these services. In various embodiments, the cloud computing provides the services via a wide area network (such as Internet) using suitable protocols. For example, a cloud computing provider provides applications over the wide area network, which can be accessed through a web browser or any other computing components. The software or components of the cloud computing architecture and corresponding data may be stored on a server at a remote position. The computing resources in the cloud computing environment may be merged or distributed at locations in a remote data center. Cloud computing infrastructures may provide the services through a shared data center, though they behave as a single access point for the users. Therefore, the cloud computing architectures may be used to provide the components and functionalities described herein from a service provider at a remote location. Alternatively, they may be provided from a conventional server or installed directly or otherwise on a client device.

[00124] The functionalities described herein can be performed, at least in part, by one or more hardware logic components. For example, and without limitation, illustrative types of hardware logic components that can be used include Field-Programmable Gate Arrays (FPGAs), Application-specific Integrated Circuits (ASICs), Application-specific Standard Products (ASSPs), System-on-a-chip systems (SOCs), Complex Programmable Logic Devices (CPLDs), and the like.

[00125] Program code for carrying out the methods of the subject matter described herein may be written in any combination of one or more programming languages. The program code may be provided to a processor or controller of a general-purpose computer, special purpose computer, or other programmable data processing apparatus such that the program code, when executed by the processor or controller, causes the functions/operations specified in the flowcharts and/or block diagrams to be implemented. The program code may be executed entirely or partly on a machine, executed as a stand-alone software package partly on the machine, partly on a remote machine, or entirely on the remote machine or server.

[00126] In the context of this disclosure, a machine-readable medium may be any tangible medium that may contain or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium may be a machine- readable signal medium or a machine-readable storage medium. A machine-readable medium may include but not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium would include an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random-access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

|00127| Further, while operations are illustrated in a particular order, this should not be understood as requiring that such operations are performed in the particular order shown or in sequential order, or that all illustrated operations are performed to achieve the desired results. In certain circumstances, multitasking and parallel processing may be advantageous Likewise, while several specific implementation details are contained in the above discussions, these should not be construed as limitations on the scope of the subject matter described herein, but rather as descriptions of features that may be specific to particular embodiments. Certain features that are described in the context of separate embodiments may also be implemented in combination in a single implementation. Rather, various features described in a single implementation may also be implemented in multiple embodiments separately or in any suitable sub-combination.

[00128] Although the subject matter has been described in language specific to structural features and/or methodological acts, it is to be understood that the subject matter specified in the appended claims is not necessarily limited to the specific features or acts described above Rather, the specific features and acts described above are disclosed as example forms of implementing the claims.

[00129] From the foregoing, it will be appreciated that specific embodiments of the presently disclosed technology have been described herein for purposes of illustration, but that various modifications may be made without deviating from the scope of the disclosure. Accordingly, the presently disclosed technology is not limited except as by the appended claims.

[00130] Embodiments of the subject matter and the functional operations described in the present disclosure can be implemented in various systems, digital electronic circuitry, or in computer software, firmware, or hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them Embodiments of the subject matter described in this specification can be implemented as one or more computer program products, i.e., one or more modules of computer program instructions encoded on a tangible and non-transitory computer readable medium for execution by, or to control the operation of, data processing apparatus. The computer readable medium can be a machine- readable storage device, a machine-readable storage substrate, a memory device, a composition of matter effecting a machine-readable propagated signal, or a combination of one or more of them. The term “data processing unit” or “data processing apparatus” encompasses all apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers. The apparatus can include, in addition to hardware, code that creates an execution environment for the computer program in question, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.

[00131] A computer program (also known as a program, software, software application, script, or code) can be written in any form of programming language, including compiled or interpreted languages, and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program does not necessarily correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files that store one or more modules, sub programs, or portions of code). A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a communication network.

[00132] Processors suitable for the execution of a computer program include, by way of example, both general and special purpose microprocessors, and any one or more processors of any kind of digital computer. Generally, a processor will receive instructions and data from a read only memory or a random access memory or both. The essential elements of a computer are a processor for performing instructions and one or more memory devices for storing instructions and data. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto optical disks, or optical disks. However, a computer need not have such devices. Computer readable media suitable for storing computer program instructions and data include all forms of nonvolatile memory, media, and memory devices, including by way of example semiconductor memory devices, e g., EPROM, EEPROM, and flash memory devices. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

2X [00133] It is intended that the specification, together with the drawings, be considered exemplary only, where exemplary means an example. As used herein, the use of “or” is intended to include “and/or”, unless the context clearly indicates otherwise.

[00134] While the present disclosure contains many specifics, these should not be construed as limitations on the scope of any disclosure or of what may be claimed, but rather as descriptions of features that may be specific to particular embodiments of particular disclosures. Certain features that are described in the present disclosure in the context of separate embodiments can also be implemented in combination in a single implementation. Conversely, various features that are described in the context of a single implementation can also be implemented in multiple embodiments separately or in any suitable sub-combination. Moreover, although features may be described above as acting in certain combinations and even initially claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a sub-combination or variation of a sub-combination.

[00135] Similarly, while operations are illustrated in the drawings in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. Moreover, the separation of various system components in the embodiments described in the present disclosure should not be understood as requiring such separation in all embodiments. Only a few embodiments and examples are described and other embodiments, enhancements and variations can be made based on what is described and illustrated in the present disclosure.

Claims

CLAIMS What is claimed is:

1. A method of multi-agent debate, comprising: obtaining a set of responses to a task expressed in a natural language, the set of responses generated by a plurality of agents in at least one round of a debate for the task; updating the set of responses by at least one of selecting responses from the set of responses, or adjusting at least one response in the set of responses; and generating a result for the task by providing the updated set of responses to the plurality of agents as an input for a subsequent round of the debate.

2 The method of claim 1, wherein selecting responses from the set of responses comprises: determining a quality metric for each response of a first plurality of responses based on the task and the response, the first plurality of responses comprised in the set of responses, the quality metric representing a relevance of the response to the task; and selecting a first number of responses from the first plurality of responses based on the quality metric for each response of the first plurality of responses.

3 The method of claim 2, wherein determining a quality metric for each response of a plurality of responses based on the task and the response comprises: generating a first representation for the task and a second representation for the response by an embedding model for the natural language; and determining a similarity between the first and second representations as the quality metric for the response.

4 The method of claim 1, wherein selecting responses from the set of responses comprises: determining a diversity metric for each pair of responses among a second plurality of responses based on the pair of responses, the second plurality of responses comprised in the set of responses, the diversity metric representing mutual information for the pair of responses; and selecting a second number of responses from the second plurality of responses based on the diversity metric for each pair of responses.

5 The method of claim 4, wherein determining a diversity metric for each pair of responses among a second plurality of responses based on the pair of responses comprises: generating a pair of representations for the pair of responses by an embedding model for the natural language; and determining the diversity metric for the pair of responses based on a similarity between the pair of representations.

6. The method of claim 1, wherein adjusting at least one response in the set of responses comprises: assigning a target agent of the plurality of agents for a given response in the set of response, the given response not generated by the target agent; providing, to the target agent, prompt information indicating to correct the given response; and deriving an adjusted response for the given response from an output of the target agent.

7. The method of claim 1 , wherein updating the set of responses comprises: selecting, from the set of responses, a first subset of responses based on a relevance of each response in the set of responses to the task; selecting, from the first subset of responses, a second subset of responses based on mutual information for each pair of responses in the first subset of responses; and determining the updated set of responses based on the second subset of responses.

8. The method of claim 7, wherein determining the updated set of responses based on the second subset of responses comprises: for a response in the second subset of responses, adjusting the response by prompting a target agent of the plurality of agents to correct the response, the response not generated by the target agent; and determining the adjusted response as one of the updated set of responses.

9. The method of claim 1 , wherein the set of responses comprise responses generated by the plurality of agents in a plurality of previous rounds of the debate.

10. An electronic device, comprising a computer processor coupled to a computer-readable memory unit, the memory unit comprising instructions that when executed by the computer processor implements a method of multi-agent debate, the method comprising: obtaining a set of responses to a task expressed in a natural language, the set of responses generated by a plurality of agents in at least one round of a debate for the task; updating the set of responses by at least one of selecting responses from the set of responses, or adjusting at least one response in the set of responses; and generating a result for the task by providing the updated set of responses to the plurality of agents as an input for a subsequent round of the debate.

11. The device of claim 10, wherein selecting responses from the set of responses comprises: determining a quality metric for each response of a first plurality of responses based on the task and the response, the first plurality of responses comprised in the set of responses, the quality metric representing a relevance of the response to the task; and selecting a first number of responses from the first plurality of responses based on the quality metric for each response of the first plurality of responses

12. The device of claim 11, wherein determining a quality metric for each response of a plurality of responses based on the task and the response comprises: generating a first representation for the task and a second representation for the response by an embedding model for the natural language; and determining a similarity between the first and second representations as the quality metric for the response.

13. The device of claim 10, wherein selecting responses from the set of responses comprises: determining a diversity metric for each pair of responses among a second plurality of responses based on the pair of responses, the second plurality of responses comprised in the set of responses, the diversity metric representing mutual information for the pair of responses, and selecting a second number of responses from the second plurality of responses based on the diversity metric for each pair of responses.

14. The device of claim 13, wherein determining a diversity metric for each pair of responses among a second plurality of responses based on the pair of responses generating a pair of representations for the pair of responses by an embedding model for the natural language; and determining the diversity metric for the pair of responses based on a similarity between the pair of representations.

15. The device of claim 10, wherein adjusting at least one response in the set of responses comprises: assigning a target agent of the plurality of agents for a given response in the set of response, the given response not generated by the target agent; providing, to the target agent, prompt information indicating to correct the given response; and deriving an adjusted response for the given response from an output of the target agent.

16 The device of claim 10, wherein updating the set of responses comprises: selecting, from the set of responses, a first subset of responses based on a relevance of each response in the set of responses to the task; selecting, from the first subset of responses, a second subset of responses based on mutual information for each pair of responses in the first subset of responses; and determining the updated set of responses based on the second subset of responses.

17 The device of claim 16, wherein determining the updated set of responses based on the second subset of responses comprises: for a response in the second subset of responses, adjusting the response by prompting a target agent of the plurality of agents to correct the response, the response not generated by the target agent; and determining the adjusted response as one of the updated set of responses.

18 The device of claim 10, wherein the set of responses comprise responses generated by the plurality of agents in a plurality of previous round of debate.

19. A computer program product, the computer program product comprising a non- transitory computer readable storage medium having program instructions embodied therewith, the program instructions executable by an electronic device to cause the electronic device to perform a method of multi-agent debate, the method comprising: obtaining a set of responses to a task expressed in a natural language, the set of responses generated by a plurality of agents in at least one round of a debate for the task; updating the set of responses by at least one of selecting responses from the set of responses, or adjusting at least one response in the set of responses; and generating a result for the task by providing the updated set of responses to the plurality of agents as an input for a subsequent round of the debate.

20. The computer program product of claim 19, wherein selecting responses from the set of responses comprises: determining a quality metric for each response of a first plurality of responses based on the task and the response, the first plurality of responses comprised in the set of responses, the quality metric representing a relevance of the response to the task; and selecting a first number of responses from the first plurality of responses based on the quality metric for each response of the first plurality of responses