WO2024251170A1 - 人机交互的数据处理方法及服务器 - Google Patents

人机交互的数据处理方法及服务器 Download PDF

Info

Publication number
WO2024251170A1
WO2024251170A1 PCT/CN2024/097611 CN2024097611W WO2024251170A1 WO 2024251170 A1 WO2024251170 A1 WO 2024251170A1 CN 2024097611 W CN2024097611 W CN 2024097611W WO 2024251170 A1 WO2024251170 A1 WO 2024251170A1
Authority
WO
WIPO (PCT)
Prior art keywords
response
quality
model
instruction
large model
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2024/097611
Other languages
English (en)
French (fr)
Inventor
张一昌
刘高
韩矞
马坚鑫
林俊旸
周畅
周靖人
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Alibaba China Co Ltd
Original Assignee
Alibaba China Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Alibaba China Co Ltd filed Critical Alibaba China Co Ltd
Publication of WO2024251170A1 publication Critical patent/WO2024251170A1/zh
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/30—Monitoring
    • G06F11/34—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
    • G06F11/3409—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment for performance assessment
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/30—Monitoring
    • G06F11/3003—Monitoring arrangements specially adapted to the computing system or computing system component being monitored
    • G06F11/302—Monitoring arrangements specially adapted to the computing system or computing system component being monitored where the computing system component is a software system
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
    • G06F18/217—Validation; Performance evaluation; Active pattern learning techniques
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F2201/00—Indexing scheme relating to error detection, to error correction, and to monitoring
    • G06F2201/865—Monitoring of software

Definitions

  • the present disclosure relates to computer technology, and in particular to a human-computer interactive data processing method and a server.
  • Natural language is an important carrier of human logic and thinking, and has a very important significance in human-computer interaction and even in the field of general artificial intelligence.
  • due to the complexity and ambiguity of natural language there has been a lack of machine facilities that directly face unconstrained natural language.
  • Large models are widely used in human-computer interaction in the field of natural language processing.
  • Large models refer to large-scale deep learning models, such as large-scale language models and multimodal models, with large-scale model parameters, usually containing hundreds of millions, tens of billions, hundreds of billions, trillions or even more than 10 trillion model parameters.
  • the performance of the model is usually generally scored based on simple dimensions such as whether the response output by the big model is helpful to the user and whether the response content is safe.
  • the evaluation dimension is single and it is impossible to accurately and comprehensively evaluate the response quality of the big model. It is not conducive to selecting high-quality models in model iteration and controlling the quality of the online model, resulting in poor quality of human-computer interaction.
  • the present invention discloses a human-computer interaction data processing method and server, which are used to solve the problem that the response quality of a large model cannot be accurately and comprehensively evaluated, which is not conducive to selecting high-quality models and controlling the quality of online models in model iteration, resulting in poor quality of human-computer interaction.
  • the present disclosure provides a data processing method for human-computer interaction, comprising:
  • the response quality information of the large model is output, and the response quality information of the large model is set to guide the online determination of the large model, or to update the optimized version of the large model, or to select a high-quality target large model.
  • the present disclosure provides a human-computer interaction data processing method, which is applied to a server and includes:
  • the interactive interface data includes a response result of the instruction output by each of the language models
  • the present disclosure provides a human-computer interaction data processing method, which is applied to a terminal device, including:
  • the interactive interface data including response results output by each of the language models, the response results being generated by: obtaining a human-computer interaction instruction, inputting the instruction into each of the language models, and outputting a response result of the instruction through each of the language models;
  • response quality information and relative evaluation information of each of the language models Receiving response quality information and relative evaluation information of each of the language models, wherein the response quality information of each of the language models is generated by evaluating the response results output by each of the language models from the dimensions of harmfulness, instruction intent coverage, factuality, and content quality, and the relative evaluation information of each of the language models is calculated based on the ranking results of the response results of the instructions output by each of the language models;
  • the present disclosure provides a server, comprising: a processor, and a memory communicatively connected to the processor; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory to implement the method described in the first aspect or the second aspect.
  • the data processing method and server for human-computer interaction obtained human-computer interaction instructions, input the instructions into a large model for realizing human-computer interaction, and output the response results of the instructions through the large model; evaluate the response results output by the large model from the dimensions of harmfulness, instruction intention coverage, factuality, and content quality, and obtain response quality information of the response results in each dimension; calculate the response quality information of the large model based on the response quality information of the response results output by the large model in each dimension, and realize the evaluation of the response quality of the large model from the dimensions of harmfulness, instruction intention coverage, factuality, and content quality.
  • the big model's response results are accurately, comprehensively, and more fine-grainedly evaluated from multiple dimensions such as authenticity and content quality, and the big model's response quality information is output.
  • the big model's response quality information is set to guide the big model's online decision, or to update the optimized version of the big model, or to select a high-quality target big model. It can accurately select high-quality models, improve the quality of iteratively updated/selected big models, and improve the quality of online models, thereby improving the accuracy of human-computer dialogue based on the big model and ensuring the quality of human-computer interaction.
  • FIG1 is a schematic diagram of an example system architecture to which the present disclosure is applicable.
  • FIG2 is a flow chart of a data processing method for human-computer interaction provided by an exemplary embodiment of the present disclosure
  • FIG3 is an example diagram of a first interactive interface provided by an exemplary embodiment of the present disclosure.
  • FIG4 is an exemplary diagram of a second interactive interface provided by an exemplary embodiment of the present disclosure.
  • FIG5 is a flow chart of a data processing method for human-computer interaction provided by an exemplary embodiment of the present disclosure
  • FIG. 6 is a schematic diagram of the structure of a server provided in an embodiment of the present disclosure.
  • Instructions refers to natural language text with a certain intention. In the human-computer interaction scenario, it refers to the questions given by the user.
  • Response result refers to the reply information output for the command.
  • Visual question answering task Given an input image and a question, determine the answer to the question from the visual information of the input image.
  • Image description task Generate description text for an input image.
  • Visual entailment task predict the semantic relevance of an input image and text, i.e., entailment, neutral, or contradiction.
  • Referential expression and comprehension task locate the image area in the input image corresponding to the input text based on the input text.
  • Image generation task Generate an image based on the input description text.
  • Text-based sentiment classification task predict the sentiment classification information of the input text.
  • Text summarization task Generate summary information of the input text.
  • Multimodal tasks refers to downstream tasks whose input and output data involve multiple modal data such as images and text, such as visual question answering tasks, image description tasks, visual implication tasks, referential expression and understanding tasks, image generation tasks, etc.
  • Multimodal pre-trained model refers to a pre-trained model whose input and output data involve multiple modal data such as images and text. After fine-tuning and training, it can be applied to multimodal task processing.
  • a large model refers to a deep learning model with large-scale model parameters, usually containing hundreds of millions, tens of billions, hundreds of billions, trillions, or even more than 10 trillion model parameters.
  • a large model can also be called a foundation model. It is pre-trained with large-scale unlabeled corpus to produce a pre-trained model with more than 100 million parameters. This model can adapt to a wide range of downstream tasks and has good generalization capabilities, such as large-scale language models (LLM) and multi-modal pre-training models.
  • LLM large-scale language models
  • the big model can be widely used in natural language processing (NLP), computer vision and other fields. Specifically, it can be applied to computer vision tasks such as visual question answering (VQA), image caption (IC), image generation, as well as natural language processing tasks such as text-based sentiment classification, text summary generation, and machine translation.
  • NLP natural language processing
  • VQA visual question answering
  • IC image caption
  • natural language processing tasks such as text-based sentiment classification, text summary generation, and machine translation.
  • the main application scenarios of the big model include digital assistants, intelligent robots, search, online education, office software, e-commerce, intelligent design, etc.
  • the big model When applied to human-computer interaction scenarios (such as intelligent robots), the big model generates responses based on the instructions given by the user. During the iteration process of the big model, it is necessary to evaluate the pros and cons of different versions of the big model in order to implement iterative updates of the big model. Before the big model goes online, it is necessary to evaluate whether the performance of the big model meets the online requirements, and to put online big models with excellent performance and avoid putting online big models with poor performance.
  • the performance of the model is usually generally scored based on simple dimensions such as whether the response is helpful to the user and whether the response content is safe. The evaluation dimension is single, and it is impossible to accurately and comprehensively evaluate the pros and cons of the performance of the big model, which is not conducive to selecting high-quality models in model iteration and controlling the quality of the online model.
  • the present disclosure provides a data processing method for human-computer interaction, which obtains human-computer interaction instructions, inputs the instructions into a big model that realizes human-computer interaction, and outputs the response results of the instructions through the big model; evaluates the response results output by the big model from the dimensions of harmfulness, instruction intention coverage, factuality, and content quality, and obtains response quality information of the response results in each dimension; calculates the response quality information of the big model based on the response quality information of the response results output by the big model in each dimension, thereby accurately, comprehensively, and more fine-grainedly evaluating the response results of the big model from multiple dimensions such as harmfulness, instruction intention coverage, factuality, and content quality, and outputs the response quality information of the big model.
  • the response quality information of the big model is set to guide the online judgment of the big model, or update the optimized version of the big model, or select a high-quality target big model, so as to accurately select a high-quality model, improve the quality of the iteratively updated/selected big model, and improve the quality of the online model, thereby improving human-computer interaction based on the big model.
  • the accuracy of the conversation ensures the quality of human-computer interaction.
  • FIG1 is a schematic diagram of an example system architecture applicable to the present disclosure.
  • the system architecture includes a first server responsible for evaluating a large model, a second server running the large model, and an end-side device.
  • a communication link between the first server and the second server which can realize a communication connection between the first server and the second server.
  • There is a communication link between the first server and the end-side device which can realize a communication connection between the first server and the end-side device.
  • the second server can be a server cluster deployed in the cloud, or a local device with computing power.
  • the second server is responsible for running the large model that implements human-computer interaction and generating response results based on the given human-computer interaction instructions.
  • One or more large models can be deployed on a second server, and multiple large models to be evaluated can be deployed on one or more second servers.
  • the terminal device is an electronic device used by the user, which can be a hardware device with network communication function, computing function and information display function, including but not limited to smart phones, tablet computers, desktop computers, servers, etc.
  • the user sends a large model evaluation request to the first server through the terminal device, and the evaluation request contains information of one or more large models to be evaluated.
  • the first server can be a server cluster deployed in the cloud, or a local device with computing power.
  • the first server is responsible for executing the data processing method for human-computer interaction provided by the present disclosure to realize the evaluation of the response quality of the large model, generate the response quality information of the large model, and guide the online determination of the large model, or update the optimized version of the large model, or select a high-quality target large model.
  • the response quality information of the large model indicates the response quality of the response result given by the large model to the input instruction, which is an evaluation value of the response quality of the response result of the large model.
  • the response quality information of the large model specifically includes the response quality information of the response result of the large model in multiple dimensions such as harmfulness, instruction intention coverage, factuality, content quality, etc., such as the evaluation scores of the response results in multiple dimensions such as harmfulness, instruction intention coverage, factuality, content quality, etc., which can accurately and comprehensively reflect the response quality of the response result from multiple dimensions.
  • the user before the big model for human-computer interaction is launched, the user sends a response quality evaluation request of the big model to be launched to the first server through the terminal device.
  • the evaluation request contains relevant information of the big model to be evaluated, such as the application program interface for calling the big model, the access address of the big model, etc.
  • the first server responds to the evaluation request, obtains the instruction of human-computer interaction, inputs the instruction into the big model, and outputs the response result of the instruction through the big model; evaluates the response result output by the big model from the dimensions of harmfulness, instruction intention coverage, factuality, and content quality, and obtains the response quality information of the response result in each dimension; calculates the response quality information of the big model based on the response quality information of the response result output by the big model in each dimension, so as to achieve a comprehensive and accurate evaluation of the response quality of the big model.
  • the response quality information of the large model can be set to guide the online decision of the large model.
  • the first server determines whether the large model meets the online conditions according to the response quality information of the large model, and outputs the online prompt information of the large model, and the online prompt information indicates whether the large model meets the online conditions.
  • the device may also send the response quality information of the large model to the end-side device.
  • the end-side device outputs the response quality information of the large model to guide the user to determine whether the large model meets the online conditions; or, the end-side device determines whether the large model meets the online conditions based on the response quality information of the large model, and outputs the online prompt information of the large model, and the online prompt information indicates whether the large model meets the online conditions.
  • the obtained new version is evaluated.
  • the user can send a response quality evaluation request of the new version of the big model to the first server through the end-side device.
  • the evaluation request contains relevant information of the new version of the big model, such as the application program interface for calling the big model, the access address of the big model, etc.
  • the first server responds to the evaluation request, obtains the instruction of human-computer interaction, inputs the instruction into the big model, and outputs the response result of the instruction through the big model; evaluates the response result output by the big model from the dimensions of harmfulness, instruction intention coverage, factuality, and content quality, and obtains the response quality information of the response result in each dimension; calculates the response quality information of the big model based on the response quality information of the response result output by the big model in each dimension, so as to achieve a comprehensive and accurate evaluation of the response quality of the new version of the big model.
  • the response quality information of the large model can be set to guide the update of the optimized version of the large model.
  • the first server compares the response quality information of the new version and the previous version based on the response quality information of the new version of the large model and the response quality information of the previous version of the large model to obtain a comparison result, and the comparison result is set to guide the update of the optimized version of the large model.
  • the first server can send the comparison result to the end-side device.
  • the end-side device outputs the comparison result of the response quality information of different versions of the large model to guide the user to select a better optimized version for iterative update of the large model.
  • the user can select a higher-quality target large model based on the evaluation comparison results of multiple large models to be selected.
  • the user can send a response quality evaluation request for multiple large models to the first server through the terminal device.
  • the evaluation request contains relevant information of multiple large models, such as the application program interface for calling the large model, the access address of the large model, etc.
  • the first server responds to the evaluation request, obtains the instruction of human-computer interaction, inputs the instruction into each large model, and outputs the response result of the instruction through each large model; evaluates the response result output by each large model from the dimensions of harmfulness, instruction intention coverage, factuality, and content quality, and obtains the response quality information of the response result output by each large model in each dimension; calculates the response quality information of each large model based on the response quality information of the response result output by each large model in each dimension, so as to comprehensively and accurately evaluate the response quality of each large model.
  • the first server compares the response quality information of each large model to obtain a comparison result of the response quality information of each large model.
  • the first server sends the comparison result of the response quality information of each large model to the terminal device.
  • the terminal device outputs the comparison result to guide the user to select a large model with better response quality as the target large model selected for use.
  • the terminal device can select a large model with better response quality based on the comparison result of the response quality information of each large model, and download and obtain the large model according to the relevant information of the selected large model, or use the large model to achieve human-computer interaction.
  • FIG2 is a flow chart of a data processing method for human-computer interaction provided by an exemplary embodiment of the present disclosure.
  • the execution subject of this embodiment is the first server in the aforementioned system architecture.
  • the specific steps of the method are as follows:
  • Step S201 obtain the human-computer interaction instruction, input the instruction into the big model, and output the response result of the instruction through the big model.
  • instructions refer to questions asked by human users during the human-computer interaction process. Based on the user's instructions, the human-computer interaction system will output a reply, which is the response result of the instruction.
  • instructions in human-computer interaction can be collected and an instruction set can be constructed by at least one of the following ways: collecting user instructions generated in a human-computer interaction system within a preset historical period, or collecting instructions in the human-computer interaction process from network resources, or manually writing instructions.
  • the instructions input by the user can be question text, or multimodal information including text, images, etc.
  • the large model is set to generate a response result of the instruction based on the instruction given by the user to realize human-computer interaction.
  • the large model as the object to be evaluated can be various language models, multimodal pre-trained models, etc., which are not specifically limited here.
  • the first server may obtain the application program interface/the service interface of the big model, input instructions into the big model by calling the application program interface/the service interface of the big model, and receive a response result of the instruction output by the big model.
  • the first server may obtain access address information of the large model, and the first server sends an instruction execution request to the second server where the large model is located, the request including the instruction to be executed.
  • the second server inputs the instruction into the large model, obtains the response result of the instruction output by the large model, and sends the response result of the instruction to the first server.
  • Step S202 Evaluate the response results output by the large model from the dimensions of harmfulness, instruction intent coverage, factuality, and content quality to obtain response quality information of the response results in each dimension.
  • harmfulness refers to whether the content of the response result is harmful, that is, whether the content of the response result has security risks.
  • the content of the response result given by the large model should be as consistent with human values as possible.
  • the harmless response result (that is, there is no security risk) means that the content of the response result does not violate the risk of human values.
  • multiple security risks of different dimensions/categories can be pre-configured, including but not limited to: content involving various types of threats to social security, content involving threats to world peace, spreading information about illegal organizations, spreading illegal content, and involving illegal acts. If the response result has any of the configured security risks of any dimension/category, the response result can be considered harmful. If the response result does not have all of the configured security risks, the response result can be considered not harmful. Harmful.
  • a command can contain one or more intents.
  • Command intent coverage refers to whether the response result covers the intent contained in the command, which can be divided into full coverage, partial coverage, and no coverage.
  • Factualness refers to whether there are factual errors in the response results, which may include but are not limited to the following situations: there are no factual errors, there are common sense factual errors that are easy to find, there are knowledge factual errors that are difficult to find and require consulting materials or thinking to find, and there are both common sense factual errors and knowledge factual errors.
  • Content quality refers to the quality of the response result content itself in terms of content connotation, layout, format, introduction, repetition, garbled characters, ambiguity, etc., which can be generally expressed as the readability and coherence of the response result content.
  • the response quality of the response results output by the large model is evaluated from multiple dimensions such as harmfulness, instruction intent coverage, factuality, content quality, etc., and response quality information of the response results in each dimension is generated to achieve accurate and comprehensive evaluation of the response quality of the response results from multiple dimensions.
  • Step S203 Calculate the response quality information of the large model according to the response quality information of the response results output by the large model in each dimension.
  • the response quality information of the large model is obtained by comprehensive calculation based on the response quality information of the response results of each instruction given by the large model in each dimension.
  • the response quality information of the response result of any instruction in each dimension can be summed up as the comprehensive response quality information of the response result; the comprehensive response quality information of the response results of each instruction given by the large model can be summed up as the response quality information of the large model.
  • a weight coefficient can be configured for each dimension respectively, and the response quality information of the response result of any instruction in each dimension can be weighted and summed as the comprehensive response quality information of the response result; the comprehensive response quality information of the response results of each instruction given by the large model can be summed as the response quality information of the large model.
  • a weight coefficient can be configured for each dimension respectively. According to the weight coefficient of each dimension, the response quality information of the response results of each instruction given by the large model in the same dimension is weighted and summed as the response quality information of the large model in this dimension. The response quality information of the large model in each dimension can be obtained, and the response quality of the response results of the large model in each dimension can be repeatedly reflected.
  • Step S204 output the response quality information of the large model, where the response quality information of the large model is set to guide the online decision of the large model, or to update the optimized version of the large model, or to select a high-quality target large model.
  • the first server visualizes the response quality information of the large model to output the evaluation result of the response quality of the large model to the user.
  • the response quality information of the large model can guide the user to make a decision whether to put the large model online; or, by comparing the response quality information of multiple large model versions, determine the high-quality large model version and perform iterative optimization of the large model; or, by comparing multiple
  • the response quality information of the large models is collected, and the target large model with higher response quality is selected as the large model used to realize human-computer interaction.
  • the first server may determine whether the large model meets the online conditions based on the response quality information of the large model; and output online prompt information of the large model, where the online prompt information indicates whether the large model meets the online conditions.
  • the online condition includes a first threshold of the response quality information of the large model. If the response quality information of the large model is greater than or equal to the first threshold, the large model meets the online condition. Otherwise, the large model does not meet the online condition.
  • the first threshold in the online condition can be customized by the user according to the needs of the specific application scenario.
  • the first server may select one of the large models with better response quality as the target large model based on the response quality information of the multiple large models by comparing the response quality information of the multiple large models, and output the information of the target large model to the terminal device.
  • the first server may select one version of the big model as the optimized version based on the response quality information of the multiple versions of the big model by comparing the response quality information of the multiple versions of the big model, and update the optimized version of the big model.
  • This embodiment obtains instructions for human-computer interaction, inputs the instructions into a big model that implements human-computer interaction, and outputs response results of the instructions through the big model; evaluates the response results output by the big model from the dimensions of harmfulness, instruction intention coverage, factuality, and content quality, and obtains response quality information of the response results in various dimensions; calculates the response quality information of the big model based on the response quality information of the response results output by the big model in various dimensions, thereby accurately, comprehensively, and more fine-grainedly evaluating the response results of the big model from multiple dimensions such as harmfulness, instruction intention coverage, factuality, and content quality, and outputs the response quality information of the big model.
  • the response quality information of the big model is set to guide the online decision of the big model, or to update the optimized version of the big model, or to select a high-quality target big model, so as to accurately select a high-quality model, improve the quality of the iteratively updated/selected big model, and improve the quality of the online model, thereby improving the accuracy of human-computer dialogue based on the big model and ensuring the quality of human-computer interaction.
  • multiple quality categories are configured to correspond to any dimension, and different quality categories correspond to different response quality information.
  • the quality category refers to the classification of the response quality of any evaluation dimension into multiple different quality levels (quality levels), and different quality categories represent different levels/quality levels of response quality, corresponding to different response quality information (such as evaluation scores).
  • quality levels quality levels
  • different quality categories represent different levels/quality levels of response quality, corresponding to different response quality information (such as evaluation scores).
  • harmfulness may include the following quality categories: no harm, harmful.
  • harmful means that the response result has at least one dimension/category of security risk.
  • multiple different dimensions/categories of security risks can be pre-configured, including but not limited to: involving various types of content that endangers social security, involving content that endangers world peace, spreading information about illegal organizations, spreading illegal content, and involving illegal acts.
  • the big model should refuse to respond.
  • the big model should respond according to the intention of the instruction. For example, when evaluating the response results of the big model, the response results of the big model can be marked as "no harm”. Harm” or “has harm” is used to evaluate the response quality of the large model response results in the harm dimension.
  • the coverage of instruction intent may include the following quality categories: fully identifying instruction intent, partially identifying instruction intent, failing to identify instruction intent, and rejecting instruction intent that should not be rejected.
  • fully identifying instruction intent means that the response result covers all intents contained in the instruction.
  • Partially identifying instruction intent means that the instruction contains multiple intents, and the response result covers at least one intent contained in the instruction, but does not cover all intents.
  • Failure to identify instruction intent means that the response result does not cover any intent contained in the instruction.
  • Rejection of instruction intent that should not be rejected means that the instruction should be harmless and should be responded to the intent contained in the instruction, but the response result given by the large model refuses to reply to the instruction.
  • the response quality of the large model response result in the dimension of covering instruction intent is evaluated.
  • Factualness can include the following quality categories: no factual errors, common sense factual errors, knowledge factual errors, and common sense and knowledge factual errors at the same time.
  • No factual errors refer to the situation where there are no factual errors in the response results.
  • Common sense factual errors refer to the situation where there are common sense factual errors in the response results that are relatively easy to find.
  • Knowledge factual errors refer to the situation where there are knowledge factual errors in the response results that are difficult to find and require reference to materials or thinking and reasoning to find.
  • the response quality of the large model response results in the factual dimension is evaluated by marking the response results of the large model as one of the quality categories of "no factual errors”, “common sense factual errors”, “knowledge factual errors”, and “common sense and knowledge factual errors at the same time” according to whether there are factual errors in the response results of the large model.
  • Content quality can include the following quality categories: good coherence, medium coherence, and poor coherence.
  • the coherence of the response results can be comprehensively evaluated from the aspects of content connotation, typesetting, format, introduction, repetition, garbled characters, ambiguity, and grammar.
  • the coherence of the response results is determined based on the quality of the response results of the large model in terms of content connotation, typesetting, format, introduction, repetition, garbled characters, ambiguity, and grammar.
  • the response quality of the large model response results in the content quality dimension is evaluated.
  • the aforementioned step S202 can be specifically implemented in the following manner: for each dimension, determine the quality category of the response result output by the large model in each dimension; and determine the response quality information of the response result in each dimension according to the quality category of the response result in each dimension.
  • any dimension corresponds to multiple quality categories, and different quality categories correspond to different response quality information.
  • the quality category of the response result output by the large model in that dimension is determined, and the response quality information corresponding to the quality category of the response result in that dimension is used as the response quality information of the response result in that dimension. In this way, the response quality information of the response result in each dimension can be determined.
  • the quality category of the response result in each dimension can be manually labeled by a labeling staff in the interactive interface through the interactive interface.
  • the body can be implemented as follows:
  • the response result and the quality category corresponding to each dimension are displayed through the first interactive interface, and an input area for the quality category of the response result in each dimension is provided.
  • the annotator can view the response result displayed in the first interactive interface and enter the quality category of the response result in each dimension in the input area.
  • the quality category of the response result in each dimension entered in the input area is obtained.
  • the first interactive interface may display one or more response results, different response results correspond to different input areas, and the annotator annotates the quality category of the response result in each dimension in the corresponding input area of the response result.
  • all quality categories of each dimension may be displayed in the input area.
  • the annotator selects the quality category of the response result in each dimension in the input area.
  • all quality categories of each dimension and the input box corresponding to each dimension may be displayed in the input area, and the annotator enters the quality category of the response result in the dimension in the input box of each dimension.
  • FIG3 shows an example of the first interactive interface, taking the five different response results given by five large models for the same instruction "If I want to meet with colleagues in London at 10 pm Beijing time, what time is the local time in London?" as an example, as shown in FIG3, the response results given by each large model are displayed separately in the first interactive interface (for example, the area with a dotted box in FIG3 is the area where the response results are displayed, and the dotted box is not displayed in the interface), but the correspondence between the response results and the large models is not displayed, so that the annotator can perform a more objective evaluation without knowing which large model the response results come from.
  • FIG3 shows five response results side by side horizontally, and the quality category of each dimension is displayed in the area below each response result, and a check box is provided. The annotator specifies the quality category of the response result in each dimension by checking the box.
  • a pre-trained classification recognition model may be used to automatically analyze the category of the response result in each dimension.
  • the harmfulness classification recognition model is used to identify the quality category of the response result in the harmfulness dimension
  • the intention coverage classification recognition model is used to identify the quality category of the response result in the instruction intention coverage dimension
  • the factuality classification recognition model is used to identify the quality category of the response result in the factuality dimension
  • the content quality classification recognition model is used to identify the quality category of the response result in the content quality dimension.
  • the input of the harm classification and recognition model is the response result, and the response result is classified according to whether there are various types of safety risks. If it is determined that the response result has at least one safety risk according to the classification result, the response result is determined to be harmful. According to the classification result, if it is determined that the response result does not have any type of safety risk, the response result is determined to be harmless.
  • the harm classification and recognition model can be obtained by training the classification model using corpus annotated with safety risk categories.
  • the input of the intention coverage classification recognition model is the instruction/instruction intention and the response result.
  • the quality category of the instruction intention coverage of the response result is classified and recognized, and the quality category of the response result in the dimension of instruction intention coverage is output.
  • the quantity category can be obtained by training the classification model using data labeled with instruction intention coverage.
  • the input of the factual classification recognition model is the response result. It identifies whether the response result contains common sense fact errors and knowledge fact errors, and determines the quality category of the response result in the factual dimension based on the recognition result. Among them, identifying whether the response result contains common sense fact errors and identifying whether the response result contains knowledge fact errors can be implemented using two models. The two models are trained using data labeled with whether there are common sense fact errors and whether there are knowledge fact errors, respectively. The two models can be isomorphic but do not share parameters.
  • the input of the content quality classification recognition model is the response result, and the output is the quality category of the response result in the content quality dimension.
  • the classification model can be trained using data with content quality classification annotation information.
  • the quality category of the response information in each dimension can be determined based on the response information of a single model, and then the quality category of the multi-modal response information can be combined to determine the quality category of the response result in each dimension.
  • the response result includes text and pictures.
  • the factual dimension it is determined that the text has common sense factual errors, and the picture has no factual errors. It can be determined that the response result has common sense factual errors.
  • the aforementioned step S203 calculates the response quality information of the large model according to the response quality information of the response results output by the large model in each dimension, which can be specifically implemented in the following manner:
  • the comprehensive quality information of each response result is calculated; according to the comprehensive quality information of each response result output by the large model, the response quality information of the large model is calculated.
  • the weight coefficient of each dimension can be configured according to experience or customized by the user.
  • the first server provides a weight configuration interface for each dimension, and the weight configuration interface is configured to configure the weight coefficient of each dimension.
  • the first server obtains the weight coefficient of each dimension configured on the weight configuration interface.
  • the response quality information corresponding to each quality category of each dimension can also be customized by the user.
  • the response result of the instruction after the response result of the instruction is outputted through the large model, the response result can also be outputted, so that the user/annotator can evaluate the comprehensive quality category of the response result and submit the comprehensive quality category.
  • the first server receives the comprehensive quality category annotated for the response result, and the comprehensive quality category includes: good, general, and poor.
  • the comprehensive quality category can be output in the first interactive interface, and a corresponding input area can be provided.
  • the options "Like”, “Dislike”, and “Fair” correspond to the comprehensive quality categories of good, bad, and fair, respectively.
  • the comprehensive quality category of the response result can be specified.
  • the The comprehensive quality information of the response results also includes:
  • the quality information interval corresponding to the comprehensive quality category marked by the response result is used as the quality information interval corresponding to the response result; the response results of the instructions are filtered to remove the instructions corresponding to the response results whose comprehensive quality information is not within the corresponding quality information interval.
  • data that does not match the comprehensive quality feedback results and the evaluation results determined by the response quality information of multiple dimensions can be filtered out, thereby reducing the impact of dirty data on the evaluation results and improving the accuracy of the response quality evaluation of large models.
  • the comprehensive quality category "good” corresponds to the quality information interval [9, 10]
  • the comprehensive quality category "general” corresponds to the quality information interval [6, 9)
  • the comprehensive quality category “poor” corresponds to the quality information interval [0, 6).
  • the received comprehensive quality category is "good”
  • the comprehensive quality information of the response result determined according to the response quality information of the response result in each dimension is 5, which is not within the quality information interval [9, 10] corresponding to the comprehensive quality category "good”.
  • step S201 may be implemented in the following manner:
  • the receiving end device sends a response quality evaluation request to multiple large models; obtains human-computer interaction instructions, inputs the instructions into multiple large models respectively, and obtains the response results of the instructions output by each large model.
  • the response results output by each large model are evaluated from the dimensions of harmfulness, instruction intent coverage, factuality, and content quality to obtain response quality information of the response results in each dimension; in the aforementioned step S203, the response quality information of each large model is calculated based on the response quality information of the response results output by each large model in each dimension.
  • the quality category of the response results output by each large model in each dimension can be determined for each response result given by each large model in response to the same instruction; and the response quality information of the response result in each dimension can be determined based on the quality category of the response result in each dimension.
  • the response results of the same instruction output by each large model and the quality category corresponding to each dimension are displayed through the first interactive interface, and an input area for the quality category of each response result in each dimension is provided (as shown in FIG3 ).
  • the correspondence between the response results and each large model is not displayed on the first interactive interface.
  • the annotator can input the quality category of each response result in each dimension in the input area without knowing which large model the response result comes from.
  • the first server obtains the quality category of each response result in each dimension entered in the input area.
  • the response quality of the response results output by each large model can also be ranked, and the relative evaluation information of each large model can be determined based on the ranking result.
  • the first server can also provide a second interactive interface, through which the various models are output The response result of the output instruction, the correspondence between the response result and the major models is not displayed on the second interactive interface.
  • the annotation personnel can sort the response results of the same instruction displayed on the second interactive interface.
  • the first server receives the sorting results of the response results of the instructions output by the major models specified in the interactive interface; based on the sorting results of the response results of the instructions output by the major models, the relative evaluation information of the response quality of the major models is calculated. Furthermore, the first server can output the response quality information and relative evaluation information of the major models to the terminal device.
  • the second interaction interface and the first interaction interface for sorting the response results may be combined into the same interaction interface, or may be implemented using different interaction interfaces respectively, which is not specifically limited here.
  • FIG4 shows an example of the second interactive interface.
  • different response results correspond to different display areas (such as the areas shown in the dotted boxes in FIG4, which are not displayed in the second interactive interface).
  • the order of each response result can be changed by dragging the position of the display area of each response result.
  • the order of each response result can also be changed by inputting the order value of the response result.
  • the position of the display area of each response result in the second interactive interface will be automatically adjusted as the order value of the response result changes.
  • a sorting algorithm can also be used to automatically sort the response results, which can greatly improve the efficiency of data processing.
  • the first server uses a pre-trained response result sorting algorithm to sort the response results of the same instruction output by each major model.
  • the first server calculates the relative evaluation information of the response quality of each major model based on the sorting results of the response results of the same instruction output by each major model.
  • the first server can output the response quality information and relative evaluation information of each major model to the terminal device.
  • the sorting algorithm can be implemented using any existing method for sorting text based on text quality, which is not specifically limited here.
  • the frequency distribution of the response results output by the large model in each ranking can be calculated as the relative evaluation information of each large model.
  • the win, loss and tie between the two large models can be calculated as the relative evaluation information of each large model.
  • the response results of the two large models A and B to the same instruction if the response result of large model A is ranked before the response result of large model B in the sorting results, then large model A wins. If the response result of large model A is ranked behind the response result of large model B in the sorting results, then large model A loses. If the response result of large model A is tied with the response result of large model B in the sorting results, the two large models are tied.
  • the number of wins, losses and ties of any of the large models can be calculated, or the winning rate of any of the large models can be calculated.
  • the relative quality of the large model among the multiple large models to be compared can be intuitively displayed.
  • the calculation of each large model Based on multiple instructions in the instruction set, the Elo rating of each large model is calculated multiple times and the average is taken as the relative evaluation information of each large model.
  • the average Elo rating of each large model By outputting the average Elo rating of each large model, the relative quality of each large model among the multiple large models to be compared can be intuitively displayed.
  • the first server may select one of the large models as the target large model based on the response quality information and/or relative evaluation information of the large models, and output the information of the target large model to the terminal device.
  • the first server may select one of the large models as the optimized version based on the response quality information and/or relative evaluation information of the large models, and update the optimized version of the large model.
  • FIG5 is a flow chart of a data processing method for human-computer interaction provided by an exemplary embodiment of the present disclosure.
  • the process of comparative evaluation of multiple language models is exemplarily described.
  • the specific steps of the method are as follows:
  • Step S501 The terminal device sends a response quality evaluation request for multiple language models to a first server.
  • multiple language models can be pre-trained language models, which are specifically applied to natural language processing (NLP), computer vision and other fields. Specifically, they can be applied to tasks in the intersection of NLP and computer vision such as visual question answering (VQA), image description (IC), visual entailment (VE), referential expression and understanding (REC), as well as tasks in the field of natural language processing such as text-based sentiment classification tasks and text summarization tasks. They can be applied to various application scenarios such as digital assistants, intelligent robots, search, online education, office software, e-commerce, and intelligent design.
  • VQA visual question answering
  • IC image description
  • VE visual entailment
  • REC referential expression and understanding
  • text-based sentiment classification tasks and text summarization tasks can be applied to various application scenarios such as digital assistants, intelligent robots, search, online education, office software, e-commerce, and intelligent design.
  • Step S502 The first server receives a response quality evaluation request for multiple language models sent by a terminal-side device.
  • Step S503 The first server obtains a human-computer interaction instruction, inputs the instruction into each language model, and outputs a response result of the instruction through each language model.
  • step S201 The specific implementation method of this step is the same as the implementation method of the aforementioned step S201. Please refer to the relevant content in the aforementioned embodiment for details, and will not be repeated here.
  • Step S504 The first server evaluates the response results output by each language model from the dimensions of harmfulness, instruction intent coverage, factuality, and content quality, and generates response quality information of each language model.
  • the specific implementation method of this step is similar to the specific implementation method of the aforementioned steps S202-S203.
  • the large model to be evaluated in step S504 is a plurality of language models specified by the user.
  • the specific implementation method refers to the relevant content in the aforementioned embodiment and will not be repeated here.
  • the response quality of the response results output by each large model is ranked, and the relative evaluation information of each large model is determined based on the ranking result.
  • the first server provides a second interactive interface, and outputs the response results of the instructions output by each large model through the second interactive interface. The correspondence between the response results and the large models is not displayed on the second interactive interface.
  • the annotator can sort the response results of the same instruction displayed on the second interactive interface.
  • the first server receives the ranking results of the response results of the instructions output by each large model specified in the interactive interface; according to the ranking results of the response results of the instructions output by each large model, the relative evaluation information of the response quality of each large model is calculated. Furthermore, the first server can output to the terminal device. Response quality information and relative evaluation information of each major model.
  • Step S505 The first server sends interactive interface data to the terminal device, where the interactive interface data includes response results of the instructions output by each language model.
  • the interactive interface data refers to the data of the second interactive interface used for sorting.
  • the first server provides a second interactive interface, and displays the second interactive interface through the terminal device.
  • the second interactive interface is configured to implement the sorting of the response results of each language model to the same instruction.
  • Figure 4 shows an example of the second interactive interface.
  • the response results output by different models correspond to different display areas, and the sorting of each response result can be changed by dragging the position of the display area of each response result.
  • the sorting of each response result can also be changed by entering the order value of the response result.
  • the position of the display area of each response result in the second interactive interface will automatically adjust as the order value of the response result changes.
  • Step S506 The terminal device receives the interactive interface data sent by the server.
  • Step S507 The terminal device displays an interactive interface according to the interactive interface data, and the interactive interface displays the response results of the instructions output by each language model, but does not display the corresponding relationship between the response results and the language model.
  • FIG4 shows an example of the second interactive interface.
  • the response results output by different models correspond to different display areas, and the order of each response result can be changed by dragging the position of the display area of each response result.
  • the order of each response result can also be changed by inputting the sequence value of the response result.
  • the position of the display area of each response result in the second interactive interface will automatically adjust with the change of the sequence value of the response result.
  • the second interactive interface shown in FIG4 displays the response results given by each large model respectively, but does not display the corresponding relationship between the response results and the large model, so that the labeling personnel can perform a more objective evaluation under the premise of not knowing which large model the response result comes from.
  • Step S508 The terminal device obtains and sends to the server the sorting result of the response results of the instructions output by each language model specified in the interactive interface.
  • the terminal device can obtain the sorting results of the response results of the current instruction output by each language model specified in the interactive interface, and send the sorting results to the first server.
  • Step S509 The first server receives the sorted result of the response results of the instructions output by each language model specified in the interactive interface and sent by the terminal side.
  • Step S510 The first server calculates relative evaluation information of the response quality of each language model according to the ranking results of the response results of the instructions output by each language model.
  • the first server can calculate the frequency distribution of the response results output by the language model in each ranking according to the ranking results of the response results output by each language model, as the relative evaluation information of each language model, the more times the top ranking is, the better the response quality of the language model.
  • the frequency distribution of the response results output by each language model in each ranking the relative quality of any language model among the multiple language models to be compared can be intuitively displayed.
  • the first server can calculate the win, loss and tie between any two language models according to the sorting results of the response results output by each language model, as the relative evaluation information of each language model. For the response results of two language models C and D for the same instruction, if the response result of language model C is ranked before the response result of language model D in the sorting results, then language model C wins. If the response result of language model C is ranked behind the response result of language model D in the sorting results, then language model C loses. If the response result of language model C is tied with the response result of language model D in the sorting results, the two language models are tied.
  • the number of wins, losses and ties of any language model can be calculated, or the winning rate of any language model can be calculated.
  • the number of wins, losses and ties of any language model, or by calculating the winning rate of any language model the relative quality of the language model among the multiple language models to be compared can be intuitively displayed.
  • the first server may calculate the Elo rating of each language model according to the ranking results of the response results of each language model to the same instruction, and calculate the Elo rating of each language model multiple times based on multiple instructions in the instruction set and take the average as the relative evaluation information of each language model.
  • the average of the Elo rating of each language model By outputting the average of the Elo rating of each language model, the relative quality of each language model among the multiple language models to be compared can be intuitively displayed.
  • Step S511 The first server outputs the response quality information and relative evaluation information of each language model to the terminal device.
  • Step S512 The terminal device receives the response quality information and relative evaluation information of each language model.
  • Step S513 The terminal-side device outputs the response quality information and relative evaluation information of each language model.
  • the instructions are input into a large model for realizing human-computer interaction, and the response results of the instructions are output by the large model; the response results output by the large model are evaluated from the dimensions of harmfulness, instruction intention coverage, factuality, and content quality, and the response quality information of the response results in each dimension is obtained; according to the response quality information of the response results output by the large model in each dimension, the response quality information of the large model is calculated, so that the response results of the large model are accurately, comprehensively, and more fine-grainedly evaluated from multiple dimensions such as harmfulness, instruction intention coverage, factuality, and content quality, and the response quality information of the large model is output; and by sorting the response quality of the response results output by each large model, the relative evaluation information of each large model is determined based on the sorting result; combined with the response quality information and relative evaluation information of the large model, the response quality of each large model can be better compared to guide the online decision of the large model, or update the optimized version of the large model
  • FIG6 is a schematic diagram of the structure of a server provided in an embodiment of the present disclosure.
  • the server includes: a memory 601 and a processor 602.
  • the memory 601 is configured to store computer-executable instructions and can be configured to store various other data to support operations on the server.
  • the processor 602 is connected to the memory 601 in communication and is configured to execute the computer-executable instructions stored in the memory 601 to implement the technical solution executed by the first server in any of the above method embodiments.
  • the specific functions and technical effects that can be achieved are similar and will not be described here. Elaborate.
  • the server further includes other components such as a firewall 603, a load balancer 604, a communication component 605, a power supply component 606, etc.
  • Fig. 6 schematically shows some components, which does not mean that the server only includes the components shown in Fig. 6.
  • the embodiment of the present disclosure also provides an end-side device, which includes: a memory and a processor.
  • the memory is configured to store computer-executable instructions and can be configured to store various other data to support operations on the end-side device.
  • the processor is communicatively connected to the memory and is configured to execute the computer-executable instructions stored in the memory to implement the technical solution executed by the end-side device in any of the above method embodiments. Its specific functions and technical effects that can be achieved are similar and will not be repeated here.
  • the embodiments of the present disclosure also provide a computer-readable storage medium, in which computer execution instructions are stored.
  • the computer execution instructions are executed by a processor, they are configured to implement the technical solution executed by the first server in any of the above method embodiments. The specific functions and technical effects that can be achieved are not repeated here.
  • the embodiments of the present disclosure also provide a computer-readable storage medium, in which computer execution instructions are stored.
  • the computer execution instructions are executed by a processor, they are configured to implement the technical solution executed by the terminal device in any of the above method embodiments. The specific functions and technical effects that can be achieved are not repeated here.
  • the embodiments of the present disclosure also provide a computer program product, which includes: a computer program, which is stored in a readable storage medium. At least one processor of an end-side device can read the computer program from the readable storage medium. At least one processor executes the computer program so that the end-side device executes the technical solution executed by the end-side device in any of the above-mentioned method embodiments. The specific functions and technical effects that can be achieved are not repeated here.
  • the embodiment of the present disclosure provides a chip, including: a processing module and a communication interface, the processing module can execute the technical solution of the first server or terminal device in the aforementioned method embodiment.
  • the chip also includes a storage module (such as a memory), the storage module is configured to store instructions, the processing module is configured to execute the instructions stored in the storage module, and the execution of the instructions stored in the storage module enables the processing module to execute the technical solution executed by the first server or terminal device in any of the aforementioned method embodiments.
  • the above-mentioned memory may be an object storage service (OSS).
  • OSS object storage service
  • the above-mentioned memory may be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.
  • SRAM static random access memory
  • EEPROM electrically erasable programmable read-only memory
  • EPROM erasable programmable read-only memory
  • PROM programmable read-only memory
  • ROM read-only memory
  • magnetic memory flash memory
  • flash memory magnetic disk or optical disk.
  • the above-mentioned communication components are configured to facilitate wired or wireless communication between the device where the communication components are located and other devices.
  • the device where the communication components are located can access wireless networks based on communication standards, such as mobile hotspots (WiFi), second-generation mobile communication systems (2G), third-generation mobile communication systems (3G), fourth-generation mobile communication systems (4G)/long-term evolution (LTE), fifth-generation mobile communication systems (5G) and other mobile communication networks, or combinations thereof.
  • the communication component receives a broadcast signal or broadcast related information from an external broadcast management system via a broadcast channel.
  • the communication component also includes a near field communication (NFC) module to facilitate short-range communication.
  • the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.
  • RFID radio frequency identification
  • IrDA infrared data association
  • UWB ultra-wideband
  • Bluetooth Bluetooth
  • the power supply assembly provides power to various components of the device where the power supply assembly is located.
  • the power supply assembly may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device where the power supply assembly is located.
  • embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
  • computer-usable storage media including but not limited to disk storage, CD-ROM, optical storage, etc.
  • each process and/or box in the flowchart and/or block diagram, as well as the combination of the process and/or box in the flowchart and/or block diagram can be implemented by computer program instructions.
  • These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device for implementing the functions specified in one or more processes in the flowchart and/or one or more boxes in the block diagram.
  • These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device that implements the functions specified in one or more processes in the flowchart and/or one or more boxes in the block diagram.
  • These computer program instructions may also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, whereby the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes in the flowchart and/or one or more boxes in the block diagram.
  • a computing device includes one or more processors (CPU), input/output interfaces, network interfaces, and memory.
  • the memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and/or non-volatile memory in the form of read-only memory (ROM) or flash memory (flash RAM).
  • RAM random access memory
  • ROM read-only memory
  • flash RAM flash memory
  • Computer-readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information.
  • the information can be computer-readable instructions, data structures, program modules or other data.
  • Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory, and so on.
  • a computer readable medium is a type of random access memory (RAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a flash memory or other memory technology, a compact disk read-only memory (CD-ROM), a digital versatile disk (DVD) or other optical storage, a magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage device, or any other non-transmission medium that can be configured to store information that can be accessed by a computing device.
  • a computer readable medium does not include transitory media such as modulated data signals and carrier waves.
  • user information including but not limited to user device information, user attribute information, etc.
  • data including but not limited to data used for analysis, stored data, displayed data, etc.
  • user information and data are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards, and provide corresponding operation entrances for users to choose to authorize or refuse.
  • user information including but not limited to user device information, user personal information, etc.
  • data including but not limited to data used for analysis, stored data, displayed data, etc.
  • user information including but not limited to user device information, user personal information, etc.
  • data including but not limited to data used for analysis, stored data, displayed data, etc.
  • the disclosed embodiment provides a data processing method and server for human-computer interaction, which obtains human-computer interaction instructions, inputs the instructions into a large model for realizing human-computer interaction, and outputs the response results of the instructions through the large model;
  • the response results output by the big model are evaluated from the dimensions of harmfulness, instruction intention coverage, factuality, and content quality to obtain response quality information of the response results in each dimension; based on the response quality information of the response results output by the big model in each dimension, the response quality information of the big model is calculated to achieve accurate, comprehensive, and more fine-grained evaluation of the response results of the big model from multiple dimensions such as harmfulness, instruction intention coverage, factuality, and content quality, and output the response quality information of the big model.
  • the response quality information of the big model is set to guide the online decision of the big model, or to update the optimized version of the big model, or to select a high-quality target big model. It can accurately select high-quality models, improve the quality of iteratively updated/selected big models, and improve the quality of online models, thereby improving the accuracy of human-computer dialogue based on the big model and ensuring the quality of human-computer interaction.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • Quality & Reliability (AREA)
  • Computing Systems (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Evolutionary Computation (AREA)
  • Evolutionary Biology (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Artificial Intelligence (AREA)
  • Mathematical Physics (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Computer Hardware Design (AREA)
  • Stored Programmes (AREA)

Abstract

本公开提供一种人机交互的数据处理方法及服务器。本公开的方法，通过获取人机交互的指令，将指令输入实现人机交互的大模型，分别从伤害性、指令意图覆盖情况、事实性、内容质量的维度，对大模型输出的响应结果进行测评，得到响应结果在各个维度的响应质量信息；根据大模型输出的响应结果在各个维度的响应质量信息，计算大模型的响应质量信息，实现从伤害性、指令意图覆盖情况、事实性、内容质量等多个维度，对大模型的响应结果进行准确、全面、更细粒度地测评，基于大模型的响应质量信息指导大模型的上线判定、或更新大模型的优化版本、或选择优质的目标大模型。

Description

人机交互的数据处理方法及服务器
交叉援引
本公开要求于2023年06月05日提交中国专利局、优先权号为202310652255.5、发明名称为“人机交互的数据处理方法及服务器”的中国专利申请的优先权，其全部内容通过引用结合在本公开中。
技术领域
本公开涉及计算机技术，尤其涉及一种人机交互的数据处理方法及服务器。
背景技术
自然语言是人类逻辑和思维的重要载体，在人机交互，甚至通用人工智能领域具有非常重大的意义。但是因为自然语言的复杂性和模糊性，一直以来缺少直接面向无约束的自然语言的机器设施。
随着人工智能的发展，大模型被广泛应用于自然语言处理领域的人机交互中。大模型是指大规模深度学习模型，例如大规模的语言模型、多模态模型等，具有大规模的模型参数，通常包含上亿、上百亿、上千亿、上万亿甚至十万亿以上的模型参数。
在大模型的迭代过程中，需要测评不同版本的大模型的优劣，以实现大模型迭代更新。在大模型上线之前，需要测评大模型的表现是否满足上线要求，以上线表现优异的大模型，避免上线表现较差的大模型。目前对于人机交互的大模型，通常在大模型输出的答复是否对用户有帮助、答复内容是否安全等简单维度，对模型的表现进行笼统地打分，测评维度单一，无法准确全面地测评大模型的响应质量，不利于模型迭代中选择优质模型、不利于控制上线模型的质量，导致人机交互质量差。
发明内容
本公开提供一种人机交互的数据处理方法及服务器，用以解决无法准确全面地测评大模型的响应质量，不利于模型迭代中选择优质模型和控制上线模型的质量，导致人机交互质量差的问题。
第一方面，本公开提供一种人机交互的数据处理方法，包括：
获取人机交互的指令，将所述指令输入大模型，通过大模型输出所述指令的响应结果；
分别从伤害性、指令意图覆盖情况、事实性、内容质量的维度，对所述大模型输出的响应结果进行测评，得到所述响应结果在各个维度的响应质量信息；
根据所述大模型输出的响应结果在各个维度的响应质量信息，计算所述大模型的响应质量信息；
输出所述大模型的响应质量信息，所述大模型的响应质量信息被设置为指导所述大模型的上线判定、或更新所述大模型的优化版本、或选择优质的目标大模型。
第二方面，本公开提供一种人机交互的数据处理方法，应用于服务器，包括：
接收端侧设备发送的对多个语言模型的响应质量测评请求；
获取人机交互的指令，将所述指令输入各所述语言模型，通过各所述语言模型输出所述指令的响应结果；
分别从伤害性、指令意图覆盖情况、事实性、内容质量的维度，对各所述语言模型输出的响应结果进行测评，并生成各所述语言模型的响应质量信息；
向端侧设备发送交互界面数据，所述交互界面数据包含各所述语言模型输出的所述指令的响应结果；
接收端侧发送的在所述交互界面内指定的各所述语言模型输出的所述指令的响应结果的排序结果；
根据各所述语言模型输出的所述指令的响应结果的排序结果，计算各所述语言模型的响应质量的相对测评信息；
向所述端侧设备输出各所述语言模型的响应质量信息和相对测评信息。
第三方面，本公开提供一种人机交互的数据处理方法，应用于端侧设备，包括：
向服务器发送对多个语言模型的响应质量测评请求；
接收服务器发送的交互界面数据，所述交互界面数据包含各所述语言模型输出的响应结果，所述响应结果是通过如下方式生成的：获取人机交互的指令，将所述指令输入各所述语言模型，通过各所述语言模型输出所述指令的响应结果；
根据所述交互界面数据显示交互界面，所述交互界面上显示各所述语言模型输出的所述指令的响应结果，所述交互界面上不显示响应结果与所述语言模型的对应关系；
获取并向服务器发送在所述交互界面内指定的各所述语言模型输出的所述指令的响应结果的排序结果；
接收各所述语言模型的响应质量信息和相对测评信息，其中各所述语言模型的响应质量信息是通过分别从伤害性、指令意图覆盖情况、事实性、内容质量的维度，对各所述语言模型输出的响应结果进行测评生成的，各所述语言模型的相对测评信息是根据各所述语言模型输出的所述指令的响应结果的排序结果计算得到的；
输出各所述语言模型的响应质量信息和相对测评信息。
第四方面，本公开提供一种服务器，包括：处理器，以及与所述处理器通信连接的存储器；所述存储器存储计算机执行指令；所述处理器执行所述存储器存储的计算机执行指令，以实现如第一方面或第二方面所述的方法。
本公开提供的人机交互的数据处理方法及服务器，通过获取人机交互的指令，将指令输入实现人机交互的大模型，通过大模型输出指令的响应结果；分别从伤害性、指令意图覆盖情况、事实性、内容质量的维度，对大模型输出的响应结果进行测评，得到响应结果在各个维度的响应质量信息；根据大模型输出的响应结果在各个维度的响应质量信息，计算大模型的响应质量信息，实现从伤害性、指令意图覆盖情况、事 实性、内容质量等多个维度，对大模型的响应结果进行准确、全面、更细粒度地测评，并输出大模型的响应质量信息，大模型的响应质量信息被设置为指导大模型的上线判定、或更新大模型的优化版本、或选择优质的目标大模型，可以准确地选择优质模型，提升迭代更新/选择的大模型的质量，提升上线模型的质量，从而提升基于大模型的人机对话的准确性，保证人机交互质量。
附图说明
此处的附图被并入说明书中并构成本说明书的一部分，示出了符合本公开的实施例，并与说明书一起用于解释本公开的原理。
图1为本公开所适用的一示例系统架构的示意图；
图2为本公开一示例性实施例提供的人机交互的数据处理方法流程图；
图3为本公开一示例性实施例提供的第一交互界面的一个示例图；
图4为本公开一示例性实施例提供的第二交互界面的一个示例图；
图5为本公开一示例性实施例提供的人机交互的数据处理方法流程图；
图6为本公开实施例提供的一种服务器的结构示意图。
通过上述附图，已示出本公开明确的实施例，后文中将有更详细的描述。这些附图和文字描述并不是为了通过任何方式限制本公开构思的范围，而是通过参考特定实施例为本领域技术人员说明本公开的概念。
具体实施方式
这里将详细地对示例性实施例进行说明，其示例表示在附图中。下面的描述涉及附图时，除非另有表示，不同附图中的相同数字表示相同或相似的要素。以下示例性实施例中所描述的实施方式并不代表与本公开相一致的所有实施方式。相反，它们是与如所附权利要求书中所详述的、本公开的一些方面相一致的装置和方法的例子。
首先对本公开所涉及的名词进行解释：
指令：指含有一定意图的自然语言文本，在人机交互场景中是指用户给出的问题。
响应结果：是指对于指令产出的回复信息。
视觉问答任务：根据输入的图像和问题，从输入图像的视觉信息中确定问题的答案。
图像描述任务：生成输入图像的描述文本。
视觉蕴涵任务：预测输入图像和文本在语义上的相关性，即蕴涵、中性或矛盾。
指代表达与理解任务：根据输入文本定位输入图像中与输入文本对应的图像区域。
图像生成任务：基于输入的描述文本生成图像。
基于文本的情感分类任务：预测输入文本的情感分类信息。
文本摘要任务：生成输入文本的摘要信息。
多模态任务：是指输入输出数据涉及图像和文本等多种模态数据的下游任务，例如视觉问答任务、图像描述任务、视觉蕴涵任务、指代表达与理解任务、图像生成任务等。
多模态预训练模型：是指输入输出数据涉及图像和文本等多种模态数据的预训练模型，经过微调训练后可以应用于多模态任务处理。
大模型是指具有大规模模型参数的深度学习模型，通常包含上亿、上百亿、上千亿、上万亿甚至十万亿以上的模型参数。大模型又可以称为基石模型/基础模型(Foundation Model)，通过大规模无标注的语料进行大模型的预训练，产出亿级以上参数的预训练模型，这种模型能适应广泛的下游任务，模型具有较好的泛化能力，例如大规模语言模型(Large Language Model，LLM)、多模态预训练模型(multi-modal pre-training model)等。
大模型在实际应用时，仅需少量样本对预训练模型进行微调即可应用于不同的任务中，大模型可以广泛应用于自然语言处理(Natural Language Processing，简称NLP)、计算机视觉等领域，具体可以应用于如视觉问答(Visual Question Answering，简称VQA)、图像描述(Image Caption，简称IC)、图像生成等计算机视觉领域任务，以及基于文本的情感分类、文本摘要生成、机器翻译等自然语言处理领域任务，大模型主要的应用场景包括数字助理、智能机器人、搜索、在线教育、办公软件、电子商务、智能设计等。
应用于人机交互场景(如智能机器人)时，大模型基于用户给出的指令生成答复。在大模型的迭代过程中，需要测评不同版本的大模型的优劣，以实现大模型迭代更新。在大模型上线之前，需要测评大模型的表现是否满足上线要求，以上线表现优异的大模型，避免上线表现较差的大模型。目前对于应用于人机交互场景的大模型，通常从答复是否对用户有帮助，答复内容是否安全等简单维度，对模型的表现进行笼统地打分，测评维度单一，无法准确全面地测评大模型表现的优劣，不利于模型迭代中选择优质模型和控制上线模型的质量。
本公开提供一种人机交互的数据处理方法，通过获取人机交互的指令，将指令输入实现人机交互的大模型，通过大模型输出指令的响应结果；分别从伤害性、指令意图覆盖情况、事实性、内容质量的维度，对大模型输出的响应结果进行测评，得到响应结果在各个维度的响应质量信息；根据大模型输出的响应结果在各个维度的响应质量信息，计算大模型的响应质量信息，从而从伤害性、指令意图覆盖情况、事实性、内容质量等多个维度对大模型的响应结果进行准确、全面、更细粒度地测评，并输出大模型的响应质量信息，大模型的响应质量信息被设置为指导大模型的上线判定、或更新大模型的优化版本、或选择优质的目标大模型，可以准确地选择优质模型，提升迭代更新/选择的大模型的质量，提升上线模型的质量，从而提升基于大模型的人机对 话的准确性，保证人机交互质量。
图1为本公开所适用的一示例系统架构的示意图。如图1所示，该系统架构包括负责测评大模型的第一服务器、运行大模型的第二服务器和端侧设备。其中，第一服务器与第二服务器间具有可通信的通信链路，能够实现第一服务器与第二服务器间的通信连接。第一服务器与端侧设备之间具有可通信的通信链路，能够实现第一服务器与端侧设备间的通信连接。
其中，第二服务器可以是部署在云端的服务器集群、或者本地具有计算能力的设备。第二服务器负责运行实现人机交互的大模型，基于给定的人机交互的指令生成响应结果。一个第二服务器上可以部署一个或者多个大模型，对于待测评的多个大模型，可以部署在一个或者多个第二服务器上。
端侧设备是用户所使用的电子设备，具体可以为具有网络通信功能、运算功能以及信息显示功能的硬件设备，其包括但不限于智能手机、平板电脑、台式电脑、服务器等。用户通过端侧设备向第一服务器发送大模型测评请求，该测评请求包含待测评的一个或者多个大模型的信息。
第一服务器可以是部署在云端的服务器集群、或者本地具有计算能力的设备。第一服务器负责执行本公开提供的人机交互的数据处理方法，以实现对大模型的响应质量的测评，生成大模型的响应质量信息，并指导大模型的上线判定、或更新大模型的优化版本、或选择优质的目标大模型。本实施例中，大模型的响应质量信息指示大模型针对输入的指令给出的响应结果的响应质量，是对大模型的响应结果的响应质量的测评值。大模型的响应质量信息具体包括大模型的响应结果在伤害性、指令意图覆盖情况、事实性、内容质量等多个维度的响应质量信息，如响应结果在伤害性、指令意图覆盖情况、事实性、内容质量等多个维度的测评分值，可以从多个维度准确、全面地反映响应结果的响应质量。
在一示例场景中，在实现人机交互的大模型上线之前，用户通过端侧设备向第一服务器发送待上线的大模型的响应质量测评请求，该测评请求包含待测评的大模型的相关信息，如调用大模型的应用程序接口、大模型的访问地址等。第一服务器响应于该测评请求，获取人机交互的指令，将指令输入大模型，通过大模型输出指令的响应结果；分别从伤害性、指令意图覆盖情况、事实性、内容质量的维度，对大模型输出的响应结果进行测评，得到响应结果在各个维度的响应质量信息；根据大模型输出的响应结果在各个维度的响应质量信息，计算大模型的响应质量信息，以实现对大模型的响应质量进行全面、准确地测评。
进一步地，大模型的响应质量信息可以被设置为指导大模型的上线判定。可选地，第一服务器根据大模型的响应质量信息，确定大模型是否满足上线条件，并输出大模型的上线提示信息，上线提示信息指示大模型是否满足上线条件。可选地，第一服务 器还可以向端侧设备发送大模型的响应质量信息。端侧设备输出大模型的响应质量信息，以指导用户判断大模型是否满足上线条件；或者，端侧设备根据大模型的响应质量信息确定大模型是否满足上线条件，并输出大模型的上线提示信息，上线提示信息指示大模型是否满足上线条件。
在另一示例场景中，在大模型迭代优化过程中，对得到的新版本进行测评。用户可以通过端侧设备向第一服务器发送新版本的大模型的响应质量测评请求，该测评请求包含新版本的大模型的相关信息，如调用大模型的应用程序接口、大模型的访问地址等。第一服务器响应于该测评请求，获取人机交互的指令，将指令输入大模型，通过大模型输出指令的响应结果；分别从伤害性、指令意图覆盖情况、事实性、内容质量的维度，对大模型输出的响应结果进行测评，得到响应结果在各个维度的响应质量信息；根据大模型输出的响应结果在各个维度的响应质量信息，计算大模型的响应质量信息，以实现对大模型新版本的响应质量的全面、准确地测评。
进一步地，大模型的响应质量信息可以被设置为指导大模型的优化版本的更新。可选地，第一服务器根据新版本的大模型的响应质量信息，以及上一版本的大模型的响应质量信息，对新版本及上一版本的响应质量信息进行比较，得到比较结果，比较结果被设置为指导更新大模型的优化版本。具体地，第一服务器可以向端侧设备发送比较结果。端侧设备输出不同版本的大模型的响应质量信息的比较结果，以指导用户选择更优的优化版本进行大模型的迭代更新。
在另一示例场景中，用户可以基于待选的多个大模型的测评对比结果，选择更加优质的目标大模型。用户可以通过端侧设备向第一服务器发送多个大模型的响应质量测评请求，该测评请求包含多个大模型的相关信息，如调用大模型的应用程序接口、大模型的访问地址等。第一服务器响应于该测评请求，获取人机交互的指令，将指令输入各个大模型，通过各个大模型输出指令的响应结果；分别从伤害性、指令意图覆盖情况、事实性、内容质量的维度，对各个大模型输出的响应结果进行测评，得到各个大模型输出的响应结果在各个维度的响应质量信息；根据各个大模型输出的响应结果在各个维度的响应质量信息，计算各个大模型的响应质量信息，以对各个大模型的响应质量进行全面、准确地测评。
进一步地，第一服务器对各个大模型的响应质量信息进行比较，得到各个大模型的响应质量信息的比较结果。第一服务器向端侧设备发送各个大模型的响应质量信息的比较结果。端侧设备输出比较结果，以指导用户选择响应质量更优的大模型，作为自己选择使用的目标大模型。可选地，端侧设备可以基于各个大模型的响应质量信息的比较结果，选择响应质量更优的大模型，并根据所选择的大模型的相关信息，下载获取该大模型，或者，使用该大模型实现人机交互。
下面以具体地实施例对本公开的技术方案以及本公开的技术方案如何解决上述 技术问题进行详细说明。下面这几个具体的实施例可以相互结合，对于相同或相似的概念或过程可能在某些实施例中不再赘述。下面将结合附图，对本公开的实施例进行描述。
图2为本公开一示例性实施例提供的人机交互的数据处理方法流程图。本实施例的执行主体为前述系统架构中的第一服务器。如图2所示，该方法具体步骤如下：
步骤S201、获取人机交互的指令，将指令输入大模型，通过大模型输出指令的响应结果。
其中，指令是指人机交互过程中人类用户发出的问题。基于用户的指令人机交互系统会输出答复，也即指令的响应结果。
该步骤中，可以通过如下至少一个途径来搜集人机交互中的指令，并构建指令集：搜集一个预设的历史时期内人机交互系统中产生的用户指令，或者从网络资源上搜集在人机交互过程中指令，或者人工编写指令。在实际应用中，用户输入的指令可以为问题文本，或者包括文本、图像等多模态信息。
本实施例中，大模型被设置为基于用户给定的指令生成指令的响应结果，实现人机交互。大模型作为待测评对象，具体可以是各类语言模型、多模态预训练模型等，此处不做具体限定。
可选地，第一服务器可以获取大模型的应用程序接口/大模型服务接口，通过调用大模型的应用程序接口/大模型服务接口，将指令输入大模型，并接收大模型输出的指令的响应结果。
可选地，第一服务器可以获取大模型的访问地址信息，第一服务器向大模型所在的第二服务器发送指令执行请求，该请求包含待执行的指令。第二服务器响应于指令执行请求，将指令输入大模型，获取到大模型输出的指令的响应结果，并将指令的响应结果发送至第一服务器。
步骤S202、分别从伤害性、指令意图覆盖情况、事实性、内容质量的维度，对大模型输出的响应结果进行测评，得到响应结果在各个维度的响应质量信息。
其中，伤害性是指响应结果的内容是否具有伤害性，也即响应结果的内容是否具有安全风险。对于大模型给出的响应结果的内容要尽可能符合人类的价值观，响应结果无伤害(也即不存在安全风险)是指响应结果的内容没有违背人类价值观的风险问题。
示例性地，可以预先配置多种不同维度/类别的安全风险，包括但不限于：涉及各类危害社会安全的内容、涉及危害世界和平的内容、传扬非法组织信息、传播非法内容、涉及违法行为。响应结果具有配置的任一维度/类别的安全风险，即可认为响应结果具有伤害性。响应结果不具有所配置的全部安全风险，即可认为响应结果不具有伤 害性。
在实际应用中，一个指令可以包含一个或者多个意图。指令意图覆盖情况是指响应结果覆盖指令所包含的意图的情况，具体可以分为完全覆盖、部分覆盖、未覆盖等情况。
事实性是指响应结果中是否存在事实错误的情况，具体可以包括但不限于如下几种情况：不存在事实性错误，存在较易发现的常识性事实错误，存在难以发现、需要查阅资料或思考才能发现的知识性事实错误，同时存在常识性事实错误和知识性事实错误。
内容质量是指响应结果内容本身在内容内涵、排版、格式、启承、重复、乱码、歧义等方面的质量情况，可以总体表现为响应结果内容的可读性、连贯性的好坏。
本实施例中，从伤害性、指令意图覆盖情况、事实性、内容质量等多个维度，对大模型输出的响应结果的响应质量进行测评，生成响应结果在各个维度的响应质量信息，以实现从多个维度准确、全面地测评响应结果的响应质量。
步骤S203、根据大模型输出的响应结果在各个维度的响应质量信息，计算大模型的响应质量信息。
该步骤中，根据大模型给出的各个指令的响应结果在各个维度的响应质量信息，综合计算得到大模型的响应质量信息。
可选地，可以将任一指令的响应结果在各个维度的响应质量信息求和，作为该响应结果的综合响应质量信息；将大模型给出的各个指令的响应结果的综合响应质量信息求和，作为大模型的响应质量信息。
可选地，可以为各个维度分别配置权重系数，将任一指令的响应结果在各个维度的响应质量信息加权求和，作为该响应结果的综合响应质量信息；将大模型给出的各个指令的响应结果的综合响应质量信息求和，作为大模型的响应质量信息。
可选地，可以为各个维度分别配置权重系数，根据各个维度的权重系数，将大模型给出的各个指令的响应结果在同一维度的响应质量信息加权求和，作为大模型在该维度的响应质量信息，可以得到大模型在各个维度的响应质量信息，可以重复体现大模型的响应结果在各个维度的响应质量。
步骤S204、输出大模型的响应质量信息，大模型的响应质量信息被设置为指导大模型的上线判定、或更新大模型的优化版本、或选择优质的目标大模型。
本实施例中，第一服务器在得到大模型的响应质量信息之后，将大模型的响应质量信息进行可视化输出，以向用户输出大模型响应质量的测评结果，大模型的响应质量信息可以指导用户做出大模型是否上线的判定；或者，通过比较多个大模型版本的响应质量信息，确定优质大模型版本，并进行大模型的迭代优化；或者，通过比较多 个大模型的响应质量信息，选择响应质量较高的目标大模型，作为实现人机交互使用的大模型。
示例性地，第一服务器可以根据大模型的响应质量信息，确定大模型是否满足上线条件；输出大模型的上线提示信息，上线提示信息指示大模型是否满足上线条件。
其中，上线条件包括大模型的响应质量信息的第一阈值，若大模型的响应质量信息大于或等于第一阈值时，则大模型满足上线条件，否则，大模型不满足上线条件。上线条件中的第一阈值可以由用户根据具体应用场景的需要进行自定义配置。
示例性地，第一服务器可以根据多个大模型的响应质量信息，通过对比多个大模型的响应质量信息分，选择其中一个响应质量较优的大模型作为目标大模型，并向端侧设备输出目标大模型的信息。
示例性地，第一服务器可以根据大模型的多个版本的响应质量信息，通过对比大模型多个版本的响应质量信息分，选择其中一个版本的大模型作为优化版本，更新大模型的优化版本。
本实施例通过获取人机交互的指令，将指令输入实现人机交互的大模型，通过大模型输出指令的响应结果；分别从伤害性、指令意图覆盖情况、事实性、内容质量的维度，对大模型输出的响应结果进行测评，得到响应结果在各个维度的响应质量信息；根据大模型输出的响应结果在各个维度的响应质量信息，计算大模型的响应质量信息，从而从伤害性、指令意图覆盖情况、事实性、内容质量等多个维度对大模型的响应结果进行准确、全面、更细粒度地测评，并输出大模型的响应质量信息，大模型的响应质量信息被设置为指导大模型的上线判定、或更新大模型的优化版本、或选择优质的目标大模型，可以准确地选择优质模型，提升迭代更新/选择的大模型的质量，提升上线模型的质量，从而提升基于大模型的人机对话的准确性，保证人机交互质量。
在一可选实施例中，对于大模型响应结果的响应质量的测评维度，配置任一维度对应多个质量类别，不同的质量类别对应不同的响应质量信息。其中，质量类别是指将任一测评维度的响应质量划分成的多个不同质量等级(质量好坏程度)的类别，不同的质量类别代表的响应质量的等级/好坏程度不同，对应不同的响应质量信息(如测评分值)。在进行该维度响应质量的测评时，通过将大模型响应结果标注为该维度的一个质量类别，即可实现对响应结果在该维度的响应质量信息的标注。
示例性地，伤害性可以包括如下质量类别：无伤害、有伤害。其中，有伤害是指响应结果存在至少一种维度/类别的安全风险。其中，可以预先配置多种不同维度/类别的安全风险，包括但不限于：涉及各类危害社会安全的内容、涉及危害世界和平的内容、传扬非法组织信息、传播非法内容、涉及违法行为。在实际应用中，对于有伤害的指令，大模型应拒绝答复。对于无伤害的指令，大模型应针对指令的意图进行答复。例如，在对大模型响应结果进行测评时，可以通过将大模型响应结果标注为“无 伤害”或者“有伤害”，来测评大模型响应结果在伤害性维度的响应质量。
指令意图覆盖情况可以包括如下质量类别：完全识别指令意图、部分识别指令意图、未能识别指令意图、不应拒绝的指令意图但拒绝。其中，完全识别指令意图是指响应结果覆盖指令所包含的全部意图。部分识别指令意图是指指令包含多个意图，响应结果覆盖指令所包含的至少一个意图，但未覆盖全部意图。未能识别指令意图是指响应结果未覆盖指令所包含的任一意图。不应拒绝的指令意图但拒绝是指指令无伤害应该针对指令所包含意图进行答复，但是大模型给出的响应结果拒绝答复指令。例如，在对大模型响应结果进行测评时，根据指令的意图和大模型的响应结果，根据响应结果对指令意图的覆盖情况，通过将大模型响应结果标注为“完全识别指令意图”、“部分识别指令意图”、“未能识别指令意图”、“不应拒绝的指令意图但拒绝”中的一种质量类别，来测评大模型响应结果在覆盖指令意图维度的响应质量。
事实性可以包括如下质量类别：无事实性错误、常识性事实错误、知识性事实错误、同时出现常识性和知识性事实错误。无事实性错误是指响应结果中不存在事实错误的情况。常识性事实错误是指响应结果中存在较易发现的常识性事实错误。知识性事实错误是指响应结果中存在难以发现、需要查阅资料或思考推理才能发现的知识性事实错误。例如，在对大模型响应结果进行测评时，根据大模型的响应结果是否存在事实错误，通过将大模型响应结果标注为“无事实性错误”、“常识性事实错误”、“知识性事实错误”、“同时出现常识性和知识性事实错误”中的一种质量类别，来测评大模型响应结果在事实性维度的响应质量。
内容质量可以包括如下质量类别：连贯性好、连贯性中、连贯性差。具体可以从内容内涵、排版、格式、启承、重复、乱码、歧义、语法等方面，对响应结果的连贯性进行综合性地测评。例如，在对大模型响应结果进行测评时，根据大模型的响应结果在内容内涵、排版、格式、启承、重复、乱码、歧义、语法等方面的质量来确定响应结果的连贯性的好坏，通过将大模型响应结果标注为“连贯性好”、“连贯性中”、“连贯性差”中的一种质量类别，来测评大模型响应结果在内容质量维度的响应质量。
本实施例中，前述步骤S202具体可以采用如下方式实现：针对各维度，确定大模型输出的响应结果在各维度的质量类别；根据响应结果在各维度的质量类别，确定响应结果在各维度的响应质量信息。
具体地，任一维度对应多个质量类别，不同的质量类别对应不同的响应质量信息。分别针对各个维度，确定大模型输出的响应结果在该维度的质量类别，并将响应结果在该维度的质量类别对应的响应质量信息，作为响应结果在该维度的响应质量信息。由此可以确定响应结果在各个维度的响应质量信息。
进一步地，在实现对各维度，确定响应结果在各维度的质量类别时，可以通过交互界面，由标注人员在交互界面中对响应结果在各维度的质量类别进行人工标注。具 体可以采用如下方式实现：
通过第一交互界面显示响应结果、以及各维度对应的质量类别，并提供对响应结果在各维度的质量类别的输入区域。标注人员可以查看第一交互界面中显示的响应结果，并在输入区域中输入响应结果在各个维度的质量类别。响应于对第一交互界面的提交操作，获取输入区域内输入的响应结果在各维度的质量类别。
可选地，第一交互界面可以显示一个或者多个响应结果，不同响应结果对应不同的输入区域，标注人员在响应结果的对应输入区域内对该响应结果在各维度的质量类别进行标注。
示例性地，输入区域内可以显示各维度的全部质量类别。标注人员通过在输入区域内选定响应结果在各维度的质量类别。或者，输入区域内可以显示各维度的全部质量类别，以及各维度对应的输入框，标注人员通过在各维度的输入框内输入响应结果在该维度的质量类别。
示例性地，图3示出了第一交互界面的一个示例，以5个大模型针对同一指令“如果我要在北京时间晚上10点和伦敦的同事开会，那么在伦敦当地时间是几点？”给出的5个不同的响应结果为例，如图3所示，第一交互界面中分别显示出各个大模型给出的响应结果(如图3中虚线框的区域为显示响应结果的区域，虚线框在界面中不显示)，但并未显示响应结果与大模型的对应关系，使得标注人员可以在未知响应结果来自哪个大模型的前提下进行更为客观的测评。图3中横向并排显示5个响应结果，在各个响应结果下方区域显示各维度的质量类别，并提供勾选框。标注人员通过勾选指定响应结果在各维度的质量类别。
在一可选实施方式中，在实现对各维度，确定响应结果在各维度的质量类别时，还可以利用预训练的分类识别模型，自动分析响应结果在各维度的类别。
具体地，利用伤害性分类识别模型，识别响应结果在伤害性维度的质量类别；利用意图覆盖分类识别模型，识别响应结果在指令意图覆盖情况维度的质量类别；利用事实性分类识别模型，识别响应结果在事实性维度的质量类别；并利用内容质量分类识别模型，识别响应结果在内容质量维度的质量类别。
其中，伤害性分类识别模型的输入是响应结果，对响应结果是否存在各类安全风险进行分类，根据分类结果如果确定响应结果存在至少一种安全风险，则确定响应结果有伤害。根据分类结果，如果确定响应结果不存在任何类型的安全风险，则确定响应结果无伤害。伤害性分类识别模型可以使用带有安全风险类别标注的语料对分类模型进行训练得到。
意图覆盖分类识别模型的输入是指令/指令的意图和响应结果，对响应结果的指令意图覆盖情况的质量类别进行分类识别，输出响应结果在指令意图覆盖情况维度的质 量类别，可以使用带有指令意图覆盖情况标注的数据对分类模型进行训练得到。
事实性分类识别模型的输入是响应结果，对响应结果是否存在常识性事实错误和是否存在知识性事实错误进行识别，并根据识别结果确定响应结果在事实性维度的质量类别。其中，识别响应结果是否存在常识性事实错误，与识别响应结果是否存在知识性事实错误，可以使用两个模型实现。两个模型分别使用带有是否存在常识性事实错误和是否存在知识性事实错误标注的数据对分类模型进行训练得到，两个模型可以同构但不共享参数。
内容质量分类识别模型的输入是响应结果，输出是响应结果在内容质量维度的质量类别，可以使用带有内容质量分类标注信息的数据对分类模型进行训练得到。
需要说明的是，输出的响应结果若包含多模态的响应信息，可以分别基于单一模型的响应信息，确定响应信息在各维度的质量类别，再综合多模态响应信息的质量类别，确定响应结果在各个维度的质量类别。例如，响应结果包括文本和图片，对于事实性维度，经识别确定文本存在常识性事实错误，图片无事实性错误，综合可以确定响应结果存在常识性事实错误。
在一可选实施例中，前述步骤S203根据大模型输出的响应结果在各个维度的响应质量信息，计算大模型的响应质量信息，具体可以采用如下方式实现：
根据大模型输出的各响应结果在各个维度的响应质量信息，以及各个维度的权重系数，计算各响应结果的综合质量信息；根据大模型输出的各响应结果的综合质量信息，计算大模型的响应质量信息。
其中，各个维度的权重系数可以根据经验值进行配置，也可以由用户自定义配置。
示例性地，第一服务器提供各个维度的权重配置界面，权重配置界面被设置为将配置各个维度的权重系数。第一服务器获取在权重配置界面上配置的各个维度的权重系数。
另外，各个维度的各个质量类别分别对应的响应质量信息，也可以由用户自定义配置，对应的响应质量信息的值越大表示该维度的响应质量越高，通过为不同质量类别设置不同的响应质量信息，可以更为精准地测评大模型响应结果的响应质量。
在一可选实施例中，在通过大模型输出指令的响应结果之后，还可以输出响应结果，使得用户/标注员对响应结果的综合质量类别进行测评，并提交综合质量类别。第一服务器接收对响应结果标注的综合质量类别，综合质量类别包括：好、一般、差。
示例性地，可以在第一交互界面中输出综合质量类别，并提供对应的输入区域。如图3中最下面一行所示的可选项“赞”“踩”“一般”分别对应于综合质量类别的好、差、一般。通过勾选可以指定响应结果的综合质量类别。
相应地，根据大模型输出的各响应结果在各个维度的响应质量信息，综合计算各 响应结果的综合质量信息之后，还包括：
根据不同的综合质量类别对应的质量信息区间，将响应结果被标注的综合质量类别对应的质量信息区间，作为响应结果对应的质量信息区间；对指令的响应结果进行过滤，去除综合质量信息不在对应质量信息区间内的响应结果对应的指令。由此可以过滤掉综合质量反馈结果与通过多个维度的响应质量信息确定测评结果不匹配的数据，从而可以提升脏数据对测评结果的影响，可以提升对大模型响应质量测评的准确性。
示例性地，假设综合质量类别“好”对应质量信息区间为[9，10]，综合质量类别“一般”对应质量信息区间为[6，9)，综合质量类别“差”对应质量信息区间为[0，6)。假设对于一个响应结果，接收到的综合质量类别为“好”，但根据响应结果在各个维度的响应质量信息确定的响应结果的综合质量信息为5，不在综合质量类别“好”对应质量信息区间为[9，10]内，可以说明针对该指令的响应结果的质量测评结果存在异常，删除该指令相关的数据，不再基于该指令相关的数据对大模型进行测评。
在一种示例应用场景中，待测评的大模型可以有多个。前述步骤S201具体可以采用如下方式实现：
接收端侧设备发送的对多个大模型的响应质量测评请求；获取人机交互的指令，将指令分别输入多个大模型，得到各大模型输出的指令的响应结果。
进一步地，在前述步骤S202中，分别从伤害性、指令意图覆盖情况、事实性、内容质量的维度，对各个大模型输出的响应结果进行测评，得到响应结果在各个维度的响应质量信息；前述步骤S203中，根据各个大模型输出的响应结果在各个维度的响应质量信息，计算各个大模型的响应质量信息。
具体地，在前述步骤S202中，可以分别针对各个大模型针对同一指令给出的响应结果，针对各维度，确定各个大模型输出的响应结果在各维度的质量类别；根据响应结果在各维度的质量类别，确定响应结果在各维度的响应质量信息。
示例性地，通过第一交互界面显示各个大模型输出的同一指令的响应结果、以及各维度对应的质量类别，并提供对各个响应结果在各维度的质量类别的输入区域(如图3所示)。第一交互界面上不显示响应结果与各大模型的对应关系。使得标注人员在未知响应结果来自于哪个大模型的前提下，在输入区域中输入各个响应结果在各个维度的质量类别。响应于对第一交互界面的提交操作，第一服务器获取输入区域内输入的各个响应结果在各维度的质量类别。
本实施例中，还可以通过对各个大模型输出的响应结果的响应质量进行排序，基于排序结果可以确定各个大模型的相对测评信息。
具体地，第一服务器还可以提供第二交互界面，通过第二交互界面输出各大模型 输出的指令的响应结果，第二交互界面上不显示响应结果与各大模型的对应关系。标注人员可以对第二交互界面上显示的同一指令的响应结果进行排序。第一服务器接收在交互界面内指定的各大模型输出的指令的响应结果的排序结果；根据各大模型输出的指令的响应结果的排序结果，计算各大模型的响应质量的相对测评信息。进一步的，第一服务器可以向端侧设备输出各大模型的响应质量信息和相对测评信息。
其中，用于对响应结果排序的第二交互界面与第一交互界面可以合并为同一交互界面，也可以分别使用不同的交互界面实现，此处不做具体限定。
示例性地，图4示出了第二交互界面的一个示例，在图3所示内容的基础上，不同响应结果对应不同的显示区域(如图4中虚线框所示区域，虚线框在第二交互界面中不显示)，可以通过拖动各个响应结果的显示区域的位置来改变各个响应结果的排序。另外，还可以通过输入响应结果的顺序值来改变各个响应结果的排序，第二交互界面中各个响应结果的显示区域的位置会随着响应结果的顺序值的变化自动调整。
在本实施例的另一可选实施方式中，还可以使用排序算法，对响应结果自动进行排序，可以大大提升数据处理的效率。具体地，第一服务器使用预训练的响应结果排序算法，对各大模型输出的同一指令的响应结果进行排序。进一步地，第一服务器根据各大模型输出的同一指令的响应结果的排序结果，计算各大模型的响应质量的相对测评信息。进一步的，第一服务器可以向端侧设备输出各大模型的响应质量信息和相对测评信息。其中，排序算法可以使用现有的任意一种对文本基于文本质量进行排序的方法实现，此处不做具体限定。
可选地，根据各个大模型输出的响应结果的排序结果，可以计算大模型输出的响应结果排在各个名次的次数分布，作为各个大模型的相对测评信息，排名靠前的次数越多，说明大模型的响应质量越好。通过输出各个大模型输出的响应结果排在各个名次的次数分布，可以直观地展示任一大模型在待对比的多个大模型中相对质量。
可选地，根据各个大模型输出的响应结果的排序结果，对于其中的任意两个大模型，可以计算两个大模型之间的胜、负和平的情况，作为各个大模型的相对测评信息。其中对于两个大模型A和B针对同一指令的响应结果，如果排序结果中大模型A的响应结果排在大模型B的响应结果前面，则大模型A胜。如果排序结果中大模型A的响应结果排在大模型B的响应结果后面，则大模型A负。如果排序结果中大模型A的响应结果与大模型B的响应结果并列，则两个大模型平。基于指令集中的多个指令，根据两个大模型针对同一指令的响应结果的排序结果，可以计算出其中任一大模型的胜、负和平的次数，或者计算任一大模型的胜率。通过任一大模型的胜、负和平的次数，或者计算任一大模型的胜率，可以直观地展示该大模型在待对比的多个大模型中相对质量。
可选地，根据各个大模型针对同一指令的响应结果的排序结果，计算各个大模型 的埃洛等级分(Elo rating)，基于指令集中多个指令，多次计算各个大模型的埃洛等级分(Elo rating)并取均值，作为各个大模型的相对测评信息。通过输出各个大模型的埃洛等级分的均值，可以直观地展示各个大模型在待对比的多个大模型中相对质量。
在一可选实施例中，第一服务器可以根据各大模型的响应质量信息和/或相对测评信息，选择其中一个大模型作为目标大模型，并向端侧设备输出目标大模型的信息。
在一可选实施例中，第一服务器可以根据各大模型的响应质量信息和/或相对测评信息，选择其中一个大模型作为优化版本，更新大模型的优化版本。
图5为本公开一示例性实施例提供的人机交互的数据处理方法流程图，本实施例中，以实现人机交互的大模型为语言模型为例，对多个语言模型的对比测评的流程进行示例性地说明。如图5所示，该方法具体步骤如下：
步骤S501、端侧设备向第一服务器发送对多个语言模型的响应质量测评请求。
其中，多个语言模型可以预训练的语言模型，具体应用于自然语言处理(NLP)、计算机视觉等领域，具体可以应用于如视觉问答(VQA)、图像描述(IC)、视觉蕴涵(VE)、指代表达与理解(REC)等NLP与计算机视觉交叉领域的任务，以及基于文本的情感分类任务和文本摘要任务等自然语言处理领域的任务，可以应用于数字助理、智能机器人、搜索、在线教育、办公软件、电子商务、智能设计等各应用场景。
步骤S502、第一服务器接收端侧设备发送的对多个语言模型的响应质量测评请求。
步骤S503、第一服务器获取人机交互的指令，将指令输入各语言模型，通过各语言模型输出指令的响应结果。
该步骤的具体实现方式与前述步骤S201的实现方式相同，具体参见前述实施例中的相关内容，此处不再赘述。
步骤S504、第一服务器分别从伤害性、指令意图覆盖情况、事实性、内容质量的维度，对各语言模型输出的响应结果进行测评，并生成各语言模型的响应质量信息。
该步骤的具体实现方式与前述步骤S202-S203的具体实现方式类似，该步骤S504中待测评的大模型是用户指定的多个语言模型，具体实现方式参见前述实施例中的相关内容，此处不再赘述。
本实施例中，通过对各个大模型输出的响应结果的响应质量进行排序，基于排序结果确定各个大模型的相对测评信息，具体通过步骤S505-S510实现，第一服务器提供第二交互界面，通过第二交互界面输出各大模型输出的指令的响应结果，第二交互界面上不显示响应结果与各大模型的对应关系。标注人员可以对第二交互界面上显示的同一指令的响应结果进行排序。第一服务器接收在交互界面内指定的各大模型输出的指令的响应结果的排序结果；根据各大模型输出的指令的响应结果的排序结果，计算各大模型的响应质量的相对测评信息。进一步的，第一服务器可以向端侧设备输出 各大模型的响应质量信息和相对测评信息。
步骤S505、第一服务器向端侧设备发送交互界面数据，交互界面数据包含各语言模型输出的指令的响应结果。
其中，交互界面数据是指用于排序的第二交互界面的数据。本实施例中第一服务器提供第二交互界面，并通过端侧设备显示第二交互界面。第二交互界面被设置为实现对各个语言模型对同一指令的响应结果的排序。图4示出了第二交互界面的一个示例，如图4所示，不同模型输出的响应结果对应不同的显示区域，可以通过拖动各个响应结果的显示区域的位置来改变各个响应结果的排序。另外，还可以通过输入响应结果的顺序值来改变各个响应结果的排序，第二交互界面中各个响应结果的显示区域的位置会随着响应结果的顺序值的变化自动调整。
步骤S506、端侧设备接收服务器发送的交互界面数据。
步骤S507、端侧设备根据交互界面数据显示交互界面，交互界面上显示各语言模型输出的指令的响应结果，交互界面上不显示响应结果与语言模型的对应关系。
示例性地，图4示出了第二交互界面的一个示例，如图4所示，不同模型输出的响应结果对应不同的显示区域，可以通过拖动各个响应结果的显示区域的位置来改变各个响应结果的排序。另外，还可以通过输入响应结果的顺序值来改变各个响应结果的排序，第二交互界面中各个响应结果的显示区域的位置会随着响应结果的顺序值的变化自动调整。图4所示第二交互界面中分别显示出各个大模型给出的响应结果，但并未显示响应结果与大模型的对应关系，使得标注人员可以在未知响应结果来自哪个大模型的前提下进行更为客观的测评。
步骤S508、端侧设备获取并向服务器发送在交互界面内指定的各语言模型输出的指令的响应结果的排序结果。
第二交互界面被提交后，端侧设备可以获取到在交互界面内指定的各语言模型输出的当前指令的响应结果的排序结果，并将排序结果发送至第一服务器。
步骤S509、第一服务器接收端侧发送的在交互界面内指定的各语言模型输出的指令的响应结果的排序结果。
步骤S510、第一服务器根据各语言模型输出的指令的响应结果的排序结果，计算各语言模型的响应质量的相对测评信息。
可选地，第一服务器可以根据各个语言模型输出的响应结果的排序结果，可以计算语言模型输出的响应结果排在各个名次的次数分布，作为各个语言模型的相对测评信息，排名靠前的次数越多，说明语言模型的响应质量越好。通过输出各个语言模型输出的响应结果排在各个名次的次数分布，可以直观地展示任一语言模型在待对比的多个语言模型中相对质量。
可选地，第一服务器可以根据各个语言模型输出的响应结果的排序结果，对于其中的任意两个语言模型，可以计算两个语言模型之间的胜、负和平的情况，作为各个语言模型的相对测评信息。其中对于两个语言模型C和D针对同一指令的响应结果，如果排序结果中语言模型C的响应结果排在语言模型D的响应结果前面，则语言模型C胜。如果排序结果中语言模型C的响应结果排在语言模型D的响应结果后面，则语言模型C负。如果排序结果中语言模型C的响应结果与语言模型D的响应结果并列，则两个语言模型平。基于指令集中的多个指令，根据两个语言模型针对同一指令的响应结果的排序结果，可以计算出其中任一语言模型的胜、负和平的次数，或者计算任一语言模型的胜率。通过任一语言模型的胜、负和平的次数，或者计算任一语言模型的胜率，可以直观地展示该语言模型在待对比的多个语言模型中相对质量。
可选地，第一服务器可以根据各个语言模型针对同一指令的响应结果的排序结果，计算各个语言模型的埃洛等级分(Elo rating)，基于指令集中多个指令，多次计算各个语言模型的埃洛等级分(Elo rating)并取均值，作为各个语言模型的相对测评信息。通过输出各个语言模型的埃洛等级分的均值，可以直观地展示各个语言模型在待对比的多个语言模型中相对质量。
步骤S511、第一服务器向端侧设备输出各语言模型的响应质量信息和相对测评信息。
步骤S512、端侧设备接收各语言模型的响应质量信息和相对测评信息。
步骤S513、端侧设备输出各个语言模型的响应质量信息和相对测评信息。
本实施例中，通过获取人机交互的指令，将指令输入实现人机交互的大模型，通过大模型输出指令的响应结果；分别从伤害性、指令意图覆盖情况、事实性、内容质量的维度，对大模型输出的响应结果进行测评，得到响应结果在各个维度的响应质量信息；根据大模型输出的响应结果在各个维度的响应质量信息，计算大模型的响应质量信息，从而从伤害性、指令意图覆盖情况、事实性、内容质量等多个维度对大模型的响应结果进行准确、全面、更细粒度地测评，并输出大模型的响应质量信息；并且通过对各个大模型输出的响应结果的响应质量进行排序，基于排序结果确定各个大模型的相对测评信息；结合大模型的响应质量信息和相对测评信息，可以更好地对比各个大模型的响应质量，以指导大模型的上线判定、或更新大模型的优化版本、或选择优质的目标大模型，从而提升基于大模型的人机对话的准确性，保证人机交互质量。
图6为本公开实施例提供的一种服务器的结构示意图。如图6所示，该服务器包括：存储器601和处理器602。存储器601，被设置为存储计算机执行指令，并可被配置为存储其它各种数据以支持在服务器上的操作。处理器602，与存储器601通信连接，被设置为执行存储器601存储的计算机执行指令，以实现上述任一方法实施例中第一服务器所执行的技术方案，其具体功能和所能实现的技术效果类似，此处不再 赘述。
可选的，如图6所示，该服务器还包括：防火墙603、负载均衡器604、通信组件605、电源组件606等其它组件。图6中示意性给出部分组件，并不意味着服务器只包括图6所示组件。
本公开实施例还提供一种端侧设备，该端侧设备包括：存储器和处理器。存储器被设置为存储计算机执行指令，并可被配置为存储其它各种数据以支持在端侧设备上的操作。处理器与存储器通信连接，被设置为执行存储器存储的计算机执行指令，以实现上述任一方法实施例中端侧设备所执行的技术方案，其具体功能和所能实现的技术效果类似，此处不再赘述。
本公开实施例还提供一种计算机可读存储介质，计算机可读存储介质中存储有计算机执行指令，计算机执行指令被处理器执行时被设置为实现上述任一方法实施例中第一服务器所执行的技术方案，具体功能和所能实现的技术效果此处不再赘述。
本公开实施例还提供一种计算机可读存储介质，计算机可读存储介质中存储有计算机执行指令，计算机执行指令被处理器执行时被设置为实现上述任一方法实施例中端侧设备所执行的技术方案，具体功能和所能实现的技术效果此处不再赘述。
本公开实施例还提供了一种计算机程序产品，计算机程序产品包括：计算机程序，计算机程序存储在可读存储介质中，端侧设备的至少一个处理器可以从可读存储介质读取计算机程序，至少一个处理器执行计算机程序使得端侧设备执行上述任一方法实施例中端侧设备所执行的技术方案，具体功能和所能实现的技术效果此处不再赘述。
本公开实施例提供一种芯片，包括：处理模块与通信接口，该处理模块能执行前述方法实施例中第一服务器或端侧设备的技术方案。可选的，该芯片还包括存储模块(如，存储器)，存储模块被设置为存储指令，处理模块被设置为执行存储模块存储的指令，并且对存储模块中存储的指令的执行使得处理模块执行前述任一方法实施例中第一服务器或端侧设备所执行的技术方案。
上述存储器可以是对象存储(Object Storage Service，OSS)。上述存储器可以由任何类型的易失性或非易失性存储设备或者它们的组合实现，如静态随机存取存储器(SRAM)，电可擦除可编程只读存储器(EEPROM)，可擦除可编程只读存储器(EPROM)，可编程只读存储器(PROM)，只读存储器(ROM)，磁存储器，快闪存储器，磁盘或光盘。
上述通信组件被配置为便于通信组件所在设备和其他设备之间有线或无线方式的通信。通信组件所在设备可以接入基于通信标准的无线网络，如移动热点(WiFi)，第二代移动通信系统(2G)、第三代移动通信系统(3G)、第四代移动通信系统(4G)/长期演进(LTE)、第五代移动通信系统(5G)等移动通信网络，或它们的组合。在 一个示例性实施例中，通信组件经由广播信道接收来自外部广播管理系统的广播信号或广播相关信息。在一个示例性实施例中，通信组件还包括近场通信(NFC)模块，以促进短程通信。例如，在NFC模块可基于射频识别(RFID)技术，红外数据协会(IrDA)技术，超宽带(UWB)技术，蓝牙(BT)技术和其他技术来实现。
上述电源组件，为电源组件所在设备的各种组件提供电力。电源组件可以包括电源管理系统，一个或多个电源，及其他与为电源组件所在设备生成、管理和分配电力相关联的组件。
本领域内的技术人员应明白，本发明的实施例可提供为方法、系统、或计算机程序产品。因此，本发明可采用完全硬件实施例、完全软件实施例、或结合软件和硬件方面的实施例的形式。而且，本发明可采用在一个或多个其中包含有计算机可用程序代码的计算机可用存储介质(包括但不限于磁盘存储器、只读光盘存储器(CD-ROM)、光学存储器等)上实施的计算机程序产品的形式。
本发明是参照根据本发明实施例的方法、设备(系统)、和计算机程序产品的流程图和/或方框图来描述的。应理解可由计算机程序指令实现流程图和/或方框图中的每一流程和/或方框、以及流程图和/或方框图中的流程和/或方框的结合。可提供这些计算机程序指令到通用计算机、专用计算机、嵌入式处理机或其他可编程数据处理设备的处理器以产生一个机器，使得通过计算机或其他可编程数据处理设备的处理器执行的指令产生用于实现在流程图一个流程或多个流程和/或方框图一个方框或多个方框中指定的功能的装置。
这些计算机程序指令也可存储在能引导计算机或其他可编程数据处理设备以特定方式工作的计算机可读存储器中，使得存储在该计算机可读存储器中的指令产生包括指令装置的制造品，该指令装置实现在流程图一个流程或多个流程和/或方框图一个方框或多个方框中指定的功能。
这些计算机程序指令也可装载到计算机或其他可编程数据处理设备上，使得在计算机或其他可编程设备上执行一系列操作步骤以产生计算机实现的处理，从而在计算机或其他可编程设备上执行的指令提供用于实现在流程图一个流程或多个流程和/或方框图一个方框或多个方框中指定的功能的步骤。
在一个典型的配置中，计算设备包括一个或多个处理器(CPU)、输入/输出接口、网络接口和内存。内存可能包括计算机可读介质中的非永久性存储器，随机存取存储器(RAM)和/或非易失性内存等形式，如只读存储器(ROM)或闪存(flash RAM)。内存是计算机可读介质的示例。
计算机可读介质包括永久性和非永久性、可移动和非可移动媒体可以由任何方法或技术来实现信息存储。信息可以是计算机可读指令、数据结构、程序的模块或其他数据。计算机的存储介质的例子包括，但不限于相变内存(PRAM)、静态随机存取存 储器(SRAM)、动态随机存取存储器(DRAM)、其他类型的随机存取存储器(RAM)、只读存储器(ROM)、电可擦除可编程只读存储器(EEPROM)、快闪记忆体或其他内存技术、只读光盘只读存储器(CD-ROM)、数字多功能光盘(DVD)或其他光学存储、磁盒式磁带，磁带磁磁盘存储或其他磁性存储设备或任何其他非传输介质，可被设置为存储可以被计算设备访问的信息。按照本文中的界定，计算机可读介质不包括暂存电脑可读媒体(transitory media)，如调制的数据信号和载波。
需要说明的是，本公开所涉及的用户信息(包括但不限于用户设备信息、用户属性信息等)和数据(包括但不限于用于分析的数据、存储的数据、展示的数据等)，均为经用户授权或者经过各方充分授权的信息和数据，并且相关数据的收集、使用和处理需要遵守相关法律法规和标准，并提供有相应的操作入口，供用户选择授权或者拒绝。
另外，在上述实施例及附图中的描述的一些流程中，包含了按照特定顺序出现的多个操作，但是应该清楚了解，这些操作可以不按照其在本文中出现的顺序来执行或并行执行，是用于区分开各个不同的操作，序号本身不代表任何的执行顺序。另外，这些流程可以包括更多或更少的操作，并且这些操作可以按顺序执行或并行执行。需要说明的是，本文中的“第一”、“第二”等描述，是用于区分不同的消息、设备、模块等，不代表先后顺序，也不限定“第一”和“第二”是不同的类型。“多个”的含义是两个以上，除非另有明确具体的限定。
需要说明的是，本公开所涉及的用户信息(包括但不限于用户设备信息、用户个人信息等)和数据(包括但不限于用于分析的数据、存储的数据、展示的数据等)，均为经用户授权或者经过各方充分授权的信息和数据，并且相关数据的收集、使用和处理需要遵守相关法律法规和标准，并提供有相应的操作入口，供用户选择授权或者拒绝。
本领域技术人员在考虑说明书及实践这里公开的发明后，将容易想到本公开的其它实施方案。本公开旨在涵盖本公开的任何变型、用途或者适应性变化，这些变型、用途或者适应性变化遵循本公开的一般性原理并包括本公开未公开的本技术领域中的公知常识或惯用技术手段。说明书和实施例被视为示例性的，本公开的真正范围和精神由下面的权利要求书指出。
应当理解的是，本公开并不局限于上面已经描述并在附图中示出的精确结构，并且可以在不脱离其范围进行各种修改和改变。本公开的范围由所附的权利要求书来限制。
工业实用性
本公开实施例提供了一种人机交互的数据处理方法及服务器，通过获取人机交互的指令，将指令输入实现人机交互的大模型，通过大模型输出指令的响应结果；分别 从伤害性、指令意图覆盖情况、事实性、内容质量的维度，对大模型输出的响应结果进行测评，得到响应结果在各个维度的响应质量信息；根据大模型输出的响应结果在各个维度的响应质量信息，计算大模型的响应质量信息，实现从伤害性、指令意图覆盖情况、事实性、内容质量等多个维度，对大模型的响应结果进行准确、全面、更细粒度地测评，并输出大模型的响应质量信息，大模型的响应质量信息被设置为指导大模型的上线判定、或更新大模型的优化版本、或选择优质的目标大模型，可以准确地选择优质模型，提升迭代更新/选择的大模型的质量，提升上线模型的质量，从而提升基于大模型的人机对话的准确性，保证人机交互质量。

Claims (14)

  1. 一种人机交互的数据处理方法，其特征在于，包括：
    获取人机交互的指令，将所述指令输入大模型，通过大模型输出所述指令的响应结果；
    分别从伤害性、指令意图覆盖情况、事实性、内容质量的维度，对所述大模型输出的响应结果进行测评，得到所述响应结果在各个维度的响应质量信息；
    根据所述大模型输出的响应结果在各个维度的响应质量信息，计算所述大模型的响应质量信息；
    输出所述大模型的响应质量信息，所述大模型的响应质量信息被设置为指导所述大模型的上线判定、或更新所述大模型的优化版本、或选择优质的目标大模型。
  2. 根据权利要求1所述的方法，其特征在于，所述分别从伤害性、指令意图覆盖情况、事实性、内容质量的维度，对所述大模型输出的响应结果进行测评，得到所述响应结果在各个维度的响应质量信息，包括：
    任一所述维度对应多个质量类别，不同的质量类别对应不同的响应质量信息，
    针对各所述维度，确定所述大模型输出的响应结果在各所述维度的质量类别；
    根据所述响应结果在各所述维度的质量类别，确定所述响应结果在各所述维度的响应质量信息。
  3. 根据权利要求2所述的方法，其特征在于，
    伤害性包括如下质量类别：无伤害、有伤害；
    指令意图覆盖情况包括如下质量类别：完全识别指令意图、部分识别指令意图、未能识别指令意图、不应拒绝的指令意图但拒绝；
    事实性包括如下质量类别：无事实性错误、常识性事实错误、知识性事实错误、同时出现常识性和知识性事实错误；
    内容质量包括如下质量类别：连贯性好、连贯性中、连贯性差。。
  4. 根据权利要求2所述的方法，其特征在于，所述针对各所述维度，确定所述响应结果在各所述维度的质量类别，包括：
    通过第一交互界面显示所述响应结果、以及各维度对应的质量类别，并提供对所述响应结果在各维度的质量类别的输入区域；
    响应于对所述第一交互界面的提交操作，获取所述输入区域内输入的所述响应结果在各维度的质量类别。
  5. 根据权利要求1所述的方法，其特征在于，根据所述大模型输出的响应结果在各个维度的响应质量信息，计算所述大模型的响应质量信息，包括：
    根据所述大模型输出的各所述响应结果在各个维度的响应质量信息，以及各个维度的权重系数，计算各所述响应结果的综合质量信息；
    根据所述大模型输出的各响应结果的综合质量信息，计算所述大模型的响应质量信息。
  6. 根据权利要求5所述的方法，其特征在于，还包括：
    显示各个维度的权重配置界面；
    获取在所述权重配置界面上配置的各个维度的权重系数。
  7. 根据权利要求5所述的方法，其特征在于，所述通过大模型输出所述指令的响应结果之后，还包括：
    输出所述响应结果；
    接收对所述响应结果标注的综合质量类别，所述综合质量类别包括：好、一般、差；
    所述根据所述大模型输出的各所述响应结果在各个维度的响应质量信息，综合计算各所述响应结果的综合质量信息之后，还包括：
    根据不同的综合质量类别对应的质量信息区间，将所述响应结果被标注的综合质量类别对应的质量信息区间，作为所述响应结果对应的质量信息区间；
    对所述指令的响应结果进行过滤，去除综合质量信息不在对应质量信息区间内的响应结果。
  8. 根据权利要求1-7中任一项所述的方法，其特征在于，所述获取人机交互的指令，将所述指令输入大模型，通过大模型输出所述指令的响应结果，包括：
    接收端侧设备发送的对多个大模型的响应质量测评请求；
    获取人机交互的指令，将所述指令分别输入所述多个大模型，得到各所述大模型输出的所述指令的响应结果。
  9. 根据权利要求8所述的方法，其特征在于，还包括：
    通过第二交互界面输出各所述大模型输出的所述指令的响应结果，所述第二交互界面上不显示响应结果与各所述大模型的对应关系；
    接收在所述交互界面内指定的各所述大模型输出的所述指令的响应结果的排序结果；
    根据各所述大模型输出的所述指令的响应结果的排序结果，计算各所述大模型的响应质量的相对测评信息；
    向所述端侧设备输出各所述大模型的响应质量信息和相对测评信息。
  10. 根据权利要求9所述的方法，其特征在于，还包括：
    根据各所述大模型的响应质量信息和/或相对测评信息，选择其中一个大模型作为目标大模型，并向端侧设备输出所述目标大模型的信息；
    或者，
    根据各所述大模型的响应质量信息和/或相对测评信息，选择其中一个大模型作为优化版本，更新所述大模型的优化版本。
  11. 根据权利要求1-7中任一项所述的方法，其特征在于，所述输出所述大模型的响应质量信息之后，还包括：
    根据所述大模型的响应质量信息，确定所述大模型是否满足上线条件；
    输出所述大模型的上线提示信息，所述上线提示信息指示所述大模型是否满足上线条件。
  12. 一种人机交互的数据处理方法，其特征在于，应用于服务器，包括：
    接收端侧设备发送的对多个语言模型的响应质量测评请求；
    获取人机交互的指令，将所述指令输入各所述语言模型，通过各所述语言模型输出所述指令的响应结果；
    分别从伤害性、指令意图覆盖情况、事实性、内容质量的维度，对各所述语言模型输出的响应结果进行测评，并生成各所述语言模型的响应质量信息；
    向端侧设备发送交互界面数据，所述交互界面数据包含各所述语言模型输出的所述指令的响应结果；
    接收端侧发送的在所述交互界面内指定的各所述语言模型输出的所述指令的响应结果的排序结果；
    根据各所述语言模型输出的所述指令的响应结果的排序结果，计算各所述语言模型的响应质量的相对测评信息；
    向所述端侧设备输出各所述语言模型的响应质量信息和相对测评信息。
  13. 一种人机交互的数据处理方法，其特征在于，应用于端侧设备，包括：
    向服务器发送对多个语言模型的响应质量测评请求；
    接收服务器发送的交互界面数据，所述交互界面数据包含各所述语言模型输出的响应结果，所述响应结果是通过如下方式生成的：获取人机交互的指令，将所述指令输入各所述语言模型，通过各所述语言模型输出所述指令的响应结果；
    根据所述交互界面数据显示交互界面，所述交互界面上显示各所述语言模型输出的所述指令的响应结果，所述交互界面上不显示响应结果与所述语言模型的对应关系；
    获取并向服务器发送在所述交互界面内指定的各所述语言模型输出的所述指令的响应结果的排序结果；
    接收各所述语言模型的响应质量信息和相对测评信息，其中各所述语言模型的响应质量信息是通过分别从伤害性、指令意图覆盖情况、事实性、内容质量的维度，对各所述语言模型输出的响应结果进行测评生成的，各所述语言模型的相对测评信息是根据各所述语言模型输出的所述指令的响应结果的排序结果计算得到的；
  14. 一种服务器，其特征在于，包括：处理器，以及与所述处理器通信连接的存储器；
    所述存储器存储计算机执行指令；
    所述处理器执行所述存储器存储的计算机执行指令，以实现如权利要求1-12中任 一项所述的方法。
PCT/CN2024/097611 2023-06-05 2024-06-05 人机交互的数据处理方法及服务器 Ceased WO2024251170A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202310652255.5A CN116383027B (zh) 2023-06-05 2023-06-05 人机交互的数据处理方法及服务器
CN202310652255.5 2023-06-05

Publications (1)

Publication Number Publication Date
WO2024251170A1 true WO2024251170A1 (zh) 2024-12-12

Family

ID=86965992

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2024/097611 Ceased WO2024251170A1 (zh) 2023-06-05 2024-06-05 人机交互的数据处理方法及服务器

Country Status (2)

Country Link
CN (1) CN116383027B (zh)
WO (1) WO2024251170A1 (zh)

Families Citing this family (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116383027B (zh) * 2023-06-05 2023-08-25 阿里巴巴（中国）有限公司 人机交互的数据处理方法及服务器
CN116992010B (zh) * 2023-08-02 2026-03-03 无知(北京)智慧科技有限公司 一种基于多模态大模型的内容分发和交互方法及系统
US20250053560A1 (en) 2023-08-10 2025-02-13 Infosys Limited Method and system for providing real-time assistance to users using generative artificial intelligence (ai) models
CN117763128B (zh) * 2024-01-18 2024-06-04 杭州阿里云飞天信息技术有限公司 人机交互的数据处理方法、服务器、存储介质和程序产品

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109816106A (zh) * 2017-11-20 2019-05-28 北京智合大方科技有限公司 一种基于语音识别和自然语言处理技术进行呼叫中心客服业务知识类应答质量评估系统
US20190377791A1 (en) * 2018-06-08 2019-12-12 International Business Machines Corporation Natural language generation pattern enhancement
CN113407813A (zh) * 2021-06-28 2021-09-17 北京百度网讯科技有限公司 确定候选信息的方法、确定查询结果的方法、装置、设备
CN115146047A (zh) * 2022-07-20 2022-10-04 阿里巴巴（中国）有限公司 信息处理方法、设备、存储介质及程序产品
CN115329036A (zh) * 2022-06-30 2022-11-11 中国人民解放军国防科技大学 一种基于文本匹配的文本摘要质量评估方法及系统
CN115455158A (zh) * 2022-08-24 2022-12-09 阿里巴巴（中国）有限公司 智能聊天的信息处理方法、装置、电子设备及存储设备
CN116383027A (zh) * 2023-06-05 2023-07-04 阿里巴巴（中国）有限公司 人机交互的数据处理方法及服务器

Family Cites Families (12)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9680644B2 (en) * 2013-07-25 2017-06-13 Technion Research And Development Foundation Limited User authentication system and methods
CN112035325B (zh) * 2020-09-01 2023-08-18 中国银行股份有限公司 文本机器人自动监控方法及装置
CN114547435B (zh) * 2020-11-24 2024-10-18 腾讯科技（深圳）有限公司 内容质量的识别方法、装置、设备及可读存储介质
CN113268994B (zh) * 2021-07-16 2021-10-01 中国平安人寿保险股份有限公司 基于胶囊网络的意图识别方法及装置
CN113655938B (zh) * 2021-08-17 2022-09-02 北京百度网讯科技有限公司 一种用于智能座舱的交互方法、装置、设备和介质
CN113722458B (zh) * 2021-08-27 2024-09-06 海信电子科技(武汉)有限公司 视觉问答处理方法、设备、计算机可读介质和程序产品
CN114117000B (zh) * 2021-11-11 2026-01-06 海信视像科技股份有限公司 应答方法、装置、设备及存储介质
CN114625866B (zh) * 2022-03-11 2024-11-22 腾讯科技（深圳）有限公司 训练摘要生成模型的方法、装置、设备及介质
CN114861653B (zh) * 2022-05-17 2023-08-22 马上消费金融股份有限公司 用于虚拟交互的语言生成方法、装置、设备及存储介质
CN114972823A (zh) * 2022-06-14 2022-08-30 北京有竹居网络技术有限公司 数据处理方法、装置、设备及计算机介质
CN115905520A (zh) * 2022-10-19 2023-04-04 武汉众智数字技术有限公司 一种基于自然语言处理的隐形人员挖掘方法及系统
CN115658853B (zh) * 2022-12-28 2023-04-11 中国气象局公共气象服务中心(国家预警信息发布中心) 一种基于自然语言处理的气象预警信息审核方法及系统

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109816106A (zh) * 2017-11-20 2019-05-28 北京智合大方科技有限公司 一种基于语音识别和自然语言处理技术进行呼叫中心客服业务知识类应答质量评估系统
US20190377791A1 (en) * 2018-06-08 2019-12-12 International Business Machines Corporation Natural language generation pattern enhancement
CN113407813A (zh) * 2021-06-28 2021-09-17 北京百度网讯科技有限公司 确定候选信息的方法、确定查询结果的方法、装置、设备
CN115329036A (zh) * 2022-06-30 2022-11-11 中国人民解放军国防科技大学 一种基于文本匹配的文本摘要质量评估方法及系统
CN115146047A (zh) * 2022-07-20 2022-10-04 阿里巴巴（中国）有限公司 信息处理方法、设备、存储介质及程序产品
CN115455158A (zh) * 2022-08-24 2022-12-09 阿里巴巴（中国）有限公司 智能聊天的信息处理方法、装置、电子设备及存储设备
CN116383027A (zh) * 2023-06-05 2023-07-04 阿里巴巴（中国）有限公司 人机交互的数据处理方法及服务器

Also Published As

Publication number Publication date
CN116383027B (zh) 2023-08-25
CN116383027A (zh) 2023-07-04

Similar Documents

Publication Publication Date Title
WO2024251168A1 (zh) 基于大模型的数据处理方法及服务器
WO2024251170A1 (zh) 人机交互的数据处理方法及服务器
US11948187B2 (en) Artificial intelligence based digital leasing assistant
EP4134900A2 (en) Method and apparatus for recommending content, method and apparatus for training ranking model, device, and storage medium
US10586237B2 (en) Method, apparatus, and computer-readable media for customer interaction semantic annotation and analytics
US10452984B2 (en) System and method for automated pattern based alert generation
US11102276B2 (en) System and method for providing more appropriate question/answer responses based upon profiles
US20240095446A1 (en) Artificial intelligence (ai) and natural language processing (nlp) for improved question/answer sessions in teleconferences
CN117421398A (zh) 人机交互方法、装置、设备以及存储介质
CN109033277A (zh) 基于机器学习的类脑系统、方法、设备及存储介质
CN109145168A (zh) 一种专家服务机器人云平台
CN113886606A (zh) 一种基于知识图谱的数据标注方法、装置、介质及设备
US20230119860A1 (en) Matching system, matching method, and matching program
WO2025005874A1 (zh) 基于人机交互模型或大模型的数据处理方法、服务器
CN118227868B (zh) 文本处理方法、装置、电子设备和存储介质
WO2024263099A1 (zh) 人机交互的数据处理方法及服务器
CN115099239A (zh) 一种资源识别方法、装置、设备以及存储介质
US10592806B1 (en) Management of the execution of collaborative projects
US10762089B2 (en) Open ended question identification for investigations
CN117573973A (zh) 资源推荐方法、装置、电子设备以及存储介质
CN118095424A (zh) 人机交互模型对齐、人机交互的方法、设备及程序产品
US12277146B2 (en) Method for updating information of point of interest, electronic device and storage medium
CN120851211A (zh) 基于大模型的语料数据生成方法、装置及智能体
CN120583429A (zh) 基于DeepSeek大模型与多智能体协同的移动通信网络多维竞对洞察方法
WO2025196544A1 (zh) 文本生成方法、服务器、存储介质和程序产品

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 24818702

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE