WO2021093821A1 - 智能助理评价、推荐方法、系统、终端及可读存储介质 - Google Patents

智能助理评价、推荐方法、系统、终端及可读存储介质 Download PDF

Info

Publication number
WO2021093821A1
WO2021093821A1 PCT/CN2020/128455 CN2020128455W WO2021093821A1 WO 2021093821 A1 WO2021093821 A1 WO 2021093821A1 CN 2020128455 W CN2020128455 W CN 2020128455W WO 2021093821 A1 WO2021093821 A1 WO 2021093821A1
Authority
WO
WIPO (PCT)
Prior art keywords
evaluation
intelligent assistant
external
comment
ability
Prior art date
Application number
PCT/CN2020/128455
Other languages
English (en)
French (fr)
Inventor
林震亚
屠要峰
郭斌
周祥生
李春霞
Original Assignee
中兴通讯股份有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 中兴通讯股份有限公司 filed Critical 中兴通讯股份有限公司
Publication of WO2021093821A1 publication Critical patent/WO2021093821A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/44—Arrangements for executing specific programs
    • G06F9/451—Execution arrangements for user interfaces
    • G06F9/453—Help systems
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
    • G06F16/33—Querying
    • G06F16/332—Query formulation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
    • G06F16/33—Querying
    • G06F16/332—Query formulation
    • G06F16/3329—Natural language query formulation or dialogue systems
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/24—Classification techniques
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/16—Sound input; Sound output
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/16—Sound input; Sound output
    • G06F3/167—Audio in a user interface, e.g. using voice commands for navigating, audio feedback
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F40/00—Handling natural language data
    • G06F40/20—Natural language analysis
    • G06F40/279—Recognition of textual entities
    • G06F40/289—Phrasal analysis, e.g. finite state techniques or chunking
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/44—Arrangements for executing specific programs
    • G06F9/451—Execution arrangements for user interfaces
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L15/00—Speech recognition
    • G10L15/22—Procedures used during a speech recognition process, e.g. man-machine dialogue

Definitions

  • the embodiments of the present application relate to but are not limited to the field of mobile communications, and specifically relate to but not limited to intelligent assistant evaluation, recommendation methods, systems, terminals, and readable storage media.
  • the dialogue system can be roughly divided into two types:
  • Non-task-oriented dialogue system also known as chat robot.
  • the task-oriented system is designed to help users complete actual and specific tasks, such as helping users find products, book hotels and restaurants, and so on.
  • the widely used method of task-oriented systems is to regard dialogue response as a pipeline.
  • the system first understands the information conveyed by humans, regards it as an internal state, and then takes a series of corresponding actions according to the strategy of the dialogue state. , And finally transform the action into a natural language expression.
  • the non-task-oriented dialogue system interacts with humans, provides reasonable response and entertainment functions, and usually focuses on conversations with people in open areas. Although the non-task-oriented system seems to be chatting, it has played a role in many real-world applications.
  • the data shows that in the online shopping scene, nearly 80% of the words are chat information, and the way to deal with these problems is closely related to the user experience.
  • the intelligent assistant evaluation and recommendation method, system, terminal, and readable storage medium provided by the embodiments of the present application.
  • the embodiment of the application provides an intelligent assistant evaluation method, which includes: evaluating a target intelligent assistant according to a preset evaluation plan; obtaining an evaluation result; and generating an evaluation report based on the evaluation result.
  • the embodiment of the present application also provides an intelligent assistant recommendation method, which includes: encapsulating the interfaces of at least two target intelligent assistants and accessing a unified management interface; acquiring the current needs of external users; and determining the corresponding Ability items and the priority of each ability item; obtain the evaluation report of the target intelligent assistant; determine the preferred intelligent assistant among the target intelligent assistants according to the ability item, the priority of the ability item, and the evaluation report; the management The interface provides the external user with an interface of the preferred intelligent assistant for the external user to use.
  • the embodiment of the present application also provides an intelligent assistant evaluation system, the intelligent assistant evaluation system includes: an evaluation module for evaluating a target intelligent assistant according to a preset evaluation plan; a first obtaining module for obtaining an evaluation result; The first generating module is used to generate an evaluation report according to the evaluation result.
  • the embodiment of the present application also provides an intelligent assistant recommendation system, including: an encapsulation module, used to encapsulate the interfaces of at least two target intelligent assistants, and access a unified management interface; and a fifth acquisition module, used to acquire external user information Current demand; a capability item determination module, used to determine the corresponding capability item and the priority of each capability item according to the current demand; the sixth acquisition module, used to acquire the evaluation report of the target intelligent assistant; a preference determination module, It is used to determine the preferred intelligent assistant among the target intelligent assistants according to the ability item, the priority of the ability item, and the evaluation report; the providing module is used for the management interface to provide the external user with the information of the preferred intelligent assistant The interface is used by the external user.
  • an encapsulation module used to encapsulate the interfaces of at least two target intelligent assistants, and access a unified management interface
  • a fifth acquisition module used to acquire external user information Current demand
  • a capability item determination module used to determine the corresponding capability item and the priority of each capability item according
  • the embodiment of the present application also provides an intelligent assistant evaluation terminal, including: a first processor, a first memory, and a first communication bus; the first communication bus is used to implement communication between the first processor and the first memory. Connection communication; the first processor is used to execute one or more first computer programs stored in the first memory to implement the steps of the intelligent assistant evaluation method according to any one of the above embodiments.
  • An embodiment of the present application also provides an intelligent assistant recommendation terminal, including: a second processor, a second memory, and a second communication bus; the second communication bus is used to implement communication between the second processor and the second memory. Connection communication; the second processor is used to execute one or more second computer programs stored in the second memory to implement the steps of the intelligent assistant recommendation method as described in the foregoing embodiment.
  • the embodiment of the present application also provides a readable storage medium, the computer readable storage medium stores one or more first computer programs, and the one or more first computer programs can be used by one or more first computer programs.
  • the processor executes to implement the steps of the intelligent assistant evaluation method described in any one of the foregoing embodiments.
  • the embodiments of the present application also provide a readable storage medium, the computer readable storage medium stores one or more second computer programs, and the one or more second computer programs can be used by one or more second computer programs.
  • the processor executes to implement the steps of the intelligent assistant recommendation method described in the foregoing embodiment.
  • FIG. 1 is a schematic flowchart of an intelligent assistant evaluation method provided in Embodiment 1 of the application;
  • FIG. 2 is a diagram of a typical target intelligent assistant architecture provided by Embodiment 1 of the application;
  • FIG. 3 is a schematic flowchart of an internal evaluation method according to Embodiment 1 of this application.
  • FIG. 5-1 is a schematic flowchart of an intention identification method provided in Embodiment 1 of this application;
  • Figure 5-2 is a schematic diagram of an example of a typical multi-object classification architecture provided in the first embodiment of this application;
  • FIG. 6 is a schematic flow chart of a method for generating a comment summary provided by Embodiment 1 of the application;
  • FIG. 7 is a schematic flowchart of an external evaluation method provided by Embodiment 1 of the application.
  • Fig. 8-1 is a schematic flowchart of a method for generating an evaluation report according to Embodiment 1 of this application;
  • Fig. 8-2 is a schematic diagram of a single-sentence modeling process provided in the first embodiment of this application.
  • Figure 8-3 is a schematic diagram of a sequence editing and modeling process provided in the first embodiment of the application.
  • FIG. 9 is a schematic flowchart of another intelligent assistant evaluation method provided by Embodiment 1 of the application.
  • FIG. 10 is a schematic flowchart of another intelligent assistant evaluation method provided by Embodiment 1 of this application.
  • FIG. 11 is a schematic flowchart of another intelligent assistant evaluation method provided by Embodiment 1 of this application.
  • FIG. 12 is a schematic flowchart of a method for recommending an intelligent assistant according to Embodiment 2 of this application;
  • FIG. 13 is a structural diagram of an intelligent assistant evaluation system provided by Embodiment 3 of this application.
  • FIG. 14 is a structural diagram of an intelligent assistant recommendation system provided by Embodiment 4 of this application.
  • FIG. 15 is a schematic structural diagram of an intelligent assistant evaluation terminal provided by Embodiment 5 of this application.
  • FIG. 16 is a schematic structural diagram of an intelligent assistant recommendation terminal provided by Embodiment 6 of this application.
  • an intelligent assistant evaluation method provided in this embodiment includes:
  • the target intelligent assistant targeted by the embodiments of the present application includes, but is not limited to, a system or device that can conduct a human-machine dialogue.
  • the dialogue is to help the user complete a specific task, such as helping the user to check the local weather, etc.; the dialogue may also be a casual chat, to accompany the user and relieve loneliness.
  • the intelligent assistant may also provide services to the user by collecting information such as the user's actions, expressions, and tone of voice, and even guide the user to perform corresponding actions.
  • the target intelligent assistants targeted in the embodiments of this application may also include, but are not limited to, devices that can communicate with other creatures that can have their own thoughts, including humans, for example, by interacting with dogs.
  • the language communication is analyzed to obtain a device that can understand the needs expressed by the dog's language, and it can also be used as the target intelligent assistant evaluated in the embodiments of the present application.
  • the target intelligent assistant in the embodiments of the present application can communicate with living beings, such as humans, through forms other than language, for example, by acquiring other biological signals of living beings, such as brain waves, for analysis. Processing, a device that gives biological feedback. For example, by analyzing the user’s biological signals including brain waves, it is understood that the user’s intention is to understand the weather tomorrow, and then through the next action, the user will be informed of the weather tomorrow by voice broadcast and/or text display. , To meet the current needs of users.
  • the solidified form of the target intelligent assistant may not be fixed.
  • the target intelligent assistant may use any device that satisfies certain conditions, such as a device with a speaker, to complete the communication with the user.
  • the target intelligent assistant in the embodiments of this application can understand source information from the user such as text, sound, voice, image, video, touch operation, etc. and complete related actions; it can also understand sensor input signals from the environment Wait for the source information and complete the relevant actions; at the same time, be able to understand the source information from the feedback and complete the relevant actions.
  • source information such as text, sound, voice, image, video, touch operation, etc. and complete related actions; it can also understand sensor input signals from the environment Wait for the source information and complete the relevant actions; at the same time, be able to understand the source information from the feedback and complete the relevant actions.
  • Figure 2 is a typical target intelligent assistant architecture diagram, which is described in detail as follows:
  • the user interface module obtains source information such as voice, text, touch, and gesture input by the user;
  • the user interface module outputs the source information flow to the information collection module
  • the information collection module outputs organized information containing context to the information understanding module
  • the information understanding module outputs the analysis result of the context information to the action decision module
  • the action decision module outputs the decision result to the information collection module to evaluate and select the optimal decision
  • the action decision module outputs the optimal decision result to the action module
  • the action module outputs feedback information to the information collection module
  • the action module outputs text, image, video, sound and other information to the user interface module
  • the user interface outputs voice, text, image, video, sound and other media streams to the user;
  • the action module outputs requests for form submission, resource acquisition or command execution to the information adaptation and exchange module;
  • the information adaptation and exchange module outputs request information such as control commands to external IoT devices;
  • the information adaptation and exchange module outputs request information such as form submission and resource acquisition to external applications
  • the information adaptation and exchange module outputs information such as commands to be executed to the robot
  • the information adaptation and exchange module obtains source information such as events and signals input by external sensors;
  • the information adaptation and exchange module obtains source information such as new knowledge or knowledge update input from external knowledge sources;
  • the information adaptation and exchange module obtains external collaboration requests, business status updates and other event source information
  • the information adaptation and exchange module outputs source information to the information collection module.
  • the smart assistant can understand source information such as text, sound, voice, image, video, and touch operations from the user and complete related actions; it can also understand source information such as sensor input signals from the environment and complete related actions ; At the same time, you can also understand the source information from the feedback and complete related actions.
  • An intelligent assistant should include at least one of the following: user interface, information collection, information understanding, action decision-making, action, information adaptation and exchange.
  • the user interface provides the user with keyboard, handwriting, touch, voice, gesture and other human-computer interaction methods for input of source information, and transmits information to the user through voice, text, image, sound, video, etc.;
  • the information collection module integrates various source information to form contextual information that the intelligent assistant can understand;
  • the information understanding module analyzes the contextual information organized by the information collection module, and predicts and generates information to support behavioral decision-making; at the same time, the module needs to learn source information such as internal and external knowledge and feedback to improve analysis and understanding Improvement of ability;
  • the action decision module selects an appropriate action or a group of actions based on the information generated by the information understanding module; at the same time, the module needs to expand its decision space and improve its planning ability based on internal and external knowledge, feedback and other source information;
  • the action module calls internal and external resources and executes corresponding actions according to the optimal decision generated by the action decision module. At the same time, the module feeds back the results of the action execution to the information collection module.
  • the information adaptation and exchange module is responsible for connecting internal and external resources and completing the data format conversion of internal and external resources.
  • the evaluation includes at least one of the following:
  • the classification of the evaluation can be divided by obtaining the source of the evaluation, and the method of obtaining the source of the evaluation can be divided by the identity of the evaluator participating in the evaluation, for example, if the evaluator’s identity is For the trained internal professional, the evaluation of the person is an internal evaluation. If the evaluator is an external user, the evaluation of the user is an external evaluation.
  • the source of the evaluation can also be divided through the evaluation transmission interface. For example, if a certain evaluation is collected through an interface provided to external users, then the evaluation is an external evaluation, and a certain evaluation is If collected through the interface provided to internal evaluators, the evaluation is an internal evaluation.
  • the evaluation of the target intelligent assistant may be only an internal evaluation or only an external evaluation. In some embodiments, the evaluation of the target intelligent assistant may also be a comprehensive evaluation that combines internal evaluation and external evaluation. It should be noted that for the evaluation of the target intelligent assistant, whether to select a pure internal evaluation or an external evaluation, or to select an internal evaluation combined with an external evaluation can be selected by those skilled in the art according to their needs.
  • the preset evaluation scheme includes an ability item to be evaluated for internal professionals to evaluate the intelligent ability of the target intelligent assistant.
  • the intelligent ability can be understood as the ability of the intelligent assistant to receive the user's needs, determine the user's needs, perform the next action that meets the user's needs, and improve its own business capabilities.
  • the internal evaluation may be based on the classification of the user's needs for the target intelligent assistant.
  • the user's needs for the target intelligent assistant can be summarized into the following four categories: emotional support, knowledge support, activity support and decision support. The following is the specific content required by various requirements:
  • Basic needs refer to small talk, and important needs refer to emotional dialogue, topic dialogue and heuristic dialogue.
  • the basic requirements are single-sentence command control, conversational form submission, and conversational form cancellation.
  • the key requirements are heuristic control, autonomous interactive control, scene linkage control, conversational form filling, and conversational form modification.
  • the basic requirements are personalized recommendation (interest-sensitive), dynamic task planning, and the key requirements are personalized recommendation (time-sensitive), personalized recommendation (association-sensitive), deductive reasoning, and task sequence planning (time-sensitive, cost-sensitive).
  • the target intelligent assistant's ability to respond to abnormal situations and autonomous learning capabilities further enhance its ability to meet user needs.
  • the working mode of the target intelligent assistant serving users can be divided into active and passive, which has a significant impact on the ability to meet user needs.
  • the intelligent ability of the target intelligent assistant includes but is not limited to at least one of the following abilities: interaction ability, decision-making ability, thing ability, and learning ability.
  • the interaction capability includes at least one of the following sub-capabilities: information feedback, information understanding, information identification, and information collection, among which:
  • Information feedback includes at least one of the following ability items: image generation, speech synthesis, abstract generation, natural language generation;
  • Information understanding includes at least one of the following ability items: dynamic theme drift, spatial understanding, emotional understanding, time understanding, video understanding, image understanding, natural language understanding (without context), natural language understanding (with context);
  • Information recognition includes at least one of the following capabilities: action recognition, emotion recognition, image recognition, speech recognition, and knowledge extraction;
  • Information collection includes at least one of the following capabilities: feedback information input, image input, video input, external event source input, text input, and voice input.
  • the decision-making ability includes at least one of the following sub-capabilities: planning, recommendation, and reasoning, where:
  • the plan includes at least one of the following capabilities: dynamic task planning, task sequence planning, and abnormal response planning;
  • Recommendation includes at least one of the following ability items: personalized recommendation;
  • Reasoning includes at least one of the following ability items: case reasoning, uncertainty reasoning, inductive reasoning, deductive reasoning.
  • the thing power includes at least one of the following sub-capabilities: third-party services, dialogue, control, task form submission, search, performance, business monitoring and handling, and knowledge questions and answers, among which:
  • Third-party services include at least one of the following capabilities: service access method, service system;
  • Dialogue includes at least one of the following ability items: multimodal dialogue, personalized dialogue, heuristic dialogue, task dialogue, emotional dialogue, small chat, active dialogue;
  • Control includes at least one of the following capabilities: scene linkage control, single-sentence command control, multi-modal control, heuristic control, autonomous interactive control;
  • the task form submission includes at least one of the following ability items: dialogue form, single biometric verification;
  • the search includes at least one of the following capabilities: vertical search, single sentence search, automatic reply search, heuristic search, image search, and drill-down search;
  • Performance includes at least one of the following capabilities: reliability, transaction process efficiency, availability, response speed, and initiative;
  • Business monitoring and handling includes at least one of the following capabilities: task exception handling, task exception notification, and task status management;
  • Knowledge Q&A includes at least one of the following ability items: open domain question and answer, context question and answer, graph question and answer, limited domain question and answer, information summary, and reading comprehension.
  • learning ability includes at least one of the following sub-ability items: feedback learning, personalized learning, algorithm optimization, and new knowledge learning, where:
  • Feedback learning includes at least one of the following ability items: online learning of user feedback;
  • Personalized learning includes at least one of the following ability items: real-time user portrait update, online feature learning;
  • Algorithm optimization includes at least one of the following capabilities: model fusion, model optimization, and small sample learning;
  • New knowledge learning includes at least one of the following ability items: new logic learning, new emotion learning, new task learning, new language expression learning, knowledge discovery, new speech learning, knowledge update, and new image learning.
  • evaluation direction of intelligent ability can also be adjusted and increased according to the development of technology, or the needs of the industry, users, etc.
  • Table 1 is a comparison table of optional low-energy-level classification standards. Of course, those skilled in the art can also make corresponding adjustments according to actual needs.
  • evaluation grade classification standard given in Table 1 is a schematic and feasible standard.
  • Table 2 gives an example of the classification of intelligent assistant capabilities. Those skilled in the art can refer to the intelligent ability of the intelligent assistant in Table 2. Divide the ranks to adjust the evaluation criteria in Table 1.
  • the preset evaluation scheme includes the ability items to be evaluated for the internal professionals to evaluate the intelligent ability of the target intelligent assistant
  • Evaluation results include internal evaluation results.
  • the preset evaluation plan includes the evaluation grade classification standard corresponding to each ability item to be evaluated, the evaluation case of the ability item to be evaluated, and the benchmark rating corresponding to the ability item to be evaluated;
  • the internal evaluation results include the actual internal rating and the percentage of compliance
  • the internal actual rating is the internal actual rating of the target intelligent assistant's ability to be evaluated when internal professionals evaluate the target intelligent assistant according to the preset evaluation plan;
  • the proportion of up-to-standard includes the ratio of the number of ability items to be evaluated whose actual rating is greater than or equal to the benchmark rating to the total number of ability items to be evaluated.
  • the internal evaluation result also includes a comprehensive rating
  • the comprehensive rating includes the calculation of the actual internal rating to obtain the comprehensive rating of the target intelligent assistant.
  • the percentage of up to standard includes the ratio of the number of ability items to be evaluated whose actual internal rating is greater than or equal to the benchmark rating, and the total number of ability items to be evaluated;
  • the preset evaluation plan can be at least a part of the evaluation equivalence division criteria as shown in Table 1, the benchmark rating of each ability item to be evaluated set by related personnel or algorithms, and the ability item to be evaluated Evaluation case.
  • the evaluation case can be a targeted command that can be executed by the target intelligent assistant based on the ability item to be evaluated.
  • the evaluation case includes at least the common format image (gif , jpg, png, etc.), request the target intelligent assistant to collect pictures, that is, to detect whether the target intelligent assistant can support the camera to take pictures, ask the target intelligent assistant to take a video, and intercept useful pictures from the video, and ask the target intelligent assistant to focus .
  • the target intelligent assistant A obtains the preset evaluation plan, and through the execution of the evaluation case of the target intelligent assistant A to be evaluated, the internal actual rating of the target intelligent assistant A's ability to be evaluated is generated.
  • the actual internal ratings of the test capability items are as follows: Case Reasoning Level 1, Uncertainty Reasoning Level 3, and Scene Linkage Control Level 5. At this time, the target intelligent assistant A’s compliance ratio is 33.3% (1/3). The above-mentioned actual internal ratings and the percentage of compliance with the standards are filled in the report template.
  • the target intelligent assistant after the target intelligent assistant obtains the preset evaluation plan, and generates the actual internal rating of each ability item to be evaluated, the target intelligent assistant further includes:
  • calculation method for calculating the internal actual rating to obtain the comprehensive rating of the target intelligent assistant can be selected by those skilled in the art as needed, such as weighted average, average number, and so on.
  • FIG. 4 is a flowchart of another internal evaluation provided by an embodiment of the application, as shown in Fig. 4:
  • the influencing factors of the intelligent ability level of the target intelligent assistant are comprehensively considered to formulate an evaluation plan that meets the needs of the target intelligent assistant.
  • the intelligent assistant product under evaluation and its characteristics should be identified, defined, and described before evaluation, including system source, purpose, and usage mode.
  • the purpose and scope of the evaluation should be determined, and the preset evaluation plan should be determined according to the ability items to be evaluated and the benchmark ratings corresponding to each ability item to be evaluated given by the evaluation grade classification standard.
  • S402 Encapsulate the interface of the target intelligent assistant and access the unified management interface.
  • the internal evaluation results include actual internal ratings.
  • the evaluation cases of the ability items to be evaluated in the preset evaluation plan are imported, and the evaluation is performed according to the evaluation cases of the ability items to be evaluated, and the evaluation is divided according to the evaluation level. The standard grades the target intelligent assistant's ability to be evaluated, and obtains the actual internal rating.
  • the target intelligent assistant's intelligent ability level is evaluated according to the evaluation purpose and the ability of the evaluated target intelligent assistant's function to meet the requirements.
  • ability items the target intelligent assistant meets and high-level ability items cover low-level ability items.
  • Table 1 the ability items in Table 1 as an example, such as: the feedback information input contains two levels of ability items 1, 4 (the same as level xx and none are counted). If the intelligent assistant meets the requirements of level 4, it will meet the requirements at the same time. 1, 4 two ability items.
  • the number of ability items that have reached the benchmark rating of the target intelligent assistant is counted, and the proportion of the ability items that have reached the benchmark rating is calculated based on the total number of evaluation ability items. That is, the actual internal rating of the target intelligent assistant is obtained to be greater than the number of ability items of the benchmark rating corresponding to the ability item, and the ratio of this number to the total number of ability items to be evaluated is obtained, and the proportion of the compliance items has been obtained.
  • the internal evaluation result also includes a comprehensive evaluation, and the comprehensive rating is calculated by calculating the actual internal rating to obtain the comprehensive rating.
  • the actual internal rating corresponding to each ability item to be tested is weighted average to obtain a comprehensive rating.
  • the weights can be set according to actual conditions and combined with the comprehensive ratings obtained through internal evaluations, so that when evaluating in the same industry or for the same purpose, the same weights of the ability items to be evaluated are used. Set a plan to ensure that the evaluation results are comparable.
  • the preset evaluation scheme includes the evaluation item of the external user evaluation target intelligent assistant, and the evaluation result of the external evaluation is the external evaluation result.
  • the external evaluation is mainly divided into two parts: intention recognition and comment summary.
  • Intention recognition is used to identify the intent of external user comments to confirm their emotional orientation;
  • the comment summary mainly combines different users’ opinions of various types. Ratings and reviews of services, giving comprehensive results.
  • evaluating the target intelligent assistant according to a preset evaluation plan, and obtaining the evaluation result includes:
  • the review summary is an external review result.
  • evaluation indicators should be determined.
  • the evaluation indicators include at least the purpose and scope of the evaluation and are in accordance with pre-designated external evaluation standards.
  • the given evaluation ability item system and ability items are used to determine the ability items to be evaluated.
  • the external user's comments on the evaluation ability item can be text or emoticons, etc.
  • the comments can also be initially screened to filter out comments that are obviously not of reference value, and retain those that meet the comment screening conditions.
  • the comment screening criteria may include, for example, expressions that are not related to the target intelligent assistant, large-scale copying of novels, essays, lyrics, etc., and large-scale copying and pasting of the same comments.
  • it can also record the number of filtered comments that do not meet the comment filtering conditions, the frequency of appearance, the user identity of the external user, the region, and the login method (WeChat, phone number, etc.), and analyze according to the record. .
  • the various ratings in this embodiment can be set according to their needs by those skilled in the art, such as simply using numbers as ratings, as shown in the table Level 1-6 in 1; It can also be Chinese or English words: good, very good, very good, etc.; it can also be an expression, such as crying, aggrieved, expressionless, smiling, laughing, crying, etc.; also You can adjust the brightness, color temperature, color, etc. through the progress bar.
  • identifying the comment intention of the comment includes:
  • the recognition model includes features, feature distances, and rating classification rules
  • the comment category may be determined by at least one of the following: comment language, such as English, Chinese, Japanese, etc.; comment composition, such as emoticons, text, pictures, text+emoticons, etc.
  • rewriting the comment according to the format includes: performing data cleaning on the comment, where the data cleaning includes, but is not limited to, segmentation, augmentation, and removal of stop words on the comment.
  • word segmentation includes recombining consecutive word sequences into word sequences according to certain specifications, for example, dividing "superficial” into “superficial” and " ⁇ ".
  • Augmentation can be understood as the addition of synonyms for a certain comment, for example, if the comment is “good”, then the comment will be expanded to “satisfactory” and so on.
  • Remove stop words Stop words are used in information retrieval to save storage space and improve search efficiency. Certain words or words will be automatically filtered before or after processing natural language data (or text). These words or Words are called Stop Words, and the removal of stop words is to remove such words.
  • rewriting the comment according to the format also includes: before performing data cleaning on the comment, unifying the data format of the comment.
  • the unification of the data format can unify the comments in accordance with the preset format rules, and the specific manner of the unification of the format can adopt related technologies known to those skilled in the art.
  • the preset format may also be specified by those skilled in the art according to actual needs.
  • obtaining the best comment recognition model corresponding to the comment category includes:
  • the target comment recognition model with the best embedding result is selected as the best comment recognition model.
  • target comment recognition model may be a model for each comment category preset by a person skilled in the art according to existing technical means.
  • model hyperparameter tuning and/or feature selection for the target comment recognition model can also be performed on the target comment recognition model.
  • obtaining the target comment recognition model includes:
  • the target comment recognition model is set up.
  • Figure 5-1 is a schematic flow diagram of an intention recognition method, as shown in Figure 5-1:
  • rewriting includes:
  • the measurement includes the distance of each category between each feature
  • the data set contains various methods such as cross-validation, as well as features such as feature splicing.
  • target comment recognition model also saves evaluation indicators and evaluation results to facilitate hyperparameter tuning and embedding.
  • S511 Perform model selection according to the embedding result.
  • Figure 5-2 provides an example of a typical multi-target classification architecture. Its core modules use multi-head attention and inception-resnet, which are briefly described below.
  • Sentence is the embedding form of the input sentence, including word embedding, character embedding, position embedding, etc.
  • Multi-head attention is a multi-head attention mechanism commonly used in generative models. This method is used to better extract sentence features.
  • pre_information is other information outside the text, such as contextual information, user ratings, etc. This kind of information is processed into the form of vector or matrix, and directly added to the result after multi-head attention.
  • the latter structure is a typical inception-resnet structure, which has proven its powerful feature extraction ability in the image field.
  • the difference is that the inception_resnet_c is disassembled because the multi-target situation is considered. Because the multi-head attention mechanism and full connection are used before using this architecture, the word vector is not split, but its abstract features are processed. It should be noted that in the multi-target classification, the features processed by more modules are more abstract and contain the previous loss function information. Therefore, the more the loss function through the process, the more fine-grained the intention corresponding to the loss function. As shown in Figure 5-2, loss_intent2 needs to be more granular than loss_intent1.
  • generating a review summary based on external ratings and review intent includes:
  • FIG. 6 provides a schematic diagram of the process of a method for generating a review summary, as shown in the review summary generating framework in FIG.
  • the framework is mainly based on a generative model. For cases with less training data, traditional extraction methods such as textteaser and textrank can be used for summarization. If the results obtained by such methods are of high quality, they can also be used as a generative framework after review. Training corpus.
  • the structure in Figure 6 mainly includes the following processes:
  • the decoder side adopts the decoder structure of the transformer.
  • the sentence processing of this part adopts the multi-head attention with mask, and the normalization adopts the layer normalization method, and then the result obtained is interacted with the final result of the encoder side by the multi-head attention method .
  • the whole process can be repeated Nx times, and the final result obtained is the final comment result.
  • the function of the review summary generation method is to automatically generate online external evaluation results for a specific service based on a large number of online user reviews and ratings given by users.
  • the external evaluation further includes:
  • test stop conditions stop obtaining comments and external ratings.
  • the test stop conditions include at least one of the following:
  • the use time of the target intelligent assistant is greater than the preset use time
  • the number of comments is greater than the preset number of comments
  • Figure 7 is a schematic flowchart of an external evaluation method for an intelligent assistant, as shown in Figure 7:
  • the influencing factors of the intelligent ability level of the target intelligent assistant are comprehensively considered to formulate an evaluation plan that meets the needs of the target intelligent assistant.
  • the intelligent assistant product under evaluation and its characteristics should be identified, defined, and described before evaluation, including system source, purpose, and usage mode.
  • the purpose and scope of the evaluation should be determined, and the preset evaluation plan should be determined according to the ability items to be evaluated and the benchmark ratings corresponding to each ability item to be evaluated given by the evaluation grade classification standard.
  • S702 Encapsulate the interface of the target intelligent assistant and access the unified management interface.
  • the ability items to be evaluated that the target intelligent assistant needs to be evaluated are provided to external users under a unified interface to support user comments and ratings.
  • the above-mentioned external users are ordinary users who use the target intelligent assistant. Their comments and ratings are given based on their own experience. There is no need for professional training for external users to make external users’ ratings or The comment standard is at the same standard. However, when necessary, some specific information of external users can be obtained for subsequent more accurate analysis of their comments and ratings. For example, to obtain the area of an external user, if the area of the external user is Xinjiang, when evaluating the information collection and voice input ability of the target intelligent assistant, if the external user in the area has a wide range of low scores and bad reviews, it can be Aimed at the training of the Xinjiang dialect for the target intelligent assistant to improve the service ability of this part of external users.
  • S704 Set test stop conditions, and stop obtaining user comments and external ratings on the target intelligent assistant's intelligent ability to be evaluated according to the test stop conditions.
  • test stop condition may be set by the user or other relevant personnel, equipment, or system before the external evaluation.
  • the test stop condition can also be set according to the actual situation after the external evaluation.
  • the test stop condition can also be set before the external evaluation starts, but during the external evaluation process, adjustments are made to form a new test stop condition.
  • test stop condition may be at least one of the following:
  • the use time of the target intelligent assistant is greater than the preset use time
  • the number of comments is greater than the preset number of comments
  • test stop condition may also be other conditions set by those skilled in the art as required.
  • comment summary includes the actual rating of the target intelligent assistant by the external user and the actual comment after processing.
  • the intelligent ability level of the target intelligent assistant is evaluated according to the evaluation purpose and the ability of the function of the evaluated target intelligent assistant to meet the requirements.
  • a comprehensive scoring method or other methods is used to form a reasonable evaluation result according to the evaluation ability item system and the ability item to be evaluated corresponding to the level of the intelligent ability of the target intelligent assistant, and the actual rating is calculated to obtain the comprehensive rating.
  • the actual rating corresponding to each ability item to be tested is weighted average to obtain a comprehensive rating.
  • generating an evaluation report based on the evaluation result includes:
  • the preset evaluation description template may be a template set by the user or the evaluator as needed, or one of multiple evaluation description templates preset by the system may be selected as the preset evaluation description template.
  • the influencing factors of the intelligent ability level of the target intelligent assistant are comprehensively considered, and an evaluation description template that meets the needs of the target intelligent assistant is formulated.
  • the preset evaluation description template may be understood as a text description of an evaluation report, and the preset evaluation description template may include but is not limited to at least one of the following:
  • the target intelligent assistant product to be evaluated and its characteristics should also be identified, defined and described, including the system source, purpose and usage method, etc., and the evaluation description template should be formulated based on the above information .
  • the evaluation result may be an evaluation result obtained from an external evaluation source, that is, a review summary, and/or an internal evaluation source, that is, an actual rating, a percentage of up to standard, and a comprehensive percentage.
  • filling the target data and target text information into the evaluation report template includes:
  • the target data and target text information are filled in the slots of the evaluation report template through single sentence modeling and sequence editing modeling.
  • the evaluation report template may be obtained by obtaining and filling in the content required for the preset evaluation report description template.
  • the preset evaluation description template can be understood as a text description of an evaluation report, and the content that the preset evaluation description template needs to fill in can include but is not limited to at least one of the following:
  • the process can use text summarization and text matching technology, that is, according to a large number of structured reports, select important texts and generate evaluations. Report template.
  • the evaluation result includes the results of internal evaluation and external evaluation, and information is extracted from the internal evaluation result and the external evaluation result to extract important target data and target text information.
  • the internal evaluation result is composed of at least one of actual rating, percentage of up to standard, and comprehensive rating
  • the external evaluation result includes a review summary, which includes actual reviews and actual ratings.
  • the final evaluation report can be generated directly by obtaining the content of the preset evaluation template and the internal and external evaluation results.
  • the entire process is a typical seq2seq problem.
  • the final evaluation report needs to give emotional orientation based on the pros and cons of the target intelligent assistant function.
  • this application can use the latest QuaSE framework, see Figure 8-2 and Figure 8-3, a specific process is as follows:
  • Figure 8-2 illustrates the modeling of a single sentence, where X and R are observed values, which represent sentences (for example, a user's evaluation of a certain function) and their corresponding values (for example, user ratings).
  • Z and Y are hidden variables, which are modeling representations of sentence content and sentence value-related attributes.
  • the optimization goal of the model is to enable the generated sentence X'to reconstruct the input sentence X to the maximum.
  • a regression function F is designed to learn the mapping relationship between the latent variable Y and the value R.
  • single-sentence modeling and sequence editing modeling models can be integrated into a unified optimization problem through end-to-end training.
  • the target data and target text information are filled into the slots of the evaluation report template.
  • evaluating the target intelligent assistant according to a preset evaluation plan includes: obtaining a preset evaluation plan; encapsulating at least one interface of the target intelligent assistant and accessing a unified management interface; and using the management interface according to the preset evaluation plan Evaluate the target intelligent assistants separately.
  • the input and output interfaces of different typical intelligent assistants are packaged and managed in a unified manner to ensure that the interfaces of different types of intelligent assistants can be used and tested online in the same manner.
  • the intelligent assistant product under evaluation and its characteristics should be identified, defined, and described, including the source, purpose, and usage of the system.
  • the purpose and scope of the evaluation should be determined, and the ability items to be evaluated should be determined based on the evaluation ability item system and ability items given by the evaluation grade division standard.
  • evaluation grading standard does not include or includes outdated ability items, it can be added or modified. All revised results can be voted by industry experts as needed. Once passed, they will be adopted as the new evaluation grading standard.
  • evaluation grade division standard can also be directly adjusted by those skilled in the art.
  • the weights of the ability items to be evaluated are set.
  • a unified index weight setting plan should be adopted to ensure that the evaluation results are comparable.
  • the evaluation report template includes the preset evaluation plan, the ability to be evaluated, the evaluation grade division standard, and the weight of the ability to be evaluated.
  • the above content is automatically generated into a structured template, and the evaluation results of each ability to be evaluated are reserved. Fill in the position.
  • S906 Encapsulate the target intelligent assistant interface that needs to be evaluated, and access the unified management interface.
  • S907 Import the data set required for evaluation, and perform evaluation based on the evaluation content in the evaluation plan.
  • the target intelligent assistant According to the preset evaluation plan, the target intelligent assistant generates the actual rating, the comprehensive rating, and the proportion of the target intelligent assistant's ability to be evaluated.
  • the target intelligent assistant's ability to be evaluated is rated, and according to the actual rating of different ratings corresponding to each level of each ability to be evaluated, the weighted sum is performed to obtain the comprehensive rating of the target intelligent assistant.
  • the weighted sum is performed to obtain the comprehensive rating of the target intelligent assistant.
  • S909 Fill in the actual rating, comprehensive rating, and percentage of compliance for each capability item to be evaluated in the evaluation report template.
  • the report text can be rewritten using the models shown in Figures 8-2 and 8-3 in combination with the evaluation objectives and content, and based on the tendency of internal expert evaluation results, to obtain the final evaluation report.
  • Fig. 10 is a schematic diagram of another method for evaluating an intelligent assistant, see Fig. 10:
  • the above information can be filled into the evaluation report description template through the scheme shown in Figure 8-2 and Figure 8-3.
  • S1002 Analyze the evaluation result, and extract target data and target text information.
  • the preset evaluation plan is automatically determined, the evaluation case is imported for internal evaluation, and the corresponding interface is opened for external evaluation to obtain internal and external evaluation results.
  • the evaluation results include internal evaluation results and external evaluation results.
  • the external evaluation result obtains external reviews and external ratings, and uses intention recognition to obtain a review summary. Finally, the internal evaluation results and external evaluation results are processed to extract target data and target text information.
  • S1003 Fill in the target data and target text information into the evaluation report template.
  • the multi-input source version of the model in Figure 8-2 and Figure 8-3 uses the multi-input source version of the model in Figure 8-2 and Figure 8-3 to automatically generate the final evaluation report based on the internal and external evaluation results.
  • the multi-input source version is obtained by customizing feature processing modules for different input data and training based on a large amount of historical evaluation data.
  • FIG. 11 is a system architecture diagram of an intelligent assistant evaluation system. Referring to FIG. 11, the flow of an intelligent assistant evaluation method executed by the system is as follows:
  • S1101 Perform data analysis and preprocessing on various evaluation data and store them in the database.
  • the evaluation data includes all kinds of ability items to be evaluated, their grade classification standards, and the main content of other evaluation target intelligent assistants, such as: basic overview of the target intelligent assistant product, evaluation purpose, evaluation object and scope, intelligent assistant Classification and definition of intelligent ability levels, evaluation assumptions and qualifications, evaluation basis, evaluation methods, evaluation procedures and implementation processes and conditions, descriptions of special matters, descriptions of restrictions on the use of evaluation reports, and evaluation report dates, etc.
  • the preset evaluation plan includes the ability items to be evaluated, evaluation cases, and evaluation grade division standards, etc.;
  • S1103 Through the encapsulated intelligent assistant interface, select a specific method for evaluation according to the preset evaluation plan;
  • S1104 If internal evaluation is required, select the corresponding evaluation content in the preset evaluation plan in the database for evaluation, and use the internal evaluation module to rate each ability item to be evaluated;
  • S1105 If external evaluation is required, open online testing and use through external interfaces, and use external evaluation modules to rate various services based on external ratings and comments of external users;
  • users use online, through the unified user interface provided, users can fully experience the services provided by various intelligent assistants.
  • the user's usage record will be fully recorded.
  • the user can score each service, which is divided into five levels: poor, poor, medium, relatively satisfied, and satisfied. At the same time, users can comment on this service.
  • S1106 Summarize the internal evaluation and external evaluation results, and automatically fill slots based on the evaluation report template formulated for this evaluation. In addition, automatically generate a short evaluation report based on the evaluation results and evaluation report templates of each capability item to be evaluated.
  • the target intelligent assistant is evaluated according to a preset evaluation scheme, the evaluation result is obtained, and an evaluation report is generated according to the evaluation result.
  • At least one of internal evaluations or external evaluations can be carried out according to the needs of users.
  • the internal evaluation has the evaluation results conducted by internal professionals from a professional perspective, and the external evaluation can obtain more samples of the actual experience of external users, providing a reference angle for a more comprehensive understanding of the intelligent assistant and the targeted improvement of the intelligent assistant.
  • the intelligent ability of the target intelligent assistant is measured and evaluated.
  • the preset evaluation scheme includes the evaluation grade classification standard.
  • the internal professional staff ranks the target intelligent assistant according to the evaluation grade classification standard.
  • the subject of the evaluation is the external user
  • the external rating and comments are the evaluations with their own colors made by the external users based on their own experience.
  • the comments and external ratings of external users are obtained, and then the comment intention is identified to form a comment summary.
  • the comment intention is identified to form a comment summary.
  • evaluating the target intelligent assistant according to the preset evaluation scheme may also include: encapsulating the interface of at least one target intelligent assistant and connecting to a unified management interface, so as to realize the evaluation of the target intelligent assistant through the management interface . It greatly saves the cost required for evaluation. Both internal professionals and external users can use the management interface to use multiple target intelligent assistants. At this time, external users and internal professionals no longer need to be one-to-one. Multiple target intelligent assistants use, evaluate, derive their evaluation results, and then generate evaluation reports. It's easier.
  • an intelligent assistant recommendation method including:
  • S1201 Encapsulate the interfaces of at least two target intelligent assistants and access a unified management interface
  • S1203 Determine the corresponding capability item and the priority of each capability item according to the current needs
  • the evaluation report can be obtained by using the method of the foregoing embodiment. Wherein, when the target intelligent assistant has an evaluation report record, the evaluation report can be used directly. Of course, real-time evaluation can also be carried out according to the ability items corresponding to the current needs of external users.
  • the evaluation can be an internal evaluation.
  • S1205 Determine the preferred intelligent assistant among the target intelligent assistants according to the ability item, the priority of the ability item, and the evaluation report;
  • the method for determining the preferred intelligent assistant may be sorted according to the rating of the corresponding ability item of each target intelligent assistant in the evaluation report. For example: the ability items corresponding to the current needs are ranked A, B, and C according to the priority; the current ability item A of the A intelligent assistant is rated 5, the ability item B is rated 4, and the ability item C is rated 6; the current B intelligence The assistant’s ability item A has a rating of 3, ability item B has a rating of 7, and ability item C has a rating of 10.
  • the determination of the preferred intelligent assistant among A and B can be determined according to preset rules. If the preset rule is to select only one ability item with the highest priority, then the A intelligent assistant is the preferred intelligent assistant.
  • each ability item of the A and B intelligent assistants can be weighted and averaged to obtain their respective weighted average ratings, and then the higher is selected.
  • the preset rules may also be other rules formulated by those skilled in the art as needed.
  • the management interface provides an interface of a preferred intelligent assistant for external users for use by external users.
  • this method is equivalent to that the external user currently has multiple target intelligent assistants to choose and use in the management interface.
  • a preferred intelligent assistant is provided for the external user, so that Its use experience can be improved. Eliminates the user's worries about trying the intelligent assistant one by one. Based on the evaluation results, this method automatically selects intelligent assistants according to the functions required by external users, thereby improving the quality of service.
  • the user can also rate and comment on each ability item, and the system will record it, so that the shortcomings of each function can be obtained and subsequent improvements can be facilitated.
  • this embodiment also provides an intelligent assistant evaluation system 1300, as shown in FIG. 13, which includes:
  • the evaluation module 1301 is used to evaluate the target intelligent assistant according to a preset evaluation plan
  • the first obtaining module 1302 is used to obtain the evaluation result
  • the first generating module 1303 is used to generate an evaluation report according to the evaluation result.
  • the evaluation module 1301 includes at least one of the following:
  • the internal evaluation module 13011 includes:
  • the second acquisition module 130111 is used to acquire a preset evaluation plan.
  • the preset evaluation plan includes the evaluation grade classification standard corresponding to each ability item to be evaluated, the evaluation case of the ability item to be evaluated, and the corresponding ability item to be evaluated Benchmark rating
  • the third obtaining module 130112 is used to obtain internal evaluation results, the internal evaluation results include actual internal ratings and percentage of compliance;
  • the internal actual rating is the internal actual rating of the target intelligent assistant's ability to be evaluated when internal professionals evaluate the target intelligent assistant according to the preset evaluation plan;
  • the proportion of up-to-standard includes the ratio of the number of ability items to be evaluated whose actual rating is greater than or equal to the benchmark rating to the total number of ability items to be evaluated.
  • the external evaluation module 13012 includes:
  • the fourth obtaining module 130121 is used to obtain external evaluation results, the external evaluation results including comments and external ratings of external users on the ability items to be evaluated;
  • the identification module 130122 is used to identify the comment intention of the comment
  • the second generating module 130123 is used to generate a review summary according to external ratings and review intentions.
  • this embodiment also provides an intelligent assistant recommendation system 1400, as shown in FIG. 14, which includes:
  • the encapsulation module 1401 is used to encapsulate the interfaces of at least two target intelligent assistants and access a unified management interface;
  • the fifth obtaining module 1402 is used to obtain the current needs of external users
  • the capability item determination module 1403 is used to determine the corresponding capability item and the priority of each capability item according to the current demand;
  • the sixth obtaining module 1404 is used to obtain the evaluation report of the target intelligent assistant
  • the optimal determination module 1405 is used to determine the optimal intelligent assistant among the target intelligent assistants according to the ability item, the priority of the ability item, and the evaluation report;
  • a providing module 1406 is used for the management interface to provide external users with a preferred intelligent assistant interface for external users to use.
  • This embodiment also provides an intelligent assistant evaluation terminal. As shown in FIG. 15, it includes a first processor 1501, a first memory 1503, and a first communication bus 1502, wherein:
  • the first communication bus 1502 is used to implement connection and communication between the first processor 1501 and the first memory 1503;
  • the first processor 1501 is configured to execute one or more computer programs stored in the first memory 1503 to implement at least one step in the evaluation of an intelligent assistant in the foregoing embodiment.
  • Embodiment 6 is a diagrammatic representation of Embodiment 6
  • This embodiment also provides an intelligent assistant recommendation terminal. As shown in FIG. 16, it includes a second processor 1601, a second memory 1603, and a second communication bus 1602, where:
  • the second communication bus 1602 is used to implement connection and communication between the second processor 1601 and the second memory 1603;
  • the second processor 1601 is configured to execute one or more computer programs stored in the second memory 1603 to implement at least one step in the intelligent assistant recommendation method in the second embodiment.
  • This embodiment also provides a computer-readable storage medium, which is included in any method or technology for storing information (such as computer-readable instructions, data structures, computer program modules, or other data). Volatile or non-volatile, removable or non-removable media.
  • Computer-readable storage media include but are not limited to RAM (Random Access Memory), ROM (Read-Only Memory, read-only memory), EEPROM (Electrically Erasable Programmable read only memory, charged Erasable Programmable Read-Only Memory) ), flash memory or other memory technology, CD-ROM (Compact Disc Read-Only Memory), digital versatile disk (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, Or any other medium that can be used to store desired information and that can be accessed by a computer.
  • the computer-readable storage medium in this embodiment can be used to store one or more first computer programs, and the stored one or more first computer programs can be executed by a processor to implement the evaluation of the intelligent assistant in the first embodiment above. At least one step of the method.
  • the computer-readable storage medium in this embodiment can be used to store one or more second computer programs, and the stored one or more second computer programs can be executed by the processor to implement the intelligent assistant recommendation in the second embodiment above. At least one step.
  • This embodiment also provides a computer program (or computer software).
  • the computer program can be distributed on a computer-readable medium and executed by a computable device, so as to implement at least one of keeping resources consistent in the first embodiment above. Steps; and in some cases, at least one step shown or described can be performed in a different order than described in the above-mentioned embodiments.
  • This embodiment also provides a computer program (or computer software).
  • the computer program can be distributed on a computer-readable medium and executed by a computable device to implement at least one of keeping resources consistent in the second embodiment above. Steps; and in some cases, at least one step shown or described can be performed in a different order than described in the above-mentioned embodiments.
  • This embodiment also provides a computer program product, including a computer readable device, and the computer readable device stores the computer program as shown above.
  • the computer-readable device in this embodiment may include the computer-readable storage medium as shown above.
  • communication media usually contain computer-readable instructions, data structures, computer program modules, or other data in a modulated data signal such as carrier waves or other transmission mechanisms, and may include any information delivery medium. Therefore, this application is not limited to any specific combination of hardware and software.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • Human Computer Interaction (AREA)
  • Software Systems (AREA)
  • Audiology, Speech & Language Pathology (AREA)
  • Health & Medical Sciences (AREA)
  • Computational Linguistics (AREA)
  • Data Mining & Analysis (AREA)
  • Multimedia (AREA)
  • Artificial Intelligence (AREA)
  • General Health & Medical Sciences (AREA)
  • Mathematical Physics (AREA)
  • Databases & Information Systems (AREA)
  • Acoustics & Sound (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Evolutionary Computation (AREA)
  • Evolutionary Biology (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
  • Management, Administration, Business Operations System, And Electronic Commerce (AREA)

Abstract

本申请实施例提供一种智能助理评价、推荐方法、系统、终端及可读存储介质，该智能助理评价方法通过根据预设评价方案对目标智能助理进行评价，获取评价结果，其中评价包括以下至少之一：内部评价、外部评价，根据该评价结果生成评价报告。本申请实施例还提供了一种智能助理推荐方法、系统、终端及可读存储介质。

Description

智能助理评价、推荐方法、系统、终端及可读存储介质
相关申请的交叉引用
本申请基于申请号为201911115568.7、申请日为2019年11月14日的中国专利申请提出，并要求该中国专利申请的优先权，该中国专利申请的全部内容在此以引入方式并入本申请。
技术领域
本申请实施例涉及但不限于移动通信领域，具体而言，涉及但不限于智能助理评价、推荐方法、系统、终端及可读存储介质。
背景技术
拥有一个虚拟助理或一个拥有足够智能的聊天伙伴系统在目前来看似乎是虚幻的，而且在很长一段时间，人们都认为这可能只在科幻电影中存在。但近年来，人机对话因其潜在的潜力和诱人的商业价值而受到越来越多研究者的关注。
随着大数据和深度学习技术的发展，创建一个自动的人机对话系统作为人们的私人助理或聊天伙伴，将不再是一个幻想。且，目前市面上也出现了一些简单的人机对话系统设备，能够与用户进行简单的对话和提供简单的服务。
其中，具体来说，对话系统大致可分为两种:
(1)任务导向型(task-oriented)对话系统
(2)非任务导向型(non-task-oriented)对话系统(也称为聊天机器人)。
面向任务的系统旨在帮助用户完成实际具体的任务，例如帮助用户找寻商品，预订酒店餐厅等。
面向任务的系统的广泛应用的方法是将对话响应视为一条管道(pipeline)，系统首先理解人类所传达的信息，将其作为一种内部状态，然后根据对话状态的策略采取一系列相应的行为，最后将动作转化为自然语言的表现形式。
虽然语言理解是通过统计模型来处理的，但是大多数已经部署的对话系统仍然使用手工的特性或手工制定的规则，用于状态和动作空间表示、意图检测和插槽填充。
非任务导向的对话系统与人类交互，提供合理的回复和娱乐消遣功能，通常情况下主要集中在开放的领域与人交谈。虽然非任务导向的系统似乎在进行聊天，但是它在许多实际应用程序中都发挥了作用。
数据显示，在网上购物场景中，近80％的话语是聊天信息，处理这些问题的方式与用户体验密切相关。
随着智能助理逐渐普及，当前针对智能助理的评价还没有一个合理的评价方式，智能助理的服务质量良莠不齐，用户体验度低，因而创建一个智能助理评价系统就变得至关重要。
发明内容
本申请实施例提供的智能助理评价、推荐方法、系统、终端及可读存储介质。
本申请实施例提供一种智能助理评价方法，包括：根据预设评价方案对目标智能助理进行评价；获取评价结果；根据所述评价结果生成评价报告。
本申请实施例还提供了一种智能助理推荐方法，包括：封装至少两个目标智能助理的接口，并接入统一的管理界面；获取外部用户的当前需求；根据所述当前需求确定相对应的能力项及各能力项的优先级；获取所述目标智能助理的评价报告；根据所述能力项、能力项的优先级、评价报告确定所述各目标智能助理中的优选智能助理；所述管理界面为所述外部用户提供所述优选智能助理的接口以供所述外部用户使用。
本申请实施例还提供了一种智能助理评价系统，所述智能助理评价系统包括：评价模块，用于根据预设评价方案对目标智能助理进行评价；第一获取模块，用于获取评价结果；第一生成模块，用于根据所述评价结果生成评价报告。
本申请实施例还提供了一种智能助理推荐系统，包括：封装模块，用于封装至少两个目标智能助理的接口，并接入统一的管理界面；第五获取模块，用于获取外部用户的当前需求；能力项确定模块，用于根据所述当前需求确定相对应的能力项及各能力项的优先级；第六获取模块，用于获取所述目标智能助理的评价报告；优选确定模块，用于根据所述能力项、能力项的优先级、评价报告确定所述各目标智能助理中的优选智能助理；提供模块，用于所述管理界面为所述外部用户提供所述优选智能助理的接口以供所述外部用户使用。
本申请实施例还提供了一种智能助理评价终端，包括：第一处理器、第一存储器及第一通信总线；所述第一通信总线用于实现第一处理器和第一存储器之间的连接通信；所述第一处理器用于执行第一存储器中存储的一个或者多个第一计算机程序，以实现如上述实施例中任一项所述的智能助理评价方法的步骤。
本申请实施例还提供了一种智能助理推荐终端，包括：第二处理器、第二存储器及第二通信总线；所述第二通信总线用于实现第二处理器和第二存储器之间的连接通信；所述第二处理器用于执行第二存储器中存储的一个或者多个第二计算机程序，以实现如上述实施例所述的智能助理推荐方法的步骤。
本申请实施例还提供了一种可读存储介质，所述计算机可读存储介质存储有一个或者多个第一计算机程序，所述一个或者多个第一计算机程序可被一个或者多个第一处理器执行，以实现如上述实施例中任一项所述的智能助理评价方法的步骤。
本申请实施例还提供了一种可读存储介质，所述计算机可读存储介质存储有一个或者多个第二计算机程序，所述一个或者多个第二计算机程序可被一个或者多个第二处理器执行，以实现如上述实施例所述的智能助理推荐方法的步骤。
本申请其他特征和相应的有益效果在说明书的后面部分进行阐述说明，且应当理解，至少部分有益效果从本申请说明书中的记载变的显而易见。
附图说明
图1为本申请实施例一提供的一种智能助理评价方法的流程示意图；
图2为本申请实施例一提供的一种典型的目标智能助理架构图；
图3为本申请实施例一内部评价方法的流程示意图；
图4为本申请实施例一提供的另一种内部评价方法的流程示意图；
图5-1为本申请实施例一提供的一种意图识别方法的流程示意图；
图5-2为本申请实施例一提供的一种典型的多目标分类架构示例示意图；
图6为本申请实施例一提供的一种评论摘要生成方法流程示意图；
图7为本申请实施例一提供的一种外部评价方法的流程示意图；
图8-1为本申请实施例一提供的一种评价报告生成方法的流程示意图；
图8-2为本申请实施例一提供的一种单句建模的流程示意图；
图8-3为本申请实施例一提供的一种序列编辑建模的流程示意图；
图9为本申请实施例一提供的另一种智能助理评价方法的流程示意图；
图10为本申请实施例一提供的另一种智能助理评价方法的流程示意图；
图11为本申请实施例一提供的另一种智能助理评价方法的流程示意图；
图12为本申请实施例二提供的一种智能助理推荐方法的流程示意图；
图13为本申请实施例三提供的一种智能助理评价系统的结构图；
图14为本申请实施例四提供的一种智能助理推荐系统的结构图；
图15为本申请实施例五提供的一种智能助理评价终端的结构示意图；
图16为本申请实施例六提供的一种智能助理推荐终端的结构示意图。
具体实施方式
为了使本申请的目的、技术方案及优点更加清楚明白，下面通过具体实施方式结合附图对本申请实施例作进一步详细说明。应当理解，此处所描述的具体实施例仅用以解释本申请，并不用于限定本申请。
实施例一：
请参见图1，本实施例提供的一种智能助理评价方法包括：
S101：根据预设评价方案对目标智能助理进行评价，获取评价结果；
S102：根据评价结果生成评价报告。
在一些实施例中，本申请实施例所针对的目标智能助理包括但不限于可以进行人机对话的系统或装置。该对话是帮助用户完成某一具体的任务，例如帮助用户查询当地天气等；该对话也可能是随意的闲聊，给用户以陪伴和排解孤独。当然智能助理还可能是通过收集用户的动作、表情、语气等信息，来为用户提供服务，甚至是引导用户进行相应的动作等。
当然，在一些实施例中，本申请实施例中所针对的目标智能助理还可以包括但不限于可以与包括人在内其他能够有自身思想的生物进行交流的装置，例如，通过对狗狗的语言交流进行分析，得到可以了解狗狗的语言所表达的需要的装置，也可以作为本申请实施例中所评价的目标智能助理。
在一些实施例中，本申请实施例中的目标智能助理与生物，例如人，的交流还可以通过除语言之外的形式，例如通过获取生物的其他一些生物信号，例如脑电波，进而进行分析处理，给予生物一些反馈的装置。例如，通过分析用户的包括脑电波在内的生物信号，进而了解用户的意图是需要了解明天的天气，进而通过下一步的动作，将明天的天气以语音播报和/或文字显示的方式告知用户，满足用户的当前需求。
在一些实施例中，目标智能助理的固化形态可以并不是固定的，例如该目标智能助理可以借助任意的满足一定条件如带有扬声器的装置，来完成与用户之间的交流。
在一些实施例中，本申请实施例中的目标智能助理能够理解来自用户的文本、声音、语 音、图像、视频、触控操作等源信息并完成相关行动；也能够理解来自环境的传感器输入信号等源信息并完成相关行动；同时，也能够理解来自反馈的源信息并完成相关行动。参见图2，图2为一种典型的目标智能助理架构图，具体说明如下：
1)用户界面模块获得用户输入的语音、文字、触控、手势等源信息；
2)用户界面模块向信息收集模块输出源信息流；
3)信息收集模块向信息理解模块输出整理好的包含上下文的信息；
4)信息理解模块向行动决策模块输出上下文信息的解析结果；
5)行动决策模块向信息收集模块输出决策结果，用以评估遴选最优决策；
6)行动决策模块向行动模块输出最优决策结果；
7)行动模块向信息收集模块输出反馈信息；
8)行动模块向用户界面模块输出文本、图像、视频、声音等信息；
9)用户界面向用户输出语音、文本、图像、视频、声音等媒体流；
10)行动模块向信息适配及交换模块输出表单提交、资源获取或命令执行等请求；
11)信息适配及交换模块向外部物联网设备输出控制命令等请求信息；
12)信息适配及交换模块向外部应用输出表单提交、资源获取等请求信息；
13)信息适配及交换模块向机器人输出待执行的命令等信息；
14)信息适配及交换模块获得外部传感器等输入的事件、信号等源信息；
15)信息适配及交换模块获得外部知识源等输入的新知识或知识更新等源信息；
16)信息适配及交换模块获得外部的协作请求、业务状态更新等其他事件源信息；
17)信息适配及交换模块向信息收集模块输出源信息。
在一些实施例中，智能助理能够理解来自用户的文本、声音、语音、图像、视频、触控操作等源信息并完成相关行动；也能够理解来自环境的传感器输入信号等源信息并完成相关行动；同时，也能够理解来自反馈的源信息并完成相关行动。一个智能助理应至少包含以下至少之一：用户界面、信息收集、信息理解、行动决策、行动、信息适配及交换六个部分。
智能助理各部分功能如下：
a)用户界面为用户提供键盘、手写、触摸、语音、手势等人机交互方式进行源信息输入，并且通过语音、文本、图像、声音、视频等方式向用户传递信息；
b)信息收集模块将各种源信息进行融合，形成智能助理可以理解的上下文信息；
c)信息理解模块对信息收集模块整理好的上下文信息进行分析，并预测和产生用以支持行为决策的信息；同时，该模块需要学习来自内外部的知识、反馈等源信息，提高分析、理解能力的提高；
d)行动决策模块根据信息理解模块产生的信息选择合适的一个或一组行动；同时，该模块需要根据来自内外部的知识、反馈等源信息实现其决策空间的扩展和规划能力的提升；
e)行动模块根据行动决策模块产生的最优决策调取内外部资源并执行相应行动，同时该模块将行动执行结果反馈给信息收集模块。
f)信息适配及交换模块负责连接内外部资源，并完成内外部资源的数据格式转换。
在一些实施例中，评价包括以下至少之一：
内部评价、外部评价。
需要说明的是，在一些实施例中，评价的分类可以通过获取评价的来源来进行划分，获取评价的来源的方式可以通过参与评价的评价者的身份来划分，例如若评价者的身份是经过培训的内部专业人员，则该人员的评价为内部评价，若评价者的身份是外部用户，则该用户的评价为外部评价。
在一些实施例中，评价的来源的划分也可以通过该评价传输的接口来划分，例如，某一条评价是通过提供给外部用户的接口收集到的，则该评价为外部评价，某一条评价是通过提供给内部评价人员的接口收集到的，则该评价为内部评价。
在一些实施例中，对目标智能助理进行评价可以是仅有内部评价或仅有外部评价。在一 些实施例中，对目标智能助理的评价也可以是结合内部评价和外部评价而进行的综合评价。需要说明的是，针对目标智能助理的评价，是选择单纯的内部评价或外部评价，还是选择内部评价结合外部评价的方式可以由本领域技术人员根据需求进行选择。
在一些实施例中，若来源包括内部评价，则预设评价方案包括内部专业人员评价目标智能助理的智能能力的待测评能力项。
需要说明的是，在一些实施例中，智能能力可以理解为智能助理接收用户的需求、判断用户需求、执行符合用户需求的下一步动作、以及提升自身业务能力的能力。
需要说明的是，在一些实施例中，内部评价可以基于用户对目标智能助理的需求的分类进行评价。其中用户对于目标智能助理的需求可以概括为以下四大类：情感支持、知识支持、活动支持及决策支持。以下为各类需求所需要的具体内容：
a)情感支持
1)通过人机交互对用户给予鼓励、关心、和爱护，打发时间，减少孤独感等负面情绪；
2)基础需求指闲聊，重要需求指情感对话、主题对话和启发式对话。
b)知识支持
1)为用户提供知识问答和知识搜索；
2)基础需求是限定域问答和单句搜索，重点需求是开放域问答和下钻式搜索。
c)活动支持
1)代替人进行日常生活中的重要活动，譬如控制家电、播放视听内容、购物、打扫卫生、信息查询等；
2)基础需求是单句指令控制、对话式表单提交、对话式表单取消，重点需求是启发式控制、自主交互式控制、场景联动控制、对话式表单填写、对话式表单修改。
d)决策支持
1)为用户做出推荐、规划等决策建议；
2)基础需求是个性化推荐(兴趣敏感)、动态任务规划，重点需求是个性化推荐(时间敏感)、个性化推荐(关联敏感)、演绎推理、任务序列规划(时间敏感、成本敏感)。
围绕这些需求展开对于智能能力框架上的其他能力的需要。目标智能助理面对异常情况的应变能力以及自主学习能力进一步提升了其满足用户需求的能力。目标智能助理为用户服务的工作模式可以分为主动和被动两种，对于满足用户需求的能力有显著影响。
在一些实施例中，目标智能助理的智能能力包括但不限于以下能力中至少之一：交互能力、决策能力、事物能力、学习能力。
在一些实施例中，交互能力包括以下子能力项中至少之一：信息反馈、信息理解、信息识别、信息收集，其中：
信息反馈包括以下能力项至少之一：图像生成、语音合成、摘要生成、自然语言生成；
信息理解包括以下能力项至少之一：动态主题漂移、空间理解情感理解、时间理解、视频理解、图像理解、自然语言理解(不含上下文)、自然语言理解(含上下文)；
信息识别包括以下能力项至少之一：动作识别、情感识别、图像识别、语音识别、知识抽取；
信息收集包括以下能力项至少之一：反馈信息输入、图像输入、视频输入、外部事件源输入、文本输入、语音输入。
在一些实施例中，决策能力包括以下子能力项中至少之一：规划、推荐、推理，其中：
规划包括以下能力项至少之一：动态任务规划、任务序列规划、异常应对规划；
推荐包括以下能力项至少之一：个性化推荐；
推理包括以下能力项至少之一：案例推理、不确定性推理、归纳推理、演绎推理。
在一些实施例中，事物力包括以下子能力项中至少之一：第三方服务、对话、控制、任务表单提交、搜索、性能、业务监控与处置、知识问答，其中：
第三方服务包括以下能力项至少之一：服务接入方式、服务体系；
对话包括以下能力项至少之一：多模态对话、个性化对话、启发式对话、任务型对话、情感对话、闲聊、主动型对话；
控制包括以下能力项至少之一：场景联动控制、单句指令控制、多模态控制、启发式控制、自主交互式控制；
任务表单提交包括以下能力项至少之一：对话式表、单生物特征验证；
搜索包括以下能力项至少之一：垂直搜索、单句搜索、回复自动搜索、启发式搜索、图像搜索、下钻式搜索；
性能包括以下能力项至少之一：可靠性、事务流程高效性、可用性、响应速度、主动性；
业务监控与处置包括以下能力项至少之一：任务异常处理、任务异常通知、任务状态管理；
知识问答包括以下能力项至少之一：开放域问答、上下文问答、图谱问答、限定域问答、信息摘要、阅读理解。
在一些实施例中，学习力包括以下子能力项中至少之一：反馈学习、个性化学习、算法优化、新知识学习，其中：
反馈学习包括以下能力项至少之一：用户反馈的在线学习；
个性化学习包括以下能力项至少之一：实时用户画像更新、在线特征学习；
算法优化包括以下能力项至少之一：模型融合、模型优化、小样本学习；
新知识学习包括以下能力项至少之一：新逻辑学习、新情感情绪学习、新任务学习、新言语表达学习、知识发现、新语音学习、知识更新、新图像学习。
需要说明的是，智能能力的测评方向还可以根据技术的发展，或者行业、用户等的需要进行调整，增加。
表1为一种可选的能力低能级划分标准对照表，当然本领域技术人员也可以根据实际需要进行相应的调整。
表1能力等级划分标准对照表
Figure PCTCN2020128455-appb-000001
Figure PCTCN2020128455-appb-000002
Figure PCTCN2020128455-appb-000003
Figure PCTCN2020128455-appb-000004
Figure PCTCN2020128455-appb-000005
Figure PCTCN2020128455-appb-000006
Figure PCTCN2020128455-appb-000007
Figure PCTCN2020128455-appb-000008
Figure PCTCN2020128455-appb-000009
Figure PCTCN2020128455-appb-000010
Figure PCTCN2020128455-appb-000011
Figure PCTCN2020128455-appb-000012
Figure PCTCN2020128455-appb-000013
Figure PCTCN2020128455-appb-000014
Figure PCTCN2020128455-appb-000015
Figure PCTCN2020128455-appb-000016
Figure PCTCN2020128455-appb-000017
Figure PCTCN2020128455-appb-000018
Figure PCTCN2020128455-appb-000019
Figure PCTCN2020128455-appb-000020
Figure PCTCN2020128455-appb-000021
Figure PCTCN2020128455-appb-000022
Figure PCTCN2020128455-appb-000023
Figure PCTCN2020128455-appb-000024
Figure PCTCN2020128455-appb-000025
Figure PCTCN2020128455-appb-000026
Figure PCTCN2020128455-appb-000027
需要说明的是，表1中给出的评价等级划分标准为一种示意的可行的标准，当然对于上述给出的智能能力评测项目各个主能力项，以及主能力项下分的具体的能力项的评价标准也可以存在一定程度上的更改，或者新增能力项，此时，表2给出了一种智能助理能力等级划分规定的示例，本领域技术人员可以根据表2的智能助理智能能力划分等级来对表1的评价标准来进行调整。
需要说明的是，智能助理智能能力等级划分规定也不是一成不变的，本领域的技术人员也可以根据需要进行适当修订。以为为表2：
表2智能助理智能能力等级划分规定
Figure PCTCN2020128455-appb-000028
Figure PCTCN2020128455-appb-000029
Figure PCTCN2020128455-appb-000030
在一些实施例中，若评价包括内部评价，则预设评价方案包括内部专业人员评价目标智能助理的智能能力的待测评能力项；
评价结果包括内部评价结果。
在一些实施例中，参见图3，其中，当评价为内部评价是，根据预设评价方案对目标智能助理进行评价，获取评价结果包括：
S301：获取预设评价方案；
在一些实施例中，该预设评价方案包括各待测评能力项所对应的评价等级划分标准、待测评能力项的测评案例，以及，待测评能力项所对应的基准评级；
S302：获取内部评价结果。
其中，内部评价结果包括内部实际评级和达标占比；
内部实际评级为内部专业人员根据预设评价方案对目标智能助理进行评测，得到的目标智能助理各待测评能力项的内部实际评级；
达标占比包括实际评级大于或等于基准评级的待测评能力项数量，与，待测评能力项总数量的比值。
在一些实施例中，内部评价结果还包括综合评级，
综合评级包括对内部实际评级进行计算，得到目标智能助理的综合评级。
需要说明的是，达标占比包括内部实际评级大于或等于基准评级的待测评能力项数量，与，待测评能力项总数量的比值；
需要说明的是，预设评价方案可以是如表1中所示的评价等价划分标准中的至少一部分、相关人员或算法所设定的各待测评能力项的基准评级、以及待测评能力项的测评案例。
需要说明的是，测评案例可以是基于待测评能力项针对性的可供目标智能助理执行的命令。例如，某一目标智能助理的图像输入能力项作为待测评能力项时，以表1的等价划分标准为例，则测评案例至少包括提供给目标智能助理并要求其输入的常见格式图像(gif,jpg,png等)，要求目标智能助理进行图片收集，也即检测目标智能助理是否可以支持摄像头拍照，要求目标智能助理进行摄像，并摄像的影片中截取有用的图片，要求目标智能助理进行对焦。
下面通过一个具体的实施例，对根据预设评价方案对目标智能助理进行评价这一过程进行更加易懂的解释说明：
针对对于当前目标智能助理A的预设评价方案为评测该智能助理的以下三个待测能力项进行评级：案例推理、不确定性推理以及场景联动控制，各能力项的等级划分标准参见表3，并确定各待测能力项的基准评级分别为：案例推理3级、不确定性推理4级以及场景联动控制2级。将目标智能助理A获取该预设评价方案，通过目标智能助理A对待测评能力项的测评案例的执行，生成目标智能助理A各待测评能力项的内部实际评级，假设目标智能助理A的各待测能力项的内部实际评级如下:案例推理1级、不确定性推理3级以及场景联动控制5级，则此时，该目标智能助理A的达标占比为33.3％(1/3)，将上述各内部实际评级及达标占比填写入报告模板中。
表3待测能力项等级划分标准对照表
Figure PCTCN2020128455-appb-000031
在一些实施例中，目标智能助理获取预设评价方案，生成目标智能助理各待测评能力项的内部实际评级之后，还包括：
对内部实际评级进行计算，得到目标智能助理的综合评级；
将综合评级填写入所述报告模板中。
需要说明的是，对内部实际评级进行计算，得到目标智能助理的综合评级的计算方法本领域技术人员可以根据需要进行选取，例如加权平均，取平均数等。
下面在通过一个具体的实施例，对本申请实施例中的内部评价过程进行进一步的说明。参见图4，图4为本申请实施例提供的另一种内部评价的流程图，如图4所示：
S401：制定预设评价方案。
在一些实施例中，根据评价目的需要，综合考虑目标智能助理的智能能力等级的影响因素，制定与其需求相符合的评价方案。可选择自行制定方案来实施评价，也可以委托专业机构或第三方制定评价方案，以期获得社会认可的结果。
在一些实施例中，评价前应识别、界定和描述被评估的智能助理产品及其特性，包括系统来源、用途和使用方式等。评价前应确定评价目的和范围，并按照评价等级划分标准所给出的待测评能力项和各待测评能力项所对应的基准评级来确定预设评价方案。
S402：封装目标智能助理的接口，接入统一的管理界面中。
S403：获取内部评价结果。
在一些实施例中，内部评价结果包括内部实际评级，在一些实施例中，导入预设评价方案中的待测评能力项的测评案例，根据待测评能力项的测评案例进行评价，根据评价等级划分标准对目标智能助理的各项待测评能力项进行评级，得到内部实际评级。
在一些实施例中，根据评价目的，结合被评价的目标智能助理的功能满足需求的能力，对目标智能助理智能能力等级进行评价。这里我们只考虑目标智能助理满足哪些能力项，高级别能力项覆盖低级别能力项。以表1中的能力项内容为例，如：反馈信息输入包含1、4两个级别的能力项(与xx级相同和无不计入内)，若该智能助理达到4级要求，则同时满足1、4两个能力项。
需要说明的是，内部评价结果还包括，达标占比。
在一些实施例中，统计目标智能助理达到基准评级的能力项数量，依据评价能力项的总数计算所达到基准评级的能力项占比。也即，获取目标智能助理的内部实际评级大于该能力项对应的基准评级的能力项的数量，将该数量与待测评能力项的总数量取比值，已得到达标占比。
在一些实施例中，内部评价结果还包括综合评价，综合评级通过对内部实际评级进行计算，得到综合评级。
在一些实施例中，是对各个待测评能力项对应的内部实际评级进行加权平均，以得到综合评级。
需要说明的是，在一些实施例中，可以根据实际情况，结合通过内部评价所得到的综合评级进行权重设定，以在同一行业或同一目的下进行评价时，采用同一的待测评能力项权重设定方案，进而保证评价结果具有可比性。
在一些实施例中，若评价包括外部评价，则预设评价方案包括外部用户评价目标智能助理的评测项目，外部评价的评价结果为外部评价结果。
需要说明的是，在一些实施例中，外部评价主要分为意图识别和评论摘要两部分，其中意图识别对外部用户评论进行意图识别，确认其情感倾向性；评论摘要主要结合不同用户对各类服务的评级和评论，给出综合性结果。
在一些实施例中，若评价为外部评价，则根据预设评价方案对目标智能助理进行评价，获取评价结果包括：
获取外部用户对待测评能力项的评论及外部评级；
识别评论的评论意图；
根据外部评级和评论意图生成评论摘要。
需要说明的是，在一些实施例中，评论摘要为外部评论结果。
需要说明的是，在一些实施例中，在获取外部用户对待测评能力项的评论及外部评级之前，还应确定评价指标，其中评价指标至少包括评价目的和范围，并按照预先指定的外部评价标准给出的评价能力项体系和能力项来确定待测评能力项。
需要说明的是，在一些实施例中，外部用户对待测评能力项的评论可以是文字或表情等，该评论也可以进行初步筛选，筛除明显不具有参考价值的评论，保留符合评论筛选条件的评论，该评论筛选条件可以包括如与目标智能助手不相关的表述，小说、散文、歌词的大段复 制等，复制黏贴的大段同样的评论。进一步的，还可就筛除到的不符合评论筛选条件的评论条数、出现的频率、外部用户的用户身份、地域、登录方式(微信、电话号码等)进行记录，并根据该记录进行分析。
需要说明的是，本实施例中的各种评级，如基准评级、外部评级、内部实际评级等的体现方式本领域的技术人员可以根据需要进行设定，例如单纯以数字为评级体现，如表1中的1-6级；还可以是中文或英文单词：好、很好、非常好等；还可以是表情，例如大哭、委屈、面无表情、微笑、大笑、笑哭等；还可以通过进度条，调整明暗、色温、颜色等等。
在一些实施例中，识别评论的评论意图包括：
获取评论的评论类别；
获取评论类别对应的格式；
根据格式对评论进行改写；
获取评论类别对应的最佳评论识别模型，识别模型包括特征、特征的距离、评价等级划分规则；
将评论录入最佳评论识别模型，得到评论的评论意图。
需要说明的是，在一些实施例中，评论类别可以是评论由以下至少一项来确定：评论语言，如英语、中文、日语等；评论构成，如表情、文字、图片、文字+表情等。
在一些实施例中，根据格式对评论进行改写包括：对评论进行数据清洗，其中，数据清洗包括但不限于对评论进行分词、增广、去停用词等。需要说明的是，分词包括将连续的字序列按照一定的规范重新组合成词序列，例如，将“表面的”划分为“表面”和“的”。增广可以理解为将某一评论的进行近义词的增加，例如，评论为“好”，则将该评论增广到“满意”等。去停用词，停用词是指为在信息检索中，为节省存储空间和提高搜索效率，在处理自然语言数据(或文本)之前或之后会自动过滤掉某些字或词，这些字或词即被称为Stop Words(停用词)，去停用词就是为了去除这类词。
需要说明的是，在一些实施例中，根据格式对评论进行改写还包括：在对评论进行数据清洗前，对评论进行数据格式统一。其中，数据格式统一可以将评论按照预设的格式规则进行统一，其格式统一的具体方式可以采用本领域技术人员所知悉的相关技术。预设的格式也可以是本领域技术人员根据实际需要进行规定的。
在一些实施例中，获取评论类别对应的最佳评论识别模型包括：
获取至少一个目标评论识别模型；
将评论嵌入各个目标评论识别模型中；
选取嵌入结果最佳的目标评论识别模型作为最佳评论识别模型。
需要说明的是，目标评论识别模型可以是本领域技术人员根据现有的技术手段预先设定的针对各个评论类别的模型。
在一些实施例中，在设定目标评论识别模型时，在初步设定目标评论识别模型后，还可以对该目标评论识别模型进行模型超参数调优和/或针对模型进行特征选择。
在一些实施例中，获取目标评论识别模型包括：
获取各评论类别的特征提取及降维方法；
对包含特征之间的各类别进行距离度量；
根据特征提取及降维方法、各特征之间的距离设定目标评论识别模型。
参见图5-1，图5-1为一种意图识别方法的流程示意图，如图5-1所示：
S501：获取评论的类别；
在一些实施例中，当获取到多条评论时，将不同问题分开，汇总数据集。
S502：获取评论的类别对应的格式；
S503：根据格式对评论进行改写；
在一些实施例中，改写包括：
统一数据格式；
和/或，
数据清洗，文本数据的分词、增广、去停用词等。
S504：获取各类别的特征的提取及降维方法；
S505：度量包含各特征之间的各类别的距离；
S506：根据特征建立模型可以读取的数据集；
需要说明的是，数据集中包含交叉验证等各类方法，同时包含特征拼接等功能。
S507：建立包括各类别问题的目标评论识别模型；
需要说明的是，目标评论识别模型还会保存评价指标及评测结果，方便超参数调优和embedding。
S508：对目标评论识别模型进行超参数调优；
S509：针对目标评论识别模型进行特征选择；
S510：采用多种方法embedding；
S511：根据embedding结果进行模型选择。
需要说明的是，对于数据量较少且文本长度较短的情况，采用诸如SVM、XGBOOST这类机器学习模型即可，对于数据量大，数据复杂的情况，采用深度学习模型可取得更好的效果，图5-2提供了一种典型的多目标分类架构示例，其核心模块采用了multi-head attention和inception-resnet，下面进行简要的说明。
Sentence为输入语句的嵌入形式，包含词嵌入、字符嵌入、位置嵌入等，multi-head attention就是生成模型中常用的多头注意力机制，这里采用该方法更好的提取句子特征。
pre_information为文本外的其它信息，如前后文信息、用户评分等。将这类信息处理为向量或是矩阵的形式，与multi-head attention后的结果直接相加。
后面的结构为典型的inception-resnet结构，该结构在图像领域已经证明了其强大的特征抽取能力。区别在于因为考虑了多目标情况，故而将inception_resnet_c拆解了出来。由于使用该架构前采用了多头注意力机制和全连接，故而并没有拆分词向量，而是对其抽象特征进行了处理。需要注意的是，多目标分类中，经过越多模块处理的特征越抽象，且包含之前的损失函数信息，故而经历流程越多的损失函数所对应的意图越细粒度。如图5-2中，loss_intent2需比loss_intent1粒度更细。
以上结构中，若要增加分类目标，则需在后面继续接相关模块，再将loss相加即可。为了模型收敛，不建议一个模型包含太多分类目标，除非各个目标强相关。若没有外部信息或是仅为单目标问题，只需将多余的模块直接去除。
在一些实施例中，根据外部评级和评论意图生成评论摘要包括：
对评论意图进行第一语句处理，得到第一语句处理结果；
获取评论意图所对应的权重；
根据权重对第一语句处理结果和外部评级进行计算得到计算结果；
对评论意图进行第二语句处理，得到第二语句处理结果；
对第二语句处理结果进行归一化处理，得到归一化结果；
将计算结果与归一化结果进行交互，生成评论摘要。
图6提供了一种评论摘要生成方法流程示意图，参见图6的评论摘要生成框架所示。该框架主要基于生成模型，对于训练数据较少的情况，可采用textteaser、textrank等传统抽取式方法进行摘要，该类方法得到的结果若质量较高，经审核后，也可以作为生成式架构的训练语料。图6中的结构主要包含以下流程：
1)将用户评论进行汇总，并对其embedding后的结果进行处理，也即对上述评述意图进行处理，处理方式还是以特征抽取为主，可采用诸如bi-GRU、multi-head attention、TCN等结构；
2)由于系统支持用户打分功能，故而将其结果向量化后与语句处理后的结果相加，需支持空，即用户没有打分的情况。评级向量化也有多种方式，典型的方法是赋予每个等级可 训练的随机向量；
3)引入外部可训练的注意力矩阵，对前一步结果进行注意力计算，得到权重。然后，将所有权重乘以上一步的结果，得到encoder端的最终结果；
4)Decoder端采用transformer的decoder结构，该部分的语句处理采用带mask的multi-head attention，归一化采用layer normalization方法，然后将得到的结果与encoder端的最终结果采用multi-head attention方法进行交互。整个流程可重复Nx遍，得到的最终结果即为最后的评论结果。
需要注意的是，模型在线使用时需要跑多轮，每轮仅使用其对应位置的那个结果。
该评论摘要生成方法的作用在于，根据线上大量用户评论及用户给的评分，自动生成对于特定服务的线上外部评价结果。
在一些实施例中，外部评价还包括：
根据测试停止条件，停止获取评论及外部评级，测试停止条件包括以下至少之一：
目标智能助理的使用时间大于预设使用时间；
评论的数量大于预设评论数量；
外部停止指令。
下面通过一个具体的实施例，对智能助理的外部评价过程加以示例性说明。参见图7，图7为一种智能助理外部评价方法的流程示意图，如图7所示：
S701：制定预设评价方案。
在一些实施例中，根据评价目的需要，综合考虑目标智能助理的智能能力等级的影响因素，制定与其需求相符合的评价方案。可选择自行制定方案来实施评价，也可以委托专业机构或第三方制定评价方案，以期获得社会认可的结果。
在一些实施例中，评价前应识别、界定和描述被评估的智能助理产品及其特性，包括系统来源、用途和使用方式等。评价前应确定评价目的和范围，并按照评价等级划分标准所给出的待测评能力项和各待测评能力项所对应的基准评级来确定预设评价方案。
S702：封装目标智能助理的接口，接入统一的管理界面中。
S703：获取外部用户对待测评能力项的评论及外部评级。
在一些实施例中，将目标智能助理所需评测的待测评能力项在统一接口下提供给外部用户，支持用户评论及评级。
需要说明的是，上述外部用户就是使用该目标智能助理的普通用户，其评论和评级均是基于其自身的使用体验而给出了，对于外部用户不需要专业的培训以使得外部用户的评级或评论标准处于同一标准。但是必要的时候，可以对外部用户的一些特定信息进行获取，以便后续对其评论及评级更准确的分析。例如，获取外部用户的地域，假使外部用户地域均为新疆，则在评价目标智能助理的信息收集，语音输入能力项时，若该地域的外部用户存在大范围的低分和差评，则可以针对的对目标智能助理对于新疆方言的训练，以提升对该部分外部用户的服务能力。
S704：设定测试停止条件，根据测试停止条件停止获取用户对目标智能助理的待评价智能能力的评论及外部评级。
需要说明的是，测试停止条件可以是在进行外部评价之前就已经由用户或其他相关人员、装置、系统设定好的。测试停止条件也可以是在进行外部评价之后，根据实际情况再行设定的。测试停止条件还可以是在外部评价开始之前就已经设定好了，但在外部评价的过程中，再进行调整，进而形成新的测试停止条件。
在一些实施例中，测试停止条件可以是以下至少之一：
目标智能助理的使用时间大于预设使用时间；
评论的数量大于预设评论数量；
外部停止指令。
当然，测试停止条件还可以是本领域技术人员根据需要设定的其他条件。
S705：获取用户评论意图。
S706：获取评论摘要。
需要说明的是评论摘要包括了外部用户对目标智能助理的实际评级以及经过处理之后的实际评论。
在一些实施例中，根据评价目的，结合被评价目标智能助理的功能满足需求的能力，对目标智能助理的智能能力等级进行评价。
在一些实施例中，根据目标智能助理的智能能力的等级对应的评价能力项体系和待测评能力项，运用综合评分法或其他方法，形成合理评价结果，对实际评级进行计算，得到综合评级。
在一些实施例中，是对各个待测评能力项对应的实际评级进行加权平均，以得到综合评级。
在一些实施例中，根据评价结果生成评价报告包括：
获取评价报告模板，评价报告模板由获取并填写预设评价报告说明模板所需填写的内容后得到；
解析评价结果，提取目标数据及目标文字信息；
将目标数据及目标文字信息填写入评价报告模板中；
生成评价报告。
需要说明的是，在一些实施例中，预设评价说明模板可以是用户或评价方根据需要设定的模板，也可以由系统预先设定的多个评价说明模板中选取一个作为预设评价说明模板。
在一些实施例中，根据评价目的的需要，综合考虑目标智能助理的智能能力等级的影响因素，制定与其需求相符合的评价说明模板。可选择自行制定方案来实施评价，也可以委托专业机构或第三方制定评估说明模板，以期获得社会认可的结果。
在一些实施例中，预设评价说明模板可以理解为一个评价报告的文字说明，该预设评价说明模板可以包括但不限于以下至少之一：
评估程序实施过程和情况、特别事项说明、评估报告日、评估依据、智能助理产品的基本概况、智能助理智能等级划分和定义、评估报告的使用限制说明、评估目的、评估方法、评估假设和限定条件、评估对象和范围等。
在一些实施例中，在制定评价说明模板之前，还应识别、界定和描述被评评的目标智能助理产品及其特性，包括系统来源、用途和使用方式等，并根据上述信息制定评价说明模板。
需要说明的是，评价结果可以是由外部评价来源得到的评价结果也即评论摘要，和/或，内部评价来源也即实际评级、达标占比、综合占比构成。
在一些实施例中，将目标数据及目标文字信息填写入评价报告模板中，包括：
获取评价报告模板的槽位；
通过单句建模和序列编辑建模来将所述目标数据及目标文字信息填写入所述评价报告模板的槽位中。
下面通过一个具体的实施例对根据评价结果生成评价报告的方法的具体流程进行进一步说明：
如图8-1所示，为了能更方便高效的输出对目标智能助理的整体评价结果，本申请设计了评价报告生成方法。其一种实现流程示意图由8-1所示：
1)获取评价报告模板。
在一些实施例中，评价报告模板可以通过获取并填写预设评价报告说明模板所需填写的内容后得到。需要说明的是，预设评价说明模板可以理解为一个评价报告的文字说明，该预设评价说明模板所需填写的内容可以包括但不限于以下至少之一：
智能助理产品的基本概况、评估目的、评估对象和范围、智能助理智能能力等级划分和定义、评估假设和限定条件、评估依据、评估方法、评估程序实施过程和情况、特别事项说 明、评估报告的使用限制说明、评估报告日；
在一些实施例中，通过获取上述各项内容，并根据上述各项内容生成评价报告模板，该过程可以使用文本摘要和文本匹配技术，即根据大量结构化报告，选取其重要的文字并生成评价报告模板。
2)解析评价结果，提取目标数据及目标文字信息。
在一些实施例中，评价结果包括内部评价和外部评价的结果，通过对内部评价结果和外部评价结果进行信息提取，以提取重要的目标数据及目标文字信息。由于内部评价结果为实际评级、达标占比以及综合评级至少之一构成，外部评价结果包括评论摘要，该评论摘要包括实际评论与实际评级，上述内部评价结果和外部评价结果本身就是结构化的，这部分内容可以通过正则的方式提取。
3)将解析后得到的目标数据及目标文字信息填写入评价报告模板的槽位中，生成评价报告。
以上为基本流程，当训练数据够多时，可直接通过获取上述的预设评价说明模板所需填写的内容以及内部、外部评价结果生成最终的评价报告，整个流程是典型的seq2seq问题。此外，最终的评价报告需要根据目标智能助理功能的优劣给出情绪倾向性，为在保证语言多样性的前提下实现该功能，在一些实施例中，本申请可以采用最新的QuaSE框架，参见图8-2和图8-3，一种具体的流程如下：
该模型包含图8-2所示意的单句建模以及图8-3所示意的序列编辑两个部分的建模。图8-2示意的为单句建模，其中X和R是观测值，分别表示句子(例如用户对某个功能的评价)以及其对应的数值(例如用户评分)。Z和Y是隐变量，是对句子内容以及句子数值相关属性的建模表示。
对于隐变量Z和Y的建模是通过生成模型的方式实现。我们设计了两个Encoder(E1和E2)和一个Decoder(D)，X以Z和Y为条件进行生成。
模型的优化目标是使得生成的句子X'能够最大限度的重建输入句子X。同时，由于优化目标积分计算困难等原因，我们采用变分的方法探寻优化目标的下界。此外，还设计了一个回归函数F来学习隐变量Y和数值R的映射关系。
参见图8-3所示意的序列编辑过建模的流程示意图，参见图8-3，首先构建一个伪平行句对数据集。对于句子编辑的建模主要包含三个部分：
1)建立句子x到句子x'的内容变化与数值变化之间的关系。原句x到目标句x'的变化肯定是增加或者减少了某些词，从而使得在数值这个属性上产生变化，即y到y'的差别。对于这个变化映射我们设计了第一个目标函数Ldiff；
2)我们提到x和x'必须在主要内容方面继续保持一致，例如必须都是在描述“情感对话功能”。所以我们引入第二个目标函数Lsim来使得z和z'尽量的相似；
3)生成过程是给定z和y来生成x(p(x|z,y))，那么改写的过程可以是给定z和y'来生成x'(p(x'|z,y'))，也可以同时是给定z'和y来生成x(p(x|z',y))，这是个双向过程。所以对于这两个生成过程引入了第三个损失函数Ld-rec。
最后，单句建模和序列编辑建模模可以融合成一个统一的优化问题通过端到端的方法进行训练。
通过上述单句建模和序列编辑建模来将目标数据和目标文字信息填写入评价报告模板的槽位中。
在一些实施例中，根据预设评价方案对目标智能助理进行评价包括：获取预设评价方案；封装至少一个目标智能助理的接口，并接入统一的管理界面；通过管理界面根据预设评价方案对目标智能助理分别进行评价。
通过上述对各个目标智能助理的接口进行封装，接入统一的管理界面，可以使得统一内部专业人员或者外部用户通过同一个管理界面对多个目标智能助理进行使用并评价，提升工作效率。且通过统一的管理界面，可以直接导入预设评价方案中的待测评能力项的测评案例， 进一步提升工作效率，更大程度减少由于待测评能力项的测评案例误差所导致的评价误差。
在一些实施例中，将不同典型智能助理的输入输出接口进行封装，并进行统一化管理，确保不同类型智能助理的接口可以采用同样的方式进行在线使用和测试。
下面通过一个具体的内部评价流程示意图，展示目标智能助理加权评级流程，并说明了在待测评能力项所对应的评价等级划分标准缺失能力项的情况下如何执行正常评价流程。参见图9，图9为另一种智能助理评价方法的流程示意图:
S901：获取预设评价方案。
确定预设评价方案方案，根据评估目的需要，综合考虑所需通过用户进行评测的服务项。可选择自行制定预设评价方案来实施评估，也可以委托专业机构或第三方制定预设评价方案，以期获得社会认可的结果。
S902：获取待测评能力项。
评估前应识别、界定和描述被评估的智能助理产品及其特性，包括系统来源、用途和使用方式等。评估前应确定评估目的和范围，并结合评价等级划分标准所给出的评价能力项体系和能力项来确定待测评能力项。
S903：确定当前评价等级划分标准。
若评价等级划分标准中不包含或包含过时的能力项，可对其新增或者修改，所有修改过的结果可以根据需要经过行业专家投票，一经通过，将被采纳作为新的评价等级划分标准。当然对于评价等级划分标准也可以由本领域技术人员进行直接调整。
S904：对待测评能力项进行权重设定。
根据实际情况，结合内部专家打分，对待测评能力项进行权重设定。当在同一行业内或同一目的下进行评价时，应采用统一的指标权重设定方案，以保证评价结果具有可比性。
S905：生成评价报告模板。
其中评价报告模板包括预设评价方案、待测评能力项、评价等级划分标准以及待测评能力项的权重，将以上内容自动生成结构化模板，并给各项待测评能力项的评价结果预留可填入的位置。
S906：封装所需评价的目标智能助理接口，并接入统一的管理界面中。
S907：导入评估所需数据集，根据评估方案中的评估内容进行评测。
S908：目标智能助理根据预设评价方案，生成目标智能助理各待测评能力项的实际评级、综合评级、达标占比。
依据评价等级划分标准对目标智能助理的各项待测评能力项进行评级，并根据每个待测评能力项各个等级对应的不同评级的实际评级，进行加权求和，得到目标智能助理的综合评级。获取实际评级大于或等于基准评级的待测评能力项数量，与，待测评能力项总数量的比值作为达标占比。
S909：将每个待测评能力项的实际评级、综合评级、达标占比填写入评价报告模板中。
在一些实施例中，可以结合评估目标及内容，根据内部专家评估的结果倾向性，采用图8-2和8-3的模型进行报告文字改写，得到最终的评价报告。
在一些实施例中，通过评价过大量智能助理后，可以通过获取到内部评价结果以及外部评价结果之后，自动对智能助理进行内部及外部评价结果处理，并生成评价报告。图10为另一种智能助理评价方法流程的示意图，参见图10：
S1001：获取评价报告模板。
可以通过获取评价报告说明模板，并根据模板的相应要求输入以下信息中的至少一个：目标智能助理产品的基本概况、评估目的、评估对象和范围、智能助理智能能力等级划分和定义、评估假设和限定条件、评估依据、评估方法、评估程序实施过程和情况、特别事项说明、评估报告的使用限制说明及评估报告日等，并对评价报告说明模板进行相应的调整，例如删除空白的槽位等得到评价报告模板。其中上述信息可以通过图8-2、图8-3所示的方案填写入评价报告说明模板中。
S1002：解析评价结果，提取目标数据及目标文字信息。
根据上述评价报告模板自动确定预设评价方案，导入测评案例进行内部评价，并开放相应接口进行外部评价，获取内部评价结果和外部评价结果。
其中，评价结果包括内部评价结果和外部评价结果。
内部评价结果根据预设评价方案进行评测后，得到实际评级、综合评级、达标占比等信息。
外部评价结果通过获取外部的评论以及外部评级，对其采用意图识别以获取评论摘要，最后对内部评价结果、外部评价结果进行处理，提取目标数据及目标文字信息。
S1003：将目标数据、目标文字信息填写入评价报告模板中。
将内部、外部评价结果，采用图8-2、图8-3中模型的多输入源版本，自动生成最终的评价报告。其中多输入源版本为给不同输入数据定制特征处理模块，并基于大量历史评测数据进行训练后得到的。
在一些实施例中，参见图11，图11为一种智能助理评价系统的系统架构图，参见图11，根据该系统执行的一种智能助理评价方法流程示意如下：
S1101：将各类评测数据进行数据解析、预处理后存入数据库中。
需要说明的是评价数据包括各类待测评能力项，及其等级划分标准，以及其他评价目标智能助理的主要内容，例如：目标智能助理产品的基本概况、评估目的、评估对象和范围、智能助理智能能力等级划分和定义、评估假设和限定条件、评估依据、评估方法、评估程序实施过程和情况、特别事项说明、评估报告的使用限制说明及评估报告日等。
S1102：根据评价任务需要确认预设评价方案，该预设评价方案包括待测评能力项和测评案例、评价等级划分标准等；
S1103：通过封装后的智能助理接口，根据预设评价方案需要选择特定的方式进行评测；
S1104：若需进行内部评价，选择数据库中预设评价方案中相应的测评内容进行评测，并通过内部评测模块对各项待测评能力项进行评级；
S1105：若需进行外部评价，通过外部接口进行开放性的在线测试使用，并基于外部用户的外部评级及评论，采用外部评测模块对各项服务进行评级；
其中，用户在线使用，通过提供的统一化的用户界面，用户可以完整体验各类智能助理所提供的服务。用户的使用记录将被完整记录，此外用户可以针对每次服务进行打分，分为差、较差、中等、较满意、满意五个等级，同时用户可以对本次服务进行评论。
S1106：汇总内部评价及外部评价结果，基于该次评测制订的评价报告模板进行自动槽位填充，此外，根据各项待测评能力项的评价结果及评价报告模板自动生成简短的评价报告。
本申请实施例通过根据预设评价方案对目标智能助理进行评价，获取评价结果，并根据评价结果生成评价报告。提供了一种基于预设评价方案中多项重要指标对智能助理进行评估，得到评价结果生成评价报告，提供了一种标准化的智能助理评价方法，该评价方法的出现可推动智能助理行业发展，促进服务质量提升，进而提升用户的体验度。
进一步的，为使该评价报告的可信度和应用的广泛性更加满足人们的要求，可以将评价的方式根据用户的需求进行内部评价或外部评价至少之一。内部评价具有内部专业人员从专业的角度进行的评价结果，外部评价能够获取更多样本的外部用户的实际体验，为更加全方位了解智能助理以及对智能助理的针对性改进提供了参考角度。
进一步的，本申请实施例中提供内部评价中，测评的是目标智能助理的智能能力，预设评价方案包括评价等级划分标准，内部专业人员根据评价等级划分标准对该目标智能助理进行评级，得到各待测评能力项的内部实际评级。
进一步的，本申请实施例中提供外部评价中，其评价的主体是外部用户，其外部评级及评论均是外部用户根据其自身的体验进行的带有自身色彩的评价。
进一步的，本申请实施例中，通过获取外部用户的评论及外部评级，进行对其进行评论意图识别，形成评论摘要。其中对于外部评价，由于在一些情况下，存在外部用户群体庞大， 且现有获取到的评论和外部评级已经足以完成外部评价了，此时可以根据测试停止条件停止获取评论和外部评级。
进一步的，根据预设评价方案对目标智能助理进行评价还可以包括：通过对至少一个目标智能助理的接口进行封装，接入到统一的管理界面，以实现通过该管理界面对目标智能助理进行评价。极大地节约评价所需要的成本，不论是内部专业人员还是外部用户，均可以通过该管理界面实现对多个目标智能助理的使用，此时，外部用户和内部专业人员不再需要分别一一对多个目标智能助理进行使用，评价，导出其评价结果，再生成评价报告。更加简便。
实施例二
请参见图12，本申请实施例提供了一种智能助理推荐方法，包括：
S1201：封装至少两个目标智能助理的接口，并接入统一的管理界面；
需要说明的是，上述封装目标智能助理的接口，以及接入统一的管理界面的方式本领域各种常用技术手段来实现。
S1202：获取外部用户的当前需求；
S1203：根据当前需求确定相对应的能力项及各能力项的优先级；
S1204：获取目标智能助理的评价报告；
需要说明的是该评价报告的获取可以采用上述实施例的方法来进行获取。其中，当该目标智能助理存在评价报告记录时，可以直接使用该评价报告。当然也可以根据外部用户的当前需求所对应的能力项来进行实时评价，该评价可以是内部评价，
和/或，
获取除该外部用户之外的其他外部用户的外部评价。
S1205：根据能力项、能力项的优先级、评价报告确定各目标智能助理中的优选智能助理；
在一些实施例中，关于优选智能助理的确定方式可以是根据评价报告中各目标智能助理的相应能力项的评级高低来进行排序。例如：当前需求所对应的能力项按优先级排序为甲、乙、丙；当前A智能助理的能力项甲的评级为5，能力项乙评级为4，能力项丙评级为6；当前B智能助理的能力项甲的评级为3，能力项乙评级为7，能力项丙评级为10；此时，针对A智能助理和B智能助理中优选智能助理的确定可以根据预设规则来确定，该预设规则假使为仅参考优先级最高的一项能力项来选取，则A智能助理为优选智能助理。假使预设规则为各个能力项的加权平均最高者确定为优选智能助理，则可以分别对A智能助理和B智能助理的各个能力项进行加权平均来获取其各自的加权平均评级，进而选择较高的作为优选智能助理。需要说明的是，预设规则还可以是本领域技术人员根据需要制定的其他规则。
S1206：管理界面为外部用户提供优选智能助理的接口以供外部用户使用。
需要说明的是，该方式相当于外部用户当前在管理界面可以有多个目标智能助理进行选择使用，通过本申请实施例的方法，结合外部用户的需求，为外部用户提供一个优选智能助理，使得其使用体验得以提升。排除了用户对智能助理一一尝试的烦恼。该方法基于评测结果，根据外部用户所需功能自动化选择智能助理，从而提高服务质量。
在一些实施例中，用户同样可针对各项能力项进行评级和评论，系统将对其进行记录，从而能得到各项功能的不足之处，方便后续改进。
实施例三：
基于上述实施例一所提供的一种智能助理评价方法，本实施例还提供了一种智能助理评价系统1300，参见图13所示，其包括：
评价模块1301，用于根据预设评价方案对目标智能助理进行评价；
第一获取模块1302，用于获取评价结果；
第一生成模块1303，用于根据评价结果生成评价报告。
在一些实施例中，评价模块1301包括以下至少之一：
外部评价模块13011、内部评价模块13012。
在一些实施例中，内部评价模块13011包括：
第二获取模块130111，用于获取预设评价方案，预设评价方案包括各待测评能力项所对应的评价等级划分标准、待测评能力项的测评案例，以及，各待测评能力项所对应的基准评级；
第三获取模块130112，用于获取内部评价结果，内部评价结果包括内部实际评级和达标占比；
内部实际评级为内部专业人员根据预设评价方案对目标智能助理进行评测，得到的目标智能助理各待测评能力项的内部实际评级；
达标占比包括实际评级大于或等于基准评级的待测评能力项数量，与，待测评能力项总数量的比值。
在一些实施例中，外部评价模块13012包括：
第四获取模块130121，用于获取外部评价结果，外部评价结果包括外部用户对待测评能力项的评论及外部评级；
识别模块130122，用于识别评论的评论意图；
第二生成模块130123，用于根据外部评级和评论意图生成评论摘要。
实施例四
基于上述实施例二所提供的一种智能助理评推荐方法，本实施例还提供了一种智能助理推荐系统1400，参见图14所示，其包括：
封装模块1401，用于封装至少两个目标智能助理的接口，并接入统一的管理界面；
第五获取模块1402，用于获取外部用户的当前需求；
能力项确定模块1403，用于根据当前需求确定相对应的能力项及各能力项的优先级；
第六获取模块1404，用于获取目标智能助理的评价报告；
优选确定模块1405，用于根据能力项、能力项的优先级、评价报告确定各目标智能助理中的优选智能助理；
提供模块1406，用于管理界面为外部用户提供优选智能助理的接口以供外部用户使用。
实施例五：
本实施例还提供了一种智能助理评价终端，参见图15所示，其包括第一处理器1501、第一存储器1503及第一通信总线1502，其中：
第一通信总线1502用于实现第一处理器1501和第一存储器1503之间的连接通信；
第一处理器1501用于执行第一存储器1503中存储的一个或者多个计算机程序，以实现上述实施例一种智能助理评价中的至少一个步骤。
实施例六：
本实施例还提供了一种智能助理推荐终端，参见图16所示，其包括第二处理器1601、第二存储器1603及第二通信总线1602，其中：
第二通信总线1602用于实现第二处理器1601和第二存储器1603之间的连接通信；
第二处理器1601用于执行第二存储器1603中存储的一个或者多个计算机程序，以实现上述实施例二中的智能助理推荐方法中的至少一个步骤。
本实施例还提供了一种计算机可读存储介质，该计算机可读存储介质包括在用于存储信息(诸如计算机可读指令、数据结构、计算机程序模块或其他数据)的任何方法或技术中实施的易失性或非易失性、可移除或不可移除的介质。计算机可读存储介质包括但不限于RAM(Random Access Memory，随机存取存储器),ROM(Read-Only Memory，只读存储器), EEPROM(Electrically Erasable Programmable read only memory，带电可擦可编程只读存储器)、闪存或其他存储器技术、CD-ROM(Compact Disc Read-Only Memory，光盘只读存储器)，数字多功能盘(DVD)或其他光盘存储、磁盒、磁带、磁盘存储或其他磁存储装置、或者可以用于存储期望的信息并且可以被计算机访问的任何其他的介质。
本实施例中的计算机可读存储介质可用于存储一个或者多个第一计算机程序，其存储的一个或者多个第一计算机程序可被处理器执行，以实现上述实施例一中的智能助理评价方法的至少一个步骤。
本实施例中的计算机可读存储介质可用于存储一个或者多个第二计算机程序，其存储的一个或者多个第二计算机程序可被处理器执行，以实现上述实施例二中的智能助理推荐的至少一个步骤。
本实施例还提供了一种计算机程序(或称计算机软件)，该计算机程序可以分布在计算机可读介质上，由可计算装置来执行，以实现上述实施例一中的保持资源一致的至少一个步骤；并且在某些情况下，可以采用不同于上述实施例所描述的顺序执行所示出或描述的至少一个步骤。
本实施例还提供了一种计算机程序(或称计算机软件)，该计算机程序可以分布在计算机可读介质上，由可计算装置来执行，以实现上述实施例二中的保持资源一致的至少一个步骤；并且在某些情况下，可以采用不同于上述实施例所描述的顺序执行所示出或描述的至少一个步骤。
应当理解的是，在某些情况下，可以采用不同于上述实施例所描述的顺序执行所示出或描述的至少一个步骤。
本实施例还提供了一种计算机程序产品，包括计算机可读装置，该计算机可读装置上存储有如上所示的计算机程序。本实施例中该计算机可读装置可包括如上所示的计算机可读存储介质。
可见，本领域的技术人员应该明白，上文中所公开方法中的全部或某些步骤、系统、装置中的功能模块/单元可以被实施为软件(可以用计算装置可执行的计算机程序代码来实现)、固件、硬件及其适当的组合。在硬件实施方式中，在以上描述中提及的功能模块/单元之间的划分不一定对应于物理组件的划分；例如，一个物理组件可以具有多个功能，或者一个功能或步骤可以由若干物理组件合作执行。某些物理组件或所有物理组件可以被实施为由处理器，如中央处理器、数字信号处理器或微处理器执行的软件，或者被实施为硬件，或者被实施为集成电路，如专用集成电路。
此外，本领域普通技术人员公知的是，通信介质通常包含计算机可读指令、数据结构、计算机程序模块或者诸如载波或其他传输机制之类的调制数据信号中的其他数据，并且可包括任何信息递送介质。所以，本申请不限制于任何特定的硬件和软件结合。
以上内容是结合具体的实施方式对本申请实施例所作的进一步详细说明，不能认定本申请的具体实施只局限于这些说明。对于本申请所属技术领域的普通技术人员来说，在不脱离本申请构思的前提下，还可以做出若干简单推演或替换，都应当视为属于本申请的保护范围。

Claims (21)

  1. 一种智能助理评价方法，包括：
    根据预设评价方案对目标智能助理进行评价，获取评价结果,所述评价包括以下至少之一：内部评价、外部评价；
    根据所述评价结果生成评价报告。
  2. 如权利要求1所述的智能助理评价方法，其中，
    若评价包括内部评价，则所述预设评价方案包括内部专业人员评价所述目标智能助理的智能能力的待测评能力项；
    所述评价结果包括内部评价结果。
  3. 如权利要求2所述的智能助理评价方法，其中，所述根据预设评价方案对目标智能助理进行评价，获取评价结果包括：
    获取所述预设评价方案，所述预设评价方案包括各所述待测评能力项所对应的评价等级划分标准、所述待测评能力项的测评案例，以及，各所述待测评能力项所对应的基准评级；
    获取所述内部评价结果，所述内部评价结果包括内部实际评级和达标占比；
    所述内部实际评级为所述内部专业人员根据所述预设评价方案对所述目标智能助理进行评测，得到的所述目标智能助理各待测评能力项的内部实际评级；
    所述达标占比包括所述内部实际评级大于或等于所述基准评级的待测评能力项数量，与，待测评能力项总数量的比值。
  4. 如权利要求3所述的智能助理评价方法，其中，所述内部评价结果还包括综合评级；
    所述综合评级包括对所述内部实际评级进行计算，得到所述目标智能助理的综合评级。
  5. 如权利要求4所述的智能助理评价方法，其中，若所述评价包括外部评价，则所述预设评价方案包括外部用户评价所述目标智能助理的待测评能力项；
    所述评价结果包括外部评价结果。
  6. 如权利要求5所述的智能助理评价方法，其中，所述根据预设评价方案对目标智能助理进行评价，获取评价结果包括：
    获取外部用户对待测评能力项的评论及外部评级；
    识别所述评论的评论意图；
    根据所述外部评级和所述评论意图生成评论摘要。
  7. 如权利要求6所述的智能助理评价方法，其中，还包括：
    根据测试停止条件，停止获取所述评论及所述外部评级，所述测试停止条件包括以下至少之一：
    所述目标智能助理的使用时间大于预设使用时间；
    所述评论的数量大于预设评论数量；
    外部停止指令。
  8. 如权利要求6所述的智能助理评价方法，其中，所述识别所述评论的评论意图包括：
    获取所述评论的评论类别；
    获取所述评论类别对应的格式；
    根据所述格式对所述评论进行改写；
    获取所述评论类别对应的最佳评论识别模型，所述识别模型包括所述特征、所述特征的距离、评价等级划分规则；
    将所述评论录入所述最佳评论识别模型，得到所述评论的评论意图。
  9. 如权利要求6所述的智能助理评价方法，其中，所述根据所述外部评级和所述评论 意图生成评论摘要包括：
    对所述评论意图进行第一语句处理，得到第一语句处理结果；
    获取所述评论意图所对应的权重；
    根据所述权重对所述第一语句处理结果和所述外部评级进行计算得到计算结果；
    对所述评论意图进行第二语句处理，得到第二语句处理结果；
    对所述第二语句处理结果进行归一化处理，得到归一化结果；
    将所述计算结果与所述归一化结果进行交互，生成评论摘要。
  10. 如权利要求1-9任一项所述的智能助理评价方法，其中，所述根据所述评价结果生成评价报告包括：
    获取评价报告模板，所述评价报告模板由获取并填写预设评价报告说明模板所需填写的内容后得到；
    解析所述评价结果，提取目标数据及目标文字信息；
    将所述目标数据及目标文字信息填写入所述评价报告模板中；
    生成评价报告。
  11. 如权利要求10所述的智能助理评价方法，其中，所述将所述目标数据及目标文字信息填写入所述评价报告模板中包括：
    获取所述评价报告模板的槽位；
    通过单句建模和序列编辑建模来将所述目标数据及目标文字信息填写入所述评价报告模板的槽位中。
  12. 如权利要求1-9任一项所述的智能助理评价方法，其中，所述根据预设评价方案对目标智能助理进行评价包括：
    封装至少一个所述目标智能助理的接口，并接入统一的管理界面；
    通过所述管理界面对所述目标智能助理分别进行评价。
  13. 一种智能助理推荐方法，包括：
    封装至少两个目标智能助理的接口，并接入统一的管理界面；
    获取外部用户的当前需求；
    根据所述当前需求确定相对应的能力项及各能力项的优先级；
    获取所述目标智能助理的评价报告；
    根据所述能力项、能力项的优先级、评价报告确定所述各目标智能助理中的优选智能助理；
    所述管理界面为所述外部用户提供所述优选智能助理的接口以供所述外部用户使用。
  14. 一种智能助理评价系统，包括：
    评价模块，用于根据预设评价方案对目标智能助理进行评价,所述评价模块包括以下至少之一：外部评价模块、内部评价模块；
    第一获取模块，用于获取评价结果；
    第一生成模块，用于根据所述评价结果生成评价报告。
  15. 如权利要求14所述的智能助理评价系统，其中，所述内部评价模块包括：
    第二获取模块，用于获取所述预设评价方案，所述预设评价方案包括各所述待测评能力项所对应的评价等级划分标准、所述待测评能力项的测评案例，以及，各所述待测评能力项所对应的基准评级；
    第三获取模块，用于获取所述内部评价结果，所述内部评价结果包括内部实际评级和达标占比；
    所述内部实际评级为所述内部专业人员根据所述预设评价方案对所述目标智能助理进行评测，得到的所述目标智能助理各待测评能力项的内部实际评级；
    所述达标占比包括所述实际评级大于或等于所述基准评级的待测评能力项数量，与，待测评能力项总数量的比值。
  16. 如权利要求14所述的智能助理评价系统，其中，所述外部评价模块包括：
    第四获取模块，用于获取所述外部评价结果，所述外部评价结果包括所述外部用户对所述待测评能力项的评论及外部评级；
    识别模块，用于识别所述评论的评论意图；
    第二生成模块，用于根据所述外部评级和所述评论意图生成评论摘要。
  17. 一种智能助理推荐系统，包括：
    封装模块，用于封装至少两个目标智能助理的接口，并接入统一的管理界面；
    第五获取模块，用于获取外部用户的当前需求；
    能力项确定模块，用于根据所述当前需求确定相对应的能力项及各能力项的优先级；
    第六获取模块，用于获取所述目标智能助理的评价报告；
    优选确定模块，用于根据所述能力项、能力项的优先级、评价报告确定所述各目标智能助理中的优选智能助理；
    提供模块，用于所述管理界面为所述外部用户提供所述优选智能助理的接口以供所述外部用户使用。
  18. 一种智能助理评价终端，包括：第一处理器、第一存储器及第一通信总线；
    所述第一通信总线用于实现第一处理器和第一存储器之间的连接通信；
    所述第一处理器用于执行第一存储器中存储的一个或者多个第一计算机程序，以实现如权利要求1至12中任一项所述的智能助理评价方法的步骤。
  19. 种智能助理推荐终端，包括：第二处理器、第二存储器及第二通信总线；
    所述第二通信总线用于实现第二处理器和第二存储器之间的连接通信；
    所述第二处理器用于执行第二存储器中存储的一个或者多个第二计算机程序，以实现如权利要求13所述的智能助理推荐方法的步骤。
  20. 一种可读存储介质，其中，所述计算机可读存储介质存储有一个或者多个第一计算机程序，所述一个或者多个第一计算机程序可被一个或者多个第一处理器执行，以实现如权利要求1至12中任一项所述的智能助理评价方法的步骤。
  21. 一种可读存储介质，其中，所述计算机可读存储介质存储有一个或者多个第二计算机程序，所述一个或者多个第二计算机程序可被一个或者多个第二处理器执行，以实现如权利要求13所述的智能助理推荐方法的步骤。
PCT/CN2020/128455 2019-11-14 2020-11-12 智能助理评价、推荐方法、系统、终端及可读存储介质 WO2021093821A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201911115568.7A CN112799747A (zh) 2019-11-14 2019-11-14 智能助理评价、推荐方法、系统、终端及可读存储介质
CN201911115568.7 2019-11-14

Publications (1)

Publication Number Publication Date
WO2021093821A1 true WO2021093821A1 (zh) 2021-05-20

Family

ID=75803918

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2020/128455 WO2021093821A1 (zh) 2019-11-14 2020-11-12 智能助理评价、推荐方法、系统、终端及可读存储介质

Country Status (2)

Country Link
CN (1) CN112799747A (zh)
WO (1) WO2021093821A1 (zh)

Cited By (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113642834A (zh) * 2021-06-29 2021-11-12 合肥工业大学 基于任务属性优先级映射的任务重要性评价方法和系统
CN114139883A (zh) * 2021-11-10 2022-03-04 云南电网有限责任公司信息中心 一种电力企业物资域评价指标的计算方法
CN114386794A (zh) * 2021-12-28 2022-04-22 中国电子技术标准化研究院华东分院 一种工业互联网服务商资源池分类分级的评估评价方法
CN116562947A (zh) * 2023-04-18 2023-08-08 成都银翼穿戴科技有限公司 一种智能发热服用户体验测试方法
CN116611896A (zh) * 2023-07-19 2023-08-18 山东省人工智能研究院 基于属性驱动解耦表征学习的多模态推荐方法
CN116738371A (zh) * 2023-08-14 2023-09-12 广东信聚丰科技股份有限公司 基于人工智能的用户学习画像构建方法及系统
CN116993296A (zh) * 2023-08-15 2023-11-03 深圳市中联信信息技术有限公司 应用于工程设计交互平台的智能监理管理系统及方法
CN117113943A (zh) * 2023-10-25 2023-11-24 湖北君邦环境技术有限责任公司 一种环评报告辅助编写方法及其系统
WO2023236873A1 (zh) * 2022-06-10 2023-12-14 进升教育有限公司 体育项目测评方法、相关设备及计算机可读存储介质
CN117493210A (zh) * 2023-11-27 2024-02-02 中国传媒大学 微服务工具评价方法及系统
CN117974722A (zh) * 2024-04-02 2024-05-03 江西师范大学 基于注意力机制和改进的Transformer的单目标跟踪系统及方法

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115017884B (zh) * 2022-01-20 2024-04-26 昆明理工大学 基于图文多模态门控增强的文本平行句对抽取方法
CN116662503B (zh) * 2023-05-22 2023-12-29 深圳市新美网络科技有限公司 私域用户场景话术推荐方法及其系统

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20140101611A1 (en) * 2012-10-08 2014-04-10 Vringo Lab, Inc. Mobile Device And Method For Using The Mobile Device
CN106933864A (zh) * 2015-12-30 2017-07-07 中国科学院深圳先进技术研究院 一种搜索引擎系统及其搜索方法
CN109712624A (zh) * 2019-01-12 2019-05-03 北京设集约科技有限公司 一种多语音助手协调方法、装置和系统
CN110210743A (zh) * 2019-05-23 2019-09-06 华侨大学 一种ai服务智商测试方法
CN110807566A (zh) * 2019-09-09 2020-02-18 腾讯科技（深圳）有限公司 人工智能模型评测方法、装置、设备及存储介质

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20140101611A1 (en) * 2012-10-08 2014-04-10 Vringo Lab, Inc. Mobile Device And Method For Using The Mobile Device
CN106933864A (zh) * 2015-12-30 2017-07-07 中国科学院深圳先进技术研究院 一种搜索引擎系统及其搜索方法
CN109712624A (zh) * 2019-01-12 2019-05-03 北京设集约科技有限公司 一种多语音助手协调方法、装置和系统
CN110210743A (zh) * 2019-05-23 2019-09-06 华侨大学 一种ai服务智商测试方法
CN110807566A (zh) * 2019-09-09 2020-02-18 腾讯科技（深圳）有限公司 人工智能模型评测方法、装置、设备及存储介质

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
LIYAN GU , WANG JUNCHENG , GAO TENG , HU YIYUAN: "Comparative Analysis on Intelligent Customer Service System", CHINA SCIENCE & TECHNOLOGY RESOURCES REVIEW, vol. 51, no. 2, 28 March 2019 (2019-03-28), pages 75 - 80, XP055811625, ISSN: 1674-1544, DOI: 10.3772/j.issn.1674-1544.2019.02.012 *

Cited By (18)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113642834B (zh) * 2021-06-29 2023-08-29 合肥工业大学 基于任务属性优先级映射的任务重要性评价方法和系统
CN113642834A (zh) * 2021-06-29 2021-11-12 合肥工业大学 基于任务属性优先级映射的任务重要性评价方法和系统
CN114139883A (zh) * 2021-11-10 2022-03-04 云南电网有限责任公司信息中心 一种电力企业物资域评价指标的计算方法
CN114139883B (zh) * 2021-11-10 2024-03-29 云南电网有限责任公司信息中心 一种电力企业物资域评价指标的计算方法
CN114386794A (zh) * 2021-12-28 2022-04-22 中国电子技术标准化研究院华东分院 一种工业互联网服务商资源池分类分级的评估评价方法
WO2023236873A1 (zh) * 2022-06-10 2023-12-14 进升教育有限公司 体育项目测评方法、相关设备及计算机可读存储介质
CN116562947A (zh) * 2023-04-18 2023-08-08 成都银翼穿戴科技有限公司 一种智能发热服用户体验测试方法
CN116611896A (zh) * 2023-07-19 2023-08-18 山东省人工智能研究院 基于属性驱动解耦表征学习的多模态推荐方法
CN116611896B (zh) * 2023-07-19 2023-10-24 山东省人工智能研究院 基于属性驱动解耦表征学习的多模态推荐方法
CN116738371B (zh) * 2023-08-14 2023-10-24 广东信聚丰科技股份有限公司 基于人工智能的用户学习画像构建方法及系统
CN116738371A (zh) * 2023-08-14 2023-09-12 广东信聚丰科技股份有限公司 基于人工智能的用户学习画像构建方法及系统
CN116993296A (zh) * 2023-08-15 2023-11-03 深圳市中联信信息技术有限公司 应用于工程设计交互平台的智能监理管理系统及方法
CN116993296B (zh) * 2023-08-15 2024-04-16 深圳市中联信信息技术有限公司 应用于工程设计交互平台的智能监理管理系统及方法
CN117113943A (zh) * 2023-10-25 2023-11-24 湖北君邦环境技术有限责任公司 一种环评报告辅助编写方法及其系统
CN117113943B (zh) * 2023-10-25 2024-02-02 湖北君邦环境技术有限责任公司 一种环评报告辅助编写方法及其系统
CN117493210A (zh) * 2023-11-27 2024-02-02 中国传媒大学 微服务工具评价方法及系统
CN117974722A (zh) * 2024-04-02 2024-05-03 江西师范大学 基于注意力机制和改进的Transformer的单目标跟踪系统及方法
CN117974722B (zh) * 2024-04-02 2024-06-11 江西师范大学 基于注意力机制和改进的Transformer的单目标跟踪系统及方法

Also Published As

Publication number Publication date
CN112799747A (zh) 2021-05-14

Similar Documents

Publication Publication Date Title
WO2021093821A1 (zh) 智能助理评价、推荐方法、系统、终端及可读存储介质
US11748555B2 (en) Systems and methods for machine content generation
Knote et al. Value co-creation in smart services: a functional affordances perspective on smart personal assistants
CN111444709B (zh) 文本分类方法、装置、存储介质及设备
Chen et al. Consumers’ perception on artificial intelligence applications in marketing communication
US20230252224A1 (en) Systems and methods for machine content generation
US9875494B2 (en) Using intents to analyze and personalize a user's dialog experience with a virtual personal assistant
US11823074B2 (en) Intelligent communication manager and summarizer
Bilquise et al. Emotionally intelligent chatbots: a systematic literature review
US20140337266A1 (en) Rapid development of virtual personal assistant applications
CN112308650B (zh) 推荐理由生成方法、装置、设备及存储介质
CN109145168A (zh) 一种专家服务机器人云平台
Chao et al. Emerging Technologies of Natural Language‐Enabled Chatbots: A Review and Trend Forecast Using Intelligent Ontology Extraction and Patent Analytics
Gao [Retracted] Research and Implementation of Intelligent Evaluation System of Teaching Quality in Universities Based on Artificial Intelligence Neural Network Model
JP7488871B2 (ja) 対話推薦方法、装置、電子機器、記憶媒体ならびにコンピュータプログラム
Shen et al. A voice of the customer real-time strategy: An integrated quality function deployment approach
RU2670781C2 (ru) Система и способ для хранения и обработки данных
JP2023053309A (ja) 情報処理システム、情報処理装置、プログラム、及び情報処理方法
Kashyap et al. Artificial Intelligence and Its Applications in E-Commerce–a Review Analysis and Research Agenda
CN115392217A (zh) 用于保持修辞流的技术
Xue et al. Bias and fairness in chatbots: An overview
Yuan [Retracted] A Classroom Emotion Recognition Model Based on a Convolutional Neural Network Speech Emotion Algorithm
CN117573946A (zh) 对话样本生成方法、聊天对话大模型训练方法及相关装置
Chen Research on Furniture Design Integrating Ming‐Style Furniture Modeling Elements and Image Sensor Data: Taking Suitable Old Furniture as an Example
US20240005911A1 (en) Systems and methods to improve trust in conversations with deep learning models

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20887217

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 20887217

Country of ref document: EP

Kind code of ref document: A1

32PN Ep: public notification in the ep bulletin as address of the adressee cannot be established

Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205A DATED 14.02.2023)

122 Ep: pct application non-entry in european phase

Ref document number: 20887217

Country of ref document: EP

Kind code of ref document: A1