WO2022253251A1

WO2022253251A1 - Method and apparatus for evaluating interaction performance of interaction system

Info

Publication number: WO2022253251A1
Application number: PCT/CN2022/096513
Authority: WO
Inventors: 刘建国; 赵培
Original assignee: 青岛海尔科技有限公司; 海尔智家股份有限公司
Priority date: 2021-06-02
Filing date: 2022-06-01
Publication date: 2022-12-08
Also published as: CN113282475B; CN113282475A

Abstract

The present application relates to the technical field of human-computer interaction, and discloses a method for evaluating interaction performance of an interaction system. The method for evaluating interaction performance of an interaction system comprises: obtaining the interaction state of each log in interaction logs of an interaction system, and labeling each log according to the interaction state; and determining a self-learning index of the interaction system according to the labeling result of each log to evaluate the interaction performance of the interaction system. The method for evaluating interaction performance of an interaction system can be used to evaluate a dynamic process of improving the performance of the interaction system by means of self-learning. Also disclosed in the present application are an apparatus for evaluating interaction performance of an interaction system.

Description

Method and device for evaluating interactive performance of interactive system

This disclosure is based on a Chinese patent application with application number 202110616138.4 and a filing date of June 2, 2021, and claims the priority of this Chinese patent application. The entire content of this Chinese patent application is hereby incorporated by reference into this disclosure.

technical field

The present disclosure relates to the technical field of human-computer interaction, for example, to a method and device for evaluating the interaction performance of an interaction system.

Background technique

Currently, users can interact with smart devices carrying interactive systems through voice, gestures, etc., to control the smart devices. The interaction success rate is an important index to measure the interactive system, which is defined as the number of successful interactive samples divided by the total number of samples. The higher the interactive success rate, the better the performance of the interactive system. The performance improvement of the interactive system is usually expressed by the interactive success rate.

In the application process of the interactive system, that is, in the interactive process of the interactive system, the evaluation index used to evaluate the performance of the interactive system is single, which is not conducive to the comprehensive evaluation of the performance of the interactive system, and is not conducive to the further improvement of the performance of the interactive system.

Contents of the invention

In order to provide a basic understanding of some aspects of the disclosed embodiments, a brief summary is presented below. The summary is not intended to be an extensive overview nor to identify key/important elements or to delineate the scope of these embodiments, but rather serves as a prelude to the detailed description that follows.

The embodiments of the present disclosure provide a method and device for evaluating the interactive performance of an interactive system, so as to solve the problem that the existing interactive success rate index is difficult for the interactive system to improve its own performance through self-learning during the interactive process of the interactive system. technical aspects of process evaluation.

In some embodiments, a method for evaluating interactive performance of an interactive system includes:

Obtain the interaction state of each log in the interaction log of the interaction system, and mark each log according to the interaction state;

According to the labeling result of each log, determine the self-learning index of described interactive system, to evaluate the interactive performance of described interactive system.

Optionally, according to the labeling result of each log, determining the self-learning index of the interactive system includes:

Obtaining the first failure log of the interaction failure within the set time period before the current evaluation period in the interaction log;

Determine the interaction success rate of the current evaluation period according to the interaction success rate of the interaction content corresponding to the first failure log in the current evaluation period;

The self-learning index is determined according to the interaction success rate of the current evaluation period.

Optionally, the current evaluation period includes multiple sub-data collection periods; determining the interaction success rate of the current evaluation period according to the interaction success rate of the interaction content corresponding to the first failure log in the current evaluation period, Including: determining the interaction success rate of each sub-data collection cycle according to the first failure log; obtaining the average value of the interaction success rates of multiple sub-data collection cycles; determining the average value as the interaction success rate of the current evaluation cycle Rate.

Optionally, the set duration is a sub-data collection period; according to the first failure log, determining the interaction success rate of each sub-data collection period includes: when the current sub-data collection period is the current evaluation period In the case of the first sub-data collection period, the interaction success rate of the interaction content corresponding to the first failure log in the current sub-data collection period is determined as the interaction success rate of the current sub-data collection period.

Optionally, determining the interaction success rate of each sub-data collection cycle according to the first failure log further includes: when the current sub-data collection cycle is not the first sub-data collection cycle of the current evaluation cycle, In the interaction log, obtain the second failure log of the interaction failure in the previous sub-data collection period of the current sub-data collection period; add the interaction content corresponding to the second failure log to the current sub-data collection period The interaction success rate within the period is determined as the interaction success rate of the current sub-data collection period.

Optionally, determining the interaction success rate of each sub-data collection period according to the first failure log includes: determining the interaction success rate of the interaction content corresponding to the first failure log in each sub-data collection period as Interaction success rate for each sub-data collection cycle.

Optionally, determining the self-learning index according to the interaction success rate of the current evaluation cycle includes: obtaining the interaction failure rate of the current evaluation cycle; determining the interaction success rate and the interaction failure rate according to the current evaluation cycle A self-learning index, so that the self-learning index is a monotonically increasing function along with the interaction success rate.

Optionally, determining the self-learning index according to the interaction success rate and interaction failure rate of the current evaluation cycle includes:

S=k'/(k'+r)

Among them, S is the self-learning index, k' is the interaction success rate, and r is the interaction failure rate.

Optionally, obtaining the interaction failure rate of the current evaluation period includes: obtaining a first number of logs in the current evaluation period in the interaction logs, and a log of interaction failures in the current evaluation period the second quantity; according to the first quantity and the second quantity, determine the interaction failure rate of the current evaluation cycle.

Optionally, obtaining the interaction status of each log in the interaction log includes: if the log in the interaction log is determined to be a reply satisfactory to the user through a preset interaction strategy, then adding the log in the interaction log to The interaction status of the log is determined as interaction success; if the log in the interaction log is determined to be an unsatisfactory reply from the user through the preset interaction policy, the interaction status of the log in the interaction log is determined as interaction failure.

In some embodiments, the device for evaluating the interaction performance of an interactive system includes a processor and a memory storing program instructions, and the processor is configured to execute the evaluation method provided by the foregoing embodiments when executing the program instructions. A method for interactive performance of interactive systems.

In some embodiments, the interactive system includes the apparatus for evaluating the interactive performance of the interactive system provided in the foregoing embodiments.

The method and device for evaluating the interactive performance of an interactive system provided by the embodiments of the present disclosure can achieve the following technical effects:

The self-learning index of the interactive system can evaluate the dynamic process of the interactive system to improve its performance through self-learning, which is beneficial to the self-learning process of the interactive system.

The foregoing general description and the following description are exemplary and explanatory only and are not intended to limit the present disclosure.

Description of drawings

One or more embodiments are exemplified by corresponding drawings, and these exemplifications and drawings do not constitute limitations to the embodiments, and elements with the same reference numerals in the drawings are regarded as similar elements, and where:

FIG. 1 is a schematic diagram of an implementation scenario of an interactive system provided by an embodiment of the present disclosure;

FIG. 2 is a schematic diagram of a method for evaluating the interactive performance of an interactive system provided by an embodiment of the present disclosure;

FIG. 3 is a schematic diagram of a process of determining a self-learning index provided by an embodiment of the present disclosure;

Fig. 4 is a schematic diagram of an apparatus for evaluating the interaction performance of an interaction system provided by an embodiment of the present disclosure.

Detailed ways

In order to understand the characteristics and technical content of the embodiments of the present disclosure in more detail, the implementation of the embodiments of the present disclosure will be described in detail below in conjunction with the accompanying drawings. The attached drawings are only for reference and description, and are not intended to limit the embodiments of the present disclosure. In the following technical description, for purposes of explanation, numerous details are set forth in order to provide a thorough understanding of the disclosed embodiments. However, one or more embodiments may be practiced without these details. In other instances, well-known structures and devices may be shown simplified in order to simplify the drawings.

The terms "first", "second" and the like in the description and claims of the embodiments of the present disclosure and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific sequence or sequence. It should be understood that the data so used may be interchanged under appropriate circumstances so as to facilitate the embodiments of the disclosed embodiments described herein. Furthermore, the terms "comprising" and "having", as well as any variations thereof, are intended to cover a non-exclusive inclusion.

Unless stated otherwise, the term "plurality" means two or more.

In the embodiments of the present disclosure, the character "/" indicates that the preceding and following objects are an "or" relationship. For example, A/B means: A or B.

The term "and/or" is an associative relationship describing objects, indicating that there can be three relationships. For example, A and/or B means: A or B, or, A and B, these three relationships.

Fig. 1 is a schematic diagram of an implementation scenario of an interactive system provided by an embodiment of the present disclosure. As shown in Figure 1, the interaction system can include an interaction model and an interaction log. The interaction model can receive request information, and after receiving the request information, the interaction model can reply to the request information; the interaction log can record the request information and its corresponding reply.

The request information here may be user request information, such as user voice instruction information, user gesture instruction information, and the like.

The response of the interaction model to the request information can be to issue control commands to control other devices, such as control commands to control smart air conditioners, control commands to control smart refrigerators, or control commands to control smart TVs. User reply.

The interactive system in the embodiments of the present disclosure needs to be implemented by relying on specific devices, and the interactive system can be implemented by terminal devices, household electrical appliances, and the like.

The terminal devices here may be smart phones, tablet computers, ultra-mobile personal computers (Ultra-mobile Personal Computer, UMPC), netbooks, personal digital assistants (Personal Digital Assistant, PDA) and other terminal devices, but are not limited thereto.

The home appliances here may be home appliances such as smart TVs, smart refrigerators, smart air conditioners, smart air conditioners, and smart refrigerators, but are not limited thereto.

Fig. 2 is a schematic diagram of a method for evaluating the interaction performance of an interaction system provided by an embodiment of the present disclosure, and the method can be executed by a terminal device or a home appliance with an interaction function. As shown in Figure 2, the methods used to evaluate the interactive performance of the interactive system include:

S201. Obtain the interaction state of each log in the interaction log of the interaction system, and mark each log according to the interaction state.

Wherein, the interaction status includes interaction success and interaction failure.

The interaction log records the request information (for example, the request information sent by the user in the form of voice or gesture) and the response of the interaction system to the request information.

In some application scenarios, obtaining the interaction status of each log in the interaction log includes: determining the interaction status of the log in the interaction log when the log in the interaction log is determined to be a satisfactory reply to the user through the preset interaction strategy is the interaction success; if the log in the interaction log is determined to be an unsatisfactory reply from the user through the preset interaction policy, the interaction status of the log in the interaction log is determined as interaction failure.

Read each log in the interaction log one by one, and judge each log one by one according to the preset interaction strategy, so as to determine the interaction status of each log as interaction success or interaction failure.

The preset interaction strategy here is the strategy used to determine whether the interaction is successful. For example, if two or more logs appear in the interaction log with the same request information but different replies, determine the interaction of these two or more logs The status is interaction failure; or, after the interactive system sends a reply corresponding to the request information, it receives the "correct" or "wrong" feedback triggered by the user. If an interactive content gets a "correct" feedback, it is determined to record the interaction The interaction status of the content log is interaction success, and if an interaction content receives an "error" feedback, then determine that the interaction status of the interaction content log is interaction failure.

Optionally, marking each log in the interaction log according to the interaction status includes: when the interaction status of a log is interaction success, marking the log as interaction success; when the interaction status of a log is interaction In case of failure, mark the log as interaction failure. After marking in this way, it is convenient to determine the failure log and the interaction success rate of the current evaluation cycle in the subsequent steps.

In some practical applications, the interaction status of each log can be recorded in the log.

S202. Determine the self-learning index of the interactive system according to the labeling result of each log, so as to evaluate the interactive performance of the interactive system.

Fig. 3 is a schematic diagram of a process of determining a self-learning index provided by an embodiment of the present disclosure. As shown in Figure 3, according to the labeling results of each log, determine the self-learning index of the interactive system, including:

S301. Obtain a first failure log of an interaction failure within a set time before the current evaluation period in the interaction log.

Each log in the interaction log also records the interaction time, and the current evaluation period has a start time and an end time. In the interaction log, firstly select the logs whose interaction time is before the start time of the current evaluation period within the set duration, and then read the interaction status of these logs sequentially. If the interaction status of the log is interaction failure, the log Determined as the first failure log. The number of logs in the first failure log may be one or more.

After the first failure log is obtained, the interaction success rate of the first failure log in the current evaluation period is obtained. For example, when the current evaluation period includes one or more sub-data collection periods, the interaction success rate of the interaction content corresponding to the first failure log in a sub-data collection period in the current evaluation period can be obtained, and the first failure log can also be obtained. The interaction success rate of the interaction content corresponding to the failure log in all sub-data collection cycles in the current evaluation cycle.

S302. Determine the interaction success rate of the current evaluation period according to the interaction success rate of the interaction content corresponding to the first failure log in the current evaluation period.

The interactive content here may be a control command for controlling other devices, for example, it may be a control command for controlling a smart air conditioner, a control command for controlling a smart refrigerator, or a control command for controlling a smart TV, or it may be other commands such as querying the weather.

The interaction content corresponding to each log in the first failure log can be performed one or more times in the current evaluation period, or not performed once.

In some application scenarios, the set duration can be longer than the current evaluation cycle, and the interaction success rate of the first failure log in the current evaluation cycle is used as the interaction success rate of the current evaluation cycle; or, the set duration can be equal to the current evaluation cycle, and the The interaction success rate of the first failure log in the current evaluation cycle is used as the interaction success rate of the current evaluation cycle; or, the set duration can be shorter than the current evaluation cycle, and the interaction success rate of the first failure log in the current evaluation cycle is used as the current evaluation Periodic interaction success rate.

Optionally, if the current evaluation period includes a sub-data collection period, the interaction success rate of the interactive content corresponding to the first failure log in the sub-data collection period can be obtained, and the interaction content corresponding to the first failure log in the sub-data collection period can be obtained. The interaction success rate in the sub-data collection cycle is used as the interaction success rate in the current evaluation cycle.

Optionally, the current evaluation period includes multiple sub-data collection periods, and correspondingly, the set duration may be one or more sub-data collection periods. The current evaluation cycle may include 2, 3, 4 or more sub-data collection cycles. For example, a sub-data collection cycle can be one week, and the current evaluation cycle can be 14 days, and the current evaluation cycle includes 2 sub-data collection cycles; or, the current evaluation cycle can be 21 days, and the current evaluation cycle includes 3 sub-data collection period; alternatively, the current evaluation period may be 28 days, in which case the current evaluation period includes 4 sub-data collection periods. On this basis, according to the interaction success rate of the interaction content corresponding to the first failure log in the current evaluation period, determine the interaction success rate of the current evaluation period, including: according to the first failure log, determine the interaction success of each sub-data collection period rate; obtain the average value of the interaction success rate of multiple sub-data collection cycles; determine the average value as the interaction success rate of the current evaluation cycle.

By adopting the above steps, the interaction success rate of the current evaluation cycle can be determined.

In some application scenarios, the duration is set as one sub-data collection cycle, and the current evaluation cycle includes multiple sub-data collection cycles; on this basis, according to the first failure log, the interaction success rate of each sub-data collection cycle is determined, including: When the current sub-data collection period is the first sub-data collection period of the current evaluation period, the interaction success rate of the interaction content corresponding to the first failure log in the current sub-data collection period is determined as the current sub-data collection period interaction success rate.

The interaction content corresponding to each log in the first failure log can be performed once or multiple times in the current sub-data collection cycle, and the total number of times the interaction content corresponding to all logs in the first failure log is carried out in the current sub-data collection cycle is used as a sample The total number of cases; the number of successful interactions in the current sub-data collection cycle of the interactive content corresponding to all logs in the first failure log is used as the number of successful cases, and the first failure can be obtained by dividing the number of successful cases by the total number of cases Interaction success rate of the interaction content corresponding to the log in the current sub-data collection cycle.

Further, according to the first failure log, determining the interaction success rate of each sub-data collection period further includes: in the interaction log, in the event that the current sub-data collection period is not the first sub-data collection period of the current evaluation period, Obtain the second failure log of the interaction failure in the previous sub-data collection period of the current sub-data collection period; determine the interaction success rate of the interaction content corresponding to the second failure log in the current sub-data collection period as the current sub-data collection period Periodic interaction success rate.

The interaction content corresponding to each log in the second failure log can be performed once or multiple times in the current sub-data collection cycle, and the total number of times the interaction content corresponding to all logs in the second failure log is carried out in the current sub-data collection cycle is used as a sample The total number of cases; the number of successful interactions in the current sub-data collection cycle of all the interactive content corresponding to the second failure log is used as the number of successful cases, and the second failure can be obtained by dividing the number of successful cases by the total number of cases Interaction success rate of the interaction content corresponding to the log in the current sub-data collection period.

Here, taking a sub-data collection cycle of 7 days and a current evaluation cycle including 4 sub-data collection cycles as an example, an exemplary description is given to obtain the interaction success rate of the current evaluation cycle.

In the current evaluation cycle, the four sub-data collection cycles are as follows in order of time: the first sub-data collection cycle, the second sub-data collection cycle, the third sub-data collection cycle and the fourth sub-data collection cycle. In order to obtain the interaction success rate of the current evaluation period, first obtain the first failure log of the interaction failure in the 7 days before the current evaluation period (the last sub-data collection period in the previous evaluation period) in the interaction log, these first failures The interactive content corresponding to the log can be repeated or non-repetitive; in the first sub-data collection cycle of the current evaluation cycle, the interaction success rate of the interactive content corresponding to the first failure log is obtained, and in the first sub-data collection period, During the data collection period, the interactive content corresponding to each log in the first failure log may be performed one or more times, or the interactive content corresponding to one or more logs in the first failure log may not be performed; the first failure log corresponding to The interaction success rate of the interactive content is taken as the interaction success rate k ₁ of the first sub-data collection period.

Obtain the second failure log of the interaction failure in the first sub-data collection period, and use the interaction success rate of the interaction content corresponding to these second failure logs in the second sub-data collection period as the interaction success of the second sub-data collection period rate k ₂ .

Obtain the second failure log of the interaction failure in the second sub-data collection period, and use the interaction success rate of the interaction content corresponding to these second failure logs in the third sub-data collection period as the interaction success rate of the third sub-data collection period k ₃ .

Obtain the second failure log of the interaction failure in the third sub-data collection period, and use the interaction success rate of the interaction content corresponding to these second failure logs in the fourth sub-data collection period as the interaction success rate of the fourth sub-data collection period k ₄ .

Then, the interaction success rate k'=(k ₁ +k ₂ +k ₃ +k ₄ )/4 in the current evaluation cycle.

By adopting the above steps, the interaction success rate of each sub-data collection cycle can be obtained.

In some application scenarios, determining the interaction success rate of each sub-data collection period according to the first failure log includes: determining the interaction success rate of the interaction content corresponding to the first failure log in each sub-data collection period as the corresponding The interaction success rate of the sub data collection cycle.

The interaction content corresponding to each log in the first failure log can be performed once or multiple times in a sub-data collection cycle, and the total number of times that the interaction content corresponding to all logs in the first failure log is carried out in the sub-data collection cycle is taken as The total number of samples; the number of times the interaction content corresponding to all the logs in the first failure log is successfully interacted in the current sub-data collection cycle is used as the number of sample successes, and the number of successful samples is divided by the total number of samples to obtain the first The interaction success rate of the interaction content corresponding to the failure log within the sub-data collection period; this process is repeated to obtain and determine the interaction success rate of each sub-data collection period in turn.

Here, taking a sub-data collection period of 7 days and the current evaluation period including 4 sub-data collection periods as an example, the interaction success rate of the current evaluation period is obtained as an example.

In the current evaluation cycle, the four sub-data collection cycles are as follows in order of time: the first sub-data collection cycle, the second sub-data collection cycle, the third sub-data collection cycle and the fourth sub-data collection cycle.

Obtain the first failure log of the interaction failure within 7 days before the current evaluation period in the interaction log, and use the interaction success rate of these first failure logs in the first sub-data evaluation period as the first sub-data evaluation period Interaction success rate k ₁ ; take the interaction success rate of these first failure logs in the second sub-data evaluation cycle as the interaction success rate k ₂ of the second sub-data collection cycle; take these first failure logs in the third sub-data The interaction success rate within the evaluation period is taken as the interaction success rate k ₃ of the third sub-data collection cycle; the interaction success rate of these first failure logs in the fourth sub-data collection cycle is taken as the interaction success rate of the fourth sub-data collection cycle rate k ₄ .

S303. Determine the self-learning index according to the interaction success rate of the current evaluation period.

In order to more accurately evaluate the dynamic process of the interactive system improving its own performance through self-learning, a concept of self-learning index is introduced here. The self-learning index is determined based on the aforementioned interaction success rate and interaction failure rate, and the self-learning index is Monotonically increasing function with interaction success rate.

Then, determining the self-learning index according to the interaction success rate of the current evaluation cycle may include: obtaining the interaction failure rate of the current evaluation cycle; determining the self-learning index according to the interaction success rate and interaction failure rate of the current evaluation cycle, so that the self-learning index is Monotonically increasing function with interaction success rate. Afterwards, the self-learning index is used as the optimization criterion for optimization.

Here, the interaction success rate of the current evaluation period may be obtained by using the method provided in the foregoing embodiments.

The optimization method here varies with different interaction models, and those skilled in the art can adopt an appropriate optimization method according to the self-learning model that is the essence of the interaction model, and details will not be repeated here.

Specifically, the self-learning index can be determined by the following formula:

S=k'/(k'+r)

Among them, S is the self-learning index, k' is the interaction success rate, and r is the interaction failure rate. In the case that the interactive failure rate r of the current evaluation period is zero, the self-learning index S reaches the upper limit.

The interaction failure rate of the current evaluation period can be obtained in the following manner: obtain the first number of logs in the current evaluation period in the interaction log, and the second number of interaction failure logs in the current evaluation period; according to the first number and The second quantity, determines the interaction failure rate for the current evaluation cycle.

The interaction failure rate of the current evaluation period can be obtained by dividing the second quantity by the first quantity.

Each log in the interaction log is marked as interaction success and interaction failure. In the interaction log, take the failure logs of the interaction failure within the set time before the current evaluation cycle, and take the interaction content corresponding to these failure logs as an example. The interaction success rate in the current evaluation cycle is evaluated. This sample is selected during the interaction process of the interactive system, which can be used to evaluate the dynamic performance of the interactive system during the interactive process, and then the interactive system can be improved through self-learning. Evaluating the dynamic process of its own performance is beneficial to the self-learning process of the interactive system.

In some embodiments, the device for evaluating the interactive performance of an interactive system includes a processor and a memory storing program instructions, and the processor is configured to execute the method for evaluating the interactive performance of an interactive system provided by the foregoing embodiments when executing the program instructions. Methods.

Fig. 4 is a schematic diagram of an apparatus for evaluating the interaction performance of an interaction system provided by an embodiment of the present disclosure. As shown in Figure 4, the means for evaluating the interactive performance of the interactive system include:

A processor (processor) 41 and a memory (memory) 42 may also include a communication interface (Communication Interface) 43 and a bus 44. Wherein, the processor 41 , the communication interface 43 , and the memory 42 can communicate with each other through the bus 44 . The communication interface 43 can be used for information transmission. The processor 41 may invoke logic instructions in the memory 42 to execute the method for evaluating the interaction performance of the interaction system provided in the foregoing embodiments.

In addition, the above-mentioned logic instructions in the memory 42 may be implemented in the form of software functional units and when sold or used as an independent product, may be stored in a computer-readable storage medium.

The memory 42, as a computer-readable storage medium, can be used to store software programs and computer-executable programs, such as program instructions/modules corresponding to the methods in the embodiments of the present disclosure. The processor 41 executes functional applications and data processing by running software programs, instructions and modules stored in the memory 42, that is, implements the methods in the foregoing method embodiments.

The memory 42 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and at least one application required by a function; the data storage area may store data created according to the use of the terminal device, and the like. In addition, the memory 42 may include a high-speed random access memory, and may also include a non-volatile memory.

An embodiment of the present disclosure provides an interactive system, including the device for evaluating the interactive performance of the interactive system provided in the foregoing embodiments.

An embodiment of the present disclosure provides a computer-readable storage medium, which stores computer-executable instructions, and the computer-executable instructions are configured to execute the method for evaluating the interaction performance of an interaction system provided in the foregoing embodiments.

An embodiment of the present disclosure provides a computer program product. The computer program product includes a computer program stored on a computer-readable storage medium. The computer program includes program instructions. When the program instructions are executed by a computer, the computer is made to execute the information provided in the foregoing embodiments. A method for evaluating the interactive performance of an interactive system.

The above-mentioned computer-readable storage medium may be a transitory computer-readable storage medium, or a non-transitory computer-readable storage medium.

The technical solutions of the embodiments of the present disclosure can be embodied in the form of software products, which are stored in a storage medium and include one or more instructions to enable a computer device (which may be a personal computer, a server, or a network equipment, etc.) to execute all or part of the steps of the methods in the embodiments of the present disclosure. The aforementioned storage medium can be a non-transitory storage medium, including: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), magnetic disk or optical disk, etc. A medium that can store program code, or a transitory storage medium.

The above description and drawings sufficiently illustrate the embodiments of the present disclosure to enable those skilled in the art to practice them. Other embodiments may incorporate structural, logical, electrical, procedural, and other changes. The examples merely represent possible variations. Individual components and functions are optional unless explicitly required, and the order of operations may vary. Portions and features of some embodiments may be included in or substituted for those of other embodiments. Also, the terms used in the present disclosure are used to describe the embodiments only and are not used to limit the claims. As used in the examples and description of the claims, the singular forms "a", "an" and "the" are intended to include the plural forms as well unless the context clearly indicates otherwise . Additionally, when used in this disclosure, the term "comprise" and its variants "comprises" and/or comprising (comprising) etc. refer to stated features, integers, steps, operations, elements, and/or The presence of a component does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components and/or groupings of these. Without further limitations, an element qualified by the statement "comprising a ..." does not preclude the presence of additional identical elements in the process, method or apparatus comprising the element. Herein, what each embodiment focuses on may be the difference from other embodiments, and the same and similar parts of the various embodiments may refer to each other. For the method, product, etc. disclosed in the embodiment, if it corresponds to the method part disclosed in the embodiment, then the relevant part can refer to the description of the method part.

Those skilled in the art can appreciate that the units and algorithm steps of the examples described in conjunction with the embodiments disclosed herein can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed by hardware or software may depend on the specific application and design constraints of the technical solution. Skilled artisans may implement the described functions using different methods for each specific application, but such implementation should not be considered as exceeding the scope of the disclosed embodiments. Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described system, device and unit can refer to the corresponding process in the foregoing method embodiment, which will not be repeated here.

In the embodiments disclosed herein, the disclosed methods and products (including but not limited to devices, equipment, etc.) can be implemented in other ways. For example, the device embodiments described above are only illustrative. For example, the division of units may only be a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components may be combined or may be Integrate into another system, or some features may be ignored, or not implemented. In addition, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces, and the indirect coupling or communication connection of devices or units may be in electrical, mechanical or other forms. A unit described as a separate component may or may not be physically separated, and a component displayed as a unit may or may not be a physical unit, that is, it may be located in one place, or may be distributed to multiple network units. Some or all of the units can be selected according to actual needs to implement this embodiment. In addition, each functional unit in the embodiments of the present disclosure may be integrated into one processing unit, each unit may exist separately physically, or two or more units may be integrated into one unit.

The flowchart and block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to embodiments of the disclosure. In this regard, each block in a flowchart or block diagram may represent a module, program segment, or part of code that includes one or more executable instruction. In some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks in succession may, in fact, be executed substantially concurrently, or they may sometimes be executed in the reverse order, depending upon the functionality involved. Each block in the block diagrams and/or flowcharts, and combinations of blocks in the block diagrams and/or flowcharts, can be implemented by a dedicated hardware-based system that performs the specified function or action, or can be implemented by dedicated hardware implemented in combination with computer instructions. .

Claims

A method for evaluating the interactive performance of an interactive system, comprising:

Obtain the interaction state of each log in the interaction log of the interaction system, and mark each log according to the interaction state;

According to the labeling result of each log, the self-learning index of the interactive system is determined to evaluate the interactive performance of the interactive system.
The method according to claim 1, wherein, according to the labeling result of each log, determining the self-learning index of the interactive system includes:

Obtaining the first failure log of the interaction failure within the set time period before the current evaluation period in the interaction log;

Determine the interaction success rate of the current evaluation period according to the interaction success rate of the interaction content corresponding to the first failure log in the current evaluation period;

The self-learning index is determined according to the interaction success rate of the current evaluation period.
The method of claim 2, wherein the current evaluation period includes a plurality of sub-data collection periods;

According to the interaction success rate of the interaction content corresponding to the first failure log in the current evaluation period, determining the interaction success rate of the current evaluation period includes:

Determine the interaction success rate of each sub-data collection cycle according to the first failure log;

Obtain the average value of the interaction success rate of multiple sub-data collection cycles;

The average value is determined as the interaction success rate of the current evaluation cycle.
The method according to claim 3, wherein the set duration is a sub-data collection period; determining the interaction success rate of each sub-data collection period according to the first failure log, comprising: in the current sub-data collection period If it is the first sub-data collection period of the current evaluation period, the interaction success rate of the interaction content corresponding to the first failure log within the current sub-data collection period is determined as the current sub-data Interaction success rate of collection cycle;

Alternatively, the interaction success rate of the interaction content corresponding to the first failure log in each sub-data collection cycle is determined as the interaction success rate of each sub-data collection cycle.
The method according to claim 4, wherein, according to the first failure log, determining the interaction success rate of each sub-data collection cycle further comprises:

If the current sub-data collection period is not the first sub-data collection period of the current evaluation period, in the interaction log, obtain the interaction failure in the previous sub-data collection period of the current sub-data collection period The second failure log of;

The interaction success rate of the interaction content corresponding to the second failure log in the current sub-data collection period is determined as the interaction success rate of the current sub-data collection period.
The method according to any one of claims 2 to 5, wherein determining the self-learning index according to the interaction success rate of the current evaluation cycle includes:

obtaining the interaction failure rate of the current evaluation period;

A self-learning index is determined according to the interaction success rate and the interaction failure rate in the current evaluation period, so that the self-learning index is a monotonically increasing function along with the interaction success rate.
The method according to claim 6, wherein, according to the interaction success rate and interaction failure rate of the current evaluation period, determining the self-learning index comprises:

S=k'/(k'+r)

Among them, S is the self-learning index, k' is the interaction success rate, and r is the interaction failure rate.
The method according to claim 6, wherein obtaining the interaction failure rate of the current evaluation period comprises:

Obtaining a first number of logs in the interaction log within the current evaluation period, and a second number of interaction failure logs in the current evaluation period;

An interaction failure rate of the current evaluation period is determined according to the first quantity and the second quantity.
The method according to any one of claims 1 to 5, wherein obtaining the interaction status of each log in the interaction log includes:

If the log in the interaction log is determined to be a satisfactory reply from the user through the preset interaction strategy, then determine the interaction status of the log in the interaction log as successful interaction;

If it is determined through the preset interaction policy that the log in the interaction log is an unsatisfactory reply from the user, the interaction status of the log in the interaction log is determined as interaction failure.
A device for evaluating the interactive performance of an interactive system, comprising a processor and a memory storing program instructions, the processor is configured to perform the following method when executing the program instructions:

Obtain the interaction state of each log in the interaction log of the interaction system, and mark each log according to the interaction state;

According to the labeling result of each log, the self-learning index of the interactive system is determined to evaluate the interactive performance of the interactive system.
The device according to claim 10, wherein, when the processor executes determining the self-learning index of the interactive system, specifically perform the following method:

Obtaining the first failure log of the interaction failure within the set time period before the current evaluation period in the interaction log;

Determine the interaction success rate of the current evaluation period according to the interaction success rate of the interaction content corresponding to the first failure log in the current evaluation period;

The self-learning index is determined according to the interaction success rate of the current evaluation period.
The apparatus of claim 11, wherein the current evaluation period comprises a plurality of sub-data collection periods;

When the processor determines the interaction success rate of the current evaluation period according to the interaction success rate of the interaction content corresponding to the first failure log in the current evaluation period, the following method is specifically performed:

Determine the interaction success rate of each sub-data collection cycle according to the first failure log;

Obtain the average value of the interaction success rate of multiple sub-data collection cycles;

The average value is determined as the interaction success rate of the current evaluation period.
The device according to claim 12, wherein the set duration is a sub-data collection period; when the processor determines the interaction success rate of each sub-data collection period according to the first failure log, specifically executes The following method: when the current sub-data collection period is the first sub-data collection period of the current evaluation period, the interaction content corresponding to the first failure log in the current sub-data collection period is successfully interacted rate, determined as the interaction success rate of the current sub-data collection period;

Alternatively, the interaction success rate of the interaction content corresponding to the first failure log in each sub-data collection cycle is determined as the interaction success rate of each sub-data collection cycle.
The device according to claim 13, wherein, when the processor determines the interaction success rate of each sub-data collection cycle according to the first failure log, it also specifically performs the following method:

If the current sub-data collection period is not the first sub-data collection period of the current evaluation period, in the interaction log, obtain the interaction failure in the previous sub-data collection period of the current sub-data collection period The second failure log of;

The interaction success rate of the interaction content corresponding to the second failure log in the current sub-data collection period is determined as the interaction success rate of the current sub-data collection period.
The device according to any one of claims 11 to 14, wherein, when the processor executes determining the self-learning index according to the interaction success rate of the current evaluation cycle, the following method is specifically executed:

obtaining the interaction failure rate of the current evaluation period;

According to the interaction success rate and the interaction failure rate of the current evaluation cycle, determine the self-learning index, so that the self-learning index is a monotonically increasing function with the interaction success rate.
The device according to claim 15, wherein, when the processor determines the self-learning index according to the interaction success rate and interaction failure rate of the current evaluation cycle, the following formula is specifically executed:

S=k'/(k ' +r)

Among them, S is the self-learning index, k' is the interaction success rate, and r is the interaction failure rate.
The device according to claim 15, wherein when the processor executes obtaining the interaction failure rate of the current evaluation period, the following method is specifically executed:

Obtaining a first number of logs in the interaction log within the current evaluation period, and a second number of interaction failure logs in the current evaluation period;

An interaction failure rate of the current evaluation cycle is determined according to the first quantity and the second quantity.
The device according to any one of claims 10 to 14, wherein, when the processor executes obtaining the interaction state of each log in the interaction log, the following method is specifically executed:

If the log in the interaction log is determined to be a satisfactory reply from the user through the preset interaction strategy, then determine the interaction status of the log in the interaction log as successful interaction;

If it is determined through the preset interaction policy that the log in the interaction log is an unsatisfactory reply from the user, the interaction status of the log in the interaction log is determined as interaction failure.
A computer-readable storage medium storing computer-executable instructions, the instruction program executing the method for evaluating the interactive performance of an interactive system according to any one of claims 1 to 9 when executed.