TWI677846B - Method and device for transferring robot customer service to manual customer service - Google Patents

Abstract

This application provides a method for transferring robot customer service to manual customer service, which includes: obtaining valid features from at least one round of conversation between the robot customer service and the user; inputting the valid features into a confidence evaluation model to obtain the current confidence evaluation value of the robot customer service and the user session; The confidence assessment model is trained using conversation samples of robotic customer service and users marked with valid features and suitable artificial points. The suitable artificial points are the appropriate points in time when the artificial customer service is used instead of the robotic customer service. In the case of artificial conditions, the user is transferred to the manual customer service. The technical solution of the present application can identify the degree of user-to-manual customer service needs based on the actual situation of the current conversation. While improving the user's satisfaction with the service, it reduces unnecessary work of manual customer service and improves the service of the customer service system. effectiveness.

Description

Method and device for transferring robot customer service to manual customer service

The present application relates to the field of data processing technology, and in particular, to a method and device for converting robotic customer service to manual customer service.

With the development of the Internet, virtual robots based on artificial intelligence technology are increasingly used in the field of enterprise user services. The robot customer service does not need to rest. It can respond to the user's questions more quickly and standardized, communicate with the user in the form of voice dialogue or text chat, and free the artificial customer service from a large number of repetitive questions and answers.

For some unconventional user questions, it is often difficult for the robot customer service to give satisfactory answers to users. At present, the most commonly used architecture of the customer service center is the coexistence of robotic customer service and manual customer service. By default, the robotic customer service first receives the user. When the robotic customer service and the user have a round of conversations (using one user speech, or one robot customer service speech and one user speech as one) When the number of rounds exceeds the predetermined threshold for out-of-labour rounds, the manual customer service is transferred. In this case, in most cases, a suitable transfer point from the robot customer service to the manual customer service cannot be captured. For example, the robot customer service is performing multiple rounds of effective interaction with the user. Although the predetermined manual round threshold is reached, Did not encounter service barriers that require human intervention, this approach will cause unnecessary amount of manual customer service When the user has urgent problems and needs to submit manual processing as soon as possible, the efficiency of the customer service system and user experience will be affected due to invalid interactions before reaching the predetermined manual round threshold.

In view of this, this application provides a method for transferring robot customer service to human customer service, which includes: obtaining valid features from at least one round of conversation between the robot customer service and the user; inputting the valid features into a confidence evaluation model to obtain the current confidence evaluation of the robot customer service and user sessions The confidence evaluation model is trained by using conversation samples of robotic customer service and users marked with valid features and suitable artificial points. The suitable artificial points are the appropriate points in time when the artificial customer service replaces the robotic customer service. When the value meets the predetermined artificial conditions, the user is transferred to the artificial customer service.

The application also provides a robot customer service to human customer service device, including: an effective feature acquisition unit for acquiring effective features from at least one round of conversation between the robot customer service and the user; a current confidence evaluation unit for inputting effective features into the confidence evaluation The model obtains the current confidence evaluation value of the conversation between the robot customer service and the user; the confidence evaluation model is trained using the conversation samples of the robot customer service and the user marked with valid features and suitable artificial points, which are suitable for artificial points The appropriate point in time is to replace the artificial customer service with the artificial customer service; the artificial customer service transfer unit is used to transfer the user to the manual customer service when the current confidence evaluation value meets the predetermined artificial conditions.

As can be seen from the above technical solutions, in the embodiments of the present application, a conversation model of a robot customer service and a user labeled with valid features and suitable artificial points is used to train a confidence evaluation model, and the ongoing conversation between the robot customer and the user is input to After the confidence evaluation model, the current confidence evaluation value output by the model is used to determine whether it is suitable to transfer to manual customer service. This can identify the degree of user to manual customer service according to the actual situation of the current session, and improve the user's satisfaction with the service. At the same time, it reduces unnecessary work of manual customer service and improves the service efficiency of the customer service system.

FIG. 1 is a flowchart of a method for transferring robot customer service to manual customer service in an embodiment of the present application; FIG. 2 is a hardware structure diagram of a device running an embodiment of the application; FIG. 3 is a robot customer service to manual operation in the embodiment of the present application The logical structure of the customer service device.

The embodiment of the present application proposes a new method of transferring robot customer service to manual customer service. In the conversation sample between the robot customer service and the user, the effective features are manually marked and the artificial points are suitable. The confidence evaluation model is trained, and the confidence after training is used. Assess the effective characteristics of the model and the current session. The service quality of robots is evaluated to find the appropriate time for transferring human service, so that the human point can be determined according to the actual progress of the current session, which can improve the user experience and avoid unnecessary workload of human service. Solve the problems in the prior art.

The embodiments of the present application can be applied to any device having computing and storage capabilities, such as a physical device or a logical device such as a mobile phone, a tablet computer, a PC (Personal Computer, personal computer), a notebook, a server, a virtual machine, etc .; Various functions in the embodiments of the present application may be implemented by two or more physical or logical devices sharing different responsibilities and cooperating with each other.

In the embodiment of the present application, the flow of the method for transferring robot customer service to manual customer service is shown in FIG. 1.

In the embodiment of the present application, a machine learning technology is used to establish a confidence evaluation model, which is used to evaluate whether a robot customer service should be transferred to a manual customer service at a certain point in the conversation between the customer service and the user. Specifically, the conversation records of a certain number of robot customer service and users are used as conversation samples, and the effective features and suitable artificial points during each conversation are manually labeled on the conversation samples, and the confidence evaluation model is trained using the conversation samples.

Among them, the effective feature is an abstract expression of various factors related to the transfer of manual requirements that appear in the conversation sample. Effective features can be generated and evaluated manually, or a combination of manual and data-picking techniques. In an implementation manner, the technical personnel can summarize and refine it according to their own work experience, historical data of the robot customer service and user conversation, and user problem solving.

In another implementation, the characteristics that can be manually summarized and refined that can describe the service quality of the robot customer service and the user's willingness to transfer labor can be used as pending features, and the pending features, suitable artificial points, and actual artificial points can be marked in On the conversation sample, combined with the service effect of each conversation in the conversation sample, a predetermined data analysis algorithm is used to evaluate and integrate the features to be determined, and a number of effective features that have a significant impact on the transfer of manual requirements and comprehensive coverage factors are obtained. For example, you can manually analyze the historical data of the robot customer's conversation with the user to mine the factors that affect the user's demand for artificial services; these factors are refined, classified, and abstracted into pending features. The pending features are from the conversation context and business. , User experience, service trajectory and other aspects comprehensively describe the above factors affecting the transfer of manual requirements; according to user feedback, user problem solving and user satisfaction, etc., each conversation in the conversation sample is obtained Using the feature selection and feature extraction algorithms to analyze the conversation samples and service effects marked with pending features, suitable artificial points and actual artificial points, the effective features are obtained.

Various data analysis algorithms that can be used for feature classification and evaluation can be used to generate effective features, which are not limited in the embodiments of the present application. In some application scenarios, the data analysis algorithm used can not only obtain effective features based on the conversation samples and service effects, but also give the weight of the impact of each effective feature on the transfer of manual requirements. In these application scenarios, The weight of each valid feature will be used for training the confidence assessment model, and / or the use of the confidence assessment model after training.

During the conversation between the robot customer service and the user, the human customer service is usually in The time after the user speaks will replace the robot customer service. In each conversation of the conversation sample, if a user speaks at a suitable point in time after the robotic customer service is replaced by a human customer service, it can be marked as suitable for the artificial point. There can be one or more suitable artificial points in a session.

The confidence evaluation model is used to obtain a confidence evaluation value based on the conversation between the robot customer service and the user. The confidence evaluation value is an estimate of the degree of manual labor required by the confidence evaluation model. Or, at some point in the conversation, the human customer service should replace the robot. The score of the degree of customer service. The embodiment of the present application does not limit the specific algorithm used in the confidence assessment model. For example, it can be an SVR (Support Vector Regression) algorithm, an LR (Logistic Regression) algorithm, or a GBDT (Logistic Regression) algorithm. Gradient Boosting Decision Tree algorithm.

After the training of the confidence evaluation model is completed, during the instant conversation between the robot customer service and the user, the confidence evaluation model can be used to determine whether the user needs to switch to manual customer service at the time after the user speaks during the conversation.

Step 110: Obtain valid features from at least one round of conversation between the robot customer service and the user.

In the embodiment of the present application, one round of speaking is made by one user, or one speaking by a robot customer service and one speaking by a user. Usually the first round of the conversation between the robot customer service and the user is the user's speech, and the second and subsequent rounds are a robot customer service speech and the user's speech. Each round ends with a user's speech, which is also the point at which the artificial customer service can replace the robotic customer service.

The valid features obtained in this step will be used as an input to the confidence assessment model to obtain the confidence assessment value at the current point in time. The target session for a predetermined number of rounds before the current point in time can be used as a basis for obtaining valid features. If the number of sessions that have been performed at the current point in time is less than the predetermined number of rounds, the entire session can be used as a basis for obtaining valid features. Sessions serve as the basis for obtaining valid features; they are not restricted.

In one implementation, effective features can be extracted directly from one to multiple rounds of conversations between the robot customer service and the user according to certain rules. For example, for text conversations, you can refer to existing technologies such as semantic analysis, keyword matching, and business content matching to set the recognition rules for each effective feature, so as to automatically obtain effective features from the above conversations. For voice conversations, you can first use speech recognition technology to It is converted into a text conversation, and the above methods are used to obtain effective features.

In another implementation, the original features can be extracted from one to multiple rounds of conversations between the robot customer service and the user, and then the original features can be split, combined, classified, and / or deleted according to the feature pre-processing rules to obtain valid features. . Similarly, for text conversations, you can refer to existing technologies such as semantic analysis, keyword matching, and business content matching to set the discovery rules for each original feature, so as to automatically extract the original features from one or more rounds of conversations; for voice conversations, you can first The speech recognition technology is used to convert the speech conversation into a text conversation, and the original features are extracted using the discovery rules of the original features.

Feature preprocessing rules describe the mapping from original features to valid features, including one or more of the following: which original feature or features can Delete, which original feature can be split into which valid features, which original feature or features can be classified as valid features representing a certain category, which original features can be combined into which valid feature, and so on.

For example, in an application scenario, the original features in the two dimensions of business and user feedback are shown in Table 1:

In the original features in Table 1, if a response from the robot customer service is a bottom answer, the answer must have a low degree of matching (if the robot customer service can find a highly matching answer, the answer will be used instead of the bottom answer. Answer the user repeatedly), assuming that the answer matching degree and whether the answer is a valid feature, you can include one in the feature pre-processing rule: if a reply can extract whether the answer is the original feature and the answer match degree original feature, Then combine the two original features into the original answer of the answer to the question of whether to use the two valid features at the same time, so as not to cause the double impact of the same fact on the confidence evaluation value.

Since the confidence assessment model is trained based on valid features, the use of the trained confidence assessment model also takes valid features as input. Because training a confidence assessment model requires a certain number of manually labeled conversation samples, the actual conversation situation in customer service may change constantly due to the increase in business, changes in business processes, and changes in popular languages. In this implementation, general and abstract features can be used as effective features, and the original features, original feature discovery rules, and feature pre-processing rules can be set according to the specific circumstances of business development and changes, so that there is no need to Changes in the situation constantly regenerate the conversation samples and train the confidence assessment model, while still maintaining the accuracy of the confidence assessment model.

The original feature discovery rules and / or feature pre-processing rules can be fixed in code in the program that completes this step, or written in a configuration file. For application scenarios where a profile exists, you can obtain the profile before executing this step, read the original feature discovery rules and / or feature pre-processing rules from it, and then apply it to the original feature extraction and / or obtain from the original features Effective features.

It should be noted that the original features may be selected from the pending features, or other features that do not exist in the pending features may be used as the original features without limitation.

In step 120, the effective features are input into the confidence evaluation model to obtain the current confidence evaluation value of the conversation between the robot customer service and the user.

Step 130: When the current confidence evaluation value meets the predetermined artificial condition, the user is transferred to the artificial customer service.

After the valid features corresponding to the current point in time between the robot customer service and the user are obtained, the valid features are input into the trained confidence assessment model to obtain the confidence value at the current point in time. If the current confidence assessment value meets the predetermined artificial conditions, it is considered that the manual customer service intervention is required at the current point in time, and the user is transferred to the manual customer service.

The predetermined artificial condition may be that the current confidence evaluation value is greater than or less than the predetermined confidence threshold, depending on whether the higher current confidence evaluation value in the actual application scenario represents a stronger degree of manual labor demand or a weaker degree of manual labor demand. The predetermined confidence threshold can be determined by a technician comprehensively considering factors such as the degree of fit between the confidence evaluation model after training and the conversation samples, the number of user conversations and the number of artificial customer service in the actual application scenario. It is also possible to set a certain standard, and the program automatically determines a predetermined confidence threshold according to the set standard. For example, the conversation samples can be input into the trained confidence assessment model to obtain the sample confidence assessment values corresponding to the artificial points in the conversation samples. A series of specific predetermined confidence thresholds are set, and when different values are selected, Coverage of sample confidence assessments at predetermined confidence thresholds Rate and accuracy rate, setting evaluation criteria for coverage rate and accuracy rate, and evaluating the best coverage rate and accuracy rate corresponding to the evaluation criteria as the predetermined confidence threshold. Among them, the coverage rate is the proportion of the sample confidence assessment values of all sample confidence assessment values corresponding to artificial points in the conversation sample that meet the predetermined artificial conditions (specific values greater than or less than the predetermined confidence threshold); the accuracy rate is The proportion of the sample confidence evaluation value that meets the artificial condition that is predetermined, corresponding to the sample confidence evaluation value suitable for the artificial point.

In addition, in the application scenario where the profile is used, the predetermined artificial condition may also be written into the profile, and the predetermined artificial condition read from the profile may be applied to step 130.

It can be seen that in the embodiment of the present application, the effective features and suitable artificial points are manually labeled in the conversation samples between the robot customer service and the user. The confidence evaluation model is trained using the labeled conversation samples, and the trained confidence evaluation model is used to The effective features in the current session are evaluated, and whether or not a manual customer service should be transferred based on the output of the model can be used to determine the artificial point according to the actual progress of the current session, which can improve the user experience and avoid manual work. Unnecessary workload of customer service, improve service efficiency of customer service system.

In an application example of this application, historical records of conversations between several robot customer service and users are used as conversation samples, and professional customer service staff analyzes the conversation samples based on different dimensions (including business dimensions, user experience dimensions, and service trajectory dimensions). Etc.) Analyze the conversation samples, summarize and refine the factors that affect the transfer of manual needs, and construct them as features to be determined.

Mark the pending features, suitable artificial points and actual artificial points in the conversation sample. According to the user feedback, user's problem solving situation and user satisfaction of each conversation in the conversation sample, use the conversation sample to determine the pending structure. The characteristics are analyzed by data, from which the effective features are evaluated and the weights of the effective features are obtained.

Mark the effective features and artificial points in the conversation samples, and use the labeled conversation samples to train the confidence evaluation model of the LR algorithm. The conversation in the conversation sample is input into the trained LR confidence assessment model, and the sample confidence assessment value corresponding to the artificial point in the conversation sample is obtained. The technicians according to the LR confidence assessment model cover the conversation sample and the accuracy rate. Determine the scheduled information threshold.

The technician writes the original feature discovery rules and feature pre-processing rules in the configuration file and saves them in a predetermined location. This application example loads the original feature discovery rules and feature preprocessing rules from the configuration file after it starts running.

After the user starts a conversation with the robot customer service, at the end of each user's speech, according to the original feature discovery rules, all original features are extracted from the conversation that has been conducted from the beginning of the session to the current point in time, and then feature preprocessing rules are applied to all The original features are mapped to one or more valid features belonging to the current point in time.

The valid features belonging to the current time point are input into the LR confidence evaluation model after training, and the current confidence evaluation value is obtained. Set the LR confidence evaluation model to obtain a higher evaluation value, indicating a stronger need for transfer labor. When the current confidence evaluation value is greater than the predetermined information threshold, transfer the user to the manual customer service. No Then the robot customer service continues to talk with the user.

Corresponding to the implementation of the above process, the embodiment of the present application further provides a device for transferring robot customer service to manual customer service. The device may be implemented by software, or may be implemented by hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, it is formed by reading a corresponding computer program instruction into a memory through a CPU (Central Process Unit, central processing unit) of a device where the device is located. At the hardware level, in addition to the CPU, memory, and non-volatile memory shown in Figure 2, the devices where the robot customer service is transferred to the human customer service usually include other hardware such as chips for wireless signal transmission and reception. And / or other hardware such as boards for network communication functions.

FIG. 3 shows a device for transferring robot customer service to manual customer service according to an embodiment of the present application, which includes an effective feature acquisition unit, a current confidence evaluation unit, and a manual customer service transfer unit. Among them, the effective feature acquisition unit is used to transfer information from the robot customer service to the user. Obtain valid features in at least one round of conversation; the current confidence assessment unit is used to input valid features into the confidence assessment model to obtain the current confidence assessment value of the robot customer service and the user session; the confidence assessment model uses valid features and is suitable for artificial points The robot customer service and the user ’s conversation samples are trained. The suitable labor point is the appropriate time to replace the robot customer service with the artificial customer service. The manual customer service transfer unit is used when the current confidence evaluation value meets the predetermined artificial conditions. Transfer to manual customer service.

In one example, the effective feature acquisition unit is specifically configured to extract the original features from at least one round of the conversation between the robot customer service and the user. The feature pre-processing rules are used to split, combine, classify, and / or delete the original features to obtain valid features.

In the above example, the device further includes a profile obtaining unit for obtaining a profile; the profile includes an original feature discovery rule and / or a feature pre-processing rule; and the effective feature obtaining unit is specifically configured to: During at least one round of conversation between the customer service and the user, the original features are extracted according to the original feature discovery rules.

Optionally, the setting file further includes: presetting artificial conditions.

In the above example, the original characteristics include: business relevance, answer matching, number of times the answer is repeated, whether the answer is a bottom line, whether the answer is a question, the user explicitly proposes a labor change, and the user has a potential labor change tendency, the user's emotional tendency, The user is explaining at least one of their questions.

In an implementation manner, the effective feature is determined by a predetermined data analysis algorithm based on a sample of a conversation between a robot and a user marked with pending features, suitable artificial points, and actual artificial points, and a service effect of the conversation among them. It is obtained after evaluation and integration; the pending features can describe the service quality of the robot customer service and the user's willingness to transfer labor.

In the above implementation manner, the effective features have respective weights and are calculated by the predetermined data analysis algorithm; the confidence evaluation model is trained according to the weights of the effective features.

Optionally, the predetermined artificial condition includes: a current confidence evaluation value is greater than or less than a predetermined confidence threshold value; the predetermined confidence threshold value is determined according to the coverage rate and accuracy rate of several sample confidence evaluation values; and the sample confidence evaluation value is Output obtained by inputting a conversation sample into a confidence assessment model; said The coverage rate is the proportion of all sample confidence evaluation values corresponding to artificial points in the conversation sample that meet the predetermined artificial conditions; the accuracy rate is all sample confidence evaluation values that meet the predetermined artificial conditions. The proportion of the sample confidence evaluation value corresponding to the appropriate artificial point.

Optionally, the confidence assessment model uses a support vector regression SVR algorithm, a logistic regression LR algorithm, or an iterative decision tree GBDT algorithm.

The above are only preferred embodiments of this application, and are not intended to limit this application. Any modification, equivalent replacement, or improvement made within the spirit and principles of this application shall be included in this application Within the scope of protection.

In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

Memory may include non-persistent memory, random access memory (RAM), and / or non-volatile memory in computer-readable media, such as read-only memory (ROM) or flash memory (flash) RAM). Memory is an example of a computer-readable medium.

Computer-readable media includes permanent and non-permanent, removable and non-removable media. Information can be stored by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), and other types of random access memory (RAM), read-only memory (ROM), electrically erasable and programmable read-only memory (EEPROM), flash memory or other memory technologies, read-only disc read-only memory (CD-ROM), Digital versatile discs (DVDs) or other optical storage, magnetic tape cartridges, magnetic tape storage or other magnetic storage devices, or any other non-transmitting media may be used to store information that can be accessed by computing devices. As defined herein, computer-readable media does not include temporary computer-readable media (transitory media), such as modulated data signals and carrier waves.

It should also be noted that the terms "including," "including," or any other variation thereof are intended to encompass non-exclusive inclusion, so that a process, method, product, or device that includes a series of elements includes not only those elements but also Other elements not explicitly listed, or those that are inherent to such a process, method, product, or device. Without more restrictions, the elements defined by the sentence "including a ..." do not exclude the existence of other identical elements in the process, method, product, or equipment that includes the elements.

Those skilled in the art should understand that the embodiments of the present application may be provided as a method, a system or a computer program product. Therefore, this application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Moreover, this application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk memory, CD-ROM, optical memory, etc.) containing computer-usable code. .

Claims (16)

A method for converting a robot customer service to a human customer service, which comprises: obtaining valid features from at least one round of conversation between the robot customer service and the user; inputting the effective features into a confidence evaluation model to obtain the current confidence evaluation value of the robot customer service and user sessions; The confidence evaluation model is trained by using a conversation sample of a robot customer service marked with effective features and suitable artificial points, which is an appropriate point in time when the artificial customer service replaces the robot customer service; when the current confidence evaluation value satisfies a predetermined When the artificial condition is found, the user is transferred to the artificial customer service, wherein the effective features are determined by a predetermined data analysis algorithm based on a sample of the conversation between the robot and the user marked with pending features, suitable for artificial points, and actual artificial points, and among them The service effect of the conversation is obtained after evaluating and integrating a number of pending features; the pending features can describe the service quality of the robot customer service and the user's willingness to transfer to labor. The method according to item 1 of the scope of patent application, wherein the obtaining valid features from at least one round of conversation between the robot customer service and the user includes: extracting original features from at least one round of conversation between the robot customer service and the user, and preprocessing according to the features The original features are split, combined, classified, and / or deleted to obtain valid features. The method according to item 2 of the scope of patent application, wherein the method further comprises: obtaining a profile; the profile includes original feature discovery rules and / or feature pre-processing rules; the slave robot customer service and the user's Extracting original features in at least one round of conversation includes extracting original features from at least one round of conversations between the robot customer service and the user according to the original feature discovery rules. The method according to item 3 of the scope of patent application, wherein the configuration file further includes: pre-determining artificial conditions. The method according to item 2 of the scope of patent application, wherein the original characteristics include: business relevance, answer matching degree, number of times the answer is repeated, whether the answer is a bottom answer, whether the answer is a question, the user explicitly proposes a labor change, and the user has potential Change at least one of an artificial tendency, a user's emotional tendency, and a user's explanation of his or her own problem. The method according to item 1 of the scope of patent application, wherein the effective features have respective weights and are calculated by the predetermined data analysis algorithm; and the confidence evaluation model is trained according to the weights of the effective features. The method according to item 1 of the scope of patent application, wherein the predetermined artificial conditions include: a current confidence evaluation value is greater than or less than a predetermined confidence threshold value; and the predetermined confidence threshold value is based on the coverage and accuracy of several sample confidence evaluation values. The accuracy rate is determined; the sample confidence evaluation value is an output obtained by inputting the conversation sample into the confidence evaluation model; and the coverage rate is the sample that satisfies the artificial condition among all the sample confidence evaluation values corresponding to the artificial points in the conversation sample. Proportion of confidence evaluation value; the accuracy rate is the proportion of the sample confidence evaluation value corresponding to the artificial point suitable among all the sample confidence evaluation values that meet the predetermined artificial conditions. The method according to item 1 of the scope of patent application, wherein the confidence evaluation model adopts a support vector regression SVR algorithm, a logistic regression LR algorithm, or an iterative decision tree GBDT algorithm. A device for converting robot customer service to manual customer service, comprising: an effective feature acquisition unit for acquiring effective features from at least one round of conversation between the robot customer service and the user; a current confidence evaluation unit for inputting effective features into a confidence evaluation model To obtain the current confidence evaluation value of the conversation between the robot customer service and the user; the confidence evaluation model is trained using the conversation samples of the robot customer service and the user marked with valid features and suitable artificial points, and the suitable artificial points are artificial customer service The appropriate time point to replace the robot customer service; the manual customer service transfer unit is used to transfer the user to the manual customer service when the current confidence evaluation value meets the predetermined artificial conditions, wherein the effective characteristics are determined by the predetermined data analysis algorithm according to To-be-determined features, conversation samples of robots and users suitable for artificial points and actual artificial points, and the service effects of the conversations are obtained after evaluating and integrating certain to-be-determined features; the to-be-determined features can describe the service quality of robot customer service And user willingness The device according to item 9 of the scope of the patent application, wherein the effective feature obtaining unit is specifically configured to: extract the original features from at least one round of the conversation between the robot customer service and the user; split the original features according to the feature preprocessing rules; Combined, classified, and / or deleted to get valid features. The device according to item 10 of the scope of patent application, wherein the device further comprises: a profile obtaining unit for obtaining a profile; the profile includes original feature discovery rules and / or feature pre-processing rules; The effective feature acquisition unit is specifically configured to extract the original features from at least one round of conversation between the robot customer service and the user according to the original feature discovery rules. The device according to item 11 of the scope of patent application, wherein the configuration file further includes: predetermined artificial conditions. The device according to item 10 of the scope of patent application, wherein the original characteristics include: business relevance, answer matching degree, number of times the answer is repeated, whether the answer is a bottom answer, whether the answer is a question, the user explicitly proposes a labor change, and the user has potential Change at least one of an artificial tendency, a user's emotional tendency, and a user's explanation of his or her own problem. The device according to item 9 of the scope of the patent application, wherein the effective features have respective weights and are calculated by the predetermined data analysis algorithm; and the confidence evaluation model is trained according to the weights of the effective features. The device according to item 9 of the scope of patent application, wherein the predetermined artificial conditions include: a current confidence evaluation value is greater than or less than a predetermined confidence threshold value; and the predetermined confidence threshold value is based on the coverage and accuracy of several sample confidence evaluation values. The accuracy rate is determined; the sample confidence evaluation value is an output obtained by inputting the conversation sample into the confidence evaluation model; and the coverage rate is the sample that satisfies the artificial condition among all the sample confidence evaluation values corresponding to the artificial points in the conversation sample. Proportion of confidence evaluation value; the accuracy rate is the proportion of the sample confidence evaluation value corresponding to the artificial point suitable among all the sample confidence evaluation values that meet the predetermined artificial conditions. The device according to item 9 of the scope of patent application, wherein the confidence evaluation model adopts a support vector regression SVR algorithm, a logical cohoming LR algorithm, or an iterative decision tree GBDT algorithm.