CN110928788A - Service verification method and device - Google Patents
Service verification method and device Download PDFInfo
- Publication number
- CN110928788A CN110928788A CN201911157503.9A CN201911157503A CN110928788A CN 110928788 A CN110928788 A CN 110928788A CN 201911157503 A CN201911157503 A CN 201911157503A CN 110928788 A CN110928788 A CN 110928788A
- Authority
- CN
- China
- Prior art keywords
- tested
- data
- customer service
- verified
- service robot
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/36—Preventing errors by testing or debugging software
- G06F11/3604—Software analysis for verifying properties of programs
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
- G06F16/33—Querying
- G06F16/332—Query formulation
- G06F16/3329—Natural language query formulation or dialogue systems
Abstract
The embodiment of the invention provides a service verification method and equipment, wherein the method comprises the following steps: determining the total number of the collection problems according to the number of the user problems of the production environment; determining the acquisition proportion of each confidence coefficient based on the distribution of the confidence coefficients of the production environment; determining the problem collection quantity of each confidence degree according to the total number of the collection problems and the collection proportion of each confidence degree; acquiring to-be-tested data of each confidence degree from user problems of the production environment based on the problem acquisition quantity of each confidence degree; inputting the data to be tested into a customer service robot to be verified; obtaining the reply of the customer service robot to be verified to the data to be tested; and verifying the service of the customer service robot to be verified according to the reply and the reply corresponding to the preset data to be tested, namely realizing service verification before the version of the intelligent customer service robot is on line, timely finding the problem of the customer service robot and avoiding the influence caused by service verification after the version is on line.
Description
Technical Field
The embodiment of the invention relates to the technical field of information processing, in particular to a service verification method and device.
Background
With the development of social economy, the living standard of people is continuously improved, and online shopping enters the daily life of people. In online shopping, when a plurality of clients exist, manual customer service is difficult to provide timely and high-quality answers for the clients, and an intelligent customer service robot is required to provide services. The service efficiency and the service quality of the intelligent customer service robot influence the experience of customers.
Generally, the service condition of the intelligent customer service robot is judged according to the operation and reply content of the customer service staff to the answer after the version of the intelligent customer service robot is on line. For example, the service quality of the intelligent customer service robot is judged according to the matching degree of the customer questions and the answer returned by the intelligent customer service robot.
However, the above method for determining the service condition of the intelligent service robot is performed after the version of the intelligent service robot is on-line, which is a post-determination. If the intelligent customer service robot needs to be adjusted, the quality of customer service may be reduced, customer experience may be affected, and normal operation of a service may be affected.
Disclosure of Invention
The embodiment of the invention provides a service verification method and service verification equipment, and aims to solve the problems that the existing method for judging the service condition of an intelligent customer service robot causes the quality of customer service to be reduced, the customer experience is influenced, and the normal operation of a service is further influenced.
In a first aspect, an embodiment of the present invention provides a service verification method, including:
determining the total number of the collection problems according to the number of the user problems of the production environment;
determining the acquisition proportion of each confidence coefficient based on the distribution of the confidence coefficients of the production environment;
determining the problem collection quantity of each confidence degree according to the total number of the collection problems and the collection proportion of each confidence degree;
acquiring to-be-tested data of each confidence degree from user problems of the production environment based on the problem acquisition quantity of each confidence degree;
inputting the data to be tested into a customer service robot to be verified;
obtaining the reply of the customer service robot to be verified to the data to be tested;
and verifying the service of the customer service robot to be verified according to the reply and the preset reply corresponding to the data to be tested.
Optionally, the method further comprises:
determining the acquisition proportion of each problem type based on the distribution of the problem types of the production environment;
determining the collection number of each problem type according to the total number of the collection problems and the collection proportion of each problem type;
the acquiring number of the problems based on each confidence coefficient and the acquiring of the to-be-tested data of each confidence coefficient from the user problems of the production environment comprise:
and acquiring to-be-tested data from the user questions of the production environment based on the question acquisition quantity of each confidence degree and the acquisition quantity of each question type.
Optionally, the verifying the service of the customer service robot to be verified according to the reply and a preset reply corresponding to the data to be tested includes:
judging whether the reply is consistent with the reply corresponding to the preset data to be tested;
and if the reply is consistent with the reply corresponding to the preset data to be tested, judging that the service verification of the customer service robot to be verified is passed.
Optionally, after the acquiring data to be tested, further comprising:
and determining the customer service robot to be verified corresponding to the data to be tested according to the corresponding relation between the preset user question and the customer service robot.
Optionally, after verifying the service of the customer service robot to be verified according to the reply and a preset reply corresponding to the data to be tested, the method further includes:
and debugging the customer service robot to be verified according to the service verification result of the customer service robot to be verified.
Optionally, after verifying the service of the customer service robot to be verified according to the reply and a preset reply corresponding to the data to be tested, the method further includes:
and adjusting the reply corresponding to the data to be tested according to the service verification result of the customer service robot to be verified and the confidence coefficient of the data to be tested.
Optionally, the method further comprises:
obtaining a verification passing proportion according to the service verification result of the customer service robot to be verified;
and determining a target confidence degree interval of the customer service robot to be verified based on the verification passing proportion.
In a second aspect, an embodiment of the present invention provides a service verification apparatus, including:
the first determining module is used for determining the total number of the collected problems according to the number of the user problems of the production environment;
determining the acquisition proportion of each confidence coefficient based on the distribution of the confidence coefficients of the production environment;
determining the problem collection quantity of each confidence degree according to the total number of the collection problems and the collection proportion of each confidence degree;
the acquisition module is used for acquiring to-be-tested data of each confidence degree from user problems of the production environment based on the problem acquisition quantity of each confidence degree;
optionally, the first determining module is further configured to:
determining the acquisition proportion of each problem type based on the distribution of the problem types of the production environment;
determining the collection number of each problem type according to the total number of the collection problems and the collection proportion of each problem type;
the obtaining module obtains to-be-tested data of each confidence degree from user problems of the production environment based on the problem collection quantity of each confidence degree, and the obtaining module comprises the following steps:
and acquiring to-be-tested data from the user questions of the production environment based on the question acquisition quantity of each confidence degree and the acquisition quantity of each question type.
The input module is used for inputting the data to be tested into the customer service robot to be verified;
the obtaining module is used for obtaining the reply of the customer service robot to be verified to the data to be tested;
and the verification module is used for verifying the service of the customer service robot to be verified according to the reply and the preset reply corresponding to the data to be tested.
Optionally, the verification module is specifically configured to:
judging whether the reply is consistent with the reply corresponding to the preset data to be tested;
and if the reply is consistent with the reply corresponding to the preset data to be tested, judging that the service verification of the customer service robot to be verified is passed.
Optionally, the method further comprises:
a second determining module for determining whether the data to be tested is obtained by the obtaining module,
and determining the customer service robot to be verified corresponding to the data to be tested according to the corresponding relation between the preset user question and the customer service robot.
Optionally, the method further comprises:
the debugging module is used for verifying the service of the customer service robot to be verified according to the reply and the reply corresponding to the preset data to be tested by the verification module,
and debugging the customer service robot to be verified according to the service verification result of the customer service robot to be verified.
Optionally, the method further comprises:
the adjusting module is used for verifying the service of the customer service robot to be verified according to the reply and the preset reply corresponding to the data to be tested by the verifying module,
and adjusting the reply corresponding to the data to be tested according to the service verification result of the customer service robot to be verified and the confidence coefficient of the data to be tested.
Optionally, further comprising;
the processing module is used for obtaining a verification passing proportion according to the service verification result of the customer service robot to be verified;
and determining a target confidence degree interval of the customer service robot to be verified based on the verification passing proportion.
In a third aspect, an embodiment of the present invention provides a service verification apparatus, including: at least one processor and memory;
the memory stores computer-executable instructions;
the at least one processor executing the computer-executable instructions stored by the memory causes the at least one processor to perform the service verification method as described above in the first aspect and various possible designs of the first aspect.
In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, where computer-executable instructions are stored, and when a processor executes the computer-executable instructions, the service verification method according to the first aspect and various possible designs of the first aspect are implemented.
According to the service verification method and the device provided by the embodiment of the invention, the total number of the acquisition problems is reasonably determined according to the number of the user problems of the production environment, the excessive or insufficient total number is avoided, and the acquisition proportion of each confidence coefficient is determined based on the distribution of the confidence coefficients of the production environment; determining the problem acquisition quantity of each confidence degree according to the total number of the acquisition problems and the acquisition proportion of each confidence degree, thereby simulating the condition of a production environment user, enabling the service verification of the robot to be close to the actual condition, and acquiring the to-be-tested data of each confidence degree from the user problems of the production environment based on the problem acquisition quantity of each confidence degree; inputting the data to be tested into the customer service robot to be verified; then, the reply of the customer service robot to be verified to the data to be tested is obtained; according to the reply of the customer service robot to be verified to the data to be tested and the reply corresponding to the preset data to be tested, the service of the customer service robot to be verified is verified, namely the service verification is realized before the version of the intelligent customer service robot is on line, the problem of the customer service robot can be found in time, the influence caused by the service verification after the version is on line is avoided, the service quality of the customer service robot after the version is on line is ensured, and the user experience is improved.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings needed to be used in the description of the embodiments or the prior art will be briefly introduced below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to these drawings without creative efforts.
Fig. 1 is a schematic structural diagram of a service verification system according to an embodiment of the present invention;
fig. 2 is a first flowchart illustrating a service verification method according to an embodiment of the present invention;
fig. 3 is a second flowchart illustrating a service verification method according to an embodiment of the present invention;
fig. 4 is a first schematic structural diagram of a service verification apparatus according to an embodiment of the present invention;
fig. 5 is a schematic structural diagram of a service verification apparatus according to an embodiment of the present invention;
fig. 6 is a schematic diagram of a hardware structure of a service authentication device according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, but not all, embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Generally, the service condition of the intelligent customer service robot is judged according to the operation and reply content of the customer service staff to the answer after the version of the intelligent customer service robot is on line. For example, the service quality of the intelligent customer service robot is judged according to the matching degree of the customer questions and the answer returned by the intelligent customer service robot.
However, the above method for determining the service condition of the intelligent service robot is performed after the version of the intelligent service robot is on-line, which is a post-determination. If the intelligent customer service robot needs to be adjusted, the customer service efficiency or quality may be reduced, customer experience is affected, and further normal operation of business is affected.
Therefore, in view of the above problems, the present embodiment provides a service verification method, which reasonably determines the total number of the collection problems according to the number of the user problems in the production environment, so as to avoid excessive or insufficient total number, and determine the collection proportion of each confidence degree based on the distribution of the confidence degrees of the production environment; determining the problem acquisition quantity of each confidence degree according to the total number of the acquisition problems and the acquisition proportion of each confidence degree, thereby simulating the condition of a production environment user, enabling the service verification of the robot to be close to the actual condition, and acquiring the to-be-tested data of each confidence degree from the user problems of the production environment based on the problem acquisition quantity of each confidence degree; inputting the data to be tested into the customer service robot to be verified; then, the reply of the customer service robot to be verified to the data to be tested is obtained; and verifying the service of the customer service robot to be verified according to the reply of the customer service robot to be verified to the data to be tested and the reply corresponding to the preset data to be tested. The service verification performed before the version of the intelligent customer service robot is on line can timely find the problem of the customer service robot, avoid the influence caused by the service verification performed after the version is on line, ensure the service quality of the customer service robot after the version is on line, and improve the user experience.
The present embodiment provides a service verification method, which may be applied to the schematic architecture diagram of the service verification system shown in fig. 1 according to the embodiment of the present invention, where as shown in fig. 1, the system provided in the present embodiment includes a terminal 101. The terminal 101 can reasonably determine the total number of the collected problems according to the magnitude of the user problems in the production environment; the acquisition proportion of each confidence coefficient can be determined based on the distribution of the confidence coefficients of the production environment; the problem collection quantity of each confidence coefficient can be determined according to the total number of the collection problems and the collection proportion of each confidence coefficient; or acquiring to-be-tested data of each confidence degree from user problems of the production environment based on the problem acquisition quantity of each confidence degree; the data to be tested can also be input into the customer service robot 102 to be verified; the reply of the customer service robot 102 to be verified to the data to be tested can also be obtained; meanwhile, the service of the customer service robot 102 to be verified can be verified according to the reply of the customer service robot 102 to be verified to the data to be tested and the reply corresponding to the preset data to be tested. The terminal 101 may be a mobile phone, a tablet, or the like. The customer service robot to be verified is any one or more robots needing service verification. The data to be tested can be determined according to actual conditions, and can also be set by related personnel, for example, the data to be tested is set as some user problem data.
The following describes the technical solution of the present invention and how to solve the above technical problems with specific examples. The following several specific embodiments may be combined with each other, and details of the same or similar concepts or processes may not be repeated in some embodiments. Embodiments of the present invention will be described below with reference to the accompanying drawings.
Fig. 2 is a first flowchart illustrating a service verification method according to an embodiment of the present invention, where an execution main body of the embodiment may be the terminal 101 in the embodiment shown in fig. 1, or may be a server, and the embodiment is not limited herein. As shown in fig. 2, the method may include:
s201: determining a total number of collection problems based on the number of user problems for the production environment.
Illustratively, the amount of user data that needs to be collected is determined by the magnitude of the amount of user problems in the production environment. For example: the production environment data is less than 1 ten thousand, and 20% of the data needs to be collected; the production environment data is hundreds of thousands of orders of magnitude, and 5% of the data needs to be collected; production data is in the million-scale, and 1% of the production data needs to be collected; production data is in the order of millions, and 0.1% of the production data needs to be collected.
The total number of the collection problems is reasonably determined according to the number of the user problems of the production environment, and the excessive or insufficient total number is avoided.
S202: and determining the acquisition proportion of each confidence coefficient based on the distribution of the confidence coefficients of the production environment.
For example, in an actual production environment, the different confidences correspond to different data acquisition proportions, and the acquisition proportions of the confidences are determined based on the distribution of the confidences of the production environment. For example: the confidence coefficient takes 0.01 as a unit, and the acquisition proportions of different confidence coefficients between 0 and 1 of the production environment are determined according to the distribution condition of the confidence coefficient of the production environment.
S203: determining the problem collection quantity of each confidence degree according to the total number of the collection problems and the collection proportion of each confidence degree;
illustratively, the production environment data is 10 ten thousand, 5% of the data needs to be collected, and the total number of collection problems is 5000; the acquisition rate with a confidence of 0.5 is 1%, and 50 pieces are acquired for the confidence.
Optionally, the method further comprises determining an acquisition proportion of each problem type at each confidence level based on the distribution of the problem types of the production environment.
The problem types can be claims, insurance, renewal, product, PR, and the like. Different problem types have different distribution ratios of the respective confidence degrees in the actual production environment.
S204: and acquiring the data to be tested of each confidence degree from the user questions of the production environment based on the question acquisition quantity of each confidence degree.
Illustratively, according to the problem collection quantity of each confidence degree, user problems of the quantity corresponding to each confidence degree are obtained from the user problems of the production environment, and the user problems, the confidence degrees and the corresponding replies are the data to be tested.
Optionally, before S204 or after S204, further comprising:
determining the acquisition proportion of each problem type based on the distribution of the problem types of the production environment;
and determining the collection quantity of each problem type according to the total number of the collection problems and the collection proportion of each problem type.
The acquiring the number of the problems based on the confidence degrees and acquiring the to-be-tested data of the confidence degrees from the user problems of the production environment comprises the following steps:
and acquiring to-be-tested data from the user questions of the production environment based on the question acquisition quantity of each confidence degree and the acquisition quantity of each question type.
Illustratively, the production environment data is 10 thousands, 5% of the data needs to be collected, the total number of the collection problems is 5000, the collection proportion with the confidence coefficient of 0.5 is 1%, and the collection number is 50; the collection proportion of claim settlement problems in the production environment is 10%, and the collection quantity is 500.
By comprehensively considering the distribution of each problem type and the distribution of each confidence coefficient, the data to be tested is obtained from the user problems of the production environment, and the simulation of the production environment can be further improved.
The data to be detected are obtained in the above mode, the condition of a production environment user can be simulated, the data to be detected are more reliable, and the follow-up steps can be more effectively used for service verification of the customer service robot to be verified.
S205: and inputting the data to be tested into the customer service robot to be verified.
Optionally, when there are a plurality of customer service robots, after acquiring the data to be tested in S205, the method further includes:
and determining the customer service robot to be verified corresponding to the data to be tested according to the corresponding relation between the preset user question and the customer service robot, and executing S206.
In a scene that the customer service robot is multiple, the customer service robot to be verified corresponding to the user problem data is determined according to the corresponding relation between the preset problem data and the customer service robot, the customer service robot to be verified can be selected in a targeted mode by combining the user problem data, and the requirements of different scenes are met.
S206: and obtaining the reply of the customer service robot to be verified to the data to be tested.
S207: and verifying the service of the customer service robot to be verified according to the reply and the preset reply corresponding to the data to be tested.
Optionally, the verifying the service of the customer service robot to be verified according to the reply and the reply corresponding to the preset data to be tested may be implemented by, but not limited to, the following manners:
judging whether the reply is consistent with the reply corresponding to the preset data to be tested;
and if the reply is consistent with the reply corresponding to the preset data to be tested, judging that the service verification of the customer service robot to be verified is passed.
Different preset similarity thresholds can be set according to different application scenes and to-be-tested data, and the preset similarity threshold is not particularly limited in this embodiment.
The following are exemplary: if the reply of the customer service robot to be verified to the data to be tested is completely consistent with the reply corresponding to the preset data to be tested; determining that the service verification of the customer service robot to be verified is passed.
And if the reply of the customer service robot to be verified to the data to be tested is inconsistent with the reply corresponding to the preset data to be tested, determining that the service verification of the customer service robot to be verified does not pass.
Whether the service verification of the customer service robot to be verified is passed or not is determined by judging whether the reply of the customer service robot to be verified to the data to be tested is consistent with the reply corresponding to the preset data to be tested or not, and the accuracy of the verification result of the service of the customer service robot to be verified is ensured.
In the service verification method provided by the embodiment, the total number of the acquisition problems is reasonably determined according to the number of the user problems of the production environment, so that the situation that the total number is too much or too little is avoided, and the acquisition proportion of each confidence coefficient is determined based on the distribution of the confidence coefficients of the production environment; determining the problem acquisition quantity of each confidence degree according to the total number of the acquisition problems and the acquisition proportion of each confidence degree, thereby simulating the condition of a production environment user, enabling the service verification of the robot to be close to the actual condition, and acquiring the to-be-tested data of each confidence degree from the user problems of the production environment based on the problem acquisition quantity of each confidence degree; inputting the data to be tested into the customer service robot to be verified; then, the reply of the customer service robot to be verified to the data to be tested is obtained; according to the reply of the customer service robot to be verified to the data to be tested and the reply corresponding to the preset data to be tested, the service of the customer service robot to be verified is verified, namely the service is verified before the version of the intelligent customer service robot is on line, the problem of the customer service robot can be found in time, the influence caused by service verification after the version is on line is avoided, the service quality of the customer service robot after the version is on line is ensured, and the user experience is improved.
Fig. 3 is a second flowchart of a service verification method according to an embodiment of the present invention, where an execution subject of this embodiment may be the terminal 201 in the embodiment shown in fig. 1, or may be a server, and this embodiment is not limited herein. As shown in fig. 3, the method includes:
s301: determining the total number of the collection problems according to the number of the user problems of the production environment;
s302: determining the acquisition proportion of each confidence coefficient based on the distribution of the confidence coefficients of the production environment;
s303: determining the problem collection quantity of each confidence degree according to the total number of the collection problems and the collection proportion of each confidence degree;
s304: and acquiring the data to be tested of each confidence degree from the user questions of the production environment based on the question acquisition quantity of each confidence degree.
S305: and inputting the data to be tested into the customer service robot to be verified.
S306: and obtaining the reply of the customer service robot to be verified to the data to be tested.
S307: and verifying the service of the customer service robot to be verified according to the reply and the preset reply corresponding to the data to be tested.
Steps S301 to S307 are the same as the implementation of steps S201 to S207, and are not described herein again.
S308: and debugging the customer service robot to be verified according to the service verification result of the customer service robot to be verified.
The method for debugging the customer service robot to be verified according to the service verification result of the customer service robot to be verified may be as follows:
counting the proportion of failed service verification of the customer service robot to be verified in the service verification result of the customer service robot to be verified, and comparing the proportion of failed service verification with the proportion of failed service verification in the past period; if the service verification fails in proportion larger than the previous period, debugging the customer service robot to be verified;
if the proportion of the service verification failure is not more than the proportion of the service verification failure in the past period, the customer service robot to be verified does not need to be debugged.
For example, in the embodiment of the present invention, taking an execution subject as an example, according to a service verification result of the verification customer service robot, debugging the verification customer service robot may be implemented in the following manner:
and debugging the verification customer service robot by a debugging module of the terminal according to the service verification result.
Alternatively, the first and second electrodes may be,
and sending a service verification result to the debugging equipment, wherein the service verification result is used for indicating the debugging equipment to debug the verification customer service robot.
Alternatively, the first and second electrodes may be,
and sending a service verification result to a preset person, wherein the service verification result is used for indicating the preset person to debug the verification customer service robot.
The service quality of the customer service robot after the customer service robot is on-line is further ensured by debugging the customer service robot to be verified according to the service verification result of the customer service robot to be verified.
S309: and adjusting the reply corresponding to the data to be tested according to the service verification result of the customer service robot to be verified and the confidence coefficient of the data to be tested.
For example, the manner of adjusting the reply corresponding to the data to be tested according to the service verification result of the customer service robot to be verified and the confidence of the data to be tested may be as follows:
and when the service verification result of the customer service robot to be verified is that the service verification fails and the confidence coefficient of the data to be tested is greater than a preset confidence coefficient threshold value, judging which of the reply of the customer service robot to be verified to the data to be tested and the reply corresponding to the preset data to be tested is more in line with the user problem, and if the reply result of the customer service robot to be verified is more in line with the user problem, adjusting the reply corresponding to the data to be tested.
Different preset confidence threshold values may be set according to different application scenarios, which is not particularly limited in this embodiment.
For example, taking the execution subject as the terminal, the adjustment of the user question data according to the service verification result of the customer service robot to be verified and the confidence of the user question may be implemented as follows:
and the adjusting module of the terminal adjusts the user problem data according to the service verification result and the confidence coefficient of the user problem.
Alternatively, the first and second electrodes may be,
and sending the confidence degrees of the service verification result and the user problem to the adjusting equipment, wherein the confidence degrees of the service verification result and the user problem are used for indicating the adjusting equipment to adjust the user problem data.
Alternatively, the first and second electrodes may be,
and sending the confidence degrees of the service verification result and the user problem to preset personnel, wherein the confidence degrees of the service verification result and the user problem are used for indicating the preset personnel to adjust the user problem data.
The user problem data are adjusted according to the service verification result of the customer service robot to be verified and the confidence coefficient of the user problem, so that the data to be tested are more reasonable, and the verification result is more accurate when service verification is carried out again.
Optionally, the embodiment of the present application may further include:
obtaining a verification passing proportion according to the service verification result of the customer service robot to be verified;
and determining a target confidence degree interval of the customer service robot to be verified based on the verification passing proportion.
Here, the verification pass ratio may be obtained with reference to step S309 described above.
Illustratively, the verification passing proportion expected by the user is 80%, in the interval with the confidence higher than 0.8, the verification passing proportion of the customer service robot to be verified can reach 80%, and the target confidence interval is 0.8-1. The interval can be fed back to related services as an option for setting the confidence of the customer service robot, so that the service quality and the service efficiency of the robot can be improved.
In this embodiment, the sequence of steps S308 and S309 is not specifically limited.
In the service verification method provided by the embodiment, the total number of the acquisition problems is reasonably determined according to the number of the user problems of the production environment, so that the situation that the total number is too much or too little is avoided, and the acquisition proportion of each confidence coefficient is determined based on the distribution of the confidence coefficients of the production environment; determining the problem acquisition quantity of each confidence degree according to the total number of the acquisition problems and the acquisition proportion of each confidence degree, thereby simulating the condition of a production environment user, enabling the service verification of the robot to be close to the actual condition, and acquiring the to-be-tested data of each confidence degree from the user problems of the production environment based on the problem acquisition quantity of each confidence degree; inputting the data to be tested into the customer service robot to be verified; then, the reply of the customer service robot to be verified to the data to be tested is obtained; according to the reply of the customer service robot to be verified to the data to be tested and the reply corresponding to the preset data to be tested, the service of the customer service robot to be verified is verified, the service verification is carried out before the version of the intelligent customer service robot is on line, the problem of the customer service robot can be found in time, the service quality of the customer service robot after the customer service robot is on line is ensured, and the user experience is improved. Meanwhile, the service quality of the customer service robot to be verified after the customer service robot is on-line is further ensured by debugging the customer service robot to be verified according to the service verification result of the customer service robot to be verified. The data to be tested is adjusted according to the service verification result of the customer service robot to be verified and the confidence coefficient of the data to be tested, so that the data to be tested is more reasonable, and the verification result is more accurate when service verification is performed again. The target confidence interval can be fed back to related services as options for setting the confidence of the customer service robot, so that the service quality and the service efficiency of the robot can be improved.
Fig. 4 is a schematic structural diagram of a service authentication apparatus provided for the embodiment of the present invention, and only shows portions related to the embodiment of the present invention for convenience of description. As shown in fig. 4, the service authentication apparatus 40 includes: a first determination module 401, an acquisition module 402, an input module 403, an obtaining module 404, and a verification module 405.
A first determining module 401, configured to determine a total number of the collection problems according to the number of the user problems of the production environment;
determining the acquisition proportion of each confidence coefficient based on the distribution of the confidence coefficients of the production environment;
determining the problem collection quantity of each confidence degree according to the total number of the collection problems and the collection proportion of each confidence degree;
an obtaining module 402, configured to obtain to-be-tested data of each confidence from user questions of the production environment based on the number of problem collections of each confidence;
an input module 403, configured to input the data to be tested into the customer service robot to be verified;
an obtaining module 404, configured to obtain a reply of the customer service robot to be verified to the to-be-tested data;
and the verification module 405 is configured to verify the service of the customer service robot to be verified according to the reply and a preset reply corresponding to the data to be tested.
The apparatus provided in the embodiment of the present invention may be used to implement the technical solutions of the above method embodiments, and the implementation principles and technical effects are similar, which are not described herein again.
Fig. 5 is a schematic structural diagram of a service verification apparatus according to an embodiment of the present invention. As shown in fig. 5, the service authentication apparatus 50 further includes, in addition to fig. 4: a second determination module 406, a debugging module 407, an adjustment module 408 and a processing module 409.
Optionally, the verification module 405 is specifically configured to:
judging whether the reply is consistent with the reply corresponding to the preset data to be tested;
and if the reply is consistent with the reply corresponding to the preset data to be tested, judging that the service verification of the customer service robot to be verified is passed.
Optionally, the first determining module 401 is further configured to:
determining the acquisition proportion of each problem type based on the distribution of the problem types of the production environment;
determining the collection number of each problem type according to the total number of the collection problems and the collection proportion of each problem type;
the obtaining module 402 obtains to-be-tested data of each confidence degree from the user questions of the production environment based on the question collection number of each confidence degree, including:
and acquiring to-be-tested data from the user questions of the production environment based on the question acquisition quantity of each confidence degree and the acquisition quantity of each question type.
Optionally, a second determining module 406, configured to, after the acquiring module 402 acquires the data to be tested,
and determining the customer service robot to be verified corresponding to the data to be tested according to the corresponding relation between the preset user question and the customer service robot.
Optionally, the debugging module 407 is configured to, after the verifying module 405 verifies the service of the customer service robot to be verified according to the reply and a preset reply corresponding to the data to be tested,
and debugging the customer service robot to be verified according to the service verification result of the customer service robot to be verified.
Optionally, the adjusting module 408 is configured to, after the verifying module 405 verifies the service of the customer service robot to be verified according to the reply and a preset reply corresponding to the data to be tested,
and adjusting the response corresponding to the data to be tested according to the service verification result of the customer service robot to be verified and the confidence coefficient of the user problem.
Optionally, the processing module 409 is configured to obtain a verification passing ratio according to a service verification result of the customer service robot to be verified;
and determining a target confidence degree interval of the customer service robot to be verified based on the verification passing proportion.
The apparatus provided in the embodiment of the present invention may be used to implement the technical solutions of the above method embodiments, and the implementation principles and technical effects are similar, which are not described herein again.
Fig. 6 is a schematic diagram of a hardware structure of a service authentication device according to an embodiment of the present invention. As shown in fig. 6, the service authentication apparatus 60 of the present embodiment includes: a processor 601 and a memory 602; wherein
A memory 602 for storing computer-executable instructions;
a processor 601 for executing computer executable instructions stored by the memory to perform the steps of the service authentication method in the above embodiments. Reference may be made in particular to the description relating to the method embodiments described above.
Alternatively, the memory 602 may be separate or integrated with the processor 601.
When the memory 602 is provided separately, the service authentication apparatus further includes a bus 603 for connecting the memory 602 and the processor 601.
An embodiment of the present invention further provides a computer-readable storage medium, where a computer execution instruction is stored in the computer-readable storage medium, and when a processor executes the computer execution instruction, the service verification method as described above is implemented.
In the embodiments provided in the present invention, it should be understood that the disclosed apparatus and method may be implemented in other ways. For example, the above-described apparatus embodiments are merely illustrative, and for example, the division of the modules is only one logical division, and other divisions may be realized in practice, for example, a plurality of modules may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or modules, and may be in an electrical, mechanical or other form.
The modules described as separate parts may or may not be physically separate, and parts displayed as modules may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the modules may be selected according to actual needs to implement the solution of the present embodiment.
In addition, functional modules in the embodiments of the present invention may be integrated into one processing unit, or each module may exist alone physically, or two or more modules are integrated into one unit. The unit formed by the modules can be realized in a hardware form, and can also be realized in a form of hardware and a software functional unit.
The integrated module implemented in the form of a software functional module may be stored in a computer-readable storage medium. The software functional module is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device) or a processor to execute some steps of the methods according to the embodiments of the present invention.
It should be understood that the Processor may be a Central Processing Unit (CPU), other general purpose Processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), etc. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. The steps of the service verification method disclosed in connection with the present invention may be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules in the processor.
The memory may comprise a high-speed RAM memory, and may further comprise a non-volatile storage NVM, such as at least one disk memory, and may also be a usb disk, a removable hard disk, a read-only memory, a magnetic or optical disk, etc.
The bus may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, an Extended ISA (Extended Industry Standard Architecture) bus, or the like. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, the buses in the figures of the present invention are not limited to only one bus or one type of bus.
The storage medium may be implemented by any type or combination of volatile or non-volatile memory devices, such as Static Random Access Memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic or optical disks. A storage media may be any available media that can be accessed by a general purpose or special purpose computer.
An exemplary storage medium is coupled to the processor such the processor can read information from, and write information to, the storage medium. Of course, the storage medium may also be integral to the processor. The processor and the storage medium may reside in an Application Specific Integrated Circuits (ASIC). Of course, the processor and the storage medium may reside as discrete components in an electronic device or host device.
Those of ordinary skill in the art will understand that: all or part of the steps for implementing the service authentication method embodiments described above may be performed by hardware associated with program instructions. The program may be stored in a computer-readable storage medium. When the program is executed, the program executes the steps of the service verification method embodiments; and the aforementioned storage medium includes: various media that can store program codes, such as ROM, RAM, magnetic or optical disks.
Finally, it should be noted that: the above embodiments are only used to illustrate the technical solution of the present invention, and not to limit the same; while the invention has been described in detail and with reference to the foregoing embodiments, it will be understood by those skilled in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some or all of the technical features may be equivalently replaced; and the modifications or the substitutions do not make the essence of the corresponding technical solutions depart from the scope of the technical solutions of the embodiments of the present invention.
Claims (10)
1. A method for service authentication, comprising:
determining the total number of the collection problems according to the number of the user problems of the production environment;
determining the acquisition proportion of each confidence coefficient based on the distribution of the confidence coefficients of the production environment;
determining the problem collection quantity of each confidence degree according to the total number of the collection problems and the collection proportion of each confidence degree;
acquiring to-be-tested data of each confidence degree from user problems of the production environment based on the problem acquisition quantity of each confidence degree;
inputting the data to be tested into a customer service robot to be verified;
obtaining the reply of the customer service robot to be verified to the data to be tested;
and verifying the service of the customer service robot to be verified according to the reply and the preset reply corresponding to the data to be tested.
2. The method of claim 1, further comprising:
determining the acquisition proportion of each problem type based on the distribution of the problem types of the production environment;
determining the collection number of each problem type according to the total number of the collection problems and the collection proportion of each problem type;
the acquiring number of the problems based on each confidence coefficient and the acquiring of the to-be-tested data of each confidence coefficient from the user problems of the production environment comprise:
and acquiring to-be-tested data from the user questions of the production environment based on the question acquisition quantity of each confidence degree and the acquisition quantity of each question type.
3. The method according to claim 1, wherein the verifying the service of the customer service robot to be verified according to the reply and a reply corresponding to the preset data to be tested comprises:
judging whether the reply is consistent with the reply corresponding to the preset data to be tested;
and if the reply is consistent with the reply corresponding to the preset data to be tested, judging that the service verification of the customer service robot to be verified is passed.
4. The method of claim 1, further comprising, after said obtaining data to be tested:
and determining the customer service robot to be verified corresponding to the data to be tested according to the corresponding relation between the preset user question and the customer service robot.
5. The method according to claim 1, after verifying the service of the customer service robot to be verified according to the reply and a reply corresponding to the preset data to be tested, further comprising:
and debugging the customer service robot to be verified according to the service verification result of the customer service robot to be verified.
6. The method according to claim 1, after verifying the service of the customer service robot to be verified according to the reply and a reply corresponding to the preset data to be tested, further comprising:
and adjusting the reply corresponding to the data to be tested according to the service verification result of the customer service robot to be verified and the confidence coefficient of the data to be tested.
7. The method of claim 1, further comprising:
obtaining a verification passing proportion according to the service verification result of the customer service robot to be verified;
and determining a target confidence degree interval of the customer service robot to be verified based on the verification passing proportion.
8. A service authentication apparatus, comprising:
the first determining module is used for determining the total number of the collected problems according to the number of the user problems of the production environment;
determining the acquisition proportion of each confidence coefficient based on the distribution of the confidence coefficients of the production environment;
determining the problem collection quantity of each confidence degree according to the total number of the collection problems and the collection proportion of each confidence degree;
the acquisition module is used for acquiring to-be-tested data of each confidence degree from user problems of the production environment based on the problem acquisition quantity of each confidence degree;
the input module is used for inputting the data to be tested into the customer service robot to be verified;
the obtaining module is used for obtaining the reply of the customer service robot to be verified to the data to be tested;
and the verification module is used for verifying the service of the customer service robot to be verified according to the reply and the preset reply corresponding to the data to be tested.
9. A service authentication device, comprising: at least one processor and memory;
the memory stores computer-executable instructions;
the at least one processor executing the memory-stored computer-executable instructions cause the at least one processor to perform the service authentication method of any of claims 1 to 7.
10. A computer-readable storage medium having computer-executable instructions stored thereon which, when executed by a processor, implement the service authentication method as claimed in any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911157503.9A CN110928788B (en) | 2019-11-22 | 2019-11-22 | Service verification method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911157503.9A CN110928788B (en) | 2019-11-22 | 2019-11-22 | Service verification method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110928788A true CN110928788A (en) | 2020-03-27 |
| CN110928788B CN110928788B (en) | 2023-09-19 |
Family
ID=69850799
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911157503.9A Active CN110928788B (en) | 2019-11-22 | 2019-11-22 | Service verification method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110928788B (en) |
Citations (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8990639B1 (en) * | 2012-05-31 | 2015-03-24 | Amazon Technologies, Inc. | Automatic testing and remediation based on confidence indicators |
| US20150095296A1 (en) * | 2013-09-27 | 2015-04-02 | Ebay Inc. | Method and apparatus for a data confidence index |
| WO2015120968A1 (en) * | 2014-02-13 | 2015-08-20 | Amadeus S.A.S. | Increasing search result validity |
| US20160070641A1 (en) * | 2014-09-08 | 2016-03-10 | Ab lnitio Technology LLC | Data-driven testing framework |
| US20170289139A1 (en) * | 2014-12-18 | 2017-10-05 | Alibaba Group Holding Limited | Device verification method and apparatus |
| US20170300401A1 (en) * | 2016-04-14 | 2017-10-19 | Vmware, Inc. | Methods and systems that identify problems in applications |
| US20180157581A1 (en) * | 2016-12-06 | 2018-06-07 | International Business Machines Corporation | Automated system testing in a complex software environment |
| CN108874823A (en) * | 2017-05-12 | 2018-11-23 | 阿里巴巴集团控股有限公司 | The implementation method and device of intelligent customer service |
| US20190005013A1 (en) * | 2017-06-28 | 2019-01-03 | Beijing Baidu Netcom Science And Technology Co., Ltd. | Conversation system-building method and apparatus based on artificial intelligence, device and computer-readable storage medium |
| CN109254917A (en) * | 2018-08-31 | 2019-01-22 | 百度在线网络技术（北京）有限公司 | Method and apparatus for test data flow system |
| CN109542780A (en) * | 2018-11-13 | 2019-03-29 | 东软集团股份有限公司 | Test method, test equipment and the storage medium of natural language processing application |
| CN109684198A (en) * | 2018-11-13 | 2019-04-26 | 平安科技（深圳）有限公司 | Data capture method to be tested, device, medium, electronic equipment |
| CN109753436A (en) * | 2019-01-09 | 2019-05-14 | 中国联合网络通信集团有限公司 | Test method, device and the storage medium of intelligent customer service system |
| CN110109829A (en) * | 2019-04-15 | 2019-08-09 | 福建天晴在线互动科技有限公司 | Intelligent dialogue automates method of calibration, storage medium |
-
2019
- 2019-11-22 CN CN201911157503.9A patent/CN110928788B/en active Active
Patent Citations (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8990639B1 (en) * | 2012-05-31 | 2015-03-24 | Amazon Technologies, Inc. | Automatic testing and remediation based on confidence indicators |
| US20150095296A1 (en) * | 2013-09-27 | 2015-04-02 | Ebay Inc. | Method and apparatus for a data confidence index |
| WO2015120968A1 (en) * | 2014-02-13 | 2015-08-20 | Amadeus S.A.S. | Increasing search result validity |
| US20160070641A1 (en) * | 2014-09-08 | 2016-03-10 | Ab lnitio Technology LLC | Data-driven testing framework |
| US20170289139A1 (en) * | 2014-12-18 | 2017-10-05 | Alibaba Group Holding Limited | Device verification method and apparatus |
| US20170300401A1 (en) * | 2016-04-14 | 2017-10-19 | Vmware, Inc. | Methods and systems that identify problems in applications |
| US20180157581A1 (en) * | 2016-12-06 | 2018-06-07 | International Business Machines Corporation | Automated system testing in a complex software environment |
| CN108874823A (en) * | 2017-05-12 | 2018-11-23 | 阿里巴巴集团控股有限公司 | The implementation method and device of intelligent customer service |
| US20190005013A1 (en) * | 2017-06-28 | 2019-01-03 | Beijing Baidu Netcom Science And Technology Co., Ltd. | Conversation system-building method and apparatus based on artificial intelligence, device and computer-readable storage medium |
| CN109254917A (en) * | 2018-08-31 | 2019-01-22 | 百度在线网络技术（北京）有限公司 | Method and apparatus for test data flow system |
| CN109542780A (en) * | 2018-11-13 | 2019-03-29 | 东软集团股份有限公司 | Test method, test equipment and the storage medium of natural language processing application |
| CN109684198A (en) * | 2018-11-13 | 2019-04-26 | 平安科技（深圳）有限公司 | Data capture method to be tested, device, medium, electronic equipment |
| CN109753436A (en) * | 2019-01-09 | 2019-05-14 | 中国联合网络通信集团有限公司 | Test method, device and the storage medium of intelligent customer service system |
| CN110109829A (en) * | 2019-04-15 | 2019-08-09 | 福建天晴在线互动科技有限公司 | Intelligent dialogue automates method of calibration, storage medium |
Non-Patent Citations (5)
| Title |
|---|
| _HOUZHI: "按照比例将大数据按照类型比例，随机提取测试数据", pages 1 * |
| 刘迎春;朱旭;谢年春;李佳;: "基于数据挖掘的专业可信回答者个性化推荐――以Stack Overflow问答社区为例", no. 05 * |
| 廖胜兰;殷实;陈小平;张波;欧阳昱;张衡;: "面向电力业务对话系统的意图识别数据集", no. 09 * |
| 张剑;童言;徐明迪;秦涛;: "轻量级主机数据采集与实时异常事件检测方法研究", no. 04 * |
| 李聪;马丽;梁昌勇;: "DFM-IA:面向B2C电子商务的多源用户兴趣数据采集机制", no. 01 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110928788B (en) | 2023-09-19 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107885656B (en) | Automatic product algorithm testing method and application server | |
| CN108932663B (en) | Method and device for simulating matched transaction | |
| CN111221726A (en) | Test data generation method and device, storage medium and intelligent equipment | |
| CN113342669A (en) | Verification method and device for shortening convergence time of chip code coverage rate | |
| CN111931047A (en) | Artificial intelligence-based black product account detection method and related device | |
| CN109377103B (en) | Evaluation method and device for course recommendation of learning platform | |
| CN110675249A (en) | Matching method, device, server and storage medium for network lending | |
| CN109472457B (en) | Loan application online reviewing method and terminal equipment | |
| CN111626498A (en) | Equipment operation state prediction method, device, equipment and storage medium | |
| CN110070383B (en) | Abnormal user identification method and device based on big data analysis | |
| CN110928788A (en) | Service verification method and device | |
| CN111340062A (en) | Mapping relation determining method and device | |
| CN109308660B (en) | Credit assessment scoring model evaluation method, apparatus, device and storage medium | |
| CN115712571A (en) | Interactive service test device, interactive service test device, computer equipment and storage medium | |
| CN110765006A (en) | Flow testing method and device, computer readable storage medium and electronic device | |
| CN115391168A (en) | Sandbox testing method, sandbox testing device, sandbox testing equipment and storage medium | |
| CN111367732A (en) | Memory application grade prediction method, system, terminal and storage medium | |
| CN108510071B (en) | Data feature extraction method and device and computer readable storage medium | |
| CN110942238A (en) | Course recommendation device and method | |
| CN110688302B (en) | Automatic field library testing method and device | |
| US11397662B2 (en) | Method for debugging computer program, device employing method, and storage medium | |
| CN117132283A (en) | Identification method, apparatus, computer device, storage medium, and program product | |
| CN116029540A (en) | Risk assessment method and device for front-end code development system | |
| CN115374009A (en) | Interface automation test method, device, equipment, medium and program product | |
| CN114003494A (en) | Automatic test method and device for data model and electronic equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |