CN116233311A - Automatic outbound testing method, device, computer equipment and storage medium - Google Patents
Automatic outbound testing method, device, computer equipment and storage medium Download PDFInfo
- Publication number
- CN116233311A CN116233311A CN202310506120.8A CN202310506120A CN116233311A CN 116233311 A CN116233311 A CN 116233311A CN 202310506120 A CN202310506120 A CN 202310506120A CN 116233311 A CN116233311 A CN 116233311A
- Authority
- CN
- China
- Prior art keywords
- dialogue
- test
- node
- intention
- outbound
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000012360 testing method Methods 0.000 title claims abstract description 175
- 238000003860 storage Methods 0.000 title claims abstract description 11
- 238000004519 manufacturing process Methods 0.000 claims description 22
- 238000011084 recovery Methods 0.000 claims description 18
- 238000004458 analytical method Methods 0.000 claims description 12
- 238000004364 calculation method Methods 0.000 claims description 12
- 238000004590 computer program Methods 0.000 claims description 7
- 238000010998 test method Methods 0.000 claims description 5
- 238000009826 distribution Methods 0.000 claims description 4
- 238000013433 optimization analysis Methods 0.000 claims description 2
- 238000000034 method Methods 0.000 abstract description 11
- 238000010586 diagram Methods 0.000 description 8
- 230000006870 function Effects 0.000 description 8
- 230000008569 process Effects 0.000 description 4
- 230000006872 improvement Effects 0.000 description 3
- 238000012545 processing Methods 0.000 description 3
- 239000000047 product Substances 0.000 description 3
- 238000011161 development Methods 0.000 description 2
- 230000003993 interaction Effects 0.000 description 2
- 239000013589 supplement Substances 0.000 description 2
- 238000004891 communication Methods 0.000 description 1
- 238000010219 correlation analysis Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 230000001934 delay Effects 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000011156 evaluation Methods 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000001737 promoting effect Effects 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 238000012216 screening Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04M—TELEPHONIC COMMUNICATION
- H04M3/00—Automatic or semi-automatic exchanges
- H04M3/22—Arrangements for supervision, monitoring or testing
- H04M3/26—Arrangements for supervision, monitoring or testing with means for applying test signals or for measuring
- H04M3/28—Automatic routine testing ; Fault testing; Installation testing; Test methods, test equipment or test arrangements therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/30—Arrangements for executing machine instructions, e.g. instruction decode
- G06F9/38—Concurrent instruction execution, e.g. pipeline or look ahead
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5061—Partitioning or combining of resources
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04M—TELEPHONIC COMMUNICATION
- H04M3/00—Automatic or semi-automatic exchanges
- H04M3/42—Systems providing special services or facilities to subscribers
- H04M3/487—Arrangements for providing information services, e.g. recorded voice services or time announcements
- H04M3/493—Interactive information services, e.g. directory enquiries ; Arrangements therefor, e.g. interactive voice response [IVR] systems or voice portals
- H04M3/4936—Speech interaction details
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/50—Indexing scheme relating to G06F9/50
- G06F2209/5017—Task decomposition
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/50—Indexing scheme relating to G06F9/50
- G06F2209/5018—Thread allocation
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02P—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN THE PRODUCTION OR PROCESSING OF GOODS
- Y02P90/00—Enabling technologies with a potential contribution to greenhouse gas [GHG] emissions mitigation
- Y02P90/30—Computing systems specially adapted for manufacturing
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Signal Processing (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Human Computer Interaction (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
- Debugging And Monitoring (AREA)
- Testing Or Calibration Of Command Recording Devices (AREA)
- Testing Of Devices, Machine Parts, Or Other Structures Thereof (AREA)
Abstract
The invention relates to the technical field of testing, and provides an automatic outbound testing method, an automatic outbound testing device, computer equipment and a storage medium, wherein the method comprises the following steps: obtaining test data of an outbound system through dialogue flow test, wherein the test data comprises a plurality of dialogue nodes and corresponding dialogue sentences in the dialogue flow; calculating the number of dialogue nodes to determine the number of intention recognition tasks performed for dialogue sentences of each dialogue node; dividing each intention recognition task into a plurality of parallel task groups according to the processor configuration of the test equipment, and executing each intention recognition task in parallel by generating a corresponding number of threads so as to obtain an intention recognition result of each dialogue node; and obtaining a test result according to the intention recognition result. The test tasks of a plurality of nodes can be executed in parallel, and the test efficiency is greatly improved.
Description
Technical Field
The present invention relates to the field of automated testing, and in particular, to an automated outbound testing method, apparatus, computer device, and storage medium.
Background
In the prior banking system, an automatic outbound system is used for promoting business operation, but the outbound system has the problem of low efficiency because sentences are selected from a preset voice library for feedback to customers through artificial intelligent recognition, and therefore, the defect of the automatic outbound system needs to be tested through high-efficiency testing, so that developers can improve better. The existing intelligent outbound system test technical scheme is that a tester writes a script for a node of a dialogue flow to be tested, configures intent of expected matching of the dialogue and the dialogue, then performs unit test, the test is not enough in test set, the test is not comprehensive enough, automatic test and targeted analysis of a process in a production link are lacking, and because of the numerous intentions of voice, the current test mode is low in efficiency, long in service time and greatly delays development and improvement progress.
Disclosure of Invention
In a first aspect, the present application provides an automated outbound testing method, comprising:
obtaining test data of an outbound system through dialogue flow test, wherein the test data comprises a plurality of dialogue nodes and corresponding dialogue sentences in the dialogue flow;
calculating the number of dialogue nodes to determine the number of intention recognition tasks performed for dialogue sentences of each dialogue node;
dividing all the intention recognition tasks into a plurality of parallel task groups according to the processor configuration of the test equipment, and executing each intention recognition task in parallel by generating a corresponding number of threads so as to obtain intention recognition results of each dialogue node;
and obtaining a test result according to the intention recognition result.
Further, according to the processor configuration of the test device, dividing all the intention recognition tasks into a plurality of parallel task groups, and generating a corresponding number of threads, wherein the method comprises the following steps:
and obtaining the core number of a processor configured by the test equipment, calculating the task group number according to the core number, and generating threads the same as the task group number.
Further, the dialog flow test includes a test phase test;
and when the test is in the test stage, obtaining the intention recognition result of each dialogue node, wherein the intention recognition result comprises the following steps:
obtaining a conversation pool of each conversation node, and identifying the intention quantity of each conversation node;
and respectively calculating the reply accuracy and the spam rate of each dialogue node, and the reply accuracy and the spam rate of the whole dialogue flow according to the conversation pool and the intention quantity.
Further, the calculating the reply accuracy and the spam rate of each dialogue node includes:
obtaining the intention quantity, the accurate quantity of the reply voice operation and the spam quantity of the dialogue node from a test log;
the calculation expression of the recovery accuracy rate is as follows:
wherein y is the recovery accuracy, M is the intention amount, and W is the recovery accuracy;
the calculation expression of the bottom rate is as follows:
wherein Z is the bottom-holding rate and X is the bottom-holding amount.
Further, when the test is performed in the test stage, the step of identifying the result according to the intention to obtain a test result includes:
and respectively comparing the calculated recovery accuracy and the spam rate with preset corresponding target values to determine whether the outbound system of the current version can work on line.
Further, the dialog flow test includes a production phase test;
when the production phase test is performed, the step of obtaining test data of the outbound system comprises the following steps:
working data in actual production work is used as test data of an outbound system;
the obtaining the intention recognition result of each dialogue node comprises the following steps:
and calculating the node access rate, the node hang-up rate and the node retry rate of each dialogue node, determining a key node according to preset indexes of the node access rate, the node hang-up rate and the node retry rate, determining a key path according to the key node, and carrying out the dialogue flow test according to the key path.
Further, when the test is performed in the production stage, the step of identifying the result according to the intention to obtain a test result includes:
each sentence of dialogue between the outbound system and the client is marked with a corresponding client label and an intention label, and the client label and the intention label are subjected to optimization analysis according to normal distribution so as to determine the support and the confidence of the client on the outbound system of the current version;
and calculating the lifting degree according to the support degree and the confidence degree, and evaluating the outbound system of the current version according to the lifting degree.
In a second aspect, the present application further provides an automated outbound testing device, comprising:
the test module is used for obtaining test data of the outbound system through dialogue flow test, wherein the test data comprises a plurality of dialogue nodes and corresponding dialogue sentences in the dialogue flow;
the planning module is used for calculating the number of the dialogue nodes to determine the number of intention recognition tasks for dialogue sentences of each dialogue node;
the parallel module is used for dividing all the intention recognition tasks into a plurality of parallel task groups according to the processor configuration of the test equipment, and executing each intention recognition task in parallel by generating a corresponding number of threads so as to obtain intention recognition results of each dialogue node;
and the analysis module is used for identifying the result according to the intention and obtaining a test result.
In a third aspect, the present application also provides a computer device comprising a processor and a memory, the memory storing a computer program which, when run on the processor, performs the automated outbound test method.
In a fourth aspect, a readable storage medium stores a computer program which, when run on a processor, performs the automated outbound testing method.
The invention discloses an automatic outbound testing method, an automatic outbound testing device, computer equipment and a storage medium, wherein the method comprises the following steps: obtaining test data of an outbound system through dialogue flow test, wherein the test data comprises a plurality of dialogue nodes and corresponding dialogue sentences in the dialogue flow; calculating the number of dialogue nodes to determine the number of intention recognition tasks performed for dialogue sentences of each dialogue node; dividing each intention recognition task into a plurality of parallel task groups according to the processor configuration of the test equipment, and executing each intention recognition task in parallel by generating a corresponding number of threads so as to obtain an intention recognition result of each dialogue node; and obtaining a test result according to the intention recognition result. The testing tasks of a plurality of nodes can be executed in parallel, the testing efficiency is greatly improved, an automatic and standardized intelligent outbound testing scheme is realized, the workload of manual testing of the business process can be reduced to the greatest extent, and the labor input cost of enterprises is saved.
Drawings
In order to more clearly illustrate the technical solutions of the present invention, the drawings that are required for the embodiments will be briefly described, it being understood that the following drawings only illustrate some embodiments of the present invention and therefore should not be considered as limiting the scope of the present invention. Like elements are numbered alike in the various figures.
Fig. 1 is a schematic flow chart of an automated outbound testing method according to an embodiment of the present application;
FIG. 2 is a schematic diagram of a parallel test method according to an embodiment of the present application;
FIG. 3 illustrates a schematic diagram of a session pool in accordance with an embodiment of the present application;
fig. 4 shows a schematic structural diagram of an automated outbound testing device according to an embodiment of the present application.
Detailed Description
The following description of the embodiments of the present invention will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present invention, but not all embodiments.
The components of the embodiments of the present invention generally described and illustrated in the figures herein may be arranged and designed in a wide variety of different configurations. Thus, the following detailed description of the embodiments of the invention, as presented in the figures, is not intended to limit the scope of the invention, as claimed, but is merely representative of selected embodiments of the invention. All other embodiments, which can be made by a person skilled in the art without making any inventive effort, are intended to be within the scope of the present invention.
The terms "comprises," "comprising," "including," or any other variation thereof, are intended to cover a specific feature, number, step, operation, element, component, or combination of the foregoing, which may be used in various embodiments of the present invention, and are not intended to first exclude the presence of or increase the likelihood of one or more other features, numbers, steps, operations, elements, components, or combinations of the foregoing.
Furthermore, the terms "first," "second," "third," and the like are used merely to distinguish between descriptions and should not be construed as indicating or implying relative importance.
Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which various embodiments of the invention belong. The terms (such as those defined in commonly used dictionaries) will be interpreted as having a meaning that is the same as the context of the relevant art and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein in connection with the various embodiments of the invention.
The technical scheme of the application is applied to the test of the outbound system of the bank system, and can be understood that when the outbound system is tested, the outbound system is tested through the set dialogue content, so that a plurality of dialogue flows are generated, namely test data, wherein each dialogue flow comprises a plurality of dialogue nodes, namely one dialogue communication formed by the outbound system and a reply utterance of a client (or a tester). Depending on the term, each dialog node may contain multiple intents, and each intent may be replied to by a different phone, so that coverage test needs to be performed for each different phone to test whether the reply condition of each phone meets the prediction. According to the method and the device, each dialogue node is synchronously tested in a parallel computing mode, so that the test can be completed rapidly.
The technical scheme of the application is described in the following specific embodiments.
Example 1
As shown in fig. 1, the automated outbound testing method of the present embodiment includes:
step S100, test data of the outbound system is obtained through dialogue flow test, wherein the test data comprises a plurality of dialogue nodes and corresponding dialogue sentences in the dialogue flow.
The dialogue flow test is that a tester performs dialogue test through a set reply sentence and an outbound system, so as to form test data mainly comprising the dialogue flow, wherein a plurality of dialogue nodes are arranged in the dialogue flow.
It should be noted that the tests in this embodiment have different stages, one is a test in the test stage, that is, an internal test when the outbound system is not formally on-line with the production system, and the other is a production test, that is, a test after the outbound system formally starts on-line operation.
The test data in the test stage are designed by testers, and the dialogue nodes are subjected to dialogue to test so as to obtain dialogue flows.
In the production stage, the conversation flow is used as test data when the outbound system actually works in daily work.
It will be appreciated that although the source of test data at different stages is different, the outbound system workflow and manner are the same and the dialog nodes are the same, so that in different stages the resulting test data is similar except for the specific content.
Step S200, calculating the number of dialogue nodes to determine the number of intention recognition tasks performed for dialogue sentences of each dialogue node.
It will be appreciated that in either test phase, the test data is a dialog flow, and that there are a plurality of dialog nodes in the dialog flow, which are the entire dialog flow between the system and the test statement. Because the test sentence has different intentions each time, the outbound system needs to answer according to the intentions, a plurality of corresponding speaking options can be provided for the intentions in each dialogue node, and according to the actual situation, the speaking options can be customized by a developer in selectable speaking. The intent recognition has to be done for these different utterances and sentences produced in the dialog throughout the test flow.
For example, in a dialog flow, there are 100 dialog nodes, each dialog node has 10 intents on average, each intent corresponds to 20 dialogs, each node supplements 10 custom dialogs, and as a result, intent recognition needs to be performed 100 times (10×20+10) =21000 times. Assuming that it takes 60 minutes ms to identify one time, in serial mode of operation, one test takes 21 minutes.
It can be seen that a shorter dialogue requires a large amount of intent recognition, which is a great burden on the test, and that more test data is exposed to the actual test and production environment, and the actual test time is longer.
The embodiment reduces the consumption time by parallel computing, and therefore, the number of dialogue nodes needs to be counted first, and after the number of dialogue nodes is obtained, the number of intention recognition tasks required to be performed in the whole dialogue flow can be calculated according to each node.
And step S300, dividing all the intention recognition tasks into a plurality of parallel task groups according to the processor configuration of the test equipment, and executing each intention recognition task in parallel by generating a corresponding number of threads so as to obtain the intention recognition result of each dialogue node.
It can be appreciated that after determining each intention recognition task in the dialog flow, the intention recognition tasks are independent and have no context association with each other, so that the processing speed can be increased and the processing time can be reduced through parallel processing. For example, 10000 intention recognition tasks are split into 10 1000 intention recognition tasks, each 1000 intention recognition task is further split into 10 100 intention recognition tasks, and each 100 intention recognition task is further split into 2 50 intention recognition subtasks.
Parallel computing is accomplished by generating multiple sub-threads that are executed simultaneously by the sub-threads, which also requires that the processor of the computer device can support, since it is intended to identify tasks that are not CPU intensive, the thread size is configured to be 2 x the processor core number +1.
Specifically, parallel running can be performed through Fork/Join, as shown in fig. 2, a large task in the figure can be understood as needing to perform 10000 intention recognition tasks. Since 10000 intention identifications are performed serially if no parallel computation is performed. The large person generates a sub-thread through the fork function, meanwhile splits the large task into a plurality of small tasks and distributes the tasks to the sub-thread, so that parallel calculation can be performed, if the split sub-task is still too large, the fork function can be used for further splitting, after all tasks are executed, the tasks are combined through a join command, and the task execution result is integrated, so that the whole parallel calculation process is finished.
After the intention recognition is finished, various data such as the intention amount, the reply call accuracy amount, the spam amount, the single-node call number, the number of connected calls, the node hang-up amount and the like are obtained. The data can be used as an intention recognition result for corresponding analysis to obtain a specific test result.
And step S400, according to the intention recognition result, obtaining a test result.
In the foregoing description, the test of the present embodiment is divided into a test stage test and a production stage test, and the test results obtained by the two different stage tests are different.
Firstly, when the test is carried out in the test stage, obtaining an intention recognition node of each dialogue node, namely, obtaining a dialogue pool of each dialogue node, and recognizing the intention quantity of each dialogue node;
and calculating the reply accuracy rate and the spam rate of each dialogue node, and the reply accuracy rate and the spam rate of the whole dialogue flow according to the conversation pool and the intention quantity.
Wherein, the speaking pool refers to a set consisting of an intention speaking and a custom speaking. As shown in FIG. 3, the intent is system inventory maintenance, with each intent maintaining a complete set of utterances. The custom call is a call supplement function provided for testers, and when the intention call used in the node does not meet the current test, the custom call can be re-customized, so that the test is more sufficient.
The recovery accuracy rate refers to the statement accuracy rate of recovery of the outbound system, and the spam rate refers to the rate that the outbound system does not recognize accurately and can only adopt a default value for recovery. It can be appreciated that the higher the recovery accuracy, the lower the spam rate, the better the stability of the outbound system.
Specifically, the calculation expression of the recovery accuracy rate is:
wherein y is the recovery accuracy, M is the intended amount, and W is the recovery accuracy.
The calculation expression of the bottom rate is as follows:
wherein Z is the bottom-holding rate and X is the bottom-holding amount.
The above calculation is that the reply accuracy and the spam rate of a dialogue node, and for the whole dialogue flow, assuming that the total intention is n, the corresponding reply accuracy and spam rate are as follows:
in the formula, i represents what kind of intention.
For the recovery accuracy and the spam rate, certain standards are provided, and the standards are configured by a tester according to actual conditions, so that after the recovery accuracy and the spam rate are calculated, the calculated value can be compared with a preset target value, if the recovery accuracy is higher than the target value and the spam rate is lower than the target value, the outbound system of the current version can be determined to work on line, otherwise, the outbound system is optimized continuously.
When the session is tested in the production stage, because the data of the outbound task can show a trend in the production environment, that is, the session flow is not accessed by all nodes, and the weight and the frequency of the node access are different, all nodes of the session flow are not required to be tested, and only key nodes of the session flow are required to be tested.
In order to determine key nodes, node access rates, node hang-up rates and node retry rates of all session nodes are calculated, the key nodes are determined according to preset values of the node access rates, the node hang-up rates and the node retry rates, key paths are determined according to the key nodes, and the session flow test is carried out according to the key paths.
The node access rate can be calculated by the number of single-node calls and the number of connected calls, and the node hang-up rate can be calculated by the number of node hang-up and the number of connected calls.
The node retry rate refers to that the node performs a session replay and performs a new round of interaction when the node does not get a response from the client or does not recognize the intention of the client after the client answers. Suppose that the repetition number at the ith occurrence of a node is K i Then the weights of the nodes are summed again as squares of the repetition times, i.e. the retry weights of the nodes are
Suppose that the retry weight of each node is J i The retry degree of a node is the sum of the squares of the retry weights of the node divided by the squares of the retry weights of all nodes multiplied by 100, i.e., the node retry rate is +.>
。
Similarly, the node access rate, the node hang-up rate and the node retry rate have preset indexes, and corresponding key dialogue nodes can be selected according to the preset indexes.
For example, the node with the node access rate of 20%, the node with the node hang-up rate of 30% and the dialogue node with the node retry rate of 30% are used as key nodes, and corresponding key paths can be obtained according to the key nodes, wherein the node with the node access rate of 20%, the node with the node hang-up rate of 30% and the node retry rate of 30% are preset indexes for screening the key nodes.
After the critical path is determined, the data set to be tested and the corresponding session pool are determined, so that the corresponding intention recognition task is also determined.
It will be appreciated that since the test is a production phase test, after the problem is found, the developer will make a corresponding adjustment, and the test is a production phase test, and the most important requirement is to evaluate whether the current version is improved compared with the previous version, so that the evaluation manner by the test phase alone is insufficient, and further judging whether the improved version is better than the previous version is also required.
Therefore, in this embodiment, during the test in the production stage, the outbound system is marked with corresponding client labels and intention labels for each call and each sentence of the client, the client labels and the intention labels are marked according to the content results of the interaction, and the labels have weight values set respectively. Specifically, the larger the customer label value obtained by the customer, the better the quality of the customer is; the larger the intent label value that the customer gets, the more interesting the customer is to the product.
Meanwhile, in actual production, the client label and the intention label conform to normal distribution:
by normal distribution probability density function calculation, the μ value and σ value different between publications can be calculated in this embodiment, and from these values, the following two kinds of analysis can be made.
Customer tag weight analysis: version with large mu value shows that the overall weight of the client label is larger, and the client quality is higher; the version with small sigma value shows that the whole client label is concentrated, the curve is high and thin, so that the higher the mu value is, the smaller the sigma value is, and the whole quality of the client can be improved.
Customer intent weight analysis: version with large mu value shows that the overall intention of the client is better and the intention of the client is high; the version with small sigma value shows that the whole client intention is concentrated, the curve is relatively high and thin, so that the version with smaller sigma value is higher as the mu value is higher, and the intention of the client can be improved.
In addition, the correlation analysis between the client intention and the client quality is also the most important, and the version with the highest correlation between the client intention and the client quality is determined by the following analysis, and the version with the highest correlation is the preferred version.
In this embodiment, the support degree and the confidence degree are calculated, the lifting degree is calculated according to the support degree and the confidence degree, and the outbound system of the current version is evaluated according to the lifting degree.
Support degree analysis:
let a & B represent the number of times that the good customer appears and the customer's intent belongs to the good intent.
N represents the total number of conversations. The calculation logic of the support S is as follows:
it will be appreciated that the higher the degree of support, the higher the probability of intent.
Confidence analysis:
let F (A & B) represent the probability that the occurrence belongs to the premium customer and to the preference, and F (A) represents the number of times the premium customer is present. From this, the confidence level can be calculated, the logic for calculating confidence level C is as follows:
and (3) lifting degree analysis:
let S (a & B) represent the degree of support that appears to be of the premium customer and of the preference, S (a) S (B) represent the product of the probability of the premium customer and the probability of the preference. The calculation logic for the lift L is as follows:
the degree of improvement L can represent the score of the automatic outbound system of the current version, after the degree of improvement L is calculated, the version with the L value larger than 1 is used for representing that the high-quality client and the preference have strong association in the version, and meanwhile, as the high-quality client can promote the preference, when the calculated L value is larger, the association of the high-quality client and the preference is more obvious, and the version with the obvious association of the high-quality client and the preference is a better version. Therefore, the version with the larger L value is used preferentially, and meanwhile, it can be understood that after the developer performs one-time updating, whether the updated version is better than the previous version or not can be known by calculating L, so that the outbound system can be updated and changed better.
According to the automatic outbound test method, the time consumption in the test process is reduced in a parallel computing mode, the test is divided into a test stage test and a production stage test, corresponding changes are made according to different requirements of the two stage tests, whether the current version can be produced or not can be rapidly and accurately identified through the test stage test, whether the updated version effect is better or not can be distinguished through the production stage test, and the test and development efficiency is greatly improved.
Example 2
As shown in fig. 4, the present application further provides an automated outbound testing device, including:
the test module 10 is configured to obtain test data of the outbound system through a session flow test, where the test data includes a plurality of session nodes and corresponding session sentences in the session flow;
a planning module 20 for calculating the number of dialogue nodes to determine the number of intention recognition tasks performed for dialogue sentences of each dialogue node;
the parallel module 30 is configured to divide all the intention recognition tasks into a plurality of parallel task groups according to the processor configuration of the test device, and execute each intention recognition task in parallel by generating a corresponding number of threads, so as to obtain an intention recognition result of each dialogue node;
and the analysis module 40 is used for identifying the result according to the intention and obtaining a test result.
The present application also provides a computer device comprising a processor and a memory, the memory storing a computer program which, when run on the processor, performs the automated outbound testing method.
The present application also provides a readable storage medium storing a computer program which when run on a processor performs the automated outbound testing method.
In the several embodiments provided in this application, it should be understood that the disclosed apparatus and method may be implemented in other manners as well. The apparatus embodiments described above are merely illustrative, for example, of the flow diagrams and block diagrams in the figures, which illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
In addition, functional modules or units in various embodiments of the invention may be integrated together to form a single part, or the modules may exist alone, or two or more modules may be integrated to form a single part.
The functions, if implemented in the form of software functional modules and sold or used as a stand-alone product, may be stored in a computer-readable storage medium. Based on such understanding, the technical solution of the present invention may be embodied essentially or in a part contributing to the prior art or in a part of the technical solution in the form of a software product stored in a storage medium, comprising several instructions for causing a computer device (which may be a smart phone, a personal computer, a server, a network device, etc.) to perform all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), a magnetic disk, or an optical disk, or other various media capable of storing program codes.
The foregoing is merely illustrative of the present invention, and the present invention is not limited thereto, and any person skilled in the art will readily recognize that variations or substitutions are within the scope of the present invention.
Claims (10)
1. An automated outbound testing method comprising:
obtaining test data of an outbound system through dialogue flow test, wherein the test data comprises a plurality of dialogue nodes and corresponding dialogue sentences in the dialogue flow;
calculating the number of dialogue nodes to determine the number of intention recognition tasks performed for dialogue sentences of each dialogue node;
dividing all the intention recognition tasks into a plurality of parallel task groups according to the processor configuration of the test equipment, and executing each intention recognition task in parallel by generating a corresponding number of threads so as to obtain intention recognition results of each dialogue node;
and obtaining a test result according to the intention recognition result.
2. The automated outbound testing method of claim 1, wherein the grouping all of the intent recognition tasks into a plurality of parallel task groups according to a processor configuration of a testing device, by generating a corresponding number of threads, comprises:
and obtaining the core number of a processor configured by the test equipment, calculating the task group number according to the core number, and generating threads the same as the task group number.
3. The automated outbound test method of claim 1, wherein the dialog flow test comprises a test phase test;
and when the test is in the test stage, obtaining the intention recognition result of each dialogue node, wherein the intention recognition result comprises the following steps:
obtaining a conversation pool of each conversation node, and identifying the intention quantity of each conversation node;
and respectively calculating the reply accuracy and the spam rate of each dialogue node, and the reply accuracy and the spam rate of the whole dialogue flow according to the conversation pool and the intention quantity.
4. The automated outbound testing method of claim 3, wherein the separately calculating the reply accuracy and spam rate for each dialog node comprises:
obtaining the intention quantity, the accurate quantity of the reply voice operation and the spam quantity of the dialogue node from a test log;
the calculation expression of the recovery accuracy rate is as follows:
wherein y is the recovery accuracy, M is the intention amount, and W is the recovery accuracy;
the calculation expression of the bottom rate is as follows:
wherein Z is the bottom-holding rate and X is the bottom-holding amount.
5. The automated outbound testing method of claim 3, wherein the identifying a result based on the intent when tested in the testing phase, comprises:
and respectively comparing the calculated recovery accuracy and the spam rate with preset corresponding target values to determine whether the outbound system of the current version can work on line.
6. The automated outbound test method of claim 1, wherein the dialog flow test comprises a production phase test;
when the production phase test is performed, the step of obtaining test data of the outbound system comprises the following steps:
working data in actual production work is used as test data of an outbound system;
the obtaining the intention recognition result of each dialogue node comprises the following steps:
and calculating the node access rate, the node hang-up rate and the node retry rate of each dialogue node, determining a key node according to preset indexes of the node access rate, the node hang-up rate and the node retry rate, determining a key path according to the key node, and carrying out the dialogue flow test according to the key path.
7. The automated outbound testing method of claim 6, wherein the identifying a result from the intent when tested at the production stage, comprises:
each sentence of dialogue between the outbound system and the client is marked with a corresponding client label and an intention label, and the client label and the intention label are subjected to optimization analysis according to normal distribution so as to determine the support and the confidence of the client on the outbound system of the current version;
and calculating the lifting degree according to the support degree and the confidence degree, and evaluating the outbound system of the current version according to the lifting degree.
8. An automated outbound testing device, comprising:
the test module is used for obtaining test data of the outbound system through dialogue flow test, wherein the test data comprises a plurality of dialogue nodes and corresponding dialogue sentences in the dialogue flow;
the planning module is used for calculating the number of the dialogue nodes to determine the number of intention recognition tasks for dialogue sentences of each dialogue node;
the parallel module is used for dividing all the intention recognition tasks into a plurality of parallel task groups according to the processor configuration of the test equipment, and executing each intention recognition task in parallel by generating a corresponding number of threads so as to obtain intention recognition results of each dialogue node;
and the analysis module is used for identifying the result according to the intention and obtaining a test result.
9. A computer device comprising a processor and a memory, the memory storing a computer program that, when run on the processor, performs the automated outbound testing method of any one of claims 1 to 7.
10. A readable storage medium, characterized in that it stores a computer program which, when run on a processor, performs the automated outbound testing method according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310506120.8A CN116233311B (en) | 2023-05-08 | 2023-05-08 | Automatic outbound testing method, device, computer equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310506120.8A CN116233311B (en) | 2023-05-08 | 2023-05-08 | Automatic outbound testing method, device, computer equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN116233311A true CN116233311A (en) | 2023-06-06 |
| CN116233311B CN116233311B (en) | 2023-07-14 |
Family
ID=86579098
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310506120.8A Active CN116233311B (en) | 2023-05-08 | 2023-05-08 | Automatic outbound testing method, device, computer equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116233311B (en) |
Citations (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111008130A (en) * | 2019-11-28 | 2020-04-14 | 中国银行股份有限公司 | Intelligent question-answering system testing method and device |
| CN111177307A (en) * | 2019-11-22 | 2020-05-19 | 深圳壹账通智能科技有限公司 | Test scheme and system based on semantic understanding similarity threshold configuration |
| CN112202978A (en) * | 2020-08-24 | 2021-01-08 | 维知科技张家口有限责任公司 | Intelligent outbound call system, method, computer system and storage medium |
| CN113065850A (en) * | 2021-04-02 | 2021-07-02 | 京东数科海益信息科技有限公司 | Speech testing method and device for intelligent outbound robot |
| CN113158690A (en) * | 2021-03-15 | 2021-07-23 | 京东数科海益信息科技有限公司 | Testing method and device for conversation robot |
| CN113259201A (en) * | 2021-06-17 | 2021-08-13 | 中国农业银行股份有限公司 | Link performance testing method and device, server and storage medium |
| CN113268370A (en) * | 2021-05-11 | 2021-08-17 | 西安交通大学 | Root cause alarm analysis method, system, equipment and storage medium |
| CN113572900A (en) * | 2021-07-22 | 2021-10-29 | 深圳追一科技有限公司 | Outbound test method, device, computer equipment and computer readable storage medium |
| CN113746990A (en) * | 2021-09-03 | 2021-12-03 | 科讯嘉联信息技术有限公司 | Outbound prediction system and method for switching from AI man-machine interaction to man-machine interaction |
| US20220129556A1 (en) * | 2020-10-28 | 2022-04-28 | Facebook, Inc. | Systems and Methods for Implementing Smart Assistant Systems |
| CN114466106A (en) * | 2021-12-23 | 2022-05-10 | 北京声智科技有限公司 | Test data generation method, device, equipment and medium of outbound system |
| CN114613350A (en) * | 2022-03-12 | 2022-06-10 | 云知声智能科技股份有限公司 | Test method, test device, electronic equipment and storage medium |
| CN114625878A (en) * | 2022-03-22 | 2022-06-14 | 中国平安人寿保险股份有限公司 | Intention identification method, interactive system and equipment |
| CN115134466A (en) * | 2022-06-07 | 2022-09-30 | 马上消费金融股份有限公司 | Intention recognition method and device and electronic equipment |
-
2023
- 2023-05-08 CN CN202310506120.8A patent/CN116233311B/en active Active
Patent Citations (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111177307A (en) * | 2019-11-22 | 2020-05-19 | 深圳壹账通智能科技有限公司 | Test scheme and system based on semantic understanding similarity threshold configuration |
| CN111008130A (en) * | 2019-11-28 | 2020-04-14 | 中国银行股份有限公司 | Intelligent question-answering system testing method and device |
| CN112202978A (en) * | 2020-08-24 | 2021-01-08 | 维知科技张家口有限责任公司 | Intelligent outbound call system, method, computer system and storage medium |
| US20220129556A1 (en) * | 2020-10-28 | 2022-04-28 | Facebook, Inc. | Systems and Methods for Implementing Smart Assistant Systems |
| CN113158690A (en) * | 2021-03-15 | 2021-07-23 | 京东数科海益信息科技有限公司 | Testing method and device for conversation robot |
| CN113065850A (en) * | 2021-04-02 | 2021-07-02 | 京东数科海益信息科技有限公司 | Speech testing method and device for intelligent outbound robot |
| CN113268370A (en) * | 2021-05-11 | 2021-08-17 | 西安交通大学 | Root cause alarm analysis method, system, equipment and storage medium |
| CN113259201A (en) * | 2021-06-17 | 2021-08-13 | 中国农业银行股份有限公司 | Link performance testing method and device, server and storage medium |
| CN113572900A (en) * | 2021-07-22 | 2021-10-29 | 深圳追一科技有限公司 | Outbound test method, device, computer equipment and computer readable storage medium |
| CN113746990A (en) * | 2021-09-03 | 2021-12-03 | 科讯嘉联信息技术有限公司 | Outbound prediction system and method for switching from AI man-machine interaction to man-machine interaction |
| CN114466106A (en) * | 2021-12-23 | 2022-05-10 | 北京声智科技有限公司 | Test data generation method, device, equipment and medium of outbound system |
| CN114613350A (en) * | 2022-03-12 | 2022-06-10 | 云知声智能科技股份有限公司 | Test method, test device, electronic equipment and storage medium |
| CN114625878A (en) * | 2022-03-22 | 2022-06-14 | 中国平安人寿保险股份有限公司 | Intention identification method, interactive system and equipment |
| CN115134466A (en) * | 2022-06-07 | 2022-09-30 | 马上消费金融股份有限公司 | Intention recognition method and device and electronic equipment |
Also Published As
| Publication number | Publication date |
|---|---|
| CN116233311B (en) | 2023-07-14 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US9883026B2 (en) | Method and apparatus for facilitating speech application testing | |
| EP3413221A1 (en) | Risk assessment method and system | |
| JP4901738B2 (en) | Machine learning | |
| Culp et al. | ada: An r package for stochastic boosting | |
| WO2020005725A1 (en) | Knowledge-driven dialog support conversation system | |
| US7295981B1 (en) | Method for building a natural language understanding model for a spoken dialog system | |
| WO2019017462A1 (en) | Satisfaction estimation model learning device, satisfaction estimation device, satisfaction estimation model learning method, satisfaction estimation method, and program | |
| CN106408325A (en) | User consumption behavior prediction analysis method based on user payment information and system | |
| CN114186697A (en) | Method and device for generating and applying deep learning model based on deep learning framework | |
| CN110556098B (en) | Voice recognition result testing method and device, computer equipment and medium | |
| CN111738017A (en) | Intention identification method, device, equipment and storage medium | |
| CN113488023B (en) | Language identification model construction method and language identification method | |
| CN116233311B (en) | Automatic outbound testing method, device, computer equipment and storage medium | |
| CN115836304A (en) | Updating trained voice robots using example-based voice robot development techniques | |
| CN111159378A (en) | Method and device for classifying problem description information | |
| CN107423140B (en) | Return code identification method and device | |
| US20220277149A1 (en) | Statistical language models for simulating communication sessions | |
| CN107229487B (en) | Code processing method and device | |
| CN114443506A (en) | Method and device for testing artificial intelligence model | |
| CN114860617A (en) | Intelligent pressure testing method and system | |
| CN114595318A (en) | Customer service reply quality evaluation method and system | |
| CN110852854B (en) | Method for generating quantitative gain model and method for evaluating risk control strategy | |
| CN114356982A (en) | Marketing compliance checking method and device, computer equipment and storage medium | |
| KR102220106B1 (en) | Method for correcting speech recognized sentence | |
| CN114548307A (en) | Classification model training method and device, and classification method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |