CN111128161A - Data processing method and device and electronic equipment - Google Patents
Data processing method and device and electronic equipment Download PDFInfo
- Publication number
- CN111128161A CN111128161A CN201911338336.8A CN201911338336A CN111128161A CN 111128161 A CN111128161 A CN 111128161A CN 201911338336 A CN201911338336 A CN 201911338336A CN 111128161 A CN111128161 A CN 111128161A
- Authority
- CN
- China
- Prior art keywords
- intention
- intention recognition
- accuracy
- test case
- program
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 19
- 238000012360 testing method Methods 0.000 claims abstract description 237
- 238000012545 processing Methods 0.000 claims abstract description 14
- 238000000034 method Methods 0.000 claims description 25
- 238000004364 calculation method Methods 0.000 claims description 12
- 238000004088 simulation Methods 0.000 claims description 11
- 238000012544 monitoring process Methods 0.000 description 10
- 238000011161 development Methods 0.000 description 3
- 238000010801 machine learning Methods 0.000 description 3
- 238000005457 optimization Methods 0.000 description 3
- 238000012795 verification Methods 0.000 description 3
- 238000010586 diagram Methods 0.000 description 2
- 230000000737 periodic effect Effects 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000005611 electricity Effects 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 238000002372 labelling Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 239000002699 waste material Substances 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G10L15/00—Speech recognition
- G10L15/22—Procedures used during a speech recognition process, e.g. man-machine dialogue
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N20/00—Machine learning
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G10L15/00—Speech recognition
- G10L15/08—Speech classification or search
- G10L15/18—Speech classification or search using natural language modelling
- G10L15/1822—Parsing for meaning understanding
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Software Systems (AREA)
- Computational Linguistics (AREA)
- Health & Medical Sciences (AREA)
- Audiology, Speech & Language Pathology (AREA)
- Human Computer Interaction (AREA)
- Acoustics & Sound (AREA)
- Multimedia (AREA)
- Artificial Intelligence (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Medical Informatics (AREA)
- Evolutionary Computation (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Mathematical Physics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Stored Programmes (AREA)
Abstract
The invention provides a data processing method, a data processing device and electronic equipment, wherein a test case and a pre-marked first intention recognition result corresponding to the test case are obtained; calling a pre-obtained intention recognition program to perform intention recognition on the test case to obtain a second intention recognition result corresponding to the test case; comparing the first intention identification result with the second intention identification result to obtain a first intention identification accuracy; and if the first intention identification accuracy is greater than a preset threshold value, determining an intention identification accuracy test result of the intention identification program. The invention can automatically calculate the intention recognition accuracy of the intention recognition program, and further can continuously optimize the performance of the intention recognition service according to the accuracy, so that the intention recognition program can serve the user faster and better.
Description
Technical Field
The present invention relates to the field of data processing, and in particular, to a data processing method and apparatus, and an electronic device.
Background
The intention recognition service is used for recognizing the intention of a text after voice recognition is carried out on the voice of a user in a conversation between a robot and the user so as to obtain the intention of the user, and is an important function in an intelligent voice robot system. The accuracy of the intention identification service can be used for reflecting the performance of the intention identification service, the intention identification service is considered to have better performance if the accuracy is high, and the intention identification service is considered to have poorer performance if the accuracy is low, so that a method for determining the accuracy of the intention identification service is urgently needed, and the performance of the intention identification service can be continuously optimized according to the accuracy, so that the intention identification service can be served to users more quickly and better.
Disclosure of Invention
In view of the above, the present invention provides a data processing method, an apparatus and an electronic device, so as to solve the problem of urgently needing a method for determining the accuracy of the intention identification service.
In order to solve the technical problems, the invention adopts the following technical scheme:
a method of data processing, comprising:
obtaining a test case and a first intention recognition result which is marked in advance and corresponds to the test case;
calling a pre-obtained intention recognition program to perform intention recognition on the test case to obtain a second intention recognition result corresponding to the test case;
comparing the first intention identification result with the second intention identification result to obtain a first intention identification accuracy;
if the first intention identification accuracy rate is larger than a preset threshold value, determining an intention identification accuracy rate test result of the intention identification program;
and running the intention recognition software in a target running environment, and testing the intention recognition accuracy of the intention recognition program in the target running environment.
Optionally, the obtaining the test case includes:
acquiring a preset test case, and acquiring a user type and/or an application scene corresponding to the test case;
and classifying the test cases according to the user types and/or the application scenes.
Optionally, invoking a pre-obtained intention recognition program to perform intention recognition on the test case to obtain a second intention recognition result corresponding to the test case, where the method includes:
generating an intention recognition simulation environment matched with a target running environment of the intention recognition program;
and under the intention recognition simulation environment, performing intention recognition on the test case by using the intention recognition program to obtain a second intention recognition result corresponding to the test case.
Optionally, the comparing the first intention recognition result and the second intention recognition result to obtain a first intention recognition accuracy includes:
determining the number of test cases with the same corresponding first intention recognition result and second intention recognition result based on the first intention recognition result and the second intention recognition result corresponding to the test cases;
acquiring the total number of the test cases;
and calculating to obtain the first intention identification accuracy according to the number and the total number.
Optionally, testing the intention recognition accuracy of the intention recognition program in the target operating environment includes:
in the target operation environment, performing intention recognition on the test case by using the intention recognition program to obtain a third intention recognition result corresponding to the test case;
comparing the first intention identification result with the third intention identification result to obtain a second intention identification accuracy;
if the difference value between the first intention identification accuracy rate and the second intention identification accuracy rate is within a preset difference value range, determining that the on-line intention test accuracy rate corresponding to the intention identification program passes;
and if the difference value between the first intention identification accuracy rate and the second intention identification accuracy rate is not within a preset difference value range, determining that the on-line intention test accuracy rate corresponding to the intention identification program fails.
Optionally, the method further comprises:
if the first intention identification accuracy rate is not greater than a preset threshold value, the intention identification program is adjusted, so that the intention identification accuracy rate of the adjusted intention identification program is greater than the preset threshold value.
A data processing apparatus comprising:
the data acquisition module is used for acquiring a test case and a first intention recognition result corresponding to the pre-marked test case;
the intention identification module is used for calling a pre-acquired intention identification program to perform intention identification on the test case to obtain a second intention identification result corresponding to the test case;
the accuracy calculation module is used for comparing the first intention recognition result with the second intention recognition result to obtain a first intention recognition accuracy;
the test result determining module is used for determining an intention recognition accuracy test result of the intention recognition program if the first intention recognition accuracy is greater than a preset threshold;
and the online testing module is used for operating the intention recognition software in a target operating environment and testing the intention recognition accuracy of the intention recognition program in the target operating environment.
Optionally, when the data obtaining module is configured to obtain the test case, the data obtaining module is specifically configured to:
the method comprises the steps of obtaining a preset test case, obtaining a user type and/or an application scene corresponding to the test case, and classifying the test case according to the user type and/or the application scene.
Optionally, the intention identifying module is configured to call a pre-obtained intention identifying program to perform intention identification on the test case, and when a second intention identifying result corresponding to the test case is obtained, specifically configured to:
and generating an intention recognition simulation environment matched with a target operation environment of the intention recognition program, and performing intention recognition on the test case by using the intention recognition program under the intention recognition simulation environment to obtain a second intention recognition result corresponding to the test case.
Optionally, the accuracy calculation module includes:
the quantity determining submodule is used for determining the quantity of the test cases with the same corresponding first intention recognition result and second intention recognition result based on the first intention recognition result and the second intention recognition result corresponding to the test cases;
the quantity obtaining submodule is used for obtaining the total quantity of the test cases;
and the accuracy calculation submodule is used for calculating the first intention identification accuracy according to the number and the total number.
Optionally, the online test module includes:
the intention identification submodule is used for carrying out intention identification on the test case by using the intention identification program in the target operation environment to obtain a third intention identification result corresponding to the test case;
the data calculation submodule is used for comparing the first intention recognition result with the third intention recognition result to obtain a second intention recognition accuracy;
a first determining submodule, configured to determine that an online intent test accuracy corresponding to the intent recognition program passes if a difference between the first intent recognition accuracy and the second intent recognition accuracy is within a preset difference range;
and the second determination submodule is used for determining that the on-line intention test accuracy corresponding to the intention identification program fails if the difference value between the first intention identification accuracy and the second intention identification accuracy is not within a preset difference value range.
Optionally, the method further comprises:
and the service adjusting module is used for adjusting the intention identification program if the first intention identification accuracy is not greater than a preset threshold value, so that the intention identification accuracy of the adjusted intention identification program is greater than the preset threshold value.
An electronic device, comprising: a memory and a processor;
wherein the memory is used for storing programs;
the processor calls a program and is used to:
obtaining a test case and a first intention recognition result which is marked in advance and corresponds to the test case;
calling a pre-obtained intention recognition program to perform intention recognition on the test case to obtain a second intention recognition result corresponding to the test case;
comparing the first intention identification result with the second intention identification result to obtain a first intention identification accuracy;
if the first intention identification accuracy rate is larger than a preset threshold value, determining an intention identification accuracy rate test result of the intention identification program;
and running the intention recognition software in a target running environment, and testing the intention recognition accuracy of the intention recognition program in the target running environment.
Compared with the prior art, the invention has the following beneficial effects:
the invention provides a data processing method, a data processing device and electronic equipment, wherein a test case and a pre-marked first intention recognition result corresponding to the test case are obtained; calling a pre-obtained intention recognition program to perform intention recognition on the test case to obtain a second intention recognition result corresponding to the test case; comparing the first intention identification result with the second intention identification result to obtain a first intention identification accuracy; and if the first intention identification accuracy is greater than a preset threshold value, determining an intention identification accuracy test result of the intention identification program. The invention can automatically calculate the intention recognition accuracy of the intention recognition program, and further can continuously optimize the performance of the intention recognition service according to the accuracy, so that the intention recognition program can serve the user faster and better.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the provided drawings without creative efforts.
Fig. 1 is a flowchart of a data processing method according to an embodiment of the present invention;
FIG. 2 is a flow chart of another data processing method according to an embodiment of the present invention;
FIG. 3 is a flowchart of a method of another data processing method according to an embodiment of the present invention;
fig. 4 is a schematic structural diagram of a data processing apparatus according to an embodiment of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
The inventor finds that the accuracy of the intention recognition program (i.e., the intention recognition service) can be used for reflecting the performance of the intention recognition program, the intention recognition program has better performance when the accuracy is high, and the intention recognition program has poorer performance when the accuracy is low, so that a method for determining the accuracy of the intention recognition program is needed, and the performance of the intention recognition program can be continuously optimized according to the accuracy, so that the intention recognition program can be served to a user more quickly and better. Therefore, when the intention recognition accuracy of the intention recognition program is determined, the intention recognition accuracy can be calculated by adopting a manual marking mode, and therefore, the method needs to be executed to monitor the intention recognition accuracy of the newly online intention recognition program every time the intention recognition program is online after being updated. In order to solve the problem of manpower waste caused by a mode of manually calculating an intention recognition accuracy, an embodiment of the present invention sets up an automation engine for executing a data processing method, and with reference to fig. 1, the data processing method may include:
s11, obtaining the test case and a first intention recognition result corresponding to the pre-marked test case.
In practical application, a plurality of test cases need to be obtained first, and the test cases can be texts obtained by historical ASR recognition (speech recognition) (the call records of human-computer interaction are actually closer to the real scene of the test, and therefore are suitable for being used as test cases), and can also be some test cases added manually, such as manually written test cases. For example, the test case may be "do you like, ask you be xxx? That is, you are now recommending a financing APP for you, do you have an interest in understanding? Some, please say, this APP is mainly related to the periodic aspect … … ".
Wherein, the above-mentioned "do you ask you for xxx? "," now recommend a financing APP for you, do you have an interest in understanding? The intelligent robot is a preset dialect which can be used by the intelligent robot, and the dialect used by the intelligent robot is different under different application scenes. For example, "do you ask you for xxx? "yes, now recommend a financing APP for you, do you have an interest in understanding? "" some, please say "" this APP relates primarily to the periodic aspect … … ". Or, "do you ask you for xxx? "not, good, that disturbed".
In general, test cases may include:
specific names of test cases (e.g., sales cases), specific contents of test cases, user types (including age, gender, whether new users are), and so forth.
To facilitate management of the test cases, the test users may be classified so as to manage the test cases by classification, and specifically, step S11 may include:
the method comprises the steps of obtaining a preset test case and a user type and/or an application scene corresponding to the test case, and classifying the test case according to the user type and/or the application scene.
The preset test cases and the types of users have been described in the above embodiments, and referring to the above specific contents, the application scenarios may be e-selling scenarios, call-receiving scenarios, and the like.
After the user types are obtained, the test cases can be classified according to the user types, for example, the test cases of the same gender are classified into one type, in addition, the test cases can also be classified only according to application scenes, for example, the test cases in the electricity marketing scene are classified into one type, and the test cases in the collection-forcing scene are classified into one type. In addition, the user type and the application scenario can be used simultaneously to classify the test cases, that is, the test cases belonging to the same user type and the same application scenario are classified into one type. The intention recognition operation can be executed together at the later stage of the test cases which are classified into one type, so that the intention recognition accuracy test of the user type and/or the application scene is facilitated.
After the test cases are determined, manually labeling the first intention recognition results corresponding to the test cases in advance, namely manually determining the first intention recognition result of each test case, wherein the first intention recognition result is used for evaluating whether the recognition result of the intention recognition program is accurate or not.
And S12, calling a pre-acquired intention recognition program to perform intention recognition on the test case, and obtaining a second intention recognition result corresponding to the test case.
In this embodiment, in order to prevent a problem from occurring in an online calling environment after a test case sample set is updated or an intention diagram is newly added, in practical application, before the intention identification program is not online, a test environment (i.e., an intention identification simulation environment) matched with a target operating environment of the intention identification program is established, and before an intelligent robot updates online each time, the intention identification program is used in an automation engine to identify an intention of a test case to obtain a second intention identification result corresponding to the test case. The intention recognition program performs intention recognition by using a machine learning method.
And S13, comparing the first intention identification result and the second intention identification result to obtain a first intention identification accuracy. The method and the device can automatically calculate the intention recognition accuracy of the test case, avoid concept confusion of the intention recognition accuracy, and the offline test result is called the first intention recognition accuracy and the online test result is called the second intention recognition accuracy.
In practical applications, referring to fig. 2, the process of obtaining the first intention recognition accuracy may include:
s21, determining the number of test cases with the same corresponding first intention recognition result and second intention recognition result based on the first intention recognition result and the second intention recognition result corresponding to the test cases.
Specifically, the first intention recognition result and the second intention recognition result corresponding to each test case are sequentially compared, if the first intention recognition result and the second intention recognition result are the same, the intention recognition result of the intention recognition program is considered to be accurate, the accumulated number is increased by one, if the intention recognition result of the intention recognition program is not the same, the intention recognition result of the intention recognition program is considered to be inaccurate, the accumulated number is not increased by one, the accumulated number is accumulated from zero, and the finally accumulated number is the number counted in the step S21.
And S22, acquiring the total number of the test cases.
And S23, calculating to obtain the first intention identification accuracy according to the number and the total number.
In practical applications, for example, the total number may be 200, and the number of the recognition results that are intended to be accurate is 10, and the accuracy is 95%.
And S14, if the first intention identification accuracy is larger than a preset threshold value, determining an intention identification accuracy test result of the intention identification program. If the first intention identification accuracy is greater than a preset threshold (manually set according to a specific test scene), which indicates that the identification accuracy of the intention identification program is high, the intention identification program passes verification at this moment, and an intention identification accuracy test result is output, wherein the intention identification accuracy test result can be a sentence such as an intention identification accuracy test pass, a test pass, and the like. After the test of the intention identifying program is passed, the intention identifying program can be on-line, namely, operated in an on-line environment (target operation environment).
In practical application, a preset threshold value which passes the test verification is configured in advance, and if the first intention identification accuracy rate is lower than the preset threshold value, the intention identification program needs to be readjusted until the intention identification accuracy rate of the adjusted intention identification program is greater than the preset threshold value. The specific adjustment process may be to manually modify some parameter values in the intention identification program, then verify the intention identification accuracy of the intention identification program again, and if not, continue to modify some parameter values in the intention identification program.
And S15, operating the intention recognition software in a target operating environment, and testing the intention recognition accuracy of the intention recognition program in the target operating environment.
If the first intention recognition accuracy is above a preset threshold, the online is allowed, that is, the operation in the target operation environment (actual online environment) is allowed. After the intention identifying program is online, the intention identifying program also needs to be tested online, and specifically, referring to fig. 3, the online testing process may include:
and S31, in the target running environment, using the intention recognition program to perform intention recognition on the test case to obtain a third intention recognition result corresponding to the test case.
Step S31 is similar to step S12, with reference to step S12, except that one is off-line and one is on-line.
And S32, comparing the first intention identification result with the third intention identification result to obtain a second intention identification accuracy.
Step S32 is similar to step S13, and reference is made to step S13 for details.
S33, judging whether the difference value between the first intention identification accuracy rate and the second intention identification accuracy rate is within a preset difference value range or not; if yes, go to step S34; if not, go to step S35.
And S34, determining that the accuracy rate of the online intention test corresponding to the intention identification program passes.
And S35, determining that the accuracy rate of the online intention test corresponding to the intention identification program fails.
In practical application, the intention identification program is realized based on machine learning, and the process of the machine learning has uncertainty, so that the accuracy rates of the intention identification under the line and on the line which are possibly finally obtained are different for the same batch of test cases, but the accuracy difference is not too large, a preset difference range is set in the embodiment, and the preset difference range can be 2% -6%.
If the difference value is within the preset range, the intention recognition program is considered to pass the online intention test; if the difference is not within the preset range, the intention identification program is considered to fail the online intention test.
In this embodiment, after the intelligent robot provided with the intention recognition program is online, the automation engine performs regression verification again online for the online environment, and in the formal online environment, the automation engine is used to execute an automation monitoring sample, and if the obtained intention recognition accuracy is not much different from the test environment (at present), it is determined that there is no problem in the online environment. And then carrying out an outbound strategy. If the difference is more, alarm prompt and readjustment intention identification program can be carried out. After the intelligent robot is on line, manual work can still check the manual call records, and then the intelligent robot is added into an automation engine after being marked, so that the test case is continuously completed and supplemented.
In the embodiment, a test case and a pre-labeled first intention recognition result corresponding to the test case are obtained; calling a pre-obtained intention recognition program to perform intention recognition on the test case to obtain a second intention recognition result corresponding to the test case; comparing the first intention identification result with the second intention identification result to obtain a first intention identification accuracy; and if the first intention identification accuracy is greater than a preset threshold value, determining an intention identification accuracy test result of the intention identification program. The invention can automatically calculate the intention recognition accuracy of the intention recognition program, and further can continuously optimize the performance of the intention recognition service according to the accuracy, so that the intention recognition program can serve the user faster and better. By the method and the device, the intention identification accuracy of the test case can be automatically calculated, the intention identification accuracy of the test case does not need to be manually calculated, and manpower is saved.
In addition, the building of the automatic test engine can help to improve the test regression efficiency, and the whole solution of automatically acquiring the test case can save the development time of the monitoring sample and save a large amount of labor and time cost. The online monitoring system can realize full-automatic regression execution aiming at optimization online and online routine monitoring, improves the labor and time cost of regression, ensures the online stability, can efficiently and timely process found problems, reduces online problems and avoids damage to users.
Optionally, on the basis of the embodiment of the data processing method, another embodiment of the present invention provides a data processing apparatus, and with reference to fig. 4, the data processing apparatus may include:
the data acquisition module 11 is configured to acquire a test case and a pre-labeled first intention recognition result corresponding to the test case;
the intention identification module 12 is configured to call a pre-obtained intention identification program to perform intention identification on the test case, so as to obtain a second intention identification result corresponding to the test case;
an accuracy calculation module 13, configured to compare the first intention recognition result and the second intention recognition result to obtain a first intention recognition accuracy;
a test result determining module 14, configured to determine an intention recognition accuracy test result of the intention recognition program if the first intention recognition accuracy is greater than a preset threshold;
and the online testing module 15 is configured to operate the intention recognition software in a target operating environment, and test the intention recognition accuracy of the intention recognition program in the target operating environment.
Further, when the data obtaining module is used for obtaining the test case, the data obtaining module is specifically used for:
the method comprises the steps of obtaining a preset test case, obtaining a user type and/or an application scene corresponding to the test case, and classifying the test case according to the user type and/or the application scene.
Further, the intention identification module is configured to call a pre-obtained intention identification program to perform intention identification on the test case, and when a second intention identification result corresponding to the test case is obtained, specifically configured to:
and generating an intention recognition simulation environment matched with a target operation environment of the intention recognition program, and performing intention recognition on the test case by using the intention recognition program under the intention recognition simulation environment to obtain a second intention recognition result corresponding to the test case.
Further, the accuracy calculation module comprises:
the quantity determining submodule is used for determining the quantity of the test cases with the same corresponding first intention recognition result and second intention recognition result based on the first intention recognition result and the second intention recognition result corresponding to the test cases;
the quantity obtaining submodule is used for obtaining the total quantity of the test cases;
and the accuracy calculation submodule is used for calculating the first intention identification accuracy according to the number and the total number.
Further, the inline test module includes:
the intention identification submodule is used for carrying out intention identification on the test case by using the intention identification program in the target operation environment to obtain a third intention identification result corresponding to the test case;
the data calculation submodule is used for comparing the first intention recognition result with the third intention recognition result to obtain a second intention recognition accuracy;
a first determining submodule, configured to determine that an online intent test accuracy corresponding to the intent recognition program passes if a difference between the first intent recognition accuracy and the second intent recognition accuracy is within a preset difference range;
and the second determination submodule is used for determining that the on-line intention test accuracy corresponding to the intention identification program fails if the difference value between the first intention identification accuracy and the second intention identification accuracy is not within a preset difference value range.
Further, still include:
and the service adjusting module is used for adjusting the intention identification program if the first intention identification accuracy is not greater than a preset threshold value, so that the intention identification accuracy of the adjusted intention identification program is greater than the preset threshold value.
In the embodiment, a test case and a pre-labeled first intention recognition result corresponding to the test case are obtained; calling a pre-obtained intention recognition program to perform intention recognition on the test case to obtain a second intention recognition result corresponding to the test case; comparing the first intention identification result with the second intention identification result to obtain a first intention identification accuracy; and if the first intention identification accuracy is greater than a preset threshold value, determining an intention identification accuracy test result of the intention identification program. The invention can automatically calculate the intention recognition accuracy of the intention recognition program, and further can continuously optimize the performance of the intention recognition service according to the accuracy, so that the intention recognition program can serve the user faster and better. By the method and the device, the intention identification accuracy of the test case can be automatically calculated, the intention identification accuracy of the test case does not need to be manually calculated, and manpower is saved.
In addition, the building of the automatic test engine can help to improve the test regression efficiency, and the whole solution of automatically acquiring the test case can save the development time of the monitoring sample and save a large amount of labor and time cost. The online monitoring system can realize full-automatic regression execution aiming at optimization online and online routine monitoring, improves the labor and time cost of regression, ensures the online stability, can efficiently and timely process found problems, reduces online problems and avoids damage to users.
It should be noted that, for the working processes of each module and sub-module in this embodiment, please refer to the corresponding description in the above embodiments, which is not described herein again.
Optionally, on the basis of the embodiments of the data processing method and apparatus, another embodiment of the present invention provides an electronic device, including: a memory and a processor;
wherein the memory is used for storing programs;
the processor calls a program and is used to:
obtaining a test case and a first intention recognition result which is marked in advance and corresponds to the test case;
calling a pre-obtained intention recognition program to perform intention recognition on the test case to obtain a second intention recognition result corresponding to the test case;
comparing the first intention identification result with the second intention identification result to obtain a first intention identification accuracy;
if the first intention identification accuracy rate is larger than a preset threshold value, determining an intention identification accuracy rate test result of the intention identification program;
and running the intention recognition software in a target running environment, and testing the intention recognition accuracy of the intention recognition program in the target running environment.
In the embodiment, a test case and a pre-labeled first intention recognition result corresponding to the test case are obtained; calling a pre-obtained intention recognition program to perform intention recognition on the test case to obtain a second intention recognition result corresponding to the test case; comparing the first intention identification result with the second intention identification result to obtain a first intention identification accuracy; and if the first intention identification accuracy is greater than a preset threshold value, determining an intention identification accuracy test result of the intention identification program. The invention can automatically calculate the intention recognition accuracy of the intention recognition program, and further can continuously optimize the performance of the intention recognition service according to the accuracy, so that the intention recognition program can serve the user faster and better. By the method and the device, the intention identification accuracy of the test case can be automatically calculated, the intention identification accuracy of the test case does not need to be manually calculated, and manpower is saved.
In addition, the building of the automatic test engine can help to improve the test regression efficiency, and the whole solution of automatically acquiring the test case can save the development time of the monitoring sample and save a large amount of labor and time cost. The online monitoring system can realize full-automatic regression execution aiming at optimization online and online routine monitoring, improves the labor and time cost of regression, ensures the online stability, can efficiently and timely process found problems, reduces online problems and avoids damage to users.
The previous description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the invention. Thus, the present invention is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims (13)
1. A data processing method, comprising:
obtaining a test case and a first intention recognition result which is marked in advance and corresponds to the test case;
calling a pre-obtained intention recognition program to perform intention recognition on the test case to obtain a second intention recognition result corresponding to the test case;
comparing the first intention identification result with the second intention identification result to obtain a first intention identification accuracy;
if the first intention identification accuracy rate is larger than a preset threshold value, determining an intention identification accuracy rate test result of the intention identification program;
and running the intention recognition software in a target running environment, and testing the intention recognition accuracy of the intention recognition program in the target running environment.
2. The data processing method of claim 1, wherein the obtaining the test case comprises:
acquiring a preset test case, and acquiring a user type and/or an application scene corresponding to the test case;
and classifying the test cases according to the user types and/or the application scenes.
3. The data processing method according to claim 1, wherein calling a pre-obtained intention recognition program to perform intention recognition on the test case to obtain a second intention recognition result corresponding to the test case includes:
generating an intention recognition simulation environment matched with a target running environment of the intention recognition program;
and under the intention recognition simulation environment, performing intention recognition on the test case by using the intention recognition program to obtain a second intention recognition result corresponding to the test case.
4. The data processing method of claim 1, wherein comparing the first intention recognition result and the second intention recognition result to obtain a first intention recognition accuracy comprises:
determining the number of test cases with the same corresponding first intention recognition result and second intention recognition result based on the first intention recognition result and the second intention recognition result corresponding to the test cases;
acquiring the total number of the test cases;
and calculating to obtain the first intention identification accuracy according to the number and the total number.
5. The data processing method of claim 1, wherein testing the intent recognition accuracy of the intent recognition program in the target execution environment comprises:
in the target operation environment, performing intention recognition on the test case by using the intention recognition program to obtain a third intention recognition result corresponding to the test case;
comparing the first intention identification result with the third intention identification result to obtain a second intention identification accuracy;
if the difference value between the first intention identification accuracy rate and the second intention identification accuracy rate is within a preset difference value range, determining that the on-line intention test accuracy rate corresponding to the intention identification program passes;
and if the difference value between the first intention identification accuracy rate and the second intention identification accuracy rate is not within a preset difference value range, determining that the on-line intention test accuracy rate corresponding to the intention identification program fails.
6. The data processing method according to any one of claims 1 to 5, further comprising:
if the first intention identification accuracy rate is not greater than a preset threshold value, the intention identification program is adjusted, so that the intention identification accuracy rate of the adjusted intention identification program is greater than the preset threshold value.
7. A data processing apparatus, comprising:
the data acquisition module is used for acquiring a test case and a first intention recognition result corresponding to the pre-marked test case;
the intention identification module is used for calling a pre-acquired intention identification program to perform intention identification on the test case to obtain a second intention identification result corresponding to the test case;
the accuracy calculation module is used for comparing the first intention recognition result with the second intention recognition result to obtain a first intention recognition accuracy;
the test result determining module is used for determining an intention recognition accuracy test result of the intention recognition program if the first intention recognition accuracy is greater than a preset threshold;
and the online testing module is used for operating the intention recognition software in a target operating environment and testing the intention recognition accuracy of the intention recognition program in the target operating environment.
8. The data processing apparatus according to claim 7, wherein the data obtaining module, when obtaining the test case, is specifically configured to:
the method comprises the steps of obtaining a preset test case, obtaining a user type and/or an application scene corresponding to the test case, and classifying the test case according to the user type and/or the application scene.
9. The data processing apparatus according to claim 7, wherein the intention identifying module is configured to invoke a pre-obtained intention identifying program to perform intention identification on the test case, and when a second intention identifying result corresponding to the test case is obtained, is specifically configured to:
and generating an intention recognition simulation environment matched with a target operation environment of the intention recognition program, and performing intention recognition on the test case by using the intention recognition program under the intention recognition simulation environment to obtain a second intention recognition result corresponding to the test case.
10. The data processing apparatus of claim 7, wherein the accuracy calculation module comprises:
the quantity determining submodule is used for determining the quantity of the test cases with the same corresponding first intention recognition result and second intention recognition result based on the first intention recognition result and the second intention recognition result corresponding to the test cases;
the quantity obtaining submodule is used for obtaining the total quantity of the test cases;
and the accuracy calculation submodule is used for calculating the first intention identification accuracy according to the number and the total number.
11. The data processing apparatus of claim 7, wherein the inline test module comprises:
the intention identification submodule is used for carrying out intention identification on the test case by using the intention identification program in the target operation environment to obtain a third intention identification result corresponding to the test case;
the data calculation submodule is used for comparing the first intention recognition result with the third intention recognition result to obtain a second intention recognition accuracy;
a first determining submodule, configured to determine that an online intent test accuracy corresponding to the intent recognition program passes if a difference between the first intent recognition accuracy and the second intent recognition accuracy is within a preset difference range;
and the second determination submodule is used for determining that the on-line intention test accuracy corresponding to the intention identification program fails if the difference value between the first intention identification accuracy and the second intention identification accuracy is not within a preset difference value range.
12. The data processing apparatus according to any one of claims 7 to 11, further comprising:
and the service adjusting module is used for adjusting the intention identification program if the first intention identification accuracy is not greater than a preset threshold value, so that the intention identification accuracy of the adjusted intention identification program is greater than the preset threshold value.
13. An electronic device, comprising: a memory and a processor;
wherein the memory is used for storing programs;
the processor calls a program and is used to:
obtaining a test case and a first intention recognition result which is marked in advance and corresponds to the test case;
calling a pre-obtained intention recognition program to perform intention recognition on the test case to obtain a second intention recognition result corresponding to the test case;
comparing the first intention identification result with the second intention identification result to obtain a first intention identification accuracy;
if the first intention identification accuracy rate is larger than a preset threshold value, determining an intention identification accuracy rate test result of the intention identification program;
and running the intention recognition software in a target running environment, and testing the intention recognition accuracy of the intention recognition program in the target running environment.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911338336.8A CN111128161A (en) | 2019-12-23 | 2019-12-23 | Data processing method and device and electronic equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911338336.8A CN111128161A (en) | 2019-12-23 | 2019-12-23 | Data processing method and device and electronic equipment |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111128161A true CN111128161A (en) | 2020-05-08 |
Family
ID=70501192
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911338336.8A Pending CN111128161A (en) | 2019-12-23 | 2019-12-23 | Data processing method and device and electronic equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111128161A (en) |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102650966A (en) * | 2011-02-24 | 2012-08-29 | 王轶辰 | Multiplexing-oriented embedded software testing method and system |
| US9922650B1 (en) * | 2013-12-20 | 2018-03-20 | Amazon Technologies, Inc. | Intent-specific automatic speech recognition result generation |
| CN110162775A (en) * | 2019-03-11 | 2019-08-23 | 腾讯科技（深圳）有限公司 | Determine the method, apparatus and computer equipment of intention assessment accuracy |
| CN110399609A (en) * | 2019-06-25 | 2019-11-01 | 众安信息技术服务有限公司 | Intension recognizing method, device, equipment and computer readable storage medium |
| CN110503143A (en) * | 2019-08-14 | 2019-11-26 | 平安科技（深圳）有限公司 | Research on threshold selection, equipment, storage medium and device based on intention assessment |
-
2019
- 2019-12-23 CN CN201911338336.8A patent/CN111128161A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102650966A (en) * | 2011-02-24 | 2012-08-29 | 王轶辰 | Multiplexing-oriented embedded software testing method and system |
| US9922650B1 (en) * | 2013-12-20 | 2018-03-20 | Amazon Technologies, Inc. | Intent-specific automatic speech recognition result generation |
| CN110162775A (en) * | 2019-03-11 | 2019-08-23 | 腾讯科技（深圳）有限公司 | Determine the method, apparatus and computer equipment of intention assessment accuracy |
| CN110399609A (en) * | 2019-06-25 | 2019-11-01 | 众安信息技术服务有限公司 | Intension recognizing method, device, equipment and computer readable storage medium |
| CN110503143A (en) * | 2019-08-14 | 2019-11-26 | 平安科技（深圳）有限公司 | Research on threshold selection, equipment, storage medium and device based on intention assessment |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109829629A (en) | Generation method, device, computer equipment and the storage medium of risk analysis reports | |
| US20150269142A1 (en) | System and method for automatically generating a dataset for a system that recognizes questions posed in natural language and answers with predefined answers | |
| CN111597818B (en) | Call quality inspection method, device, computer equipment and computer readable storage medium | |
| US20220327452A1 (en) | Method for automatically updating unit cost of inspection by using comparison between inspection time and work time of crowdsourcing-based project for generating artificial intelligence training data | |
| CN114360581A (en) | Method and device for identifying equipment fault and electronic equipment | |
| CN111124898B (en) | Question-answering system testing method and device, computer equipment and storage medium | |
| CA3182191A1 (en) | Voice quality inspection method and device, computer equipment and storage medium | |
| CN112506757A (en) | Automatic test method, system, computer device and medium thereof | |
| CN110348215B (en) | Abnormal object identification method, abnormal object identification device, electronic equipment and medium | |
| CN113569988B (en) | Algorithm model evaluation method and system | |
| CN117196322B (en) | Intelligent wind control method, intelligent wind control device, computer equipment and storage medium | |
| CN113919432A (en) | Classification model construction method, data classification method and device | |
| CN111128161A (en) | Data processing method and device and electronic equipment | |
| CN116384680A (en) | Visual monitoring method, device and equipment for research and development quality and readable storage medium | |
| CN115859065A (en) | Model evaluation method, device, equipment and storage medium | |
| CN114493340A (en) | Skill data processing method and device, computer equipment and storage medium | |
| CN115309638A (en) | Method and device for assisting model optimization | |
| CN115660451A (en) | Supplier risk early warning method, device, equipment and medium based on RPA | |
| CN114155412A (en) | Deep learning model iteration method, device, equipment and storage medium | |
| CN114546864A (en) | Software quality evaluation method, device, equipment and storage medium | |
| CN113312261A (en) | Test case screening method, test case screening equipment, storage medium and device | |
| CN114817077B (en) | Method, system and network equipment for judging test quality and specification | |
| CN116681421A (en) | Method and device for generating work order defect solution and related products | |
| CN113656325B (en) | Method, device, electronic equipment and computer storage medium for predicting software defects | |
| CN110096430B (en) | Third party SDK access test method, device, terminal and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| CB02 | Change of applicant information |
Address after: 401121 b7-7-2, Yuxing Plaza, No.5 Huangyang Road, Yubei District, Chongqing Applicant after: Chongqing duxiaoman Youyang Technology Co.,Ltd. Address before: 201800 room j1328, 3 / F, building 8, 55 Huiyuan Road, Jiading District, Shanghai Applicant before: SHANGHAI YOUYANG NEW MEDIA INFORMATION TECHNOLOGY Co.,Ltd. |
|
| CB02 | Change of applicant information | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20211215 Address after: 100193 Room 606, 6 / F, building 4, West District, courtyard 10, northwest Wangdong Road, Haidian District, Beijing Applicant after: Du Xiaoman Technology (Beijing) Co.,Ltd. Address before: 401121 b7-7-2, Yuxing Plaza, No.5 Huangyang Road, Yubei District, Chongqing Applicant before: Chongqing duxiaoman Youyang Technology Co.,Ltd. |
|
| TA01 | Transfer of patent application right | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20200508 |
|
| RJ01 | Rejection of invention patent application after publication |