CN112988567B - Crowdsourcing test automated evaluation method and device - Google Patents
Crowdsourcing test automated evaluation method and device Download PDFInfo
- Publication number
- CN112988567B CN112988567B CN202110104907.2A CN202110104907A CN112988567B CN 112988567 B CN112988567 B CN 112988567B CN 202110104907 A CN202110104907 A CN 202110104907A CN 112988567 B CN112988567 B CN 112988567B
- Authority
- CN
- China
- Prior art keywords
- test case
- standard test
- evaluated
- keyword set
- score
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/36—Preventing errors by testing or debugging software
- G06F11/3668—Software testing
- G06F11/3672—Test management
- G06F11/3684—Test management for test design, e.g. generating new test cases
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/36—Preventing errors by testing or debugging software
- G06F11/3668—Software testing
- G06F11/3672—Test management
- G06F11/3688—Test management for test execution, e.g. scheduling of test suites
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Computer Hardware Design (AREA)
- Quality & Reliability (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Debugging And Monitoring (AREA)
Abstract
The invention discloses a crowdsourcing test automatic evaluation method and a device, wherein the automatic evaluation method comprises the following steps: converting each standard test case in the standard test case set into a first keyword set, and converting each test case to be evaluated in the test case set to be evaluated into a second keyword set; acquiring the intersection of the first keyword set and the second keyword set, and mapping each test case to be evaluated to a unique corresponding standard test case according to the intersection; acquiring a union set of the first keyword set and the corresponding second keyword set, and calculating the score of each standard test case according to the union set; and calculating the average value of the total scores, and evaluating the test case set to be evaluated according to the average value. The invention provides a crowdsourcing test automatic evaluation method and device, which are used for solving the technical problem that the crowdsourcing test cannot be evaluated accurately in the prior art and realizing accurate crowdsourcing test automatic evaluation.
Description
Technical Field
The invention relates to the technical field of software testing, in particular to a crowdsourcing test automated evaluation method and device.
Background
In the internet era, various kinds of software are developed, the software has defects in different degrees without exception, and the purpose of software testing is to discover the defects as far as possible before a system is on line, so that the defects can be corrected in time to reduce the influence caused by the defects. In the traditional software test, a few testers need to perform a large number of tests on one software, and the tests consume a great deal of energy of the few testers, are long in time and often have limitations. The crowdsourcing test is a test method derived for solving the defects that the traditional software test is long in time consumption, large in human and material resource consumption and limited in test. The crowdsourcing test is low in cost, and the test task is released on the crowdsourcing platform, so that users on the crowdsourcing platform can participate in the crowdsourcing test task with low reward, and the test cost is effectively reduced. Registered users on a crowdsourcing platform are distributed around the world, internet access equipment, operating systems and programming languages used by the internet personnel are different, and problems such as compatibility problems of software possibly existing in more real environments can be found for the same test task. The information fed back is collected by using crowdsourcing test, so that a large number of possible defects of the software can be obtained, and the software is improved in a targeted manner. However, the low cost of the crowdsourcing test also causes the level of the personnel participating in the crowdsourcing test to be uneven, and some testers can submit low-quality test reports for quickly obtaining reward, so that the quality of the test information collected by the test task is poor, and the test information is difficult to be used for improving the software quality in a targeted manner.
The inventor of the application finds that in the prior art, a large number of test cases are evaluated by a manual means, and the crowdsourcing test cannot be accurately evaluated due to the fact that the number of the test cases is large and the credibility of participants of the crowdsourcing test does not have specific quantitative indexes.
Disclosure of Invention
The invention provides a crowdsourcing test automatic evaluation method and device, which are used for solving the technical problem that a large number of test cases in crowdsourcing tests cannot be accurately evaluated through manual means in the prior art, so that accurate automatic evaluation of the crowdsourcing tests is realized.
The first embodiment of the invention provides a crowdsourcing test automated evaluation method, which comprises the following steps:
converting each standard test case in the standard test case set into a first keyword set, and converting each test case to be evaluated in the test case set to be evaluated into a second keyword set;
acquiring an intersection of a first keyword set corresponding to each standard test case and the second keyword set of each test case to be evaluated, and mapping each test case to be evaluated to the only corresponding standard test case according to the intersection;
acquiring a union set of the first keyword set of each standard test case and the corresponding second keyword set of the test case to be evaluated, and calculating to obtain the score of each standard test case according to the union set;
and calculating the total score of all the standard test cases according to the score of each standard test case, calculating the average value of the total score, and evaluating the test case set to be evaluated according to the average value.
Further, each standard test case in the standard test case set is converted into a first keyword set, and each test case to be evaluated in the test case set to be evaluated is converted into a second keyword set, which specifically includes:
converting each standard test case in the standard test case set into a first keyword set by adopting a natural language processing technology; and converting each test case to be evaluated in the test case set to be evaluated into a second keyword set by adopting a natural language processing technology.
Further, mapping each test case to be evaluated to the unique corresponding standard test case according to the intersection specifically includes:
and acquiring the maximum intersection of each first keyword set and each second keyword set, and mapping the test case to be evaluated corresponding to the maximum intersection to a unique corresponding standard test case.
Further, the obtaining a score of each standard test case according to the union specifically includes:
and calculating the coverage rate of the union set and the first keyword set of the standard test case, and converting the coverage rate into the score of each corresponding standard test case according to a preset conversion rule.
A second embodiment of the present invention provides a crowdsourcing test automated evaluation device, including: the device comprises a conversion module, a mapping module, a calculation module and an evaluation module;
the conversion module is used for converting each standard test case in the standard test case set into a first keyword set and converting each test case to be evaluated in the test case set to be evaluated into a second keyword set;
the mapping module is used for acquiring an intersection of a first keyword set corresponding to each standard test case and the second keyword set of each test case to be evaluated, and mapping each test case to be evaluated to the only corresponding standard test case according to the intersection;
the calculation module is used for acquiring a union set of the first keyword set of each standard test case and the corresponding second keyword set of the test case to be evaluated, and calculating the score of each standard test case according to the union set;
and the evaluation module is used for calculating the total score of all the standard test cases according to the score of each standard test case, calculating the average value of the total score, and evaluating the test case set to be evaluated according to the average value.
Further, the conversion module comprises means for:
converting each standard test case in the standard test case set into a first keyword set by adopting a natural language processing technology; and converting each test case to be evaluated in the test case set to be evaluated into a second keyword set by adopting a natural language processing technology.
Further, the mapping module includes instructions for:
and acquiring the maximum intersection of each first keyword set and each second keyword set, and mapping the test case to be evaluated corresponding to the maximum intersection to a unique corresponding standard test case.
Further, the computing module includes instructions for:
and calculating the coverage rate of the union set and the first keyword set of the standard test case, and converting the coverage rate into the score of each corresponding standard test case according to a preset conversion rule.
The invention provides a crowdsourcing test automatic evaluation method and device, aiming at solving the technical problem that in the prior art, a large number of test cases are evaluated by a manual means, so that crowdsourcing tests cannot be evaluated accurately.
Drawings
Fig. 1 is a schematic flow chart of an automated evaluation method for crowdsourcing test according to an embodiment of the present invention;
FIG. 2 is a diagram illustrating a natural language processing technique according to an embodiment of the present invention for obtaining keywords of a test case;
FIG. 3 is a process diagram of a standard test case for mapping test cases to be evaluated according to an embodiment of the present invention;
FIG. 4 is a schematic diagram illustrating a coverage of a test case for obtaining a standard provided by an embodiment of the present invention;
FIG. 5 is a schematic diagram of obtaining scores for each standard test case according to an embodiment of the present invention;
fig. 6 is a schematic structural diagram of an automated evaluation apparatus for crowdsourcing test according to an embodiment of the present invention.
Detailed Description
The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only a part of the embodiments of the present application, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
In the description of the present application, it is to be understood that the terms "first", "second" are used for descriptive purposes only and are not to be construed as indicating or implying relative importance or implying any number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of the present application, "a plurality" means two or more unless otherwise specified.
In the description of the present application, it is to be noted that, unless otherwise explicitly specified or limited, the terms "mounted," "connected," and "connected" are to be construed broadly, e.g., as meaning either a fixed connection, a removable connection, or an integral connection; can be mechanically or electrically connected; they may be connected directly or indirectly through intervening media, or they may be interconnected between two elements. The specific meaning of the above terms in the present application can be understood in a specific case by those of ordinary skill in the art.
Referring to fig. 1-5, a first embodiment of the invention provides an automated evaluation method for crowdsourcing test as shown in fig. 1, comprising:
s1, converting each standard test case in the standard test case set into a first keyword set, and converting each test case to be evaluated in the test case set to be evaluated into a second keyword set;
s2, acquiring the intersection of the first keyword set corresponding to each standard test case and the second keyword set of each test case to be evaluated, and mapping each test case to be evaluated to the only corresponding standard test case according to the intersection;
s3, acquiring a union set of the first keyword set of each standard test case and the corresponding second keyword set of the test case to be evaluated, and calculating to obtain the score of each standard test case according to the union set;
and S4, calculating the total score of all the standard test cases according to the score of each standard test case, calculating the average value of the total score, and evaluating the test case set to be evaluated according to the average value.
In the embodiment of the invention, the score of each standard test case is calculated, the total score is obtained after all the scores are accumulated and summed, the total score is divided by the total number of the standard test cases in the standard test case set to obtain the average score, the evaluation case set to be evaluated is evaluated according to the average score, and when the average score is higher than the preset threshold score, the reliable evaluation conclusion of the test case set to be evaluated is obtained.
The embodiment of the invention converts each standard test case in the standard test case set into the first keyword set, converts each test case to be evaluated in the test case set to be evaluated into the second keyword set, maps each test case to be evaluated to the only corresponding standard test case according to the intersection of the first keyword set and the second keyword set, calculates the score of each standard test case according to the mapping relation, and evaluates the test case set to be evaluated by averaging the total scores of all the standard test cases. According to the embodiment of the invention, the standard test cases with the same number of keywords as the most number are obtained for the test cases to be evaluated according to the mapping relation between the standard test cases and the test cases to be evaluated, the evaluation of the test case set to be evaluated is realized by calculating the average score of the standard test cases, the test cases to be evaluated are evaluated by taking the standard test case data as the basis, the automatic evaluation of the test case set to be evaluated accurately can be realized, and the evaluation accuracy and reliability of the crowdsourcing test can be improved.
As a specific implementation manner of the embodiment of the present invention, converting each standard test case in a standard test case set into a first keyword set, and converting each test case to be evaluated in a test case set to be evaluated into a second keyword set, specifically includes:
and converting each standard test case in the standard test case set into a first keyword set by adopting a natural language processing technology, and converting each test case to be evaluated in the test case set to be evaluated into a second keyword set by adopting the natural language processing technology.
In the embodiment of the invention, aiming at a crowdsourced test object, a standard test case set is obtained and is { TS1, TS2 … TSm }, wherein each standard test case is TSj; the test case set to be evaluated is { TC1, TC2 … TCn }, each test case to be evaluated converts each standard test case into a first keyword set { TSjKW1, TSjKW2 … TSjKWp } (j ═ 1,2 … m) by using a natural language processing technology; each test case to be evaluated is converted into a second keyword set { tciww 1, tciww 2 … TCiKWq } (i ═ 1,2 … n). Referring to fig. 2, a schematic diagram of obtaining a keyword of a test case by using a natural language processing technology according to an embodiment of the present invention is shown. Wherein, the Testcase Standard represents a Standard test case; the Testcase Standard Keywords set represents a first keyword set of a Standard test case; testcase represents a test case to be evaluated; the Testcase Keywords set represents a second keyword set of the test case to be evaluated; NLP denotes natural language processing.
As a specific implementation manner of the embodiment of the present invention, mapping each test case to be evaluated to a unique corresponding standard test case according to an intersection includes:
and acquiring the maximum intersection of each first keyword set and each second keyword set, and mapping the test case to be evaluated corresponding to the maximum intersection to the only corresponding standard test case.
In the embodiment of the invention, an intersection is taken between the second keyword set { tciww 1, tciww 2 … TCiKWq } (i ═ 1,2 … n) and the first keyword set { TSjKW1, TSjKW2 … TSjKWp } (j ═ 1,2 … m), m × n intersections are taken together, the standard test cases TSk with the same number as the keywords of the test cases to be evaluated and the maximum number are obtained according to all the intersections, and each test case to be evaluated is mapped to the only corresponding standard test case. Such as: TC1 → TS2, TC2 → TS2, TC3 → TS1, TC4 → TS3, …, TCm → TSn. According to the embodiment of the invention, each test case to be evaluated is mapped to the only corresponding standard test case, evaluation is carried out according to the mapping relation between the standard test case and the test case to be evaluated, and the evaluation accuracy and reliability of the test case to be evaluated can be effectively improved. Please refer to fig. 3, which is a process diagram of mapping a Standard test case to a test case to be evaluated according to an embodiment of the present invention, wherein a Testcase Standard Keywords represents Keywords of the Standard test case; keywords count of associations represents the same number of Keywords in the intersection; and TC-TS mapping represents mapping from the test case to be evaluated to the standard test case.
As a specific implementation manner of the embodiment of the present invention, obtaining the score of each standard test case according to the union specifically includes:
and calculating the coverage rate of the union set and the first keyword set of the standard test case, and converting the coverage rate into the score of each corresponding standard test case according to a preset conversion rule.
In the embodiment of the present invention, each standard test case TSk is merged with the intersection TSk ≤ TCp, TSk ≤ TCq … of the second keyword set of all test cases to be evaluated { TCm, TCp, TCq … } corresponding thereto to obtain a third keyword set { TCmKWx, …, TCpKWy, …, TCqKWz, … }, the number of keywords in the third keyword set is divided with the number of keywords in the keyword set { tskkkw 1, tskkkw 2 … TSkKWp } converted by the standard test case TSk to obtain the coverage rate of the first keyword set of the standard test case, the coverage rate is converted into the score of the standard test case according to a preset conversion rule, and the scores of all standard test cases are calculated. Fig. 4 is a schematic diagram illustrating a coverage condition of a standard test case obtained according to an embodiment of the present invention, where TC-TS Keywords represent Keywords of a test case to be evaluated mapped to the standard test case. Please refer to fig. 5, which is a schematic diagram illustrating obtaining scores of each Standard test case according to an embodiment of the present invention, wherein a Testcase Standard Keywords count represents the number of Keywords of the Standard test case; the TC-TS Keywords score represents the keyword score that the test case to be evaluated maps to the standard test case.
The embodiment of the invention has the following beneficial effects:
the embodiment of the invention converts each standard test case in a standard test case set into a first keyword set, converts each test case to be evaluated in the test case set to be evaluated into a second keyword set, maps each test case to be evaluated to a unique corresponding standard test case according to the intersection of the first keyword set and the second keyword set, calculates the score of each standard test case according to the mapping relation, calculates the average value according to the total score of all the standard test cases, evaluates the test case set to be evaluated, obtains the standard test cases with the most keywords according to the mapping relation of the standard test cases and the test cases to be evaluated, evaluates the test case set to be evaluated by calculating the average score of the standard test cases, evaluates the test cases to be evaluated by taking the data of the standard test cases as the basis, the automatic evaluation of the test case set to be evaluated accurately can be realized, and the evaluation accuracy and reliability of the crowdsourcing test can be improved.
Referring to fig. 6, a second embodiment of the present invention provides an automatic evaluation apparatus for crowdsourcing test shown in fig. 6, comprising: a conversion module 10, a mapping module 20, a calculation module 30 and an evaluation module 40;
the conversion module 10 is configured to convert each standard test case in the standard test case set into a first keyword set, and convert each test case to be evaluated in the test case set to be evaluated into a second keyword set;
the mapping module 20 is configured to obtain an intersection of the first keyword set corresponding to each standard test case and the second keyword set of each test case to be evaluated, and map each test case to be evaluated to a unique corresponding standard test case according to the intersection;
the calculation module 30 is configured to obtain a union set of the first keyword set of each standard test case and the corresponding second keyword set of the test case to be evaluated, and calculate a score of each standard test case according to the union set;
and the evaluation module 40 is configured to calculate a total score of all the standard test cases according to the score of each standard test case, calculate an average value of the total scores, and evaluate the test case set to be evaluated according to the average value.
In the embodiment of the invention, the score of each standard test case is calculated, the total score is obtained after all the scores are accumulated and summed, the total score is divided by the total number of the standard test cases in the standard test case set to obtain the average score, the evaluation case set to be evaluated is evaluated according to the average score, and when the average score is higher than the preset threshold score, the reliable evaluation conclusion of the test case set to be evaluated is obtained.
The embodiment of the invention converts each standard test case in a standard test case set into a first keyword set, converts each test case to be evaluated in the test case set to be evaluated into a second keyword set, maps each test case to be evaluated to a unique corresponding standard test case according to the intersection of the first keyword set and the second keyword set, calculates the score of each standard test case according to the mapping relation, calculates the average value according to the total score of all the standard test cases, evaluates the test case set to be evaluated, obtains the standard test cases with the most keywords according to the mapping relation of the standard test cases and the test cases to be evaluated, evaluates the test case set to be evaluated by calculating the average score of the standard test cases, evaluates the test cases to be evaluated by taking the data of the standard test cases as the basis, the automatic evaluation of the test case set to be evaluated accurately can be realized, and the evaluation accuracy and reliability of the crowdsourcing test can be improved.
As a specific implementation manner of the embodiment of the present invention, the conversion module 10 includes:
converting each standard test case in the standard test case set into a first keyword set by adopting a natural language processing technology; and converting each test case to be evaluated in the test case set to be evaluated into a second keyword set by adopting a natural language processing technology.
In the embodiment of the invention, aiming at a test object, a standard test case set is obtained as { TS1, TS2 … TSm }, wherein each standard test case is TSj; the test case set to be evaluated is { TC1, TC2 … TCn }, each test case to be evaluated converts each standard test case into a first keyword set { TSjKW1, TSjKW2 … TSjKWp } (j ═ 1,2 … m) by using a natural language processing technology; each test case to be evaluated is converted into a second keyword set { tciww 1, tciww 2 … TCiKWq } (i ═ 1,2 … n). Referring to fig. 2, a schematic diagram of obtaining a keyword of a test case by using a natural language processing technology according to an embodiment of the present invention is shown. Wherein, the Testcase Standard represents a Standard test case; the Testcase Standard Keywords set represents a first keyword set of a Standard test case; testcase represents a test case to be evaluated; the Testcase Keywords set represents a second keyword set of the test case to be evaluated; NLP denotes natural language processing.
As a specific implementation manner of the embodiment of the present invention, the mapping module 20 includes:
and acquiring the maximum intersection of each first keyword set and each second keyword set, and mapping the test case to be evaluated corresponding to the maximum intersection to the only corresponding standard test case.
In the embodiment of the invention, an intersection is taken between the second keyword set { tciww 1, tciww 2 … TCiKWq } (i ═ 1,2 … n) and the first keyword set { TSjKW1, TSjKW2 … TSjKWp } (j ═ 1,2 … m), m × n intersections are taken together, the standard test cases TSj with the same number and the most number as the keywords of the test cases to be evaluated are obtained according to all the intersections, and each test case to be evaluated is mapped to the only corresponding standard test case. Such as: TC1 → TS2, TC2 → TS2, TC3 → TS1, TC4 → TS3, …, TCm → TSn. According to the embodiment of the invention, each test case to be evaluated is mapped to the only corresponding standard test case, evaluation is carried out according to the mapping relation between the standard test case and the test case to be evaluated, and the evaluation accuracy and reliability of the test case to be evaluated can be effectively improved. Please refer to fig. 3, which is a process diagram of mapping a Standard test case to a test case to be evaluated according to an embodiment of the present invention, wherein a Testcase Standard Keywords represents Keywords of the Standard test case; keywords count of associations represents the same number of Keywords in the intersection; and TC-TS mapping represents mapping from the test case to be evaluated to the standard test case.
As a specific implementation manner of the embodiment of the present invention, the calculating module 30 includes:
and calculating the coverage rate of the union set and the first keyword set of the standard test case, and converting the coverage rate into the score of each corresponding standard test case according to a preset conversion rule.
In the embodiment of the present invention, each standard test case TSk is merged with the intersection TSk ≦ TCm, TSk ≦ TCp, TSk ≦ TCq … of the second keyword set of all test cases to be evaluated { TCm, TCp, TCq … } corresponding thereto to obtain the third keyword set { TCmKWx, …, TCpKWy, …, TCqKWz, … }, the number of keywords in the third keyword set is divided by the number of keywords in the keyword set { tskkkw 1, tskkkw 2 … kkkwp } converted by the standard test case TSk to obtain the coverage rate of the first keyword set of the standard test case, the coverage rate is converted into the score of the standard test case according to the preset conversion rule, and the scores of all standard test cases are calculated. Fig. 4 is a schematic diagram illustrating a coverage condition of a standard test case obtained according to an embodiment of the present invention, where TC-TS Keywords represent Keywords of a test case to be evaluated mapped to the standard test case. Please refer to fig. 5, which is a schematic diagram illustrating obtaining scores of each Standard test case according to an embodiment of the present invention, wherein a Testcase Standard Keywords count represents the number of Keywords of the Standard test case; the TC-TS Keywords score represents the keyword score that the test case to be evaluated maps to the standard test case.
The embodiment of the invention has the following beneficial effects:
the embodiment of the invention converts each standard test case in the standard test case set into the first keyword set, converts each test case to be evaluated in the test case set to be evaluated into the second keyword set, maps each test case to be evaluated to the only corresponding standard test case according to the intersection of the first keyword set and the second keyword set, calculates the score of each standard test case according to the mapping relation, and evaluates the test case set to be evaluated by averaging the total scores of all the standard test cases. According to the embodiment of the invention, the standard test cases with the same number of keywords as the most number are obtained for the test cases to be evaluated according to the mapping relation between the standard test cases and the test cases to be evaluated, the evaluation of the test case set to be evaluated is realized by calculating the average score of the standard test cases, the test cases to be evaluated are evaluated by taking the standard test case data as the basis, the automatic evaluation of the test case set to be evaluated accurately can be realized, and the evaluation accuracy and reliability of the crowdsourcing test can be improved.
The foregoing is a preferred embodiment of the present invention, and it should be noted that it would be apparent to those skilled in the art that various modifications and enhancements can be made without departing from the principles of the invention, and such modifications and enhancements are also considered to be within the scope of the invention.
Claims (6)
1. An automated evaluation method for crowdsourcing tests, comprising:
converting each standard test case in the standard test case set into a first keyword set, and converting each test case to be evaluated in the test case set to be evaluated into a second keyword set;
acquiring an intersection of a first keyword set corresponding to each standard test case and the second keyword set of each test case to be evaluated, and mapping each test case to be evaluated to the only corresponding standard test case according to the intersection;
acquiring a union set of the first keyword set of each standard test case and the corresponding second keyword set of the test case to be evaluated, and calculating to obtain the score of each standard test case according to the union set; calculating the score of each standard test case according to the union set, wherein the calculation specifically comprises the following steps: calculating the coverage rate of the union set and the first keyword set of the standard test case, and converting the coverage rate into the score of each corresponding standard test case according to a preset conversion rule;
calculating the total score of all the standard test cases according to the score of each standard test case, calculating the average value of the total score, and evaluating the test case set to be evaluated according to the average value; calculating the score of each standard test case, accumulating and summing all the scores to obtain a total score, and dividing the total score by the total number of the standard test cases in the standard test case set to obtain an average score.
2. The automated evaluation method for crowdsourcing test according to claim 1, wherein converting each standard test case in the standard test case set into a first keyword set, and converting each test case in the test case set to be evaluated into a second keyword set, specifically comprises:
converting each standard test case in the standard test case set into a first keyword set by adopting a natural language processing technology; and converting each test case to be evaluated in the test case set to be evaluated into a second keyword set by adopting a natural language processing technology.
3. The automated evaluation method for crowdsourcing test according to claim 1, wherein the mapping each test case to be evaluated to the uniquely corresponding standard test case according to the intersection specifically comprises:
and acquiring the maximum intersection of each first keyword set and each second keyword set, and mapping the test case to be evaluated corresponding to the maximum intersection to a unique corresponding standard test case.
4. An automated crowdsourcing test evaluation device, comprising: the device comprises a conversion module, a mapping module, a calculation module and an evaluation module;
the conversion module is used for converting each standard test case in the standard test case set into a first keyword set and converting each test case to be evaluated in the test case set to be evaluated into a second keyword set;
the mapping module is used for acquiring an intersection of a first keyword set corresponding to each standard test case and the second keyword set of each test case to be evaluated, and mapping each test case to be evaluated to the only corresponding standard test case according to the intersection;
the calculation module is used for acquiring a union set of the first keyword set of each standard test case and the corresponding second keyword set of the test case to be evaluated, and calculating the score of each standard test case according to the union set; the method is specifically used for: calculating the coverage rate of the union set and the first keyword set of the standard test case, and converting the coverage rate into the score of each corresponding standard test case according to a preset conversion rule;
the evaluation module is used for calculating the total score of all the standard test cases according to the score of each standard test case, calculating the average value of the total score, and evaluating the test case set to be evaluated according to the average value; the method is specifically used for calculating the score of each standard test case, accumulating and summing all the scores to obtain a total score, and dividing the total score by the total number of the standard test cases in the standard test case set to obtain an average score.
5. The crowdsourcing test automated evaluation device of claim 4, wherein the translation module comprises instructions to:
converting each standard test case in the standard test case set into a first keyword set by adopting a natural language processing technology; and converting each test case to be evaluated in the test case set to be evaluated into a second keyword set by adopting a natural language processing technology.
6. The crowdsourcing test automated evaluation device of claim 4, wherein the mapping module comprises means for:
and acquiring the maximum intersection of each first keyword set and each second keyword set, and mapping the test case to be evaluated corresponding to the maximum intersection to a unique corresponding standard test case.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110104907.2A CN112988567B (en) | 2021-01-26 | 2021-01-26 | Crowdsourcing test automated evaluation method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110104907.2A CN112988567B (en) | 2021-01-26 | 2021-01-26 | Crowdsourcing test automated evaluation method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112988567A CN112988567A (en) | 2021-06-18 |
| CN112988567B true CN112988567B (en) | 2022-02-15 |
Family
ID=76345464
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110104907.2A Active CN112988567B (en) | 2021-01-26 | 2021-01-26 | Crowdsourcing test automated evaluation method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112988567B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113918471A (en) * | 2021-11-19 | 2022-01-11 | 中国工商银行股份有限公司 | Test case processing method and device and computer readable storage medium |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106294182A (en) * | 2016-08-24 | 2017-01-04 | 腾讯科技（深圳）有限公司 | A kind of determine many survey the feedback method of effectiveness, test equipment and system |
| CN108874655A (en) * | 2017-05-15 | 2018-11-23 | 华为技术有限公司 | A kind of method and device handling crowdsourcing test data |
| US10162734B1 (en) * | 2016-07-20 | 2018-12-25 | Intuit Inc. | Method and system for crowdsourcing software quality testing and error detection in a tax return preparation system |
| CN110069409A (en) * | 2019-04-12 | 2019-07-30 | 网宿科技股份有限公司 | A kind of Evaluation of Software Quality and system |
| CN111899740A (en) * | 2020-07-23 | 2020-11-06 | 深圳慕智科技有限公司 | Voice recognition system crowdsourcing test case generation method based on test requirements |
Family Cites Families (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8794971B2 (en) * | 2010-10-09 | 2014-08-05 | Yellowpages.Com Llc | Method and system for assigning a task to be processed by a crowdsourcing platform |
| US10223244B2 (en) * | 2015-09-15 | 2019-03-05 | Accenture Global Solutions Limited | Test plan inspection platform |
| CN108304328B (en) * | 2018-02-05 | 2021-04-27 | 苏州大学 | Text description generation method, system and device for crowdsourcing test report |
| CN111090585B (en) * | 2019-12-09 | 2021-06-01 | 中国科学院软件研究所 | Crowd-sourcing task closing time automatic prediction method based on crowd-sourcing process |
-
2021
- 2021-01-26 CN CN202110104907.2A patent/CN112988567B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10162734B1 (en) * | 2016-07-20 | 2018-12-25 | Intuit Inc. | Method and system for crowdsourcing software quality testing and error detection in a tax return preparation system |
| CN106294182A (en) * | 2016-08-24 | 2017-01-04 | 腾讯科技（深圳）有限公司 | A kind of determine many survey the feedback method of effectiveness, test equipment and system |
| CN108874655A (en) * | 2017-05-15 | 2018-11-23 | 华为技术有限公司 | A kind of method and device handling crowdsourcing test data |
| CN110069409A (en) * | 2019-04-12 | 2019-07-30 | 网宿科技股份有限公司 | A kind of Evaluation of Software Quality and system |
| CN111899740A (en) * | 2020-07-23 | 2020-11-06 | 深圳慕智科技有限公司 | Voice recognition system crowdsourcing test case generation method based on test requirements |
Non-Patent Citations (1)
| Title |
|---|
| 移动应用众包测试质量影响因素分析;成静 等;《计算机应用》;20180930(第9期);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112988567A (en) | 2021-06-18 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Fincham et al. | Counting clicks is not enough: Validating a theorized model of engagement in learning analytics | |
| US11415469B2 (en) | Edge-cloud collaboration platform for intelligent coking monitoring of cracking furnace tubes and working method thereof | |
| CN114937264B (en) | Transformer manufacturing assembly process monitoring and analyzing system based on artificial intelligence | |
| CN112988567B (en) | Crowdsourcing test automated evaluation method and device | |
| KR102232880B1 (en) | Method for evaluating inspector of crowdsourcing based projects for collecting image or video for artificial intelligence training data generation | |
| CN112905435A (en) | Workload evaluation method, device and equipment based on big data and storage medium | |
| CN109190674A (en) | The generation method and device of training data | |
| CN109214625A (en) | A kind of oil tank evaluation method for failure and device | |
| CN112434717B (en) | Model training method and device | |
| CN109359803B (en) | Reliability evaluation method and device | |
| CN117745222A (en) | Method, device, equipment, medium and product for auditing annotation content oriented to large model scene | |
| CN111967276B (en) | Translation quality evaluation method and device, electronic equipment and storage medium | |
| CN113361780A (en) | Behavior data-based crowdsourcing tester evaluation method | |
| CN102081124B (en) | System and method for identifying high-speed peripheral equipment interconnected signal | |
| CN113159537B (en) | Assessment method and device for new technical project of power grid and computer equipment | |
| Prastyo et al. | Development of the Human Resources Information System (HRIS) Based on Usability Analysis with Usefulness, Satisfaction, Ease to Use (USE) Questionaire and Cognitive Walkthrough Methods | |
| CN118503654B (en) | Data analysis method and system for monitoring landslide accumulation | |
| CN118053101B (en) | Method, system, equipment and medium for detecting aviation winder | |
| CN111507639B (en) | Financing risk analysis method and device | |
| WO2022219810A1 (en) | Information presentation device, information presentation method, and program | |
| CN115204699A (en) | Test data generation method and device based on portrait, computer equipment and medium | |
| EP4287198A1 (en) | Method and system for determining which stage a user performance belongs to | |
| CN118747294A (en) | Artificial intelligence management method for identifying artificial intelligence generated content | |
| CN113095711A (en) | Training quality evaluation method and system for remotely training nursing staff | |
| CN116312076A (en) | Drilling engineering examination method and system based on VR equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |