CN115587726A - Crowdsourcing test task allocation method based on capability matching - Google Patents
Crowdsourcing test task allocation method based on capability matching Download PDFInfo
- Publication number
- CN115587726A CN115587726A CN202110764872.5A CN202110764872A CN115587726A CN 115587726 A CN115587726 A CN 115587726A CN 202110764872 A CN202110764872 A CN 202110764872A CN 115587726 A CN115587726 A CN 115587726A
- Authority
- CN
- China
- Prior art keywords
- task
- test
- crowdsourcing
- worker
- factor
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000012360 testing method Methods 0.000 title claims abstract description 86
- 238000000034 method Methods 0.000 title claims abstract description 28
- 238000011156 evaluation Methods 0.000 claims description 16
- 238000004364 calculation method Methods 0.000 claims description 3
- 238000012358 sourcing Methods 0.000 claims 1
- 239000013598 vector Substances 0.000 abstract description 3
- 238000012512 characterization method Methods 0.000 abstract 1
- 238000013136 deep learning model Methods 0.000 abstract 1
- 238000007781 pre-processing Methods 0.000 abstract 1
- 230000011218 segmentation Effects 0.000 abstract 1
- 239000011159 matrix material Substances 0.000 description 12
- 230000006978 adaptation Effects 0.000 description 11
- 230000000694 effects Effects 0.000 description 4
- 230000007547 defect Effects 0.000 description 3
- 230000003044 adaptive effect Effects 0.000 description 2
- 230000000295 complement effect Effects 0.000 description 2
- 238000000692 Student's t-test Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000001364 causal effect Effects 0.000 description 1
- 239000002131 composite material Substances 0.000 description 1
- 230000001419 dependent effect Effects 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 238000012552 review Methods 0.000 description 1
- 238000012216 screening Methods 0.000 description 1
- 238000013522 software testing Methods 0.000 description 1
- 238000012353 t test Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q10/00—Administration; Management
- G06Q10/06—Resources, workflows, human or project management; Enterprise or organisation planning; Enterprise or organisation modelling
- G06Q10/063—Operations research, analysis or management
- G06Q10/0631—Resource planning, allocation, distributing or scheduling for enterprises or organisations
- G06Q10/06311—Scheduling, planning or task assignment for a person or group
- G06Q10/063112—Skill-based matching of a person or a group to a task
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q10/00—Administration; Management
- G06Q10/06—Resources, workflows, human or project management; Enterprise or organisation planning; Enterprise or organisation modelling
- G06Q10/063—Operations research, analysis or management
- G06Q10/0639—Performance analysis of employees; Performance analysis of enterprise or organisation operations
- G06Q10/06398—Performance of employee with respect to a job function
Landscapes
- Business, Economics & Management (AREA)
- Human Resources & Organizations (AREA)
- Engineering & Computer Science (AREA)
- Strategic Management (AREA)
- Educational Administration (AREA)
- Economics (AREA)
- Entrepreneurship & Innovation (AREA)
- Development Economics (AREA)
- Game Theory and Decision Science (AREA)
- Marketing (AREA)
- Operations Research (AREA)
- Quality & Reliability (AREA)
- Tourism & Hospitality (AREA)
- Physics & Mathematics (AREA)
- General Business, Economics & Management (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
A deep criminal phase prediction method based on case facts uses a deep learning model in criminal phase prediction in the judicial field, the case facts are processed into feature vectors, and the criminal phase serves as a label. The method mainly comprises three steps, wherein the first step is text preprocessing, extracted case facts are subjected to word segmentation and characterization engineering, a processed sequence is used as case characteristic representation, the criminal period is divided into 5 types, and tags are made for each case. The second step is to take the processed data set as input, train with fastText algorithm to get the prediction model of the criminal phase, the last step is to process the test set according to the mode of the first step, use the model got from the second step to predict the criminal phase, then compare with the actual label. The invention can basically and accurately predict the criminal period label based on case facts and provides reference for the workers to propose criminal measuring suggestions.
Description
Technical Field
The invention belongs to the field of software testing, aims at the research of task allocation in a crowdsourcing test process, focuses on how to effectively allocate a test task to a proper crowdsourcing worker, comprehensively considers the comprehensive capabilities of candidate crowdsourcing workers in terms of capability, experience, willingness degree, credibility degree and the like by combining the task, gives formal representation of a strategy in detail, and is an effective task allocation method.
Background
The crowdsourcing test is a technology for testing software by utilizing a crowdsourcing idea, and task distributors and task completers are connected together through tasks. The advantages of short test period, low cost and high efficiency, etc., the crowdsourcing test has gained wide attention in the industry and academia. In current testing practice, testing tasks are typically performed by a random set of workers, not all crowdsourced workers are eligible to perform it for a particular testing task, and different testing tasks require different requirements on worker experience, domain knowledge, etc., and unadapted crowdsourced workers may miss real bugs, introduce false bugs, which not only reduces the quality of the test results, but also increases worker employment costs and test report review costs.
In order to solve the problems, the invention focuses on how to effectively distribute the test tasks to proper crowdsourcing workers, and provides a crowdsourcing test task distribution method based on capability matching, and the requirements of the candidate crowdsourcing workers on the aspects of capability, experience, willingness, credibility and the like are comprehensively considered by combining the test tasks, so that the method has important significance for helping a contracting party to reduce the input test cost, improve the overall test quality and promote the operating efficiency of a crowdsourcing test platform.
The method is mainly applied to test task allocation in the whole crowdsourcing test activity process, the requirements of task attributes (including task types, task complexity and the like) on crowdsourcing workers are analyzed, the candidate crowdsourcing workers are evaluated by taking four dimensions of experience, willingness degree, capability and credibility as adaptation factors, and the influence of each factor on task allocation decision is different. The weights of the factors are usually set by a task requester, and the task requester may have fuzzy knowledge about the task completion progress and the actual participation amount of crowdsourcing workers, so that the weight of each adaptive factor in the whole distribution mechanism cannot be determined by an accurate numerical value. Based on the method, the weights of the adaptation factors can be determined by using a fuzzy analytic hierarchy process according to the relative importance degree (represented by triangular fuzzy numbers) between every two adaptation factors described by the requesting party, so that fuzzy information is quantized, and a crowdsourcing test platform is helped to accurately make reasonable matching standards of tasks and crowdsourcing workers.
Disclosure of Invention
The invention aims to comprehensively consider the requirements of candidate crowdsourcing workers in the aspects of capability, experience, willingness degree, credibility degree and the like in combination with tasks, focuses on how to effectively distribute test tasks to the appropriate crowdsourcing workers, and provides reference for task distribution of crowdsourcing tests.
In order to achieve the above purpose, the invention provides a crowdsourcing test task allocation method based on capability matching, which mainly comprises the following four steps:
1) Analyzing the attribute of the test task according to the test requirement provided by the task contracting party, and analyzing the attribute of the test task, including task type, task complexity and the like;
2) Depicting the portrait of crowdsourcing workers according to the task attribute information obtained by analyzing in the step 1), depicting the portrait of crowdsourcing workers from four aspects of capability, experience, willingness degree and credibility, and measuring the influence degree of each factor on the distribution of test tasks by adopting triangular fuzzy numbers;
3) Determining the relative weight of the matching factors according to the relative importance degree between every two influencing factors described by the contracting party, and determining the weight of the influencing factors by using a fuzzy analytic hierarchy process, thereby quantifying fuzzy information and helping the contracting party to make a correct decision;
4) And calculating a comprehensive evaluation value according to the data set obtained in the step 1) 2) 3), and calculating the sum of products of each influence factor value and the corresponding weight of each influence factor value, so as to measure the comprehensive matching degree of the candidate crowdsourcing worker corresponding to the execution of the test task.
The invention has the beneficial effects that: the method provides reference for task allocation of crowdsourcing tests, the crowdsourcing worker screening basis is a result obtained according to comprehensive evaluation, multi-dimensional allocation influence factors are considered, the method is more convincing, a new debate basis is provided for crowdsourcing test platforms to select high-quality crowdsourcing workers, and meanwhile the method has important significance for helping the crowdsourcing test platforms to reduce input test cost, improve overall test quality and promote operation efficiency of the crowdsourcing test platforms.
Drawings
FIG. 1 is an overall flow chart of the present invention
Detailed Description
In order to clearly understand the technical contents of the present invention, the detailed procedures and operation details of each step in the framework will be described in detail hereinafter.
1. Capability adaptation value
For test task t, worker capability fitness is evaluated with a score, the higher the score, the higher his task performance fitness. Suppose that a candidate worker participating in task t has w 1 ，w 2 ，...，w N Then for each candidate worker w for I E {1, 2.. N }, then I The capability adaptation values relative to the test task t are:
wherein x represents worker w I Number of test reports submitted, S score Indicating worker w I Score of the xth report in the test report submitted, S Tosco Represents the total score of a single report.
2. Empirical fit value
Crowdsourcing workers with more similar experience to the testing task, i.e., workers with more relevant experience, are more likely to find defects. Therefore, we can consider that for the test task t, the worker passes throughThe fitness of the test is evaluated by a numerical value, the higher the numerical value, the higher his fitness for the task to perform. Suppose that a candidate worker participating in task t has w 1 ，w 2 ，...，w N For each candidate worker w of I e {1,2,. N }, then I The empirical fit values with respect to the test task t are:
wherein, # TP T Indicates worker w I Number of times of participation in t-test type dependent crowdsourcing tests, N 0 Indicates worker w I Total number of all crowdsourced test campaigns attended.
3. Willingness degree adaptation value
Workers with high initiative are more likely to find defects. In other words, it can be understood that the stronger the willingness of crowdsourcing workers to participate, the greater the probability of discovering defects. Thus we can consider that for the test task t, the worker's willingness to fit is evaluated with the total number of test reports submitted, the higher the number, the higher his task execution fit. Suppose that a candidate worker participating in task t has w 1 ，w 2 ，...，w N Then for each candidate worker w for I E {1, 2.. N }, then I The willingness adaptation values with respect to the test task t are:
S wan (w I ，t)＝Sigmoid(#TC t ) (3)
wherein, # TC t Indicating worker w I The total number of test reports submitted in all of the participating crowd-sourced testing activities.
4. Goodness of credit adaptation value
The task allocation mechanism based on credibility may improve the overall test quality, so we can consider that the fitness of the test task t and the worker credibility is evaluated by a numerical value, and the higher the numerical value is, the higher the fitness of the task execution of the test task t and the worker credibility is. Suppose that a candidate worker participating in task t has w 1 ，w 2 ，...，w N Then for each candidate worker w for I E {1, 2.. N }, then I Relative to the testThe reputation adaptation value of the task t is as follows:
wherein, # ResgiP I Indicates worker w I Total number of enrollment in crowdsourcing tests, # ActuP I Indicating worker w I The number of times the user actually participates in the crowdsourced test activity.
5. Adaptation factor influencing weights
The task allocation using capacity, experience, willingness degree and credibility degree of the crowdsourcing test are used for evaluating candidate crowdsourcing workers, and factors in each aspect have different influences on task allocation decision. Because, factor influence weight is defined to measure the influence degree of each factor, and omega is used respectively cap ，ω exp ，ω wan ，ω cre And the evaluation weights of the capability, experience, willingness degree and reputation degree are expressed. Each weight value is in the range of [0,1 ]]The sum of the four being 1, i.e. ω cap +ω exp +ω wan +ω cre And =1. The larger the value is, the larger the influence of the factor is, the value of 0 indicates that the factor does not work, and the value of 1 indicates that the factor plays a decisive role.
The weight of the above influencing factors is generally set by the contracting party, but in practice, the contracting party may have a vague knowledge of the task completion progress and the proportion of the number of workers required in the whole crowdsourcing test activity, and it is difficult to clearly give an accurate value. Based on the above, the fuzzy analytic hierarchy process is used to determine the weight of each two influencing factors described by the contracting party according to the relative importance degree between the two influencing factors, so as to quantify the fuzzy information and help the contracting party to make a correct decision, and the specific operation steps are as follows:
(1) Determining the causal relationship among all factors to construct a multilevel (multi-level) hierarchical structure model, comparing every two of the elements of the same level (level) with the elements of the higher level as a criterion, and judging the relative importance degree to obtain a fuzzy judgment matrix F, namely:
wherein,
and f is ij +f ji ＝1，i，j＝1，2，...，n，f ij Showing the fuzzy relation of the ith element relative to the jth element of the lower layer.
(2) And (5) calculating the relative importance among the elements in the F, and solving the formula as follows.
From this, the weight vector w of the fuzzy judgment matrix F can be calculated F ＝(w 1 ，w 2 ，...，w n ) T Wherein w is i Is not less than 0 and w 1 +w 2 +…+w n ＝1。
(3) The above formula was checked for consistency. When the deviation consistency is too large, the calculation result is not reliable as a decision basis. The consistency principle of the fuzzy judgment matrix is checked by adopting the compatibility of the fuzzy judgment matrix in the following method.
a. Compatibility index
If there is a fuzzy judgment matrix
And
the compatibility indexes of the two are as follows:
b. feature matrix
By the formula (6) through W ij ＝W i W i +W j (i, j =1, 2.. Times.n) is converted to obtain the blurFeature matrix W of judgment matrix F * ＝(W ij ) n×n 。
And judging by adopting the compatibility index according to the description F of the task requester on the adaptation factors, wherein when I (F, W) is less than or equal to F, the judgment matrix can be considered to meet the consistency. And the smaller the value of F is, the higher the requirement of the task requester on the consistency of the fuzzy judgment matrix is, and the value of F is usually 0.1. When m (m is more than or equal to 1) decision makers (task requesters) give out fuzzy judgment matrixes with the same adaptive factor, the consistency check work of the fuzzy complementary judgment matrixes needs to meet the following two conditions:
first, m fuzzy judgment matrices F are examined x Satisfactory consistency of (c):
wherein
Is F x Corresponding to the weight vector.
Second, the satisfactory compatibility between the judgment matrices is checked:
I(F x ，F y ) F is less than or equal to F, x is not equal to y; the above expression of x, y =1,2,. ·, m (9) may indicate that in a case where the fuzzy complementary decision matrix Fx = (x =1,2,. ·, m) is consistently acceptable, their comprehensive decision matrix is also consistently acceptable.
(4) And all the alternative schemes are prioritized by calculating the comprehensive importance degree, so that a scientific decision basis is provided for a decision maker to select an optimal scheme.
6. Comprehensive evaluation value
The calculation of the comprehensive evaluation value is based on the evaluation values of the above various factors, and is used for measuring the comprehensive matching degree of the specific crowdsourcing worker corresponding to the execution of the test task t. For a particular candidate w of a task t to be performed I The integrated evaluation value of (a):
S(w I )＝S cap (w I ，t)*ω cap +S exp (w I ，t)*ω exp
+S wan (w I ，t)*ω wan +S cre (w I ，t)*ω cre (10)
i.e. the composite evaluation value is equal to the sum of the products of the respective influencing factor values and their corresponding weights.
Claims (5)
1. A crowdsourcing test task allocation method based on capability matching is characterized in that a multi-dimensional capability matching method is applied to task allocation of crowdsourcing tests, a final comprehensive evaluation value is obtained by calculating the sum of products of each influence factor value and corresponding weight of each influence factor value so as to screen out proper crowdsourcing workers, and the complete steps are as follows:
1) Analyzing the attribute of the test task according to the test requirement provided by the task contracting party, and analyzing the attribute of the test task, including task type, task complexity and the like;
2) Depicting the portrait of crowdsourcing workers according to the task attribute information obtained by analyzing in the step 1), depicting the portrait of crowdsourcing workers from four aspects of capability, experience, willingness degree and credibility, and measuring the influence degree of each factor on the distribution of test tasks by adopting triangular fuzzy numbers;
3) Determining the relative weight of the matching factors according to the relative importance degree between every two influencing factors described by the contracting party, and determining the weight of the influencing factors by using a fuzzy analytic hierarchy process, thereby quantifying fuzzy information and helping the contracting party to make a correct decision;
4) And calculating a comprehensive evaluation value according to the data set obtained in the step 1) 2) 3), and calculating the sum of products of each influence factor value and the corresponding weight of each influence factor value, so as to measure the comprehensive matching degree of the candidate crowdsourcing worker corresponding to the execution of the test task.
2. The method as claimed in claim 1, wherein in step 1), the software test crowdsourcing task published on the crowdsourcing platform comprises information of task publishers and test description of natural language provided by the test task publishersAnd (5) trial requirements. The test task is a test task set composed of three-level pages, and can be expressed as T = { T = { (T) 1 ，t 2 ...，t i ，...}(t i Representing the ith tertiary page). When testing task T for each of tasks T i The task types and the task difficulties are almost consistent, and the attributes of the task T are uniformly analyzed.
3. The method for allocating crowdsourcing test task based on capability matching as claimed in claim 1, wherein in step 2), the users willing to execute the test task registered on the crowdsourcing test platform are each accompanied by their identity information (sex, belonging unit, etc.) and historical test submission data (new registered user does not have), such as test report score, test report submission time, registration and actual task participation times, etc., and the worker likeness is characterized by the calculation of these data from four dimensions of capability, experience, willingness degree and credibility degree, which is expressed as w (t) = < S cap ，S exp ，S wan ，S cre Where t denotes the test task to be adapted, S cap Representing an executable capability evaluation value of a worker for a test task t, S exp An empirical evaluation value S representing the executable of a worker for a test task t wan Evaluation value, S, representing willingness of worker to test task t cre Representing the credibility evaluation value of the worker for the test task t; crowd sourcing of all worker components set of workers W = { W = 1 ，w 2 ，...，w j ，...}(w j Indicating the jth worker in the worker pool).
4. The method for distributing the crowdsourcing test tasks based on the capability matching as claimed in claim 1, wherein in step 3), the task distribution of the crowdsourcing test uses the candidate crowdsourcing workers to evaluate in terms of capability, experience, willingness and credibility, and each factor has different influence on the task distribution decision. Because, factor influence weight is defined to measure the influence degree of each factor, and omega is used respectively cap ，ω exp ，ω wan ，ω cre And the evaluation weights of the capability, experience, willingness degree and reputation degree are expressed. Each weight value is in the range of [0,1 ]]The sum of the four is 1, i.e. omega cap +ω exp +ω wan +ω cre And =1. The larger the value is, the larger the influence of the factor is, the value of 0 indicates that the factor does not work, and the value of 1 indicates that the factor plays a decisive role.
5. The method for distributing crowdsourcing test tasks based on capability matching as claimed in claim 1, wherein in step 4), the specific candidate w for the task t to be performed is calculated according to the data set obtained in step 1) 2) 3) I The comprehensive evaluation value of (1).
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110764872.5A CN115587726A (en) | 2021-07-06 | 2021-07-06 | Crowdsourcing test task allocation method based on capability matching |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110764872.5A CN115587726A (en) | 2021-07-06 | 2021-07-06 | Crowdsourcing test task allocation method based on capability matching |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115587726A true CN115587726A (en) | 2023-01-10 |
Family
ID=84771966
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110764872.5A Pending CN115587726A (en) | 2021-07-06 | 2021-07-06 | Crowdsourcing test task allocation method based on capability matching |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115587726A (en) |
-
2021
- 2021-07-06 CN CN202110764872.5A patent/CN115587726A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Zhou et al. | Type-2 fuzzy multi-objective DEA model: An application to sustainable supplier evaluation | |
| Auer et al. | Optimal project feature weights in analogy-based cost estimation: Improvement and limitations | |
| Chen et al. | Project delivery system selection of construction projects in China | |
| Liu et al. | Personnel selection using interval 2‐tuple linguistic VIKOR method | |
| CN112396211B (en) | Data prediction method, device, equipment and computer storage medium | |
| Shukla et al. | Estimating software maintenance effort: a neural network approach | |
| Izadikhah | Using the Hamming distance to extend TOPSIS in a fuzzy environment | |
| Mobin et al. | Integrating FAHP with COPRAS-G method for supplier selection (case study: an Iranian manufacturing company) | |
| CN115545388A (en) | Bridge state evaluation method and device, computer equipment and storage medium | |
| Wang et al. | A decision model for information systems outsourcing: using a multicriteria method | |
| CN113435713B (en) | Risk map compiling method and system based on GIS technology and two-model fusion | |
| Kalvakolanu et al. | Measuring job satisfaction levels of airport employees using entropy, critic and TOPSIS methods | |
| Javidnia et al. | Identifying factors affecting acceptance of new technology in the industry using hybrid model of UTAUT and FUZZY DEMATEL | |
| GB2583176A (en) | Prediction device, prediction program, and prediction method for predicting human judgments | |
| CN109190798B (en) | Cloud manufacturing service combination method | |
| Piwowarski et al. | Application of VMCM method (Vector Measure Construction Methods) to estimate consumer’s quality of life in EU countries–dynamic perspective | |
| CN115587726A (en) | Crowdsourcing test task allocation method based on capability matching | |
| Ennaceur et al. | Multi-criteria decision making method with belief preference relations | |
| CN113393023B (en) | Mold quality evaluation method, apparatus, device and storage medium | |
| CN105844374A (en) | Multi-index comprehensive evaluation method and multi-index comprehensive evaluation device based on hierarchical structure model | |
| CN108846577B (en) | Group task allocation method based on context analysis | |
| Amre et al. | Decision Making Approach in Recruitment using Neuro-Fuzzy System | |
| Teena et al. | Comprehensive trust based service selection model in federated cloud | |
| Wang et al. | Research on Credit Decision Issues of the Small and Medium-Sized Enterprises Based on TOPSIS and Hierarchical Cluster Analysis | |
| CN113379271B (en) | Open source platform-oriented abandoned contribution takeover recommendation method, device and equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |