CN108923951B - Crowdsourcing-based task allocation method for website barrier-free detection system - Google Patents
Crowdsourcing-based task allocation method for website barrier-free detection system Download PDFInfo
- Publication number
- CN108923951B CN108923951B CN201810425552.5A CN201810425552A CN108923951B CN 108923951 B CN108923951 B CN 108923951B CN 201810425552 A CN201810425552 A CN 201810425552A CN 108923951 B CN108923951 B CN 108923951B
- Authority
- CN
- China
- Prior art keywords
- detection
- counter
- cost
- detector
- task
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/14—Network analysis or design
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/14—Network analysis or design
- H04L41/145—Network analysis or design involving simulating, designing, planning or modelling of a network
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/14—Network analysis or design
- H04L41/147—Network analysis or design for predicting network behaviour
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/08—Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters
- H04L43/0805—Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters by checking availability
- H04L43/0817—Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters by checking availability by checking functioning
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Environmental & Geological Engineering (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
A task allocation method of a crowdsourcing-based website barrier-free detection system comprises the following steps: firstly, initializing a detection point counter and a detector counter to be 0; secondly, obtaining a detector with a minimum value from the prediction Cost matrix Cost (i, j); setting the corresponding numerical value of the task completion matrix as 1, adding 1 to the corresponding task counter and the counter of the detector, then updating the predicted cost matrix data, and then jumping to the second step, or else executing the fourth step; fourthly, if all testers have average detection points corresponding to the detection point counters, updating Cost (i, j) to be infinite; fifthly, for all the detection points, the corresponding counter values of the detectors are the total number of the detectors, and Cost (i, j) is updated to be infinite; and finally, for all the detection points, if the counter of the detector corresponding to each detection point is the total number of the detectors, the algorithm is interrupted, otherwise, the second step is carried out.
Description
Technical Field
The invention relates to the technical field of task allocation of a website barrier-free detection system in a crowdsourcing system.
Background
The vigorous development of the internet society leads the new trend of social informatization. Information exchange via the internet plays an indispensable role in information dissemination today, so that obtaining information on an equal basis is increasingly the target of pursuit of human society, and the position of information in the information society without hindrance is also increasingly important.
According to statistical data of the Chinese disabled people association, the number of various disabled people in China is about 8502 ten thousands by the end of 2010. With the high popularization of the internet and the increasing importance of the internet in daily life, for a large number of special people such as the elderly and the disabled, the lack of vision and hearing directly prevents the people from normally acquiring information on the internet. Therefore, it is important to make this part of people obtain internet information without obstacles.
The barrier-free detection system aims to find out interaction barriers encountered by disabled people when the disabled people acquire website information. Since some check points require compliance testing of people, it is costly to test a website. To address this issue, a crowd-sourced based system may be used in website unobstructed detection to capture the participant's contribution. However, some barrier-free tests are relatively complex and require some experience. Whereas accuracy is not ideal if inexperienced testers participate in complex, unobstructed testing. To address this issue, we propose a task scheduling policy (EDBA) based on result detection to better balance the tester's participation and experience. By using the detected historical data and the suggestions of experts, a minimum cost model is trained by a machine learning method to obtain an optimal task distribution map. Experiments in the Chinese website barrier-free detection system show that the method obtains high accuracy in website barrier-free detection. Meanwhile, the balancing strategy of the EDBA also enables new and old testers to participate in the barrier-free detection effectively.
Thousands of disabled users experience problems when surfing the internet. While much effort has been made to improve the surfing experience, there are still many problems for them to obtain information over a network. Many web site barrier-free detection methods and tools have been found to solve the problem of barrier-free access to web pages. The consistency test proposed by WAI is a very effective detection method for the problem of no obstacle, and the method can realize automatic and manual test methods based on guidelines. Although automatic testing is widely used, it does not cover all the detection points. Due to the lack of humanized settings, automatic testing cannot deal with situations with scenes and semantics. Such as for verification codes, error suggestions, and detection points that require human simulation operations, keyboard traps, skip navigation, etc. Manual testing is a remedy for uncovered monitoring points in automatic testing. But its cumbersome work makes it impractical to detect relatively large web sites.
The main challenge of web page barrier-free detection is that consistency detection of larger web sites will cost a huge expense. The proposal of the crowdsourcing-based barrier-free detection system (CB-WAES) is to overcome the bottleneck problem of the volunteer population in manual detection. However, many testing tasks require the subject to have a certain level of expertise and experience. This makes it a difficult problem to design an efficient task allocation strategy for crowd-sourced barrier-free detection systems. Most existing task allocation strategies based on crowdsourcing adopt random task allocation, and neglect the experience condition of a detected person. Although the detected person may perform well in a simpler detection test, the random task allocation may be ineffective in detecting a specific web page without any obstacle. Similarly, if the detected person is not good at detecting the content, we can not obtain the ideal detection result. For example, if we assign the detection points of keyboard traps to people with mobility disorders, or assign the detection points of verification codes to blind testers, they will have difficulty in completing the test.
To address this problem, the paper proposes a strategy called outcome-based detection (EDBA) for crowd-sourced barrier-free detection-based systems. Through historical detection data and expert suggestions, a cost model is trained through a least square method and a gradient decreasing mode to analyze experience and capability of a tested person, and then an optimal task distribution map is obtained through a greedy algorithm according to the analysis. Hence we can get a reasonable task allocation for the detection points. The EDBA is applied to a Chinese website barrier-free detection system and high detection precision is obtained. Meanwhile, the balance task allocation strategy enables both the old and the new to effectively participate in the detection process.
Disclosure of Invention
The invention provides a task allocation method of a crowdsourcing-based website barrier-free detection system, which aims to overcome the defects in the prior art.
The invention discloses a task allocation method of a crowdsourcing-based website barrier-free detection system, which comprises the following steps of:
1) and (5) training a cost model. Before proceeding with the task allocation strategy, we first train the Cost matrix Cost (i, j) with the least variance and gradient decreasing model from the historical data through machine learning.
2) Initialization: initializing the checkpoint counter and the checker counter to 0
3) Obtaining a minimum value detector: and obtaining the testers with the minimum value from the prediction Cost matrix Cost (i, j), if a plurality of testers exist, selecting the testers with the minimum website detection tasks in the task queue, and if the detection tasks are the same, randomly selecting the testers.
4) And (3) analyzing a detection result: if the detection point counter is smaller than the maximum detection point number of each detector and the number of the detectors is smaller than the total number of the detectors, setting the corresponding numerical value of the task completion matrix as 1, adding 1 to the corresponding task counter and the detector counter, then updating the predicted cost matrix data, and then skipping to the step 3; otherwise, execution of the algorithm is aborted. If the counter of the corresponding detection point of each detector is the average detection point number of the sub-detectors, the Cost (i, j) is updated to be infinite; if the counter values of the corresponding testers are the total number of testers for the detected points, the Cost (i, j) is updated to infinity.
The prediction cost matrix in the step 1) basically comprises the following components:
a cost matrix: prediction Cost matrix Cost (i, j): the examiner corresponds to eachWhen detecting the point, it has the corresponding detection cost value. I.e. Cost ═ θ (1) (E)h/TWh)+θ(2)(Gh/TWh)+θ(3)(Th/TWh) And θ means θ ═ CTC)-1CT(CAV), and C ═ Eh/(TWh),Gh/(TWh),Th/(TWh)]，EhThe number of web pages, G, in which erroneous detection occurs in the corresponding detection in the detection processhMeans the number of pages, T, discarded for detection in the detection processhRefers to detecting the number of web pages, TW, overtime in the detection processhRefers to the number of all web pages participating in the detection, TWhComprises Ah、Eh、Gh、ThAnd A ishThe number of the web pages detected normally is referred to; in the formula of θ, CTReferring to the transposition of the matrix C, CAV refers to the estimated cost value of the web page, and the cost value required by an expert for detecting a certain web page is estimated.
The detection point counter and the detector counter in the step 2) basically comprise the following components:
detection point counter: and defining a corresponding task counter (TaskCount (i) for each detector to represent the number of currently finished detection points, wherein i represents the number of the detectors.
The counter of the detector: and defining a corresponding detector counter for each detection point, wherein the counter is marked as WorkerCount (j), and represents the number of the detectors completing the test at the detection point, and j represents the number of the detection point.
The total detection number, the total number of detectors, the number of detectors required by the task, the average detection point number of the detectors and the task completion matrix in the step 4) are basically composed as follows:
total number of detections. The total number of detections can be used to measure the detection progress of the user and is marked as N.
Total number of subjects tested: i.e. the number of all users participating in the test, is denoted as M.
Number of detectors required for the task: i.e. the number of examiners required for each examination task, is denoted as K.
Average number of detection points of the examiners: the total number of detected objects/the average number of detected objects in the task is expressed as
A task completion matrix: and each detector completes setting 1 and does not complete setting 0 for the indication of whether the test application matrix corresponding to all the detection points is completed.
The invention has the advantage that the detection task is more effectively distributed in the crowdsourcing-based barrier-free detection system. Especially for detection tasks requiring corresponding special skills, the random allocation strategy will bring about a serious detection error problem. After the task allocation is carried out, the method can greatly avoid that the user is allocated to the detection task which is not good at the user, and greatly improve the detection quality of the website.
Drawings
FIG. 1 is a flow chart of the method of the present invention.
Detailed Description
The invention is further illustrated with reference to the accompanying drawings:
a task allocation method of a website barrier-free detection system in a crowdsourcing system comprises the following steps:
1) and (5) training a cost model. Before proceeding with the task allocation strategy, we first train the Cost matrix Cost (i, j) with the least variance and gradient decreasing model from the historical data through machine learning.
2) Initialization: initializing the checkpoint counter and the checker counter to 0
3) Obtaining a minimum value detector: and obtaining the testers with the minimum value from the prediction Cost matrix Cost (i, j), if a plurality of testers exist, selecting the testers with the minimum website detection tasks in the task queue, and if the detection tasks are the same, randomly selecting the testers.
4) And (3) analyzing a detection result: if the detection point counter is smaller than the maximum detection point number of each detector and the number of the detectors is smaller than the total number of the detectors, setting the corresponding numerical value of the task completion matrix as 1, adding 1 to the corresponding task counter and the detector counter, then updating the predicted cost matrix data, and then skipping to the step 3; otherwise, execution of the algorithm is aborted. If the counter of the corresponding detection point of each detector is the average detection point number of the sub-detectors, the Cost (i, j) is updated to be infinite; if the counter values of the corresponding testers are the total number of testers for the detected points, the Cost (i, j) is updated to infinity.
The prediction cost matrix in the step 1) basically comprises the following components:
a cost matrix: prediction Cost matrix Cost (i, j): the detector has corresponding detection cost value when corresponding to each detection point. I.e. Cost ═ θ (1) (E)h/TWh)+θ(2)(Gh/TWh)+θ(3)(Th/TWh) And θ means θ ═ CTC)-1CT(CAV), and C ═ Eh/(TWh),Gh/(TWh),Th/(TWh)]，EhThe number of web pages, G, in which erroneous detection occurs in the corresponding detection in the detection processhMeans the number of pages, T, discarded for detection in the detection processhRefers to detecting the number of web pages, TW, overtime in the detection processhRefers to the number of all web pages participating in the detection, TWhComprises Ah、Eh、Gh、ThAnd A ishThe number of the web pages detected normally is referred to; in the formula of θ, CTReferring to the transposition of the matrix C, CAV refers to the estimated cost value of the web page, and the cost value required by an expert for detecting a certain web page is estimated.
The detection point counter and the detector counter in the step 2) basically comprise the following components:
detection point counter: and defining a corresponding task counter (TaskCount (i) for each detector to represent the number of currently finished detection points, wherein i represents the number of the detectors.
The counter of the detector: and defining a corresponding detector counter for each detection point, wherein the counter is marked as WorkerCount (j), and represents the number of the detectors completing the test at the detection point, and j represents the number of the detection point.
The data in step 3) basically consist of:
the detector is: user participating in website detection
And (3) task queue: website detection task queue of detector
The total detection number, the total number of detectors, the number of detectors required by the task, the average detection point number of the detectors and the task completion matrix in the step 4) are basically composed as follows:
total number of detections. The total number of detections can be used to measure the detection progress of the user and is marked as N.
Total number of subjects tested: i.e. the number of all users participating in the test, is denoted as M.
Number of detectors required for the task: i.e. the number of examiners required for each examination task, is denoted as K.
Average number of detection points of the examiners: the total number of detected objects/the average number of detected objects in the task is expressed as
A task completion matrix: and each detector completes setting 1 and does not complete setting 0 for the indication of whether the test application matrix corresponding to all the detection points is completed.
The embodiments described in this specification are merely illustrative of implementations of the inventive concept and the scope of the present invention should not be considered limited to the specific forms set forth in the embodiments but rather by the equivalents thereof as may occur to those skilled in the art upon consideration of the present inventive concept.
Claims (3)
1. A task allocation method of a crowdsourcing-based website barrier-free detection system comprises the following steps:
1) training a cost model; before a task allocation strategy is carried out, firstly training a prediction Cost matrix Cost (i, j) with a minimum variance and a gradient decreasing model according to historical data through machine learning; the prediction cost matrix basically comprises the following components:
prediction Cost matrix Cost (i, j): when the detector corresponds to each detection point, the detector has corresponding detection Cost, namely Cost ═ theta (1) (E)h/TWh)+θ(2)(Gh/TWh)+θ(3)(Th/TWh) And θ means θ ═ CTC)-1CT(CAV), and C ═ Eh/(TWh),Gh/(TWh),Th/(TWh)]，EhThe number of web pages, G, in which erroneous detection occurs in the corresponding detection in the detection processhMeans the number of pages, T, discarded for detection in the detection processhRefers to detecting the number of web pages, TW, overtime in the detection processhRefers to the number of all web pages participating in the detection, TWhComprises Ah、Eh、Gh、ThAnd A ishThe number of the web pages detected normally is referred to; in the formula of θ, CTReferring to the transposition of C, CAV refers to the estimated cost value of a webpage, and is the estimation of the cost value required by an expert for detecting a certain webpage;
2) initialization: initializing a detection point counter and a detector counter to be 0;
3) obtaining a minimum value detector: obtaining the testers with the minimum value from the prediction Cost matrix Cost (i, j), if a plurality of testers exist, selecting the testers with the fewest website detection tasks in the task queue, and if the detection tasks are the same, randomly selecting the testers;
4) and (3) analyzing a detection result: if the detection point counter is smaller than the maximum detection point number of each detector and the number of the detectors is smaller than the total number of the detectors, setting the corresponding numerical value of the task completion matrix as 1, adding 1 to the corresponding task counter and the detector counter, then updating the predicted cost matrix data, and then skipping to the step 3; otherwise, stopping the algorithm execution; if the counter of the corresponding detection point of each detector is the average detection point number of the sub-detectors, the Cost (i, j) is updated to be infinite; if the counter values of the corresponding testers are the total number of testers for the detected points, the Cost (i, j) is updated to infinity.
2. The method for task allocation for a crowdsourced website accessibility detection system as claimed in claim 1, wherein: the detection point counter and the detector counter in the step 2) basically comprise the following components:
detection point counter: defining a corresponding task counter for each detector, and recording the task counter as TaskCount (i) to represent the number of currently finished detection points, wherein i represents the number of the detector;
the counter of the detector: and defining a corresponding detector counter for each detection point, wherein the counter is marked as WorkerCount (j), and represents the number of the detectors completing the test at the detection point, and j represents the number of the detection point.
3. The method for task allocation for a crowdsourced website accessibility detection system as claimed in claim 1, wherein: the total detection number, the total number of detectors, the number of detectors required by the task, the average detection point number of the detectors and the task completion matrix in the step 4) basically comprise the following components:
total number of detections: the total detection number can be used for measuring the detection progress of the user and is marked as N;
total number of subjects tested: namely, the number of all users participating in detection is recorded as M;
number of detectors required for the task: the number of testers required by each detection task is marked as K;
average number of detection points of the examiners: the total number of detected objects/the average number of detected objects in the task is expressed as
A task completion matrix: and each detector completes setting 1 and does not complete setting 0 for the indication of whether the test application matrix corresponding to all the detection points is completed.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810425552.5A CN108923951B (en) | 2018-05-07 | 2018-05-07 | Crowdsourcing-based task allocation method for website barrier-free detection system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810425552.5A CN108923951B (en) | 2018-05-07 | 2018-05-07 | Crowdsourcing-based task allocation method for website barrier-free detection system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN108923951A CN108923951A (en) | 2018-11-30 |
| CN108923951B true CN108923951B (en) | 2020-08-04 |
Family
ID=64403529
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810425552.5A Active CN108923951B (en) | 2018-05-07 | 2018-05-07 | Crowdsourcing-based task allocation method for website barrier-free detection system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108923951B (en) |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105824941A (en) * | 2016-03-21 | 2016-08-03 | 浙江大学 | WAQM-based optimal sampling method for network barrier-free detection |
| CN106327090A (en) * | 2016-08-29 | 2017-01-11 | 安徽慧达通信网络科技股份有限公司 | Real task allocation method applied to preference crowd-sourcing system |
| CN106777014A (en) * | 2016-12-08 | 2017-05-31 | 浙江大学 | A kind of accessible Detection task distribution method in self adaptation website based on classification |
| CN107220780A (en) * | 2017-06-16 | 2017-09-29 | 佛山科学技术学院 | Heterogeneous task Perform sequence optimization method in mass-rent system |
| CN107909262A (en) * | 2017-11-13 | 2018-04-13 | 北京航空航天大学 | A kind of acquisition methods of crowdsourcing cost complexity |
Family Cites Families (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20130197954A1 (en) * | 2012-01-30 | 2013-08-01 | Crowd Control Software, Inc. | Managing crowdsourcing environments |
| US20140108103A1 (en) * | 2012-10-17 | 2014-04-17 | Gengo, Inc. | Systems and methods to control work progress for content transformation based on natural language processing and/or machine learning |
| US11120373B2 (en) * | 2014-07-31 | 2021-09-14 | Microsoft Technology Licensing, Llc | Adaptive task assignment |
-
2018
- 2018-05-07 CN CN201810425552.5A patent/CN108923951B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105824941A (en) * | 2016-03-21 | 2016-08-03 | 浙江大学 | WAQM-based optimal sampling method for network barrier-free detection |
| CN106327090A (en) * | 2016-08-29 | 2017-01-11 | 安徽慧达通信网络科技股份有限公司 | Real task allocation method applied to preference crowd-sourcing system |
| CN106777014A (en) * | 2016-12-08 | 2017-05-31 | 浙江大学 | A kind of accessible Detection task distribution method in self adaptation website based on classification |
| CN107220780A (en) * | 2017-06-16 | 2017-09-29 | 佛山科学技术学院 | Heterogeneous task Perform sequence optimization method in mass-rent system |
| CN107909262A (en) * | 2017-11-13 | 2018-04-13 | 北京航空航天大学 | A kind of acquisition methods of crowdsourcing cost complexity |
Also Published As
| Publication number | Publication date |
|---|---|
| CN108923951A (en) | 2018-11-30 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Noltemeyer et al. | Relationship between school suspension and student outcomes: A meta-analysis | |
| Canzian et al. | Trajectories of depression: unobtrusive monitoring of depressive states by means of smartphone mobility traces analysis | |
| French et al. | Iterative purification and effect size use with logistic regression for differential item functioning detection | |
| Thomson | Qualitative research: validity | |
| JP2020509498A5 (en) | ||
| TWI567685B (en) | System and method of truly reflecting ability of testee through online test and storage medium storing the method | |
| Wang et al. | Temporal trends analysis of human brucellosis incidence in mainland China from 2004 to 2018 | |
| Barnett et al. | Reliability and validity of speech & pause measures during passage reading in ALS | |
| Clow et al. | A field-based indicator for determining the likelihood of Ixodes scapularis establishment at sites in Ontario, Canada | |
| He et al. | The effect of sample size on item parameter estimation for the partial credit model | |
| CN108923951B (en) | Crowdsourcing-based task allocation method for website barrier-free detection system | |
| US10665123B2 (en) | Smart examination evaluation based on run time challenge response backed by guess detection | |
| CN111680569B (en) | Attendance rate detection method, device, equipment and storage medium based on image analysis | |
| Medina et al. | Predicting pharmacy curriculum outcomes assessment performance using admissions, curricular, demographics, and preparation data | |
| Thompson | Testing and mnemonic strategies | |
| CN111507632A (en) | Method and device for evaluating developability and electronic equipment | |
| CN112348408A (en) | Online teachers and materials training and evaluation method | |
| US20200185110A1 (en) | Computer-implemented method and an apparatus for use in detecting malingering by a first subject in one or more physical and/or mental function tests | |
| Brandt et al. | Advances in Rasch modeling: New applications and directions: Guest Editorial | |
| Stupar et al. | A test-retest reliability study of the whiplash disability questionnaire in patients with acute whiplash-associated disorders | |
| Kaur et al. | Causal inference in higher education: Building better curriculums | |
| CN111460128B (en) | Computerized self-adaptive testing method based on cognitive diagnosis | |
| Barz et al. | Development of a skills-based instrument to measure scientific reasoning in medicine across different levels of expertise | |
| Scanlan et al. | The development and pilot testing of an occupational therapy group participation rating scale for inpatient mental health settings | |
| KR20140105112A (en) | Language evaluation system through network and its method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |