CN112508400B - Self-generation method of crowdsourcing collaborative iteration task - Google Patents

Abstract

The invention provides a self-generating method of a crowdsourcing collaborative iterative task, which firstly automatically distributes tasks for the first time in a form of filling blank questions, then redistributes the recovered results in a way of judging questions and converges the results, and finally generates the iterative task again in a way of selecting questions after sequencing and screening the converged results, and finally obtains the task result. The method provided by the invention ensures that the range of extracting the questions is as wide as possible when the questions are generated, so as to reduce the cost; and the problems of high repetition rate of task contents and the like are solved, and meanwhile, the process control is increased to improve the result quality of the task.

Description

Self-generation method of crowdsourcing collaborative iteration task

Technical Field

The invention relates to the technical field of computers, in particular to a self-generation method of a crowdsourcing collaborative iteration task.

Background

The crowdsourcing activity is started to rise along with the development and application of the internet, the phenomenon of crowdsourcing is firstly explained by a 'weck business model' proposed by a domestic scholars in 2055, the 'crowdsourcing' business development model is proposed by a U.S. reporter Howe in 2006, and the definition of the 'crowdsourcing' is similar, namely that part of tasks once completed by employees of a company is transferred to the public, and the task or work is started to be completed by the strength of the public. That is, enterprises distribute work, discover creatives, or solve technical problems using the internet.

At present, the crowdsourcing has wide application, for example, the crowdsourcing has good application examples in the aspects of image recognition, semantic annotation, picture search, relevance annotation, translation and the like. In order to enable crowdsourcing technology to play social production activities in the Internet at presentThe method plays a better role, and the establishment of a reasonable and effective crowdsourcing platform is one of the approaches for solving the problems. Existing research divides crowdsourcing participants into 3 roles, the sponsor, the crowdsourcing platform and the receiver. The subcontractor and the task provider submit the tasks to be completed to the crowdsourcing platform, and set the reward of the tasks, the deadline for completing the tasks and the like; the crowdsourcing platform is used for processing the tasks sent by the packet sender, dividing the tasks into subtasks and sending the subtasks to the masses (cooperative workers), and finally processing the results of the tasks executed by the masses and returning the results to the packet sender; and the receiver selects tasks according to own preference and completes the tasks by utilizing idle time. Accordingly, the art has become more and more accepted by those in the industry as a way that the packager can utilize the technology to achieve a higher quality of task results at a lower cost in time and money. However, in the process of completing the task, the receiver has the defects of insufficient capacity and 'taking a free car'^[1]That is, the task is completed without or with less effort to achieve a higher return, resulting in a poor task result. Therefore, in the crowd-sourced cooperative task, it is often necessary to utilize an iterative task generation technique^[3]And the task results obtained by the cooperative workers for the first time are iteratively distributed again, so that the task result quality is improved, and the cost brought by task distribution is reduced.

At present, the collaborative iterative task generation technology at home and abroad is mainly researched in two aspects. On one hand, the method is an iterative control technology, and focuses on designing a corresponding algorithm to iterate a certain task process. For example, in order to ensure the satellite mission planning to be correctly implemented, some scholars carry out iterative control on the mission blocks, and the efficiency and the reliability of the mission block generation are improved^[3]. Literature reference^[4]And eliminating samples with small contribution by using an iteration mechanism in the face classification algorithm, so that the algorithm has higher accuracy and stability. In the iterative control technique, researchers also optimize the iterative technique in the experiment to achieve the intended purpose. For example, two-dimension code generation is used for completing recommendation tasks in a Hash collaborative filtering algorithm by using an iterative technology^[5]. Literature reference^[6]Developed by softwareAnd the triple iteration model optimizes the iteration process of task allocation and improves the development efficiency of software. Yet to be learned^[7]The quality of the image is qualitatively and quantitatively evaluated and improved by using an iterative generation technology. The most common method is to carry out integral optimization on independent tasks by utilizing an iteration generation technology^[8]. Therefore, the iterative control technology can effectively reduce the cost and improve the efficiency and the task data quality, but the method is only suitable for the whole stage and the more complete task block in the task and is not suitable for the scattered task and the content of a certain task.

In this regard, currently, many researchers are working on another aspect of the collaborative iterative task generation technology, that is, iterating the content of the task. The method mainly obtains the final result by iterating the initially obtained task result again. In this way, the following two cases can be classified according to the coverage boundary of the content:

1) and iterating the whole task content. The method focuses on iterating the whole task result obtained for the first time or the whole task result obtained for the first time after being screened and used as an input value. The learners use the method in the image reconstruction technology, and the reconstruction result of the previous time is taken as new input to form an iterative algorithm, thereby improving the quality of the reconstructed image^[9]. When the whole task content is iterated, the quality of the initially obtained task result cannot be guaranteed, so that some researchers randomly generate the initial task result during research^[10]The method of (1) evaluating the quality of the task after the second iteration, or iterating the content of the whole task to extract data, so as to optimize the data generation process^[11](ii) a To document^[12]The known value is used as an input, the obtained result is subjected to two continuous iterations, and the results of the two iterations are compared, so that the effectiveness of the algorithm is proved.

2) An iteration is performed on a certain portion of the content. The method carries out iterative analysis on a certain part of contents in the task result, and is mainly used for obtaining a part of data results or diagnosing whether errors occur or not and judging the reasons of the errors^[13]. Literature reference^[14]Using iterationThe generation technique takes a certain value or vector as an iterative input to get a higher quality feature line. Bugni et al^[15]The estimated values in the algorithm are iterated to select the optimal value, and the efficiency is improved. The iterative generation technology is utilized to iterate the task content, so that the result quality of the task can be effectively improved, but the task content is easy to repeat in the iterative process, and redundant data is generated. To overcome the generation of redundant information during the experiment, the literature^[16]And performing parallel associated learning on each subtask by using an iteration technology, and simultaneously taking a generated result as the input of the next iteration so as to improve the performance of the new model.

Whether the whole task process is iterated or the contents of a certain task are iterated, the problems of low task generation efficiency, high content repetition rate, high cost and the like exist, and the method is also the key point and the difficulty point of the current research. The self-generation method of the collaborative iterative task mainly solves the following problems:

(1) the problem of high repetition rate of extracting task content in task iteration is solved

When a task result is obtained by using a traditional iterative task generation technology, the situation of repeated extraction of task content often exists, so that on one hand, the quality and efficiency of the iterative task result are reduced, and on the other hand, the task cost is increased.

(2) Solve the problem of result quality reduction caused by worker factors in task execution

Due to the heterogeneity of cooperative workers, the performance of the distributed tasks is insufficient, the tasks are not tried or cheated, and the quality of the task results is low. The method eliminates the screening of the workers in the task distribution, and how to judge the working quality of the workers in the task execution, thereby preventing the problem of the quality reduction of the final result of the task.

[1] Wangying, Chuibiepeng, childhood.A reputation based on-line incentive mechanism for mobile crowdsourcing systems [ J ] computer applications, 2016(18): 2121-.

[2]VallonC,Borrelli F.Task Decomposition for Iterative Learning Model Predictive Control[J].2019.

[3] Design and implementation of a satellite task interpretation closed loop simulation verification system [ J ] computer measurement and control, 2019,027(001):271-274.

[4] Li Guangyao, Wang Shi is the same, face recognition [ J ] based on sparse representation and elastic network computer application, 2017(03): 295-.

[5] Zhanyingjiu, Houyangzhong, pottery, a two-stage joint hash collaborative filtering algorithm [ J ] computer engineering, 2018,44(12):316-320.DOI:10.19678/j.issn.1000-3428.0049038.

[6] A triple iteration model (J) developed by MarangJun, Luo assist female agile software, electronic technology and software engineering, 2017,000(006) P.52-54.

[7]Thomas A K,Southard R,Curran J,et al.Comparing Fourth Generation Statistical Iterative Reconstruction Technique to Standard Filtered Back Projection in Pediatric Head Computed Tomography Examinations[J].Journal of Computer Assisted Tomography,2017:1.

[8]Hu B,Xie N,Zhao T,et al.Dynamic Task Scheduling Via Policy Iteration Scheduling Approach for Cloud Computing[J].Ksii Transactions on Internet&Information Systems,2017,11(3):1265-1278.

[9] Dongfanglin, Hongmingjia, Zhang Hai Biao, et al, cardiac magnetic resonance cine imaging method combining adjacent frame prediction [ J ]. automated chemical newspaper, 2018.

[10] The Zhang Yan-lan, the permissive force, the Wan Jian force, an iterative least square method [ J ] based on the F-J linear-nonlinear model solution, Wuhan university report of information science, 2019,44(12), 1816 and 1822.

[11]GeorgakarakosG,Lilius J.Recursive Task Generation for Scalable SDF Graph Execution on Multicore Processors[C]//PDP2020.2020.

[12]Khan N A,Mohammadi M,Stankovic I.Sparse reconstruction based on iterative TF domain filtering and Viterbi based IF estimation algorithm[J].Signal processing,2020,166(Jan.):107260.1-107260.12.

[13]Hui X U,Xuejun Z,Jilai L I.Hardware in the loop simulation test system for belt conveyor control system[J].Industry and Mine Automation,2017.

[14] Yangxankang, Panbodong, Tongweihua, grid surface characteristic line extraction algorithm [ J ] based on L0 optimization, computer engineering, 2019,45(7).

[15]BugniF,Bunting J.On the iterated estimation of dynamic discrete choice games[J].Social ence Electronic Publishing,2018.

[16] Liao auspicious writing, Chenze, Guilin, et al, argumentation mining method [ J ] computer science report based on multitask iterative learning, 2018.

Disclosure of Invention

The invention aims to solve the defects in the prior art and provide a crowdsourcing collaborative iteration task self-generation method which can reduce the task repetition rate and improve the task result quality under the condition of lower cost.

A self-generation method of a crowdsourcing collaborative iterative task comprises the following steps:

the method comprises the following steps: receiving a task issued by a packet sender, generating a gap filling question according to a gap filling question extraction method to reduce the question extraction repetition rate, and issuing the gap filling question to a packet receiver so that the packet receiver can complete the task; then collecting the task result of filling the blank questions completed by the receiver;

step two: generating a judgment question for the task result of the filling-up question according to a judgment question extraction method to improve the quality of the task result and further reduce the extraction repetition rate, and then distributing the judgment question to a packet receiver again for the packet receiver to complete the task; collecting the task results of the judgment questions completed by the bag receiver again;

step three: preprocessing the task result of the judgment question to obtain a task result of the judgment question with high quality; secondly, generating selection questions from the preprocessed task results of the judgment questions, controlling the result quality again, reducing the question extraction repetition rate, and finally, releasing the selection questions again to the packet receiver for the packet receiver to complete the tasks; and collecting the task results of the choice questions completed by the bag receiver to obtain the final task result.

Further, in the method for self-generating a crowdsourcing collaborative iterative task, in the first step, the method for extracting a null filling question generates a null filling question, and the method for extracting a null filling question includes the following steps:

step 1: constructing a content library of tasks to be completed; the Fillnum mark is used for each piece of content in the content library to represent the number of times of extracting the piece of content, the initial number is 0, and 1 is added to each extraction in the subsequent steps; one content of the task content library is used as a topic of a filling-in-the-blank question;

step 2: arranging all the filling-in-the-blank questions in an ascending order according to the marks Fillnum;

and step 3: selecting the first strip after the Fillnum is arranged in an ascending order, sending the first strip as a vacancy filling question to a packet receiver, and meanwhile, increasing the Fillnum value by 1; and collecting the task result of filling the blank questions completed by the receiver;

and 4, step 4: repeating the step 2 to the step 3 until the number of the extracted filling-up questions reaches the requirement upper limit of the task, and stopping the extraction;

and 5: acquiring a task result set of blank filling questions fed back by a receiver; the task result set of the gap filling questions consists of the extracted gap filling questions and answers corresponding to the gap filling questions.

Further, in the method for self-generating a crowdsourcing collaborative iteration task, in the second step, the task result of the gap filling question is generated into a judgment question according to a judgment question extraction method, and the method for generating the gap filling question comprises the following steps:

step 1: extracting an answer and a question corresponding to the answer from the task result set of the blank filling questions as the content of the issued judgment questions to be issued, marking the extracted answer with tofnum, representing the number of times of extracting the answer, wherein the initial value is 0, and increasing 1 in each extraction in the subsequent steps; judging results are distinguished in a grade mode;

step 2: arranging answers corresponding to all the questions in an ascending order of tofnum;

and step 3: selecting a first answer after tofnum is arranged in an ascending order and a question corresponding to the answer as a judgment question to be issued; simultaneously increasing the tofnum value of the answer by 1;

and 4, step 4: and (4) repeating the step (2-3) until the number of the extracted judgment questions reaches the upper limit of the number of the task extraction, and stopping the extraction.

Further, in the above method for self-generating a crowdsourcing collaborative iteration task, in the third step, the preprocessing includes performing statistical screening processing on the task results of the judgment questions:

step 1: counting the number of supported persons with different judgment levels for each collected judgment question to generate a statistical table;

the factors included in the statistical table are: the extracted judgment questions, answers corresponding to the judgment questions, judgment conditions corresponding to the answers, and scores corresponding to the answers calculated according to the judgment conditions;

the judging condition comprises the following steps: the number of persons who are judged to be out of check, the number of persons who are judged to be in check, the number of persons who are judged to be good, and the number of persons who are judged to be excellent;

the calculation formula of the score is as follows: goodness rate + excellence rate;

step 2: and sorting the scores in the statistical table from top to bottom, screening out a plurality of answers with high scores and a question corresponding to each answer as the question of the selected question, and issuing the question to a package receiver.

Further, as described above, in the crowd-sourced collaborative iterative task self-generation method, several of the answers include several choices depending on the choice question, and if there are several choices, there are several corresponding answers.

Further, in the above method for self-generating a crowdsourcing collaborative iteration task, in the third step, the preprocessed task results of the judgment questions are generated into the choice questions, and the method for generating the choice questions includes:

step 1: selecting a certain question and a plurality of answers corresponding to the question from the screened judgment question statistical table as selected question issuing contents to issue again, and marking the extracted question by choicenum and sur _ choice; wherein choicenum represents the number of times the question is extracted, sur _ choice represents that the question is set to true when meeting the condition of forming the choice question, and true represents that the question can be distributed to the receiver as the choice question;

step 2: arranging all the questions in ascending order according to choicenum;

and step 3: selecting a first question and a plurality of answers screened by the first question which satisfy sur _ choice as true after choicenum is arranged in an ascending order, serving as selection questions to be issued to a packet receiver, and increasing the choicenum value of the extracted question by 1; collecting the task results of the choice questions completed by the bag receiver;

and 4, step 4: repeating the step 2-3 until the number of the selected questions reaches the upper limit of the number of the task extraction, and stopping the extraction;

and 5: counting the number of people who select the answer according to the collected answers of the choice questions, and calculating a final scoring value according to the weight by combining the score;

the final scoring value calculation formula is: score weight 1+ select number weight 2;

step 6: and (4) sorting the answers with the first ranking and the questions corresponding to the answers from high to low according to the final scoring values to be used as final task results.

Has the advantages that:

(1) aiming at the problem of high repetition rate of task content extraction, a random extraction mode is not adopted in design, but an index is designed in a task list, ascending sequencing is carried out in a digital mode, and a first sequenced task is selected as a topic according to the extraction condition when a task is extracted. Therefore, the repetition rate of task extraction is ensured to be low, and the extracted task content coverage can be as wide as possible;

(2) aiming at the problem of low quality of task results caused by the untimely behavior of workers in tasks, when a task self-generating method is designed, the task results submitted by the workers are subjected to secondary iteration and distributed to different workers in the form of judgment questions, and the obtained results are selected to obtain the optimal results in the mode of selecting the questions, so that the problem that the quality of the results is influenced by the malicious behavior or the burst factor of cooperative workers is solved.

Drawings

FIG. 1 is a framework diagram of a crowdsourced participant;

FIG. 2 is a flow chart of a method for extracting a filling-in-blank question according to the present invention;

FIG. 3 is a flow chart of a method for generating judgment questions according to the present invention;

FIG. 4 is a flow chart of a choice question generation method of the present invention.

Detailed Description

In order to make the objects, technical solutions and advantages of the present invention more apparent, the technical solutions of the present invention are described clearly and completely below, and it is obvious that the described embodiments are some, not all embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

FIG. 1 is a framework diagram of a crowdsourced participant; as shown in fig. 1, the crowdsourcing participant comprises: the crowdsourcing platform can obtain a task result after the following three steps are finished according to the task content sent by the subcontractor and the self-generating method provided by the application, and the result is fed back to the subcontractor.

The invention mainly aims at the technical aspect of an iterative process, because the iterative aspect of task content is researched less and iterative content repeated extraction phenomenon exists in most cases, the invention provides an improved method for improving the repeated extraction phenomenon existing in task extraction, the improved method takes the crowdsourcing cooperative iterative task self-generation method provided by the application and takes the crowdsourcing platform as an execution main body, and the three steps are as follows:

the method comprises the following steps: receiving a task issued by a packet sender, generating a gap filling question according to a gap filling question extraction method shown in the figure 2, and issuing the gap filling question to a packet receiver so that the packet receiver can complete the task; then collecting the task result of filling the blank questions completed by the receiver;

step two: the task result of the filling-in-blank question is released to a packet receiver again in the form of a judgment question according to the judgment question extraction method shown in the figure 3 so that the packet receiver can complete the task; collecting the task results of the judgment questions completed by the bag receiver again;

step three: after the task results of the judgment questions are subjected to statistical screening processing, selecting questions are generated according to the graph 4 and are distributed to the packet receiver again in the form of the selecting questions so that the packet receiver can complete the tasks; and collecting the task results of the choice questions completed by the bag receiver to obtain the final task result.

When the crowdsourcing platform extracts the filling-in-the-blank topic, the extraction method is shown in fig. 2, and the method ensures that each topic is extracted at least once and avoids repeated extraction. The method is described as follows:

step 1: constructing a content library of tasks to be completed; the Fillnum mark is used for each piece of content in the content library to represent the number of times of extracting the piece of content, the initial number is 0, and 1 is added to each extraction in the subsequent steps; one content of the task content library is used as a topic of a filling-in-the-blank question;

step 2: arranging all the filling-in-the-blank questions in an ascending order according to the marks Fillnum;

and step 3: selecting the first strip after the Fillnum is arranged in an ascending order, sending the first strip as a vacancy filling question to a packet receiver, and meanwhile, increasing the Fillnum value by 1; and collecting the task result of filling the blank questions completed by the receiver;

and 4, step 4: repeating the step 2 to the step 3 until the number of the extracted filling-up questions reaches the requirement upper limit (namely the requirement value) of the task, namely stopping the extraction;

and 5: and acquiring a task result set of the gap filling questions fed back by the receiver, wherein the task result set of the gap filling questions consists of the extracted gap filling question and an answer corresponding to each gap filling question.

Specifically, when the iterative task self-generation method is designed, in order to ensure that the task extraction range is wide enough, a field fillnum is newly added in the task list library before the task is extracted, the field represents the times of the task appearing in the blank filling question, the default value is 0, when the task is extracted as the title of the blank filling question, the tasks are sorted according to the ascending order of the numbers in the field, and the first task meeting the extraction necessary conditions is selected. The fillnum value of this task after it appears as a null-fill topic is incremented by one. And circulating operation when the next gap filling question is extracted until the gap filling question does not appear any more. Therefore, the first step of the invention mainly solves the problem of how to reduce the repetition rate of content extraction when the task content iteration is carried out. In the step, the iterative content is extracted according to the value of the fillnum field, and after the extraction is finished, the value is added by one, and the extraction is possible until the fillnum value of the residual content is the same as the fillnum value of the residual content, so that the problem of high extraction repetition rate is solved, and the probability of re-extraction is reduced.

In the second step, the crowdsourcing platform generates a judgment question for the blank filling question result obtained in the first step according to the method shown in fig. 3, and sends the judgment question again to the receiver in a judgment question mode to complete the task. The method for generating the crowdsourcing platform judgment questions comprises the following steps:

step 1: extracting an answer and a question corresponding to the answer from the task result set of the blank filling questions as the content of the issued judgment questions to be issued, marking the extracted answer with tofnum, representing the number of times of extracting the answer, wherein the initial value is 0, and increasing 1 in each extraction in the subsequent steps; the judgment results are classified in a hierarchical manner, for example: excellent, good, passing and failing.

Step 2: arranging answers corresponding to all the questions in an ascending order of tofnum;

and step 3: selecting a first answer after tofnum is arranged in an ascending order and a question corresponding to the answer as a judgment question to be issued; simultaneously increasing the tofnum value of the answer by 1;

and 4, step 4: and (4) repeating the step (2-3) until the number of the extracted judgment questions reaches the upper limit of the number of the task extraction, and stopping the extraction.

Specifically, the design of the judgment questions mainly aims to judge the answer quality obtained by the cooperative workers in the gap filling questions for the first time, and the judgment questions have four grades: the worker can sort all answers according to four grades, thereby obtaining better results.

After the cooperative workers answer the blank filling questions, a result table is generated in the system, and all answers saved after the cooperative workers answer the blank filling questions are recorded in the result table. And simultaneously, a new field of tofnum is added in the table, the number of times of all answers appearing in the judgment questions is represented by a field, the default value is 0, when the answers of the tasks are extracted as the questions of the judgment questions, the answers are sorted according to the ascending order of numbers in the tofnum, the first answer meeting the extraction necessary conditions is selected, and the task corresponding to the answer is selected from the task library as the question of the judgment questions. This answer is added to the tofnum value after the title of the judgment question. And when the next judgment question is extracted, the operation is circulated until the judgment question does not appear any more.

Therefore, the purpose of step two in the present application is mainly to perform the first iteration on the task result obtained for the first time. In order to obtain a high-quality task result, when the task result is iterated for the first time, the problem that the iteration content repetition rate is too high is solved.

In the third step, the statistical screening processing of the task results of the judgment questions specifically comprises the following steps:

step 1: counting the number of supporters with different judgment grades (excellence, good, passing and failing) for each collected judgment question to generate a statistical table; thus, the factors included in the statistical table are: the extracted judgment questions, answers corresponding to the judgment questions, judgment conditions corresponding to the answers, and scores corresponding to the answers calculated according to the judgment conditions;

the judging condition comprises the following steps: the number of persons who are judged to be out of check, the number of persons who are judged to be in check, the number of persons who are judged to be good, and the number of persons who are judged to be excellent;

the calculation formula of the score is as follows: goodness rate + excellence rate; for example: good rate 0.4+ excellent rate 0.6

Step 2: sorting the scores in the statistical table from top to bottom, screening out a plurality of answers with high scores and a question corresponding to each answer as a question of a selected question, and issuing the questions to a package receiver; the plurality of choices includes several choices, and if the choice is one out of four, the answers of the four judgment questions are screened out as choices of the choice.

In the third step, after the result of the judgment questions is screened and counted, the selection questions are generated according to the method shown in fig. 4 and then are issued to the receiver to complete the task. The method for processing the selection questions by the crowdsourcing platform comprises the following steps:

step 1: and selecting a certain question and a plurality of answers corresponding to the question from the screened judgment question statistical table as selected question issuing contents to issue again, and marking the extracted question by choicenum and sur _ choice. Wherein choicenum represents the number of times the question is extracted, sur _ choicerepresents that the question meets the condition for forming the choice question, namely when the number of choice choices of the choice question (for example: 4) is reached, sur _ choicevalue is set to true, which represents that the question can be distributed to the receiver as the choice question;

step 2: arranging all the questions in ascending order according to choicenum;

and step 3: selecting a first question and a plurality of (number of options) answers screened by the first question, which satisfy sur _ choice as true after choicenum is arranged in an ascending order, as choice questions to be issued to a package receiver, and adding 1 to the choicenum value of the extracted question; collecting the task results of the choice questions completed by the bag receiver;

and 4, step 4: repeating the step 2-3 until the number of the selected questions reaches the upper limit of the number of the task extraction, and stopping the extraction;

and 5: counting the number of people who select the answer according to the collected answers of the choice questions, and calculating a final score value according to the weight by combining the score, wherein the final score value calculation formula is as follows: score weight 1+ selector weight 2, for example: score 0.6+ choosen 0.4;

step 6: and ranking the first answer and the question corresponding to the answer in a high-to-low ranking mode according to the final scoring value, namely the final task result.

Specifically, when the choice questions are designed, the better results in the judgment questions are judged again, the results obtained through the iterative judgment questions are used as options, and the workers obtain the optimal results after selection. After the cooperative workers finish the judgment questions, a judgment question statistical table is generated in the system, and the condition that the result answers of each task are judged by the workers is recorded in the table, wherein the condition comprises the number of people who judge that the answers are not qualified, the number of qualified people, the number of good people, the number of excellent people and the respective occupation ratio. And finally, sorting all answers of the task in a descending order according to the total scores of the ratio of the good number to the outstanding number.

Two fields are added in the task library: choicenum and Sur _ choice. And when the answers in the task exceed four, the value of the Sur _ choice automatically becomes true to indicate that the condition is met, and the task can appear as the choice question. Choicenum indicates the number of times the task appears in the choice question, and the default value is 0. And when the selected question is extracted, under the condition that the sum _ choice is true, sorting according to ascending order of Choicenum numbers, and selecting the first task and the answers four before the total ranking of the answers in the judgment question statistical table corresponding to the task as the questions of the selected question. This task is selected as the choice topic followed by the Choicenum value plus one. And when the next choice question is extracted, the operation is circulated until the choice question does not appear any more.

Therefore, the third step of the application mainly aims at the fact that the results of the primarily obtained task results after the first iteration judgment are gathered and screened and then iterated again, and therefore the final task result is selected. High-quality task results are obtained at a low cost while the problem of high repetition rate caused when task contents are extracted is solved.

In conclusion, the method provided by the invention not only solves the problem of high problem extraction repetition rate, but also improves the quality of task execution results.

Topic extraction repetition rate analysis

And (3) extracting according to descending order of values of preset fields by utilizing a collaborative iteration task self-generating technology when the titles are extracted every time, and adding one to the preset field value of the task after extraction is finished. The preset field has the same value until all tasks in the task library or results answered by workers are extracted, the extraction is started from the first piece of data in the next round of task extraction, and the operation is repeated sequentially. By adopting the method, the problem extracted in the next iteration can be prevented from being consistent with the problem extracted in the previous iteration, the repetition rate of task extraction can be reduced, and the maximization of the iteration range of the task can be ensured.

Task execution result quality analysis

Due to the heterogeneity of cooperative workers, whether the worker has cheating behavior or the result quality is reduced due to some burst factors cannot be guaranteed in the task execution process, and whether the task result quality is effective cannot be guaranteed. And finally, selecting the task result ranked in the first four in the same task, and performing iteration and redistribution to the workers to obtain the final judgment result. The final task result is selected after being judged for many times, and the optimal result can be effectively judged, so that the result quality of the cooperative task is improved, and the problem of low task result quality caused by workers is solved.

In summary, the application designs and realizes a task content self-generation method based on a mixed topic type aiming at the problems of content repetition, low result quality and low efficiency caused by task iteration in a crowdsourcing and collaboration system. The method can effectively solve the problems of high repetition rate caused by iterative content extraction, low result quality caused by cooperative worker factors in the task and the like, and can more efficiently obtain more accurate and higher-quality task results at lower cost. The method provides ideas and method reference for self-generation and iteration control of the crowdsourcing collaborative iteration task.

Example (b):

the content library of tasks to be completed constructed in this embodiment is exemplified by the following 3 items:

1. how do the weather today?

2. Now several points?

3. What book you like to see?

Question for filling one blank

The states of the three contents before the beginning of the gap filling question are shown in table 1: (Fillnum indicates the number of times the content has appeared in the gap filling question)

TABLE 1

Numbering	Content providing method and apparatus	Fillnum
			1	How do the weather today?	0
2	Now several points?	0
			3	What book you like to see?	0

Beginning of gap filling: (default three pieces of content are all used as fill-in-blank question if there are three pieces of content but only two pieces of content are needed to be distributed to the user, then only two times are selected in a loop)

For the first time:

the first piece of content (how is the weather of today:

TABLE 2

Numbering	Content providing method and apparatus	Fillnum
			1	How do the weather today?	1
2	Now several points?	0
			3	What book you like to see?	0

And (3) for the second time:

the states of the three pieces of content sorted in ascending order according to the field fillnum are shown in table 3:

TABLE 3

Numbering	Content providing method and apparatus	Fillnum
			2	Now several points?	0
3	You likeWhat book is seen?	0
			1	How do the weather today?	1

Select the first piece of content (now a few

The states of the three pieces of content after completion are as in table 4:

TABLE 4

Numbering	Content providing method and apparatus	Fillnum
			1	How do the weather today?	1
2	Now several points?	1
			3	What book you like to see?	0

And thirdly:

the states of the three pieces of content sorted in ascending order according to the field fillnum are shown in table 5:

TABLE 5

Numbering	Content providing method and apparatus	Fillnum
			3	What book you like to see?	0
1	How do the weather today?	1
			2	Now several points?	1

Select the first piece of content (what book you like to see

The states of the three pieces of content after completion are as in table 6:

TABLE 6

Numbering	Content providing method and apparatus	Fillnum
			1	How do the weather today?	1
2	Now several points?	1
			3	What book you like to see?	1

And ending the whole content selection period of the filling-up question and entering a second period.

The answers corresponding to the judgment questions (the matching degree of the judgment content and the answers is divided into four grades, namely, failing, passing, good and excellent) are obtained after three contents in the gap filling questions are distributed to the users.

For the first piece of content (how do today's weather: (tofnum represents the number of times the answer appears in the judgment question, score represents the score obtained by the answer)

TABLE 6

Numbering	Answer to the question	Tofnum	score
				1	Kunming is a very good weather today	0	60
2	Shanghai today rains	0	65
				3	Beijing is very cloudy today	0	70
4	Nanjing today's many miles in sunny days	0	40
				5	Sichuan is snowing today	0	20

For the second piece of content (now a few:

TABLE 7

Numbering	Answer to the question	Tofnum	score
				1	Now at 9 am	0	72
2	Now at 12 am	0	43
				3	Now at 3 pm	0	57
4	Now 10 o' clock in the evening	0	84
				5	Now 2 am	0	62

For the third piece of content (what book you like to see:

TABLE 8

Numbering	Answer to the question	Tofnum	score
				1	"Xiaowangzi	0	63
2	'Sanmaoliliulang' recording	0	26
				3	Half-hour cartoon	0	48
4	Kite chasing man	0	56
				5	Three of us	0	39

All answers and states obtained after distributing the task content in the gap filling question are as in table 9:

TABLE 9

Starting a judgment question:

for the first time:

sorting according to the ascending order of the fields tofnum, selecting a first answer (how good the weather is today) to search for corresponding content (how the weather is today.

All answer states after completion are as in table 10:

watch 10

Numbering	Answer to the question	Content number corresponding to answer	Tofnum
				1	Kunming is a very good weather today	1	1
2	Shanghai today rains	1	0
				3	Beijing is very cloudy today	1	0
4	Nanjing today's many miles in sunny days	1	0
				5	Sichuan is snowing today	1	0
6	Now at 9 am	2	0
				7	Now at 12 am	2	0
8	Now at 3 pm	2	0
				9	Now 10 o' clock in the evening	2	0
10	Now 2 am	2	0
				11	"Xiaowangzi	3	0
12	'Sanmaoliliulang' recording	3	0
				13	Half-hour cartoon	3	0
14	Kite chasing man	3	0
				15	Three of us	3	0

And (3) for the second time:

the status of all answers sorted in ascending order by field tofnum is shown in table 11:

TABLE 11

Numbering	Answer to the question	Content number corresponding to answer	Tofnum
				2	Shanghai today rains	1	0
3	Beijing is very cloudy today	1	0
				4	Nanjing today's many miles in sunny days	1	0
5	Sichuan is snowing today	1	0
				6	Now at 9 am	2	0
7	Now at 12 am	2	0
				8	Now at 3 pm	2	0
9	Now 10 o' clock in the evening	2	0
				10	Now 2 am	2	0
11	"Xiaowangzi	3	0
				12	'Sanmaoliliulang' recording	3	0
13	Half-hour cartoon	3	0
				14	Kite chasing man	3	0
15	Three of us	3	0
				1	Kunming is a very good weather today	1	1

And selecting a first answer (Shanghai and raining today) to search corresponding contents (how do the weather of today.

All answer states after completion are as in table 12:

TABLE 12

Numbering	Answer to the question	Content number corresponding to answer	Tofnum
				1	Kunming is a very good weather today	1	1
2	Shanghai today rains	1	1
				3	Beijing is very cloudy today	1	0
4	Nanjing today's many miles in sunny days	1	0
				5	Sichuan is snowing today	1	0
6	Now at 9 am	2	0
				7	Now at 12 am	2	0
8	Now at 3 pm	2	0
				9	Now 10 o' clock in the evening	2	0
10	Now it is2 am	2	0
				11	"Xiaowangzi	3	0
12	'Sanmaoliliulang' recording	3	0
				13	Half-hour cartoon	3	0
14	Kite chasing man	3	0
				15	Three of us	3	0

Third time … …

……

……

The fifteenth time

The status of all answers sorted in ascending order by field tofnum is shown in table 13:

watch 13

Numbering	Answer to the question	Content number corresponding to answer	Tofnum
				15	Three of us	3	0
1	Kunming is a very good weather today	1	1
				2	Shanghai today rains	1	1
3	Beijing is very cloudy today	1	1
				4	Nanjing today's many miles in sunny days	1	1
5	Sichuan is snowing today	1	1
				6	Now at 9 am	2	1
7	Now at 12 am	2	1
				8	Now at 3 pm	2	1
9	Now 10 o' clock in the evening	2	1
				10	Now 2 am	2	1
11	"Xiaowangzi	3	1
				12	'Sanmaoliliulang' recording	3	1
13	Half-hour cartoon	3	1
				14	Kite chasing man	3	1

The first answer (three of us) is selected to find the corresponding content (what book you like to see.

All answer states after completion are as in table 14:

TABLE 14

Numbering	Answer to the question	Content number corresponding to answer	Tofnum
				1	Kunming is a very good weather today	1	1
2	Shanghai provinceToday rain	1	1
				3	Beijing is very cloudy today	1	1
4	Nanjing today's many miles in sunny days	1	1
				5	Sichuan is snowing today	1	1
6	Now at 9 am	2	1
				7	Now at 12 am	2	1
8	Now at 3 pm	2	1
				9	Now 10 o' clock in the evening	2	1
10	Now 2 am	2	1
				11	"Xiaowangzi	3	1
12	'Sanmaoliliulang' recording	3	1
				13	Half-hour cartoon	3	1
14	Kite chasing man	3	1
				15	Three of us	3	1

At this point, the whole question judging period is completed, and the second period is entered.

Question selection

Assuming that after the content and the answers are distributed to the users in the second-stage judgment questions, the judgment result of each answer is obtained, which is specifically shown in table 15:

(the score is 0.4+ 0.6 of good rate)

Watch 15

When the content and the result are judged by the user and then the received result is more than or equal to 4, the content can be distributed to the user for selection as a choice. A field (sur _ choice ═ true) indicates that the piece of content can appear in the choice question as a choice question.

Assuming that all answers of three contents (5 answers per content) appear in the judgment questions and are distributed to the user for judgment, and a corresponding judgment result is obtained, the state of the contents is as shown in table 16:

TABLE 16

Numbering	Content providing method and apparatus	choicenum	sur_choice
				1	How do the weather today?	0	True
2	Now several points?	0	True
				3	What book you like to see?	0	True

Choice question start:

for the first time:

selecting the first content after sorting according to the ascending order of the field choicenum (how like today is weather

Meanwhile, corresponding answers are searched in a judgment question result table, and the result is as shown in a table 17:

TABLE 17

And (4) selecting the first four answers and contents as the questions of the selection questions in descending order of scores, and distributing the questions to the user to enable the user to select an answer which the user considers to be correct. So the case of the ranked answers is as in table 18:

watch 18

The first four answers and the first piece of content are selected as the title of the choice question, and the results are shown in table 19:

watch 19

How do the weather today? (Contents)
	Shanghai today raining (option one)
Sichuan snowing today (option two)
	Kunming today's weather is very good (option three)
Nanjing today Wanli sunny day (four options)

The state of all contents at this time is shown in table 20:

watch 20

Numbering	Content providing method and apparatus	choicenum	sur_choice
				1	How do the weather today?	1	True
2	Now several points?	0	True
				3	What book you like to see?	0	True

And (3) for the second time:

select the first piece of content (now a few

Meanwhile, corresponding answers are searched in a judgment question result table and sorted according to the descending order of scores, and the results are shown in a table 21:

TABLE 21

Numbering	Answer to the question	Content number corresponding to answer	Not shorter than the number of grids	And the number of cells	Good number	Number of excellences	Score of
								10	Now 2 am	2	0	0	5	5	0.5
8	Now at 3 pm	2	1	0	3	5	0.42
								7	Now at 12 am	2	1	2	2	5	0.38
9	Now 10 o' clock in the evening	2	1	3	2	4	0.32
								6	Now at 9 am	2	1	8	0	1	0.06

The first four answers and the first piece of content are selected as the title of the choice question, and the results are shown in table 23:

TABLE 23

Now several points? (Contents)
	Now 2 am (option one)
Now 3 pm (option two)
	At noon 12 o' clock (option three)
Now 10 o' clock evening (option four)

The state of all contents at this time is shown in table 24:

watch 24

Numbering	Content providing method and apparatus	choicenum	sur_choice
				1	How do the weather today?	1	True
2	Now several points?	1	True
				3	What book you like to see?	0	True

And thirdly:

select the first piece of content (now a few

Meanwhile, corresponding answers are searched in a judgment question result table and sorted according to the descending order of scores, and the result is shown in a table 25:

TABLE 25

Numbering	Answer to the question	Content number corresponding to answer	Not shorter than the number of grids	And the number of cells	Good number	Number of excellences	Score of
								11	"Xiaowangzi	3	1	1	2	6	0.44
12	'Sanmaoliliulang' recording	3	1	3	3	3	0.3
								15	Three of us	3	2	5	2	1	0.12
13	Half-hour cartoon	3	9	0	0	1	0.06
								14	Kite chasing man	3	1	8	1	0	0.04

The first four answers and the first piece of content are selected as the title of the choice question, and the results are shown in table 26:

watch 26

The state of all contents at this time is shown in table 27:

watch 28

Numbering	Content providing method and apparatus	choicenum	sur_choice
				1	How do the weather today?	1	True
2	Now several points?	1	True
				3	What book you like to see?	1	True

And the content period of the whole choice question is completed, and the second period is entered.

Therefore, assume that the final choice questions are given in Table 29: the final score is a combination of score and the number of people selected, calculated as: score 0.6+ number 0.4)

Watch 29

The end result of the content is as in table 30:

watch 30

Numbering	Content providing method and apparatus	Answer to the question
			1	Today' sHow does the weather?	Shanghai today rains
2	Now several points?	Now 10 o' clock in the evening
			3	What book you like to see?	"Xiaowangzi

Finally, it should be noted that: the above examples are only intended to illustrate the technical solution of the present invention, but not to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions of the embodiments of the present invention.

Claims (5)

1. A self-generation method of a crowdsourcing collaborative iterative task is characterized by comprising the following steps:

the method comprises the following steps: receiving a task issued by a packet sender, generating a gap filling question according to a gap filling question extraction method to reduce the question extraction repetition rate, and issuing the gap filling question to a packet receiver so that the packet receiver can complete the task; then collecting the task result of filling the blank questions completed by the receiver;

step two: generating a judgment question for the task result of the filling-up question according to a judgment question extraction method to improve the quality of the task result and further reduce the extraction repetition rate, and then distributing the judgment question to a packet receiver again for the packet receiver to complete the task; collecting the task results of the judgment questions completed by the bag receiver again;

step three: preprocessing the task result of the judgment question to obtain a task result of the judgment question with high quality; secondly, generating selection questions from the preprocessed task results of the judgment questions, controlling the result quality again, reducing the question extraction repetition rate, and finally, releasing the selection questions to the packet receiver again for the packet receiver to complete the tasks; collecting the task results of the choice questions completed by the bag receiver to obtain the final task result;

in the third step, the preprocessing comprises the following steps of carrying out statistical screening processing on the task results of the judgment questions:

step 1: counting the number of supported persons with different judgment levels for each collected judgment question to generate a statistical table;

the factors included in the statistical table are: the extracted judgment questions, answers corresponding to the judgment questions, judgment conditions corresponding to the answers, and scores corresponding to the answers calculated according to the judgment conditions;

the judging condition comprises the following steps: the number of persons who are judged to be out of check, the number of persons who are judged to be in check, the number of persons who are judged to be good, and the number of persons who are judged to be excellent;

the calculation formula of the score is as follows: goodness rate + excellence rate;

step 2: and sorting the scores in the statistical table from top to bottom, screening out a plurality of answers with high scores and a question corresponding to each answer as the question of the selected question, and issuing the question to a package receiver.

2. The method for self-generating a crowdsourcing collaborative iterative task according to claim 1, wherein in the first step, the method for extracting a filling-up question generates a filling-up question, and the method for extracting a filling-up question comprises the following steps:

step 1: constructing a content library of tasks to be completed; the Fillnum mark is used for each piece of content in the content library to represent the number of times of extracting the piece of content, the initial number is 0, and 1 is added to each extraction in the subsequent steps; one content of the task content library is used as a topic of a filling-in-the-blank question;

step 2: arranging all the filling-in-the-blank questions in an ascending order according to the marks Fillnum;

and step 3: selecting the first strip after the Fillnum is arranged in an ascending order, sending the first strip as a vacancy filling question to a packet receiver, and meanwhile, increasing the Fillnum value by 1; and collecting the task result of filling the blank questions completed by the receiver;

and 4, step 4: repeating the step 2 to the step 3 until the number of the extracted filling-up questions reaches the requirement upper limit of the task, and stopping the extraction;

and 5: acquiring a task result set of blank filling questions fed back by a receiver; the task result set of the gap filling questions consists of the extracted gap filling questions and answers corresponding to the gap filling questions.

3. The method for self-generating a crowdsourcing cooperative iterative task according to claim 1, wherein in the second step, the task result of the task of filling the vacancy question is generated into a judgment question according to a judgment question extraction method, and the method for self-generating the task result of filling the vacancy question comprises the following steps:

step 1: extracting an answer and a question corresponding to the answer from the task result set of the blank filling questions as the content of the issued judgment questions to be issued, marking the extracted answer with tofnum, representing the number of times of extracting the answer, wherein the initial value is 0, and increasing 1 in each extraction in the subsequent steps; judging results are distinguished in a grade mode;

step 2: arranging answers corresponding to all the questions in an ascending order of tofnum;

and step 3: selecting a first answer after tofnum is arranged in an ascending order and a question corresponding to the answer as a judgment question to be issued; simultaneously increasing the tofnum value of the answer by 1;

and 4, step 4: and (4) repeating the step (2-3) until the number of the extracted judgment questions reaches the upper limit of the number of the task extraction, and stopping the extraction.

4. The method of claim 1, wherein the plurality of answers includes several choices depending on the choice question, and if there are several choices, there are several answers.

5. The method for self-generating the crowdsourcing collaborative iterative task according to claim 1, wherein in the third step, the preprocessed task results of the judgments are generated into the choice questions, and the method for generating the choice questions comprises:

step 1: selecting a certain question and a plurality of answers corresponding to the question from the screened judgment question statistical table as selected question issuing contents to issue again, and marking the extracted question by choicenum and sur _ choice; wherein choicenum represents the number of times the question is extracted, sur _ choice represents that the question is set to true when meeting the condition of forming the choice question, and true represents that the question can be distributed to the receiver as the choice question;

step 2: arranging all the questions in ascending order according to choicenum;

and step 3: selecting a first question and a plurality of answers screened by the first question which satisfy sur _ choice as true after choicenum is arranged in an ascending order, serving as selection questions to be issued to a packet receiver, and increasing the choicenum value of the extracted question by 1; collecting the task results of the choice questions completed by the bag receiver;

and 4, step 4: repeating the step 2-3 until the number of the selected questions reaches the upper limit of the number of the task extraction, and stopping the extraction;

and 5: counting the number of people who select the answer according to the collected answers of the choice questions, and calculating a final scoring value according to the weight by combining the score;

the final scoring value calculation formula is: score weight 1+ select number weight 2;

step 6: and (4) sorting the answers with the first ranking and the questions corresponding to the answers from high to low according to the final scoring values to be used as final task results.

Images