CN107767055A - A kind of mass-rent result assemblage method and device based on collusion detection - Google Patents

Abstract

The invention discloses a crowdsourcing result aggregation method and device based on collusion detection. The method includes: collecting the answer sets of each worker for the task set from the crowdsourcing platform; calculating the aggregation result of the answer set, and calculating the Aggregating the result and the consistency parameter of each worker's answer; determining the repeated answer set from the answer set, and calculating the worker ability change rate corresponding to each repeated answer set based on the consistency parameter of the answer of each worker; for A repeated answer set whose rate of change of worker ability is less than or equal to a preset threshold is determined to be normally generated and the repeated answer set is retained in the answer set; for a repeated answer set whose rate of change of worker ability is greater than a preset threshold , determining that the repeated answer set is generated by collusion and deleting the repeated answer set from the answer set; obtaining an updated answer set, and calculating an aggregation result of the updated answer set.

Description

A crowdsourcing result aggregation method and device based on collusion detection

technical field

The present invention relates to the technical field of crowdsourcing, in particular to a method and device for converging crowdsourcing results based on collusion detection.

Background technique

Crowdsourcing is a rapidly developing field that aims to use human cognitive advantages to solve problems that are difficult for computers. Crowdsourcing general platforms such as CrowdFlower and AMT are widely used for general data processing tasks such as sentiment analysis, handwriting recognition and image annotation. Since workers may return low-quality results, a central issue in crowdsourcing is to guarantee the quality of results. A widely adopted method to control quality is result pooling, which first assigns each task to multiple workers and then uses inference algorithms to pool the results returned by the workers. Taking image labeling as an example, an image is assigned to multiple workers, and then these workers each provide labels describing the content of the image. Finally, a high-quality result is pooled from all collected labels by voting or other inference methods.

In crowdsourcing, in order to get more rewards and pay less labor, conspirators form collusion teams outside the platform through SMS, WeChat, phone calls, forums and even face-to-face communication. In a colluding team, only one worker tackles a task and the other workers copy his answers. All workers in the final team provide the same answer. These malicious repeated answers will dominate the answers provided by normal workers in the result aggregation, reducing the quality of the results. For example, if a task is assigned to five workers, if three of them collude, the results will be aggregated using the majority voting method, and the final pooled result will be equal to the result provided by the colluders.

From the above, it can be seen that duplicate answers produced by collusion are detrimental to the result quality of general tasks on general platforms. However, existing collusion detection algorithms cannot effectively detect and eliminate the negative impact of such collusion.

Contents of the invention

In order to solve the above technical problems, embodiments of the present invention provide a method and device for gathering crowdsourcing results based on collusion detection.

The crowdsourcing result aggregation method based on collusion detection provided by the embodiment of the present invention includes:

Collect the answer sets of each worker for the set of tasks from the crowdsourcing platform;

calculating an aggregated result of the set of answers, and calculating a consistency parameter between the aggregated result and each worker's answer;

Determining a repeated answer set from the answer set, and calculating a worker capability change rate corresponding to each repeated answer set based on the consistency parameters of the answers of the workers;

For a repeated answer set whose worker ability change rate is less than or equal to a preset threshold, determine that the repeated answer set is normally generated and retain the repeated answer set in the answer set;

For repeated answer sets whose rate of change of worker ability is greater than a preset threshold, determine that the repeated answer sets are produced by collusion and delete the repeated answer sets from the answer sets;

After retaining or deleting each repeated answer set, an updated answer set is obtained, and an aggregation result of the updated answer set is calculated.

In the embodiment of the present invention, the calculation of the consistency parameter of the aggregation result and the answers of each worker includes:

The consensus parameter for the aggregated results and the answers of each worker is calculated based on the following formula:

Among them, P _i is the aggregation result and the consistency parameter of worker i and L _i is the answer returned by worker i corresponding to the task set, Aggregated results for answer sets.

In the embodiment of the present invention, the calculation of the worker ability change rate corresponding to each repeated answer set based on the consistency parameters of the answers of each worker includes:

calculating the first variance of the consistency parameter of the answers of the workers while retaining duplicate answer sets in the answer sets;

calculating the second variance of the consistency parameter of the answers of the workers when the repeated answer sets are deleted in the answer sets;

Based on the first variance and the second variance, calculate the worker capability change rate corresponding to the repeated answer set.

In the embodiment of the present invention, when the calculation retains repeated answer sets in the answer set, the first variance of the consistency parameter of the answers of the workers includes:

Keep duplicate answer sets in the answer set by the following formula , the first variance of the consistency parameter of each worker's answer:

Among them, Var(P) is the first variance, E(P) is the average value of the consistency parameters of each worker, P _i is the aggregation result and the consistency parameter of worker i and collection of answers.

In the embodiment of the present invention, when the calculation deletes duplicate answer sets in the answer set, the second variance of the consistency parameter of the answers of the workers includes:

Delete duplicate answer sets in the answer set by the following formula , the second variance of the consistency parameter of each worker's answer:

Among them, Var(P ^k ) is the second variance, Ε(P ^k ) is the average value of the consistency parameters of each worker, P _i ^k is the aggregation result and the consistency parameter of worker i and to delete collection of answers.

In the embodiment of the present invention, the calculation of the rate of change of worker ability corresponding to the repeated answer set based on the first variance and the second variance includes:

Calculate duplicate answer sets based on the following formula Corresponding rate of change in worker capacity:

in, is the rate of change in worker capacity.

The crowdsourcing result aggregation device based on collusion detection provided by the embodiment of the present invention includes:

The collection module is used to collect the answer sets of each worker for the set of tasks from the crowdsourcing platform;

a consistency calculation module, configured to calculate the aggregation result of the answer set, and calculate the consistency parameter between the aggregation result and the answers of each worker;

A worker ability change rate module, configured to determine a repeated answer set from the answer set, and calculate the worker ability change rate corresponding to each repeated answer set based on the consistency parameters of the answers of each worker;

The collusion detection module is used to determine that the repeated answer set is normally generated and retain the repeated answer set in the answer set for the repeated answer set whose worker ability change rate is less than or equal to the preset threshold; for the worker ability change rate A duplicate answer set greater than a preset threshold, determining that the duplicate answer set is generated by collusion and deleting the duplicate answer set from the answer set;

The aggregation module is configured to obtain an updated answer set after retaining or deleting each repeated answer set, and calculate an aggregation result of the updated answer set.

In the embodiment of the present invention, the consistency calculation module is specifically used to calculate the consistency parameters of the aggregation result and each worker's answer based on the following formula:

Among them, P _i is the aggregation result and the consistency parameter of worker i and L _i is the answer returned by worker i corresponding to the task set, Aggregated results for answer sets.

In the embodiment of the present invention, the worker ability change rate module includes:

a first variance calculation unit, configured to calculate the first variance of the consistency parameter of the answers of the workers when the repeated answer set is reserved in the answer set;

The second variance calculation unit is used to calculate the second variance of the consistency parameter of the answers of the workers when the duplicate answer set is deleted from the answer set;

A worker capability change rate calculation unit, configured to calculate the worker capability change rate corresponding to the repeated answer set based on the first variance and the second variance.

In the embodiment of the present invention, the first variance calculation unit is specifically used to calculate and retain repeated answer sets in the answer set by the following formula: , the first variance of the consistency parameter of each worker's answer:

Among them, Var(P) is the first variance, E(P) is the average value of the consistency parameters of each worker, P _i is the aggregation result and the consistency parameter of worker i and collection of answers.

In the embodiment of the present invention, the second variance calculation unit is specifically used to calculate and delete duplicate answer sets in the answer set by the following formula: , the second variance of the consistency parameter of each worker's answer:

Among them, Var(P ^k ) is the second variance, Ε(P ^k ) is the average value of the consistency parameters of each worker, P _i ^k is the aggregation result and the consistency parameter of worker i and to delete collection of answers.

In the embodiment of the present invention, the worker ability change rate calculation unit is specifically used to calculate repeated answer sets based on the following formula Corresponding rate of change in worker capacity:

in, is the rate of change in worker capacity.

Adopting the technical solutions of the embodiments of the present invention, (1) Different from spatio-temporal crowdsourcing and social network scenarios, in general platforms, the characteristics of answers to general tasks are unknown. Therefore, the embodiment of the present invention introduces the concept of consistency between workers' answers and aggregation results to describe the influence of repeated answers produced by collusion on result aggregation.

(2) Different from the collusion detection algorithm based on similarity in the e-commerce platform, the embodiment of the present invention proposes a collusion detection method based on the change rate of worker performance, which can determine the collusion in the answer set containing normal repeated answers Duplicate answers generated. (3) The embodiment of the present invention proposes a crowdsourcing result aggregation method for collusion detection, which can effectively eliminate the negative impact of collusion behavior on result aggregation.

Description of drawings

FIG. 1 is a schematic diagram of a crowdsourcing framework based on collusion detection according to an embodiment of the present invention;

FIG. 2 is a schematic flow diagram of a crowdsourcing result aggregation method based on collusion detection according to an embodiment of the present invention;

3 is a schematic diagram of the structural composition of a crowdsourcing result aggregation device based on collusion detection according to an embodiment of the present invention;

Fig. 4 is a schematic diagram of the result of the worker ability change rate module according to the embodiment of the present invention.

Detailed ways

In order to understand the characteristics and technical contents of the embodiments of the present invention in more detail, the implementation of the embodiments of the present invention will be described in detail below in conjunction with the accompanying drawings. The attached drawings are only for reference and description, and are not intended to limit the embodiments of the present invention.

Existing collusion detection algorithms cannot effectively detect and eliminate the negative impact of collusion, mainly for the following reasons:

(1) Collusion detection algorithms in spatio-temporal crowdsourcing and social networks need to extract some features of the data to detect collusion. However, these features are difficult to obtain on general-purpose crowdsourcing platforms.

(2) The detection algorithm in the e-commerce platform is mainly based on the similarity between the answers provided by each pair of workers to detect collusion. Due to the repeated answers of tasks in the general platform are divided into normal repetitions and collusion repetitions. Workers show high ability in some simple tasks, when many of the repeated answers are normally produced. Testing for collusion based on the similarity of answers can misjudge normally repeated answers as those produced by collusion.

(3) In auction platforms, bidders often collude to obtain high returns at low costs. Such algorithms are mainly based on game theory to detect collusion, which is difficult to apply to general tasks on general platforms.

To sum up, for general tasks on common platforms, existing algorithms cannot effectively detect and eliminate the harm of collusion to generate duplicate answers to the quality of results. In view of the existing problems, the technical solution of the embodiment of the present invention proposes a crowdsourcing quality control method based on collusion detection.

Fig. 1 is a schematic diagram of a crowdsourcing framework based on collusion detection according to an embodiment of the present invention. As shown in Fig. 1, the framework includes the following steps:

(1) Requesters post tasks to crowdsourcing platforms, such as MechanicalTurk, where requesters reward workers according to the quality of their answers.

(2) Tasks are assigned to workers according to the scheduling policy and user-specified platform constraints.

(3) In fact, some workers are not independent and can even collaborate on some crowdsourcing tasks outside the platform. Workers may collude with each other behind the scenes. For example, workers plagiarize others who are crowdsourcing the same work through online forums. After the task processing, answers are collected and some noisy answers are eliminated, e.g. some answers are obviously irrelevant to the pictures in the image labeling task.

(4) This step involves collusion detection and aggregation of results. After completing the collection of answers from all the workers, the embodiment of the present invention uses a collusion detection mechanism to detect collusion, and then filters out duplicate answers generated by colluders. After result filtering, embodiments of the present invention use an aggregation method to reason about the final result of each task and submit it to the requester.

The core of the framework of the embodiment of the present invention is step (4), which includes the collusion detection method proposed by the embodiment of the present invention, and then adopts the result reasoning method, which can reason high-quality results even in the case of collusion.

The crowdsourcing framework for collusion detection proposed by the embodiments of the present invention effectively solves the problem that it is difficult for existing result aggregation algorithms to effectively eliminate the harm of collusion to result aggregation. Different from the general crowdsourcing framework, the workers in the crowdsourcing framework proposed by the embodiment of the present invention are no longer independent, but may communicate or even collude with each other. In addition, the result reasoning part of the framework includes the process of serial port detection.

The technical solutions of the embodiments of the present invention generally include three steps: collusion detection, result filtering, and result aggregation, which are described below.

Step 1: Collusion Detection

(1) Calculate the consistency between the aggregation results and the worker's answer: when the worker completes the task processing, first collect the answer returned by the worker, assuming that for the task set The set of answers returned by workers is Assume A repeated answer set for one of the answer sets. The purpose of the embodiment of the present invention is to judge whether the repeated set is produced by collusion, and on this basis answer set Pooling is performed to obtain high-quality results.

Aggregate the answer set using the majority voting method to obtain the aggregated result The embodiment of the present invention provides a calculation formula for the consistency between the aggregation result and worker i's answer:

Among them, L _i is worker i corresponding to task set The collection of answers returned.

(2) Calculate the change rate of worker ability corresponding to each repeated answer set: For a repeated answer set, the change rate of worker ability mainly measures the overall performance of the repeated answer set on the consistency of worker answers and aggregation results. In the embodiment of the present invention, the variance change of the overall consistency before and after deleting duplicate answer sets is used to formalize the change rate of the worker's ability. First, calculate the set of retained duplicate answers When , the variance of worker answer consistency:

Remove Duplicate Answer Sets available Similarly, the calculation removes duplicate answer sets The variance of worker answer agreement when :

Then the above two formulas can be used to obtain the rate of change in worker capacity:

(3) Judging whether repeated answers are produced by collusion: when When it is less than or equal to the threshold Threshold, it is considered a duplicate set Repeat the answer as normal. when When it is greater than Threshold, it is considered to be a duplicate set Repeat answer for collusion.

In the above scheme, when calculating the repeated answer set When the variance of the worker's answer consistency, it can also be obtained by using the results of other pooling algorithms For example, pooling methods for probabilities.

Step 2: Result filtering

Repeat the above steps for a pair of answer sets All sets of duplicates in . Duplicate answers judged to be collusion will be deleted, and answers judged to be normal duplicates will be kept.

Step 3: Aggregation of Results

Use the existing result aggregation algorithm to aggregate the answer set to get the final result.

Fig. 2 is a schematic flowchart of a method for converging crowdsourcing results based on collusion detection according to an embodiment of the present invention. As shown in Fig. 2 , the method for converging crowdsourcing results based on collusion detection includes the following steps:

Step 201: Collect the answer sets of each worker for the set of tasks from the crowdsourcing platform.

Step 202: Calculate the aggregation result of the answer set, and calculate the consistency parameter between the aggregation result and the answers of each worker.

In the embodiment of the present invention, the consistency parameter between the aggregation result and each worker's answer is calculated based on the following formula:

Among them, P _i is the aggregation result and the consistency parameter of worker i and L _i is the answer returned by worker i corresponding to the task set, Aggregated results for answer sets.

Step 203: Determine repeated answer sets from the answer sets, and calculate the worker capability change rate corresponding to each repeated answer set based on the consistency parameters of the answers of the workers.

In the embodiment of the present invention, the calculation of the worker ability change rate corresponding to each repeated answer set based on the consistency parameters of the answers of each worker includes:

calculating the first variance of the consistency parameter of the answers of the workers while retaining duplicate answer sets in the answer sets;

calculating the second variance of the consistency parameter of the answers of the workers when the repeated answer sets are deleted in the answer sets;

Based on the first variance and the second variance, calculate the worker capability change rate corresponding to the repeated answer set.

Wherein, the duplicate answer set is retained in the answer set by the following formula , the first variance of the consistency parameter of each worker's answer:

Among them, Var(P) is the first variance, E(P) is the average value of the consistency parameters of each worker, P _i is the aggregation result and the consistency parameter of worker i and collection of answers.

Delete duplicate answer sets in the answer set by the following formula , the second variance of the consistency parameter of each worker's answer:

Among them, _Var (P ^k ) is the second variance, Ε (P ^k ) is the average value of the consistency parameters of each worker, P _i ^k is the aggregation result and the consistency parameter of worker i and to delete collection of answers.

Calculate duplicate answer sets based on the following formula Corresponding rate of change in worker capacity:

in, is the rate of change in worker capacity.

Step 204: For the repetitive answer set whose worker capability change rate is less than or equal to the preset threshold, determine that the repeated answer set is normally generated and keep the repeated answer set in the answer set.

Step 205: For the repeated answer sets whose rate of change of worker ability is greater than a preset threshold, determine that the repeated answer sets are generated by collusion and delete the repeated answer sets from the answer sets.

Step 206: After retaining or deleting each repeated answer set, an updated answer set is obtained, and an aggregation result of the updated answer set is calculated.

The collusion detection method proposed by the embodiment of the present invention can detect the collusion group with high precision according to the results given by the workers. The change in the variance of the consistency of workers' answers and results before and after deleting a duplicate answer set is used to formalize the change rate of worker ability, and the scale of the change rate of worker ability is used to detect collusion. The collusion results proposed in the embodiment of the present invention are deleted and then aggregated, which can greatly improve the accuracy of the aggregated results. Different from the existing aggregation algorithm, the result aggregation method proposed by the embodiment of the present invention includes the detection of collusion behavior and can effectively eliminate its negative impact on the result aggregation and improve the quality of the result.

FIG. 3 is a schematic diagram of the structure and composition of a crowdsourcing result aggregation device based on collusion detection according to an embodiment of the present invention. As shown in FIG. 3 , the device includes:

The collection module 301 is used to collect the answer sets of each worker for the set of tasks from the crowdsourcing platform;

Consistency calculation module 302, configured to calculate the aggregation result of the answer set, and calculate the consistency parameter between the aggregation result and the answers of each worker;

The worker ability change rate module 303 is used to determine the repeated answer set from the answer set, and calculate the worker ability change rate corresponding to each repeated answer set based on the consistency parameters of the answers of the workers;

The collusion detection module 304 is used to determine that the repeated answer set is normally generated and retain the repeated answer set in the answer set for the repeated answer set whose worker ability change rate is less than or equal to the preset threshold; rate is greater than a preset threshold, determine that the repeated answer set is collusively generated and delete the repeated answer set from the answer set;

The aggregation module 305 is configured to obtain an updated answer set after retaining or deleting each repeated answer set, and calculate an aggregation result of the updated answer set.

In one embodiment of the present invention, the consistency calculation module 302 is specifically used to calculate the consistency parameters of the aggregation result and the answers of each worker based on the following formula:

Among them, P _i is the aggregation result and the consistency parameter of worker i and L _i is the answer returned by worker i corresponding to the task set, Aggregated results for answer sets.

In one embodiment of the present invention, as shown in Figure 4, the worker ability change rate module 303 includes:

The first variance calculation unit 3031 is used to calculate the first variance of the consistency parameter of the answers of the workers when the repeated answer sets are reserved in the answer sets;

The second variance calculation unit 3032 is used to calculate the second variance of the consistency parameter of the answers of the workers when duplicate answer sets are deleted from the answer set;

The worker ability change rate calculation unit 3033 is configured to calculate the worker ability change rate corresponding to the repeated answer set based on the first variance and the second variance.

In one embodiment of the present invention, the first variance calculation unit 3031 is specifically used to calculate the repeated answer set in the answer set by the following formula: , the first variance of the consistency parameter of each worker's answer:

Among them, Var(P) is the first variance, E(P) is the average value of the consistency parameters of each worker, P _i is the aggregation result and the consistency parameter of worker i and collection of answers.

In one embodiment of the present invention, the second variance calculation unit 3032 is specifically used to calculate and delete duplicate answer sets in the answer set by the following formula: , the second variance of the consistency parameter of each worker's answer:

Among them, Var(P ^k ) is the second variance, Ε(P ^k ) is the average value of the consistency parameters of each worker, P _i ^k is the aggregation result and the consistency parameter of worker i and to delete collection of answers.

In one embodiment of the present invention, the worker ability change rate calculation unit 3033 is specifically used to calculate repeated answer sets based on the following formula Corresponding rate of change in worker capacity:

in, is the rate of change in worker capacity.

Those skilled in the art should understand that the implementation functions of each module in the crowdsourcing result aggregation device based on collusion detection shown in FIG. The implementation functions of each module in the collusion detection-based crowdsourcing result aggregation device shown can be realized by a program running on a processor, or by a specific logic circuit.

The technical solutions described in the embodiments of the present invention may be combined arbitrarily if there is no conflict.

In the several embodiments provided by the present invention, it should be understood that the disclosed methods and smart devices can be implemented in other ways. The device embodiments described above are only illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods, such as: multiple units or components can be combined, or May be integrated into another system, or some features may be ignored, or not implemented. In addition, the mutual coupling, or direct coupling, or communication connection between the various components shown or discussed may be through some interfaces, and the indirect coupling or communication connection of devices or units may be in electrical, mechanical or other forms. of.

The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units; Part or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional unit in each embodiment of the present invention can be integrated into one processing unit, or each unit can be used as a single unit, or two or more units can be integrated into one unit; the above-mentioned integration The unit can be realized in the form of hardware or in the form of hardware plus software functional unit.

Those of ordinary skill in the art can understand that all or part of the steps for realizing the above-mentioned method embodiments can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the Including the steps of the foregoing method embodiments; and the foregoing storage medium includes: a removable storage device, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, etc. A medium on which program code can be stored.

Alternatively, if the above-mentioned apparatus in the embodiment of the present invention is realized in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for Make a computer device (which may be a personal computer, a server, or a network device, etc.) execute all or part of the methods described in various embodiments of the present invention. The aforementioned storage media include: various media capable of storing program codes such as removable storage devices, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), magnetic disks or optical disks.

The above is only a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Anyone skilled in the art can easily think of changes or substitutions within the technical scope disclosed in the present invention. Should be covered within the protection scope of the present invention.

Claims (12)

1. A crowdsourcing result aggregation method based on collusion detection, characterized in that the method comprises: Collect the answer sets of each worker for the set of tasks from the crowdsourcing platform; calculating an aggregated result of the set of answers, and calculating a consistency parameter between the aggregated result and each worker's answer; Determining a repeated answer set from the answer set, and calculating a worker capability change rate corresponding to each repeated answer set based on the consistency parameters of the answers of the workers; For a repeated answer set whose worker ability change rate is less than or equal to a preset threshold, determine that the repeated answer set is normally generated and retain the repeated answer set in the answer set; For repeated answer sets whose rate of change of worker ability is greater than a preset threshold, determine that the repeated answer sets are produced by collusion and delete the repeated answer sets from the answer sets; After retaining or deleting each repeated answer set, an updated answer set is obtained, and an aggregation result of the updated answer set is calculated. 2. The method for converging crowdsourcing results based on collusion detection according to claim 1, wherein the calculation of the consistency parameters of the converging results and the answers of each worker comprises: The consensus parameter for the aggregated results and the answers of each worker is calculated based on the following formula: Among them, P _i is the aggregation result and the consistency parameter of worker i and L _i is the answer returned by worker i corresponding to the task set, Aggregated results for answer sets. 3. The crowdsourcing result aggregation method based on collusion detection according to claim 2, characterized in that, based on the consistency parameters of the answers of the workers, the rate of change of worker ability corresponding to each repeated answer set is calculated ,include: calculating the first variance of the consistency parameter of the answers of the workers while retaining duplicate answer sets in the answer sets; calculating the second variance of the consistency parameter of the answers of the workers when the repeated answer sets are deleted in the answer sets; Based on the first variance and the second variance, calculate the worker capability change rate corresponding to the repeated answer set. 4. The crowdsourcing result aggregation method based on collusion detection according to claim 3, characterized in that, when the calculation retains a repeated answer set in the answer set, the consistency parameter of the answers of each worker is First variance, including: Keep duplicate answer sets in the answer set by the following formula , the first variance of the consistency parameter of each worker's answer: Among them, Var(P) is the first variance, E(P) is the average value of the consistency parameters of each worker, P _i is the aggregation result and the consistency parameter of worker i and collection of answers. 5. The crowdsourcing result aggregation method based on collusion detection according to claim 3 or 4, characterized in that, when the calculation deletes duplicate answer sets in the answer set, the consistency of the answers of each worker The second variance of the parameters, including: Delete duplicate answer sets in the answer set by the following formula , the second variance of the consistency parameter of each worker's answer: Among them, Var(P ^k ) is the second variance, Ε(P ^k ) is the average value of the consistency parameters of each worker, P _i ^k is the aggregation result and the consistency parameter of worker i and to delete collection of answers. 6. The crowdsourcing result aggregation method based on collusion detection according to claim 5, characterized in that, based on the first variance and the second variance, the worker capability corresponding to the repeated answer set is calculated rate of change, including: Calculate duplicate answer sets based on the following formula Corresponding rate of change in worker capacity: in, is the rate of change in worker capacity. 7. A crowdsourcing result aggregation device based on collusion detection, characterized in that the device comprises: The collection module is used to collect the answer sets of each worker for the set of tasks from the crowdsourcing platform; a consistency calculation module, configured to calculate the aggregation result of the answer set, and calculate the consistency parameter between the aggregation result and the answers of each worker; A worker ability change rate module, configured to determine a repeated answer set from the answer set, and calculate the worker ability change rate corresponding to each repeated answer set based on the consistency parameters of the answers of each worker; The collusion detection module is used to determine that the repeated answer set is normally generated and retain the repeated answer set in the answer set for the repeated answer set whose worker ability change rate is less than or equal to the preset threshold; for the worker ability change rate A duplicate answer set greater than a preset threshold, determining that the duplicate answer set is generated by collusion and deleting the duplicate answer set from the answer set; The aggregation module is configured to obtain an updated answer set after retaining or deleting each repeated answer set, and calculate an aggregation result of the updated answer set. 8. The crowdsourcing result aggregation device based on collusion detection according to claim 7, wherein the consistency calculation module is specifically used to calculate the consistency of the aggregation result and each worker's answer based on the following formula parameter: Among them, P _i is the aggregation result and the consistency parameter of worker i and L _i is the answer returned by worker i corresponding to the task set, Aggregated results for answer sets. 9. The device for converging crowdsourcing results based on collusion detection according to claim 7, wherein said worker capability change rate module comprises: a first variance calculation unit, configured to calculate the first variance of the consistency parameter of the answers of the workers when the repeated answer set is reserved in the answer set; The second variance calculation unit is used to calculate the second variance of the consistency parameter of the answers of the workers when the duplicate answer set is deleted from the answer set; A worker capability change rate calculation unit, configured to calculate the worker capability change rate corresponding to the repeated answer set based on the first variance and the second variance. 10. The device for converging crowdsourcing results based on collusion detection according to claim 9, wherein the first variance calculation unit is specifically used to calculate and retain duplicate answer sets in the answer set by the following formula , the first variance of the consistency parameter of each worker's answer: Among them, Var(P) is the first variance, E(P) is the average value of the consistency parameters of each worker, P _i is the aggregation result and the consistency parameter of worker i and collection of answers. 11. The crowdsourcing result aggregation device based on collusion detection according to claim 9 or 10, wherein the second variance calculation unit is specifically used to calculate and delete repeated answers in the answer set by the following formula gather , the second variance of the consistency parameter of each worker's answer: Among them, Var(P ^k ) is the second variance, Ε(P ^k ) is the average value of the consistency parameters of each worker, P _i ^k is the aggregation result and the consistency parameter of worker i and to delete collection of answers. 12. The device for converging crowdsourcing results based on collusion detection according to claim 11, characterized in that the calculation unit for the rate of change of worker ability is specifically used to calculate the repeated answer set based on the following formula Corresponding rate of change in worker capacity: in, is the rate of change in worker capacity.