KR102155789B1 - Method and apparatus for acquiring work result of high reliability through crosscheck - Google Patents

Abstract

The present invention relates to a method and a device for acquiring a work result of high reliability through cross-check. The method includes: a step of assigning the same work to n workers (n being a natural number equal to or greater than 3); a step of receiving the input of n work results from the n workers; a step of checking whether the n work results match without exception; a step of re-assigning the same work to m new additional workers in the case of non-matching of the n work results (m being the number of work results corresponding to a minority opinion among the n work results); a step of receiving the input of m re-work results from the m new additional workers; a step of checking whether the work result corresponding to a majority opinion among the m re-work results and the n work results matches; and a step of using only the work result corresponding to the majority opinion and the re-work result as effective data in a case where the work result corresponding to the majority opinion and the re-work result match. The work is an alternative work in which the worker selects one out of two options.

Description

Method and device for obtaining reliable work results through cross check {METHOD AND APPARATUS FOR ACQUIRING WORK RESULT OF HIGH RELIABILITY THROUGH CROSSCHECK}

The present invention relates to a method and apparatus for obtaining reliable work results through cross check.

Recently, more and more companies collect and process large amounts of data on a crowdsourcing basis that engages the general public in some process of corporate activities. In other words, by opening a project and allowing the general public, that is, workers to participate in the project, necessary information is collected through the work results completed by the workers.

At this time, in order to collect more reliable information, the company assigns the work result completed by the worker to the inspector to perform the inspection work.

Specifically, when one project is opened, a plurality of tasks are assigned to each of the plurality of workers. Each worker performs a plurality of tasks assigned to it, and provides the task results. Thereafter, a plurality of inspection tasks are assigned to each of the plurality of inspectors, and each inspector performs a plurality of inspection tasks assigned to them.

Currently, in the inspection method of the inspector, it is a principle to perform a total inspection on the results of a plurality of work for one project.

However, since the number of inspectors participating in one project is considerably smaller than the number of workers, there is a problem that the inspector's work is heavy because numerous inspection tasks are assigned to one inspector.

In addition, since the entire work result is based on full inspection, the inspection is performed by the inspector even though the inspection is not necessary, and there is a problem that it is difficult for the inspector to efficiently perform the inspection.

Unexamined Patent Publication No. 10-2014-0095956, 2014.08.04

The problem to be solved by the present invention is to provide a method and apparatus for obtaining reliable work results through cross-check that determines whether to use a corresponding work result as valid data based on whether a plurality of work results are matched.

The problems to be solved by the present invention are not limited to the problems mentioned above, and other problems that are not mentioned will be clearly understood by those skilled in the art from the following description.

The method of obtaining reliable work results through a cross check according to an aspect of the present invention for solving the above-described problems is the step of allocating the same task to n workers (n is a natural number of 3 or more), n tasks from n workers Receiving a result, checking whether all the n work results match, and if the n work results are inconsistent, reassigning the same work to m new additional workers (m is among the n work results) The number of work results corresponding to a minority opinion), receiving m rework results from the m new additional workers, the m rework results and the work results corresponding to the majority opinion among the n work results coincide Checking whether or not, and when the rework result and the work result corresponding to the majority opinion match, using only the rework result and the work result corresponding to the majority opinion as valid data, the work is It is characterized in that it is an alternative type of work that allows the operator to select one of two options.

In some embodiments of the present invention, when the n number of job results match, the step of using the n job results as valid data without inspecting the n number of job results may be further included.

In some embodiments of the present invention, when the rework result and the work result corresponding to the multiple opinions are inconsistent, allocating the same work to an inspector, receiving an inspection result from the inspector, and the inspection result Accordingly, the step of determining which of the n work results and the m rework results is to be used as valid data may be further included.

In some embodiments of the present invention, when the rework results and work results corresponding to the majority opinion are inconsistent, the step of invalidating the n work results and the m rework results may be further included.

In some embodiments of the present invention, n may be an odd number.

In some embodiments of the present invention, n may be determined according to reliability.

In some embodiments of the present invention, the task may be an answer to a question that requires subjective evaluation of an operator.

A computer program according to another aspect of the present invention for solving the above-described problems is combined with a computer as hardware to execute the method of obtaining a reliable work result through the cross check, and is stored in a medium.

The device for obtaining reliable work results through a cross check according to another aspect of the present invention for solving the above-described problems transmits the same work to n workers and m additional workers, and n jobs from the n workers A communication unit that receives results and receives m rework results from the m new additional workers (n is a natural number of 3 or more, m is the number of work results corresponding to a minority opinion among n work results), and the n A storage unit that stores the n jobs input from the workers and the m rework results input from the m new additional workers, determines whether all the n jobs input from the n workers match, and the n A determination unit that determines whether the work results corresponding to the majority opinion among the number of tasks coincide with the m rework results, and when all the n tasks match, selects the n work results as valid data, and the n If the work results corresponding to multiple opinions among the m work results match the m rework results, the rework results and the work results corresponding to the majority opinion are selected as valid data, and the n workers determined according to reliability And a selection unit for selecting the m new additional workers according to whether or not all of the n jobs match.

Other specific details of the present invention are included in the detailed description and drawings.

According to the method and apparatus for obtaining reliable work results through cross-check of the present invention as described above, the work results are received from a plurality of workers for the same work, and the work results are valid data based on whether the plurality of work results match. As it is used as a product, it can provide the project result with high reliability to the client.

In addition, if multiple work results match, the work result is immediately used as valid data without going through the inspection work, so the number of inspection work assigned to the inspector is generally reduced, reducing the work and related costs of the inspector. Only when the reliability of the result is low, the inspector can efficiently perform the inspection work.

In addition, if multiple work results match, the work result is used as valid data without going through the inspection work, so the number of inspection work assigned to the inspector is generally reduced, and the entire project due to excessive and redundant inspection work by the inspector Can prevent the bottleneck of

The effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the following description.

1 is a conceptual diagram of a crowd sourcing service according to an embodiment of the present invention.
2 is a flowchart illustrating a crowdsourcing-based project progress process according to an embodiment of the present invention.
3 is a flowchart of a method of obtaining a reliable work result through cross check according to an embodiment of the present invention.
4 is an exemplary view for explaining a task requiring objective evaluation according to an embodiment of the present invention.
5 and 6 are exemplary diagrams for explaining a task requiring subjective evaluation according to an embodiment of the present invention.
7 is a block diagram of an apparatus for obtaining reliable work results through cross check according to an embodiment of the present invention.

Advantages and features of the present invention, and a method of achieving them will become apparent with reference to the embodiments described below in detail together with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but may be implemented in a variety of different forms, only the present embodiments are intended to complete the disclosure of the present invention, It is provided to fully inform the technician of the scope of the present invention, and the present invention is only defined by the scope of the claims.

The terms used in the present specification are for describing exemplary embodiments and are not intended to limit the present invention. In this specification, the singular form also includes the plural form unless specifically stated in the phrase. As used in the specification, “comprises” and/or “comprising” do not exclude the presence or addition of one or more other elements other than the mentioned elements. Throughout the specification, the same reference numerals refer to the same elements, and “and/or” includes each and all combinations of one or more of the mentioned elements. Although "first", "second", and the like are used to describe various elements, it goes without saying that these elements are not limited by these terms. These terms are only used to distinguish one component from another component. Therefore, it goes without saying that the first component mentioned below may be the second component within the technical idea of the present invention.

Unless otherwise defined, all terms (including technical and scientific terms) used in the present specification may be used as meanings that can be commonly understood by those of ordinary skill in the art to which the present invention belongs. In addition, terms defined in a commonly used dictionary are not interpreted ideally or excessively unless explicitly defined specifically.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a conceptual diagram of a crowd sourcing service according to an embodiment of the present invention.

Referring to FIG. 1, a crowdsourcing service is performed by being composed of a client 10, a service provider 20, and a public 30.

The sponsor 10 refers to a company or individual who requests a crowdsourcing-based project (hereinafter, referred to as a project).

The requester 10 requests a project for the purpose of collecting source data or annotating data. Collection of source data refers to collecting raw data such as recorded voice collection and photo collection. Data annotation refers to inputting relevant annotation data into source data such as text, photos, and videos. For example, the data annotation may include finding an entity in a given fingerprint or finding a similar sentence, but is not limited thereto.

The service provider 20 refers to a company that provides crowdsourcing services.

When the service provider 20 receives a request for a project for a product or service from the client 10, the service provider 20 allocates a task for the project to the general public 30 and receives the work result from the public 30. Thereafter, the final product extracted based on the work result is provided to the client 10.

At this time, the service provider 20 provides a crowd sourcing service to the client 10 and the public 30 through a crowd sourcing platform (hereinafter, the platform). That is, when the service provider 20 receives a request for a project from the client 10, the service provider 20 opens the project on the platform. Thereafter, when the work result for the open project is received from the public 30, the project is terminated on the platform, and the final product may be extracted and provided to the client 10.

The public 30 refers to the general public participating in the project open on the platform. Here, the public 30 may participate in a project opened on the platform through an application or website provided by the service provider 20.

The public 30 is composed of a worker 32 and an inspector 34.

The worker 32 decides to participate in a specific project among a plurality of projects open on the platform. Thereafter, the operator 32 performs an operation such as collection of source data or data annotation, and transmits it to the platform.

The inspector 34 decides to participate in a specific project among a plurality of projects open on the platform. Thereafter, the inspector 34 inspects the result of the work performed by the operator 32. As a result of performing the inspection, the inspector 34 may pass or reject the job, and may input a reason for rejection when returning the job.

2 is a flowchart illustrating a crowdsourcing-based project progress process according to an embodiment of the present invention.

First, the client 10 requests a project to the service provider 20 (S41).

Thereafter, the service provider 20 opens the requested project on the platform (S42). At this time, the service provider 20 determines the grade in consideration of the difficulty level of the project before opening the project. In other words, depending on the level of difficulty, it is decided to expose the project to the public 30 or higher. Accordingly, it is possible to increase the reliability of the work result of the project.

Thereafter, the service provider 20 allocates a task to the worker 32 of the corresponding grade or higher according to the grade of the project (S43).

Thereafter, the worker 32 performs the assigned task (S44). In this case, the worker 32 may input the reason for the inability to work without performing the work for a work in which the work itself is impossible for some reason.

Thereafter, the service provider 20 receives the work result from the worker 32 (S45), and assigns the inspection work for the work result to the inspector 34 (S46).

Thereafter, the inspector 34 performs the assigned inspection (S47). At this time, the inspector 34 decides to pass the inspection when it is determined that the work has been properly performed, and determines rejection when it is determined that the inspection work is wrong. When deciding to reject, the inspector 34 inputs the rejection reason for what reason the task was judged to be wrong.

Thereafter, the service provider 20 receives the inspection result from the inspector 34 (S48).

If the inspection result is Pass, the service provider 20 uses the work result as valid data, and based on this, extracts the final product at the end of the project.

If the inspection result is rejected, the service provider 20 may internally perform the inspection again or assign the job again to the worker 32 to perform the job.

Thereafter, when the project period ends or sufficient valid data is secured, the service provider 20 terminates the project (S49), calculates a final result based on the secured valid data, and provides it to the client 10 ( S50).

At this time, before the end of the project, the service provider 20 evaluates the performance results of the worker 32 and the inspector 34, calculates the work cost and the inspection cost according to the evaluation, to the worker 32 and the inspector 34. give.

In Figs. 1 and 2, it is simply expressed as a client, service provider, worker, and inspector, but these refer to computer devices or telecommunication devices such as smartphones, tablets, PDAs, laptops, desktops, servers, etc. operated by each participant. .

3 is a flowchart of a method for obtaining a reliable work result through cross check according to an embodiment of the present invention, and FIG. 4 is an exemplary diagram for explaining a task requiring objective evaluation according to an embodiment of the present invention, 5 and 6 are exemplary diagrams for explaining a task requiring subjective evaluation according to an embodiment of the present invention.

Steps S110 to S180 shown in FIG. 3 may be understood as being performed by a platform server (hereinafter, referred to as a server) operated by the service provider 20, but are not limited thereto.

3, in step S110, the server allocates the same task to a plurality of workers 32. Here, the work means an alternative type of work that selects one of two options.

That is, the plurality of workers 32 assigned the same task will perform the task by selecting one of the two options according to their own judgment.

The server selects n workers 32 who will perform the same task and assigns the same task. Here, n means a natural number of 3 or more.

In one embodiment, n may be an odd number of 3 or more. The server may allocate the same task to three or more odd numbers (ie, 3, 5, 7, etc.) of the workers 32. As will be described later, this is to distinguish the majority opinion.

In this case, the server may determine n according to the reliability required by the client 10. The larger n, that is, the more workers 32 assigned to the same task, the more task results can be obtained for the same task, thereby increasing the reliability of the task result.

The alternative task assigned to the worker 32 may be a task requiring objective evaluation or a task requiring subjective evaluation. In particular, in the case of a task that requires subjective evaluation, a large number of workers participate in the same task, so that highly reliable data formed by multiple opinions can be secured.

4, a fingerprint is given to an operator 32, and a question based on the fingerprint is given. The worker 32 objectively judges whether the given question is a correct question or an incorrect question with reference to the fingerprint, and then answers.

For example, the worker 32 may select "O" by referring to the contents of the fingerprint for the question "Is Eric Thames nationality of the United States?"

Referring to FIG. 5, when two sentences with similar meanings are given to the worker 32, the worker 32 subjectively determines whether the two sentences have the same meaning or different meanings, and then answers.

For example, if it is determined that the meanings of "looks gentle and kind" and "looks good and looks good" are determined to be the same, the worker 32 may select "same meaning (O)", or "looks gentle and kind" and "looks good and good" If it is determined that the meaning of "looks pretty" is different, the worker 32 may select "different meaning (X)".

Referring to FIG. 6, when a portrait photo (specifically, a face photo) is given to the worker 32, the worker 32 subjectively determines the gender of the person by viewing the given photo and then responds.

For example, you are given a picture of which gender is ambiguous because you are wearing a hat, and you are asked "Please choose your gender". In contrast, the worker 32 may select "male" or "female" according to the judgment.

The data secured through the exemplary tasks of FIGS. 4 to 6 may be used as learning data for artificial intelligence or machine learning.

Subsequently, referring again to FIG. 3, in step S120, the server receives n job results from n workers 32.

When n is 3, the server receives three job results from three workers 32.

For example, if the task is an alternative task of selecting O or X, the server will receive a result in one of "OOO", "XXX", "OOX", and "OXX".

Subsequently, in step S130, the server checks whether all n work results match. In other words, it is to cross-check n work results to check whether they all match.

For example, if the job is an alternative job of selecting O or X, and n is 3, the server will respond to the job results of “OOO” and “XXX”, where all three workers (32) agree. It is determined that the opinions of the three workers 32 are inconsistent with respect to the work results of "OOX" and "OXX".

When all n job results match, the server uses n job results as valid data. Details about this will be described later in step S180.

Subsequently, in step S140, when the n number of job results are inconsistent, the server reassigns the same job to m new additional workers 32. Here, m denotes the number of work results corresponding to a minority opinion among n work results.

That is, when the results of n tasks are "OOX", the server judges O as a majority opinion and X as a minority opinion. Accordingly, the server selects the number of work results corresponding to X, that is, one additional worker 32 and reassigns the same work.

In addition, when n is 5, if the server receives the work results of "OOOOX" or "OOOXX" from 5 workers, the server judges X as a minority opinion, and the number of work results corresponding to X, that is, one One or two new additional workers 32 are selected and the same task is reassigned.

For example, if the task of determining whether two similar sentences are identical as shown in FIG. 5 is assigned to n workers 32, whether two similar sentences are identical to m new additional workers 32 The task of judging will be assigned.

Subsequently, in step S150, the server receives m rework results from m new additional workers 32.

For example, when m is 1 (that is, when the number of work results corresponding to a minority opinion among n work results is one), the server receives one rework result from one additional worker 32 .

At this time, if the job is an alternative job of selecting O or X, the server receives the rework result of O or X from one additional worker 32.

Subsequently, in step S160, the server checks whether the work results corresponding to the majority opinion among the m rework results and n work results match. In other words, among the m rework results and n work results, the work results corresponding to the majority opinion are cross-checked to see if they match.

That is, when n is 3 and m is 1, the server checks whether two work results corresponding to the majority opinion and one rework result match each other.

For example, if the task is an alternative task of selecting O or X, the two task results corresponding to the majority opinion may be "OO" or "XX". At this time, it is determined whether the result of one rework is O or X.

Specifically, if two work results are "OO" and one rework result is O, the server determines that the rework result and the work result corresponding to the majority opinion match.

When the rework result and the work result corresponding to the majority opinion match, the server uses the rework result and the work result corresponding to the majority opinion as valid data. Details about this will be described later in step S180.

On the other hand, if two work results are "OO" and one rework result is X, the server determines that the rework result and the work result corresponding to the majority opinion are inconsistent.

Subsequently, in step S170, when the rework result and the work result corresponding to the majority opinion are inconsistent, the server allocates the same work to the inspector 34.

For example, the server allocates to the inspector 34 an alternative task of selecting "O" or "X". That is, the inspector 34 is assigned the same task assigned to a plurality of workers and determines whether "O" is correct or "X" is correct.

Thereafter, the server secures the final data based on the inspection result input from the inspector 34. Details about this will be described later in step S180.

In addition, as an embodiment, in step S170, when the rework result and the work result corresponding to the majority opinion are inconsistent, the server may invalidate the work result and the rework result.

That is, if n is 3 and m is 1, if one rework result and two work results corresponding to the majority opinion are inconsistent, the server can invalidate one rework result and all three work results. . As a result, all work results are invalidated.

After invalidating all of the work results, the server may perform the same process from the beginning by assigning the same work to another n number of workers 32. Alternatively, the server may assign the task to the inspector 34 to perform the task directly. In general, since the inspector 34 is selected as a person with long experience and high reliability compared to the operator 32, the work result of the inspector 34 can be used as valid data as it is.

Subsequently, in step S180, the server uses the work result that passed the cross check or the work result based on the inspection result as valid data.

In summary, there are three cases in which the server uses the job result as valid data.

First, in step S130, if all of the n job results match, the server uses the n job results as valid data.

For example, if the job is an alternative job of selecting O or X, and n is 3, the results of the three jobs received from the three workers 32 are all matched as “OOO” or “XXX”. Once confirmed, the server does not inspect the results of the three jobs and immediately uses them as valid data. Here, the valid data means data used to generate the final product at the end of the project. That is, if the results of the three jobs are "OOO", the job results for that job are used as "O".

Second, in step S160, when the rework result and the work result corresponding to the majority opinion match, the server uses the rework result and the work result corresponding to the majority opinion as valid data.

For example, if the task is an alternative task of choosing O or X, and n is 3 and m is 1, then the result of one rework is "O", and the result of two tasks corresponding to the majority opinion is If a match is found to be "OO", the server uses only one rework result and two work results corresponding to majority opinion as valid data.

In other words, the work result for the work is used as "O", and one work result (X) corresponding to a minority opinion among the three work results is discarded.

At this time, the server inspects the rework result and the work result corresponding to the majority opinion, that is, (rather, because the majority opinion may be wrong), checks whether the work result for the work is "O" by the inspector. Instead, use it as valid data immediately.

Third, in step S170, according to the inspection result for the same job, the server determines which job result is to be used as valid data among all the job results (n job results and m rework results).

In one embodiment, when the inspection result is input as "O" from the inspector 34 for the alternative type of work, the server only results in the work corresponding to "O" among m rework results and n work results. Use as valid data.

For example, if the majority opinion is X", the minority opinion is "O", and one rework result is entered as "O", the server will display the decimal number according to the inspection result ("O"). Only one work result ("O") and one rework result ("O") corresponding to the opinion is used as valid data.

7 is a block diagram of an apparatus for obtaining work results using a cross check according to an embodiment of the present invention.

Referring to FIG. 7, an apparatus 200 for obtaining a reliable work result through a cross check (hereinafter, a device for obtaining a work result) includes a communication unit 210, a storage unit 220, a determination unit 230, and a selection unit ( 240).

The communication unit 210 transmits the same job to n workers and receives n job results from n workers. In addition, when rework is required, the communication unit 210 transmits the same task to m additional workers and receives m rework results from m new additional workers.

Here, n is a natural number that is an odd number of 3 or more, and m is the number of work results corresponding to a minority opinion among n work results.

The storage unit 220 serves to store n jobs input from n workers and m rework results input from m new additional workers.

The determination unit 230 determines whether all n jobs input from n workers match, and determines whether a job result corresponding to a majority opinion among the n jobs matches m rework results.

When all of the n jobs match, the selection unit 240 serves to select n job results as valid data.

In addition, the selection unit 240 plays a role of extracting the rework result and the work result corresponding to the majority opinion as valid data when the work results corresponding to the majority opinion among the n tasks match m rework results. do.

In addition, the selection unit 240 selects n workers determined according to the reliability, and selects m new additional workers according to whether all n jobs match.

The operation result obtaining apparatus 200 described with reference to FIG. 7 may be provided as a component of the above-described server.

The method of obtaining a reliable work result through a cross check according to an embodiment of the present invention described above may be implemented as a program (or application) to be executed in combination with a computer, which is hardware, and stored in a medium.

The above-described program includes C, C++, JAVA, Ruby, which can be read by a processor (CPU) of the computer through the device interface of the computer, in order for the computer to read the program and execute the methods implemented as a program. It may include a code (Code) coded in a computer language such as machine language. Such code may include a functional code related to a function that defines necessary functions for executing the methods, and includes a control code related to an execution procedure necessary for the processor of the computer to execute the functions according to a predetermined procedure. can do. In addition, these codes may further include additional information required for the processor of the computer to execute the functions, or code related to a memory reference to which location (address address) of the internal or external memory of the computer should be referenced. have. In addition, when the processor of the computer needs to communicate with any other computer or server in the remote in order to execute the functions, the code uses the communication module of the computer to determine how It may further include a communication-related code for whether to communicate, what kind of information or media should be transmitted and received during communication.

The stored medium is not a medium that stores data for a short moment, such as a register, cache, memory, etc., but a medium that stores data semi-permanently and can be read by a device. Specifically, examples of the storage medium include, but are not limited to, ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, and the like. That is, the program may be stored in various recording media on various servers to which the computer can access, or on various recording media on the user's computer. In addition, the medium may be distributed over a computer system connected through a network, and computer-readable codes may be stored in a distributed manner.

In the above, embodiments of the present invention have been described with reference to the accompanying drawings, but those of ordinary skill in the art to which the present invention pertains can be implemented in other specific forms without changing the technical spirit or essential features. You can understand. Therefore, the embodiments described above are illustrative in all respects, and should be understood as non-limiting.

10: Client
20: service provider
30: public
32: worker
34: inspector
200: inspector monitoring device
210: communication department
220: storage unit
230: judgment unit
240: sorting unit

Claims (9)

As a method performed by a computer,
Assigning the same task to n workers (n is a natural number of 3 or more);
receiving n job results from n workers;
Checking whether all of the n work results match;
Reassigning the same job to m new additional workers when the n job results are inconsistent (m is the number of job results corresponding to a small number of n job results);
Receiving m rework results from the m new additional workers;
Checking whether the m rework results match all of the work results corresponding to multiple opinions among the n work results; And
When the rework result and the work result corresponding to the majority opinion all coincide, using the work result corresponding to the majority opinion as valid data,
The task is to select an answer to a question that requires subjective evaluation, and is an alternative task in which the n workers select one of two options according to their own judgment,
If the rework result and the work result corresponding to the majority opinion are inconsistent,
Allocating the same task to an inspector;
Receiving an inspection result from the inspector; And
According to the inspection result, the operation result corresponding to the majority opinion or the operation result corresponding to the minority opinion, further comprising the step of determining whether to use as valid data, obtaining a highly reliable work result through cross-check Way. The method of claim 1,
If the above n task results match,
A method for obtaining reliable work results through cross check, further comprising the step of using as valid data without performing the inspection of the n work results. delete The method of claim 1,
If the rework result and the work result corresponding to the majority opinion are inconsistent,
And invalidating the n number of work results and the m number of rework results. The method of claim 1,
Wherein n is an odd number, a method of obtaining a reliable work result through a cross check. The method of claim 1,
Wherein n is determined according to the reliability, a method of obtaining a reliable work result through a cross check. delete A computer program that is combined with a computer as hardware and stored in a medium to execute the method of obtaining a reliable work result through the cross check of any one of claims 1, 2, 4, 5 and 6 . As a reliable work result acquisition device through cross check,
Sending the same job to n workers, m new additional workers, and inspectors, receiving n job results from the n workers, receiving m rework results from the m new additional workers, and the inspector A communication unit that receives the inspection result from (n is a natural number of 3 or more, m is the number of work results corresponding to a minority opinion among n work results);
A storage unit for storing the n work results input from the n workers, the m rework results input from the m new additional workers, and the inspection results;
A determination unit determining whether all the n work results inputted from the n workers are identical, and determining whether the m rework results match all work results corresponding to multiple opinions among the n work results; And
The same task is assigned to the n number of workers, and if all the n task results match, the n task results are selected as valid data, and when the n task results are inconsistent, the same task is performed. Reassignment to m new additional workers, and if the m rework results all match the work results corresponding to the majority opinion among the n work results, the work results corresponding to the majority opinion are selected as valid data, If the rework result and the work result corresponding to the majority opinion are inconsistent, the same work is assigned to the inspector, and according to the inspection result, the work result corresponding to the majority opinion or the work corresponding to the minor opinion Includes a selection unit for selecting the result as valid data,
The task is to select an answer to a question that requires subjective evaluation, characterized in that the n workers select one of two options according to their own judgment, characterized in that, through cross-check Device for obtaining reliable work results.

Images