JP2021034053A - Method and device for obtaining highly-reliable work result through cross-checking - Google Patents

Abstract

To provide a method and a device for obtaining a highly-reliable work result through cross-checking.SOLUTION: A method for obtaining a highly-reliable work result through cross-checking comprises the steps of: assigning the same work to n number of workers (n is a natural number of 3 or more); receiving n number of work results from the n number of workers; checking whether the n number of work results all match; if the n number of work results do not match, reassigning the same work to m number of new additional workers (m is the number of work results corresponding to the minority opinion of the n number of work results); receiving m number of reworked results from the m number of new additional workers; checking whether the m number of reworked results match with the work results corresponding to the majority opinion among the n number of work results; and if the reworked results match with the work results corresponding to the majority opinion, using only the reworked results and the work results corresponding to the majority opinion as valid data. The above work is selected by a worker from among two options.SELECTED DRAWING: Figure 3

Description

The present invention relates to a method and an apparatus for obtaining a highly reliable work result through a cross check.

In recent years, an increasing number of companies are collecting and processing large amounts of data based on crowdsourcing that involves the general public in some processes of corporate activities. That is, a company collects necessary information through the work results completed by a worker by opening one project and allowing the general public, that is, a worker, to participate in the project.

At this time, in order to collect more reliable information, the company assigns the work result completed by the worker to the inspector to perform the acceptance inspection work.

Specifically, when one project is opened, a plurality of tasks are assigned to each of the plurality of workers. Each worker performs a plurality of assigned tasks and provides work results. After that, a plurality of acceptance inspection operations for the work results are assigned to the plurality of inspectors, and the inspectors perform the plurality of acceptance inspection operations assigned to each.

Currently, in the acceptance method of inspectors, in principle, 100% inspection is basically performed for a plurality of work results for one project.

However, since the number of inspectors participating in one project is significantly smaller than the number of workers, a large number of inspectors are assigned to one inspector, and there is a problem that the work of the inspectors becomes overloaded. is there.

In addition, since 100% inspection is the basis for all work results, acceptance inspection is performed by the inspector even though the work does not require acceptance inspection, so that the inspector can efficiently perform the acceptance inspection work. There is a problem that there is difficulty.

Korean Publication No. 10-2014-095956 (2014.08.04)

Therefore, the present invention has been made in view of the above circumstances, and an object thereof is through a cross-check in which it is determined whether or not the relevant work result is used as valid data depending on whether or not the plurality of work results match. It is an object of the present invention to provide a method and an apparatus for obtaining a highly reliable work result.

The problem to be solved by the present invention is not limited to the problem mentioned above, and other problems not mentioned can be clearly understood by ordinary engineers from the following description.

The method of obtaining highly reliable work results through cross-check according to one aspect of the present invention for solving the above-mentioned problems is a stage of assigning the same work to n workers (n is 3 or more). (Natural number), the stage of receiving input of n work results from n workers, the stage of confirming whether all the n work results match, and the stage where the n work results do not match. The stage of reassigning the same work to m new additional workers (m is the number of work results corresponding to the minority opinion among n work results) and the new addition of m names. The stage of receiving input of m rework results from the worker and the stage of confirming whether the m rework results and the work results corresponding to the majority opinions of the n work results match. When the rework result and the work result corresponding to the majority opinion match, the work includes a step of using only the rework result and the work result corresponding to the majority opinion as valid data. It is characterized in that it is an alternative type of work that allows a person to select one of two options.

In one embodiment of the present invention, when the n work results match each other, the step of using the n work results as valid data without performing acceptance inspection can be further included.

In one embodiment of the present invention, when the rework result and the work result corresponding to the majority opinion do not match, the step of assigning the same work to the inspector and the input of the acceptance result from the inspector are received. It can further include a step and a step of determining which of the n work results and the m rework results is used as valid data according to the acceptance inspection result.

In one embodiment of the present invention, when the rework result and the work result corresponding to the majority opinion do not match, the step of invalidating the n work results and the m rework results is further added. Can include.

In one embodiment of the invention, the n can be odd.

In one embodiment of the invention, the n can be determined according to reliability.

In one embodiment of the invention, the work may be an answer to a question that requires a subjective evaluation of the operator.

A computer program according to another aspect of the present invention for solving the above-mentioned problems is combined with a computer as hardware to execute a highly reliable work result acquisition method through the cross-check, and is a medium. Stored in.

A highly reliable work result acquisition device through cross-checking according to still another aspect of the present invention for solving the above-mentioned problems allows n workers and m new additional workers to perform the same work. (N is a natural number of 3 or more, n is a natural number of 3 or more, and is transmitted to, receives n work results from the n workers, and receives m rework results from the newly added m workers. m is the number of work results corresponding to a minority opinion out of n work results), the n work input from the n workers, and input from the newly added m workers. It is determined whether the storage unit that stores the m rework results and the n works input from the n workers all match, and the majority opinion of the n works. When the judgment unit for determining whether the work result corresponding to the above corresponds to the m rework results and the n work all match, the n work results are selected as valid data and described as described above. When the work result corresponding to the majority opinion among the n works matches the m rework results, the rework result and the work result corresponding to the majority opinion are selected as valid data and trusted. It includes a sorting unit that selects the n workers determined according to the degree and selects a new additional worker of the m depending on whether or not all the n works match.

Other specific matters of the present invention are included in the detailed description and drawings.

According to the method and apparatus for obtaining highly reliable work results through cross-checks of the present invention as described above, a plurality of work results are received from a plurality of workers for the same work. Since the corresponding work result is used as valid data depending on whether or not they match, the effect is that highly reliable project results can be provided to the client.

In addition, when multiple work results match, the corresponding work result is not performed for acceptance work and is immediately used as valid data. Therefore, the number of acceptance work assigned to the inspector is generally reduced, and the work of the inspector and related matters. The cost is reduced, and the inspector can efficiently perform the acceptance inspection only when the reliability of the work result is low.

Furthermore, when a plurality of work results match, the corresponding work result is not subjected to the acceptance inspection work and is immediately used as valid data. Therefore, the number of acceptance inspection operations assigned to the inspector is generally reduced, and the acceptance inspection work of the inspector is performed. It is possible to prevent the bottleneck phenomenon of the entire project due to excess and accumulation of data.

The effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned above can be clearly understood by ordinary engineers from the following description.

It is a conceptual diagram of the crowdsourcing service which concerns on one Example of this invention. It is a flowchart explaining the progress process of the project of the crowdsourcing platform which concerns on one Example of this invention. It is a flowchart of a highly reliable work result acquisition method through the cross check which concerns on one Example of this invention. It is explanatory drawing explaining the work which requires the objective evaluation which concerns on one Example of this invention. It is an example figure (1) explaining the work which requires the subjective evaluation which concerns on one Example of this invention. It is explanatory drawing (2) explaining the work which requires subjective evaluation which concerns on one Example of this invention. It is a block diagram of the work result acquisition apparatus with high reliability through a cross check which concerns on one Example of this invention.

The advantages and features of the present invention, and the methods for achieving them, will be clarified with reference to the examples described in detail with the accompanying drawings. However, the present invention is not limited to the examples disclosed below, and can be realized in various forms different from each other. However, the present embodiment is provided to complete the disclosure of the present invention and to allow ordinary engineers in the technical field to which the present invention belongs to fully understand the scope of the present invention, and the present invention is claimed. It is only defined by the scope.

The terms used herein are for illustration purposes only and are not intended to limit the invention. In the present specification, the singular type also includes a plural type unless otherwise specified. As used herein, "comprises" and / or "comprising" does not preclude the presence or addition of one or more other components other than those mentioned. The same drawing reference throughout the specification indicates the same component, and "and / or" includes each of the mentioned components and any combination of one or more. Of course, even if "first", "second", etc. are used to describe various components, these components are not limited by these terms. These terms are used solely to distinguish one component from the other. Therefore, it goes without saying that the first component referred to below can also be the second component within the technical idea of the present invention.

Unless otherwise defined, all terms used herein (including technical and scientific terms) are used as meanings commonly understood by ordinary engineers in the technical field to which the present invention belongs. Also, terms defined in commonly used dictionaries are not ideally or over-interpreted unless explicitly specifically defined.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

FIG. 1 is a conceptual diagram of a crowdsourcing service according to an embodiment of the present invention.

With reference to FIG. 1, the crowdsourcing service is composed of 10 clients, 20 service providers, and 30 masses.

The client 10 means a company or an individual who requests a crowdsourcing-based project (hereinafter referred to as a project).

The requester 10 requests a project for the purpose of collecting source data or data annotation. Collecting source data means collecting unprocessed data, such as collecting recorded audio, collecting photographs, and so on. Data annotation means entering annotation data related to source data such as text, photos, and videos. For example, data annotation includes, but is not limited to, searching for an individual or a similar sentence from a given text.

The service provider 20 means a company that provides a crowdsourcing service.

When the service provider 20 receives a project request for a product or service from the client 10, the service provider 20 allocates the work for the project to the general public 30 and receives the work result from the public 30. After that, the output (final product) extracted based on the work result is provided to the client 10.

At this time, the service provider 20 provides the crowdsourcing service to the client 10 and the public 30 via the crowdsourcing platform (hereinafter referred to as the platform). That is, when the service provider 20 receives a project request from the client 10, the service provider 20 opens the project on the platform. After that, when the work result for the opened project is provided by the public 30, the project can be completed on the platform, the output (final product) can be extracted and provided to the client 10.

Mass 30 means the general public participating in a project opened on the platform. Here, the public 30 can participate in a project opened on the platform through an application or website provided by the service provider 20.

The mass 30 is composed of a worker 32 and an inspector 34.

Worker 32 decides to participate in a specific project among a plurality of projects opened on the platform. After that, the worker 32 performs work such as collecting source data or data annotation, and sends this to the platform.

The inspector 34 decides to participate in a specific project among a plurality of projects opened on the platform. After that, the inspector 34 inspects the work result performed by the worker 32. As a result of the acceptance inspection, the inspector 34 can perform pass certification (passage) or remand of the work, and can input the reason for the remand when the work is remanded.

FIG. 2 is a flowchart illustrating a progress process for a crowdsourcing-based project according to an embodiment of the present invention.

First, the client 10 requests the service provider 20 for the project (S41).

After that, the service provider 20 opens the requested project on the platform (S42). At this time, the service provider 20 determines the grade in consideration of the difficulty level of the project before opening the project. That is, depending on the difficulty level, it is determined what grade or higher the public 30 is exposed to the project. This makes it possible to increase the reliability of the work results of the project.

After that, the service provider 20 allocates work to workers 32 of the corresponding grade or higher according to the grade of the project (S43).

After that, the worker 32 will perform the assigned work (S44). At this time, the worker 32 can input the reason why the work is impossible without performing the work for the work whose work itself is impossible for some reason.

After that, the service provider 20 receives the work result provided by the worker 32 (S45), and assigns the acceptance inspection work for the work result to the inspector 34 (S46).

After that, the inspector 34 performs the assigned acceptance inspection (S47). At this time, the inspector 34 decides to pass the acceptance if it is determined that the work has been performed normally, and decides to remand if it is determined by the acceptance inspection that the work is incorrect. When deciding to remand, the inspector 34 inputs the reason for the remand as to why the work was determined to be incorrect.

After that, the service provider 20 receives the acceptance result from the inspector 34 (S48).

If the acceptance result is pass certification (pass), the service provider 20 uses the corresponding work result as valid data, and extracts the output (final product) at the end of the project based on this.

If the acceptance result is a remand, the service provider 20 can internally perform the acceptance again or assign the work to the worker 32 again to perform the work.

After that, when the project period ends or sufficient valid data is secured, the service provider 20 ends the project (S49) and calculates the final result based on the secured valid data. And provide it to the client 10 (S50).

At this time, before the end of the project, the service provider 20 evaluates the execution results of the worker 32 and the inspector 34, calculates the work cost and the inspector cost according to the evaluation, and provides them to the worker 32 and the inspector 34. To do.

In FIGS. 1 and 2, the terms client, service provider, worker, and inspector are simply expressed, but these are smartphones, tablet PCs, PDAs, laptop PCs, and desktop PCs operated by each participant. , A computer device such as a server, or a telecommunications device.

FIG. 3 is a flowchart of a method for obtaining a highly reliable work result through a cross-check according to an embodiment of the present invention, and FIG. 4 shows an objective evaluation according to an embodiment of the present invention. FIG. 5 and FIG. 6 are illustrations illustrating the required work, and FIGS. 5 and 6 are illustrations illustrating the work required for subjective evaluation according to an embodiment of the present invention.

It can be understood that the steps S110 to S180 shown in FIG. 3 are performed by a platform server (hereinafter referred to as a server) operated by the service provider 20, but the steps S110 to S180 are not limited thereto.

Referring to FIG. 3, in step S110, the server assigns the same work to a plurality of workers 32. Here, the work means an alternative type work in which one is selected from the two options.

That is, a plurality of workers 32 to which the same work is assigned will select one from the two options and perform the work at their own discretion.

The server selects n workers 32 who perform the same work and assigns the same work. Here, n means a natural number of 3 or more.

As an embodiment, n can be an odd number of 3 or more. The server can assign the same work to three or more odd (ie, 3, 5, 7, etc.) workers 32. This is to separate the majority opinions, as will be described later.

At this time, the server can determine n according to the reliability required by the requester 10. The larger n is, that is, the more the same work is assigned to more workers 32, the more work results can be obtained for the same work, so that the reliability of the work results can be increased.

The alternative type of work assigned to the worker 32 may be a work that requires an objective evaluation or a work that requires a subjective evaluation. In particular, in the case of work that requires subjective evaluation, it is possible to secure highly reliable data formed by a large number of opinions by having a large number of workers participate in the same work.

With reference to FIG. 4, the worker 32 is given a text and a question based on the text. The worker 32 answers after objectively judging whether the given question is a correct question or an incorrect question with reference to the text.

For example, the worker 32 can select "○" by referring to the content of the text in response to the question "Is Eric Thames' nationality the United States?"

Referring to FIG. 5, when the worker 32 is given two sentences having similar meanings, the worker 32 answers after subjectively judging whether the two sentences have the same meaning or different meanings.

For example, if it is determined that the meanings of "looks honest and good" and "looks good and beautiful" are the same, the worker 32 can select "same meaning (○)", and also. When it is determined that the meanings of "looks honest and good" and "looks good and beautiful" are different, the worker 32 can select "different meaning (x)".

Referring to FIG. 6, when a person photograph (specifically, a face photograph) is given to the worker 32, the worker 32 looks at the given photograph and subjectively judges the gender of the person. answer.

For example, a photo with a hat on and gender indistinguishable is provided and the question "Please select a gender" is given. On the other hand, the worker 32 can select "boy" or "girl" depending on the judgment.

The data secured through the exemplary work of FIGS. 4 to 6 can be utilized as learning data for artificial intelligence or machine learning.

Next, referring to FIG. 3 again, in step S120, the server receives input of n work results from n workers 32.

When n is 3, the server receives input of three work results from three workers 32.

For example, if the work is an alternative type of work that selects ○ or ×, the server will be among "○○○", "XXX", "○○ ×", and "○ ××". Receive the results in one form of.

Next, in step S130, the server checks whether all n work results match. That is, the n work results are cross-checked to see if they all match.

For example, if the work is an alternative type of work that selects ○ or ×, and n is 3, the server is the work of three people for the work results of “○○○” and “XXX”. It is judged that the opinions of the three workers 32 are all in agreement, and it is judged that the opinions of the three workers 32 are not in agreement with respect to the work results of "○○ ×" and "○ ××".

If all n work results match, the server uses n work results as valid data. Detailed contents regarding this will be described later in step S180.

Next, in step S140, if the n work results do not match, the server reassigns the same work to m new additional workers 32. Here, m means the number of work results corresponding to a minority opinion out of n work results.

That is, when n work results are "○○ ×", the server determines that ○ is a majority opinion and × is a minority opinion. Therefore, the server selects the number of work results corresponding to x, that is, one new additional worker 32, and reassigns the same work.

Further, when n is 5, if the server is provided with the work result of "○○○○ ×" or "○○○ ××" by five workers, the server judges × as a minority opinion and determines that × is a minority opinion. The number of work results corresponding to x, that is, one or two new additional workers 32 is selected and the same work is reassigned.

For example, if n workers 32 are assigned the task of determining whether two similar sentences are the same, as shown in FIG. 5, the m new additional workers 32 are also assigned. Similarly, the task of determining whether two similar sentences are the same is assigned.

Next, in step S150, the server receives input of m rework results from m new additional workers 32.

For example, if m is 1 (ie, if the number of work results corresponding to the minority opinion is 1 out of n work results), the server will re-start from one new additional worker 32. Receive input of work results.

At this time, if the work is an alternative type work that selects ○ or ×, the server will be provided with the rework result of ○ or × from one new additional worker 32. ..

Next, in step S160, the server confirms whether the m rework results and the work results corresponding to the majority of the n work results match. That is, the results of the m rework results and the work results corresponding to the majority opinions among the n work results are cross-checked to confirm whether they match.

That is, when n is 3 and m is 1, the server confirms whether the two work results corresponding to the majority opinion and the one rework result match each other.

For example, when the work is an alternative type work in which ○ or × is selected, the two work results corresponding to the majority opinion may be “○○” or “XX”. .. At this time, it is determined whether or not one rework result matches depending on whether it is ○ or ×.

Specifically, if the two work results are "○○" and one rework result is ○, the server determines that the rework result and the work result corresponding to the majority opinion match. ..

When the rework result and the work result corresponding to the majority opinion match, the server uses the rework result and the work result corresponding to the majority opinion as valid data. Detailed contents regarding this will be described later in step S180.

On the other hand, if the two work results are "○○" and one rework result is ×, the server determines that the rework result and the work result corresponding to the majority opinion do not match.

Next, in step S170, if the rework result and the work result corresponding to the majority opinion do not match, the server assigns the same work to the inspector 34.

For example, the server assigns the inspector 34 an alternative task of selecting "○" or "x". That is, the inspector 34 is assigned the same work assigned to the plurality of workers, and determines whether "○" is correct or "x" is correct.

After that, the server secures the final data based on the acceptance result input from the inspector 34. Detailed contents regarding this will be described later in step S180.

Further, as an embodiment, if the rework result and the work result corresponding to the majority opinion do not match in step S170, the server can invalidate the work result and the rework result.

That is, when n is 3 and m is 1, if one rework result and two work results corresponding to the majority opinion do not match, the server outputs one rework result and three work results. Both can be disabled. As a result, all work results are invalidated.

After invalidating any of the work results, the server can assign the same work to another n workers 32 and perform the same process from the beginning. Alternatively, the server may assign work to the inspector 34 to perform the work directly. Normally, the inspector 34 is selected as a person who has many years of experience and high reliability as compared with the worker 32, so that the work result of the inspector 34 can be used as it is as valid data.

Next, in step S180, the server uses the work result that has passed the cross check or the work result based on the acceptance inspection result as valid data.

To summarize, there are three cases where the server uses the work result as valid data.

First, in step S130, when all n work results match, the server uses n work results as valid data.

For example, when the work is an alternative type work in which ○ or × is selected and n is 3, the three work results received from the three workers 32 are “○○○” or “×”. If it is confirmed that all of them match, the server immediately uses it as valid data without performing acceptance inspection for the three work results. Here, valid data means data used to generate an output (final product) at the end of a project. That is, if the three work results are "○○○", the work result for the corresponding work is used as "○".

Second, in step S160, when the rework result and the work result corresponding to the majority opinion match, the server uses the rework result and the work result corresponding to the majority opinion as valid data.

For example, if the work is an alternative type work that selects ○ or ×, n is 3, and m is 1, one rework result is “○”, which corresponds to a majority opinion. When it is confirmed that the two work results are "○○" and match, the server uses only one rework result and two work results corresponding to the majority opinions as valid data.

That is, the work result for the corresponding work is used as "○", and one of the three work results (x) corresponding to the minority opinion is discarded.

At this time, the server does not inspect the rework result and the work result corresponding to the majority opinion, that is, the work result for the corresponding work is "" by the inspector (rather, because the majority opinion may be wrong). Immediately use it as valid data without performing acceptance inspection to see if "○" is correct.

Third, in step S170, the server uses which of the entire work results (n work results and m rework results) as valid data according to the acceptance inspection result for the same work. Determine if.

As an embodiment, when the acceptance result is input from the inspector 34 that "○" is correct for the alternative type work, the server has m rework results and n work results. Of these, only the work results corresponding to "○" are used as valid data.

For example, if the majority opinion is "x", the minority opinion is "○", and one rework result is "○" among the three work results, the server receives the acceptance result ("○". According to (○)), only one work result (“○”) and one rework result (“○”) corresponding to the minority opinion are used as valid data.

FIG. 7 is a block diagram of a work result acquisition device utilizing a cross check according to an embodiment of the present invention.

Referring to FIG. 7, the highly reliable work result acquisition device 200 (hereinafter referred to as the work result acquisition device) through the cross check includes a communication unit 210, a storage unit 220, a determination unit 230, and a sorting unit 240.

The communication unit 210 plays a role of transmitting the same work to n workers and receiving n work results from the n workers. When rework is required, the communication unit 210 sends the same work to m new additional workers and receives m rework results from m new additional workers. Play a role.

Here, n is a natural number that is an odd number of 3 or more, and m is the number of work results that correspond to a minority opinion among the n work results.

The storage unit 220 serves to store the results of n operations input from n workers and m re-operations input from m new additional workers.

The judgment unit 230 determines whether all the n works input from the n workers match, and m rework results of the work results corresponding to the majority opinions among the n works. It plays a role in determining whether the result is consistent.

The sorting unit 240 plays a role of sorting n work results as valid data when all n works match.

Further, when the work result corresponding to the majority opinion among the n works matches the rework result of m pieces, the sorting unit 240 uses the rework result and the work result corresponding to the majority opinion as valid data. It plays a role in extracting.

Further, the sorting unit 240 has a role of selecting n workers determined according to the reliability and selecting m new additional workers depending on whether or not all the n works match. Fulfill.

The work result acquisition device 200 described with reference to FIG. 7 can be provided as a component of the server described above.

The highly reliable work result acquisition method through cross-checking according to the embodiment of the present invention described above is executed by being combined with a computer as hardware, and thus is executed by a program (or application). It can be embodied and stored in a medium.

The above-mentioned program is C, C ++, JAVA, which can be read by the processor (CPU) of the computer through the device interface of the computer so that the computer can read the program and execute the method embodied by the program. A code encoded in a computer language such as (registered trademark), Ruby, or machine language can be included. Such code can include a function that defines the functions required to execute the method and related functional code (Funcational Code), and each of the functions is performed by the processor of the computer. It can include control code related to the execution procedure, which is necessary for executing the execution according to a predetermined procedure. In addition, such code should refer to additional information or media necessary for the processor of the computer to execute each of the functions at any position (address) in the internal or external memory of the computer. Further memory reference related code can be included. In addition, when the processor of the computer needs to communicate with some other computer or server which is remote (remote) in order to execute each of the functions, the code is the communication of the computer. Modules can be used to further include communication-related codes for how to communicate with some (any) remote computer, server, etc., what information and media should be sent and received during communication, etc. ..

The stored medium is not a medium that stores data for a short time such as a register, a cache, or a memory, but a medium that stores data semipermanently and can be read by an apparatus. Specifically, examples of the storage medium include, but are not limited to, ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, and the like. That is, the program can be stored in various recording media on various servers to which the computer can be connected or in various recording media on the user's computer. Further, the media may be distributed to computer systems connected to each other by a network, and a code that can be read by a computer in a distributed manner may be stored.

Although the embodiments of the present invention have been described above with reference to the accompanying drawings, ordinary engineers in the technical field to which the present invention belongs can use the present invention without changing its technical idea or essential features. It can be understood that it can be carried out in a concrete form of. Therefore, the examples described above should be understood as exemplary in all respects and not restrictive.

10 Client 20 Service provider 30 Mass 32 Worker 34 Inspector 200 Inspector Monitoring device 210 Communication unit 220 Storage unit 230 Judgment unit 240 Sorting unit

Claims (7)

It ’s a computer-based method,
At the stage of assigning the same work to n workers (n is a natural number of 3 or more),
At the stage of receiving input of n work results from n workers,
At the stage of confirming whether all the n work results match, and
If the n work results do not match, the same work is reassigned to m new additional workers (m is the number of work results corresponding to the minority opinion of the n work results). When,
At the stage of receiving input of m rework results from the newly added worker of m names,
At the stage of confirming whether the m rework results match all the work results corresponding to the majority opinions among the n work results, and
When the rework result and the work result corresponding to the majority opinion all match, the step of using the work result corresponding to the majority opinion as valid data is included.
The work is to select an answer to a question that requires subjective evaluation, and asks the n workers to select one of two options at their own discretion. It ’s an alternative type of work,
When the rework result and the work result corresponding to the majority opinion do not match,
At the stage of assigning the same work to the inspector,
At the stage of receiving the input of the acceptance result from the inspector,
Depending on the acceptance result, the stage of determining whether to use the work result corresponding to the majority opinion or the work result corresponding to the minority opinion as valid data, and
Reliable work result acquisition method through cross-check, including further. When the n work results match each other,
The method for obtaining highly reliable work results through a cross-check according to claim 1, further comprising a step of using the n work results as valid data without performing acceptance inspection. When the rework result and the work result corresponding to the majority opinion do not match,
The highly reliable work result acquisition method through a cross-check according to claim 1, further comprising a step of invalidating the n work results and the m rework results. The highly reliable work result acquisition method through the cross check according to claim 1, wherein n is an odd number. The method for obtaining a highly reliable work result through a cross check according to claim 1, wherein the n is determined according to the reliability. A computer program that is combined with a computer that is hardware and stored in a medium to execute a highly reliable work result acquisition method through the cross-check according to any one of claims 1 to 5. It is a highly reliable work result acquisition device through cross-checking.
The same work is sent to n workers, m new additional workers, and an inspector, n work results are received from the n workers, and the m new additions are received. Communication unit that receives m rework results from the worker and receives the acceptance result from the inspector (n is a natural number of 3 or more, m is a work result corresponding to a minority opinion among n work results) Number of) and
A storage unit for storing the n work results input from the n workers, the m rework results input from the newly added m workers, and the acceptance inspection result.
It is determined whether or not all the n work results input from the n workers match, and the m rework results correspond to the majority opinions of the n work results. And a judgment unit that judges whether all match
When the same work is assigned to the n workers, the n work results all match, the n work results are selected as valid data, and the n work results do not match. When the same work is reassigned to the newly added worker with the m name, and the m rework results all match the work results corresponding to the majority opinion among the n work results, the above. The work result corresponding to the majority opinion is selected as valid data, and if the rework result and the work result corresponding to the majority opinion do not match, the same work is assigned to the inspector and the acceptance result is adjusted. Including the work result corresponding to the majority opinion and the sorting unit for selecting any of the work results corresponding to the minority opinion as valid data.
The work selects an answer to a question that requires subjective evaluation, and allows the n workers to select one of two options at their own discretion. A highly reliable work result acquisition device through cross-check, which is an alternative type of work.

Images