KR102195964B1 - Method for operating self check process of worker of crowdsourcing based project for artificial intelligence training data generation - Google Patents

Abstract

Provided is a method for operating a self-checking process of a worker of a crowdsourcing-based project for generating artificial intelligence training data based on a rejection ratio to increase a progress ratio of the project. According to the present invention, the method comprises the following steps: assigning a plurality of work of a crowdsourcing-based project (hereinafter, referred to as a project) to a plurality of workers to request execution of the work; receiving a plurality of work results from the plurality of workers; assigning the plurality of work results to a plurality of inspectors to request execution of inspection; receiving a plurality of inspection results of the plurality of work results from the plurality of inspectors as inspection pass or rejection; calculating a rejection ratio for each worker based on the plurality of inspection results; selecting at least one self-checking candidate among the plurality of workers based on the rejection ratio; and requesting the self-checking candidate to perform recheck and rework for the work results of the self-checking candidate, which are remaining without being inspected.

Description

How to operate the self-check process of workers based on the rejection rate of a crowdsourcing-based project for generating artificial intelligence learning data{METHOD FOR OPERATING SELF CHECK PROCESS OF WORKER OF CROWDSOURCING BASED PROJECT FOR ARTIFICIAL INTELLIGENCE TRAINING DATA GENERATION}

The present invention relates to a method of operating a self-check process of workers based on the rejection rate of a crowdsourcing-based project for generating artificial intelligence learning data.

Recently, more and more companies collect and process large amounts of data on a crowdsourcing basis that engages the general public in some process of corporate activities. In other words, by opening a project and allowing the general public, that is, workers to participate in the project, necessary information is collected through the work results completed by the workers.

Specifically, when one project is opened, multiple tasks are assigned to multiple workers. Each worker performs the assigned task and provides the task result. And, a plurality of work results are assigned to a plurality of inspectors. Each inspector performs an inspection on the assigned work results.

However, there are far more workers currently participating in the project than inspectors. Accordingly, as each worker performs the work, the work result input from the plurality of workers increases, and the inspection bottleneck occurs due to insufficient number of inspectors to inspect it.

Accordingly, among the plurality of job results submitted by the worker, the first input job result is initially inspected, and the later input job result is inspected later. In other words, there is a time difference between inspecting the results of a plurality of work submitted by the worker.

If the inspection result of the initially inspected work result is rejected due to the inspection time difference, the operator cannot correct the input work result later and wait until the inspection result, even though he recognizes that his work method is wrong. Things happen.

In addition, since the inspector's inspection must be completed in order for the operator to correct the wrong work result, there is a problem that inspector resources are wasted in a situation where the inspector is insufficient.

Unexamined Patent Publication No. 10-2014-0095956, 2014.08.04.

The problem to be solved by the present invention is to provide a method of operating a self-check process of workers based on the rejection rate of a crowdsourcing-based project for generating artificial intelligence learning data.

However, the problem to be solved by the present invention is not limited to the problem as described above, and other problems may exist.

A method of operating a self-check process of workers based on the rejection rate of a crowdsourcing-based project according to an aspect of the present invention for solving the above-described problems, assigns a plurality of tasks of a crowdsourcing-based project (hereinafter, a project) to a plurality of workers. Requesting to perform a task, receiving a plurality of work results from the plurality of workers, assigning the plurality of work results to a plurality of inspectors and requesting to perform the inspection, the plurality of tasks from the plurality of inspectors Receiving a plurality of inspection results for the result as passing or rejecting the inspection, calculating a rejection rate for each worker based on the multiple inspection results, and self-checking at least one of the plurality of workers based on the rejection rate Selecting a subject and requesting the self-check subject to reconfirm and perform rework on the remaining work results that have not been inspected by the inspector of the self-check subject.

In some embodiments of the present invention, the step of calculating the rejection rate for each worker based on the plurality of inspection results includes the initial inspection result compared to the number of work results inspected by the inspector so far for each operator. The ratio of the number of cases entered as rejection may be calculated as the rejection rate.

In some embodiments of the present invention, the step of selecting one or more self-check targets from among the plurality of workers based on the rejection rate comprises: among the plurality of workers, the self-check target person with the rejection rate higher than the reference rejection rate Can be selected as.

In some embodiments of the present invention, the step of selecting one or more self-check targets from among the plurality of workers based on the rejection rate includes the number of work results inspected by the inspector from among the plurality of workers so far. The one or more self-check targets may be selected for more workers.

In some embodiments of the present invention, the reference rejection rate may be determined according to the difficulty of the project.

In some embodiments of the present invention, the method may further include the step of excluding residual work results that have not been inspected by the inspector of the self-check subject from inspection.

In some embodiments of the present invention, the method may further include requesting, from the self-check subject, requesting the self-check subject to confirm the total number of rejection reasons and a work result inputted as a rejection.

In some embodiments of the present invention, the step of requesting the self-check target person to reconfirm and perform rework on the remaining work results that have not been checked by the examiner of the self-check target person, the self-check target person confirms the total number Upon completion, it is possible to request the self-check target person to reconfirm and perform rework on the remaining work results that have not been checked by the inspector of the self-check target person.

In some embodiments of the present invention, the method comprises the steps of, when the self-check subject completes the reconfirmation and rework, returning the remaining work results that have not been inspected by the inspector of the self-check subject to the inspection target. It may further include.

A computer program according to another aspect of the present invention for solving the above-described problem is combined with a computer that is hardware to execute a self-check process operation method of workers based on the rejection rate of the crowdsourcing-based project, and on a computer-readable recording medium. Is saved.

Other specific details of the present invention are included in the detailed description and drawings.

According to the present invention described above, by reconfirming and reworking the work result of the person who has not yet been inspected, based on the reason for rejection of the work result inspected as a rejection by the operator, the inspector reconfirmed and corrected the work result through rework Since the inspection is sufficient, the inspection time required by the inspector is reduced, and accordingly, the inspector resources can be saved.

In addition, the result of the work returned by the inspector through the reconfirmation and rework of the worker is reduced, so that the operator rework the rejected work result, and a series of processes in which the inspector re-inspect the rework result is omitted, and the waiting time is reduced. By disappearing, the project progress can be improved.

The effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the following description.

1 is a conceptual diagram of a crowdsourcing service according to an embodiment of the present invention.
2 is a flowchart illustrating a crowdsourcing-based project progress process according to an embodiment of the present invention.
3 is a flowchart of a method of operating a self-check process of a worker based on rejection rate of a crowdsourcing-based project according to an embodiment of the present invention.
4 is an exemplary view for explaining a comparison between the number of inspected work results and the reference number according to an embodiment of the present invention.
5 is a flowchart illustrating a comparison between a worker's rejection rate and a reference rejection rate according to an embodiment of the present invention.
6 is an exemplary view for explaining reconfirmation and rework of a residual work result according to an embodiment of the present invention.
7 is an exemplary diagram for explaining a different process depending on whether or not to select a self-check subject according to an embodiment of the present invention.
FIG. 8 is a diagram illustrating an apparatus for operating a self-check process of a worker based on a rejection rate for a crowdsourcing-based project according to an embodiment of the present invention.

Advantages and features of the present invention, and a method of achieving them will become apparent with reference to the embodiments described below in detail together with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but may be implemented in a variety of different forms, only the present embodiments are intended to complete the disclosure of the present invention, It is provided to fully inform the technician of the scope of the present invention, and the present invention is only defined by the scope of the claims.

The terms used in the present specification are for describing exemplary embodiments and are not intended to limit the present invention. In this specification, the singular form also includes the plural form unless specifically stated in the phrase. As used in the specification, “comprises” and/or “comprising” do not exclude the presence or addition of one or more other elements other than the mentioned elements. Throughout the specification, the same reference numerals refer to the same elements, and “and/or” includes each and all combinations of one or more of the mentioned elements. Although "first", "second", and the like are used to describe various elements, it goes without saying that these elements are not limited by these terms. These terms are only used to distinguish one component from another component. Therefore, it goes without saying that the first component mentioned below may be the second component within the technical idea of the present invention.

Unless otherwise defined, all terms (including technical and scientific terms) used in the present specification may be used as meanings that can be commonly understood by those of ordinary skill in the art to which the present invention belongs. In addition, terms defined in a commonly used dictionary are not interpreted ideally or excessively unless explicitly defined specifically.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a conceptual diagram of a crowdsourcing service according to an embodiment of the present invention.

Referring to FIG. 1, a crowdsourcing service is performed by being composed of a client 10, a service provider 20, and a public 30.

The sponsor 10 refers to a company or individual who requests a crowdsourcing-based project (hereinafter, referred to as a project).

The requester 10 requests a project for the purpose of collecting source data or annotating data for generating artificial intelligence learning data. The data generated through the project can be used as learning data for arbitrary machine learning such as supervised learning, unsupervised learning, and reinforcement learning. Collection of source data refers to collecting raw data such as recorded voice collection and photo collection. Data annotation refers to inputting relevant annotation data into source data such as text, photos, and videos. For example, the data annotation may include finding an entity in a given fingerprint or finding a similar sentence, but is not limited thereto. Meanwhile, the types of the above-described projects are only one embodiment, and various projects may be handled in the present invention according to the design of the client.

The service provider 20 refers to a company that provides crowdsourcing services.

When the service provider 20 receives a request for a project for a product or service from the client 10, the service provider 20 allocates a task for the project to the general public 30 and receives the work result from the public 30. Thereafter, the final product extracted based on the work result is provided to the client 10.

At this time, the service provider 20 provides a crowdsourcing service to the client 10 and the public 30 through a crowdsourcing platform (hereinafter, the platform). That is, when the service provider 20 receives a project from the client 10, it opens the project on the platform. Thereafter, when the work result for the open project is received from the public 30, the project is terminated on the platform, and the final product may be extracted and provided to the client 10.

The public 30 refers to the general public participating in the project open on the platform. Here, the public 30 may participate in a project opened on the platform through an application or website provided by the service provider 20.

The public 30 is composed of a worker 32 and an inspector 34.

The worker 32 decides to participate in a specific project among a plurality of projects open on the platform. Thereafter, the operator 32 performs an operation such as collection of source data or data annotation, and transmits it to the platform.

The inspector 34 decides to participate in a specific project among a plurality of projects open on the platform. Thereafter, the inspector 34 inspects the result of the work performed by the operator 32. As a result of performing the inspection, the inspector 34 may perform inspection pass processing or rejection processing, and input a rejection reason during rejection processing. In the case of passing the inspection, it is not necessary to rework and re-examination due to this, so passing inspection has the same meaning as completion of inspection.

2 is a flowchart illustrating a crowdsourcing-based project progress process according to an embodiment of the present invention.

First, the requester 10 requests one or more projects to the service provider 20 (S11).

Thereafter, the service provider 20 opens the requested project on the platform (S12). At this time, the service provider 20 may determine the grade in consideration of the difficulty of the project before opening the project. That is, depending on the difficulty level, it is possible to determine which level or higher public 30 to expose the project. Accordingly, it is possible to increase the reliability of the work result of the project.

Thereafter, the service provider 20 requests the work by assigning the work to the workers 32 of the corresponding level or higher according to the level of the project (S13).

Thereafter, the worker 32 performs the assigned task (S14). In this case, the worker 32 may input the reason for the inability to work without performing the work for a work in which the work itself is impossible for some reason.

Thereafter, the service provider 20 receives the work result from the worker 32 (S15), and assigns the work result to the inspector 34 to request the inspection (S16).

Thereafter, the inspector 34 performs the assigned inspection (S17). At this time, the inspector 34 decides to pass the inspection when it is determined that the work has been properly performed, and if it is determined that the inspection work is wrong, the inspector 34 rejects it. During rejection processing, the inspector 34 inputs the rejection reason for what reason the job was judged to be wrong.

Thereafter, the service provider 20 receives the inspection result from the inspector 34 (S18).

If the inspection result passes the inspection, the service provider 20 uses the work result as valid data, and based on this, extracts the final product at the end of the project.

When the inspection result is rejection processing, the service provider 20 may internally perform the inspection again, or assign the work to the worker 32 again to perform the rework. When reworking, re-examination by an inspector is required.

Thereafter, when the project period ends or sufficient valid data is secured, the service provider 20 terminates the project (S19), calculates a final result based on the secured valid data, and provides it to the client 10 ( S20).

At this time, before the end of the project, the service provider 20 evaluates the performance results of the worker 32 and the inspector 34, calculates the work cost and the inspection cost according to the evaluation, to the worker 32 and the inspector 34. give.

In FIGS. 1 and 2, it is simply expressed as a requester 10, a service provider 20, an operator 32, and an inspector 34, but these are smartphones, tablets, PDAs, laptops, desktops, operated by each participant. It means a computer device such as a server or a telecommunication device.

The general progress process described in FIG. 2 has a problem in that it is difficult to predict the progress of the project due to the waiting time generated by repeating the process of work-review-rework-review, and accordingly, in operating the project, within the target period. Situations such as data collection failure occur.

In the present invention, in order to minimize the process of inspection-rework-re-review as described above, the self-check subject is asked to check and correct the remaining work results of the person who have not yet been inspected. Accordingly, the inspector can inspect the work results with high reliability, thereby saving the inspector's resources and consequently increasing the project progress rate.

In the following, referring to Figs. 3 to 6, the ratio of the number of cases in which the initial inspection result is entered as rejection is calculated as the rejection rate of the worker, and based on this, the self-check target person among a plurality of workers is selected to The explanation will be made on enabling more efficient inspection in performing the initial inspection by multiple inspectors by reconfirming and reworking the remaining work results.

3 is a flowchart of a method of operating a self-check process of a worker based on rejection rate of a crowdsourcing-based project according to an embodiment of the present invention. 4 is an exemplary view for explaining a comparison between the number of inspected work results and the reference number according to an embodiment of the present invention. 5 is a flowchart illustrating a comparison between a worker's rejection rate and a reference rejection rate according to an embodiment of the present invention. 6 is an exemplary view for explaining reconfirmation and rework of a residual work result according to an embodiment of the present invention.

Meanwhile, the steps shown in FIG. 3 may be understood as being performed by a platform server (hereinafter, referred to as a server) operated by the service provider 20, but are not limited thereto.

In addition, a plurality of workers 32 or a plurality of inspectors 34 perform work using a predetermined terminal device. The terminal device of the operator 32 or the inspector 34 may be a computer device or a telecommunication device such as a smartphone, tablet, PDA, laptop, desktop, etc., but is not limited thereto.

First, referring to FIG. 3, the server assigns a plurality of tasks of a crowdsourcing-based project (hereinafter, a project) to a plurality of workers to request the task performance (S110), and receives a plurality of task results from a plurality of workers. (S120), assigning the plurality of work results to a plurality of inspectors to request inspection (S130), and receiving a plurality of inspection results for the plurality of work results from the plurality of inspectors as passing or rejecting the inspection (S140) ).

Here, the plurality of workers 32 and the plurality of inspectors 34 refer to all workers and inspectors participating in the same project.

Since the steps S110 to S140 overlap with the steps S13 to S18 described above with reference to FIG. 2, a detailed description will be omitted.

Subsequently, the server calculates the rejection rate for each worker 32 based on the plurality of inspection results (S150).

The server calculates the ratio of the number of cases in which the initial inspection result is input as rejection to the number of work results inspected by the inspector 34 so far for each worker 32 as rejection rate. For example, out of the total work results submitted by worker A, if the number of work results inspected by the inspector 32 so far is 10, and among them, if the initial inspection result is 3 cases entered as rejection, worker A The rejection rate of can be calculated as 30 percent.

The rejection rate of the worker 32 is updated based on the initial inspection result input after the work result of the worker 32 is initially inspected by the inspector 34, respectively.

On the other hand, the server separately calculates the rejection rate in which all the initial inspection results and rework inspection results input by the inspector 34 so far for each worker 32 are reflected, and evaluates the worker 32 at the end of the project. Can be utilized. However, it is not limited thereto, and the worker 32 may be evaluated using the rejection rate reflecting only the initial inspection result.

In the present invention, since the subject to self-check is selected based on the initial inspection result, the rejection rate described below means the ratio of the number of cases in which the initial inspection result is entered as rejection.

Subsequently, the server selects one or more self-check targets from among the plurality of workers based on the rejection rate (S160).

Specifically, among the plurality of workers 32, the server has the number of work results inspected by the inspector 34 so far greater than the reference number, and the rejection rate of the inspected work results is higher than the reference rejection rate. Select as a self-check target.

First, the server compares the number of work results inspected by the inspector 34 for each worker 32 with the reference number, and compares the number of work results inspected by the inspector 34 among the plurality of workers 32 so far. The number of workers 32 is extracted more than the reference number.

Here, the reference number means the minimum number of cases in which the rejection rate of the inspected work result becomes significant. This is because it is difficult to determine whether the worker 32 is properly performing the work through the rejection rate of the inspected work result until the number of cases exceeds the reference number.

Referring to FIG. 4, assuming that the number of reference cases is 10, out of a total of 50 work results submitted by worker A, 20 work results have been inspected by the inspector 34 so far, and the total number of work submitted by worker B If there are 15 work results that have been inspected by the inspector 34 out of 50 work results, and 8 out of 40 work results submitted by worker C, that have been inspected by the inspector 34, The server extracts worker A and worker B whose number of inspected work results exceeds the standard number of 10.

Thereafter, the server selects a self-check target person by comparing the rejection rate for the plurality of workers 32 extracted based on the reference number.

That is, the server compares the rejection rate of the extracted work results of each of the workers 34 to the present with the reference rejection rate, and retrieves one or more workers 32 whose rejection rate is higher than the reference rejection rate. Select as a self-check target.

Here, the standard rejection rate is determined according to the difficulty of the project.

The higher the difficulty of the project, the lower the standard rejection rate can be determined. This is to enable most of the workers 32 to be selected as self-check targets by setting the standard rejection rate low because, if the difficulty of the project is high, there is a very high possibility that the worker 32 will perform the task incorrectly. Accordingly, most of the workers 32 can check the remaining work results of the person who have not yet been inspected, so that the inspector 34 can save resources.

Referring to FIG. 5, out of 20 work results inspected so far by worker A, the number of work results rejected by the inspection result is 10, and jobs in which the inspection result is rejected out of 15 work results inspected by worker B so far. If the number of results is 5, worker A's rejection rate is calculated as 50 percent, and worker B's rejection rate is calculated as 34 percent. Assuming that the standard rejection rate is 40%, the server selects worker A whose rejection rate is higher than the reference rejection rate as a self-check target.

Referring back to FIG. 3, the server requests the self-check target person to re-check and perform rework on the remaining work results that have not been checked by the self-check target person's inspector (S170).

Before requesting reconfirmation and rework, the server first excludes the remaining work results that have not been reviewed by the inspector 34 of the self-check subject from the subject of inspection. This is to enable the operator 32, who is a self-check target, to first reconfirm and perform rework on the remaining work results before the remaining work results are inspected by the inspector 34.

In addition, before requesting reconfirmation and rework execution, the server requests the self-check target person to confirm the total number of work results and reasons for rejection, in which the initial inspection result of the self-check target person is rejected.

Referring to FIG. 5, the server may request to confirm the total number of work results of a total of 10 cases in which the inspection result of worker A selected as a self-check target is rejected.

The self-check subject performs rework on the rejected task result by confirming the rejection reason entered by the examiner 34 for the task result inputted as rejection, and correcting the task result based on the rejection reason. .

When the self-check target person completes the verification of the total number, that is, rework on the rejected work result, the server requests the self-check target person to reconfirm and perform rework on the remaining work result of the self check target.

Referring to FIG. 5, the server may request reconfirmation and rework from the self-check target person for 30 remaining work results that have not yet been reviewed among 50 work results of the self-check target person.

Since the self-check target learns the correct work method through rework on the rejected work results, it is possible to recheck and rework the remaining work results correctly.

When the self-check subject completes the reconfirmation and rework, the server returns the remaining work results that have not been inspected by the inspector 34 of the self-check subject to the inspection subject.

Referring to FIG. 6, a recheck and rework (self-check) of the remaining work results that have not yet been inspected by the inspector 34 were requested from worker A, who is a self-check target. At this time, as described above, the remaining work results are excluded from the inspection target.

Worker A judges whether or not the remaining work results are properly worked based on learning, and corrects the work results determined to be incorrect.

That is, as shown in FIG. 6, the worker A determines that eight remaining work results are incorrectly worked out of 10 remaining work results that have not yet been inspected, and corrects the eight remaining work results. In addition, the remaining two work results are determined to have already been properly worked and are not corrected.

The server returns the remaining work results corrected by the self-check target person and the remaining work results that have not been corrected (determined that they have been properly worked) back to the subject of inspection.

The remaining work results returned to the inspection target are assigned to the inspector 34 and inspected. The inspector 34 can inspect the work result with increased reliability through reconfirmation and rework of the self-check target. That is, the number of cases immediately passing through the inspection than the rejected cases can be performed quickly, and accordingly, the inspection bottleneck can be eliminated.

In the above, it has been described that after the total check of the rejected work results is completed, the remaining work results are rechecked and reworked, but according to the embodiment, the total check of the rejected work results and the remaining work results are checked and Rework can also be done in parallel.

7 is an exemplary diagram for explaining a different process depending on whether or not to select a self-check target according to an embodiment of the present invention.

As described above, if the number of work results inspected so far is greater than the reference number, and the rejection rate of the work results inspected so far is higher than the reference rejection rate, it is selected as a self-check target.

The rejected work results of the worker 32 selected as the self-check target are fully checked (ie, reworked) by the self-check target.

On the other hand, the residual work result of the worker 32 selected as a self-check target is excluded from the subject of inspection, and is re-checked and reworked by the self-check target person.

Although not shown in FIG. 7, the server not only excludes the remaining work results of the self-check target from the subject of inspection, but also self-checks until the self-check target reconfirms and rework (self-check) for the remaining work results. The assignment of new tasks to the subject may be discontinued. This is because there is a concern that the number of rejected work results may increase if new tasks are continuously assigned before the self-check is completed.

Thereafter, when the self-check subject completes reconfirmation and rework, the reconfirmation and rework of the remaining work results are returned to the inspection subject again. At the same time, the server resumes assigning new tasks to self-check subjects.

In this case, the server may distribute and pay the inspection cost for the remaining work results for which the reconfirmation and rework has been completed, to the inspector 34 and the worker 32 who is a self-check target.

In other words, the server recognizes that the self-check subject has re-checked and re-worked the remaining work results as being inspected, and distributed and paid the inspection cost as compensation for the re-checking and re-working.

For example, if the work unit price is 500 won and the inspection unit price is 600 won, the self-check target worker 32 receives the work cost for the remaining work results at 500+(600/2)=800 won, and the inspector ( 34) can be paid 600/2=300 won for the inspection fee for the remaining work results.

On the other hand, even if the number of work results inspected so far is greater than the reference number, if the rejection rate of the work results inspected so far is lower than the standard rejection rate, the subject is not selected as a self-check.

The rejected work result of the worker 32 that has not been selected as a self-check target is fully confirmed (ie, reworked) by the worker 32. In other words, regardless of whether or not to be selected as a self-check target, the overall confirmation of the rejected work result is performed in the same manner.

On the other hand, the remaining work results of the worker 32 not selected as a self-check target are continuously inspected by the inspector 34.

Since the rejection rate is updated as the number of inspected work results accumulates, if the rejection rate of the inspected work results is higher than the standard rejection rate, the worker 32 is also selected as a self-check target, and the subsequent process is performed. .

Meanwhile, in the above description, steps S11 to S170 may be further divided into additional steps or may be combined into fewer steps according to an embodiment of the present invention. In addition, some steps may be omitted as necessary, and the order between steps may be changed. In addition, even if other contents are omitted, the contents of FIG. 8 to be described later can also be applied to the method of operating the self-check process of workers based on the rejection rate of the crowdsourcing-based project of FIGS. 1 to 7.

Hereinafter, an apparatus 200 for operating a self-check process of a worker based on a rejection rate of a crowdsourcing-based project according to an embodiment of the present invention will be described with reference to FIG. 8.

FIG. 8 is a diagram illustrating an apparatus for operating a self-check process of a worker based on a rejection rate for a crowdsourcing-based project according to an embodiment of the present invention.

Referring to FIG. 8, the self-check process operating device 200 (hereinafter, the self-check process operating device) of a worker based on the rejection rate of a crowdsourcing-based project according to the communication module 210, a memory 220, and a processor 230 Includes.

The communication module 210 transmits a crowdsourcing-based work for one project to a plurality of workers 32 to request a task to be performed, and receives a work result from the plurality of workers 32. The work results received from the plurality of workers 32 are transmitted to the plurality of inspectors 34 to request inspection, and the inspection results are received from the plurality of inspectors 34.

A program for selecting the self-check target person by calculating the rejection rate for each worker based on the data received from the communication module 210 in the memory 220, and requesting reconfirmation and rework of the remaining work result to the self-check target person Is saved.

The processor 230 executes a program stored in the memory 220. As the processor 230 executes the program stored in the memory 220, the self-check target is selected among the plurality of workers 32 based on the number and rejection rate of the job results inspected by the inspector for each worker. After selecting and excluding the remaining work results of the selected self-check subject from the subject of inspection, requesting reconfirmation and rework to the self-check subject, and when the attendees and rework are completed, the remaining work results of the self-check subject are subject to re-examination By returning to, it is possible to request an inspection from the inspector 34.

The self-check process operating apparatus 200 described with reference to FIG. 8 may be provided as a component of the above-described server.

The method of operating the self-check process of a worker based on the rejection rate of a crowdsourcing-based project according to an embodiment of the present invention described above is implemented as a program (or application) to be executed in combination with a computer, which is hardware, and stored in a medium. I can.

The above-described program includes C, C++, JAVA, Ruby, which can be read by a processor (CPU) of the computer through the device interface of the computer, in order for the computer to read the program and execute the methods implemented as a program. It may include a code (Code) coded in a computer language such as machine language. Such code may include a functional code related to a function defining necessary functions for executing the methods, and a control code related to an execution procedure necessary for the processor of the computer to execute the functions according to a predetermined procedure. can do. In addition, such code may further include additional information required for the processor of the computer to execute the functions or code related to a memory reference to which location (address address) of the internal or external memory of the computer should be referenced. have. In addition, when the processor of the computer needs to communicate with any other computer or server in the remote in order to execute the functions, the code uses the communication module of the computer to determine how It may further include a communication-related code for whether to communicate or what information or media to transmit and receive during communication.

The stored medium is not a medium that stores data for a short moment, such as a register, cache, memory, etc., but a medium that stores data semi-permanently and can be read by a device. Specifically, examples of the storage medium include, but are not limited to, ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, and the like. That is, the program may be stored in various recording media on various servers to which the computer can access, or on various recording media on the user's computer. In addition, the medium may be distributed over a computer system connected through a network, and computer-readable codes may be stored in a distributed manner.

The above description of the present invention is for illustrative purposes only, and those of ordinary skill in the art to which the present invention pertains will be able to understand that it can be easily modified into other specific forms without changing the technical spirit or essential features of the present invention. will be. Therefore, it should be understood that the embodiments described above are illustrative in all respects and not limiting. For example, each component described as a single type may be implemented in a distributed manner, and similarly, components described as being distributed may also be implemented in a combined form.

The scope of the present invention is indicated by the claims to be described later rather than the detailed description, and all changes or modified forms derived from the meaning and scope of the claims and their equivalent concepts should be interpreted as being included in the scope of the present invention. do.

10: Client
20: service provider
30: public
32: worker
34: inspector
200: self-check process operation device
210: communication module
220: memory
230: processor

Claims (10)

As a method performed by a computer,
Assigning a plurality of tasks of a crowdsourcing-based project (hereinafter, referred to as a project) to a plurality of workers and requesting the task to be performed;
Receiving a plurality of work results from the plurality of workers;
Assigning the plurality of work results to a plurality of inspectors and requesting to perform inspection;
Receiving a plurality of inspection results for the plurality of work results from the plurality of inspectors as passing or rejecting the inspection;
Calculating a rejection rate for each worker based on the plurality of inspection results;
Selecting one or more self-check targets from among the plurality of workers based on the rejection rate;
Requesting the self-check target person to confirm the total number of the rejection reason and the result of the initial inspection of the self-check target person; And
When the self-check target person completes the verification of the total number,
Including the step of requesting the self-check subject to reconfirm the remaining work results not inspected by the inspector of the self-check subject and to perform rework according to the reconfirmation result,
The step of selecting one or more self-check targets from among the plurality of workers based on the rejection rate,
Selecting a worker whose rejection rate is higher than a reference rejection rate among the plurality of workers as the self-check target,
How to operate the self-check process of workers based on the rejection rate of crowdsourcing-based projects. The method of claim 1,
The step of calculating the rejection rate for each worker based on the plurality of inspection results,
Calculating the ratio of the number of cases in which the initial inspection result was input as rejection to the number of work results inspected by the inspector so far for each worker, as the rejection rate,
How to operate the self-check process of workers based on the rejection rate of crowdsourcing-based projects. delete The method of claim 1,
The step of selecting one or more self-check targets from among the plurality of workers based on the rejection rate,
Selecting the one or more self-check targets for workers in which the number of work results inspected by the inspector so far among the plurality of workers is greater than the reference number,
How to operate the self-check process of workers based on the rejection rate of crowdsourcing-based projects. The method of claim 1,
The standard rejection rate is determined according to the difficulty of the project,
How to operate the self-check process of workers based on the rejection rate of crowdsourcing-based projects. The method of claim 1,
Further comprising the step of excluding the remaining work results that have not been inspected by the inspector of the self-check subject from the inspection object,
How to operate the self-check process of workers based on the rejection rate of crowdsourcing-based projects. delete delete The method of claim 6,
When the self-check subject completes the reconfirmation and rework performance according to the reconfirmation result, returning the remaining work result not inspected by the inspector of the self-check subject to the inspection subject,
How to operate the self-check process of workers based on the rejection rate of crowdsourcing-based projects. Computer-readable in order to perform the self-check process operation method of workers based on the rejection rate of any one of claims 1, 2, 4 to 6, and 9 in any one of claims 1, 2, 4 to 6, and 9 in combination with a computer A computer program stored on a recording medium.

Images