KR102195606B1 - Method for improving reliability by selective self check of worker of crowdsourcing based project for artificial intelligence training data generation - Google Patents

Abstract

Provided is a method for increasing reliability by selective self-checking of worker of a crowdsourcing-based project for generating artificial intelligence training data to increase the number of inspectors who participate in the product. According to the present invention, the method comprises the following steps: assigning a plurality of work of a crowdsourcing-based project to a plurality of workers to request execution of the work; receiving a plurality of work results from the plurality of workers; inquiring a recheck and rework (hereinafter, referred to as self-checking) intention for the work result of the worker to each worker and selectively receiving the self-checking intension from each worker; requesting execution of self-checking for the work result of the worker to the worker (hereinafter, referred to as a self-checking worker) who has the self-checking intention; receiving a work result completing self-checking from the self-checking worker; assigning a plurality of work results including the work result completing the self-checking to a plurality of inspectors to request execution of inspection; and receiving a plurality of inspection results for the plurality of work results from the plurality of inspectors as inspection pass or rejection.

Description

A method of improving reliability through selective self-check of a crowdsourcing-based project for generating artificial intelligence learning data {METHOD FOR IMPROVING RELIABILITY BY SELECTIVE SELF CHECK OF WORKER OF CROWDSOURCING BASED PROJECT FOR ARTIFICIAL INTELLIGENCE TRAINING DATA GENERATION}

The present invention relates to a method for improving reliability through selective self-check of a crowdsourcing-based project for generating artificial intelligence learning data.

Recently, more and more companies collect and process large amounts of data on a crowdsourcing basis that engages the general public in some process of corporate activities. In other words, by opening a project and allowing the general public, that is, workers to participate in the project, necessary information is collected through the work results completed by the workers.

Specifically, when one project is opened, multiple tasks are assigned to multiple workers. Each worker performs the assigned task and provides the task result. After that, multiple work results are assigned to multiple inspectors. Each inspector performs an inspection on the assigned work results.

However, there are many cases in which the quality of the work result input from the operator is low due to various reasons such as unscrupulous work performance, unfamiliarity with the guide and mistakes.

Since the inspector must inspect the work results of low quality, the time required for inspection per one work result is longer. In addition, since most of the work results of low quality are rejected, the income that the inspector can obtain for the time it takes to perform the inspection becomes very low.

Accordingly, there is a problem in that the work result input from the operator increases, but the number of inspectors to inspect it decreases, resulting in an inspection bottleneck.

In addition, as the inspection bottleneck occurs, there is a problem that the work progress rate decreases, eventually failing to collect target data from the project or delaying the project schedule.

Unexamined Patent Publication No. 10-2014-0095956, 2014.08.04.

The problem to be solved by the present invention is to provide a method of improving reliability through selective self-check of a crowdsourcing-based project for generating artificial intelligence learning data.

However, the problem to be solved by the present invention is not limited to the problem as described above, and other problems may exist.

A method of improving reliability through selective self-check of a crowdsourcing-based project according to an aspect of the present invention for solving the above-described problems is to allocate a plurality of tasks of a crowdsourcing-based project (hereinafter, a project) to a plurality of workers. A step of requesting to perform a task, receiving a plurality of work results from the plurality of workers, reconfirming and reworking their work results (hereinafter, self-check) to each worker, and each worker Selectively inputting the self-check intention from the self-check, requesting the self-checking worker (hereinafter referred to as the self-check worker) to perform a self-check on the result of the self-check, and performing a self-check from the self-check worker. Receiving input of completed work results, assigning the plurality of work results including the self-check completed work results to a plurality of inspectors to request inspection performance, and the plurality of work results from the plurality of inspectors And receiving a plurality of inspection results as passing or rejecting the inspection.

In some embodiments of the present invention, the step of inquiring each worker for a self-check intention for his or her work result, and selectively receiving the self-check intention from each worker, comprises a predetermined number of Whenever the work result of the unit of is received, the self-check intention for the work result of the unit may be inquired.

In some embodiments of the present invention, the predetermined number of units may be determined according to the difficulty of the project.

In some embodiments of the present invention, the step of receiving the result of completing the self-check from the self-check operator includes receiving the corrected work result by reworking according to the reconfirmation result by the self-check operator, or not being modified. You can receive the results of the unsuccessful work.

In some embodiments of the present invention, the method includes the step of selectively receiving an inspection intention for a job result of completing the self-check or a job result not performing the self-check by the operator from the plurality of inspectors. It may contain more.

In some embodiments of the present invention, the method further comprises calculating an inspection cost for the plurality of inspection results of the plurality of inspectors, wherein the inspection cost for the inspection result input as the inspection pass is calculated. What is to be done may be to calculate a first inspection cost for a result of the self-check completed to be lower than a second inspection cost for a result of the job not performing the self-check.

In some embodiments of the present invention, calculating the inspection cost for the plurality of inspection results of the plurality of inspectors may include calculating the inspection cost for the inspection result input as a rejection, completing the self-check The third inspection cost for one work result and the fourth inspection cost for work results that have not been subjected to the self-check may be equally calculated.

In some embodiments of the present invention, the method further comprises the step of calculating a work cost for the plurality of work results of the plurality of workers, wherein the inspection result is Calculating the work cost may be to calculate a first work cost for a work result that has completed the self-check higher than a second work cost for a work result that has not performed the self-check.

In some embodiments of the present invention, among the plurality of work results, a work result in a self-checking state may be excluded from the subject of the inspection by the inspector.

A computer program according to another aspect of the present invention for solving the above-described problems is combined with a computer as hardware to execute a method of improving reliability through selective self-check of workers of the crowdsourcing-based project, and a computer-readable recording medium Is stored in.

Other specific details of the present invention are included in the detailed description and drawings.

According to the present invention described above, the reliability of the work result can be increased by allowing the worker to re-check and rework (self-check) the work result that the worker has done.

As the reliability of the work result is higher, the time it takes for the inspector to inspect the work result decreases, so the income compared to the time required for inspection increases, motivating the inspector, and accordingly, the number of inspectors participating in the project increases.

As the number of inspectors increases, the inspection bottleneck is eliminated, the work progress rate increases, and accordingly, the amount of data targeted within the project period can be collected.

The effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the following description.

1 is a conceptual diagram of a crowdsourcing service according to an embodiment of the present invention.
2 is a flowchart illustrating a crowdsourcing-based project progress process according to an embodiment of the present invention.
3 is a flowchart of a method for improving reliability through selective self-check of a crowdsourcing-based project worker according to an embodiment of the present invention.
4 is an exemplary diagram for explaining inquiring about a self-check intention to an operator according to an embodiment of the present invention.
5 is an exemplary diagram for explaining that a result of a work in progress of a self-check according to an embodiment of the present invention is excluded from the subject of inspection.
6 is an exemplary diagram for explaining that a result of a self-check completed task is included in a subject to be inspected according to an embodiment of the present invention.
7 is an exemplary view for explaining calculating an inspection cost according to an inspection result according to an embodiment of the present invention.
8 is an exemplary diagram for explaining calculating a work cost according to an inspection result according to an embodiment of the present invention.
9 is an exemplary diagram illustrating a rejection rate of a result of a self-check completed according to an embodiment of the present invention.
10 is a diagram for describing an apparatus for improving reliability through selective self-check of a worker of a crowdsourcing-based project according to an embodiment of the present invention.

Advantages and features of the present invention, and a method of achieving them will become apparent with reference to the embodiments described below in detail together with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but may be implemented in a variety of different forms, only the present embodiments are intended to complete the disclosure of the present invention, It is provided to fully inform the technician of the scope of the present invention, and the present invention is only defined by the scope of the claims.

The terms used in the present specification are for describing exemplary embodiments and are not intended to limit the present invention. In this specification, the singular form also includes the plural form unless specifically stated in the phrase. As used in the specification, “comprises” and/or “comprising” do not exclude the presence or addition of one or more other elements other than the mentioned elements. Throughout the specification, the same reference numerals refer to the same elements, and “and/or” includes each and all combinations of one or more of the mentioned elements. Although "first", "second", and the like are used to describe various elements, it goes without saying that these elements are not limited by these terms. These terms are only used to distinguish one component from another component. Therefore, it goes without saying that the first component mentioned below may be the second component within the technical idea of the present invention.

Unless otherwise defined, all terms (including technical and scientific terms) used in the present specification may be used as meanings that can be commonly understood by those of ordinary skill in the art to which the present invention belongs. In addition, terms defined in a commonly used dictionary are not interpreted ideally or excessively unless explicitly defined specifically.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a conceptual diagram of a crowdsourcing service according to an embodiment of the present invention.

Referring to FIG. 1, a crowdsourcing service is performed by being composed of a client 10, a service provider 20, and a public 30.

The sponsor 10 refers to a company or individual who requests a crowdsourcing-based project (hereinafter, referred to as a project).

The requester 10 requests a project for the purpose of collecting source data or annotating data for generating artificial intelligence learning data. The data generated through the project can be used as learning data for arbitrary machine learning such as supervised learning, unsupervised learning, and reinforcement learning. Collection of source data refers to collecting raw data such as recorded voice collection and photo collection. Data annotation refers to inputting relevant annotation data into source data such as text, photos, and videos. For example, the data annotation may include finding an entity in a given fingerprint or finding a similar sentence, but is not limited thereto. Meanwhile, the types of the above-described projects are only one embodiment, and various projects may be handled in the present invention according to the design of the client.

The service provider 20 refers to a company that provides crowdsourcing services.

When the service provider 20 receives a request for a project for a product or service from the client 10, the service provider 20 allocates a task for the project to the general public 30 and receives the work result from the public 30. Thereafter, the final product extracted based on the work result is provided to the client 10.

At this time, the service provider 20 provides a crowdsourcing service to the client 10 and the public 30 through a crowdsourcing platform (hereinafter, the platform). That is, when the service provider 20 receives a request for a project from the client 10, the service provider 20 opens the project on the platform. Thereafter, when the work result for the open project is received from the public 30, the project is terminated on the platform, and the final product may be extracted and provided to the client 10.

The public 30 refers to the general public participating in the project open on the platform. Here, the public 30 may participate in a project opened on the platform through an application or website provided by the service provider 20.

The public 30 is composed of a worker 32 and an inspector 34.

The worker 32 decides to participate in a specific project among a plurality of projects open on the platform. Thereafter, the operator 32 performs an operation such as collection of source data or data annotation, and transmits it to the platform.

The inspector 34 decides to participate in a specific project among a plurality of projects open on the platform. Thereafter, the inspector 34 inspects the result of the work performed by the operator 32. As a result of performing the inspection, the inspector 34 may perform inspection pass processing or rejection processing, and input a rejection reason during rejection processing. In the case of passing the inspection, it is not necessary to rework and re-examination due to this, so passing inspection has the same meaning as completion of inspection.

2 is a flowchart illustrating a crowdsourcing-based project progress process according to an embodiment of the present invention.

First, the requester 10 requests one or more projects to the service provider 20 (S11).

Thereafter, the service provider 20 opens the requested project on the platform (S12). At this time, the service provider 20 may determine the grade in consideration of the difficulty of the project before opening the project. That is, depending on the difficulty level, it is possible to determine which level or higher public 30 to expose the project. Accordingly, it is possible to increase the reliability of the work result of the project.

Thereafter, the service provider 20 requests the work by assigning the work to the workers 32 of the corresponding level or higher according to the level of the project (S13).

Thereafter, the worker 32 performs the assigned task (S14). In this case, the worker 32 may input the reason for the inability to work without performing the work for a work in which the work itself is impossible for some reason.

Thereafter, the service provider 20 receives the work result from the worker 32 (S15), and assigns the work result to the inspector 34 to request the inspection (S16).

Thereafter, the inspector 34 performs the assigned inspection (S17). At this time, the inspector 34 decides to pass the inspection when it is determined that the work has been properly performed, and if it is determined that the inspection work is wrong, the inspector 34 rejects it. During rejection processing, the inspector 34 inputs the rejection reason for what reason the job was judged to be wrong.

Thereafter, the service provider 20 receives the inspection result from the inspector 34 (S18).

If the inspection result passes the inspection, the service provider 20 uses the work result as valid data, and based on this, extracts the final product at the end of the project.

When the inspection result is rejection processing, the service provider 20 may internally perform the inspection again, or assign the work to the worker 32 again to perform the rework. When reworking, re-examination by an inspector is required.

Thereafter, when the project period ends or sufficient valid data is secured, the service provider 20 terminates the project (S19), calculates a final result based on the secured valid data, and provides it to the client 10 ( S20).

At this time, before the end of the project, the service provider 20 evaluates the performance results of the worker 32 and the inspector 34, calculates the work cost and the inspection cost according to the evaluation, to the worker 32 and the inspector 34. give.

In FIGS. 1 and 2, it is simply expressed as a requester 10, a service provider 20, an operator 32, and an inspector 34, but these are smartphones, tablets, PDAs, laptops, desktops, operated by each participant. It means a computer device such as a server or a telecommunication device.

In crowdsourcing-based projects, the quality of the work results submitted by the workers 32 greatly affects the overall speed of the project. The lower the quality of the work result of the worker 32, the longer the time required for the inspection by the inspector 34, and accordingly, the speed of the project is slowed down, and as a result, the schedule of the project is delayed.

There are various reasons for the deterioration of the quality of the work result, among them, the quality of the work result is often deteriorated due to the mistake of the worker 32.

Hereinafter, referring to FIGS. 3 to 9, each worker 32 selectively performs a self-check before inspecting the work result in order to prevent the quality of the work result from deteriorating due to the error of the worker 32. By doing so, we will explain how to increase the reliability of the work results.

3 is a flowchart of a method for improving reliability through selective self-check of a crowdsourcing-based project worker according to an embodiment of the present invention. 4 is an exemplary diagram for explaining inquiring about a self-check intention to an operator according to an embodiment of the present invention. 5 is an exemplary diagram for explaining that a result of a work in progress of a self-check according to an embodiment of the present invention is excluded from the subject of inspection. 6 is an exemplary diagram for explaining that a result of a self-check completed task is included in a subject to be inspected according to an embodiment of the present invention.

Meanwhile, the steps shown in FIG. 3 may be understood as being performed by a platform server (hereinafter, referred to as a server) operated by the service provider 20, but are not limited thereto.

In addition, a plurality of workers 32 or a plurality of inspectors 34 perform work using a predetermined terminal device. The terminal device of the operator 32 or the inspector 34 may be a computer device or a telecommunication device such as a smartphone, tablet, PDA, laptop, desktop, etc., but is not limited thereto.

First, referring to FIG. 3, the server assigns a plurality of tasks of a crowdsourcing-based project (hereinafter, a project) to a plurality of workers 32 to request the task execution (S110), and a plurality of tasks from the plurality of workers 32 The operation result of is received (S120).

In one embodiment, the server may request to perform a task by assigning a predetermined number of tasks to each worker 32 participating in the project at one time. Each worker 32 sequentially performs an assigned predetermined number of tasks and inputs a task result. The server sequentially stores job results submitted from each of the plurality of workers 32 according to the input order.

In another embodiment, the server may request to perform the task by assigning one task to each worker 32 participating in the project. That is, when each worker 32 performs one assigned task and inputs a task result, another task is assigned from the server. The server sequentially stores job results submitted from each of the plurality of workers 32 according to the input order.

4, when the server assigns tasks to workers A, B, and C and requests to perform tasks, the server is sequentially based on the order in which task results are input from each of workers A, B and C, not the order in which tasks are assigned. Can be saved as

Referring to FIG. 3 again, the server inquires each worker 32 for re-checking and reworking (hereinafter, self-check) intention for his or her work results, and selectively requests a self-check intention from each worker 32. The input is received (S130).

Specifically, each time the server receives a predetermined number of work results from each of the workers 32, it inquires of each worker 32 for a self-checking intention for the work result of the unit.

When the predetermined number is 5, the server inquires each worker 32 for a self-check intention at each point in time when the work results input from each worker 32 are accumulated in five units. That is, each time the server accumulates five job results input for each worker 32 (for example, when the 5th job result is input, when the 10th job result is input, etc.), each worker ( 32) Ask your self-check doctor.

Referring to FIG. 4, the server receives work results from workers A, B, and C and sequentially stores them, and to each of workers A, B, and C at the time when 5 work results of workers A, B and C are accumulated. You can ask for a self-check doctor. In FIG. 4, it is shown that each worker 32 inquires about the self-check intention at different times. However, when the times when a predetermined number of units are accumulated are the same, the server asks each worker 32 at the same time. You can ask a doctor to check.

At this time, the predetermined number of units is determined according to the difficulty of the project.

The higher the difficulty of the project, the smaller the number of units can be determined by the server. The higher the difficulty of the project, the higher the probability that the worker 32 makes a mistake when performing the task, so that the self-check can be performed at relatively short intervals.

On the other hand, as the difficulty of the project is lower, the server may determine the number of units as a larger number. The lower the difficulty of the project, the lower the probability that the worker 32 makes a mistake when performing the task, so that the self-check can be performed at relatively wide intervals.

Referring back to FIG. 3, the server requests a self-checking worker (hereinafter referred to as a self-check worker) with a self-check intention to perform a self-check on the result of the self-check (S140).

When the worker 32 accepts the self-check inquiry, the server requests the self-check performance while providing the job results in a predetermined number of units submitted by the corresponding worker 32, that is, the self-check worker. .

In this case, the status of the work result of the self-check operator in the unit is classified as'self-check in progress', and is excluded from the subject of inspection by the inspector.

For example, if the predetermined number is 5, the server excludes 5 work results of self-check workers from the subject of inspection, and reconfirms and rework (self-check) of the 5 work results to self-check workers. request.

On the other hand, when the worker 32 rejects the self-check inquiry, the server immediately inspects the work results of a predetermined number of units submitted by the worker 32, that is, a non-self-check worker who does not intend to self-check. Include in the target of execution.

At this time, the state of the work result of the unit of the non-self-check operator is classified as'self-check not in progress', and is immediately included in the subject of inspection by the inspector 34.

For example, if the predetermined number is 5, the server immediately includes 5 work results of non-self-check workers into the subject to be inspected, and sequentially allocates the 5 work results to the inspector 34.

As described above, the server determines whether to immediately include or exclude the work results of a predetermined number of units of each worker 32 in the subject to be inspected according to the self-check selection of the worker 32.

Referring to FIG. 5, when the server receives an acceptance of the self-check intention inquiry from the worker A, the status of the work results (i.e., five work results) of a predetermined number of units of the worker A is'self-check in progress'. It is classified as, and the results of 5 work are excluded from the subject of inspection.

On the other hand, when the server receives rejection of the self-check intention inquiry from workers B and C, the work results of a predetermined number of units of worker B (that is, five work results) and a predetermined number of units of worker C The status of the work results (that is, 5 work results) is classified as'self-check not in progress', and each of the 5 work results is immediately included in the subject to be inspected.

At this time, the server does not determine whether to include the subject of inspection for each work result before the work results of the worker 32 are accumulated by the unit. That is, if the predetermined number is 5, when the work results of the worker 32 are accumulated to less than 5, it is not determined whether to include the inspection target for each work result, but 5 work results have been accumulated. When the operator 32 selects the self-check, it is determined at once whether to include the subject to perform the inspection of the five work results.

Referring back to FIG. 3, the server receives a job result of completing the self-check from the self-check worker (S150).

Specifically, the server receives a reworked and corrected work result according to the reconfirmation result by the self-check operator, or receives an unmodified work result.

That is, the self-check operator reconfirms his or her work result, and if it is determined that there is a problem, the work result is correctly reworked to input the corrected work result, and if it is determined that there is no problem, the work result is inputted as it is.

When the self-check of the work results of a predetermined number of units is completed, the server classifies the status of the work results of the unit by changing the status of the work results from'self-check in progress' to'self-check complete'.

Referring to FIG. 6, a result of a task whose status is changed to'self-check completed' is included in the subject to be inspected.

At this time, the server sequentially stores the work result of completing the self-check of inspector A. That is, even if the work result before the self-check by inspector A is input before the work results of inspectors B and C, the work result after self-check is input later than the work result of workers B and C who did not self-check. The server sequentially places the work results of inspector A's self-check in the back order.

Referring back to FIG. 3, the server requests the inspection by assigning a plurality of job results including the job result for completing the self-check to a plurality of inspectors (S160).

Before allocating a plurality of work results to a plurality of inspectors 34, the server selectively selects from the plurality of inspectors 34 to the work results that have completed self-check by the operator 32 or the work results that have not been self-checked. Receive an input of your intention for inspection.

Depending on the inspection intention entered for each inspector 34, the server may allocate only the task results that have completed self-check to each inspector 34, or allocate only the task results that have not been self-checked, or Task results that have been checked and tasks that have not been self-checked can be assigned together.

Subsequently, the server receives a plurality of inspection results for the plurality of work results from the plurality of inspectors 34 as passing or rejecting the inspection (S170).

That is, each inspector 34 performs inspection on the task result of completing the self-check assigned according to the user's choice or the task result without self-check, but if there is no problem with the task result, it is entered as the inspection pass. , If there is a problem, input the inspection result as rejection.

Hereinafter, with reference to FIGS. 7 and 8, a description will be given of calculating the inspection cost and the work cost according to the inspection result of the self-checking work result and the self-checking work result.

7 is an exemplary view for explaining calculating an inspection cost according to an inspection result according to an embodiment of the present invention. 8 is an exemplary diagram for explaining calculating a work cost according to an inspection result according to an embodiment of the present invention.

The server calculates the inspection cost for the plurality of inspection results of the plurality of inspectors 34.

When the inspector 34 inputs the inspection result for the self-check completed work result as the inspection pass, the server calculates a first inspection cost lower than the inspection unit price for the self-check completed work result to the inspector 34 give.

When the inspector 34 inputs the inspection result for the result of the work that has not been self-checked as the inspection pass, the server calculates a second inspection cost corresponding to the inspection unit price for the work result that has not been self-checked, and the inspector ( 34).

On the other hand, if the inspector 34 inputs the result of the self-check completed or the result of the non-self-check as a rejection, the server is responsible for the third inspection cost and self-check for the result of the self-check completed. The fourth inspection cost for the result of the work that has not been checked is equally calculated and paid to each inspector 34.

In addition, the server calculates a work cost for a plurality of work results of the worker 32.

When the inspector 34 inputs the inspection result for the self-check completed work result as the inspection pass, the server calculates the first work cost higher than the work unit price for the self-check completed work result to the self-check operator. give.

When the inspector 34 inputs the inspection result for the result of the work that has not been self-check as the inspection pass, the server calculates the second work cost corresponding to the work unit cost for the work result that has not been self-check. -Pay to self-check workers.

On the other hand, when the inspector 34 inputs the result of the self-check completed or the result of the non-self-check as a rejection, the server sends the self-check operator and the non-self-check operator to each of the corresponding job results. No work costs are paid for.

As described above, the work result of completing the self-check is rechecked and reworked by the self-check operator. Therefore, the server acknowledges that the self-check operator has performed the inspection on his or her work result, and the inspector 34 judges that it has inspected the work result that is relatively not difficult, and inspects the work result that has completed the self-check. When the result is the inspection pass, the operator 32 for the case is paid a work cost higher than the work unit cost, and the inspector 34 is paid a work cost lower than the inspection unit price.

Referring to FIG. 7, the inspector inspected the work results of not performing 5 self-checks and the work results of completing 5 self-checks. In this case, it is assumed that the inspection unit price is 600 won.

If the inspection result for the result of the work that did not proceed with the self-check passed the inspection, the inspector 34 was paid an inspection fee, 600 won, corresponding to the inspection unit price.

On the other hand, if the inspection result for the self-check completed work result passed the inspection, the inspector 34 was paid an inspection cost, 300 won, which is lower than the inspection unit price.

On the other hand, if the inspection result is rejected, the inspector 34 did not receive the inspection cost for all the work results regardless of whether or not the self-check.

According to the selection of the inspection intention, the inspector can obtain a relatively high income by inspecting only cases that have not undergone self-check, and can perform relatively large inspections by inspecting only cases that have completed the self-check. In addition, by inspecting the cases that have not been self-checked and those that have completed the self-checking together, it is possible to obtain an opportunity to perform more inspections while obtaining a relatively high income.

Referring to FIG. 8, the inspector inspected the work results of not performing 5 self-checks and the work results of completing 5 self-checks. In this case, it is assumed that the unit cost of work is 500 won.

If the result of the inspection for the work result that did not proceed with the self-check passed the inspection, the worker 32 was paid a work cost of 500 won corresponding to the work unit cost.

On the other hand, when the inspection result of the self-check completed work result passed the inspection, the worker 32 was paid a work cost, 800 won, which is higher than the work unit price.

On the other hand, when the inspection result is rejected, the worker 32 has not been paid the work cost for all work results regardless of whether the self-check or not.

According to the self-check selection, the worker can obtain a relatively high income for the case of completing the self-check. For this reason, by deliberately selecting a self-check and performing a task, there is a high possibility that fraudulent workers who try to earn more income will occur.

Accordingly, by selecting a fraudulent worker for the purpose of money among self-check workers and giving a penalty, it is possible to prevent a situation in which a higher inspection cost is paid to the fraudulent worker.

Hereinafter, referring to FIG. 9, a description will be given of selecting a self-check worker for the purpose of money as an illegal worker and granting a penalty.

9 is an exemplary diagram illustrating a rejection rate of a result of a self-check completed according to an embodiment of the present invention.

Based on the inspection result of each self-check worker's self-check completed work result, the server selects the self-check worker as an illegal worker if the rejection rate of the self-check completed work result of a specific self-check worker is more than the standard percentage. .

Referring to FIG. 9, assuming that the predetermined number is 5, it can be seen that the self-check operator has completed two self-checks. At the time of the first self-check, all five work results are inspected, and the rejection rate of the work results completed by the corresponding worker is calculated as 0%. Thereafter, at the second self-check, two of the five work results are rejected, and the rejection rate of the work results completed by the corresponding worker is calculated as 20%.

At this time, if the reference percentage is 10%, since the rejection rate of the self-check completed work result is calculated as 20%, the worker is selected as an illegal worker.

The server penalizes self-check workers who are selected as illegal workers. For example, the server classifies workers who are selected as fraudulent workers as non-self-checking workers, and does not proceed with inquiries about self-check intention for job results submitted by fraudulent workers, so that fraudulent workers can perform self-check. You can do it.

In the above, for convenience, it has been described that the worker 32 continuously performs self-check on each of the five work results, but the worker 32 performs self-check on the previous five work results according to the selection, and then Self-check may not be performed for the results of five tasks. The server may calculate the rejection rate of the self-check completed work result as the work results completed by the worker 32 are accumulated.

On the other hand, the server should calculate the rejection rate of the self-check completed job result and the rejection rate of the entire job result for each worker 32 separately.

When the worker 32 alternately selects acceptance and rejection of the self-check inquiry, the server rejects the rejection rate that reflects only the job result of completing the self-check submitted by the worker 32 (rejection of the job result of completing the self-check Rate) and the rejection rate (rejection rate of the entire job result) that reflects both the result of the work completed with the self-check and the result of the work without the self-check are calculated.

Accordingly, when the rejection rate of the job result completed by the corresponding worker 32 and the rejection rate of the entire job result is equal to or greater than the reference percentage, each of the corresponding worker 32 gives a different penalty.

As described above, if the rejection rate of the job result of completing the self-check of the worker 32 exceeds the reference percentage, the server classifies the worker 32 as an unself-checkable worker, and the corresponding worker 32 Make it impossible to perform self-check on work results. That is, the worker 32 is limited only to self-check, and can continue to perform the task.

However, if the rejection rate of the entire work result of the worker 32 exceeds the reference percentage, the server determines the worker 32 as a defective worker and gives the worker 32 a job blocking penalty. That is, the worker 32 is unable to perform the task itself in the project or for a predetermined period.

Meanwhile, in the above description, steps S11 to S170 may be further divided into additional steps or may be combined into fewer steps according to an embodiment of the present invention. In addition, some steps may be omitted as necessary, and the order between steps may be changed. In addition, even if other contents are omitted, the contents of FIG. 10 to be described later can also be applied to a method of improving reliability through selective self-check of a worker of a crowdsourcing-based project of FIGS. 1 to 9.

Hereinafter, an apparatus 200 for improving reliability through selective self-check of a crowdsourcing-based project operator according to an embodiment of the present invention will be described with reference to FIG. 10.

10 is a diagram for describing an apparatus for improving reliability through selective self-check of a worker of a crowdsourcing-based project according to an embodiment of the present invention.

Referring to FIG. 10, a device 200 (hereinafter, a device for improving reliability) for improving reliability through a selective self-check of an operator of a crowdsourcing-based project according to the present invention includes a communication module 210, a memory 220, and a processor 230. ).

The communication module 210 transmits a crowdsourcing-based work for one project to a plurality of workers 32 to request a task to be performed, and receives a work result from the plurality of workers 32. At this time, the communication module 210 receives the result of the self-check completed from the self-check operator.

The work results received from the plurality of workers 32 including the self-check workers (self-checked work results and self-checked work results) are transmitted to the plurality of inspectors 34 to request inspection, and a plurality of The inspection result is received from the inspector 34 of.

The memory 220 stores a program for improving the reliability of a work result by performing a self-check based on the data received from the communication module 210 according to an operator's self-check intention.

The processor 230 executes a program stored in the memory 220. As the processor 230 executes the program stored in the memory 220, it inquires each worker 32 for a self-check intention for his or her work result, and asks each inspector 34 for the self-checking result or Inquiry about the inspection intention for the work result that has not been self-check, and based on the intention of the operator 32 and the inspector 34, the inspector for the self-check completed work result or the work result without self-check ( Depending on the inspection result in 34), the work cost and inspection cost can be calculated differently.

The reliability improvement apparatus 200 described with reference to FIG. 10 may be provided as a component of the above-described server.

The method for improving reliability through selective self-check of a worker for a crowdsourcing-based project according to an embodiment of the present invention described above is implemented as a program (or application) to be executed in combination with a computer, which is hardware, and stored in a medium. Can be.

The above-described program includes C, C++, JAVA, Ruby, which can be read by a processor (CPU) of the computer through the device interface of the computer, in order for the computer to read the program and execute the methods implemented as a program. It may include a code (Code) coded in a computer language such as machine language. Such code may include a functional code related to a function defining necessary functions for executing the methods, and a control code related to an execution procedure necessary for the processor of the computer to execute the functions according to a predetermined procedure. can do. In addition, such code may further include additional information required for the processor of the computer to execute the functions or code related to a memory reference to which location (address address) of the internal or external memory of the computer should be referenced. have. In addition, when the processor of the computer needs to communicate with any other computer or server in the remote in order to execute the functions, the code uses the communication module of the computer to determine how It may further include a communication-related code for whether to communicate or what information or media to transmit and receive during communication.

The stored medium is not a medium that stores data for a short moment, such as a register, cache, memory, etc., but a medium that stores data semi-permanently and can be read by a device. Specifically, examples of the storage medium include, but are not limited to, ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, and the like. That is, the program may be stored in various recording media on various servers to which the computer can access, or on various recording media on the user's computer. In addition, the medium may be distributed over a computer system connected through a network, and computer-readable codes may be stored in a distributed manner.

The above description of the present invention is for illustrative purposes only, and those of ordinary skill in the art to which the present invention pertains will be able to understand that it can be easily modified into other specific forms without changing the technical spirit or essential features of the present invention. will be. Therefore, it should be understood that the embodiments described above are illustrative in all respects and not limiting. For example, each component described as a single type may be implemented in a distributed manner, and similarly, components described as being distributed may also be implemented in a combined form.

The scope of the present invention is indicated by the claims to be described later rather than the detailed description, and all changes or modified forms derived from the meaning and scope of the claims and their equivalent concepts should be interpreted as being included in the scope of the present invention. do.

10: Client
20: service provider
30: public
32: worker
34: inspector
200: reliability improvement device
210: communication module
220: memory
230: processor

Claims (10)

As a method performed by a computer,
Assigning a plurality of tasks of a crowdsourcing-based project (hereinafter, referred to as a project) to a plurality of workers and requesting the task to be performed;
Receiving a plurality of work results from the plurality of workers;
Inquiring each worker for reconfirmation of his/her work result and rework (hereinafter, self-check) intention according to the reconfirmation result, and receiving a selective self-check intention by each worker from each worker;
Requesting a self-checking worker (hereinafter referred to as a self-check worker) to perform a self-check on the result of the self-check;
Receiving a job result of completing the self-check from the self-check operator;
Allocating the plurality of work results, including the work results of completing the self-check, to a plurality of inspectors and requesting to perform inspection; And
And receiving a plurality of inspection results for the plurality of work results from the plurality of inspectors as passing or rejecting the inspection,
Further comprising the step of receiving, from the plurality of inspectors, an intention to selectively inspect a task result of completing the self-check by the worker or a result of a task not performing the self-check,
Further comprising the step of calculating an inspection cost for the plurality of inspection results of the plurality of inspectors,
Calculating the inspection cost for the inspection result inputted through inspection is to make the first inspection cost for the work result completed with the self-check lower than the second inspection cost for the work result not performing the self-check. Calculating,
A method of improving reliability through selective self-check by workers of crowdsourced projects. The method of claim 1,
Inquiring each worker for a self-check intention for his or her work result, and receiving a selective self-check intention by each worker from each worker,
Inquiring the self-check intention for the work result of the unit whenever a predetermined number of work results are input from each worker,
A method of improving reliability through selective self-check of workers in crowdsourced projects. The method of claim 2,
The predetermined number of units is determined according to the difficulty of the project,
A method of improving reliability through selective self-check of workers in crowdsourced projects. The method of claim 1,
The step of receiving the result of the self-check completion from the self-check operator,
Depending on the reconfirmation result by the self-check operator, reworked and corrected work results are input, or unmodified work results are input,
A method of improving reliability through selective self-check of workers in crowdsourced projects. delete delete The method of claim 1,
Calculating an inspection cost for the plurality of inspection results of the plurality of inspectors,
Calculating the inspection cost for the inspection result entered as a rejection equals the third inspection cost for the result of the work completed with the self-check and the fourth inspection cost for the result of the work not performing the self-check. Calculating,
A method of improving reliability through selective self-check of workers in crowdsourced projects. The method of claim 1,
Further comprising the step of calculating a work cost for the plurality of work results of the plurality of workers,
Calculating the work cost for the work result inputted as the inspection result passes through the inspection is to calculate the first work cost for the work result for which the self-check has been completed, and the second work cost for the work result not performing the self-check That yields higher than the cost of work,
A method of improving reliability through selective self-check of workers in crowdsourced projects. The method of claim 1,
Excluding the work results in the self-checking state among the plurality of work results from the subject of the inspection performed by the inspector,
A method of improving reliability through selective self-check of workers in crowdsourced projects. In order to perform the method of improving reliability through selective self-check of the crowdsourcing-based project of any one of claims 1 to 4 and 7 to 9 in combination with a computer, a computer-readable recording medium is provided. Stored computer programs.

Images