KR102164837B1 - Method for cost-effectively managing inspection quality of crowdsourcing based projects for artificial intelligence training data generation - Google Patents

Abstract

Provided is a method for cost-effective inspection quality management of a project based on crowdsourcing. The method comprises the steps of: assigning a plurality of tasks of a project to a plurality of workers and requesting the tasks to be performed; receiving a plurality of task results from the plurality of workers; allocating the plurality of task results to a plurality of inspectors and requesting to perform inspection; receiving an inspection pass or rejection as a plurality of inspection results for the plurality of work results from the plurality of inspectors; and managing the inspection quality of each inspector using the inspection results.

Description

A cost-effective inspection quality management method for crowdsourcing-based projects for generating artificial intelligence learning data {METHOD FOR COST-EFFECTIVELY MANAGING INSPECTION QUALITY OF CROWDSOURCING BASED PROJECTS FOR ARTIFICIAL INTELLIGENCE TRAINING DATA GENERATION}

The present invention relates to a cost-effective quality control method for crowdsourcing-based projects for generating artificial intelligence learning data.

Recently, more and more companies collect and process large amounts of data on a crowdsourcing basis that engages the general public in some process of corporate activities. In other words, by opening a project and allowing the general public, that is, workers to participate in the project, necessary information is collected through the work results completed by the workers.

At this time, in order to collect more reliable information, the company assigns the work result completed by the worker to the inspector to perform the inspection work.

Specifically, when one project is opened, a plurality of tasks are assigned to each of the plurality of workers. Each worker performs a plurality of tasks assigned to it, and provides the task results. Thereafter, a plurality of inspection tasks are assigned to each of the plurality of inspectors, and each inspector performs a plurality of inspection tasks assigned to them.

On the other hand, the quality of the project is determined by the work result of the worker, but it is also determined by the inspection quality of the inspector. If the inspector enters the work result to be rejected as the inspection pass, the work result is not re-examined, and the quality of the project decreases as such cases increase.

In particular, the inspection fee paid to the inspector is paid only when the inspector enters the inspection pass, not the rejection, but an unfaithful or unfaithful inspector can use it to enter the inspection pass recklessly. There is a need for a method of detecting.

Unexamined Patent Publication No. 10-2014-0095956, 2014.08.04.

The problem to be solved by the present invention is to check the rejection rate of inspectors at regular intervals and allocate verification work results to the inspectors based on this, thereby cost-effective inspection of crowdsourcing-based projects that can determine whether the inspector is likely to be illegally inspected It is to provide a quality control method.

However, the problem to be solved by the present invention is not limited to the problem as described above, and other problems may exist.

In order to solve the above problems, the cost-effective quality control method for a crowdsourcing-based project according to an aspect of the present invention allocates a plurality of tasks of a crowdsourcing-based project (hereinafter, referred to as a project) to a plurality of workers and requests to perform tasks The step of, receiving a plurality of work results from the plurality of workers, assigning the plurality of work results to a plurality of inspectors and requesting to perform inspection, a plurality of the plurality of work results from the plurality of inspectors And receiving an inspection pass or rejection as an inspection result, and managing the inspection quality of each inspector using the inspection result. At this time, the step of assigning the plurality of work results to a plurality of inspectors and requesting to perform the inspection includes: m (where m is a natural number of 2 or more) actual work results are assigned to the inspector and periodically whenever the inspection is requested , Based on the rejection rate of the m number of actual work results of the inspector, n (where n is a natural number of 1 or more) verification task results are assigned to the inspector and requesting to perform the inspection, wherein The step of managing the inspection quality of each inspector using the inspection results includes the step of managing inspection quality of the m actual work results of the inspector using the inspection results for the n verification work results. do.

In some embodiments of the present invention, the step of assigning the m actual work results to the inspector and requesting to perform the inspection by periodically assigning n verification work results to the inspector whenever the inspection is requested to be performed, the inspector If the rejection rate for the m actual work results of is less than the predetermined standard rejection rate, the n verification work results are assigned to the inspector for verification of the inspection results for the m actual work results to perform the inspection. The requesting step and, if the rejection rate for the m actual work results of the inspector is more than the reference rejection rate, verification of the inspection results for the m actual work results is omitted, and the n verification work results are displayed. It may include the step of not requesting to perform the inspection by assigning it to the inspector.

In some embodiments of the present invention, the predetermined reference rejection rate may be an average rejection rate for the m actual work results of all inspectors of the project.

In some embodiments of the present invention, when the inspection results for the m actual work results are rejected, the rejected jobs are reassigned to a corresponding worker to request rework, and the inspection fee is not paid to the inspector, When the inspection result for the n verification work results is rejected, the rejected job may be reassigned to a corresponding worker and the inspection fee may be paid to the inspector without requesting rework.

In some embodiments of the present invention, the result of the verification operation may be a result of a job that has been rejected because the inspection has already been performed by another inspector.

In some embodiments of the present invention, the step of managing the inspection quality for the m actual work results of the inspector using the inspection results for the n verification work results, comprises: If the inspection result is not rejected, the step of determining the inspection quality of the m actual work results of the inspector as inappropriate.

In some embodiments of the present invention, when the number of nonconformities reaches a predetermined reference number based on a result of determining the quality of inspection for the m actual work results of the inspector, determining the inspector as a defect inspector. It may contain more.

In some embodiments of the present invention, a penalty for restricting the defect inspector from participating in the inspection of the project for a predetermined limited period, or recovering the inspection cost for the m actual work results paid to the defect inspector It may further include the step of giving.

In some embodiments of the present invention, if the inspector determines the inspection quality of the m actual work results as inadequate, the step of reassigning the m actual work results of the inspector to another inspector to request re-review. It may contain more.

A computer program according to another aspect of the present invention for solving the above-described problems is combined with a computer as hardware to execute a cost-effective inspection quality control method of the crowdsourcing-based project, and is stored in a computer-readable recording medium.

Other specific details of the present invention are included in the detailed description and drawings.

According to the present invention described above, the rejection rate of the inspector is checked at regular intervals, and when the checked rejection rate is lower than the standard rejection rate, the verification work result is assigned to the inspector to determine the possibility of fraudulent inspection in the corresponding period. have.

In addition, if there are multiple times that the number of times of fraudulent inspection as described above is determined, a penalty is given by designating the inspector as a defect inspector, thereby solving the problem of delays in the project schedule due to the deterioration of the project quality and additional inspections caused by the defect inspector can do.

The effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the following description.

1 is a conceptual diagram of a crowdsourcing service according to an embodiment of the present invention.
2 is a flowchart illustrating a crowdsourcing-based project progress process according to an embodiment of the present invention.
3 is a flowchart of a cost-effective inspection quality management method for a crowdsourcing-based project according to an embodiment of the present invention.
4 is a flowchart illustrating a process of requesting to perform an inspection by assigning a plurality of work results to a plurality of inspectors.
5 is a diagram illustrating an example in which actual work results and verification work results are allocated.
6 is a diagram illustrating an example in which assignment of a verification task result is omitted.
7 is a diagram illustrating an example in which a penalty is given to a defect inspector.
8 is a view for explaining an inspection quality management apparatus according to an embodiment of the present invention.

Advantages and features of the present invention, and a method of achieving them will become apparent with reference to the embodiments described below in detail together with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but may be implemented in a variety of different forms, only the present embodiments are intended to complete the disclosure of the present invention, It is provided to fully inform the technician of the scope of the present invention, and the present invention is only defined by the scope of the claims.

The terms used in the present specification are for describing exemplary embodiments and are not intended to limit the present invention. In this specification, the singular form also includes the plural form unless specifically stated in the phrase. As used in the specification, “comprises” and/or “comprising” do not exclude the presence or addition of one or more other elements other than the mentioned elements. Throughout the specification, the same reference numerals refer to the same elements, and “and/or” includes each and all combinations of one or more of the mentioned elements. Although "first", "second", and the like are used to describe various elements, it goes without saying that these elements are not limited by these terms. These terms are only used to distinguish one component from another component. Therefore, it goes without saying that the first component mentioned below may be the second component within the technical idea of the present invention.

Unless otherwise defined, all terms (including technical and scientific terms) used in the present specification may be used as meanings that can be commonly understood by those of ordinary skill in the art to which the present invention belongs. In addition, terms defined in a commonly used dictionary are not interpreted ideally or excessively unless explicitly defined specifically.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a conceptual diagram of a crowdsourcing service according to an embodiment of the present invention.

Referring to FIG. 1, a crowdsourcing service is performed by being composed of a client 10, a service provider 20, and a public 30.

The sponsor 10 refers to a company or individual who requests a crowdsourcing-based project (hereinafter, referred to as a project).

The requester 10 requests a project for the purpose of collecting source data or annotating data for generating artificial intelligence learning data. The data generated through the project can be used as learning data for arbitrary machine learning such as supervised learning, unsupervised learning, and reinforcement learning. Collection of source data refers to collecting raw data such as recorded voice collection and photo collection. Data annotation refers to inputting relevant annotation data into source data such as text, photos, and videos. For example, the data annotation may include finding an entity in a given fingerprint or finding a similar sentence, but is not limited thereto. Meanwhile, the types of the above-described projects are only one embodiment, and various projects may be handled in the present invention according to the design of the client.

The service provider 20 refers to a company that provides crowdsourcing services.

When the service provider 20 receives a request for a project for a product or service from the client 10, the service provider 20 allocates a task for the project to the general public 30 and receives the work result from the public 30. Thereafter, the final product extracted based on the work result is provided to the client 10.

At this time, the service provider 20 provides a crowdsourcing service to the client 10 and the public 30 through a crowdsourcing platform (hereinafter, the platform). That is, when the service provider 20 receives a request for a project from the client 10, the service provider 20 opens the project on the platform. Thereafter, when the work result for the open project is received from the public 30, the project is terminated on the platform, and the final product may be extracted and provided to the client 10.

The public 30 refers to the general public participating in the project open on the platform. Here, the public 30 may participate in a project opened on the platform through an application or website provided by the service provider 20.

The public 30 is composed of a worker 32 and an inspector 34.

The worker 32 decides to participate in a specific project among a plurality of projects open on the platform. Thereafter, the operator 32 performs an operation such as collection of source data or data annotation, and transmits it to the platform.

The inspector 34 decides to participate in a specific project among a plurality of projects open on the platform. Thereafter, the inspector 34 inspects the result of the work performed by the operator 32. As a result of performing the inspection, the inspector 34 may perform inspection pass processing or rejection processing, and input a rejection reason during rejection processing. In the case of passing the inspection, it is not necessary to rework and re-examination due to this, so passing inspection has the same meaning as completion of inspection.

2 is a flowchart illustrating a crowdsourcing-based project progress process according to an embodiment of the present invention.

First, the requester 10 requests one or more projects to the service provider 20 (S11).

Thereafter, the service provider 20 opens the requested project on the platform (S12). At this time, the service provider 20 may determine the grade in consideration of the difficulty of the project before opening the project. That is, depending on the difficulty level, it is possible to determine which level or higher public 30 to expose the project. Accordingly, it is possible to increase the reliability of the work result of the project.

Thereafter, the service provider 20 requests the work by assigning the work to the workers 32 of the corresponding level or higher according to the level of the project (S13).

Thereafter, the worker 32 performs the assigned task (S14). In this case, the worker 32 may input the reason for the inability to work without performing the work for a work in which the work itself is impossible for some reason.

Thereafter, the service provider 20 receives the work result from the worker 32 (S15), and assigns the work result to the inspector 34 to request the inspection (S16).

Similarly, according to an embodiment of the present invention, only a suitable project may be exposed to the inspector 34 out of the total projects being performed according to the level set according to the difficulty of the project or the qualification requirements of the inspector 34.

Thereafter, the inspector 34 performs the assigned inspection (S17). At this time, if the inspector 34 determines that the work has been properly performed, the inspector 34 decides to complete the inspection, and if it determines that the inspection operation is wrong, the inspector 34 rejects it. During rejection processing, the inspector 34 inputs the rejection reason for what reason the job was judged to be wrong.

Thereafter, the service provider 20 receives the inspection result from the inspector 34 (S18).

When the inspection result is the inspection completion, the service provider 20 uses the work result as valid data, and based on this, extracts the final product at the end of the project.

When the inspection result is rejection processing, the service provider 20 may internally perform the inspection again, or assign the work to the worker 32 again to perform the rework. When reworking, re-examination by an inspector is required.

Thereafter, when the project period ends or sufficient valid data is secured, the service provider 20 terminates the project (S19), calculates a final result based on the secured valid data, and provides it to the client 10 ( S20).

At this time, before the end of the project, the service provider 20 evaluates the performance results of the worker 32 and the inspector 34, calculates the work cost and the inspection cost according to the evaluation, to the worker 32 and the inspector 34. give.

In FIGS. 1 and 2, it is simply expressed as a requester 10, a service provider 20, an operator 32, and an inspector 34, but these are smartphones, tablets, PDAs, laptops, desktops, operated by each participant. It means a computer device such as a server or a telecommunication device.

Hereinafter, a cost-effective inspection quality management method for a crowdsourcing-based project according to an embodiment of the present invention will be described with reference to FIGS. 3 to 7.

3 is a flowchart of a cost-effective inspection quality management method for a crowdsourcing-based project according to an embodiment of the present invention. 4 is a flow chart illustrating a process of requesting to perform the inspection by assigning a plurality of work results to a plurality of inspectors 34. 5 is a diagram illustrating an example in which actual work results and verification work results are allocated. 6 is a diagram illustrating an example in which assignment of a verification task result is omitted. 7 is a diagram illustrating an example in which a penalty is given to a defect inspector.

Meanwhile, the steps illustrated in FIGS. 3 and 4 may be understood as being performed by a platform server (hereinafter, referred to as a server) operated by the service provider 20, but are not limited thereto.

In addition, a plurality of workers 32 or a plurality of inspectors 34 perform work using a predetermined terminal device. The terminal device of the operator 32 or the inspector 34 may be a computer device or a telecommunication device such as a smartphone, tablet, PDA, laptop, desktop, etc., but is not limited thereto.

First, referring to FIG. 3, the server assigns a plurality of tasks of a crowdsourcing-based project (hereinafter referred to as a project) to a plurality of workers 32 and requests the task to be performed (S110), and a plurality of workers 32 A plurality of work results are inputted from (S120).

Here, one project may contain multiple jobs, and the plurality of jobs may be grouped into a plurality of tasks according to their type, or each job may be a different type of job or the same job. , But is not limited thereto.

Next, the server assigns the plurality of work results to the plurality of inspectors 34 to request the inspection performance (S130), and receives the plurality of inspection results for the plurality of job results from the plurality of inspectors 34 (S140). ). At this time, the inspector 34 may input the inspection pass or rejection as a plurality of inspection results.

Referring to FIG. 4, in an embodiment of the present invention, a plurality of work results are assigned to a plurality of inspectors 34 to request inspection. First, m received from the operator 32 (where m is 2 or more) The actual work result of (natural number) is assigned to the inspector 34 and the inspection is requested (S131).

And the server periodically calculates the rejection rate for the m actual work results of the inspector 34 (S132), and based on the calculated rejection rate, n (where n is a natural number of 1 or more) of verification work results Assigned to the inspector 34 to request inspection.

In this case, the rejection rate in an embodiment of the present invention is a value calculated by dividing the number of initial rejections among m actual job results by m, and the number of re rejections due to rework is not reflected.

Referring to FIG. 5, for example, when the actual work result received from the worker is 20 (m=20) and the assigned verification work result is set to 1 (n=1), the server is 20 actual work. Whenever the result is assigned to the inspector, one verification work result can be assigned together. At this time, m, which corresponds to the number of actual work results, is a cycle for allocating the verification work results to the inspector.

On the other hand, the actual work result refers to the work result in which the correct answer does not exist as the test result has not been reviewed by another inspector, and the work result for verification refers to the work result in which the correct answer to the test result exists. The correct answer to the result of the inspection on the result of the verification work may be confirmed as rejection.

In one embodiment, the verification work result is a work result assigned to and performed by the worker 32, similar to the actual work result, and refers to a work result that was first rejected by another inspector. In other words, the server may extract a case in which the first inspection result is rejected among the cases assigned to another inspector as a verification work result and provide it to the corresponding inspector 34. Accordingly, the inspector 34 cannot distinguish between the actual work result and the verification work result. However, the verification result is not necessarily limited thereto, and a separately prepared one may be used by the requester 20.

In one embodiment, if there is a rejected job result among the inspection results for m actual job results, the server requests re-assignment to the worker 32 corresponding to the rejected job to perform rework, and the inspector 34 You may not pay the inspection fee. That is, the inspection cost for the actual work result is paid to the inspector 34 only when the inspector 34 inputs the inspection pass, not the rejection.

On the contrary, if the result of the inspection for n verification work results is rejected, the server does not request reassignment and rework to the corresponding worker 32 for the rejected job, and pays the inspection fee to the inspector 34. can do. That is, in the case of the inspection of the result of the verification work, even if the inspector 34 inputs the rejection, the purpose of inspection quality management has been achieved. In this case, the inspection cost is paid to the inspector 34. In this way, as the inspection cost is paid for the result of the verification work, the inspector 34 cannot distinguish whether the assigned work result is the actual work result or the inspection work result when the work result is assigned. The results of the inspection work are inspected without distinction. On the other hand, since the verification work results correspond to the actual work results of other inspectors, rework and re-inspection due to rejection are performed by the operator and other inspectors.

Unlike the embodiment in which the above inspection cost is paid, the inspection cost may not be paid because the results of n verification tasks were previously inspected by another inspector. In this case, the inspector 34 can recognize that the verification work results are assigned every specific cycle, and the verification work results are included in a random position between m actual work results to be assigned next, as in the following embodiments. And can be assigned. At the end of the project, the inspector 34 can check how many verification work results have been assigned to him/her. In the case of this embodiment, it is desirable to explain and recognize that the results of the verification work may be assigned to the inspector 34 before the start of the project.

Referring back to FIG. 4, in an embodiment of the present invention, the server compares the rejection rate for m actual work results of the inspector 34 with a predetermined reference rejection rate (S133), and when the comparison result is less than the reference rejection rate (S133-Y), in order to verify the inspection results for m actual work results, n verification work results may be assigned to the inspector 34 to request the inspection performance (S134).

In contrast, when the rejection rate for m actual work results of the inspector 34 is greater than or equal to the predetermined standard rejection rate (S133-N), the server omits the verification of the inspection results for m actual work results and performs n verifications. It is possible not to request the inspection performance by allocating the results of the work to the inspector 34 (S135).

In the above embodiment, since the high rejection rate means that the examiner is excellent, an embodiment of the present invention may omit the assignment of the verification work result in this case. When the inspector 34 enters the rejection as a result of the inspection of the work result, the inspector 34 must perform re-examination for the rework case again, which is not rejected, but continues to be repeated until it passes. Soon becomes a burden on the inspector (34). Therefore, the inspector 34 does not deliberately reject a case that has passed the inspection, and only rejects the case when there is a reason for rejection, so that the rejection rate is high means that the inspection quality is high.

Referring to FIG. 6, in an exemplary embodiment of the present invention, when the rejection rate in a corresponding period is high, assignment of a verification task result may be omitted only for a corresponding period.

For example, the server calculates the rejection rate for the first 20 cases (the 1st to 20th cases) allocated to the inspector 34, and if the rejection rate in the corresponding cycle is high, the assignment of the verification work result is omitted. can do. However, if the rejection rate for the next 20 cases (the 21st to the 40th case) is low, the assignment of the verification work result corresponding to this cycle can be allocated without omitting. If the rejection rate of the inspector 34 continues to be high every cycle, the assignment of the verification work result may be continuously omitted. This process can be repeated until the end of the project, or can be repeated until a cycle of multiple times.

As described above, in an embodiment of the present invention, it is possible to determine whether an inappropriate inspection is possible through the assignment of the verification work results, and at the same time, in the case of an excellent inspector, the cost due to the verification work results increases by omitting the assignment of the verification work results. You can adjust the degree to which you do.

In one embodiment, n verification work results may be assigned after m actual work results. For example, as a result of comparison with the reference rejection rate for the first cycle, if the rejection rate is greater than or equal to the reference rejection rate, the results of n verification tasks to be assigned are assigned after the results of m actual tasks performed in the first cycle. If it is necessary to allocate verification work results in the second cycle, n verification work results are allocated after m actual work results performed in the second cycle.

In another embodiment, n verification work results may be included and assigned to random positions of m actual work results. For example, if the rejection rate is greater than or equal to the reference rejection rate as a result of comparison with the reference rejection rate for the first cycle, the results of n verification tasks to be assigned afterwards are included in the m actual task results assigned to the second cycle and assigned. Can be.

On the other hand, the high or low rejection rate for a specific inspector is determined by the reference rejection rate and the comparison result, and the reference rejection rate may be set as the average rejection rate for m actual work results of all inspectors of the project.

In addition, the standard rejection rate may be determined in advance, which means that the actual work result for the period is completed and before the assignment decision for the verification work result is made.

For example, the standard rejection rate of the first 20 cases may be calculated as the average rejection rate of all inspectors for the first 20 cases (a total of 20 cases corresponding to the 1st to 20th cases). And the standard rejection rate of the next 20 cases can be calculated as the average rejection rate of all inspectors for the next 20 cases (a total of 20 cases corresponding to the 21st to 40th cases).

In addition, the average rejection rate may be determined for the examinees 34 who have completed m inspections, and the examinees 34 who have not completed m inspections may not be considered when calculating the average rejection rate. In the above example, the average rejection rate is calculated for the inspectors 34 who have completed 20 inspections, and the inspectors 34 who have performed only 10 inspections in the corresponding cycle are excluded.

Accordingly, the average rejection rate may be calculated differently for each examinee 34. That is, the calculation time of the average rejection rate is the time when each examiner 34 has completed the inspection of m actual work results, so the average rejection to determine whether or not the examiner A and examiner B are assigned the results of the verification work. Rate points can be different. For example, for inspector A, the average rejection rate calculated for a total of 10 inspectors can be determined as the standard rejection rate, and inspector B, who reached the period later than inspector A, targets a total of 11 or more inspectors. The average rejection rate calculated as may be determined as the reference rejection rate.

If, among the inspectors, the inspector 34 who first completes the inspection of m actual work results, the average rejection rate cannot be set as the reference rejection rate because the cycle of other inspectors has not arrived. In this case, the rejection rate initially set by the requester 20 may be used as the reference rejection rate, and may be compared with the rejection rate of the corresponding inspector 34.

Meanwhile, in an embodiment of the present invention, a plurality of actual work results may be allocated to the inspector 34, and one or more work results for verification may be allocated. The object of the present invention is to verify the inspector 34, and if the number of verification work results allocated to the inspector 34 becomes more than the actual number of work results, the project schedule is delayed and the cost burden increases. Therefore, it is preferable that the number of actual work results is at least two. In one embodiment, the number of actual work results is preferably set to exceed the number of work results for verification, but is not limited thereto.

Referring to FIG. 3 again, the server manages the quality of the inspection of each inspector 34 by using the inspection result (S150).

In one embodiment, the server manages the quality of m actual work results of the inspector 34 by using the inspection results for n verification work results.

In this case, when the inspection results for the n verification work results are not rejected, the server may determine the quality of the inspection for the m actual work results of the inspector 34 as inappropriate.

That is, at the time of inspection of the actual work result, the inspector 34 receives the inspection fee only when the pass is entered, not the rejection, so the unfaithful or unfaithful inspector 34 receives the inspection result in order to receive the inspection fee. It is also possible to enter the inspection result for the recklessly inspection pass.

In order to extract the inspection cases of the inspector 34, an embodiment of the present invention allocates the verification work results, and if the inspection results are not rejected, the server inspects m actual work results for the period. Quality can be judged as nonconforming

In the above embodiment, when the number of nonconformities reaches a predetermined reference number based on the result of the quality of inspection for m actual work results of the inspector 34, the server may determine the inspector as a defect inspector.

In addition, the server may restrict the defect inspector from participating in the inspection of the project for a predetermined limited period, or grant a penalty for reimbursing the inspection cost for m actual work results paid to the defect inspector.

For example, referring to FIG. 7, when the predetermined number of times is 3, a penalty is given to an inspector who is determined to be inadequate for the first 20 cases, the second 20 cases, and the third 20 cases. As an example of the penalty, the inspector 34 may not participate in the project for a predetermined period or for a predetermined period, and may participate in the inspection of the project again as the period passes.

Alternatively, when it is determined as a defective inspector, the total number of actual work results that can be assigned afterwards may be reduced, or the number of m number of actual work results assigned may be reduced.

In the above embodiment, if the server determines that the quality of the inspection of m actual work results of the inspector 34 is inadequate, the server may request re-review by reassigning the m actual work results of the inspector 34 to other inspectors. .

If it is judged as inappropriate, m actual work results may be assigned to any one or more than one reviewer, and other reviewers are assigned as normal actual work results without being distinguished by requests for reassignment and re-review. do.

On the other hand, if the work results inspected by other inspectors are provided as verification work results, the inspection results by other inspectors may not be perfect, so the number of cases in which the inspection results for the verification work results passed, not rejected. In the case of occurrence of multiple times, the server may notify the requestor 20 to confirm it. If the correct answer to the verification work result is pass rather than rejection, the server deletes the number of times counted as inappropriate to the inspector 34.

Meanwhile, in the above description, steps S110 to S150 may be further divided into additional steps or may be combined into fewer steps, according to an embodiment of the present invention. In addition, some steps may be omitted as necessary, and the order between steps may be changed. In addition, even if other contents are omitted, the contents of FIG. 8 to be described later can be applied to the cost-effective inspection quality management method of the crowdsourcing-based project of FIGS. 1 to 7.

Hereinafter, a cost-effective inspection quality management apparatus 200 (hereinafter referred to as inspection quality apparatus) for a crowdsourcing-based project according to an embodiment of the present invention will be described.

8 is a view for explaining the inspection quality management apparatus 200 according to an embodiment of the present invention.

Referring to FIG. 8, the inspection quality management apparatus 200 includes a communication module 210, a memory 220, and a processor 230.

The communication module 210 transmits a crowdsourcing-based work for one project to a plurality of workers 32 to request a task to be performed, and receives a work result from the plurality of workers 32. Further, the work results received from the plurality of workers 32 are transmitted to the plurality of inspectors 34 to request inspection, and the inspection results are received from the plurality of inspectors 34.

The memory 220 stores a program for managing the inspection quality of each inspector 34 using the inspection result.

The processor 230 executes a program stored in the memory 220. As the processor 230 executes the program stored in the memory 220, the m actual job results of the inspector 34 are periodically assigned to the inspector 34 to request the inspection. Calculate the rejection rate for, and based on this, assign n verification work results to the inspector 34 to request the inspection performance, and then use the inspection results for the n verification work results to be used by the inspector 34 It manages the quality of inspection for m actual work results.

The cost-effective inspection quality control method for a crowdsourcing-based project according to an embodiment of the present invention described above may be implemented as a program (or application) to be executed in combination with a computer, which is hardware, and stored in a medium.

The above-described program includes C, C++, JAVA, Ruby, which can be read by a processor (CPU) of the computer through the device interface of the computer, in order for the computer to read the program and execute the methods implemented as a program. It may include a code (Code) coded in a computer language such as machine language. Such code may include a functional code related to a function defining necessary functions for executing the methods, and a control code related to an execution procedure necessary for the processor of the computer to execute the functions according to a predetermined procedure. can do. In addition, such code may further include additional information required for the processor of the computer to execute the functions or code related to a memory reference to which location (address address) of the internal or external memory of the computer should be referenced. have. In addition, when the processor of the computer needs to communicate with any other computer or server in the remote in order to execute the functions, the code uses the communication module of the computer to determine how It may further include a communication-related code for whether to communicate, what kind of information or media should be transmitted and received during communication.

The stored medium is not a medium that stores data for a short moment, such as a register, cache, memory, etc., but a medium that stores data semi-permanently and can be read by a device. Specifically, examples of the storage medium include, but are not limited to, ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, and the like. That is, the program may be stored in various recording media on various servers to which the computer can access, or on various recording media on the user's computer. In addition, the medium may be distributed over a computer system connected through a network, and computer-readable codes may be stored in a distributed manner.

The above description of the present invention is for illustrative purposes only, and those of ordinary skill in the art to which the present invention pertains will be able to understand that it can be easily modified into other specific forms without changing the technical spirit or essential features of the present invention. will be. Therefore, it should be understood that the embodiments described above are illustrative in all respects and not limiting. For example, each component described as a single type may be implemented in a distributed manner, and similarly, components described as being distributed may also be implemented in a combined form.

The scope of the present invention is indicated by the claims to be described later rather than the detailed description, and all changes or modified forms derived from the meaning and scope of the claims and their equivalent concepts should be interpreted as being included in the scope of the present invention. do.

10: Client
20: service provider
30: public
32: worker
34: inspector
200: Inspection quality control device
210: communication module
220: memory
230: processor

Claims (10)

As a method performed by a computer,
Assigning a plurality of tasks of a crowdsourcing-based project (hereinafter, referred to as a project) to a plurality of workers and requesting the task to be performed;
Receiving a plurality of job results from the plurality of workers;
Assigning the plurality of work results to a plurality of inspectors and requesting to perform inspection;
Receiving an inspection pass or rejection as a plurality of inspection results for the plurality of work results from the plurality of inspectors; And
Including the step of managing the inspection quality of each inspector using the inspection result,
Assigning the plurality of work results to a plurality of inspectors and requesting to perform the inspection,
Whenever m (where m is a natural number of 2 or more) actual work results are allocated to the inspector and requested to perform the inspection, periodically, based on the rejection rate of the m actual work results of the inspector, n (only , n is a natural number of 1 or more) including the step of requesting to perform the inspection by selectively assigning the verification work result to the inspector,
The step of managing the inspection quality of each inspector using the inspection result,
And managing the inspection quality of the m actual work results of the inspector by using the inspection results for the n verification work results,
The correct answer to the inspection result for the verification work result is confirmed as rejection,
A cost-effective inspection quality control method for crowdsourced projects. The method of claim 1,
The step of allocating the m actual work results to the inspector and requesting to perform the inspection by periodically allocating n verification work results to the inspector whenever the inspection is requested,
If the checker's rejection rate for the m actual work results is less than a predetermined standard rejection rate, the n verification work results are assigned to the examiner to verify the inspection results for the m actual work results. The step of requesting to be performed, and
If the checker's rejection rate for the m actual work results is more than the standard rejection rate, verification of the inspection results for the m actual work results is omitted and the n verification work results are assigned to the examiner for inspection. Including the step of not requesting to be performed,
A cost-effective inspection quality control method for crowdsourced projects. The method of claim 2,
The predetermined standard rejection rate is,
The average rejection rate for the m actual work results of all inspectors of the project,
A cost-effective inspection quality control method for crowdsourced projects. The method of claim 1,
If the inspection results for the m actual work results are rejected, the rejected work is reassigned to the corresponding worker to request rework, and the inspection fee is not paid to the inspector,
When the inspection results for the n verification work results are rejected, reassigning the rejected work to a corresponding worker to perform rework, and paying the inspection cost to the inspector,
A cost-effective inspection quality control method for crowdsourced projects. The method of claim 1,
The result of the verification operation,
The result of the work that has already been checked and returned by another inspector,
A cost-effective inspection quality control method for crowdsourced projects. The method of claim 1,
The step of managing the inspection quality of the m actual work results of the inspector by using the inspection results for the n verification work results,
Comprising the step of determining that all the inspection quality of the m actual work results of the inspector as non-conformity when the inspection results for the n verification work results are not rejected,
A cost-effective inspection quality control method for crowdsourced projects. The method of claim 6,
When the number of times determined as non-conformity reaches a predetermined reference number based on the determination result of the inspection quality of the m actual work results of the inspector, determining the inspector as a defect inspector,
A cost-effective inspection quality control method for crowdsourced projects. The method of claim 7,
Restricting the defect inspector from participating in the inspection of the project for a predetermined limited period, or giving the defect inspector a penalty for recovering the inspection cost for the m actual work results paid,
A cost-effective inspection quality control method for crowdsourced projects. The method of claim 6,
If it is determined that the quality of the inspection of the m actual work results of the inspector is all inadequate, the step of reassigning the m actual work results of the inspector to another inspector and requesting to perform re-examination,
A cost-effective inspection quality control method for crowdsourced projects. A computer program combined with a computer and stored in a computer-readable recording medium for executing the cost-effective inspection quality control method of a crowdsourcing-based project according to any one of claims 1 to 9.

Images