KR102155750B1 - Method for managing inspection quality using returned work result of crowdsourcing based project for artificial intelligence training data generation - Google Patents

Abstract

Provided is a method for managing inspection quality using returned work results of a crowdsourcing-based project for generating artificial intelligence training data to reduce expenses for making a work result for verification. According to the present invention, the method comprises the following steps of: assigning a plurality of works of a crowdsourcing-based project to a plurality of workers to request execution; receiving a plurality of work result from the plurality of workers; assigning the plurality of work results to a plurality of inspectors to receive inspection; receiving a plurality of inspection results for the plurality of work results from the plurality of inspectors; separating the plurality of inspection results into a pass and a return and collecting the returned work results; and using the returned work results to manage inspection quality of each inspector. The step of managing the inspection quality of each inspector comprises the following steps of: assigning the work result (hereinafter, referred to as a work result for verification) returned by other inspectors to the inspector to request inspection; receiving an inspection result for the work result for verification from the inspector; and when the inspection result for the work result for verification is not determined as a return, determining the inspection quality of the inspector as inadequate.

Description

[METHOD FOR MANAGING INSPECTION QUALITY USING RETURNED WORK RESULT OF CROWDSOURCING BASED PROJECT FOR ARTIFICIAL INTELLIGENCE TRAINING DATA GENERATION}

The present invention relates to an inspection quality management method using the rejected work result of a crowdsourcing-based project for generating artificial intelligence learning data.

Recently, more and more companies collect and process large amounts of data on a crowdsourcing basis that engages the general public in some process of corporate activities. In other words, by opening a project and allowing the general public, that is, workers to participate in the project, necessary information is collected through the work results completed by the workers.

At this time, in order to collect more reliable information, the company can evaluate the reliability of the worker's work data by assigning the result of work completed by the worker to the inspector to perform the inspection.

However, if the inspector does not properly perform the inspection function and performs the inspection in a manner that unconditionally passes the work result of the worker, the quality of the work data is deteriorated, as well as the damage of paying the inspection cost to the defective inspector. In addition, when such a situation is prevalent, the damage is further spread because the defective workers are mass-produced and the cost for the defective work is also paid.

Therefore, it is necessary to test and verify the inspector by presenting the inspection inspection work for the so-called trap function to the inspector, but there is a problem that time and financial costs are incurred in creating the result of the verification work in which the correct answer exists. This is because, in the case of a project with a large amount of work data, it is necessary to create the work result for verification at a proportional scale.

Unexamined Patent Publication No. 10-2014-0095956, 2014.08.04.

The problem to be solved by the present invention is to provide an inspection quality management method using the rejected work results of a crowdsourcing-based project for generating artificial intelligence learning data.

However, the problem to be solved by the present invention is not limited to the problem as described above, and other problems may exist.

In an aspect of the present invention for solving the above-described problems, the quality control method for inspection using the rejected work results of a crowdsourcing-based project is a request for performing work by assigning a plurality of tasks of a crowdsourcing-based project to a plurality of workers. The step of, receiving a plurality of work results from the plurality of workers, assigning the plurality of work results to a plurality of inspectors and requesting to perform inspection, a plurality of the plurality of work results from the plurality of inspectors Receiving an input of inspection results, dividing the plurality of inspection results into inspection completion and rejection, collecting rejected operation results, and managing inspection quality of individual inspectors using the rejected operation results, and , The step of managing the inspection quality of the individual inspectors includes the steps of requesting to perform inspection by assigning the job result returned by another inspector (hereinafter, the verification job result) to the inspector, and the verification job from the inspector And receiving an input of an inspection result for the result; and if the inspection result for the verification work result is not rejected, determining the quality of the inspection by the inspector as inappropriate.

In some embodiments of the present invention, the verification work result may be assigned to the inspector together with an actual work result that has not been inspected by another inspector.

In some embodiments of the present invention, when the inspection result for the actual work result is rejected, the rejected job is reassigned to a corresponding worker to request rework, and the inspection fee is not paid to the inspector, and the verification If the result of the inspection on the result of the application is rejected, the rejected job may be reassigned to a corresponding worker and the inspection fee may be paid to the inspector without requesting rework.

In some embodiments of the present invention, the step of requesting the inspection by assigning the verification work result to the inspector may be repeatedly performed according to a predetermined period or a period of a predetermined inspection quantity.

In some embodiments of the present invention, when the inspection result for the verification work result is not rejected, the cycle may be shortened in relation to the inspector.

In some embodiments of the present invention, when the inspection result for the verification work result is rejected, the period may be extended in relation to the inspector.

In some embodiments of the present invention, when the result of the inspection for the result of the verification work is not rejected, the result of the verification work may be continuously assigned to the inspector to request the inspection.

In some embodiments of the present invention, the method determines that the inspector is a defective inspector when the nonconformity ratio is greater than or equal to a predetermined value or the number of nonconformities reaches a predetermined number based on a result of the inspection quality of the inspector. It may further include the step of.

In some embodiments of the present invention, the method may further include limiting the defect inspector from participating in the inspection for a predetermined period of time, or granting a penalty for reimbursing the inspection cost paid to the defective inspector.

In some embodiments of the present invention, the method may further include the step of requesting re-examination by reassigning the work result assigned to the defective inspector to another inspector.

A computer program according to another aspect of the present invention for solving the above-described problem is combined with a computer that is hardware to execute a quality control method for inspection using the rejected work results of the crowdsourcing-based project, and to a computer-readable recording medium. Is saved.

Other specific details of the present invention are included in the detailed description and drawings.

According to the present invention described above, there is no need to separately create a verification work result for verifying an inspector because the already rejected work result is utilized, and thus the cost of producing the verification work result can be reduced.

In addition, by monitoring the inspectors using the result of the already rejected work, it is possible to identify defect inspectors in real time and give penalty to the defect inspectors, thereby minimizing the damage caused by the defective inspectors and increasing the quality of the work data. At the same time, it is possible to cultivate reliable inspectors by monitoring inspectors to determine integrity inspectors.

In addition, by performing retraining of an inspector who is identified as a defective inspector as a result of monitoring, the quality of subsequent inspection can be improved.

In addition, when the error rate is high in relation to the specific verification work result, it is determined that there is a problem such as uncertainty in the inspection guide related to the specific verification work result, and the inspection guide can be reviewed and improved.

The effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the following description.

1 is a conceptual diagram of a crowdsourcing service according to an embodiment of the present invention.
2 is a flowchart illustrating a crowdsourcing-based project progress process according to an embodiment of the present invention.
3 is a flowchart of a quality control method for inspection using rejected work results of a crowdsourcing-based project according to an embodiment of the present invention.
4 is a flow chart of a method of determining an inspector's inspection quality as inappropriate according to an embodiment of the present invention.
FIG. 5 is an exemplary diagram for explaining an inspection quality determination period according to a predetermined period according to an embodiment of the present invention.
6 is an exemplary diagram for explaining an inspection quality determination cycle according to a predetermined inspection quantity according to an embodiment of the present invention.
7 is an exemplary diagram for explaining determining a defect inspector based on a nonconformity ratio according to an embodiment of the present invention.
8 is an exemplary view for explaining determining a defect inspector based on the number of nonconformities according to an embodiment of the present invention.
9 is a view for explaining an apparatus for managing inspection quality using rejected work results of a crowdsourcing-based project according to an embodiment of the present invention.

Advantages and features of the present invention, and a method of achieving them will become apparent with reference to the embodiments described below in detail together with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but may be implemented in a variety of different forms, only the present embodiments are intended to complete the disclosure of the present invention, It is provided to fully inform the technician of the scope of the present invention, and the present invention is only defined by the scope of the claims.

The terms used in the present specification are for describing exemplary embodiments and are not intended to limit the present invention. In this specification, the singular form also includes the plural form unless specifically stated in the phrase. As used in the specification, “comprises” and/or “comprising” do not exclude the presence or addition of one or more other elements other than the mentioned elements. Throughout the specification, the same reference numerals refer to the same elements, and “and/or” includes each and all combinations of one or more of the mentioned elements. Although "first", "second", and the like are used to describe various elements, it goes without saying that these elements are not limited by these terms. These terms are only used to distinguish one component from another component. Therefore, it goes without saying that the first component mentioned below may be the second component within the technical idea of the present invention.

Unless otherwise defined, all terms (including technical and scientific terms) used in the present specification may be used as meanings that can be commonly understood by those of ordinary skill in the art to which the present invention belongs. In addition, terms defined in a commonly used dictionary are not interpreted ideally or excessively unless explicitly defined specifically.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a conceptual diagram of a crowdsourcing service according to an embodiment of the present invention.

Referring to FIG. 1, a crowdsourcing service is performed by being composed of a client 10, a service provider 20, and a public 30.

The sponsor 10 refers to a company or individual who requests a crowdsourcing-based project (hereinafter, referred to as a project).

The requester 10 requests a project for the purpose of collecting source data or annotating data for generating artificial intelligence learning data. The data generated through the project can be used as learning data for arbitrary machine learning such as supervised learning, unsupervised learning, and reinforcement learning. Collection of source data refers to collecting raw data such as recorded voice collection and photo collection. Data annotation refers to inputting relevant annotation data into source data such as text, photos, and videos. For example, the data annotation may include finding an entity in a given fingerprint or finding a similar sentence, but is not limited thereto. Meanwhile, the types of the above-described projects are only one embodiment, and various projects may be handled in the present invention according to the design of the client.

The service provider 20 refers to a company that provides crowdsourcing services.

When the service provider 20 receives a request for a project for a product or service from the client 10, the service provider 20 allocates a task for the project to the general public 30 and receives the work result from the public 30. Thereafter, the final product extracted based on the work result is provided to the client 10.

At this time, the service provider 20 provides a crowdsourcing service to the client 10 and the public 30 through a crowdsourcing platform (hereinafter, the platform). That is, when the service provider 20 receives a request for a project from the client 10, the service provider 20 opens the project on the platform. Thereafter, when the work result for the open project is received from the public 30, the project is terminated on the platform, and the final product may be extracted and provided to the client 10.

The public 30 refers to the general public participating in the project open on the platform. Here, the public 30 may participate in a project opened on the platform through an application or website provided by the service provider 20.

The public 30 is composed of a worker 32 and an inspector 34.

The worker 32 decides to participate in a specific project among a plurality of projects open on the platform. Thereafter, the operator 32 performs an operation such as collection of source data or data annotation, and transmits it to the platform.

The inspector 34 decides to participate in a specific project among a plurality of projects open on the platform. Thereafter, the inspector 34 inspects the result of the work performed by the operator 32. As a result of performing the inspection, the inspector 34 may perform inspection pass processing or rejection processing, and input a rejection reason during rejection processing. In the case of passing the inspection, it is not necessary to rework and re-examination due to this, so passing inspection has the same meaning as completion of inspection.

2 is a flowchart illustrating a crowdsourcing-based project progress process according to an embodiment of the present invention.

First, the requester 10 requests one or more projects to the service provider 20 (S11).

Thereafter, the service provider 20 opens the requested project on the platform (S12). At this time, the service provider 20 may determine the grade in consideration of the difficulty of the project before opening the project. That is, depending on the difficulty level, it is possible to determine which level or higher public 30 to expose the project. Accordingly, it is possible to increase the reliability of the work result of the project.

Thereafter, the service provider 20 requests the work by assigning the work to the workers 32 of the corresponding level or higher according to the level of the project (S13).

Thereafter, the worker 32 performs the assigned task (S14). In this case, the worker 32 may input the reason for the inability to work without performing the work for a work in which the work itself is impossible for some reason.

Thereafter, the service provider 20 receives the work result from the worker 32 (S15), and assigns the work result to the inspector 34 to request the inspection (S16).

Thereafter, the inspector 34 performs the assigned inspection (S17). At this time, the inspector 34 decides to complete the inspection when it is determined that the work has been properly performed, and if it is determined that the inspection work is wrong, the inspector 34 rejects it. During rejection processing, the inspector 34 inputs the rejection reason for what reason the job was judged to be wrong.

Thereafter, the service provider 20 receives the inspection result from the inspector 34 (S18).

When the inspection result is the inspection completion, the service provider 20 uses the work result as valid data, and based on this, extracts the final product at the end of the project.

When the inspection result is rejection processing, the service provider 20 may internally perform the inspection again, or assign the work to the worker 32 again to perform the rework. When reworking, re-examination by an inspector is required.

Thereafter, when the project period ends or sufficient valid data is secured, the service provider 20 terminates the project (S19), calculates a final result based on the secured valid data, and provides it to the client 10 ( S20).

At this time, before the end of the project, the service provider 20 evaluates the performance results of the worker 32 and the inspector 34, calculates the work cost and the inspection cost according to the evaluation, to the worker 32 and the inspector 34. give.

In FIGS. 1 and 2, it is simply expressed as a requester 10, a service provider 20, an operator 32, and an inspector 34, but these are smartphones, tablets, PDAs, laptops, desktops, operated by each participant. It means a computer device such as a server or a telecommunication device.

In the following, referring to Figs. 3 to 8, when requesting for inspection by assigning work results to each inspector, in addition to the actual work results that can know whether the inspection passes or return is necessary only through inspection, the verification work results for which the correct answer is determined are periodically assigned together. And, based on the inspection results for the verification work results, explain the management of the inspection quality of each inspector.

3 is a flowchart of a quality control method for inspection using rejected work results of a crowdsourcing-based project according to an embodiment of the present invention. 4 is a flow chart of a method of determining an inspector's inspection quality as inappropriate according to an embodiment of the present invention. FIG. 5 is an exemplary diagram for explaining an inspection quality determination period according to a predetermined period according to an embodiment of the present invention. 6 is an exemplary diagram for explaining an inspection quality determination cycle according to a predetermined inspection quantity according to an embodiment of the present invention. 7 is an exemplary diagram for explaining determining a defect inspector based on a nonconformity ratio according to an embodiment of the present invention. 8 is an exemplary view for explaining determining a defect inspector based on the number of nonconformities according to an embodiment of the present invention.

Meanwhile, the steps shown in FIG. 3 may be understood as being performed by a platform server (hereinafter, referred to as a server) operated by the service provider 20, but are not limited thereto.

In addition, a plurality of workers 32 or a plurality of inspectors 34 perform work using a predetermined terminal device. The terminal device of the operator 32 or the inspector 34 may be a computer device or a telecommunication device such as a smartphone, tablet, PDA, laptop, desktop, etc., but is not limited thereto.

First, referring to FIG. 3, the server assigns a plurality of tasks of a crowdsourcing-based project to a plurality of workers 32 to request the task performance (S110), and receives a plurality of task results from the plurality of workers 32. (S120), a plurality of work results are assigned to a plurality of inspectors 34 to request inspection performance (S130), and a plurality of inspection results for a plurality of work results are input from the plurality of inspectors 34 (S140) .

Here, the plurality of workers 32 and the plurality of inspectors 34 refer to all workers and inspectors participating in the same project.

Since the steps S110 to S140 overlap with the steps S13 to S18 described above with reference to FIG. 2, a detailed description will be omitted.

Subsequently, the server divides the plurality of inspection results received from the plurality of inspectors 34 into inspection completion and rejection, and collects the rejected work results (S150).

As described above, the server uses the reviewed work result as valid data for extracting the final product at the end of the project.

In addition, as described above, the server requests rework by assigning the task again to the worker 32 for the rejected task result.

At the same time as requesting rework of the rejected job result, the server collects the rejected job result.

In this case, the server generates and stores the job result-inspector matching data by matching the returned job result and the inspector 34 who processed the rejected job result with each other. This is to prevent repeated assignments to the inspectors who have already been rejected when assigning the rejected work results to each inspector 34 for the inspection quality control of the individual inspectors afterwards. For example, if inspector A and the rejected work result a are matched and stored, then, when inspecting quality control by individual inspectors, inspector A will not be assigned the job result a that has already been rejected by inspector A.

Subsequently, the server manages the quality of the individual inspector's inspection by using the returned job result (S160).

Here, the rejected work result means the work result that another inspector has already performed the inspection as'rejected'.

In other words, the server can verify each inspector by using the result of work already returned by other inspectors as the work result for inspection.

Since the correct answer for the verification work result is set as'reject', the server can determine whether the inspector 34 is properly performing the verification according to the verification result of the verification work result. This is because the inspection cost is paid to the inspector 34 only when the inspection result is the inspection completion, so that the defective inspector for the purpose of money unconditionally performs the inspection result as inspection completion.

Accordingly, the server may determine that the inspector 34, who has completed the inspection for the verification work result, is not properly performing the inspection, and may determine that the quality of the inspection result performed by the corresponding inspector 34 is inappropriate.

Specifically, referring to FIG. 4, the server allocates a result of a job rejected by another inspector 34 (hereinafter, a result of a verification operation) to the inspector 34 to request the inspection (S161), and the inspector 34 ), the inspection result for the verification work result is input (S162), and if the verification result for the verification work result is not rejected, the quality of the inspection of the inspector 34 is determined as inadequate (S163). Hereinafter, steps S161 to S163 will be described in detail.

The server allocates the verification work result to each of the inspectors 34 based on the inspector-work result matching data stored in step S150. For example, as described above, when the inspector A matches the work result a, the server may allocate the work result a to the inspectors except for the inspector A.

In addition, the server generates and stores a verification task result assignment list for each examiner by matching the verification task results and the verified examiners 34 by receiving the verification task results. This is because the trap function is lost if the result of the verification work already presented to the specific inspector 34 is presented to the same specific inspector 34 again. Therefore, the verification work result presented once to a specific inspector 34 is not repeatedly presented to the same specific inspector 34 again.

In addition, when allocating the verification work result to the inspector 34, the server also allocates the actual work result. Here, the actual work result refers to a work result that has not been inspected by another inspector 34, which is first inspected by the inspector 34. If the corresponding inspector 34 treats the inspection result for the actual work result as a rejection, the rejected actual work result is later assigned to another inspector as the verification work result.

At this time, the inspector 34 cannot distinguish between the actual work result and the verification work result. That is, the inspector 34 performs the inspection by recognizing the inspected verification work result as an actual work result. As a result, the verification result can function as a trap for the defect inspector.

If the result of the inspection by the inspector 34 on the actual work result is rejected, the server requests the operator 32 who has performed the rejected job to perform rework by reassigning the job as described above, and the inspector 34 ) Do not pay the inspection fee. Since the actual work result is used as valid data for extracting the final product when the inspection is completed, the server pays the inspection cost only when the inspection result for the actual work result is the inspection completion.

If the result of the inspection by the inspector 34 for the verification result is rejected, the server does not request to perform rework by reassigning the job to the worker 32 who performed the rejected job, but to the inspector 34 Pay the inspection fee. This is because the result of the verification work has already been rejected by the inspector who has already inspected it for the first time, and the rework has already been requested from the worker 32 who performed the rejected operation at the time of the initial rejection. In addition, since the verification work result is to determine whether the inspector 34 is properly performing the inspection by using the rejected work result, the inspection is performed when the inspector 34 treats the verification work result as a rejection. It is judged that it is performing properly and the inspection fee is paid as compensation accordingly.

In addition, the server allocates the results of the verification work to each inspector 34 for each period. In each cycle, the server may allocate one verification work result to the inspector 34, or continuously allocate a plurality of verification work results to the inspector 34. At this time, the period may be applied equally to each inspector 34 at first, and then may be shortened or extended by a predetermined ratio according to the inspection result of the verification work result of each inspector 34.

Specifically, if the inspection result of the inspection result of the inspector 34 is not rejected, a shortened period is applied to the inspector 34 thereafter. This is because the inspector 34 incorrectly performed the inspection due to the completion of the inspection for the result of the verification that the correct answer is set as rejection, the server determines that the inspection quality of the inspector 34 is inappropriate, and the inspector 34 ) Can be monitored. As the cycle is shortened, defect inspectors can be quickly identified.

Conversely, if the result of the inspection for the verification work result of the inspector 34 is rejected, an extended period is applied to the inspector 34 afterwards. This is because the inspector 34 has properly performed the inspection for the result of the verification that the correct answer is set as rejection, so the server determines that the inspection quality of the inspector 34 is appropriate and monitors the inspector 34 at a relatively long period. can do. As the period is extended, the inspector 34 can concentrate on the actual inspection work.

In an embodiment, the server may set a predetermined period of time. Assuming that the predetermined period is set to 4 days, with reference to FIG. 5, it will be described that the period is changed according to the inspection result of each inspector's verification work result.

Referring to FIG. 5A, 4 days after the project is opened, the server performs a first quality determination for inspector A. As Inspector A treats the inspection result for the verification work result as a rejection, the server determines that Inspector A's inspection quality is appropriate. Accordingly, the server extends the period of inspector A to 8 days, and performs the second inspection quality determination for inspector A relatively leisurely. Thereafter, the server may shorten or extend a period for determining the quality of the third inspection according to the result of determining the quality of the second inspection.

Referring to FIG. 5B, 4 days after the project is opened, the server performs a first quality determination for inspector B. As Inspector B treats the inspection result of the verification work result as inspection completion rather than rejection, the server determines that Inspector B's inspection quality is inadequate. Accordingly, the server shortens the cycle of inspector B to 2 days, and performs the second inspection quality determination for inspector B relatively tightly. Thereafter, the server may shorten or extend a period for determining the quality of the third inspection according to the result of determining the quality of the second inspection.

As described above, the initial period is the same for all inspectors, and the period of each inspector varies differently according to the result of the inspection quality determination. The server performs the next inspection quality judgment for each inspector according to a cycle that varies differently for each inspector.

In another embodiment, the server may set a predetermined number of inspections in cycles. Assuming that the predetermined number of inspections is set to 4, with reference to FIG. 6, it will be described that the period is changed according to the inspection result of each inspector's verification work result.

Referring to FIG. 6A, after the project is opened, when the inspector A performs inspection on the results of four actual work, the server performs a primary inspection quality determination for the inspector A. As Inspector A treats the inspection result for the verification work result as a rejection, the server determines that Inspector A's inspection quality is appropriate. Accordingly, the server extends the cycle of inspector A to eight, and performs the second inspection quality determination for inspector A relatively leisurely. Thereafter, the server may shorten or extend a period for determining the quality of the third inspection according to the result of determining the quality of the second inspection.

Referring to FIG. 6B, after the project is opened, if the inspector B performs inspection on the results of four actual work, the server performs the first inspection quality determination for the inspector B. As Inspector B treats the inspection result of the verification work result as inspection completion rather than rejection, the server determines that Inspector B's inspection quality is inadequate. Accordingly, the server shortens the cycle of inspector B to two, and performs the second inspection quality determination for inspector B relatively tightly. Thereafter, the server may shorten or extend a period for determining the quality of the third inspection according to the result of determining the quality of the second inspection.

As described above, the initial period is the same for all inspectors, and the period of each inspector varies differently according to the result of the inspection quality determination. The server performs the next inspection quality judgment for each inspector according to a cycle that varies differently for each inspector.

In the following, with reference to FIGS. 7 and 8, when the specific inspector 34 judges the quality of the inspection, when the inspection result for the verification work result is not rejected, but when the inspection is completed, the corresponding inspector 34 is used for verification. Explain about requesting inspection by assigning work results continuously.

That is, if the inspector 34 incorrectly performs the inspection result for the initial verification work result as the inspection completion, the server takes into account the case that the inspector 34 accidentally performs the inspection incorrectly, Are assigned consecutively. Thereafter, the server may determine the corresponding inspector 34 as a defect inspector according to the inspection results of the plurality of verification work results.

In one embodiment, the server may determine the inspector 34 as a defective inspector based on the ratio.

If it is determined that the inspection result for the initial verification work result of the inspector 34 is inadequate, the server continuously allocates a predetermined number of verification work results to the corresponding inspector 34 and requests the inspection. When the nonconformity ratio for the predetermined number of verification work results is equal to or greater than a predetermined value, the server determines the corresponding inspector 34 as a defect inspector.

Referring to FIG. 7A, assuming that the predetermined number is 10 and the predetermined value is 70%, if it is determined that the inspection result for the initial verification operation result of the inspector 34 is inappropriate, the server performs the initial verification operation. Ten verification work results, including the results, are successively assigned to the inspector 34. The inspector 34 incorrectly performed 8 out of 10 verification results. Accordingly, the ratio of nonconformity of the inspector 34 is calculated as 80%, and the server determines the inspector 34 as a defective inspector.

Referring to FIG. 7B, assuming that the predetermined number is 10 and the predetermined value is 70%, if it is determined that the inspection result for the initial verification work result of the inspector 34 is inappropriate, the server performs the initial verification work. Ten verification work results, including the results, are successively assigned to the inspector 34. The inspector 34 incorrectly performed 3 of the 10 verification work results. Accordingly, the nonconformity ratio of the inspector 34 is calculated as 30%, and the server determines that the inspector 34 is not a defective inspector.

The server allocates the actual work result to the inspector 34, which is determined not to be a defective inspector, so that the inspector 34 continues the inspection, and applies the shortened period to the point of time when the shortened period arrives again. 34) inspection and quality judgment.

In another embodiment, the server may determine the inspector 34 as a defect inspector based on the number of times.

If it is determined that the inspection result for the initial verification work result of the inspector 34 is inadequate, the server allocates the verification work results to the corresponding inspector 34 one by one, and determines whether or not each verification work result is inadequate. . When the number of times of nonconformity to the result of the verification operation reaches a predetermined number, the server determines the corresponding inspector 34 as a defect inspector.

Referring to FIG. 8A, assuming that the predetermined number of times is 7, if it is determined that the inspection result for the initial verification work result (first verification work result) of the inspector 34 is inappropriate, the server performs the verification work. Assign results one by one in succession. That is, the server allocates the second verification work result to the inspector 34, and if it is determined that the inspection result for the second verification work result is inappropriate, assigns the third verification work result to the inspector 34 Judge whether it is not suitable according to While repeating this process, when the number of times judged as inadequate reaches 7, the server determines the corresponding inspector 34 as a defective inspector.

Referring to FIG. 8B, assuming that the predetermined number of times is 7, if it is determined that the inspection result for the initial verification work result (first verification work result) of the inspector 34 is inappropriate, the server performs the verification work. Assign results one by one in succession. That is, the server allocates the second verification work result to the inspector 34, and if it is determined that the inspection result for the second verification work result is inappropriate, assigns the third verification work result to the inspector 34 Judge whether it is not suitable according to At this time, if the inspection result for the third verification work result is rejected (the inspection result is determined to be appropriate), it is determined that the inspector 34 has properly performed the inspection before reaching a predetermined number of times, and the server (34) is judged not to be a defective inspector.

On the other hand, the server may not continuously allocate the verification work results to the inspector 34. In this case as well, the server may determine the inspector 34 as a defective inspector based on a rate or number of times. Each period, the inspector 34 is assigned one verification task result, and as the period is accumulated, the nonconformity ratio is more than a predetermined value or the number of nonconformities is more than a predetermined number, and is determined as a defect inspector. In the case of the number-based defect inspector determination, it can be applied only after the inspector 34 has been assigned the minimum number of verification work results for the determination.

The server allocates the actual work result to the inspector 34, which is judged not to be a defective inspector, so that the inspector 34 continues the inspection, and applies the shortened period to the point of time when the shortened period arrives again. 34) inspection and quality judgment.

In FIGS. 7 and 8, it has been described that the predetermined number and the predetermined number include the initial verification operation result, but the predetermined number and the predetermined number may not include the initial verification operation result.

According to an embodiment, the server may provide a penalty to the inspector 34 determined to be a defective inspector.

For example, the server may limit the qualification of the inspection so that the inspector 34, which is determined as a defective inspector, cannot participate in the inspection for a predetermined period. Accordingly, it is possible to minimize damage caused by a defect inspector and improve the quality of work data. The predetermined period includes until the end of the project.

In addition, the server may recover the inspection cost already paid to the defect inspector 34. For example, the inspection cost paid as the defect inspector 34 previously completed the inspection on the actual work result and the inspection cost paid as it rejected the previous verification work result may be refunded.

According to an embodiment, the server may request re-review by reassigning the actual work result assigned to the defect inspector 34 to another inspector.

For example, the defect inspector 34 may request to perform re-inspection of all the work results that have been inspected, or the defect inspector 34 may request to perform re-inspection only for the result of the inspection completed among the work results inspected by the defect inspector 34. have.

This is to obtain high quality work data by having a sincere inspector re-inspection because the inspection quality of the defective inspector has already been determined to be inadequate.

Meanwhile, in the above description, steps S11 to S163 may be further divided into additional steps or may be combined into fewer steps, according to an embodiment of the present invention. In addition, some steps may be omitted as necessary, and the order between steps may be changed. In addition, even if other contents are omitted, the contents of FIG. 9 to be described later may also be applied to the inspection quality management method using the rejected operation results of FIGS. 1 to 8.

Hereinafter, a description will be given of an apparatus 200 for managing inspection quality using the rejected work result of a crowdsourcing-based project according to an embodiment of the present invention.

9 is a view for explaining an apparatus for managing inspection quality using rejected work results of a crowdsourcing-based project according to an embodiment of the present invention.

Referring to FIG. 9, an apparatus 200 (hereinafter, a quality control apparatus for inspection) that manages inspection quality using rejected work results of a crowdsourcing-based project includes a communication module 210, a memory 220, and a processor 230. ).

The communication module 210 transmits a crowdsourcing-based work for one project to a plurality of workers 32 to request a task to be performed, and receives a work result from the plurality of workers 32. Further, the work results received from the plurality of workers 32 are transmitted to the plurality of inspectors 34 to request inspection, and the inspection results are received from the plurality of inspectors 34.

The memory 220 stores a program for managing the quality of inspection using the rejected work result of the crowdsourcing-based project based on the data received from the communication module 210.

The processor 230 executes a program stored in the memory 220. As the processor 230 executes the program stored in the memory 220, it allocates the verification work result to each inspector at each period, and determines the inspector as a defective inspector according to the inspection result of the verification work result By giving, it is possible to manage the inspection quality of individual inspectors.

The inspection quality management apparatus 200 described with reference to FIG. 9 may be provided as a component of the above-described server.

The inspection quality management method using the rejected work result of the crowdsourcing-based project according to an embodiment of the present invention described above is implemented as a program (or application) to be combined with a computer as hardware and stored in a medium. I can.

The above-described program includes C, C++, JAVA, Ruby, which can be read by a processor (CPU) of the computer through the device interface of the computer, in order for the computer to read the program and execute the methods implemented as a program. It may include a code (Code) coded in a computer language such as machine language. Such code may include a functional code related to a function that defines necessary functions for executing the methods, and includes a control code related to an execution procedure necessary for the processor of the computer to execute the functions according to a predetermined procedure. can do. In addition, these codes may further include additional information required for the processor of the computer to execute the functions, or code related to a memory reference to which location (address address) of the internal or external memory of the computer should be referenced. have. In addition, when the processor of the computer needs to communicate with any other computer or server in the remote in order to execute the functions, the code uses the communication module of the computer to determine how It may further include a communication-related code for whether to communicate, what kind of information or media should be transmitted and received during communication.

The stored medium is not a medium that stores data for a short moment, such as a register, cache, memory, etc., but a medium that stores data semi-permanently and can be read by a device. Specifically, examples of the storage medium include, but are not limited to, ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, and the like. That is, the program may be stored in various recording media on various servers to which the computer can access, or on various recording media on the user's computer. In addition, the medium may be distributed over a computer system connected through a network, and computer-readable codes may be stored in a distributed manner.

The above description of the present invention is for illustrative purposes only, and those of ordinary skill in the art to which the present invention pertains can understand that it is possible to easily transform it into other specific forms without changing the technical spirit or essential features of the present invention. will be. Therefore, it should be understood that the embodiments described above are illustrative in all respects and not limiting. For example, each component described as a single type may be implemented in a distributed manner, and similarly, components described as being distributed may also be implemented in a combined form.

The scope of the present invention is indicated by the claims to be described later rather than the detailed description, and all changes or modified forms derived from the meaning and scope of the claims and their equivalent concepts should be interpreted as being included in the scope of the present invention. do.

10: Client
20: service provider
30: public
32: worker
34: inspector
200: Inspection quality control device
210: communication module
220: memory
230: processor

Claims (11)

As a method performed by a computer,
Allocating a plurality of tasks of a crowdsourcing-based project to a plurality of workers and requesting the task to be performed;
Receiving a plurality of job results from the plurality of workers;
Assigning the plurality of work results to a plurality of inspectors and requesting to perform inspection;
Receiving a plurality of inspection results for the plurality of work results from the plurality of inspectors;
Dividing the plurality of inspection results into inspection completion and rejection, and collecting rejected work results; And
Including the step of managing the inspection quality of the individual inspector using the returned work result,
The step of managing the inspection quality of the individual inspectors,
Assigning the result of the work rejected by another inspector (hereinafter, the result of the work for verification) to the inspector and requesting to perform the inspection,
Receiving an inspection result for the verification work result from the inspector; and
If the inspection result for the verification work result is not rejected, comprising the step of determining the inspection quality of the inspector as inappropriate,
The verification work result is assigned to the inspector together with the actual work results that have not been inspected by other inspectors,
If the result of the inspection for the actual work result is rejected, the rejected job is reassigned to the corresponding worker to request rework, and the inspection fee is not paid to the inspector,
If the result of the inspection on the result of the verification work is rejected, the rejected job is reassigned to a corresponding worker, without requesting to perform rework, and paying the inspection fee to the inspector,
Inspection quality management method using rejected work results of crowdsourcing-based projects. delete delete The method of claim 1,
The step of requesting to perform the inspection by allocating the verification work result to the inspector,
Repeatedly performed according to a predetermined period or period of a predetermined inspection quantity,
Inspection quality management method using rejected work results of crowdsourcing-based projects. The method of claim 4,
If the inspection result for the verification work result is not rejected, shortening the cycle in relation to the inspector,
Inspection quality management method using rejected work results of crowdsourcing-based projects. The method of claim 4,
If the inspection result for the verification work result is rejected, extending the period in relation to the inspector,
Inspection quality management method using rejected work results of crowdsourcing-based projects. The method of claim 1,
If the result of the inspection on the result of the verification work is not rejected, requesting the inspection by continuously assigning the result of the verification work to the inspector,
Inspection quality management method using rejected work results of crowdsourcing-based projects. The method of claim 1,
When the nonconformity ratio is greater than or equal to a predetermined value or the number of nonconformities reaches a predetermined number based on a result of determining the quality of the inspection by the inspector, determining the inspector as a defective inspector,
Inspection quality management method using rejected work results of crowdsourcing-based projects. The method of claim 8,
Restricting the defective inspector from participating in the inspection for a predetermined period of time, or giving a penalty for redeeming the inspection cost paid to the defective inspector,
Inspection quality management method using rejected work results of crowdsourcing-based projects. The method of claim 8,
Re-allocating the work result assigned to the defective inspector to another inspector to request re-examination,
Inspection quality management method using rejected work results of crowdsourcing-based projects. A computer program stored in a computer-readable recording medium to perform an inspection quality control method using the rejected work result of the crowdsourcing-based project of any one of claims 1, 4 to 10 in combination with a computer.

Images