KR102155751B1 - Method for deciding to authorize inspection by inspcetion test of crowdsourcing based project for artificial intelligence training data generation - Google Patents

Abstract

Provided is a method of granting an inspection authority through an inspection test of a crowdsourcing-based project for generating artificial intelligence learning data. The method includes the steps of: assigning a plurality of tasks of a crowdsourcing-based project to a plurality of workers and requesting to perform the task; receiving a plurality of work results from the plurality of workers; assigning the plurality of work results to a plurality of inspectors and requesting to perform inspection; receiving a plurality of inspection results of the plurality of work results from the plurality of inspectors as passing or rejecting the inspection; and granting or withdrawing the inspection authority based on the plurality of work results and the plurality of inspection results.

Description

{METHOD FOR DECIDING TO AUTHORIZE INSPECTION BY INSPCETION TEST OF CROWDSOURCING BASED PROJECT FOR ARTIFICIAL INTELLIGENCE TRAINING DATA GENERATION}

The present invention relates to a method for determining the authorization of inspection through an inspection test of a crowdsourcing-based project for generating artificial intelligence learning data.

Recently, more and more companies collect and process large amounts of data on a crowdsourcing basis that engages the general public in some process of corporate activities. In other words, by opening a project and allowing the general public, that is, workers to participate in the project, necessary information is collected through the work results completed by the workers.

And, in order to collect more reliable information, the company assigns the results of work completed by the worker to the inspector to perform the inspection work.

As the amount of data required by the project increases, the number of workers participating in the project increases. As a result, the number of work results that must be inspected increases, so the number of inspectors participating in the project must increase in proportion to the number of workers.

However, nowadays, in selecting the inspectors to participate in the project, the project manager directly checks the information of each inspector and decides whether to grant the opportunity to participate in the project, so it is time for the manager to select a large number of inspectors by himself. There is a problem that it is inefficient in terms of physical properties.

In addition, even if the work performance is excellent, the inspection performance ability is sometimes inferior. Therefore, selecting an inspector according to the subjective judgment of the manager has a problem that the ability of the inspector participating in the project cannot be convinced. The probability of this deteriorating situation will increase.

Unexamined Patent Publication No. 10-2014-0095956, 2014.08.04.

The problem to be solved by the present invention is to provide a method of granting inspection authority through an inspection test of a crowdsourcing-based project for generating artificial intelligence learning data.

However, the problem to be solved by the present invention is not limited to the problem as described above, and other problems may exist.

In an aspect of the present invention for solving the above-described problems, a method of granting inspection authority through an inspection test of a crowdsourcing-based project is a method of assigning a plurality of tasks of a crowdsourcing-based project to a plurality of workers and requesting the performance of the task. Step, receiving a plurality of work results from the plurality of workers, assigning the plurality of work results to a plurality of inspectors and requesting to perform inspection, a plurality of inspections of the plurality of work results from the plurality of inspectors Receiving a result as an inspection pass or rejection, and granting or recovering an inspection authority based on the plurality of work results and the plurality of inspection results, wherein the granting or retrieving the inspection authority may include: Based on the work result of, selecting a predetermined worker from among the plurality of workers as a subject to perform the inspection test (hereinafter, hereinafter, the inspection test subject), an inspector who will provide the inspection result for the inspection test among the plurality of inspectors ( Hereinafter, the step of selecting a first inspection test provider), based on the inspection result, the m first inspection test work results and n second inspection test work results that have been inspected by the first inspection test provider (m And n is a natural number), requesting to perform an inspection test by providing the m first inspection test work results and the n second inspection test work results to the inspection test subject, the inspection test The step of receiving the results of the m first inspection test work results and the inspection test results for the n second inspection test work results from the subject, and whether to grant the inspection authority to the inspection test subject according to the inspection test result Including the step of determining, wherein the inspection result of the inspection test subject for the m first inspection test work results and the n second inspection test work results is an inspection result of the first inspection test provider If it matches all, to the subject of the above inspection An inspection authority is granted, and the m first inspection test work result is a work result determined by the first inspection test provider as the inspection result passed, and the n second inspection test work results are the first The inspection result by the inspection test provider may be the result of the work set as rejected.

In some embodiments of the present invention, the step of selecting a predetermined worker from among the plurality of workers as a subject for the inspection test includes performing a reference number of tasks among the plurality of workers, and the rejection rate of the task result of the reference number is It may be to select an operator with a standard percentage or less as an inspection test subject.

In some embodiments of the present invention, the inspection test result of the inspection test subject for the m first inspection test work results and the n second inspection test work results is the inspection result of the first inspection test provider If one or more of the items do not match, the first inspection test provider may be selected as the inspection test subject without granting the inspection authority to the inspection test subject.

In some embodiments of the present invention, the method comprises, when the first inspection test provider is selected as the inspection test subject, an inspector who will provide inspection results for inspection among a plurality of inspectors excluding the first inspection test provider (hereinafter , Selecting a second inspection test provider), extracting m third inspection test work results and n fourth inspection test work results inspected by the second inspection test provider based on the inspection results Step, requesting to perform an inspection test by providing the m third inspection test work results and the n fourth inspection test work results to the first inspection test provider, the m from the first inspection test provider Receiving the third inspection test result of the proposal and the inspection test result for the n fourth inspection test results, and determining whether to withdraw the inspection authority of the first inspection test provider according to the inspection test result Including the step of, wherein the inspection results of the first inspection test provider for the m third inspection test work results and the n fourth inspection test work results are the inspection results of the second inspection test provider If all match with, the first inspection test provider maintains the inspection authority, and the m cases of the third inspection test work result is the work result determined by the second inspection test provider as the inspection result passed the inspection, and the The results of the n fourth inspection tests may be job results in which the inspection result is determined to be rejected by the second inspection test provider.

In some embodiments of the present invention, the inspection test results of the first inspection test provider for the m third inspection test work results and the n fourth inspection test work results are determined by the second inspection test provider. If one or more discrepancies with the inspection result, the inspection authority of the first inspection test provider may be recovered, and the second inspection test provider may be selected as the inspection test subject.

In some embodiments of the present invention, when selecting an inspector who will provide an inspection result for an inspection test, it may be selected from among inspectors who have not previously provided an inspection result for an inspection test.

In some embodiments of the present invention, m and n may be determined according to the difficulty of the project.

In some embodiments of the present invention, the step of extracting m first inspection test work results and n second inspection test work results inspected by the first inspection test provider may include: It may be to extract by excluding the work result of the subject to be inspected.

In some embodiments of the present invention, the step of extracting m first inspection test work results and n second inspection test work results inspected by the first inspection test provider may include: Based on the inspection result initially inspected by, the m first inspection test work results and the n second inspection test work results may be extracted.

A computer program according to another aspect of the present invention for solving the above-described problems is combined with a computer as hardware to execute a method of granting an inspection authority through an inspection test of the crowdsourcing-based project, and stored in a computer-readable recording medium. do.

Other specific details of the present invention are included in the detailed description and drawings.

According to the present invention described above, by providing the operator with the work result already inspected by the inspector to perform the inspection test, and granting the inspection authority to the operator according to the operator's inspection test result, the operator whose inspection level has been verified without the intervention of the manager. Can be quickly selected as an inspector.

In addition, by performing the inspection test not only for the operator but also for inspectors whose inspection level is suspicious, and recovering the inspection authority of the inspector according to the inspection test result of the inspector, it is possible to automatically convert an inspector with a lower inspection level into an operator without intervention of the manager I can. Accordingly, it is possible to maintain a high level of inspection quality of the project.

The effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the following description.

1 is a conceptual diagram of a crowdsourcing service according to an embodiment of the present invention.
2 is a flowchart illustrating a crowdsourcing-based project progress process according to an embodiment of the present invention.
3 is a flow chart of a method for determining an authorization authorization for a crowdsourcing-based project through an inspection test according to an embodiment of the present invention.
4 is a flowchart of a method of granting an inspection authority when the subject of an inspection test is an operator according to an embodiment of the present invention.
5 is a flowchart of a method of recovering an inspection authority when the subject of an inspection test is an inspector according to an embodiment of the present invention.
6 and 7 are exemplary diagrams for explaining extracting a work result for an inspection test according to an embodiment of the present invention.
8 is an exemplary view for explaining a case of granting an inspection authority or maintaining an inspection authority according to an embodiment of the present invention.
9 is an exemplary diagram for explaining a case in which an inspection authority is not granted or an inspection authority is recovered according to an embodiment of the present invention.
10 is a flowchart illustrating a process in which an inspection test is repeatedly performed according to an embodiment of the present invention.
11 is a diagram for explaining an apparatus for determining granting of an inspection authority through an inspection test of a crowdsourcing-based project according to an embodiment of the present invention.

Advantages and features of the present invention, and a method of achieving them will become apparent with reference to the embodiments described below in detail together with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but may be implemented in a variety of different forms, only the present embodiments are intended to complete the disclosure of the present invention, It is provided to fully inform the technician of the scope of the present invention, and the present invention is only defined by the scope of the claims.

The terms used in the present specification are for describing exemplary embodiments and are not intended to limit the present invention. In this specification, the singular form also includes the plural form unless specifically stated in the phrase. As used in the specification, “comprises” and/or “comprising” do not exclude the presence or addition of one or more other elements other than the mentioned elements. Throughout the specification, the same reference numerals refer to the same elements, and “and/or” includes each and all combinations of one or more of the mentioned elements. Although "first", "second", and the like are used to describe various elements, it goes without saying that these elements are not limited by these terms. These terms are only used to distinguish one component from another component. Therefore, it goes without saying that the first component mentioned below may be the second component within the technical idea of the present invention.

Unless otherwise defined, all terms (including technical and scientific terms) used in the present specification may be used as meanings that can be commonly understood by those of ordinary skill in the art to which the present invention belongs. In addition, terms defined in a commonly used dictionary are not interpreted ideally or excessively unless explicitly defined specifically.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a conceptual diagram of a crowdsourcing service according to an embodiment of the present invention.

Referring to FIG. 1, a crowdsourcing service is performed by being composed of a client 10, a service provider 20, and a public 30.

The sponsor 10 refers to a company or individual who requests a crowdsourcing-based project (hereinafter, referred to as a project).

The requester 10 requests a project for the purpose of collecting source data or annotating data for generating artificial intelligence learning data. The data generated through the project can be used as learning data for arbitrary machine learning such as supervised learning, unsupervised learning, and reinforcement learning. Collection of source data refers to collecting raw data such as recorded voice collection and photo collection. Data annotation refers to inputting relevant annotation data into source data such as text, photos, and videos. For example, the data annotation may include finding an entity in a given fingerprint or finding a similar sentence, but is not limited thereto. Meanwhile, the types of the above-described projects are only one embodiment, and various projects may be handled in the present invention according to the design of the client.

The service provider 20 refers to a company that provides crowdsourcing services.

When the service provider 20 receives a request for a project for a product or service from the client 10, the service provider 20 allocates a task for the project to the general public 30 and receives the work result from the public 30. Thereafter, the final product extracted based on the work result is provided to the client 10.

At this time, the service provider 20 provides a crowdsourcing service to the client 10 and the public 30 through a crowdsourcing platform (hereinafter, the platform). That is, when the service provider 20 receives a request for a project from the client 10, the service provider 20 opens the project on the platform. Thereafter, when the work result for the open project is received from the public 30, the project is terminated on the platform, and the final product may be extracted and provided to the client 10.

The public 30 refers to the general public participating in the project open on the platform. Here, the public 30 may participate in a project opened on the platform through an application or website provided by the service provider 20.

The public 30 is composed of a worker 32 and an inspector 34.

The worker 32 decides to participate in a specific project among a plurality of projects open on the platform. Thereafter, the operator 32 performs an operation such as collection of source data or data annotation, and transmits it to the platform.

The inspector 34 decides to participate in a specific project among a plurality of projects open on the platform. Thereafter, the inspector 34 inspects the result of the work performed by the operator 32. As a result of performing the inspection, the inspector 34 may perform inspection pass processing or rejection processing, and input a rejection reason during rejection processing. In the case of passing the inspection, it is not necessary to rework and re-examination due to this, so passing inspection has the same meaning as completion of inspection.

2 is a flowchart illustrating a crowdsourcing-based project progress process according to an embodiment of the present invention.

First, the requester 10 requests one or more projects to the service provider 20 (S11).

Thereafter, the service provider 20 opens the requested project on the platform (S12). At this time, the service provider 20 may determine the grade in consideration of the difficulty of the project before opening the project. That is, depending on the difficulty level, it is possible to determine which level or higher public 30 to expose the project. Accordingly, it is possible to increase the reliability of the work result of the project.

Thereafter, the service provider 20 requests the work by assigning the work to the workers 32 of the corresponding level or higher according to the level of the project (S13).

Thereafter, the worker 32 performs the assigned task (S14). In this case, the worker 32 may input the reason for the inability to work without performing the work for a work in which the work itself is impossible for some reason.

Thereafter, the service provider 20 receives the work result from the worker 32 (S15), and assigns the work result to the inspector 34 to request the inspection (S16).

Thereafter, the inspector 34 performs the assigned inspection (S17). At this time, the inspector 34 decides to pass the inspection when it is determined that the work has been properly performed, and if it is determined that the inspection work is wrong, the inspector 34 rejects it. During rejection processing, the inspector 34 inputs the rejection reason for what reason the job was judged to be wrong.

Thereafter, the service provider 20 receives the inspection result from the inspector 34 (S18).

If the inspection result passes the inspection, the service provider 20 uses the work result as valid data, and based on this, extracts the final product at the end of the project.

When the inspection result is rejection processing, the service provider 20 may internally perform the inspection again, or assign the work to the worker 32 again to perform the rework. When reworking, re-examination by an inspector is required.

Thereafter, when the project period ends or sufficient valid data is secured, the service provider 20 terminates the project (S19), calculates a final result based on the secured valid data, and provides it to the client 10 ( S20).

At this time, before the end of the project, the service provider 20 evaluates the performance results of the worker 32 and the inspector 34, calculates the work cost and the inspection cost according to the evaluation, to the worker 32 and the inspector 34. give.

In FIGS. 1 and 2, it is simply expressed as a requester 10, a service provider 20, an operator 32, and an inspector 34, but these are smartphones, tablets, PDAs, laptops, desktops, operated by each participant. It means a computer device such as a server or a telecommunication device.

In general, since the number of workers participating in a project is large due to a large difference compared to the number of inspectors, there is a difficulty due to the lack of inspectors such as slowing the project speed and lowering the quality of inspection in operating the project.

Therefore, the present invention requests the operator who is judged to have excellent work level among the workers participating in the project to perform the inspection test, and the inspection authority so that the worker whose inspection level has been verified by passing the inspection test can perform the inspection in the project. By giving, it is possible to increase the number of insufficient inspectors.

At the same time, the present invention performs an inspection test on the inspectors participating in the project, so that the inspector who does not pass the inspection test recovers the inspection authority so that the inspection cannot be performed in the project, so that only inspectors whose inspection level is verified To be able to perform. At this time, the inspector whose inspection authority has been withdrawn may be converted to a worker to perform work on the project, and may be granted inspection authority again by performing an inspection test at a time when a predetermined condition is satisfied.

Hereinafter, referring to FIGS. 3 to 9, a description will be given of selecting an operator or an inspector as a subject for inspection and performing an inspection test, granting an inspection authority to the operator according to the inspection test result, and recovering the inspection authority to the inspector. Do it.

3 is a flow chart of a method for determining an authorization authorization for a crowdsourcing-based project through an inspection test according to an embodiment of the present invention. 4 is a flowchart of a method of granting an inspection authority when the subject of an inspection test is an operator according to an embodiment of the present invention. 5 is a flowchart of a method of recovering an inspection authority when the subject of an inspection test is an inspector according to an embodiment of the present invention. 6 and 7 are exemplary diagrams for explaining extracting a work result for an inspection test according to an embodiment of the present invention. 8 is an exemplary view for explaining a case of granting an inspection authority or maintaining an inspection authority according to an embodiment of the present invention. 9 is an exemplary diagram for explaining a case in which an inspection authority is not granted or an inspection authority is recovered according to an embodiment of the present invention.

Meanwhile, the steps illustrated in FIGS. 3 to 5 may be understood as being performed by a platform server (hereinafter, referred to as a server) operated by the service provider 20, but are not limited thereto.

In addition, a plurality of workers 32 or a plurality of inspectors 34 perform work using a predetermined terminal device. The terminal device of the operator 32 or the inspector 34 may be a computer device or a telecommunication device such as a smartphone, tablet, PDA, laptop, desktop, etc., but is not limited thereto.

First, referring to FIG. 3, the server assigns a plurality of tasks of a crowdsourcing-based project to a plurality of workers 32 and requests the task to be performed (S100), and receives a plurality of task results from the plurality of workers 32. (S200), a plurality of work results are assigned to a plurality of inspectors 34 to request inspection performance (S300), and a plurality of inspection results for a plurality of work results are input from the plurality of inspectors 34 (S400) .

Here, the plurality of workers 32 and the plurality of inspectors 34 refer to all workers and inspectors participating in the same project.

Since the steps S110 to S140 overlap with the steps S13 to S18 described above with reference to FIG. 2, a detailed description will be omitted.

Subsequently, the server grants or recovers the inspection authority based on the plurality of work results and the plurality of inspection results (S500).

First, with reference to FIG. 4, a process of performing an inspection test for an operator selected as an inspection test subject will be described.

The server selects a predetermined worker 32 among the plurality of workers 32 as a subject to perform the inspection test (hereinafter, hereinafter, the inspection test subject) based on the results of the plurality of tasks (S510).

That is, the server performs a reference number of tasks among the plurality of workers 32 participating in the project, and selects a worker 32 whose rejection rate of the task result of the reference number is less than or equal to the standard percentage as the subject of the inspection test.

For example, if the reference count is 10 and the reference percentage is 20%, then workers A, B, C, D, and E each performed 10 or more tasks, each with a rejection rate of 10%, 15%, and 30%. When calculated as 50% and 40%, the server can select only workers A and B as the subject of the inspection test.

Subsequently, the server selects an inspector (hereinafter, referred to as a first inspection test provider) to provide an inspection result for an inspection test from among the plurality of inspectors (S520).

That is, the server selects an inspector who will provide the inspection result for inspection to the operator 32 selected as the inspection test subject.

At this time, the first inspection test provider should be selected from among inspectors who have not previously provided inspection results for inspection tests. In other words, an inspector who has not previously provided inspection results for inspection tests is selected as the first inspection test provider, and the work results inspected by the selected first inspection test provider are provided as inspection results for inspection tests to the subject of inspection tests. Request to perform the inspection test.

Since the first inspection test provider has not previously provided inspection results for inspection tests, it is necessary to determine whether or not the accuracy of the inspection results for inspection tests provided by the first inspection test provider is based on the inspection test results of the inspection test subjects. I can. As a result, it is possible to perform an inspection test on the operator 32, which is the subject of the inspection test, and at the same time, verify the inspector 34, which is the first inspection test provider.

Subsequently, the server extracts m first inspection test work results and n second inspection test work results that have been inspected by the first inspection test provider based on the inspection result (S530).

At this time, m and n are natural numbers and are determined according to the difficulty of the project.

As an example, in the case of focusing on the speed of the project, m and n may be determined as small numbers as the difficulty of the project increases. Since the inspection test is not an actual task or performing inspection, if the inspection subject spends a lot of time performing the inspection test, it will affect the speed of the project and the project will proceed slowly.

In another embodiment, in the case of focusing on the quality of the project, m and n may be determined as a larger number as the difficulty of the project increases. If a small number of work tests for inspection tests are provided so that the inspection level cannot be judged, the quality of the project is degraded, and the cost and labor are wasted because a worker or an inspector with a low inspection level performs the inspection.

In addition, the m results of the first inspection test are the result of the inspection determined by the first inspection test provider as the inspection pass, and the results of the second inspection test of n cases are the inspection results by the first inspection test provider. This is the result of the work determined as a rejection.

Referring to FIG. 6, the inspection result of the first inspection test provider includes a plurality of job results that have passed the inspection and a plurality of rejected job results. At this time, if m is 3 and n is 2, the server extracts the three work results for which the inspection results pass the inspection as the work results for the first inspection test, and the two work results whose inspection results are rejected are the second inspection. Extracted as a test result.

In addition, in extracting the m first inspection test work results and n second inspection test work results that have been reviewed by the first inspection test provider, the server extracts the results of the inspection test subject by excluding the work results.

The inspector 34 inspects each operation result input from the plurality of workers 32 as passing or rejecting the inspection. When the inspector 34, the first inspection test provider, inspects the work result of the operator 32 as the subject of the inspection test as a rejection, the operator 32 as the subject of the inspection test has received the inspection result for the work result in advance. Therefore, when the work result that the person has performed is provided as the work result for the inspection test, the accuracy in judging the inspection level of the subject of the inspection test decreases. Accordingly, the server must extract the work result for the inspection test by excluding the work result of the subject of the inspection test among the inspection results of the first inspection test provider.

Referring to FIG. 7, when m is 3 and n is 2, the server extracts three work results in which the inspection result passed the inspection as the work result for the first inspection test, and the two work results in which the inspection result is rejected Is extracted as the work result for the second inspection test. At this time, since the two work results, which are rejected by the first inputted inspection results, are the work results of the subject of the inspection test, the server extracts the two work results, which are rejected by the inspection results input later, as the work results for the second inspection test.

6 and 7 show that m first inspection test work results and n second inspection test work results are sequentially extracted according to the input order, but the server randomly receives m It is also possible to extract the work results for the first inspection test of the case and the work results for the second inspection test of n cases.

Further, the server extracts m first inspection test work results and n second inspection test work results based on the inspection results initially inspected by the first inspection test provider.

Since the initial and re-inspectors may be different for the same work depending on the delay of the project or the inspection performance of the inspector 32, the inspection is based on the case initially inspected by the inspector 34, the first inspection test provider. The results of the test work should be extracted.

Referring to FIG. 4 again, the server requests the inspection test by providing m first inspection test results and n second inspection test results to the inspection test subject (S540), and m from the inspection test subject. The first inspection test result of the proposal and the inspection test result for the n second inspection test work results are input (S550), and according to the inspection test result, it is determined whether or not to grant the inspection authority to the inspection test subject (S560). ).

If the inspection results of the inspection test subject for m first inspection test results and n second inspection test results match all of the inspection results of the first inspection test provider, the server inspects the inspection test subjects. Grant authority.

Referring to FIG. 8, when m is 3 and n is 2, the subject of the inspection test performed inspection by passing all inspection results for the first inspection test work results of three cases, and the work results for the inspection number 2 test two cases. All were reviewed as companions.

Since the inspection test result of the inspection test subject is entirely identical to the inspection result of the first inspection test provider, the server grants inspection authority to the operator 32 as the inspection test subject.

The operator 32 who has passed the inspection test and is granted inspection authority has the inspection level verified and plays the role of an inspector in the project. In addition, the operator 32 who has been granted inspection authority afterwards performs the inspection as an inspector, and is selected as an inspection test provider to perform inspection again. According to the inspection test result, the inspection authority is recovered and the operator ( 32).

If the inspection test results of the inspection test subject for m first inspection test results and n second inspection test results are inconsistent with the inspection results of the first inspection test provider, the server sends to the inspection test subjects. Does not grant inspection authority.

Referring to FIG. 9, when m is 3 and n is 2, the subject of the inspection test performed inspection by passing all inspection results for the first inspection test results of three cases, and one of the work results for inspection 2 inspection tests in two cases. The work result for the second inspection test of the case was reviewed as a rejection, and the work result for the second inspection test was reviewed by passing the inspection.

Since the inspection test result of the inspection test subject is inconsistent with the inspection result of the second inspection test provider and one case, the server does not grant the inspection authority to the inspection test subject.

The worker 32 who has not passed the inspection test and has not been granted inspection authority continues to perform the role as a worker in the project.

Hereinafter, a process of performing an inspection test for an inspector will be described with reference to FIG. 5.

If the inspection test results of the inspection test subject for m first inspection test results and n second inspection test results are inconsistent with the inspection results of the first inspection test provider, the server performs the first inspection test. The provider is selected as the subject of the inspection test (S561).

That is, if the inspection test result of the inspection test subject and the inspection result of the first inspection test provider do not match at least one, the server determines that the operator 32 as the inspection test subject is not capable of performing the inspection, as described above. The inspection authority is not granted to the worker 32. In addition, the server is suspicious of the level of inspection of the inspector 34, which is the first inspection test provider, so that the first inspection test provider is selected as the inspection test subject, and the inspection test is performed on the first inspection test provider.

The server selects an inspector (hereinafter, referred to as a second inspection test provider) to provide an inspection result for an inspection test among a plurality of inspectors other than the first inspection test provider (S562).

In other words, since the first inspection test provider, who is the inspector 34, has been selected as the subject of the inspection test, an inspector who will provide inspection results for inspection tests excluding the first inspection test provider among the plurality of inspectors 34 participating in the project ( 34) is selected as the second inspection test provider.

At this time, the second inspection test provider should be selected from among inspectors 34 who have not previously provided inspection results for inspection tests. An inspector 34 who has not previously provided inspection results for inspection tests is selected as the second inspection test provider, and the work results inspected by the selected second inspection test provider are provided as inspection results for inspection tests, and inspection tests Request the first inspection test provider to perform the inspection test.

Since the second inspection test provider has not previously provided inspection results for inspection tests, the accuracy of the inspection results for inspection tests provided by the second inspection test provider is determined according to the inspection results of the first inspection test provider. I can judge. As a result, it is possible to perform an inspection test on the first inspection test provider, which is the subject of the inspection test, and at the same time, verify the inspector 34, which is the second inspection test provider.

Based on the inspection result, the server extracts m third inspection test work results and n fourth inspection test work results inspected by the second inspection test provider (S563).

At this time, m and n are natural numbers and are determined according to the difficulty of the project. It is determined according to the difficulty of the project as described above.

In addition, the m results of the third inspection test are the result of the inspection determined by the second inspection test provider as the inspection pass, and the results of the fourth inspection test of n cases are the inspection results by the second inspection test provider. This is the result of the work determined as a rejection.

Referring to FIG. 6, the inspection result of the second inspection test provider includes a plurality of job results that have passed the inspection and a plurality of rejected job results. At this time, if m is 3 and n is 2, the server extracts the 3 work results for which the inspection results pass the inspection as the work results for the 3rd inspection test, and the 2 work results whose inspection results are rejected are the 4th inspection. Extracted as a test result.

In Fig. 6, it is shown that m third inspection test results and n fourth inspection test results are sequentially extracted according to the input order, but the server randomly receives m third inspection results regardless of the input order. It is also possible to extract the work results for the inspection test and the work results for the fourth inspection test of n cases.

In addition, the server extracts m third inspection test results and n fourth inspection results based on the inspection results initially inspected by the second inspection test provider.

Since the initial and re-inspectors may be different for the same task due to delays in the project or the inspection performance of the inspector (32), the inspection test is based on the case initially inspected by the inspector, the second inspection test provider. You need to extract the results.

Referring to FIG. 5 again, the server requests the first inspection test provider to perform the inspection test by providing m cases of the third inspection test work results and the n fourth inspection test work results (S564), and the first inspection Receiving the inspection test results for m third inspection test work results and n fourth inspection test work results from the test provider (S565), and whether to recover the inspection authority of the first inspection test provider according to the inspection test results Whether or not it is determined (S566).

When the inspection results of the first inspection test provider (the inspection test subject) for the m third inspection test work results and the n fourth inspection test work results match all of the inspection results of the second inspection test provider, The server maintains the inspection authority of the first inspection test provider.

Referring to FIG. 8, when m is 3 and n is 2, the first inspection test provider, who is the subject of the inspection test, has inspected all of the results of three third inspection tests by passing inspection, and two fourth inspections All of the test work results were rejected.

Since the verification test results of the first verification test provider are all identical to the verification results of the second verification test provider, the server maintains the verification authority of the first verification test provider as the subject of the verification test.

The inspector (the first inspection test provider) who has passed the inspection test and maintained the inspection authority has the inspection level verified and continues to play the role of the inspector in the project.

When the inspection results of the first inspection test provider (the subject of the inspection test) for m third inspection test results and n fourth inspection test results are inconsistent with the inspection results of the second inspection test provider , The server recovers the inspection authority of the first inspection test provider.

Referring to FIG. 9, when m is 3 and n is 2, the first inspection test provider, who is the subject of the inspection test, has inspected all the results of the third inspection test by passing the inspection, and the fourth inspection has two cases. Among the test work results, one fourth inspection test work result was reviewed as a rejection, and one fourth inspection test work result was reviewed by passing the inspection.

Since the inspection result of the first inspection test provider is inconsistent with the inspection result of the second inspection test provider for one case, the server recovers the inspection authority of the first inspection test provider, which is the subject of the inspection test.

The inspector (the first inspection test provider) whose inspection authority has been withdrawn due to failing to pass the inspection test is converted to the worker 32 in the project and plays the role of the worker 32.

Then, the first inspection test provider, who has been converted to the worker 32 afterwards, performs a task as a worker, performs a reference number of tasks as described above, and determines the condition that the rejection rate of the task result of the reference number is less than or equal to the reference percentage. When it is satisfied, a request to perform the inspection test can be received from the server. When the first inspection test provider who has been converted to the worker 32 passes the inspection test, he or she is granted inspection authority again and can perform the inspection as the inspector 34 in the project.

In addition, the inspection results of the first inspection test provider (the inspection test subject) for the m third inspection test work results and the n fourth inspection test results are inconsistent with the inspection results of the second inspection test provider. If so, the server selects the second inspection test provider as the inspection test target.

That is, if the inspection result of the first inspection test provider and the inspection result of the second inspection test provider do not match at least one, the server determines that the inspection level of the inspector 34, which is the first inspection test provider, is falling as described above. The inspection authority granted to the first inspection test provider is recovered. In addition, the server is suspicious of the level of inspection of the second inspection test provider, the inspector 34, selects the second inspection test provider as an inspection test subject, and performs an inspection test on the second inspection test provider.

According to an embodiment of the present invention, the server may pay a predetermined inspection cost for a work result for an inspection test whose inspection result is determined to pass the inspection to the inspector 34 selected as an inspection test provider.

The inspector 34 selected as the inspection test provider is selected as the inspection test subject according to the inspection test result of the inspection test subject, regardless of will, and undergoes the inspection test. In other words, despite being a sincere inspector, the inspection level is suspected and the inspection test is performed.

Therefore, when the inspector 34 selected as the inspection test target passes the inspection test, the server determines that the inspector 34 has performed cross inspection for the work result for the inspection test, which is set as the inspection pass. Compensation can be provided, such as paying a predetermined inspection fee. Here, the reason why the predetermined inspection cost is paid only for the inspection test work result determined as the inspection pass is because the inspection cost is not paid for the work result whose inspection result is rejected.

Hereinafter, with reference to FIG. 10, it will be described that the inspection test for the operator and the inspection test for the inspector are repeated to increase the number of inspectors and maintain the inspection quality of inspectors during the project.

10 is a flowchart illustrating a process in which an inspection test is repeatedly performed according to an embodiment of the present invention.

The server determines that the worker A satisfies the inspection test condition among the plurality of workers 32, selects the worker A as the subject of the inspection test, and requests the inspector A to perform the inspection test.

Worker A accepts the request to perform the inspection test if he wants to perform the inspection in the project, and rejects the request to perform the inspection test if he wants to continue to perform the work.

When the worker A accepts the request to perform the inspection test, the server provides the job result of the inspection by the inspector A selected as the inspection test provider to the operator A as the inspection test work result.

When the inspection results of worker A match all of the inspection results of inspector A, the server grants inspection authority to worker A, and worker A switches to inspector. In this way, the server can automatically increase the number of inspectors whose inspection level has been verified through the inspection test to the corresponding project.

In addition, if the results of the inspection test by operator A match all of the inspection results of inspector A, the server determines that inspector A is properly performing the inspection, and can even complete the verification for inspector A.

On the other hand, if the inspection result of worker A is inconsistent with the inspection result of inspector A, the server determines that the inspection level of worker A is lower and does not grant the inspection authority to worker A. As a result, the server can automatically deprive the inspection opportunity for workers whose inspection level is low through the inspection test.

In addition, if the inspection result of operator A is inconsistent with the inspection result of inspector A at least, the server suspects the inspection level of inspector A and selects inspector A as the subject of inspection.

The server provides the inspection results to inspector A by inspector B selected as the inspection test provider as the inspection test results.

If inspector A's inspection results match all of inspector B's inspection results, the server determines that inspector A is performing the inspection correctly, and maintains inspector A's inspection authority.

At this time, the server pays a predetermined inspection fee to the inspector A for the inspection test result of which the correct answer is determined as the job pass. This is because, as Inspector A conducts an inspection test on the work result inspected by Inspector B, it is recognized that the work result has been cross-inspected.

In addition, if the inspection results of inspector A match all of the inspection results of inspector B, the server judges that inspector B is properly performing the inspection and can even complete the verification for inspector B.

On the other hand, if the inspection result of inspector A is inconsistent with the inspection result of inspector B at least, the server determines that inspector A's inspection level is lower and retrieves inspector A's inspection authority.

Inspector A, whose inspection authority has been withdrawn, performs the work as a worker in the project, and then performs the inspection test when the inspection test conditions are satisfied.

In addition, if the inspection result of inspector A is inconsistent with the inspection result of inspector B at least, the server suspects inspector B's inspection level and selects inspector B as the subject of inspection.

The server provides the inspection results to inspector B by inspector C selected as the inspection test provider as the inspection test results.

If inspector B's inspection results match all of inspector C's inspection results, the server determines that inspector B is performing the inspection correctly, and maintains inspector B's inspection authority.

At this time, the server pays a predetermined inspection fee to the inspector B for the inspection test result of which the correct answer is determined as the job pass. This is because, as Inspector B performs an inspection test on the work result inspected by Inspector C, it is recognized that the work result has been cross-inspected.

In addition, if the inspection results of inspector B match all of the inspection results of inspector C, the server determines that inspector C is properly performing the inspection, and can even complete the verification for inspector C.

On the other hand, if the inspection result of inspector B is inconsistent with the inspection result of inspector C at least, the server determines that inspector B's inspection level is low and retrieves inspector B's inspection authority.

Inspector B, whose inspection authority has been withdrawn, performs the work as a worker in the project, and then performs the inspection test when the inspection test conditions are satisfied.

In addition, if the inspection result of inspector B is inconsistent with the inspection result of inspector C, the server suspects the inspection level of inspector C and selects inspector C as the subject of inspection.

As described above, the operator who passed the inspection test through the repetitive inspection test is converted to an inspector and automatically increases the inspectors whose inspection level has been verified, and the inspector who does not pass the inspection test is converted to an operator to verify the inspection level. By automatically reducing the number of unsuccessful inspectors, the inspection quality of inspectors participating in the project can be maintained at a high level.

Meanwhile, in the above description, steps S11 to S566 may be further divided into additional steps or may be combined into fewer steps, according to an embodiment of the present invention. In addition, some steps may be omitted as necessary, and the order between steps may be changed. In addition, even if other contents are omitted, the contents of FIG. 11 to be described later may also be applied to the method of granting inspection authority through the inspection test of the crowdsourcing-based project of Figs. 1 to 10.

Hereinafter, with reference to FIG. 11, a description will be given of the apparatus 600 for determining the authorization of the inspection authority through the inspection test of the crowdsourcing-based project according to an embodiment of the present invention.

11 is a diagram for explaining an apparatus for determining granting of an inspection authority through an inspection test of a crowdsourcing-based project according to an embodiment of the present invention.

Referring to FIG. 11, the device 600 (hereinafter, the device for granting permission to be inspected) for determining the authorization of the inspection through the inspection test of a crowdsourcing-based project is a communication module 610, a memory 620, and a processor 630. Includes.

The communication module 610 transmits a crowdsourcing-based work for one project to a plurality of workers 32 to request a task to be performed, and receives work results from the plurality of workers 32. Further, the work results received from the plurality of workers 32 are transmitted to the plurality of inspectors 34 to request inspection, and the inspection results are received from the plurality of inspectors 34.

In addition, the communication module 610 transmits the work result for the inspection test to the operator 32 or the inspector 34, which is the subject of the inspection test, and requests the inspection test from the operator 32 or the inspector 34. Receive the inspection result for the work result.

The memory 620 stores a program for determining the number of or granting of the inspection authority through the inspection test based on the data received from the communication module 610.

The processor 630 executes a program stored in the memory 620. As the processor 630 executes the program stored in the memory 620, among the work results already inspected by the inspector 34, which is an inspection test provider, the inspection results are m inspection test results and inspections that have passed the inspection. The inspection test is performed by extracting the results of n inspection tests whose results are rejected, and providing m inspection test work results and n inspection test work results to the operator (32) or inspector (34) who are subject to inspection test And, according to the result of the inspection test, the inspection authority is granted to the operator 32, which is the subject of the inspection test, or the inspection authority of the inspector 34 is recovered.

The apparatus 600 for granting inspection authority described with reference to FIG. 11 may be provided as a component of the server described above.

In the above-described method of granting review authority through an inspection test of a crowdsourcing-based project according to an embodiment of the present invention described above, the method may be implemented as a program (or application) to be executed in combination with a computer that is hardware and stored in a medium. have.

The above-described program includes C, C++, JAVA, Ruby, which can be read by a processor (CPU) of the computer through the device interface of the computer, in order for the computer to read the program and execute the methods implemented as a program. It may include a code (Code) coded in a computer language such as machine language. Such code may include a functional code related to a function that defines necessary functions for executing the methods, and includes a control code related to an execution procedure necessary for the processor of the computer to execute the functions according to a predetermined procedure. can do. In addition, these codes may further include additional information required for the processor of the computer to execute the functions, or code related to a memory reference to which location (address address) of the internal or external memory of the computer should be referenced. have. In addition, when the processor of the computer needs to communicate with any other computer or server in the remote in order to execute the functions, the code uses the communication module of the computer to determine how It may further include a communication-related code for whether to communicate, what kind of information or media should be transmitted and received during communication.

The stored medium is not a medium that stores data for a short moment, such as a register, cache, memory, etc., but a medium that stores data semi-permanently and can be read by a device. Specifically, examples of the storage medium include, but are not limited to, ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, and the like. That is, the program may be stored in various recording media on various servers to which the computer can access, or on various recording media on the user's computer. In addition, the medium may be distributed over a computer system connected through a network, and computer-readable codes may be stored in a distributed manner.

The above description of the present invention is for illustrative purposes only, and those of ordinary skill in the art to which the present invention pertains can understand that it is possible to easily transform it into other specific forms without changing the technical spirit or essential features of the present invention. will be. Therefore, it should be understood that the embodiments described above are illustrative in all respects and not limiting. For example, each component described as a single type may be implemented in a distributed manner, and similarly, components described as being distributed may also be implemented in a combined form.

The scope of the present invention is indicated by the claims to be described later rather than the detailed description, and all changes or modified forms derived from the meaning and scope of the claims and their equivalent concepts should be interpreted as being included in the scope of the present invention. do.

10: Client
20: service provider
30: public
32: worker
34: inspector
600: Approved authorization decision device
610: communication module
620: memory
630: processor

Claims (10)

As a method performed by a computer,
Allocating a plurality of tasks of a crowdsourcing-based project to a plurality of workers and requesting the task to be performed;
Receiving a plurality of job results from the plurality of workers;
Assigning the plurality of work results to a plurality of inspectors and requesting to perform inspection;
Receiving a plurality of inspection results for the plurality of work results from the plurality of inspectors as passing or rejecting the inspection; And
And granting or recovering an inspection authority based on the plurality of work results and the plurality of inspection results,
The step of granting or recovering the inspection authority,
Selecting a predetermined worker from among the plurality of workers as a subject to perform an inspection test (hereinafter, an inspection test subject) based on the plurality of work results;
Selecting an inspector (hereinafter, referred to as a first inspection test provider) to provide an inspection result for an inspection test, from among the plurality of inspectors;
Extracting m first inspection test work results and n second inspection test work results (m and n are natural numbers) inspected by the first inspection test provider based on the inspection results;
Requesting to perform an inspection test by providing the m first inspection test work results and the n second inspection test work results to the inspection test subject;
Receiving an inspection test result of the m first inspection test work results and the n second inspection test work results from the inspection test subject; And
And determining whether to grant an inspection authority to the inspection test subject according to the inspection test result,
The m first inspection test work results are work results determined by the first inspection test provider as the inspection result passed, and the n second inspection test work results are reviewed by the first inspection test provider. The result is the result of the work determined as rejection,
If the inspection results of the inspection subject for the m first inspection test work results and the n second inspection test work results match all of the inspection results of the first inspection test provider, the inspection Authorize the test subject to inspect,
When the inspection result of the inspection test subject for the m first inspection test work results and the n second inspection test work results is inconsistent with the inspection results of the first inspection test provider, the Without granting inspection authority to the inspection test subject, the first inspection test provider is selected as the inspection test subject,
When the first inspection test provider is selected as the inspection test subject,
Selecting an inspector (hereinafter, referred to as a second inspection test provider) to provide inspection results for inspection tests from among a plurality of inspectors other than the first inspection test provider;
Extracting m third inspection test work results and n fourth inspection test work results inspected by the second inspection test provider based on the inspection results;
Requesting to perform an inspection test by providing the m third inspection test work results and the n fourth inspection test work results to the first inspection test provider;
Receiving an inspection test result of the m third inspection test work results and the n fourth inspection test work results from the first inspection test provider; And
Further comprising the step of determining whether to withdraw the inspection authority of the first inspection test provider according to the inspection test result,
The m results of the third inspection test are the result of the inspection determined by the second inspection test provider as the inspection pass, and the results of the fourth inspection test of the n cases are inspected by the second inspection test provider. The result is the result of the work determined as rejection,
When the inspection results of the first inspection test provider with respect to the m third inspection test work results and the n fourth inspection test work results match all of the inspection results of the second inspection test provider, Maintain the inspection authority of the first inspection test provider,
When the inspection results of the first inspection test provider for the m third inspection test results and the n fourth inspection test results are inconsistent with the inspection results of the second inspection test provider , Recovering the inspection authority of the first inspection test provider, and selecting the second inspection test provider as the inspection test subject,
How to determine the authorization of inspection through inspection tests for crowdsourcing-based projects. The method of claim 1,
The step of selecting a predetermined worker from among the plurality of workers as an inspection test subject,
Among the plurality of workers, performing a reference number of jobs, and selecting an operator whose rejection rate of the job result of the reference number is less than or equal to a reference percentage as an inspection test subject,
How to determine the authorization of inspection through inspection tests for crowdsourcing-based projects. delete delete delete The method of claim 1,
When selecting an inspector to provide inspection results for inspection tests,
To select from among inspectors who have never provided inspection results for inspection tests before,
How to determine the authorization of inspection through inspection tests for crowdsourcing-based projects. The method of claim 1,
The m and n are determined according to the difficulty of the project,
How to determine the authorization of inspection through inspection tests for crowdsourcing-based projects. The method of claim 1,
The step of extracting m first inspection test work results and n second inspection test work results inspected by the first inspection test provider,
To extract by excluding the work result of the inspection test subject inspected by the first inspection test provider,
How to determine the authorization of inspection through inspection tests for crowdsourcing-based projects. The method of claim 1,
The step of extracting m first inspection test work results and n second inspection test work results inspected by the first inspection test provider,
To extract the m first inspection test work results and the n second inspection test work results based on the inspection results initially inspected by the first inspection test provider,
How to determine the authorization of inspection through inspection tests for crowdsourcing-based projects. A computer stored in a computer-readable recording medium in order to perform the method of granting inspection authority through the inspection test of any one of claims 1, 2, 6 to 9 in combination with a computer program.

Images