KR102156581B1 - Method for providing test crowdsoursing works based on real crowdsoursing works and apparatus thereof - Google Patents

Abstract

The present invention relates to a method and a device for providing verification-purpose crowdsourcing work based on actual work. The method performed by a computer includes: a step of establishing a verification-purpose crowdsourcing work pool; an actual crowdsourcing group work result receiving step of receiving an actual crowdsourcing group work result from a worker terminal, wherein the actual crowdsourcing group work includes at least one actual crowdsourcing work; a step of selecting at least one verification-purpose crowdsourcing work corresponding to the actual crowdsourcing group work in the verification-purpose crowdsourcing work pool and transmitting the same to the worker terminal; a step of receiving the result of the at least one verification-purpose crowdsourcing work from the worker terminal; and a step of determining whether the result of the actual crowdsourcing group work is valid based on the result of the at least one verification-purpose crowdsourcing work.

Description

Method and apparatus to provide crowdsourcing work for verification based on actual work {METHOD FOR PROVIDING TEST CROWDSOURSING WORKS BASED ON REAL CROWDSOURSING WORKS AND APPARATUS THEREOF}

The present invention relates to a method and apparatus for providing a crowdsourcing operation for verification based on an actual operation.

In machine learning, what is more important than learning methods and algorithms is the data for learning. Therefore, securing a lot of high-quality learning data is very important in the field of machine learning.

To obtain training data, it is necessary to collect source data or annotate data. For collecting source data or annotating data, where machine learning is researched or developed, personnel may be employed to collect or annotate source data.

However, in many cases, collecting source data or annotating data is something that anyone can do without specialized knowledge, so to reduce labor costs, distribute data to multiple workers through the Internet and collect source data or annotate data. A platform in the form of paying compensation according to the situation has appeared. For example, Amazon's Mechanical Turk.

However, in the case of the existing platform, there is a problem that it takes a lot of cost and time in the process of inspecting whether an operator has sincerely performed source data collection or data annotation work. Therefore, there is a need to develop a technology that can increase the reliability of learning data by efficiently collecting source data or inspecting data annotation work while maintaining the advantages of the existing platform.

Unexamined Patent Publication No. 10-2012-0149930, 2012.12.20

The problem to be solved by the present invention is to obtain and provide a verification crowdsourcing job provided in a verification method for a practical crowdsourcing job.

The problems to be solved by the present invention are not limited to the problems mentioned above, and other problems that are not mentioned will be clearly understood by those skilled in the art from the following description.

A method of providing a verification crowdsourcing task based on a practical task according to an embodiment of the present invention for solving the above-described problems is a method performed by a computer, comprising the steps of: establishing a crowdsourcing task pool for verification, Receives a result of a practical crowdsourcing group job from an operator terminal, wherein the practical crowdsourcing group job is composed of one or more practical crowdsourcing jobs, receiving a practical crowdsourcing group job result, the practical crowd Selecting one or more verification crowdsourcing jobs corresponding to a sourcing group job from the verification crowdsourcing job pool and transmitting them to the worker terminal receiving results of the one or more verification crowdsourcing jobs from the worker terminal And determining whether a result of the practical crowdsourcing group job is valid based on the results of the one or more verification crowdsourcing jobs, wherein the verification crowdsourcing job pool comprises: a plurality of verification crowdsourcing Including the work, and the step of establishing the verification crowdsourcing work pool, the verification pass that has passed the review of the inspector among the results of one or more practical crowdsourcing jobs previously performed in real time while the crowdsourcing work project is in progress. Extracting the result of the crowdsourcing operation, generating a crowdsourcing operation for verification based on actual work based on the result of the crowdsourcing operation and a check box that satisfies a predetermined condition among the extracted check box and the result of the crowdsourcing operation, and the above And including the generated field-based verification crowdsourcing job into the verification crowdsourcing job pool.

In one embodiment, the predetermined condition corresponds to a predetermined ranking based on the time taken for inspection.

In one embodiment, the step of generating the verification crowdsourcing operation comprises: extracting a check box that satisfies a predetermined condition among the extracted inspection box and a result of the crowdsourcing operation, and the result of the crowdsourcing operation, wherein the predetermined condition is Requesting a re-examination of a satisfactory inspection box and a result of the crowdsourcing operation to one or more inspectors different from the inspector, and the re-review result passed the inspection by all of the one or more inspectors, based on the result of the crowdsourcing operation And creating a crowdsourcing task for verification based on actual work.

In one embodiment, the step of transmitting at least one verification crowdsourcing job to the worker terminal includes at least one verification crowdsourcing job excluding verification crowdsourcing jobs already provided on the same worker terminal as the worker terminal. Is to send to.

In one embodiment, when a specific verification crowdsourcing job in the verification crowdsourcing job pool is generated based on the results of one or more practical crowdsourcing jobs received from the same worker terminal as the worker terminal, at least one verification In the step of transmitting a crowdsourcing job for the operator to the operator terminal, one or more crowdsourcing jobs for verification other than the specific verification crowdsourcing job are transmitted to the operator terminal.

In one embodiment, the step of constructing the verification crowdsourcing job pool further includes including, in the verification crowdsourcing job pool, a verification crowdsourcing job previously generated by the manager before the crowdsourcing job project is executed. .

In one embodiment, the step of selecting from the verification crowdsourcing job pool and transmitting it to the operator terminal comprises: after the previously generated verification crowdsourcing job is exhausted, the actual job-based verification crowdsourcing job is performed by the operator. It is to send to the terminal.

A method for providing a verification crowdsourcing job based on a practical job according to an embodiment of the present invention for solving the above-described problem is, according to the validity of the result of the practical crowdsourcing group job, the practical crowd The step of determining whether to proceed with the inspection result of the sourcing group operation further comprises the step of determining whether to proceed with the inspection result of the practical crowdsourcing group operation, wherein the result of the verification crowdsourcing operation is determined in advance. If the answer is consistent with the answer, it is decided to proceed with the inspection of the result of the work of the relevant practical crowdsourcing group, and if the result of the above verification crowdsourcing work is inconsistent with the predetermined answer, the work of the relevant practical crowdsourcing group It is decided not to proceed with the inspection by deleting the result.

A computer program for providing a crowdsourcing task for verification based on a practical task according to another embodiment of the present invention for solving the above-described problem is combined with a computer that is hardware, in order to execute the method of any one of the above methods. Is saved.

The apparatus for providing a verification crowdsourcing job based on a practical job according to another embodiment of the present invention for solving the above-described problem is a verification crowdsourcing job pool construction unit that builds a verification crowdsourcing job pool, from an operator terminal. Receives the results of a practical crowdsourcing group job, wherein the practical crowdsourcing group job is composed of one or more practical crowdsourcing jobs, a practical crowdsourcing group job result receiving unit, to the practical crowdsourcing group job A verification crowdsourcing job transmission unit that selects one or more corresponding verification crowdsourcing jobs from the verification crowdsourcing job pool and transmits them to the worker terminal, and the results of the one or more verification crowdsourcing jobs from the worker terminal And a validity determination unit that determines whether or not a result of the practical crowdsourcing group operation is valid, based on a received, verification crowdsourcing operation result receiving unit and a result of the one or more verification crowdsourcing operations, and the verification The crowdsourcing job pool for verification includes a plurality of crowdsourcing jobs for verification, and the verification crowdsourcing job pool construction unit includes one or more practical crowdsourcing jobs previously performed in real time while the crowdsourcing job project is in progress. Among the results, the inspection box that passed the inspection by the inspector and the result of the crowdsourcing operation are extracted, and the inspection box that satisfies a predetermined condition among the extracted inspection bottles and the results of the crowdsourcing operation, and for verification based on actual work based on the results of the crowdsourcing operation A crowdsourcing job is created, and the generated crowdsourcing job for verification based on a practical job is included in the verification crowdsourcing job pool.

Other specific details of the present invention are included in the detailed description and drawings.

According to the present invention, even if all the pre-generated verification crowdsourcing jobs are provided to the operator, the verification crowdsourcing job is continuously updated based on the actual job, so that new verification crowdsourcing jobs can be continuously provided to the operator. have.

Further, according to the present invention, the crowdsourcing work data for practical use is provided as a crowdsourcing work for verification, so that the crowdsourcing work for verification can be utilized as evaluation data of a work more suitable for the project.

In addition, according to the present invention, by providing a crowdsourcing operation for verification that has a long working time among the crowdsourcing work that has passed the inspection to a specific inspector and that all inspections have been passed by other inspectors as a crowdsourcing operation for verification, it is highly difficult and accurate. Through high verification crowdsourcing, the evaluation results for workers can be more accurately calculated.

The effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the following description.

1 is a conceptual diagram of a crowdsourcing service according to an embodiment of the present invention.
2 is a flowchart illustrating a process of a crowdsourcing project according to an embodiment of the present invention.
3 is a schematic flowchart of a method for providing a crowdsourcing operation for verification based on an actual operation according to an embodiment of the present invention.
4 is a schematic diagram of a crowdsourcing operation for an image including an object.
5 is a diagram for explaining a method of constructing a crowdsourcing work pool for verification.
6 is a diagram for explaining a method of providing a crowdsourcing job for practical use and a crowdsourcing job for verification.
7 and 8 are exemplary diagrams for explaining a method of acquiring a crowdsourcing job for verification from a result of an actual crowdsourcing job.
FIG. 9 is an exemplary diagram for providing a practical crowdsourcing operation and a verification crowdsourcing operation after a result of an actual crowdsourcing operation is invalid.
10 is an exemplary diagram for providing a practical crowdsourcing operation and a verification crowdsourcing operation after a result of a practical crowdsourcing operation is valid.
11 is a schematic configuration diagram of an apparatus for providing a crowdsourcing job for verification based on a practical job according to another embodiment of the present invention.

Advantages and features of the present invention, and a method of achieving them will become apparent with reference to the embodiments described below in detail together with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but may be implemented in a variety of different forms, only the present embodiments are intended to complete the disclosure of the present invention, It is provided to fully inform the technician of the scope of the present invention, and the present invention is only defined by the scope of the claims.

The terms used in the present specification are for describing exemplary embodiments and are not intended to limit the present invention. In this specification, the singular form also includes the plural form unless specifically stated in the phrase. As used in the specification, “comprises” and/or “comprising” do not exclude the presence or addition of one or more other elements other than the mentioned elements. Throughout the specification, the same reference numerals refer to the same elements, and “and/or” includes each and all combinations of one or more of the mentioned elements. Although "first", "second", and the like are used to describe various elements, it goes without saying that these elements are not limited by these terms. These terms are only used to distinguish one component from another component. Therefore, it goes without saying that the first component mentioned below may be the second component within the technical idea of the present invention.

Unless otherwise defined, all terms (including technical and scientific terms) used in the present specification may be used as meanings that can be commonly understood by those of ordinary skill in the art to which the present invention belongs. In addition, terms defined in a commonly used dictionary are not interpreted ideally or excessively unless explicitly defined specifically.

In the present specification, an operator refers to an individual or group that performs a crowdsourcing job provided from a verification crowdsourcing job providing device based on an actual job and receives a predetermined compensation. Projects that include crowdsourcing work include various types of crowdsourcing work, such as collection of source data or data annotation, for example, if the crowdsourcing work is annotating work, it is used for verification based on actual work. The crowdsourcing work providing device provides an image with a specific object or text to the workers, and the workers designate the range of the object or text included in the image, and an annotation that inputs what the object or text is included in the specified range. Do the processing work. The result of the annotation processing operation performed by the workers is transmitted to the automatic inspection device for annotating the data using the verification data.

The inspector refers to an individual or organization that performs inspection on the results of crowdsourcing jobs uploaded by workers. Depending on the embodiment, the inspector is not a person, and may be understood as encompassing a trained model, program, terminal, etc. that can automatically perform the inspection.

However, in the case of performing the inspection on all crowdsourcing work results, the bottleneck of generating learning data may occur in the inspection process depending on the number of inspectors, and development of a method to improve this is required.

Therefore, the method and apparatus for providing a crowdsourcing work for verification based on the actual work of the present invention is a method of automatically pre-verifying the crowdsourcing work performed by the operator before the inspector inspects it. On the basis of this, the inspector can only inspect the results of the practical crowdsourcing work of the worker who has passed a predetermined standard before proceeding with the inspection.

In addition, by configuring the crowdsourcing work for verification based on the crowdsourcing work that passed the inspection and passed the inspection among the results of the previously performed crowdsourcing work, as the crowdsourcing work for verification is continuously updated, the verification crowd is provided to the operator. When providing sourcing work, it is possible to increase the effectiveness of providing a crowdsourcing work for verification by providing a new non-overlapping crowdsourcing work for verification.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a conceptual diagram of a crowdsourcing service according to an embodiment of the present invention.

Referring to FIG. 1, a crowdsourcing service is performed by being composed of a client 10, a service provider 20, and a public 30.

The sponsor 10 refers to a company or individual who requests a crowdsourcing project (hereinafter, referred to as a project).

The requester 10 requests a project for the purpose of collecting source data or annotating data. Collection of source data refers to collecting raw data such as recorded voice collection and photo collection. Data annotation refers to inputting relevant annotation data into source data such as text, photos, and videos. For example, the data annotation may include finding an entity in a given fingerprint or finding a similar sentence, but is not limited thereto. Meanwhile, the types of the above-described projects are only one embodiment, and various projects may be handled in the present invention according to the design of the client.

The service provider 20 refers to a company that provides crowdsourcing services.

When the service provider 20 receives a request for a project for a product or service from the client 10, the service provider 20 allocates a task for the project to the general public 30 and receives the work result from the public 30. Thereafter, the final product extracted based on the work result is provided to the client 10.

At this time, the service provider 20 provides a crowdsourcing service to the client 10 and the public 30 through a crowdsourcing platform (hereinafter, the platform). That is, when the service provider 20 receives a request for a project from the client 10, the service provider 20 opens the project on the platform. Thereafter, when the work result for the open project is received from the public 30, the project is terminated on the platform, and the final product may be extracted and provided to the client 10.

The public 30 refers to the general public participating in the project open on the platform. Here, the public 30 may participate in a project opened on the platform through an application or website provided by the service provider 20.

The public 30 is composed of a worker 32 and an inspector 34.

The worker 32 decides to participate in a specific project among a plurality of projects open on the platform. Thereafter, the operator 32 performs an operation such as collection of source data or data annotation, and transmits it to the platform.

The inspector 34 decides to participate in a specific project among a plurality of projects open on the platform. Thereafter, the inspector 34 inspects the result of the work performed by the operator 32. As a result of performing the inspection, the inspector 34 may pass or reject the job, and may input a reason for rejection when returning the job.

2 is a flowchart illustrating a process of a crowdsourcing project according to an embodiment of the present invention.

First, the requester 10 requests one or more projects to the service provider 20 (S11).

Thereafter, the service provider 20 opens the requested project on the platform (S12). At this time, the service provider 20 may determine the grade in consideration of the difficulty of the project before opening the project. That is, depending on the difficulty level, it is possible to determine which level or higher public 30 to expose the project. Accordingly, it is possible to increase the reliability of the work result of the project.

Thereafter, the service provider 20 allocates a task to the workers 32 of the corresponding grade or higher according to the grade of the project (S13).

Thereafter, the worker 32 performs the assigned task (S14). In this case, the worker 32 may input the reason for the inability to work without performing the work for a work in which the work itself is impossible for some reason.

Thereafter, the service provider 20 receives the work result from the worker 32 (S15), and assigns the inspection work on the work result to the inspector 34 (S16).

Thereafter, the inspector 34 performs the assigned inspection (S17). At this time, the inspector 34 decides to pass the inspection when it is determined that the work has been properly performed, and determines rejection when it is determined that the inspection work is wrong. When deciding to reject, the inspector 34 inputs the rejection reason for what reason the task was judged to be wrong.

Thereafter, the service provider 20 receives the inspection result from the inspector 34 (S18).

If the inspection result is Pass, the service provider 20 uses the work result as valid data, and based on this, extracts the final product at the end of the project.

If the inspection result is rejected, the service provider 20 may internally perform the inspection again or assign the job again to the worker 32 to perform the job.

Thereafter, when the project period ends or sufficient valid data is secured, the service provider 20 terminates the project (S19), calculates a final result based on the secured valid data, and provides it to the client 10 ( S20).

At this time, before the end of the project, the service provider 20 evaluates the performance results of the worker 32 and the inspector 34, calculates the work cost and the inspection cost according to the evaluation, to the worker 32 and the inspector 34. give.

Meanwhile, in FIGS. 1 and 2, the client 10, the service provider 20, the worker 32, and the inspector 34 are simply expressed, but these are smartphones, tablets, PDAs, and laptops operated by each participant. It means a computer device or a telecommunication device such as a desktop or a server.

3 is a schematic flowchart of a method for providing a crowdsourcing operation for verification based on an actual operation according to an embodiment of the present invention.

Referring to FIG. 3, a method of providing a verification crowdsourcing job based on an actual job is a step of constructing a verification crowdsourcing job pool (S110), an actual crowdsourcing group job result receiving step (S130), one Transmitting the above verification crowdsourcing job to the operator terminal (S150), receiving the results of one or more verification crowdsourcing jobs from the operator terminal (S170), and determining whether the results of the practical crowdsourcing group job are valid It includes a step (S190).

The step of establishing a crowdsourcing work pool for verification (S110) includes establishing a crowdsourcing work pool for verification in real time to provide the work of the project and/or before providing the work of the project to the worker.

A detailed description of the step (S110) of building a crowdsourcing job pool for verification will be described later in the description of FIGS. 5 to 8.

The practical crowdsourcing group work result reception step (S130) is to receive the results of the practical crowdsourcing group job from the operator terminal, and the practical crowdsourcing group job is composed of one or more practical crowdsourcing jobs. The practical crowdsourcing work consists of the work requested by the customer who requested the crowdsourcing work.

Here, the crowdsourcing operation includes various types of crowdsourcing operations such as collection of source data or data annotation, as described above. An example of the crowdsourcing operation is an annotation operation processing operation, and a description thereof will be described later in the description of FIG. 4.

The customer requests a certain crowdsourcing job from the manager of the method and device to provide a crowdsourcing job for verification based on the actual job. The manager requests a certain crowdsourcing job requested by one or more workers, and pays compensation according to the work result of the worker.

A practical crowdsourcing group job composed of one or more practical crowdsourcing jobs will be described in FIG. 6.

In the step (S150) of transmitting at least one verification crowdsourcing job to an operator terminal (S150), at least one verification crowdsourcing job corresponding to a practical crowdsourcing group job is selected from the verification crowdsourcing job pool and transmitted to the operator terminal. .

Since the verification crowdsourcing job pool includes one or more verification crowdsourcing jobs, verification crowdsourcing jobs are selected on a predetermined basis or randomly and transmitted to the operator terminal. In other words, one or more crowdsourcing jobs for verification are provided to the operator terminal per one practical crowdsourcing group job, and at this time, one or more crowdsourcing jobs for verification are selected from the pool of verification crowdsourcing jobs. will be.

Since the answer to each verification crowdsourcing task is predetermined, the computer must receive the predetermined answer from the operator terminal.

The step of transmitting one or more verification crowdsourcing jobs to the operator terminal (S150) is whether or not one or more verification crowdsourcing jobs generated from the results of practical crowdsourcing jobs received from the same worker, or one or more already provided to the same worker. One or more crowdsourcing jobs for verification can be sent, considering whether or not it is a verification crowdsourcing job.

In one embodiment, the step of transmitting one or more verification crowdsourcing jobs to the operator terminal (S150) is to perform one or more verification crowdsourcing jobs to the operator terminal excluding the verification crowdsourcing jobs already provided in the same operator terminal as the operator terminal. Send. This is because, in the case of crowdsourcing for verification already provided to the same operator terminal, the effectiveness of the verification decreases because the operator knows the answer in advance.

In another embodiment, when a specific verification crowdsourcing job in the verification crowdsourcing job pool is generated based on the results of one or more practical crowdsourcing jobs received from the same worker terminal as the corresponding worker terminal, one or more verification purposes In the step of transmitting the crowdsourcing job to the operator terminal (S150), at least one verification crowdsourcing job excluding the specific verification crowdsourcing job is transmitted to the operator terminal. Even in the case of the verification crowdsourcing job created based on the results of the actual crowdsourcing job received from the same operator terminal, the verification effect is inferior because the operator knows the answer in advance, so the verification crowdsourcing job is also excluded. shall.

In the step S150 of transmitting one or more verification crowdsourcing jobs to the operator terminal, the computer may call up one or more verification crowdsourcing jobs processed within the verification crowdsourcing job pool. Here, processing may mean generating a data set by using one or more crowdsourcing tasks and answers from one or more crowdsourcing tasks.

The computer selects one or more verification crowdsourcing jobs from the verification crowdsourcing job pool. The computer selects from the pool of verification crowdsourcing jobs one or more verification crowdsourcing jobs corresponding to the actual crowdsourcing jobs. Therefore, the crowdsourcing work for verification and the crowdsourcing work for practical use can be performed as work with the same category. In addition, the computer may select one or more of the practical crowdsourcing tasks as the verification crowdsourcing task. By selecting a verification crowdsourcing job and a practical crowdsourcing job into the same category, or selecting one or more of the practical crowdsourcing jobs as verification crowdsourcing jobs, the operator can perform the actual crowdsourcing job by himself during the crowdsourcing job. They do not recognize whether they are doing the verification or crowdsourcing work for verification. In addition, the results of the verification crowdsourcing work can represent the results of the practical crowdsourcing group work.

One or more practical crowdsourcing jobs are stored and managed in a verification crowdsourcing job pool, and one or more verification crowdsourcing jobs are selected from one or more practical crowdsourcing jobs stored in the verification crowdsourcing job pool. The number of one or more verification crowdsourcing jobs may be determined corresponding to the number of the one or more practical crowdsourcing jobs stored in the verification crowdsourcing job pool.

The crowdsourcing work for verification generated in the present invention is as described later in Figs. 6 to 8, and the crowdsourcing work for verification that is provided in advance before creating the crowdsourcing work for verification obtained from the actual work is all requested from the customer. It is chosen for practical crowdsourcing work. Workers perform some of the entire verification crowdsourcing work while performing the actual crowdsourcing work, and depending on the results of the actual crowdsourcing work performed by the worker, it is decided whether to perform other parts of the entire verification crowdsourcing work. do.

The database stores and manages one or more practical crowdsourcing jobs received from the customer's terminal, stores and manages verification crowdsourcing jobs and answers selected from the database control unit (not shown), and stores one or more Stores and manages answers to practical crowdsourcing tasks.

In step S170 of receiving the results of one or more verification crowdsourcing jobs from the operator terminal (S170), the results of one or more verification crowdsourcing jobs corresponding to the practical crowdsourcing group jobs are received from the operator terminal.

The pre-determined answers corresponding to the verification crowdsourcing job and verification crowdsourcing job consist of a dataset of verification crowdsourcing jobs. The verification crowdsourcing job can be selected from a crowdsourcing job for practical use, which is a predetermined crowdsourcing job requested by the customer. The predetermined answer corresponding to the verification crowdsourcing operation may be, in one embodiment, an answer previously made by the administrator of the verification device for a practical crowdsourcing job using the verification crowdsourcing job for the verification crowdsourcing job. The manager can divide a predetermined crowdsourcing task requested from a customer by a predetermined field and configure a data set of crowdsourcing tasks and answers for each field.

However, for the verification crowdsourcing work and corresponding predetermined answers, even if the manager creates the verification crowdsourcing work selection and answers before requesting the operator to perform the project, it is performed on one project. The number of tasks to be performed is very large, and because there are cases where it is necessary to provide more frequent verification crowdsourcing jobs to workers with many rejected jobs, the number of verification crowdsourcing jobs created in advance may be insufficient.

If all pre-created verification crowdsourcing jobs are provided and redundantly created verification crowdsourcing jobs are provided, the operator can easily answer the verification crowdsourcing job by memorizing the answer of the job. Therefore, it is possible to draw conclusions that cannot be solved for the essential task of providing verification crowdsourcing.

Accordingly, the present invention enables to continuously provide a new verification crowdsourcing operation even if all of the pre-created verification crowdsourcing jobs are provided.At least one verification crowdsourcing job of the present invention passes one or more inspections. This is one or more within the pool of verification crowdsourcing jobs consisting of crowdsourcing jobs.

At this time, the inspection pass crowdsourcing job is a job that has been inspected by an inspector among one or more crowdsourcing jobs that have been previously performed, and the one or more crowdsourcing jobs that have been previously performed are jobs performed before the actual crowdsourcing group work.

The crowdsourcing work for verification acquired based on the actual work will be described in detail with reference to FIGS. 7 and 8.

In the step of determining whether the results of the practical crowdsourcing group work are valid (S190), it is determined whether the results of the practical crowdsourcing group work are valid based on the results of one or more verification crowdsourcing jobs.

The computer determines whether to classify the results of one or more practical crowdsourcing operations as valid or invalid data.

When the result of the verification crowdsourcing operation matches the predetermined answer, the computer treats the result of the practical crowdsourcing group work as valid, and when the result of the verification crowdsourcing operation is inconsistent with the predetermined answer, Treat the results of a crowdsourcing group work for real use as invalid. Therefore, since the crowdsourcing for practical use defined as a result of an invalid crowdsourcing operation is not inspected by the inspector, the amount inspected by the inspector can be reduced. In addition, the practical crowdsourcing group work defined as a result of effective crowdsourcing work is inspected by an inspector.

That is, although not shown separately, the method of providing a crowdsourcing task for verification based on the actual work of the present invention includes the steps of establishing a crowdsourcing task pool for verification (S110) to the results of the actual crowdsourcing group work. In the step S190 of determining the validity of (S190), according to whether the result of the actual crowdsourcing group work is valid, determining whether to proceed with the inspection of the result of the actual crowdsourcing group work.

The step of deciding whether to proceed with the inspection is, if the result of the verification crowdsourcing work matches the predetermined answer, it is decided to proceed with the inspection of the results of the practical crowdsourcing group work, and the result of the verification crowdsourcing work If is inconsistent with the predetermined answer, it is decided not to proceed with the inspection by deleting the result of the actual crowdsourcing group work.

4 is a schematic diagram illustrating annotation processing for an image including an object.

Referring to FIG. 4, an exemplary embodiment of performing annotation processing on an image including a bed as a crowdsourcing operation is illustrated. The practical crowdsourcing work and the verification crowdsourcing work may be in the form of FIG. 4.

The image 200 shown in FIG. 4 includes at least one object. The operator may select a specific area 210 of the image 200 and input or select information 220 describing an object included in the selected area 210. For example, a plurality of objects may be included in one image 200, and an operator may perform a crowdsourcing operation of annotating one or more objects included in the image 200.

In the embodiment shown in FIG. 4, the result of the crowdsourcing operation as the annotation processing operation is the coordinates of the specific region 210 and information on the objects included in the specific region 210 (for example, in the specific region 210 ). Information that the included object is a bed) may be included.

5 to 8 are detailed descriptions of the step (S110) of building a crowdsourcing job pool for verification.

In the step of constructing the verification crowdsourcing job pool (S110), the verification crowdsourcing job pool is an embodiment, and will consist of a practical job-based verification crowdsourcing job derived based on the results of the practical crowdsourcing job. In another embodiment, it may be configured to further include a crowdsourcing task for verification generated in advance by an administrator prior to performing the crowdsourcing project project together with the crowdsourcing task for verification of the above embodiment.

In the case of constructing a crowdsourcing work pool for verification by further including a crowdsourcing job for verification previously created by the manager before performing the crowdsourcing work project, selecting from the crowdsourcing work pool for verification and sending it to the operator terminal ( In step S150), after the pre-generated verification crowdsourcing job is exhausted, the actual job-based verification crowdsourcing job is transmitted to the operator terminal.

Therefore, in the early stage of the crowdsourcing project, as a crowdsourcing for verification, a pre-generated crowdsourcing for verification is provided, and when the previously generated crowdsourcing for verification is exhausted, the crowdsourcing for verification based on actual work is performed. By providing it, it is possible to verify the results of the actual crowdsourcing work by continuously providing new verification crowdsourcing work to the worker from the initial stage to the last stage of the work.

5 is a diagram for explaining a method of constructing a crowdsourcing work pool for verification.

Referring to FIG. 5, a method of constructing a crowdsourcing job pool for verification includes a step of extracting a check box and a result of a crowdsourcing job (S111), a step of generating a crowdsourcing job for verification based on an actual job (S113), and a practice Including a crowdsourcing job for job-based verification in the crowdsourcing job pool for verification (S115).

The step (S111) of extracting the results of the check box and the crowdsourcing work is the result of one or more practical crowdsourcing work previously performed in real time while the crowdsourcing work project is in progress. To extract the results.

In the step of creating a crowdsourcing task for verification based on actual work (S113), a check box that satisfies a predetermined condition among the check boxes extracted in step S111 and the results of the crowdsourcing work, and a crowdsourcing work based on the actual work based on the results of the crowdsourcing work Is to create.

Here, the predetermined condition corresponds to a predetermined ranking based on the time required for inspection.

Creating a crowdsourcing task based on the actual task based on the result of the check box and the crowdsourcing task is, in one embodiment, the result of the check box and the crowdsourcing task itself can create a crowdsourcing task based on the actual task, and the check box and the crowd It is possible to create a crowdsourcing job based on actual work according to the results of the re-examination by requesting re-examination of the results of the sourcing work.

At this time, in the case of re-examination by other inspectors, the step of creating a crowdsourcing job for verification based on actual work is to extract a inspection box that satisfies a predetermined condition among the extracted inspection boxes and the results of the crowdsourcing job and the results of the crowdsourcing job. The step of requesting re-review of the inspection box that satisfies a predetermined condition and the result of the crowdsourcing operation from one or more inspectors different from the inspector who conducted the inspection, and the re-examination result that has passed the inspection by all of at least one inspector It may include the step of creating a crowdsourcing task for verification based on the actual task based on the result of the pass crowdsourcing task.

In the step (S115) of including the field work-based verification crowdsourcing task in the verification crowdsourcing task pool (S115), the actual task-based verification crowdsourcing task generated in step S113 is included in the verification crowdsourcing task pool.

6 is a diagram for explaining a method of providing a crowdsourcing job for practical use and a crowdsourcing job for verification.

Referring to FIG. 6, since the arrow indicates the order of provision, it can be seen that after the practical crowdsourcing group task 50 is provided to the worker, the verification crowdsourcing task 60 is provided.

The practical crowdsourcing group job 50 includes one or more practical crowdsourcing jobs, and in FIG. 6, six practical crowdsourcing jobs are exemplarily shown (51, 52, 53, 54, 55, 56). Done. However, since one or more practical crowdsourcing jobs may be included in the practical crowdsourcing group job, less or more than that of FIG. 6 may be included.

In addition, although only one unit in which the verification crowdsourcing work 60 is provided is shown in FIG. 6, the work of the practical crowdsourcing work 51 to 56 of the project is completed and passed until inspection. Provisioning operations may be provided repeatedly. That is, after providing a crowdsourcing group task 50 for practical use, a crowdsourcing task 60 for verification is provided, and after providing the crowdsourcing group task 50 for practical use, the crowdsourcing task 60 for verification may be repeatedly provided. .

7 and 8 are exemplary diagrams for explaining a method of acquiring a crowdsourcing job for verification from a result of an actual crowdsourcing job.

Since the project of the present invention is not performed by one worker, but is crowdsourcing, multiple workers are working, so before a specific worker performs the actual crowdsourcing work, the actual crowdsourcing work (71, 72, 73, 74, 75, 76) may exist. When a specific worker first performs a practical crowdsourcing job, there is no previous practical crowdsourcing job (71~76), but in this case, the verification crowdsourcing job created in advance by the administrator as described above. Can provide. However, in FIGS. 7 and 8, generation of a crowdsourcing task for verification is described based on the actual field work corresponding to the case where the previously performed actual crowdsourcing tasks 71 to 76 exist.

The crowdsourcing work pool for verification can be updated in real time, and when the crowdsourcing work (81~84) that passes the inspection in real time is extracted, the crowdsourcing work (81~84) that passed the inspection is extracted as a crowdsourcing work pool for verification. Can be updated.

At this time, the extracted inspection pass crowdsourcing operation may include, in one embodiment, a simple inspection pass crowdsourcing operation 81 to 84 that have simply passed the inspection, and in another embodiment, the description of FIG. 7 to be described later. As shown in FIG. 8, the crowdsourcing process 81-83 extracted in a predetermined ranking may be included, and in another embodiment, the inspection passing through the inspection of other inspectors among the predetermined rankings as described in FIG. 8 to be described later. May include a pass-through crowdsourcing operation 81.

In addition, the verification crowdsourcing job pool may further include a newly extracted inspection pass crowdsourcing job, excluding verification crowdsourcing jobs used by workers.

In one embodiment, if the crowdsourcing work pool for verification is different for each worker, excluding the crowdsourcing work that has already passed the inspection and that has already been provided to the worker, crowdsourcing the number of newly extracted crowdsourcing jobs that have passed the inspection as excluded. To include more in the working pool.

The above embodiment may correspond to an example in which the number of verification crowdsourcing jobs included in the verification crowdsourcing job pool is specified (eg, the number of verification crowdsourcing jobs is maintained at 100).

In another embodiment, the crowdsourcing work pool for verification may be cumulatively built, and when the crowdsourcing work pool for verification is the same for each worker, every time a crowdsourcing job for verification is provided, the work performed by the same worker is excluded. And provide.

The above embodiment corresponds to the case where the number of verification crowdsourcing jobs is not specified, and in the case where the number of verification crowdsourcing jobs is not specified, the inspection pass, which is excluded, is not considered, but for verification. Whenever the crowdsourcing job 60 is extracted, the continuously extracted verification crowdsourcing job 60 may be included in the verification crowdsourcing job pool.

The number of previously performed practical crowdsourcing jobs 71-76, inspection passing crowdsourcing jobs 80, and inspection passing crowdsourcing jobs 81-84 shown in FIG. 7 is arbitrary, and any one or more may be included. .

Referring to FIG. 7, the previously performed crowdsourcing for practical use (71 to 76) is a crowdsourcing for practical use previously performed by workers, and corresponds to a job before the inspection by an inspector is completed. When the inspection of the previously performed practical crowdsourcing work (71~76) is completed, the previously performed practical crowdsourcing work (73, 74, 75, 76) that has passed the inspection is passed through the crowdsourcing work ( 80).

The computer arranges the classified inspection pass crowdsourcing task 80 in the order of the longest time required for inspection, as in the example in FIG. 7, and the inspection pass crowdsourcing corresponding to a predetermined priority in the order of the longest inspection time. Only tasks (81-83) are extracted.

Longer inspection time means that you need to perform the work more carefully than other tasks, so if the inspection time is extracted in the long order, separate inspection time for defective jobs in accordance with the purpose of providing crowdsourcing for verification It is more effective because it can not take.

In addition, previously performed crowdsourcing tasks for practical use (71 to 76) and crowdsourcing tasks that have passed inspection (81 to 84) will be continuously generated in real time, and the criteria for extracting a predetermined ranking are set by an administrator or a computer. May be set in advance, and may be extracted in units of number or time.

The inspection pass crowdsourcing jobs 81 to 83 extracted by a predetermined rank according to FIG. 7 may be immediately included in the verification crowdsourcing job pool.

However, since there may be errors in the inspection, such as not being able to concentrate on the work, the inspector may be included in the crowdsourcing work pool for verification through an additional inspector's inspection in order to prevent this and provide a more verified crowdsourcing work 60. The final verification crowdsourcing operation 60 can be extracted.

Referring to FIG. 8, it can be seen how to extract the crowdsourcing work 60 for verification through additional inspection work by other inspectors from the inspection pass crowdsourcing jobs 81 to 83 extracted in a predetermined order in FIG. 7.

First, if the inspector who performed the inspection in FIG. 7 is referred to as the first inspector, in FIG. 8, additionally, the second inspector 34-1 and the third inspector 34-2 undergo inspection. In this case, only two inspectors are shown to be additionally inspected in FIG. 8, but the number of inspectors performing additional inspection is at least one, and may be predetermined, set by the administrator, or determined based on a set schedule. .

In more detail with respect to FIG. 8, the inspection is passed by the first inspector in FIG. 7, and the inspection passing crowdsourcing jobs 81 to 83 extracted in a predetermined order are the second inspector 34-1 and Have the third inspector (34-2) additionally inspect it.

The second inspector (34-1) passed the inspection only for the crowdsourcing work that passed the inspection of 81 and 82, and did not pass the crowdsourcing work that passed the inspection of 83.

The third inspector (34-2) passed the inspection only for the crowdsourcing work that passed the inspection of 81, and did not pass the inspection for the crowdsourcing work that passed the inspection of 82 and 83.

Therefore, the work that both the second and third inspectors (34-1) and the third inspector (34-2), which are additional inspectors, passed, is a crowdsourcing that passed the inspection of 81, and only the crowdsourcing that passed the inspection was for verification As a crowdsourcing job 60, it is included in the crowdsourcing job pool for verification.

Although not shown separately, the method for providing a verification crowdsourcing job based on a practical job of the present invention is an additional practical crowd after the step (S190) of determining whether the result of the practical crowdsourcing group job is valid. It may further include receiving a result of the sourcing group operation and receiving a result of the additional verification crowdsourcing operation.

1) If the results of n practical crowdsourcing operations are invalid

Immediately after defining the results of n practical crowdsourcing jobs as invalid crowdsourcing jobs, the computer sends n-i (where i is a natural number of 1 or more) additional practical crowdsourcing jobs to the operator's terminal.

The computer receives the results of n-i additional hands-on crowdsourcing tasks from the operator's terminal. Then, one or more additional verification crowdsourcing jobs, which are prepared in advance, corresponding to n-i additional practical crowdsourcing jobs are called.

The computer transmits the one or more additional verification crowdsourcing jobs to the operator's terminal immediately after receiving the results of n-i additional practical crowdsourcing jobs. Then, the result of one or more additional verification crowdsourcing jobs is received from the operator's terminal.

If the operator does not properly perform the crowdsourcing for verification, then the crowdsourcing for verification is transmitted more quickly.After that, the order to receive the crowdsourcing for verification is determined by i, and i is invalid crowdsourcing. It changes according to the number of tasks defined as tasks. For example, i may be a value obtained by dividing the number of tasks defined as previously invalid crowdsourcing tasks by a predetermined natural number of 1 or more. Alternatively, i may be a natural number greater than or equal to a predetermined number corresponding to the number of jobs defined as a previous invalid crowdsourcing job.

If i is less than a predetermined criterion, the worker who performed the corresponding crowdsourcing task and verification crowdsourcing task is defined and classified as an untrusted worker, and the computer performs one or more practical crowdsourcing tasks on the worker's terminal. Do not send. For example, when i is less than 2, workers can no longer perform crowdsourcing for practical use. Defining a worker as an untrusted worker means that it is unreliable for one particular crowdsourcing task, and if a new or existing client requests a new crowdsourcing task from the manager, the previous crowdsourcing task will be unreliable. Workers classified as missing workers can also perform new practical crowdsourcing tasks.

The customer request receiving unit (not shown) may receive the reliability of the crowdsourcing operation requested from the customer's terminal. i may be a value changed according to the reliability of the crowdsourcing operation requested from the customer terminal. i may be a value obtained by dividing the number of tasks defined as invalid crowdsourcing by a predetermined number. For example, i may be the number of invalid crowdsourcing jobs divided by 2. The predetermined number can be determined by the reliability of crowdsourcing operations requested from customers. For example, when the reliability of the crowdsourcing work requested by the customer is high, a predetermined number can be made a low number. As a result, i becomes a relatively high number, and the timing at which the verification crowdsourcing operation is transmitted can be relatively accelerated.

2) When the results of n practical crowdsourcing jobs are valid

Immediately after defining the results of n practical crowdsourcing jobs as valid crowdsourcing jobs, the computer transmits n+j additional practical crowdsourcing jobs (where j is a natural number of 1 or more) to the operator's terminal.

The computer receives the results of n+j additional practical crowdsourcing operations from the operator's terminal.

Then, one or more additional verification crowdsourcing jobs, which are prepared in advance, corresponding to n+j additional practical crowdsourcing jobs are invoked.

Immediately after receiving the results of the n+j additional practical crowdsourcing jobs, the computer sends one or more additional verification crowdsourcing jobs to the operator's terminal. Then, the result of one or more additional verification crowdsourcing jobs is received from the operator's terminal.

If the operator properly performs the crowdsourcing for verification, the crowdsourcing for verification will be transmitted later.After that, the order of receiving the crowdsourcing for verification is determined by j, and j is defined as a valid crowdsourcing job. It changes according to the number of jobs done. For example, j may be a value obtained by dividing the number of jobs defined as a previously valid crowdsourcing job by a predetermined natural number of 1 or more. Alternatively, j may be a natural number equal to or greater than one predetermined corresponding to the number of tasks defined as a previous effective crowdsourcing task.

If j is more than a predetermined criterion, the operator who performed the practical crowdsourcing task and the verification crowdsourcing task is defined and classified as a trusted worker, and the computer performs one or more crowdsourcing tasks for verification on the worker's terminal. Do not send. Defining a worker as a trusted worker means that they can be trusted for one particular crowdsourcing job, and if a new or existing customer requests a new crowdsourcing job from the manager, they can be trusted in the previous crowdsourcing job. Workers classified as existing workers may also need to perform a new verification crowdsourcing task.

If an operator is classified and defined as a trusted operator, do not proceed with receiving the results of one or more additional verification crowdsourcing operations corresponding to one or more additional practical crowdsourcing operations. Therefore, the result of additional practical crowdsourcing work by a trusted worker is defined as a valid crowdsourcing work without the verification process as a verification crowdsourcing work.

The customer request receiving unit (not shown) receives the reliability of the crowdsourcing operation requested from the customer's terminal, and j may be a value changed corresponding to the reliability of the crowdsourcing operation requested by the customer. j may be a value obtained by dividing the number of effective crowdsourcing operations by a predetermined number. For example, j may be the number of valid crowdsourcing operations divided by 2. The predetermined number can be determined by the reliability of crowdsourcing operations requested from customers. For example, when the reliability of crowdsourcing work requested by a customer is high, a predetermined number can be made a high number. As a result, j becomes a relatively low number, and the time point at which the verification crowdsourcing task is transmitted can be relatively accelerated.

FIG. 9 is an exemplary diagram for providing a practical crowdsourcing operation and a verification crowdsourcing operation after a result of an actual crowdsourcing operation is invalid.

Referring to FIG. 9, when the result of the verification crowdsourcing operation does not match a predetermined answer, the result of the actual crowdsourcing operation is defined and classified as invalid.

If the operator does not properly perform the verification crowdsourcing job #1(310), the result of n practical crowdsourcing jobs performed before verification crowdsourcing job #1(310) is defined as an invalid crowdsourcing job. And classified. In addition, the operator performs a crowdsourcing task for practical use n-i times lower than that of the previous n times, and performs crowdsourcing task #2 (320) for verification. Similarly, if the verification crowdsourcing operation #2 (320) is not properly performed, the ni number of practical crowds performed after the verification crowdsourcing operation #1 (310) and before the verification crowdsourcing operation #2 (320) The result of a sourcing operation is defined and classified as an invalid crowdsourcing operation.

Therefore, if the operator fails to properly perform the verification crowdsourcing work, the result of the previous practical crowdsourcing work is defined and classified as an invalid crowdsourcing work. And, the reliability of the worker's ability is evaluated lower than before, and the corresponding worker receives the crowdsourcing work for verification faster than before. The verification crowdsourcing job can be transmitted relatively quickly or relatively late depending on the result of the previous verification crowdsourcing job. And, when i is less than or equal to a predetermined value, the worker becomes an unreliable worker, and after that, the crowdsourcing work for verification and the crowdsourcing work for practical use are not allowed.

10 is an exemplary diagram for providing a practical crowdsourcing operation and a verification crowdsourcing operation after a result of a practical crowdsourcing operation is valid.

Referring to FIG. 10, when the result of the verification crowdsourcing operation matches a predetermined answer, the result of the practical crowdsourcing operation is defined and classified as valid.

If the operator properly performs the verification crowdsourcing operation #1 (410), the results of n practical crowdsourcing operations performed before the verification crowdsourcing operation #1 (410) are defined and classified as valid crowdsourcing operations. . In addition, the operator performs a crowdsourcing task for practical use n+j times higher than that of the previous n times, and performs crowdsourcing task #2 420 for verification. Likewise, if the verification crowdsourcing operation #2 (420) is properly performed, n+j practices performed after the verification crowdsourcing operation #1 (410) and before the verification crowdsourcing operation #2 (420) are performed. The outcome of a crowdsourcing operation is defined and classified as a valid crowdsourcing operation.

Accordingly, if the operator properly performs the verification crowdsourcing work, the result of the previous practical crowdsourcing work is defined and classified as a valid crowdsourcing work. And, the reliability of the worker's ability is evaluated higher than before, and the corresponding worker receives the crowdsourcing task for verification later than before. The verification crowdsourcing job can be transmitted relatively quickly or relatively late depending on the result of the previous verification crowdsourcing job. And, when j is greater than or equal to a predetermined value, the worker becomes a reliable worker, and the result of the practical crowdsourcing job is defined as an effective crowdsourcing job without performing the verification crowdsourcing job thereafter.

11 is a schematic configuration diagram of an apparatus for providing a crowdsourcing job for verification based on a practical job according to another embodiment of the present invention.

Referring to FIG. 11, the verification crowdsourcing job providing device based on the actual job includes a verification crowdsourcing job pool construction unit 510, a crowdsourcing group job result receiving unit 520 for practical use, and a crowdsourcing job transmission unit for verification. (530), a verification crowdsourcing operation result receiving unit 540 and a validity determination unit 550.

Components of the apparatus for providing a crowdsourcing task for verification based on the actual task of FIG. 11 correspond to the components of the method described with reference to FIGS. 1 to 10. Hereinafter, in describing the function or operation of each component of the device, a redundant description will be omitted.

The verification crowdsourcing job pool construction unit 510 builds a verification crowdsourcing job pool consisting of verification crowdsourcing jobs.

The practical crowdsourcing group job result receiving unit 520 receives the practical crowdsourcing group job result from the operator terminal.

The verification crowdsourcing job transmission unit 530 selects one or more verification crowdsourcing jobs corresponding to the practical crowdsourcing group jobs from the verification crowdsourcing job pool and transmits them to the operator terminal.

The verification crowdsourcing job result receiving unit 540 receives the results of one or more verification crowdsourcing jobs from the operator terminal.

The validity determination unit 550 determines whether the results of the actual crowdsourcing group task are valid based on the results of one or more verification crowdsourcing tasks.

The steps of a method or algorithm described in connection with an embodiment of the present invention may be implemented directly in hardware, implemented as a software module executed by hardware, or a combination thereof. Software modules include Random Access Memory (RAM), Read Only Memory (ROM), Erasable Programmable ROM (EPROM), Electrically Erasable Programmable ROM (EEPROM), Flash Memory, hard disk, removable disk, CD-ROM, or It may reside on any type of computer-readable recording medium well known in the art to which the present invention pertains.

In the above, embodiments of the present invention have been described with reference to the accompanying drawings, but those of ordinary skill in the art to which the present invention pertains can be implemented in other specific forms without changing the technical spirit or essential features. You can understand. Therefore, the embodiments described above are illustrative in all respects, and should be understood as non-limiting.

10: Client
20: service provider
30: public
32: worker
34: inspector
50: Practical crowdsourcing group work
60: Crowdsourcing for verification
71, 72, 73, 74, 75, 76: pre-processed practical crowdsourcing work
80, 81, 82, 83, 84: crowdsourcing work that passed inspection
510: Verification crowdsourcing work pool construction unit
520: Real-purpose crowdsourcing group work result receiver
530: Crowdsourcing task transmitter for verification
540: Crowdsourcing work result receiver for verification
550: validity determination unit

Claims (10)

As a method performed by a computer,
Establishing a crowdsourcing job pool for verification;
Receiving a result of a practical crowdsourcing group job from an operator terminal, wherein the practical crowdsourcing group job is composed of one or more practical crowdsourcing jobs, receiving a practical crowdsourcing group job result;
Selecting one or more verification crowdsourcing jobs corresponding to the practical crowdsourcing group jobs from the verification crowdsourcing job pool and transmitting them to the operator terminal;
Receiving results of the one or more verification crowdsourcing operations from the operator terminal; And
Based on the results of the one or more verification crowdsourcing operations, determining whether the results of the practical crowdsourcing group work are valid or not,
The verification crowdsourcing job pool includes a plurality of verification crowdsourcing jobs,
The step of building the verification crowdsourcing work pool,
Extracting the results of the crowdsourcing work and the check boxes that have passed the inspection of the inspector from among the results of one or more practical crowdsourcing jobs previously performed in real time while the crowdsourcing work project is in progress;
Generating a crowdsourcing task for verification based on actual work based on a check box that satisfies a predetermined condition among the extracted check box and the result of the crowdsourcing operation and a result of the crowdsourcing operation; And
Including the generated crowdsourcing task for verification based on actual field work in the verification crowdsourcing task pool,
The predetermined conditions are:
It corresponds to a predetermined ranking based on the time spent for inspection,
The step of creating the verification crowdsourcing job,
Extracting a check box and a result of the crowdsourcing operation satisfying a predetermined condition from among the extracted check box and the result of the crowdsourcing operation;
Requesting re-examination of the inspection pass satisfying the predetermined condition and the result of the crowdsourcing operation from one or more inspectors different from the inspector; And
Including the step of generating a crowdsourcing task for verification based on actual work based on a result of a crowdsourcing task that has passed the re-check by all of the one or more inspectors as a result of the re-check,
How to provide a crowdsourcing task for verification based on real-world work. delete delete The method of claim 1,
Transmitting one or more verification crowdsourcing jobs to the operator terminal,
Transmitting at least one verification crowdsourcing job to the worker terminal excluding verification crowdsourcing jobs already provided on the same worker terminal as the worker terminal,
How to provide a crowdsourcing task for verification based on real-world work. The method of claim 1,
When a specific verification crowdsourcing job in the verification crowdsourcing job pool is generated based on the results of one or more practical crowdsourcing jobs received from the same worker terminal as the worker terminal,
Transmitting one or more verification crowdsourcing jobs to the operator terminal,
To transmit one or more verification crowdsourcing jobs excluding the specific verification crowdsourcing job to the operator terminal,
How to provide a crowdsourcing task for verification based on real-world work. The method of claim 1,
The step of building the verification crowdsourcing work pool,
The crowdsourcing task further comprising including in the crowdsourcing task pool for verification, a crowdsourcing task for verification previously generated by the manager prior to performing the project,
How to provide a crowdsourcing task for verification based on real-world work. The method of claim 6,
The step of selecting from the verification crowdsourcing job pool and transmitting it to the operator terminal,
After the previously generated verification crowdsourcing job is exhausted, transmitting the actual job-based verification crowdsourcing job to the operator terminal,
How to provide a crowdsourcing task for verification based on real-world work. The method of claim 1,
The step of determining whether to proceed with the inspection of the result of the practical crowdsourcing group job, according to whether the result of the practical crowdsourcing group job is valid,
The step of determining whether to proceed with the inspection of the results of the practical crowdsourcing group work,
If the result of the verification crowdsourcing work is consistent with the predetermined answer, it is decided to proceed with the inspection of the results of the practical crowdsourcing group work,
If the result of the verification crowdsourcing operation is inconsistent with the predetermined answer, it is determined not to proceed with the inspection by deleting the corresponding practical crowdsourcing group work result,
How to provide a crowdsourcing task for verification based on real-world work. A computer program for providing a crowdsourcing task for verification based on actual work, which is combined with a computer as hardware and stored in a medium to execute the method of any one of claims 1, 4 to 8. A verification crowdsourcing job pool construction unit for building a verification crowdsourcing job pool;
Receives a result of a practical crowdsourcing group job from an operator terminal, wherein the practical crowdsourcing group job is composed of one or more practical crowdsourcing jobs, a practical crowdsourcing group job result receiving unit;
A verification crowdsourcing job transmission unit for selecting one or more verification crowdsourcing jobs corresponding to the practical crowdsourcing group jobs from the verification crowdsourcing job pool and transmitting them to the operator terminal;
A verification crowdsourcing job result receiving unit for receiving the results of the one or more verification crowdsourcing jobs from the operator terminal; And
And a validity determination unit that determines whether a result of the practical crowdsourcing group operation is valid, based on the results of the one or more verification crowdsourcing operations,
The verification crowdsourcing job pool includes a plurality of verification crowdsourcing jobs,
The verification crowdsourcing work pool construction unit,
Among the results of one or more practical crowdsourcing work previously performed in real time while the crowdsourcing work project is in progress, the check box that passed the check by the checker and the result of the crowdsourcing work are extracted,
Create a crowdsourcing task for verification based on actual work based on the check box that satisfies a predetermined condition among the extracted check box and the results of the crowdsourcing work, and the result of the crowdsourcing work,
Including the generated field-based verification crowdsourcing job in the verification crowdsourcing job pool,
The predetermined conditions are:
It corresponds to a predetermined ranking based on the time spent for inspection,
The verification crowdsourcing work pool construction unit,
Extracting the test container and the result of the crowdsourcing operation satisfying a predetermined condition among the extracted test container and the results of the crowdsourcing operation,
Requesting a re-examination of the inspection pass satisfying the predetermined condition and the result of the crowdsourcing operation from one or more inspectors different from the inspector,
To generate a crowdsourcing task for verification based on actual work based on a result of a crowdsourcing task that has passed the re-check by all of the one or more inspectors as a result of the re-check,
A device that provides crowdsourcing work for verification based on actual work.

Images