KR102155747B1 - Method for setting minimum re-work and re-inspection time based on the number of objects returned of crowdsourcing based project for artificial intelligence training data generation - Google Patents

Abstract

The present invention relates to a method for setting minimum re-work time and re-inspection time based on the number of returned objects of a crowdsourcing-based project for artificial intelligence learning data generation. The method includes: a step of assigning crowdsourcing-based work with respect to one project to a worker and performing work request; a step of receiving the input of a work result from the worker by activating a work result input interface after the passage of first minimum work time; a step of requesting inspection of the work result by an inspector and receiving the input of an inspection result from the inspector; a step of requesting re-work by the worker with respect to the returned work result in a case where the result of the inspection by the inspector is return; a step of calculating an object-based return rate with respect to the returned work result; and a step of receiving the input of a re-work result from the worker by activating the work result input interface after the passage of second minimum work time corresponding to the object-based return rate.

Description

A method of setting the minimum rework and re-review time based on the number of objects rejected in a crowdsourcing-based project for artificial intelligence learning data generation {METHOD FOR SETTING MINIMUM RE-WORK AND RE-INSPECTION TIME BASED ON THE NUMBER OF OBJECTS RETURNED OF CROWDSOURCING BASED PROJECT FOR ARTIFICIAL INTELLIGENCE TRAINING DATA GENERATION}

The present invention relates to artificial intelligence learning data generation, and more particularly, to a method of setting a minimum rework and re-review time based on the number of objects rejected in a crowdsourcing-based project.

Recently, more and more companies collect and process large amounts of data on a crowdsourcing basis that engages the general public in some process of corporate activities. In other words, by opening a project and allowing the general public, that is, workers to participate in the project, necessary information is collected through the work results completed by the workers.

And, in order to collect more reliable information, the company assigns the results of work completed by the worker to the inspector to perform the inspection work.

Specifically, when one project is opened, each task is assigned to a plurality of workers. Each worker performs a plurality of tasks assigned to it, and provides the task results.

Thereafter, a plurality of inspection tasks are assigned to each of the plurality of inspectors, and each inspector performs the assigned plurality of inspection tasks.

However, there is an emergency work progress and inspection by some workers and inspectors, and measures to prevent this are necessary.

Unexamined Patent Publication No. 10-2014-0095956, 2014.08.04.

The problem to be solved by the present invention is to provide a method of setting a minimum rework and re-review time based on the number of objects returned.

However, the problem to be solved by the present invention is not limited to the problem as described above, and other problems may exist.

A method according to an aspect of the present invention for solving the above-described problem is the step of allocating a crowdsourcing-based task for one project to a worker and requesting a task, a task result input interface after the first minimum working time has elapsed. Activating and receiving a work result from the operator, requesting an inspector to inspect the work result, receiving the inspection result from the inspector, and when the inspection result by the inspector is rejection processing, the rejected operation result Activating the work result input interface after requesting rework from the operator for the job, calculating an object-based rejection rate for the rejected job result, and after a second minimum working time corresponding to the object-based rejection rate has elapsed And receiving a rework result from the operator.

The method according to another aspect of the present invention for solving the above-described problem is the step of assigning a crowdsourcing-based task for one project to a worker and requesting a task, receiving a task result from the worker, and the inspector. Requesting an inspection on the work result, activating the inspection result input interface after the first minimum inspection time has elapsed, and receiving the job result from the inspector, and if the inspection result by the inspector is rejection processing, the rejected job Requesting a rework from a worker for a result, calculating an object-based rejection rate for the rejected work result, receiving a rework result from the worker, and a second corresponding to the object-based rejection rate And activating an inspection result input interface after the minimum inspection time has elapsed, and receiving a re-examination result from the inspector.

Other specific details of the present invention are included in the detailed description and drawings.

According to the present invention described above, when reworking is performed due to rejection processing as a result of the inspection, the minimum operation time set at the initial progress of the corresponding operation is not applied equally, that is, the rejected operation is not uniformly applied, By calculating the object-based rejection rate of and applying the object-based rejection rate to set a new minimum working time, unnecessary waiting time for workers can be reduced during rework.

In addition, according to the present invention, in the case of re-examination, as in the case of rework, the minimum inspection time set at the initial progress of the inspection is not applied equally when re-examination due to the rejection processing and rework of the inspection result, that is, By not uniformly applying the minimum inspection time, calculating the object-based rejection rate of the rejected job, and setting a new minimum inspection time by applying the object-based rejection rate, it is possible to reduce unnecessary waiting time for inspectors during re-examination.

In conclusion, there is an advantage that the overall progress of the project can be improved by shortening the rework and re-review time due to rejection processing.

The effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the following description.

1 is a conceptual diagram of a crowdsourcing service according to an embodiment of the present invention.
2 is a flowchart illustrating a crowdsourcing-based project progress process according to an embodiment of the present invention.
3 is a flowchart of a method of setting a minimum rework time based on the number of objects returned according to an embodiment of the present invention.
4 is a flowchart of a method of setting a minimum re-examination time based on the number of objects returned according to an embodiment of the present invention.
5A and 5B are diagrams for explaining an example of setting a re-examination time according to an embodiment of the present invention.
6 is a diagram illustrating an apparatus for setting a minimum rework and re-review time based on the number of objects returned according to an embodiment of the present invention.

Advantages and features of the present invention, and a method of achieving them will become apparent with reference to the embodiments described below in detail together with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but may be implemented in a variety of different forms, only the present embodiments are intended to complete the disclosure of the present invention, It is provided to fully inform the technician of the scope of the present invention, and the present invention is only defined by the scope of the claims.

The terms used in the present specification are for describing exemplary embodiments and are not intended to limit the present invention. In this specification, the singular form also includes the plural form unless specifically stated in the phrase. As used in the specification, “comprises” and/or “comprising” do not exclude the presence or addition of one or more other elements other than the mentioned elements. Throughout the specification, the same reference numerals refer to the same elements, and “and/or” includes each and all combinations of one or more of the mentioned elements. Although "first", "second", and the like are used to describe various elements, it goes without saying that these elements are not limited by these terms. These terms are only used to distinguish one component from another component. Therefore, it goes without saying that the first component mentioned below may be the second component within the technical idea of the present invention.

Unless otherwise defined, all terms (including technical and scientific terms) used in the present specification may be used as meanings that can be commonly understood by those of ordinary skill in the art to which the present invention belongs. In addition, terms defined in a commonly used dictionary are not interpreted ideally or excessively unless explicitly defined specifically.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a conceptual diagram of a crowdsourcing service according to an embodiment of the present invention.

Referring to FIG. 1, a crowdsourcing service is performed by being composed of a client 10, a service provider 20, and a public 30.

The sponsor 10 refers to a company or individual who requests a crowdsourcing-based project (hereinafter, referred to as a project).

The requester 10 requests a project for the purpose of collecting source data or annotating data. Collection of source data refers to collecting raw data such as recorded voice collection and photo collection. Data annotation refers to inputting relevant annotation data into source data such as text, photos, and videos. For example, the data annotation may include finding an entity in a given fingerprint or finding a similar sentence, but is not limited thereto. Meanwhile, the types of the above-described projects are only one embodiment, and various projects may be handled in the present invention according to the design of the client.

The service provider 20 refers to a company that provides crowdsourcing services.

When the service provider 20 receives a request for a project for a product or service from the client 10, the service provider 20 allocates a task for the project to the general public 30 and receives the work result from the public 30. Thereafter, the final product extracted based on the work result is provided to the client 10.

At this time, the service provider 20 provides a crowdsourcing service to the client 10 and the public 30 through a crowdsourcing platform (hereinafter, the platform). That is, when the service provider 20 receives a request for a project from the client 10, the service provider 20 opens the project on the platform. Thereafter, when the work result for the open project is received from the public 30, the project is terminated on the platform, and the final product may be extracted and provided to the client 10.

The public 30 refers to the general public participating in the project open on the platform. Here, the public 30 may participate in a project opened on the platform through an application or website provided by the service provider 20.

The public 30 is composed of a worker 32 and an inspector 34.

The worker 32 decides to participate in a specific project among a plurality of projects open on the platform. Thereafter, the operator 32 performs an operation such as collection of source data or data annotation, and transmits it to the platform.

The inspector 34 decides to participate in a specific project among a plurality of projects open on the platform. Thereafter, the inspector 34 inspects the result of the work performed by the operator 32. As a result of performing the inspection, the inspector 34 may pass or reject the inspection, and may input a reason for rejection during rejection processing.

2 is a flowchart illustrating a crowdsourcing-based project progress process according to an embodiment of the present invention.

First, the requester 10 requests one or more projects to the service provider 20 (S11).

Thereafter, the service provider 20 opens the requested project on the platform (S12). At this time, the service provider 20 may determine the grade in consideration of the difficulty of the project before opening the project. That is, depending on the difficulty level, it is possible to determine which level or higher public 30 to expose the project. Accordingly, it is possible to increase the reliability of the work result of the project.

Thereafter, the service provider 20 requests the work by assigning the work to the workers 32 of the corresponding level or higher according to the level of the project (S13).

Thereafter, the worker 32 performs the assigned task (S14). In this case, the worker 32 may input the reason for the inability to work without performing the work for a work in which the work itself is impossible for some reason.

Thereafter, the service provider 20 receives the work result from the worker 32 (S15), and assigns the work result to the inspector 34 to request the inspection (S16).

Thereafter, the inspector 34 performs the assigned inspection (S17). At this time, the inspector 34 decides to complete the inspection when it is determined that the work has been properly performed, and if it is determined that the inspection work is wrong, the inspector 34 rejects it. During rejection processing, the inspector 34 inputs the rejection reason for what reason the job was judged to be wrong.

Thereafter, the service provider 20 receives the inspection result from the inspector 34 (S18).

When the inspection result is the inspection completion, the service provider 20 uses the work result as valid data, and based on this, extracts the final product at the end of the project.

When the inspection result is rejection processing, the service provider 20 may internally perform the inspection again, or assign the work to the worker 32 again to perform the rework. When reworking, re-examination by an inspector is required.

Thereafter, when the project period ends or sufficient valid data is secured, the service provider 20 terminates the project (S19), calculates a final result based on the secured valid data, and provides it to the client 10 ( S20).

At this time, before the end of the project, the service provider 20 evaluates the performance results of the worker 32 and the inspector 34, calculates the work cost and the inspection cost according to the evaluation, to the worker 32 and the inspector 34. give.

In FIGS. 1 and 2, it is simply expressed as a requester 10, a service provider 20, an operator 32, and an inspector 34, but these are smartphones, tablets, PDAs, laptops, desktops, operated by each participant. It means a computer device such as a server or a telecommunication device.

On the other hand, in the current system, in order to prevent an emergency operation and inspection, the minimum time required to perform one operation and inspection is set.

For example, if the minimum working time is 1 minute, the operator can complete the work only after 1 minute, and if the minimum inspection time is 30 seconds, the inspector can complete the inspection after 30 seconds.

The operation result input interface was activated only after the minimum operation time had elapsed, and the inspection result input interface was activated only after the minimum inspection time had elapsed.

However, in a project that includes many objects in a single task, if it is rejected because of only a few objects, a situation occurs in which the worker has to wait as much as the minimum work time unnecessarily in the process of reworking. There is a problem of slowing down.

For example, if there are 30 objects per source and the minimum work time is 5 minutes, and the work result is rejected by 1 object, it takes less than 1 minute to rework 1 object, but at least You have to wait 5 minutes due to the working time setting.

This problem occurs equally to the inspector, and as a result, there is a problem that the fatigue of the inspector increases rapidly.

Therefore, it is necessary to separately apply the minimum rework time and re-review time for the rejected work.

Hereinafter, a method and apparatus for setting a minimum rework and re-review time based on the number of returned objects according to an embodiment of the present invention will be described with reference to FIGS. 3 to 6.

3 is a flowchart of a method of setting a minimum rework time based on the number of objects returned according to an embodiment of the present invention. 4 is a flowchart of a method of setting a minimum re-examination time based on the number of objects returned according to an embodiment of the present invention. 5A and 5B are diagrams for explaining an example of setting a re-examination time according to an embodiment of the present invention.

Meanwhile, the steps illustrated in FIGS. 3 and 4 may be understood as being performed by a platform server (hereinafter, referred to as a server) operated by the service provider 20, but are not limited thereto.

In addition, a plurality of workers 32 or a plurality of inspectors 34 perform work using a predetermined terminal device. Terminal devices of the operator 32 and the inspector 34 may be a computer device or a telecommunication device such as a smart phone, tablet, PDA, laptop, desktop, etc., but is not limited thereto.

For convenience of explanation, some steps overlapping with FIG. 2 are not shown in FIGS. 3 and 4.

First, referring to FIG. 3, the server creates a crowdsourcing-based task for one project, assigns it to a plurality of workers 32, and requests a task (S110).

Here, one project consists of a plurality of data sets, and one data set contains a plurality of tasks. The plurality of jobs may be grouped into a plurality of tasks according to types, or each job may be a different type of job or the same job, but is not limited thereto.

In this case, in one embodiment of the present invention, each task is configured to include one or more objects.

An object is a unit of work that composes a work, and refers to a unit of work that can determine whether to pass or reject inspection within a work. For example, in the case of completing a plurality of sentences according to a scenario, one sentence may be one object, and in the case of annotating an area that satisfies a specific condition in a specific image, one area that satisfies the condition is one It may be an object of. In other words,

When the operator 32 receives a rejection processing decision from the inspector 34 for any one of a plurality of objects constituting one job, the entire job is rejected.

These objects can be pre-set and configured by the project sponsor and assigned to the workers as tasks.

Next, after the first minimum work time has elapsed, the completed work results are inputted from the plurality of workers 32 (S120).

In this case, according to an embodiment of the present invention, an interface for receiving a work result may be activated only after the first minimum work time has elapsed. That is, in order for the worker 32 to transmit the work result to the server, the first minimum work time must elapse.

For example, the interface for inputting work results is separate from the interface provided to the operator to perform work such as collection of source data or data annotation, and an interface for completing input of the work result or moving to the next work. Can be

In an embodiment, the first minimum working time may be determined based on a standard hourly wage and a unit price for each job.

As an example, the first minimum work time may be determined based on the number of work performed per hour that can earn a predetermined multiple of the standard hourly wage. If the minimum hourly wage is 8,350 won, which is the minimum hourly wage, and 1.5 times the predetermined multiple is applied as the standard hourly wage, it is determined based on the amount of work performed per hour that can earn 12,525 won.

The unit price for each job is a unit price that is applied for each job. When the difficulty is high, the unit cost is high, and when the difficulty is low, the unit price is decreased. When the difficulty of the work is high, a higher quality is required, so that the first minimum work time is also required, and when the work difficulty is low, the opposite is true.

Accordingly, the first minimum working time may be determined as in Equation 1 below.

[Equation 1]

1st minimum working time = 3600 sec/((standard hourly wage * constant rate)/unit price per job)

If the unit price for each job is set to 1,000 won, the first minimum working time in the above example is about 287 seconds, which is about 5 minutes.

As the work result is input from the worker 32, the test result is assigned to a plurality of inspectors 34 to receive the test result, and the rejected work result is requested to be reworked from the worker 32 Do (S130).

Next, the object-based rejection rate is calculated for the rejected work result (S140), and the completed rework result is input from the reworked worker 32 after the second minimum work time has elapsed (S150).

Like the first minimum working time, according to an embodiment of the present invention, the interface for receiving the rework result may be activated only after the second minimum working time has elapsed. That is, in order for the worker 32 to transmit the rework result to the server, the second minimum working time must elapse.

In an embodiment, the second minimum working time may be determined by applying an object-based rejection rate to the first minimum working time.

Here, the object-based rejection rate is a rejection rate based on the number of rejected objects among one or more objects included in the rejected job, and is different from the rejection rate, which is usually measured in worker units or inspectors. Typically, the rejection rate is 0.3 when one worker performs 10 tasks and 3 of them are rejected.

The second minimum working time to which the object-based rejection rate is applied may be calculated according to Equation 2.

[Equation 2]

Second minimum working time = 3600 seconds/((standard hourly wage * constant rate)/unit price per job)*object-based rejection rate

In the above example, if one rejected job includes 10 objects and 3 of them are rejected, the object-based rejection rate is 0.3.

Applying this to Equation 2, the second minimum working time is 86 seconds, which is about 1 minute and 30 seconds.

In other words, the operator 32 had to wait for about 5 minutes in the past to input the rework result of modifying only three objects, but the actual object-based rejection rate based on the object unit It has the advantage of reducing the overall work time, as it takes only about 1 minute and 30 seconds, which is the time required for the work.

On the other hand, if the work unit price for all tasks belonging to one project is the same, the first minimum work time is the same for all tasks, but the second minimum work time is object-based rejection rate because object-based rejection rate is applied. If these are different, they may be set differently for each task.

In contrast, when the work unit cost for each work belonging to one project is different, the first and second minimum work hours may be set differently for each work.

This embodiment of the present invention not only shortens the rework time of the worker 32, but also applies the same to the inspector 34, so that the re-examination time can be shortened.

Referring to FIG. 4, the server generates a sourcing-based job for one project, assigns it to a plurality of workers 32 to request a job, receives a job result from the worker 32, and receives a plurality of inspectors 34 ) To request inspection (S210).

Next, after the first minimum inspection time has elapsed, the completed inspection results are received from the plurality of inspectors 34 (S220).

In this case, according to an exemplary embodiment of the present invention, an interface for inputting an inspection result may be activated only after the first minimum inspection time has elapsed. That is, the inspector 34 can transmit the inspection result to the server only after the first minimum inspection time has elapsed.

For example, the interface for inputting the inspection result is separate from the interface in which the operator's work results are provided to the inspector to perform the inspection, and the result of the inspection of the operator's work result is determined or'reviewed' It may be an interface for determining and inputting'reject'.

In an embodiment, the first minimum inspection time may be determined based on a standard hourly wage and a unit price per inspection.

As an example, the first minimum inspection time may be determined based on the number of inspections performed per hour that can earn a predetermined multiple of the standard hourly wage. If the minimum hourly wage is 8,350 won, which is the minimum hourly wage, and 1.5 times the predetermined multiple is applied as the standard hourly wage, it is determined based on the number of inspections performed per hour that can earn 12,525 won.

The unit price for each inspection is a unit price that is applied for each inspection. If the inspection difficulty is high, the inspection unit price increases, and when the difficulty level is low, the inspection unit price decreases. If the inspection difficulty is high, a higher quality is required, so the first minimum inspection time is also required, and when the inspection difficulty is low, the opposite is true.

Accordingly, the first minimum inspection time may be determined as in Equation 3 below.

[Equation 3]

1st minimum inspection time = 3600 seconds/((standard hourly wage * constant rate)/unit price per inspection)

If the unit price for each inspection is set to 1,500 won, the first minimum inspection time in the example above is about 431 seconds, which is about 7 minutes.

When the inspection result is input, rework is requested from the worker 32 for the rejected work result, and next, an object-based rejection rate is calculated for the rejected work result (S230).

When the rework result is received from the worker 32, the re-review is assigned to the inspector 34 with respect to the rework result, and re-review is requested (S240).

After the second minimum inspection time has elapsed, the completed re-examination result is input from the re-examined inspector 34 (S250).

Like the first minimum inspection time, in an embodiment of the present invention, an interface for inputting a re-examination result may be activated only after the second minimum inspection time has elapsed. That is, the inspector 34 may submit the re-examination result to the server after the second minimum inspection time has elapsed.

In an embodiment, the second minimum inspection time may be determined by applying an object-based rejection rate to the first minimum inspection time.

In this case, the second minimum inspection time to which the object-based rejection rate is applied may be calculated according to Equation 4 as in the case of the second minimum working time.

[Equation 4]

2nd minimum inspection time = 3600 seconds/((standard hourly wage * constant rate)/unit price per inspection)* object-based rejection rate

5A and 5B show the inspection screen of the inspector 34, one job is composed of 40 objects, and after the first inspection and rejection, the rework result of the worker 32 is received and re-examined. It represents a situation that needs to be done.

The inspector 34 only needs to re-examine 9 objects in a state in which 9 objects out of 40 objects are rejected as a result of the first inspection for the corresponding job.

At the time of the initial inspection, the inspector 34 inspects the entire work by dedicating about 7 minutes according to the first minimum inspection time.

However, if the second minimum inspection time is not applied, there arises a problem in that the inspection can be completed only when 7 minutes of time is unconditionally allocated even in a situation where only 9 objects are inspected.

In order to solve this problem, an embodiment of the present invention applies only about 1 minute and 30 seconds according to the second minimum inspection time calculated by applying 22.5%, which is the rejection rate per object, to the first minimum inspection time. You will be able to submit the re-examination results.

On the other hand, as in the case of the first and second minimum working hours, if the inspection unit price for all tasks belonging to one project is the same, the first minimum inspection time is the same for all tasks, but the second minimum inspection time is the object Since the rejection rate based on is applied, if the object-based rejection rate is different, it may be set differently for each task.

In contrast, when the inspection unit price for each job belonging to one project is different, the first and second minimum inspection times may be set differently for each job.

Meanwhile, in the above description, steps S11 to S240 may be further divided into additional steps or may be combined into fewer steps, according to an embodiment of the present invention. In addition, some steps may be omitted as necessary, and the order between steps may be changed. In addition, even if other contents are omitted, the contents of FIG. 6 to be described later may also be applied to the rework and re-examination time setting methods of Figs. 1 to 5B.

Hereinafter, with reference to FIG. 6, an apparatus 100 for setting a minimum rework and re-review time based on the number of rejected objects according to an embodiment of the present invention will be described.

6 is a diagram for explaining an apparatus 100 for setting a minimum rework and re-review time based on the number of objects returned according to an embodiment of the present invention.

Referring to FIG. 6, an apparatus 100 for setting a minimum rework and re-review time based on the number of returned objects includes a communication module 110, a memory 120, and a processor 130.

The communication module 110 transmits a crowdsourcing-based work for one project to a plurality of workers 32 and receives work results and rework results from the plurality of workers 32. Then, the work results and rework results input from the plurality of workers 32 are transmitted to the plurality of inspectors 34, and the inspection results and re-examination results are received from the plurality of inspectors 34.

A program for setting a minimum rework and re-review time based on the number of objects included in the work is stored in the memory 120.

The processor 130 executes a program stored in the memory 120. As the processor 130 executes the program stored in the memory 120, it is possible to input the work result after the first minimum work time has elapsed from the plurality of workers 32, and the second minimum work time has elapsed. After that, make it possible to input the rework result.

Further, the processor 130 enables the input of the inspection result after the first minimum inspection time has elapsed from the plurality of inspectors 34, and inputs the re-review result after the second minimum inspection time has elapsed.

Here, the second minimum work time and the second minimum inspection time are calculated based on the number of rejected objects among objects included in the rejected job.

The apparatus 100 for setting the minimum rework and re-review time based on the number of rejected objects described with reference to FIG. 6 may be provided as a component of the above-described server.

The method of setting the minimum rework and re-review time based on the number of returned objects according to the embodiment of the present invention described above is implemented as a program (or application) to be executed in combination with a computer that is hardware Can be stored in.

The above-described program includes C, C++, JAVA, Ruby, which can be read by a processor (CPU) of the computer through the device interface of the computer, in order for the computer to read the program and execute the methods implemented as a program. It may include a code (Code) coded in a computer language such as machine language. Such code may include a functional code related to a function that defines necessary functions for executing the methods, and includes a control code related to an execution procedure necessary for the processor of the computer to execute the functions according to a predetermined procedure. can do. In addition, these codes may further include additional information required for the processor of the computer to execute the functions, or code related to a memory reference to which location (address address) of the internal or external memory of the computer should be referenced. have. In addition, when the processor of the computer needs to communicate with any other computer or server in the remote in order to execute the functions, the code uses the communication module of the computer to determine how It may further include a communication-related code for whether to communicate, what kind of information or media should be transmitted and received during communication.

The stored medium is not a medium that stores data for a short moment, such as a register, cache, memory, etc., but a medium that stores data semi-permanently and can be read by a device. Specifically, examples of the storage medium include, but are not limited to, ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, and the like. That is, the program may be stored in various recording media on various servers to which the computer can access, or on various recording media on the user's computer. In addition, the medium may be distributed over a computer system connected through a network, and computer-readable codes may be stored in a distributed manner.

The above description of the present invention is for illustrative purposes only, and those of ordinary skill in the art to which the present invention pertains will be able to understand that it can be easily modified into other specific forms without changing the technical spirit or essential features of the present invention. will be. Therefore, it should be understood that the embodiments described above are illustrative in all respects and not limiting. For example, each component described as a single type may be implemented in a distributed manner, and similarly, components described as being distributed may also be implemented in a combined form.

The scope of the present invention is indicated by the claims to be described later rather than the detailed description, and all changes or modified forms derived from the meaning and scope of the claims and their equivalent concepts should be interpreted as being included in the scope of the present invention. do.

10: Client
20: service provider
30: public
32: worker
34: inspector
100: Minimum rework and retest time setting device
110: communication module
120: memory
130: processor

Claims (12)

As a method performed by a crowdsourcing platform server computer (hereinafter, a server),
Requesting a job by allocating, by the server, a crowdsourcing-based job for one project to a worker;
Receiving, by the server, a job result input interface from the worker by activating a job result input interface after a first minimum job limit time elapses;
Receiving, by the server, a request for inspection of the work result from an inspector, and receiving an inspection result from the inspector;
Requesting, by the server, an operator for rework of the rejected job result when the result of the inspection by the inspector is rejection processing;
Calculating, by the server, an object-based rejection rate for the rejected job result; And
And receiving, by the server, activating a work result input interface after a second minimum work limit time corresponding to the object-based rejection rate has elapsed, and receiving a rework result from the worker,
The object-based rejection rate is calculated based on the number of rejected objects among one or more objects included in the job,
How to set a minimum rework time based on the number of objects dropped. The method of claim 1,
The first minimum work time limit is determined based on a standard hourly wage and a unit price per work,
How to set a minimum rework time based on the number of objects dropped. The method of claim 1,
The second minimum work limit time is determined by applying the object-based rejection rate to the first minimum work limit time,
How to set a minimum rework time based on the number of objects dropped. delete The method of claim 1,
The first minimum work time limit for the first work and the second work having the same work unit cost are set the same,
According to different object-based rejection rates, the second minimum task time limit of the first task and the second task having the same work unit cost are set differently from each other,
How to set a minimum rework time based on the number of objects dropped. As a method performed by a crowdsourcing platform server computer (hereinafter, a server),
Requesting a job by allocating, by the server, a crowdsourcing-based job for one project to a worker;
Receiving, by the server, a work result from the worker;
Requesting, by the server, an inspector to inspect the work result;
Receiving, by the server, a work result from the inspector by activating an inspection result input interface after a first minimum inspection time has elapsed;
Requesting, by the server, an operator for rework of the rejected job result when the result of the inspection by the inspector is rejection processing;
Calculating, by the server, an object-based rejection rate for the rejected job result;
Receiving, by the server, a rework result from the worker; And
And receiving, by the server, activating an inspection result input interface after a corresponding second minimum inspection time has elapsed according to the object-based rejection rate, and receiving a re-examination result from the inspector,
The object-based rejection rate is calculated based on the number of rejected objects among one or more objects included in the job,
How to set the minimum retest time based on the number of rejected objects. The method of claim 6,
The first minimum inspection time limit is determined based on a standard hourly wage and a unit price per inspection,
How to set the minimum retest time based on the number of rejected objects. The method of claim 6,
The second minimum inspection time limit is determined by applying the object-based rejection rate to the first minimum inspection time limit,
How to set the minimum retest time based on the number of rejected objects. delete The method of claim 6,
The first minimum inspection time limit for the first task and the second task having the same work unit cost is set equally,
According to different object-based rejection rates, the second minimum inspection time limit of the first task and the second task having the same work unit cost are set differently,
How to set the minimum retest time based on the number of rejected objects. A computer program stored in a computer-readable recording medium to execute a method for setting a minimum rework time based on the number of objects rejected in any one of claims 1 to 3 and 5. A computer program stored in a computer-readable recording medium to execute a method for setting a minimum re-examination time based on the number of rejected objects in any one of claims 6 to 8 and 10.

Images