KR102195607B1 - Method for setting the deadline for rework and granting a penalty for crowdsourcing based project for artificial intelligence training data generation - Google Patents

Abstract

Provided is a method for setting a deadline for rework and assigning penalty for a crowdsourcing-based project for generating artificial intelligence training data to prevent the project from being delayed. According to the present invention, the method comprises the following steps: assigning a plurality of works of a crowdsourcing-based project to a plurality of workers to request execution of work; receiving a plurality of work results from the plurality of workers; allocating the plurality of work results to a plurality of inspectors to request execution of inspection; receiving a plurality of inspection results for the plurality of work results from the plurality of inspectors as inspection pass or rejection; when the result of the inspection for the work result is a rejection, assigning rework corresponding to the work result to a worker who has entered the work result to request execution of rework; setting a deadline of the rework to a point in time when a preset predetermined period has elapsed from the time when the worker is requested to execute the rework; and when the worker does not start executing the rework until the deadline of the rework, canceling the rework assigned to the worker and assigning the rework to another worker to request execution of the rework.

Description

{METHOD FOR SETTING THE DEADLINE FOR REWORK AND GRANTING A PENALTY FOR CROWDSOURCING BASED PROJECT FOR ARTIFICIAL INTELLIGENCE TRAINING DATA GENERATION} for the creation of artificial intelligence learning data

The present invention relates to a method for setting a limit for rework of a crowdsourcing-based project for generating artificial intelligence learning data and giving a penalty.

Recently, more and more companies collect and process large amounts of data on a crowdsourcing basis that engages the general public in some process of corporate activities. In other words, by opening a project and allowing the general public, that is, workers to participate in the project, necessary information is collected through the work results completed by the workers.

When one project is opened, multiple tasks are assigned to each of the multiple workers. Each worker performs a plurality of tasks assigned to it, and provides the task results. Thereafter, a plurality of inspection tasks are assigned to each of the plurality of inspectors, and each inspector performs a plurality of inspection tasks assigned to them.

At this time, the work that was rejected as a result of the inspection by the inspector requires the corresponding worker to rework. In this case, if a number of rework is assigned while the worker does not perform the rework quickly, a bottleneck for performing the rework occurs from the worker, which leads to a delay in the entire project.

In addition, when a number of rework is assigned, if sufficient time for performing rework for each rework is not guaranteed, a problem that may lead to a worker's departure from rework while performing rework occurs.

Unexamined Patent Publication No. 10-2014-0095956, 2014.08.04.

The problem to be solved by the present invention is to provide a method for setting a limit period for rework and granting a penalty for a crowdsourcing-based project for generating artificial intelligence learning data.

However, the problem to be solved by the present invention is not limited to the problem as described above, and other problems may exist.

In order to solve the above-described problems, a method for setting a limit for rework and granting a penalty for a crowdsourcing-based project according to an aspect of the present invention is by assigning a plurality of tasks of a crowdsourcing-based project (hereinafter, a project) to a plurality of workers. Requesting to perform a task, receiving a plurality of work results from the plurality of workers, assigning the plurality of work results to a plurality of inspectors to request inspection, and the plurality of work results from the plurality of inspectors Receiving a plurality of inspection results for the inspection pass or rejection, and if the inspection result for the work result is rejected, rework corresponding to the work result is assigned to the worker who entered the work result and reworked Requesting to perform, setting a limit for rework to a point in time when a predetermined period of time has elapsed from the time when the worker requested to perform the rework, and the worker performs the rework until the limit for rework If not initiated, canceling the assignment of the rework to the worker, assigning the rework to another worker, and requesting to perform the rework.

In addition, the step of canceling the assignment of the rework to the worker and allocating the rework to another worker to request the rework is performed by the worker until the Nth (where, N is a natural number of 1 or more) rework limit If the N-th rework is initiated and the N-th rework is being performed, even if the operator does not initiate the N+M-th rework by the N+Mth (where M is a natural number of 1 or more) rework limit, Assigning the N+Mth rework to the worker may not be canceled.

In addition, the step of setting the rework limit period includes calculating the N-th time required for rework by the worker, and a predetermined multiple of the N-th time required for rework from the time when the N-th rework result is input. It may include the step of resetting the N+1 th rework limit period to one point in time.

In addition, the step of resetting the N+1th rework limit period may include a predetermined amount of the Nth rework time required from the time when the N+1th rework limit and the Nth rework result are received before resetting. Comparing the time when a multiple of the has elapsed, and as a result of the comparison, if the N+1th rework limit is earlier than the reset, the N+1th rework limit is reset, and before the reset If the N+1 th rework limit period is not earlier, the step of not resetting the N+1 th rework limit period may be included.

In addition, the step of canceling the assignment of the rework to the worker and allocating the rework to another worker to request the rework performance, by the worker, the N+1 until the reset N+1 th rework limit deadline. If the N+1th rework is being performed by starting the second rework, even if the worker does not start performing the N+Lth rework by the N+Lth (however, L is a natural number of 2 or more) , It is possible not to cancel the assignment of the N+L-th rework to the worker.

In addition, the step of canceling the assignment of the rework to the worker and assigning the rework to another worker to request rework, the worker performs the rework until the reset N+1th rework limit deadline If not initiated, the assignment of the N+1th rework to the worker may be canceled, and the N+1th rework may be assigned to another worker to request rework.

In addition, the step of setting the rework limit time limit, if the operator does not start performing the rework until the reset N+1th rework limit time limit, calculating the N-th time required for rework of the worker And resetting the N+2 th rework limit to a point in time when a predetermined multiple of the N th rework time has elapsed from the reset N+1 th rework limit.

In addition, the step of resetting the N+2 th rework limit period may include the N+2 th rework limit time limit and the N-th rework time required from the reset N+1 th rework limit time limit. Comparing a time point at which a predetermined multiple has elapsed, and if the comparison result, the N+2 th rework limit period is earlier before the reset, the N+2 th rework limit period is reset, and the reset If the previous limit of the N+2 rework is not earlier, the step of not resetting the limit of the N+2 rework may be included.

In addition, the predetermined multiple may be determined in advance according to the difficulty of the project.

A computer program according to another aspect of the present invention for solving the above-described problems is combined with a computer that is hardware to execute the method of setting a rework limit and granting a penalty for the crowdsourcing-based project, and is stored in a computer-readable recording medium. do.

Other specific details of the present invention are included in the detailed description and drawings.

According to the present invention described above, by canceling rework assignments to workers who have not faithfully performed rework among workers participating in a specific project and assigning them to new workers, the bottleneck of performing rework is quickly solved and finally To avoid project delays.

In addition, when an operator begins to perform a specific rework, the penalty related to the rework restriction deadline for rework assigned to the next turn is exempted, so that the operator is not bound by the rework restriction deadline for rework assigned to the next turn. By doing so, it is possible to induce a worker to initiate sequential rework, and consequently, to achieve project completion through rapid rework progress.

In addition, by calculating the time required for rework and resetting the rework limit for the next rework based on this, the operator can be guaranteed the minimum time required to perform the next rework. As a result, even when a number of rework is assigned to an operator, a sufficient time limit for rework is given for each rework, thereby preventing rework deviation and increasing the quality of rework and the efficiency of performing rework. have.

The effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the following description.

1 is a conceptual diagram of a crowdsourcing service according to an embodiment of the present invention.
2 is a flowchart illustrating a crowdsourcing-based project progress process according to an embodiment of the present invention.
3 is a flowchart schematically showing a method of setting a limit for rework and granting a penalty for a crowdsourcing-based project according to an embodiment of the present invention.
FIG. 4 is an exemplary diagram of a time limit for each rework when two rework are sequentially assigned to an operator.
5 is a flowchart schematically illustrating a method of resetting a rework limit period of an N+1 th rework according to an embodiment of the present invention.
6 is an exemplary diagram illustrating a method of resetting a rework limit period of an N+1 th rework according to an embodiment of the present invention.
FIG. 7 is an exemplary diagram illustrating a method of resetting a rework limit time limit for an N+1th rework based on a value twice the Nth rework required time according to an embodiment of the present invention.
8 is a view for explaining that the rework limitation period is shortened by resetting the N+1th rework limitation period.
9 is a schematic flowchart of a method of resetting an N+1th rework limit compared to a rework limit before resetting according to an embodiment of the present invention.
FIG. 10 is an exemplary diagram of a time limit for each rework when three rework is sequentially assigned to an operator.
11 is a flowchart schematically illustrating a method of resetting a rework limit period for an N+2 rework according to an embodiment of the present invention.
12 is an exemplary view illustrating a method of resetting a rework limit period for an N+2 rework according to an embodiment of the present invention.
13 is a view for explaining that the rework limitation period is shortened by resetting the N+2th rework limitation period.
14 is a flowchart schematically illustrating a method of resetting the N+2th rework limit period compared to the rework limit period before resetting according to an embodiment of the present invention.
FIG. 15 is an exemplary diagram of a rework limitation period of each rework when a plurality of rework is assigned to an operator.
16 is an exemplary view illustrating a method of resetting a rework limit time limit for a plurality of rework according to an embodiment of the present invention.
17 is a diagram for explaining an apparatus for setting a limit for rework and granting a penalty for a crowdsourcing-based project according to an embodiment of the present invention.

Advantages and features of the present invention, and a method of achieving them will become apparent with reference to the embodiments described below in detail together with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but may be implemented in a variety of different forms, only the present embodiments are intended to complete the disclosure of the present invention, It is provided to fully inform the technician of the scope of the present invention, and the present invention is only defined by the scope of the claims.

The terms used in the present specification are for describing exemplary embodiments and are not intended to limit the present invention. In this specification, the singular form also includes the plural form unless specifically stated in the phrase. As used in the specification, “comprises” and/or “comprising” do not exclude the presence or addition of one or more other elements other than the mentioned elements. Throughout the specification, the same reference numerals refer to the same elements, and “and/or” includes each and all combinations of one or more of the mentioned elements. Although "first", "second", and the like are used to describe various elements, it goes without saying that these elements are not limited by these terms. These terms are only used to distinguish one component from another component. Therefore, it goes without saying that the first component mentioned below may be the second component within the technical idea of the present invention.

Unless otherwise defined, all terms (including technical and scientific terms) used in the present specification may be used as meanings that can be commonly understood by those of ordinary skill in the art to which the present invention belongs. In addition, terms defined in a commonly used dictionary are not interpreted ideally or excessively unless explicitly defined specifically.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a conceptual diagram of a crowdsourcing service according to an embodiment of the present invention.

Referring to FIG. 1, a crowdsourcing service is performed by being composed of a client 10, a service provider 20, and a public 30.

The sponsor 10 refers to a company or individual who requests a crowdsourcing-based project (hereinafter, referred to as a project).

The requester 10 requests a project for the purpose of collecting source data or annotating data for generating artificial intelligence learning data. The data generated through the project can be used as learning data for arbitrary machine learning such as supervised learning, unsupervised learning, and reinforcement learning. Collection of source data refers to collecting raw data such as recorded voice collection and photo collection. Data annotation refers to inputting relevant annotation data into source data such as text, photos, and videos. For example, the data annotation may include finding an entity in a given fingerprint or finding a similar sentence, but is not limited thereto. Meanwhile, the types of the above-described projects are only one embodiment, and various projects may be handled in the present invention according to the design of the client.

The service provider 20 refers to a company that provides crowdsourcing services.

When the service provider 20 receives a request for a project for a product or service from the client 10, the service provider 20 allocates a task for the project to the general public 30 and receives the work result from the public 30. Thereafter, the final product extracted based on the work result is provided to the client 10.

At this time, the service provider 20 provides a crowdsourcing service to the client 10 and the public 30 through a crowdsourcing platform (hereinafter, the platform). That is, when the service provider 20 receives a request for a project from the client 10, the service provider 20 opens the project on the platform. Thereafter, when the work result for the open project is received from the public 30, the project is terminated on the platform, and the final product may be extracted and provided to the client 10.

The public 30 refers to the general public participating in the project open on the platform. Here, the public 30 may participate in a project opened on the platform through an application or website provided by the service provider 20.

The public 30 is composed of a worker 32 and an inspector 34.

The worker 32 decides to participate in a specific project among a plurality of projects open on the platform. Thereafter, the operator 32 performs an operation such as collection of source data or data annotation, and transmits it to the platform.

The inspector 34 decides to participate in a specific project among a plurality of projects open on the platform. Thereafter, the inspector 34 inspects the result of the work performed by the operator 32. As a result of performing the inspection, the inspector 34 may perform inspection pass processing or rejection processing, and input a rejection reason during rejection processing. In the case of passing the inspection, it is not necessary to rework and re-examination due to this, so passing inspection has the same meaning as completion of inspection.

2 is a flowchart illustrating a crowdsourcing-based project progress process according to an embodiment of the present invention.

First, the requester 10 requests one or more projects to the service provider 20 (S11).

Thereafter, the service provider 20 opens the requested project on the platform (S12). At this time, the service provider 20 may determine the grade in consideration of the difficulty of the project before opening the project. That is, depending on the difficulty level, it is possible to determine which level or higher public 30 to expose the project. Accordingly, it is possible to increase the reliability of the work result of the project.

Thereafter, the service provider 20 requests the work by assigning the work to the workers 32 of the corresponding level or higher according to the level of the project (S13).

Thereafter, the worker 32 performs the assigned task (S14). In this case, the worker 32 may input the reason for the inability to work without performing the work for a work in which the work itself is impossible for some reason.

Thereafter, the service provider 20 receives the work result from the worker 32 (S15), and assigns the work result to the inspector 34 to request the inspection (S16).

Thereafter, the inspector 34 performs the assigned inspection (S17). At this time, the inspector 34 decides to pass the inspection when it is determined that the work has been properly performed, and if it is determined that the inspection work is wrong, the inspector 34 rejects it. During rejection processing, the inspector 34 inputs the rejection reason for what reason the job was judged to be wrong.

Thereafter, the service provider 20 receives the inspection result from the inspector 34 (S18).

If the inspection result passes the inspection, the service provider 20 uses the work result as valid data, and based on this, extracts the final product at the end of the project.

When the inspection result is rejection processing, the service provider 20 may internally perform the inspection again, or assign the work to the worker 32 again to perform the rework. When reworking, re-examination by an inspector is required.

Thereafter, when the project period ends or sufficient valid data is secured, the service provider 20 terminates the project (S19), calculates a final result based on the secured valid data, and provides it to the client 10 ( S20).

At this time, before the end of the project, the service provider 20 evaluates the performance results of the worker 32 and the inspector 34, calculates the work cost and the inspection cost according to the evaluation, to the worker 32 and the inspector 34. give.

In FIGS. 1 and 2, it is simply expressed as a requester 10, a service provider 20, an operator 32, and an inspector 34, but these are smartphones, tablets, PDAs, laptops, desktops, operated by each participant. It means a computer device such as a server or a telecommunication device.

Hereinafter, a method of setting a limit for rework and granting a penalty according to an embodiment of the present invention will be described with reference to FIGS. 3 to 15. A detailed description of the overlapping content in relation to the crowdsourcing service configuration and the crowdsourcing-based project process described with reference to FIG. 1 and FIG. 2 will be omitted.

Meanwhile, the steps shown in FIGS. 3, 5, 9, 11 and 14 may be understood as being performed by a platform server (hereinafter, referred to as a server) operated by the service provider 20, but are limited thereto. It does not become.

In addition, a plurality of workers 32 or a plurality of inspectors 34 perform work using a predetermined terminal device. The terminal device of the operator 32 or the inspector 34 may be a computer device or a telecommunication device such as a smartphone, tablet, PDA, laptop, desktop, etc., but is not limited thereto.

First, referring to FIG. 3, the server assigns a plurality of tasks of a crowdsourcing-based project (hereinafter, a project) to a plurality of workers 32 to request the task execution (S100), and a plurality of tasks from the plurality of workers 32 Receives the results of the work (S110), assigns a plurality of work results to a plurality of inspectors (34) to request inspection performance (S120), and a plurality of inspection results for the plurality of work results from the plurality of inspectors (34) It receives the input to pass or reject the inspection (S130).

If the result of the inspection on the work result is rejected, the rework corresponding to the work result is assigned to the worker who has entered the work result and requests to perform the rework (S140). Then, a rework limit period is set to a point in time when a predetermined period of time has elapsed from the time when the worker is requested to perform the rework (S150).

Rework is assigned to the worker subject to the commencement of rework within the limit of rework. If the worker does not start the rework assigned by the rework limit deadline, the assignment of the rework to the worker is canceled, and the rework is assigned to another worker to request the rework (S160). A penalty for canceling rework assignments is given to workers who have not started rework, and by promptly assigning the rework to a new worker, the bottleneck caused by delay in rework is solved, and finally, the schedule delay for the entire project is prevented. It is to do.

FIG. 4 is an exemplary diagram of a time limit for each rework when two rework are sequentially assigned to an operator.

Referring to FIG. 4, as an example, an N-th rework at 09:00 on April 13, 2020 and an N+1th rework at 10:00 are assigned to an operator. This is because each was rejected as a result of the inspection by the inspector. The time limit for the rework is 6 hours, and the operator must complete the Nth rework by 15:00 and the N+1th rework by 16:00, respectively. If the worker does not start performing the Nth rework by 15:00, the assignment to the worker will be canceled for the Nth rework of the worker, and the assignment to another worker will be requested to perform the rework. Reference numeral 40 denotes a period in which the Nth rework can be performed, and 50 denotes a period in which the N+1th rework can be performed.

Here, N is a natural number greater than or equal to 1.

The predetermined time set in advance for setting the limit for rework may be different depending on the difficulty of the project, the remaining period of the project, the number of workers 32 participating in the project, and the like.

For example, for a project with relatively high difficulty, 12 hours may be set as a predetermined time, and for a project with relatively low difficulty, 6 hours may be set as a predetermined time. In addition, when the remaining period of the project remains relatively large, a predetermined time may be set shorter than when the remaining period of the project remains relatively small. In addition, in the case of a project with a large number of participating workers 32, a predetermined time may be set shorter than in the case of a project with a small number of participating workers 32.

Hereinafter, it is assumed that the predetermined time set in advance related to the limit of rework of the project is 6 hours.

In one embodiment of the present invention, the step S160, if the worker starts performing the N-th rework until the N-th (however, N is a natural number of 1 or more) rework limit and is performing the N-th rework, the Even if the worker does not start performing the N+Mth rework by the N+Mth (where M is a natural number of 1 or more) rework limit, the assignment of the N+Mth rework to the worker is not canceled.

When the time interval between the Nth rework limit and the N+Mth rework limit is not large, that is, there may be cases where rework is concentrated. This is the case where the work result of one worker is inspected and returned by several inspectors. In this case, it is because it is an excessive and excessive penalty to cancel the allocation for the N+Mth rework because the N+Mth rework limit has expired during the Nth rework. Therefore, when the worker starts performing the N-th rework until the N-th rework limit 40 and is performing the N-th rework, each rework is initiated until the N+M-th rework limit following this. If not, the corresponding rework assignment will not be cancelled.

For example, if an operator currently needs to perform the 10th to 15th rework, and the operator has initiated and is performing the 10th rework by the 10th rework limit, (10th rework Due to the performance), even if the worker's 11th to 15th rework performance cannot be initiated until the corresponding rework limit deadline, respectively, the penalty for assigning rework to the worker is not given.

This is because the above-described penalty for workers is a sanctioning means for workers who delay project completion by not commencing rework performance until the rework limit deadline, causing a bottleneck for rework, and not for sincere workers.

In addition, in the process of faithfully performing rework, if a penalty is given until the limit for the next rework of the next order has expired, the rework in progress will be hastily and the quality of the rework may be degraded. .

On the other hand, referring to FIG. 4, the worker started performing the N-th rework at 14:30, before the N-th rework limit. At this time, if the worker completes the Nth rework at 15:30, the time given to the worker to do the N+1th rework until 16:00 is only 30 minutes.

Even though the worker started to perform the rework by the given Nth rework limit, due to the time taken to perform the Nth rework, the worker is disadvantaged in not fully utilizing the N+1th rework limit. do.

In order to prevent the disadvantage of the worker, it is necessary to reset the limit period for the N+1 th rework. The method of resetting the N+1th rework limit is as follows.

5 is a flowchart schematically illustrating a method of resetting a rework limit period of an N+1 th rework according to an embodiment of the present invention.

Referring to Figure 5, as an embodiment of the present invention, the step S150 is the step of calculating the N-th rework time required by the worker (S151) and the N-th rework required from the time point when the N-th rework result is input. And resetting the N+1th rework limit to a point in time when a predetermined multiple of the time has elapsed (S152).

Specifically, if a worker is assigned a plurality of rework, if the worker starts to perform the rework in a specific sequence among the rework assigned to the worker, the time (TR) required to perform the rework in the sequence is calculated and the result Based on, the rework limit of the next rework is reset.

6 is an exemplary view illustrating a method of resetting a rework limit period for an N+1 th rework according to an embodiment of the present invention.

Referring to FIG. 6, as in the example of FIG. 4 above, the worker's N-th rework was completed by the worker starting at 14:30 and inputting the N-th rework result at 15:30. At this time, assuming that the predetermined multiple is 1, the N+1th rework from 15:30 when the Nth rework result is input to 16:30, which is the time when 1 hour, which is the Nth rework time (TR), has elapsed. Reset the deadline.

Therefore, the operator can be guaranteed sufficient time to perform the N+1th rework even after completing the Nth rework.

FIG. 7 is an exemplary diagram illustrating a method of resetting a rework limit time limit for an N+1th rework based on a value twice the Nth rework required time according to an embodiment of the present invention.

Referring to FIG. 7, for example, when a predetermined multiple is 2, the N+1 th rework limit is 2 of the N th rework time (TR) from 15:30 at which the N th rework result is input. It is reset to 17:30, the point at which 2 hours have elapsed.

Meanwhile, according to an embodiment of the present invention, a predetermined multiple as a reference for resetting the rework time limit may be predetermined according to the difficulty of the project. A detailed description of this will be described later, and in the following examples, it is assumed that a predetermined multiple is set to 1 according to the difficulty of the project.

On the other hand, there may be a result that the limit period of the reset N+1 th rework is shorter than before the reset. 8 is a view for explaining that the rework limitation period is shortened by resetting the N+1th rework limitation period.

Referring to FIG. 8, unlike the previous examples, when the operator starts performing the N-th rework at 13:30 and completes it by inputting the rework result at 14:30, when the rework limit is uniformly reset, The N+1 th rework limit is reset from 14:30 when the N th rework result is input to 15:30 1 hour has passed.

This is shorter than the rework limit until 16:30 before resetting, and the result of resetting the N+1th rework limit is rather disadvantageous to the worker 32. Therefore, in order to prevent this, as an exception handling for the exception, the N+1 rework limit and the Nth rework result are input before resetting the N+1 rework limit. A time point at which a predetermined multiple of the required work time has elapsed may be compared, and based on the result, it may be determined whether to reset the N+1th rework limit period.

9 is a schematic flowchart of a method of resetting an N+1th rework limit compared to a rework limit before resetting according to an embodiment of the present invention.

9, as an embodiment of the present invention, the step S152 includes the N+1 th rework limit and the N th rework time taken from a time when the N+1 th rework result is received before resetting. Comparing the point in time when a predetermined multiple has elapsed (S153) and as a result of the comparison, if the N+1th rework limit is earlier than the reset, the N+1th rework limit is reset, and the If the N+1 th rework limit time limit is not earlier before resetting, the step of not resetting the N+1 th rework limit time limit (S154).

Therefore, referring to FIG. 8 again, in this case, the N+1 th rework limit of 16:00 before resetting is less than 15:30 1 hour after 14:30 when the N th rework result was input. Since it is not faster, the N+1 th rework limit is not reset.

On the other hand, in an embodiment of the present invention, when the N+1th rework limit is reset based on the Nth rework time required, step S160 is performed by the operator until the reset N+1th rework limit. If the N+1th rework is initiated and the N+1th rework is being performed, the worker performs the N+Lth rework until the N+Lth (but L is a natural number of 2 or more) rework limit deadline. Even if not undertaken, the assignment of the N+Lth rework to the worker is not canceled. The above-described penalty exemption applies even when the limit for rework is reset.

Therefore, if the operator 32 properly starts performing the N+1th rework until the reset N+1th rework limit deadline, each subsequent rework up to the N+Lth rework is corresponding to each. Penalties are not applied even if rework is not initiated by the rework limit.

For example, if the worker 32 currently needs to perform the 10th to 15th rework, and as described above, the 11th rework limit is reset by calculating the 10th rework time required, the worker ( 32), if the 11th rework execution is initiated and rework is being performed by the 11th rework limit, which is reset, the 12th to 15th rework subsequent to the rework cannot be initiated until the corresponding rework limit. Even so, no penalty for assigning rework to the worker is given.

On the other hand, even in the case of the N+1th rework in which the rework limit period is reset according to an embodiment of the present invention, the penalty for the worker's failure to start rework is still applied.

Specifically, when the N+1th rework limit is reset, in step S160, the server does not start performing the rework until the reset N+1th rework limit, the N+1th Rework is assigned to the worker, and the N+1th rework is assigned to another worker to request rework.

FIG. 10 is an exemplary diagram of a time limit for each rework when three rework is sequentially assigned to an operator.

Referring to FIG. 10, based on 1 hour (14:30~15:30), which is the time taken for the Nth rework, the time limit for the N+1th rework is 15: when the Nth rework result is input. It was reset from 30 to 16:30 after 1 hour. Reference numeral 51 denotes a period in which the N+1th rework can be performed due to reset, and reference numeral 60 denotes a period in which the N+2th rework can be performed before resetting. At this time, if the worker does not start performing the N+1th rework by 16:30, the reset N+1th rework limit, the assignment of the N+1th rework is canceled and assigned to another worker.

11 is a flowchart schematically illustrating a method of resetting a rework limit period for an N+2 rework according to an embodiment of the present invention.

Referring to FIG. 11, in an embodiment of the present invention, the step S150 includes the step of calculating the N-th rework time required by the worker (S151) and the reset N+1 th rework limit time limit. And a step (S155) of resetting the N+2 th rework limit to a point in time when a predetermined multiple of the required rework time has elapsed.

In other words, if the operator does not start performing rework by the reset N+1th rework limit, the point at which the N+1th rework result is input to reset the N+2th rework limit or N+1 Since the time required for the second rework cannot be calculated, the rework time limit for the N+2nd rework is reset based on the reset time limit for the N+1th rework and the time required for the Nth rework.

Referring back to FIG. 10, at 15:30 when the worker inputs the N-th rework result, the worker's N+1-th rework limit is reset to 16:30. In addition, it can be seen that the reworkable period of 30 minutes has been given to the worker from the reset N+1 th rework limit to 17:00, which is the N+2 th rework limit.

On the other hand, if the operator could not perform the N+1 rework due to unavoidable reasons, but want to rework the N+2 rework again, 30 minutes is too short compared to 6 hours, so the N+2 rework It is necessary to reset the deadline.

12 is an exemplary diagram illustrating a method of resetting a rework limit period for an N+2 rework according to an embodiment of the present invention.

Referring to FIG. 12, the N+2 th rework limit is from 16:30, which is the reset N+1 th rework limit, to 17:30, when 1 hour, which is the Nth rework time, has elapsed. Is reset.

On the other hand, in another embodiment of the present invention, when the operator starts performing rework until the reset N+1 th rework limit, the N+2 th rework limit may also be reset.

Specifically, when the worker initiates rework by the reset N+1th rework deadline and performs rework, the N+1th rework time is calculated, and the N+1th rework result from the worker The N+2 th rework limit period may be reset to a point in time when a predetermined multiple of the calculated N+1 th rework time has elapsed from the input time point.

That is, whenever the operator starts to perform rework appropriately until the rework limit deadline and completes the rework, the rework limit period of the next sequence may be reset again and updated based on the rework required time. Of course, as described above, if the rework limit period is rather shortened, it is not reset.

In addition, in another embodiment of the present invention, the server stores the rework required time calculated in step S151, stores the calculated rework time each time the operator completes a new rework, and accumulated stored previous rework Compared to the required time, it is also possible to reset the rework time limit based on the longest rework time required.

13 is a view for explaining that the rework limitation period is shortened by resetting the N+2th rework limitation period.

Referring to FIG. 13, a result that the reset limit of the N+2 th rework is shorter than the N+2 th rework limit before the reset may occur.

14 is a flowchart schematically illustrating a method of resetting the N+2th rework limit period compared to the rework limit period before resetting according to an embodiment of the present invention.

Referring to FIG. 14, in order to prevent this, in an embodiment of the present invention, the step S155 includes the N+2 th rework limit and the reset N+1 th rework limit before resetting. Comparing the point in time when a predetermined multiple of the N-th rework time has elapsed (S156), and if the comparison result, the N+2th rework limit 60 before the reset is faster, the N+ Reset the second rework limit (60), and do not reset the N+2nd rework limit (60) if the N+2nd rework limit (60) is not earlier than the reset It may include step S157.

Specifically, referring again to FIG. 13, the limit period for the N+1th rework has been reset to 16:30. The worker's N+2nd rework limit is 17:30, when 1 hour has elapsed from the reset N+1st rework limit of 16:30 to the Nth rework time (TR) calculated in step S151 at 16:30. Will be reset to. However, this is a case that is shorter than the rework limit of 19:00 based on the pre-set 6 hours, and rather results in a more unfavorable condition for the worker to perform the rework.

Therefore, before resetting the N+2th rework limit, the server is the N+2th rework limit at 19:00 and the N+1th rework limit at 16:30, which is the Nth rework time required. It compares 17:30, which is the time one hour has elapsed, and decides whether to reset the N+2th rework limit. In this case, since 19:00, which is the N+2th rework limit before resetting, is not earlier, the N+2th rework limit will not be reset.

This applies not only when the operator does not start performing rework by the reset N+1th rework deadline, but also when rework is performed.

That is, the server compares the time when a predetermined multiple of the time required for the N+1 rework has elapsed from the time when the N+2 rework limit and the N+1 rework result is received before resetting. , As a result of the comparison, if the N+2 th rework limit time limit before resetting is earlier, the N+2 th rework limit time limit is reset, and the N+2 th rework limit time limit before the reset is further If it is not fast, the N+2 th rework limit may not be reset.

In addition, according to an embodiment of the present invention, in step S155, when there is a rework (N+3, N+4...) following this assigned to the worker as well as the N+2th rework, the subsequent rework It is also possible to reset the rework time limit to be substantially the same as described above.

FIG. 15 is an exemplary view of a rework limit time limit for each rework when a plurality of rework is assigned to an operator, and FIG. 16 is a rework limit time limit for a plurality of rework according to an embodiment of the present invention. It is an exemplary diagram about how to do it.

Referring to FIGS. 15 and 16, even when a plurality of rework is assigned to an operator, the rework limit period for subsequent rework may be reset based on the time required for the previous rework.

As described above, according to an exemplary embodiment of the present invention, a predetermined multiple as a reference for resetting the rework time limit may be determined according to the difficulty of the project.

The higher the difficulty of the project, the greater the number of tasks returned as a result of the inspector's review and the number of rework assigned to each worker. As a result, the burden on the worker's rework increases, which leads to a deterioration in the quality of the rework result.

Therefore, in this case, in order to relieve the burden of rework on the worker and improve the quality of the rework result, a predetermined multiple can be set based on the difficulty of the project. The predetermined multiple is set in advance when the project is opened.

Hereinafter, an apparatus for setting a limit for rework of a crowdsourcing-based project and granting a penalty according to an embodiment of the present invention will be described with reference to FIG. 17.

17 is a diagram for explaining an apparatus for setting a limit for rework and granting a penalty for a crowdsourcing-based project according to an embodiment of the present invention.

Referring to FIG. 17, an apparatus 200 (hereinafter, referred to as a device) for setting a limit for rework of a crowdsourcing-based project and granting a penalty includes a communication module 210, a memory 220, and a processor 230.

The communication module 210 transmits a crowdsourcing-based work for one project to a plurality of workers 32 to request a task to be performed, and receives a work result from the plurality of workers 32. The work results received from the plurality of workers 32 are transmitted to the plurality of inspectors 34 to request inspection, and the inspection results are received from the plurality of inspectors 34. At this time, with respect to the rejected inspection result, the corresponding rework is transmitted to the worker 32 to request rework, and the rework result is received from the worker 32.

In the memory 220, a program for setting a rework limit period based on the data received from the communication module 210 and granting a penalty is stored.

The processor 230 executes a program stored in the memory 220. As the processor 230 executes the program stored in the memory 220, if the worker does not start performing rework by the rework limit deadline, the rework is canceled and assigned to another worker to perform rework. You can ask for In addition, the processor 230 may perform the method of setting a limit for rework of a crowdsourcing-based project described with reference to FIGS. 3 to 16 and granting a penalty.

The device 200 described with reference to FIG. 17 may be provided as a component of the above-described server.

The method for setting a limit for rework and granting a penalty for a crowdsourcing-based project according to an embodiment of the present invention described above is implemented as a program (or application) to be executed in combination with a computer, which is hardware, and is a computer-readable recording medium. Can be stored in.

The above-described program includes C, C++, JAVA, Ruby, which can be read by a processor (CPU) of the computer through the device interface of the computer, in order for the computer to read the program and execute the methods implemented as a program. It may include a code (Code) coded in a computer language such as machine language. Such code may include a functional code related to a function defining necessary functions for executing the methods, and a control code related to an execution procedure necessary for the processor of the computer to execute the functions according to a predetermined procedure. can do. In addition, such code may further include additional information required for the processor of the computer to execute the functions or code related to a memory reference to which location (address address) of the internal or external memory of the computer should be referenced. have. In addition, when the processor of the computer needs to communicate with any other computer or server in the remote in order to execute the functions, the code uses the communication module of the computer to determine how It may further include a communication-related code for whether to communicate or what information or media to transmit and receive during communication.

The stored medium is not a medium that stores data for a short moment, such as a register, cache, memory, etc., but a medium that stores data semi-permanently and can be read by a device. Specifically, examples of the storage medium include, but are not limited to, ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, and the like. That is, the program may be stored in various recording media on various servers to which the computer can access, or on various recording media on the user's computer. In addition, the medium may be distributed over a computer system connected through a network, and computer-readable codes may be stored in a distributed manner.

The above description of the present invention is for illustrative purposes only, and those of ordinary skill in the art to which the present invention pertains will be able to understand that it can be easily modified into other specific forms without changing the technical spirit or essential features of the present invention. will be. Therefore, it should be understood that the embodiments described above are illustrative in all respects and not limiting. For example, each component described as a single type may be implemented in a distributed manner, and similarly, components described as being distributed may also be implemented in a combined form.

The scope of the present invention is indicated by the claims to be described later rather than the detailed description, and all changes or modified forms derived from the meaning and scope of the claims and their equivalent concepts should be interpreted as being included in the scope of the present invention. do.

10: Client
20: service provider
30: public
32: worker
34: inspector
200: Rework limit period setting and penalty grant device
210: communication module
220: memory
230: processor

Claims (10)

As a method performed by a computer,
Allocating a plurality of tasks of a crowdsourcing-based project to a plurality of workers and requesting the task to be performed;
Receiving a plurality of work results from the plurality of workers;
Assigning the plurality of work results to a plurality of inspectors and requesting to perform inspection;
Receiving a plurality of inspection results for the plurality of work results from the plurality of inspectors as passing or rejecting the inspection;
Requesting rework by assigning rework corresponding to the work result to a worker who inputs the work result when the inspection result of the work result is rejected;
Setting a time limit for limiting rework to a time point when a predetermined period of time has elapsed from the time when the worker is requested to perform the rework; And
If the worker does not start performing the rework by the rework limit deadline, canceling the assignment of the rework to the worker and assigning the rework to another worker to request the rework performance,
If the worker starts performing the Nth rework by the Nth (where N is a natural number of 1 or more) rework limit and is performing the Nth rework, the worker is N+ until the N+1th rework limit. Even if you do not start performing the first rework, you do not cancel the assignment of the N+1th rework to the worker,
The step of setting the rework limit period,
Calculating the N-th time required for rework of the worker, and
And resetting the N+1th rework limit to a time when a predetermined multiple of the Nth rework time has elapsed from the time when the Nth rework result is input,
The step of canceling the assignment of the rework to the worker and assigning the rework to another worker to request rework,
If the worker does not start performing the rework by the reset N+1th rework limit deadline, canceling the assignment of the N+1th rework to the worker, and performing the N+1th rework to another worker Assigned to request rework,
How to set a deadline for rework and grant penalties for crowdsourced projects. The method of claim 1,
The step of canceling the assignment of the rework to the worker and assigning the rework to another worker to request rework,
If the worker starts performing the Nth rework by the Nth rework limit and is performing the Nth rework, the worker is N+Mth (where M is a natural number of 2 or more) until the rework limit is N+ Even if the M-th rework is not initiated, it does not cancel the assignment of the N+M-th rework to the worker,
How to set a deadline for rework and grant penalties for crowdsourced projects. delete The method of claim 1,
The step of resetting the N+1 th rework limit period,
Comparing a time point when a predetermined multiple of the N-th time required for rework has elapsed from the time when the N+1th rework limit and the Nth rework result is received before resetting, and
As a result of the comparison, if the N+1th rework limit period before the reset is earlier, the N+1th rework limit period is reset, and the N+1th rework limit period before the reset is faster. If not, including not resetting the N + 1 th rework limit,
How to set a deadline for rework and grant penalties for crowdsourced projects. The method of claim 1,
The step of canceling the assignment of the rework to the worker and assigning the rework to another worker to request rework,
If the worker starts performing the N+1th rework by the reset N+1th rework limit and is performing the N+1th rework, the worker is N+Lth (however, L is 2 Or more natural number) Even if the N+L-th rework performance is not started by the rework limit deadline, the assignment of the N+L-th rework to the worker is not canceled,
How to set a deadline for rework and grant penalties for crowdsourced projects. delete The method of claim 1,
The step of setting the rework limit period,
If the operator does not start performing the rework by the reset N+1th rework limit,
Resetting the N+2 th rework limit to a point in time when a predetermined multiple of the N-th rework required time has elapsed from the reset N+1 th rework limit time limit,
How to set a deadline for rework and grant penalties for crowdsourced projects. The method of claim 7,
The step of resetting the N+2th rework limit period,
Comparing a time point at which a predetermined multiple of the N-th rework required time elapses from the N+2 th rework limit time limit and the reset N+1 th rework limit time limit before resetting,
As a result of the comparison, if the N+2th rework limit is earlier before the reset, the N+2th rework limit is reset, and the N+2th rework limit before the reset is faster. If not, including not resetting the N + 2 th rework limit,
How to set a deadline for rework and grant penalties for crowdsourced projects. The method of claim 1,
The predetermined multiple is determined in advance according to the difficulty of the project,
How to set a deadline for rework and grant penalties for crowdsourced projects. In conjunction with a computer, executing the method of setting a limit for rework of a crowdsourcing-based project according to any one of paragraphs 1, 2, 4, 5, 7 to 9 and granting a penalty Computer program stored on a computer-readable recording medium

Images