KR102244702B1 - Method for paying appropriate cost corresponding to the inspector's inspection performance of crowdsourcing based projects for artificial intelligence training data generation - Google Patents

Abstract

The present invention relates to a method for paying an appropriate inspection cost for inspection execution by an inspector of a crowdsourcing-based project for artificial intelligence training data generation. The method includes: a work execution request step including requesting work execution by assigning reassignment work of a crowdsourcing-based project (hereinafter, project) to a second worker; a step of receiving the input of work results from the workers; an inspection execution request step including requesting inspection execution by assigning the work result input from the second worker to an inspector assigned with a work result input from a first worker; a step of receiving the input of inspection results regarding the work results as inspection pass or return from the inspectors; a step of paying the inspector an inspection cost of a second inspection unit price exceeding a first inspection unit price related to a general work result in relation to the work result of the reassignment work; a step of requesting rework execution to a worker who has input a work result in which the inspection result is input as return; and a step of cancelling, when the worker requested to perform rework execution (hereinafter, first worker) does not initiate the rework execution within a rework limited period, the assignment of the work requiring rework execution with respect to the first worker.

Description

{METHOD FOR PAYING APPROPRIATE COST CORRESPONDING TO THE INSPECTOR'S INSPECTION PERFORMANCE OF CROWDSOURCING BASED PROJECTS FOR ARTIFICIAL INTELLIGENCE TRAINING DATA GENERATION}

The present invention relates to a method of paying an appropriate inspection cost for an inspector's inspection performance of a crowdsourcing-based project for generating artificial intelligence learning data.

Recently, more and more companies collect and process large amounts of data on a crowdsourcing basis that engages the general public in some process of corporate activities. In other words, by opening a project and allowing the general public, that is, workers to participate in the project, necessary information is collected through the results of work completed by the workers.

When one project is opened, multiple tasks are assigned to each of the multiple workers. Each worker performs a plurality of tasks assigned to it, and provides the task results. At this time, in order to collect more reliable information, the company assigns the work result completed by the worker to the inspector to perform the inspection work.

On the other hand, the inspection fee is paid to the inspector in exchange for performing the inspection. However, the inspection fee is paid only when the inspection result entered by the inspector passes the inspection. That is, if the inspector inputs the inspection result as a rejection, compensation for the inspection performance is not received.

Therefore, if the operator continues to input work results that do not meet the inspection pass criteria, or if the operator does not perform rework on the work results returned to the operator by the inspector, the inspector has faithfully performed the inspection. You will not receive an inspection cost corresponding to the effort and time spent performing the inspection. This problem inevitably occurs in a crowdsourcing system that collects valid data, which is the final product, based on the work results of workers. However, if these problems continue to be turned away, the inspector will lose the motivation to perform the inspection, which will lead to a deterioration in the quality of the inspection results.

Therefore, there is a need for a method of providing the inspectors of the project with an appropriate inspection cost suitable for performing the inspection.

Unexamined Patent Publication No. 10-2014-0095956, 2014.08.04.

The problem to be solved by the present invention is to provide a method of paying an appropriate inspection cost for an inspector's inspection performance of a crowdsourcing-based project for generating artificial intelligence learning data.

However, the problem to be solved by the present invention is not limited to the problem as described above, and other problems may exist.

In order to solve the above-described problems, the method of paying an appropriate inspection cost for the inspection performance of an inspector of a crowdsourcing-based project according to an aspect of the present invention is a method performed by a computer, and includes a crowdsourcing-based project (hereinafter, "project"). Assigning a plurality of tasks to a plurality of workers and requesting to perform a task, receiving a plurality of work results from the plurality of workers, and assigning the plurality of work results to a plurality of inspectors and requesting to perform the inspection , Receiving a plurality of inspection results for the plurality of work results from the plurality of inspectors as inspection pass or rejection, the step of paying an inspection fee to the inspector who inputs the inspection result as inspection pass, and the inspection result is Step of requesting rework to the worker who has entered the result of the work entered as a rejection, and when the worker who has been requested to perform the rework (hereinafter referred to as the "first worker") does not start performing the rework within the rework limit period, the above Including the step of canceling the assignment of the task that needs to be reworked to the first worker, and the step of requesting the task by assigning a plurality of tasks of the project to a plurality of workers, the assignment to the first worker Reassigning the canceled job (hereinafter, “reassignment job”) to a second worker other than the first worker among the plurality of workers to perform the job, and the plurality of job results to a plurality of inspectors The step of assigning and requesting to perform the inspection includes requesting to perform the inspection by assigning the job result input from the second worker to the inspector who has assigned the job result input from the first worker, and inspecting the inspection result. In the step of paying the inspection fee to the inspector entered as a pass, the inspection cost of the first inspection unit is paid to the inspector in relation to the work result of the general work other than the reassignment work, and the work result of the reassignment work In relation to, the inspection fee of the second inspection unit price higher than the first inspection unit price is paid to the inspector. Do it.

In some embodiments of the present invention, the step of receiving a plurality of work results from the plurality of workers includes receiving a rework result from the first worker who started performing the rework within the rework limit period, and , In the step of paying the inspection cost to the inspector who inputs the inspection result as inspection pass, the inspection cost of the first inspection unit price is paid to the inspector in relation to the rework result as a work result of the general operation. I can.

In some embodiments of the present invention, payment of the inspection cost may be reserved to an inspector who inputs the inspection result as a rejection.

In some embodiments of the present invention, the first inspection unit price is composed of a basic inspection unit price, the second inspection unit price is composed of the basic inspection unit price and an additional inspection unit price, and the above-described method periodically increases the additional inspection unit price. Further comprising the step of adjusting, wherein the step of periodically adjusting the additional inspection unit price includes counting the number of payment processing times of the inspection cost of the second inspection unit price until a current period, the number of payment processing times and the additional inspection unit price Calculating the expenditure of the inspection cost preliminary cost on the basis of, and calculating the possible expenditure of the inspection cost preliminary expenditure based on the progress rate of the project and the total amount of the inspection cost reserve, and comparing the inspection cost preliminary expenditure with the inspection cost preliminary expenditure Thus, it may include the step of adjusting the additional inspection unit price.

In some embodiments of the present invention, the step of adjusting the additional inspection unit price by comparing the inspection cost preliminary cost expenditure amount and the inspection cost preliminary cost expenditure amount, when the inspection cost preliminary expenditure amount is less than the inspection cost preliminary expenditure possible amount, the The additional inspection unit price may be adjusted by the amount available for expenditure on the inspection cost preliminary cost relative to the number of payment processing times.

In some embodiments of the present invention, the step of adjusting the additional inspection unit price by comparing the inspection cost preliminary expenditure amount and the inspection cost preliminary expenditure available amount, the ratio of the inspection cost preliminary expenditure to the inspection cost reserve expenditure available amount is a reference ratio If less than that, the additional inspection unit price may be adjusted by the amount available for expenditure of the preliminary cost for the inspection cost compared to the number of payment processing times.

In some embodiments of the present invention, the step of periodically adjusting the additional inspection unit price may further include calculating a progress rate of the project based on the total number of inspection pass inputs relative to the total number of work of the project.

In some embodiments of the present invention, the first inspection unit price is composed of a basic inspection unit price, the second inspection unit price is composed of a basic inspection unit price and an additional inspection unit price, The step of calculating the time from the time when the work was canceled to the time when the work was requested to be performed by reassigning the work to the second worker as a waiting time for payment of the inspection cost, wherein the additional inspection unit price is calculated based on the waiting time for the payment of the inspection cost. Can be.

In some embodiments of the present invention, the additional inspection unit price may be calculated using the following equation.

Additional inspection unit price = Waiting time for payment of inspection fee X Unit cost X predetermined ratio

A computer program according to another aspect of the present invention for solving the above-described problems, in combination with a computer, which is hardware, executes a method of paying an appropriate inspection cost for an inspector's inspection performance of the crowdsourcing-based project, and a computer-readable recording medium Can be stored in.

Other specific details of the present invention are included in the detailed description and drawings.

According to the present invention described above, by reassigning the reassignment work with a history that has been previously assigned to another worker to the inspector who has previously performed the inspection on the work result of another worker, the inspection cost is reduced due to the previous failure of rework by another worker. A higher inspection fee is paid to inspectors who have not been paid. Therefore, the inspector has an expectation and a sense of relief that the inspection fee will be paid in the future even if the inspection fee is not paid due to an unscrupulous worker. This may lead to a positive effect of improving the quality of the inspection results by providing an environment in which the inspector can concentrate on conducting the inspection.

In addition, according to the present invention, the additional inspection unit price is adjusted by periodically comparing the inspection cost preliminary cost expenditure and the possible inspection cost expenditure. This allows the client of the project, who has no choice but to estimate and calculate the budget related to the additional inspection unit cost before opening the project, to efficiently manage the budget. In addition, if there is a surplus budget, it is possible to inspire the inspector to inspect by increasing the additional inspection unit cost.

Most of all, if you perform an inspection on a reassignment work that has been rejected, a higher inspection cost can be paid, so you will be able to induce the inspector to perform the inspection more sincerely and closely.

The effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the following description.

1 is a conceptual diagram of a crowdsourcing service according to an embodiment of the present invention.
2 is a flowchart illustrating a crowdsourcing-based project progress process according to an embodiment of the present invention.
3 is a flowchart schematically showing a method of paying an inspection fee to an inspector of a crowdsourcing-based project.
4 is a flowchart schematically illustrating a method of requesting a task to be performed by a worker based on a rejected task result according to an embodiment of the present invention.
5 is an exemplary view showing canceling a job for a worker who has not initiated rework within a rework limit period according to an embodiment of the present invention.
6 is an exemplary view illustrating a method of paying an inspection cost to an inspector who has performed inspection of a work result related to a reassignment operation according to an embodiment of the present invention.
7 is an exemplary diagram for explaining an inspection unit price differently paid according to an inspector's inspection performance according to an embodiment of the present invention.
8 is a flowchart schematically illustrating a method of periodically adjusting an additional inspection unit price according to an embodiment of the present invention.
9 is a diagram for explaining a method of periodically adjusting an additional inspection unit price according to an embodiment of the present invention.
10 is a diagram for explaining an apparatus for paying an appropriate inspection cost for an inspector's inspection performance of a crowdsourcing-based project according to an embodiment of the present invention.

Advantages and features of the present invention, and a method of achieving them will become apparent with reference to the embodiments described below in detail together with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but may be implemented in a variety of different forms. It is provided to fully inform the skilled person of the scope of the present invention, and the present invention is only defined by the scope of the claims.

The terms used in the present specification are for describing exemplary embodiments and are not intended to limit the present invention. In this specification, the singular form also includes the plural form unless specifically stated in the phrase. As used herein, “comprises” and/or “comprising” do not exclude the presence or addition of one or more other elements other than the mentioned elements. Throughout the specification, the same reference numerals refer to the same elements, and "and/or" includes each and all combinations of one or more of the mentioned elements. Although "first", "second", and the like are used to describe various elements, it goes without saying that these elements are not limited by these terms. These terms are only used to distinguish one component from another component. Therefore, it goes without saying that the first component mentioned below may be the second component within the technical idea of the present invention.

Unless otherwise defined, all terms (including technical and scientific terms) used in the present specification may be used with meanings that can be commonly understood by those of ordinary skill in the art to which the present invention belongs. In addition, terms defined in a commonly used dictionary are not interpreted ideally or excessively unless explicitly defined specifically.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a conceptual diagram of a crowdsourcing service according to an embodiment of the present invention.

Referring to FIG. 1, the crowdsourcing service is performed by being composed of a client 10, a service provider 20, and a public 30.

The sponsor 10 refers to a company or individual who requests a crowdsourcing-based project (hereinafter, referred to as a project).

The client 10 requests a project for the purpose of collecting source data or annotating data for generating artificial intelligence learning data. The data generated through the project can be used as learning data for arbitrary machine learning such as supervised learning, unsupervised learning, and reinforcement learning. Collection of source data means collecting raw data such as recorded voice collection and photo collection. Data annotation refers to inputting relevant annotation data into source data such as text, photos, and videos. For example, the data annotation may include finding an entity in a given fingerprint or finding a similar sentence, but is not limited thereto. Meanwhile, the types of the above-described projects are only one embodiment, and various projects may be handled in the present invention according to the design of the client.

The service provider 20 refers to a company that provides crowdsourcing services.

When the service provider 20 receives a request for a project for a product or service from the client 10, the service provider 20 allocates a task for the project to the general public 30 and receives the work result from the public 30. Thereafter, the final product extracted based on the work result is provided to the requester 10.

At this time, the service provider 20 provides a crowdsourcing service to the client 10 and the public 30 through a crowdsourcing platform (hereinafter, the platform). That is, when the service provider 20 receives a request for a project from the client 10, the service provider 20 opens the project on the platform. Thereafter, when the work result for the open project is provided by the public 30, the project is terminated on the platform, and the final product may be extracted and provided to the client 10.

The public 30 refers to the general public participating in the project open on the platform. Here, the public 30 may participate in a project opened on the platform through an application or website provided by the service provider 20.

The public 30 is composed of a worker 32 and an inspector 34.

The worker 32 decides to participate in a specific project among a plurality of projects open on the platform. Thereafter, the operator 32 performs an operation such as collection of source data or data annotation, and transmits it to the platform.

The inspector 34 decides to participate in a specific project among a plurality of projects opened on the platform. Thereafter, the inspector 34 inspects the result of the work performed by the operator 32. As a result of performing the inspection, the inspector 34 may perform inspection pass processing or rejection processing, and input a rejection reason during rejection processing. In the case of passing the inspection, rework and the resulting re-inspection are not required, so passing the inspection has the same meaning as the completion of the inspection.

2 is a flowchart illustrating a crowdsourcing-based project progress process according to an embodiment of the present invention.

First, the client 10 requests one or more projects to the service provider 20 (S11).

Thereafter, the service provider 20 opens the requested project on the platform (S12). At this time, the service provider 20 may determine the grade in consideration of the difficulty of the project before opening the project. That is, depending on the difficulty level, it is possible to determine which level or higher public 30 to expose the project. Accordingly, it is possible to increase the reliability of the work result of the project.

Thereafter, the service provider 20 requests the work by assigning the work to the workers 32 of the corresponding level or higher according to the class of the project (S13).

Thereafter, the worker 32 performs the assigned task (S14). In this case, the worker 32 may input the reason for the inability to work without performing the work for a work in which the work itself is impossible for some reason.

Thereafter, the service provider 20 receives the work result from the worker 32 (S15), and assigns the work result to the inspector 34 to request the inspection (S16).

Thereafter, the inspector 34 performs the assigned inspection (S17). At this time, the inspector 34 decides to pass the inspection when it is determined that the work has been properly performed, and if it is determined that the inspection work is wrong, the inspector 34 rejects it. During rejection processing, the inspector 34 inputs the rejection reason for what reason the job was judged to be wrong.

Thereafter, the service provider 20 receives the inspection result from the inspector 34 (S18).

If the inspection result passes the inspection, the service provider 20 uses the work result as valid data, and based on this, extracts the final product at the end of the project.

If the result of the inspection is rejection processing, the service provider 20 may internally perform the inspection again, or assign the work to the worker 32 again to perform the rework. When reworking, re-examination by an inspector is required.

Thereafter, when the project period ends or sufficient valid data is secured, the service provider 20 terminates the project (S19), calculates a final result based on the secured valid data, and provides it to the client 10 ( S20).

At this time, before the end of the project, the service provider 20 evaluates the performance results of the worker 32 and the inspector 34, calculates the work cost and the inspection cost according to the evaluation, to the worker 32 and the inspector 34. give.

In Figures 1 and 2, it is simply expressed as a requester 10, a service provider 20, an operator 32, and an inspector 34, but these are smartphones, tablets, PDAs, laptops, desktops operated by each participant. It means a computer device such as a server or a telecommunication device.

Hereinafter, a method of paying an appropriate inspection cost for an inspector's inspection performance of a crowdsourcing-based project according to an embodiment of the present invention will be described with reference to FIGS. 3 to 10. A detailed description thereof will be omitted for overlapping contents related to the crowdsourcing service configuration and the crowdsourcing-based project progress process described above with reference to FIGS. 1 and 2.

Meanwhile, the steps shown in FIGS. 3, 4 and 8 may be understood as being performed by a platform server (hereinafter, referred to as a server) operated by the service provider 20, but are not limited thereto.

3 is a flow chart schematically showing a method of paying an inspection fee to an inspector of a crowdsourcing-based project, and FIG. 4 schematically shows a method of requesting a worker to perform a task of a rejected task result according to an embodiment of the present invention. It is a flow chart.

First, referring to FIG. 3, a plurality of tasks of a project are assigned to a plurality of workers to request the task performance (S110), and a plurality of task results are inputted from a plurality of workers (S120).

Here, one project may contain multiple jobs, and the plurality of jobs may be grouped into a plurality of tasks according to their type, or each job may be a different type of job or the same job. , But is not limited thereto.

Next, referring to FIG. 3 again, a plurality of work results are assigned to a plurality of inspectors to request for inspection (S130), and then a plurality of inspection results for the plurality of work results are input from a plurality of inspectors as passing or rejecting the inspection. Receive (S140).

At this time, it is decided whether to pay the inspection fee to the inspector according to the inspection result input from the inspector. Specifically, referring to FIG. 3, it is determined whether the inspection result input from the inspector is the inspection pass or rejection (S150). Then, the inspection cost is paid to the inspector who inputs the inspection result as the inspection pass (S160). On the other hand, the payment of the inspection fee is reserved for the inspector who inputs the inspection result as a rejection, and the server requests the operator who has entered the rejected operation result to perform rework (S170). However, when the result of the inspection of the rework result of the worker (the worker who has been requested to perform the rework) is input as the inspection pass, the inspection fee is also paid to the inspector whose inspection cost has been reserved (S160).

On the other hand, looking at the figure, rework is assigned to a worker on the condition that the rework begins to be performed within the rework limit period. At this time, it is determined whether the worker has initiated the rework assigned by the rework limit deadline (S180), and if the worker does not start performing the rework within the rework limit deadline, the assignment of the task to the worker requested to perform the rework is canceled. Do (S190). This is to solve a bottleneck caused by delays in rework by giving a penalty for canceling work assignment to a worker who has not started rework, and quickly assigning the work to a new worker.

On the other hand, by canceling the assignment of the work to the worker in step S190, the inspector who performed the inspection on the result of the work received a penalty of not receiving the inspection fee definitively even though the inspection was performed after the inspection fee payment was suspended. .

5 is an exemplary view showing canceling a job for a worker who has not initiated rework within a rework limit period according to an embodiment of the present invention.

Referring to FIG. 5, inspector 1 performed a task assigned to worker 1 (Data ID: 1001) and inspected the input task result. And inspector 1 entered the inspection result as rejection. At this time, payment of the inspection fee to inspector 1 was reserved. On the other hand, the server requested the operator to perform rework on the job corresponding to the rejected job result. However, because worker 1 did not start to perform the rework within the rework limit, the server canceled the assignment of the task (Data ID: 1001) to worker 1. Eventually, Inspector 1, whose payment of the inspection fee was deferred, will not receive the final inspection fee as the assignment of the task is cancelled. In other words, Inspector 1 does not receive a reasonable compensation corresponding to the time (Ts) and effort it took to perform the inspection.

As such, whether or not to pay the inspection fee to the inspector is not determined solely by the inspector's ability to perform inspection. The inspector can receive the inspection fee only when the assigned work result has passed inspection, which means that the payment of the inspection fee is also determined by the job performance ability of the worker who entered the job result and the willingness to perform the rework. .

In the end, even if the inspector faithfully conducts the inspection with a lot of time and effort, the inspection cost if the operator does not enter the work result corresponding to the inspection pass by improving the work method, or does not start rework within the rework limit period. Are not paid. This problem inevitably occurs in a crowdsourcing system that requires an unspecified number of workers to participate in the project and produce the final product, the source data, based on the results of their work. This is because the source data is created as a result of the work that has finally passed the inspection.

Therefore, an embodiment of the present invention proposes a method of paying an appropriate inspection cost corresponding to the inspection performance to the inspector in order to solve the damage of the inspector who does not receive the inspection cost appropriate for the inspection performance caused by such a crowdsourcing system. do. Hereinafter, an embodiment of the present invention in relation to this will be described in detail.

Referring back to FIG. 4, in an embodiment of the present invention, if the worker (hereinafter referred to as "the first worker") requested to perform the rework does not start performing the rework within the rework limit period, the above The assignment of a task requiring rework is canceled (S190). At this time, data on the initial work result input after the first worker is assigned a work is also deleted. In addition, the task in which the assignment to the first worker is canceled (hereinafter, “reassignment task”) is reassigned to a second worker other than the first worker among the plurality of workers to request the task to be performed (S110).

Specifically, in the step (S110) of assigning a plurality of tasks to a plurality of workers and requesting to perform a task (S110), the reassignment task is reassigned to a second worker other than the first worker among the plurality of workers on the crowdsourcing platform to request the task to be performed Includes doing. At this time, from the perspective of the second worker, it is the same as receiving a request to perform a task by being assigned a random task among a plurality of tasks in the project, but the reassignment task is clearly distinguished from other general tasks.

The reassignment work refers to a work that has already been requested to perform work by another worker, but has been canceled due to the worker's failure to perform the reassignment work. In the case of the reassignment work, there is a history of the inspector already performing the inspection on the work result of the work, and the inspector has not been paid the inspection cost corresponding to the inspection performance. Accordingly, in an embodiment of the present invention, unlike other general tasks, in the case of a reassignment task, an inspector who will perform an inspection on the result of the corresponding operation is determined, and the inspection cost is also paid differently from the general operation. This will be described in detail later.

Meanwhile, according to an embodiment of the present invention, when the second worker inputs the work result after performing the work, the work result input from the second worker is assigned to the inspector who assigned the work result received from the first worker to perform the inspection. Request (S130).

Specifically, step S130 includes requesting to perform the inspection by assigning the work result input from the second worker to the inspector who has assigned the work result input from the first worker.

6 is an exemplary view for explaining a method of paying an inspection cost to an inspector who has performed inspection of a work result related to a reassignment operation according to an embodiment of the present invention.

Referring to FIG. 6, after the assignment of the work (Data ID: 1001) to worker 1 was canceled due to the uncommitted rework within the rework limit period, the corresponding work (reassignment work) was reassigned to worker 2. At this time, the work result for the reassignment task input by worker 2 is reassigned to inspector 1 who assigned the work result input from worker 1 (the first worker).

On the other hand, in an embodiment of the present invention, the step of paying the inspection cost to the inspector who inputs the inspection result as the inspection pass is, in relation to the work result of the general operation, not the reassignment operation, the first inspection unit price to the inspector. The inspection cost is paid, and, in relation to the work result of the rearrangement operation, the inspection cost of the second inspection unit price, which is greater than the first inspection unit price, is paid to the inspector.

Referring to FIG. 6 again, the inspector 1 was assigned to the work result for the reassignment job input by the operator 2 and performed the inspection. If inspector 1 inputs the inspection result as inspection passed, inspector 1 is paid an inspection fee that was not received at the time the inspection was performed on the work result of worker 1. At this time, the inspection cost paid to inspector 1 corresponds to the second inspection unit price, which is a higher cost than the first inspection unit price paid as a price for performing inspection on the work results of general work.

Inspector 1 repeatedly performed the inspection for the same task (Data ID: 1001). Above all, it performed inspection on the work results input by different workers. This is different from the inspection of work results and rework results entered by the same operator. In the case of the result of the work that the inspector rejected, it is sufficient to review only the correction or improvement of the part pointed out by the inspector. However, in the case of work results input by different workers, the inspector has to spend more time and effort because it can have completely different contents depending on the work method, tendency, and prior knowledge of the worker. Therefore, a larger inspection unit price is paid as inspection cost to inspectors who have performed inspection on the work result of the reassignment work. In addition, paying the larger inspection unit price as inspection cost is to provide a reasonable compensation to the inspector who has lost the expected profit of the inspection fee payment as the assignment of work to the first worker is canceled in step S190.

Through this, even if the work for the first worker is canceled, the inspector can have a sense of relief and expectation that someday, even if the work for the first worker is canceled, the inspection cost for the corresponding work can be compensated. In this way, the inspector can relieve anxiety about non-payment of inspection costs and focus more on inspection performance. In addition, by paying the inspection cost of the higher second inspection unit price to the inspector whose inspection fee has been deferred, it can inspire the inspector to perform the inspection, and as a result, it will lead to the effect of improving the quality of the inspection result.

On the other hand, according to an embodiment of the present invention, step S120 includes receiving the rework result from the first worker who started performing the rework within the rework limited period, and the inspection result to the inspector who entered the inspection pass. In the step of paying the inspection cost, the inspection cost of the first inspection unit price may be paid to the inspector in relation to the rework result as a work result of a general operation.

7 is an exemplary diagram for explaining an inspection unit price differently paid according to an inspector's inspection performance according to an embodiment of the present invention.

Referring to FIG. 7, a task result for a task assigned to worker 1 (Data ID: 1001) was input. In addition, if the inspector 1 assigned the work result enters the inspection result as the inspection pass, the first inspection unit price is paid as the inspection cost. If inspector 1 rejects the work result and then inputs the re-inspection result for the rework result input by operator 1 as the inspection pass, the first inspection unit price will be paid as the inspection cost in the same way. This is because, as described above, the inspection of the rework result does not require much time and effort from the inspector.

On the other hand, according to an embodiment of the present invention, payment of the inspection fee may be deferred to an inspector who inputs the inspection result as a rejection.

Even if the inspector enters the inspection result for the work result of the reassignment operation input by the second worker as a rejection, payment of the inspection fee to the inspector is reserved. As described above, this is because the inspection cost means not only a reward for the time and effort required by the inspector to perform the inspection, but also a means of confirming that the work result of the operator has been confirmed as the final product.

On the other hand, although not clearly shown in the drawings, in an embodiment of the present invention, the first inspection unit price may be composed of a basic inspection unit price, and the second inspection unit price may be composed of a basic inspection unit price and an additional inspection unit price. At this time, the additional inspection unit price may be adjusted periodically.

Specifically, the inspector who inspects the work result for the reassignment job input by the second worker receives the second inspection unit price as the inspection cost. At this time, the second inspection unit price is determined by summing the additional inspection unit price to the first inspection unit price (basic inspection unit price) paid to the inspector who performed the inspection result for the general work, not the reassignment work.

Before opening a project, it is difficult for the sponsor of the project to accurately predict the number of times the second inspection unit price will be paid. This is because it is impossible to accurately predict the number of rework that occurs in relation to the reassignment work for which the second inspection unit is paid, and the number of relocation work that occurs when the worker does not start rework within the rework limit period. Therefore, the budget related to the additional inspection unit price is inevitably estimated and calculated by the project sponsor. On the other hand, the budget related to the basic inspection unit price can be clearly selected. This is because the budget can be calculated based on the number of work results corresponding to the final product, that is, the number of work results requested by the project sponsor. For example, assuming that the final output, source data, is 1000 and the basic inspection unit price is KRW 2,000, the inspection cost budget related to the basic inspection unit price can be calculated as KRW 2,000,000.

Accordingly, according to an embodiment of the present invention, by periodically adjusting the additional inspection unit cost, the budget related to the additional inspection unit cost estimated and calculated before opening a project can be efficiently and appropriately managed. In addition, if there is a surplus budget within the budget related to the additional inspection unit price, a higher additional inspection unit price can be paid to the inspector, thereby inspiring the inspector's desire for inspection.

Hereinafter, a specific method of periodically adjusting the additional inspection unit price will be described with reference to FIG. 8.

8 is a flowchart schematically illustrating a method of periodically adjusting an additional inspection unit price according to an embodiment of the present invention, and FIG. 9 is a flowchart illustrating a method of periodically adjusting an additional inspection unit price according to an embodiment of the present invention. It is a drawing for.

Referring to FIG. 8, first, the number of times of payment processing of the inspection cost of the second inspection unit price up to the current period is counted (S200). As described above, since the second inspection unit price consists of a basic inspection unit price and an additional inspection unit price, the number of times the second inspection unit price is paid for inspection is the same as the number of payments of the additional inspection unit price.

And, based on the number of payment processing counts and the additional inspection unit price counted in step S200, the expenditure of the inspection cost reserve is calculated (S210). Here, the expenditure of the preliminary expenses for inspection means the total amount of the additional inspection unit price paid to the inspector up to the current cycle. Referring to FIG. 9, if the number of times the second inspection unit price is paid as inspection cost until the current cycle is 500 times, and the additional inspection unit price is 1,000 won, the inspection cost preliminary expenditure will be calculated as 100,000 won.

After calculating the inspection cost reserve cost, the possible amount of the inspection cost reserve cost is calculated based on the progress of the project and the total amount of the inspection cost reserve cost (S220). Here, the total amount of preliminary costs for inspection costs means the budget for additional inspection unit costs estimated and calculated by the client of the project. For example, assuming that the total project budget is 50 million won, the total preliminary cost for inspection can be calculated as 10 million won, which is approximately 20% of the total project budget.

Inspection cost The reserve cost expenditure is the maximum amount that could be paid for the additional inspection unit cost, calculated based on the progress of the project up to the current cycle. For example, referring to FIG. 9, assuming that the progress rate of the project up to the current cycle is 10%, the available amount of the inspection cost reserve cost is 1 million won out of the total inspection cost reserve cost of 10 million won. In other words, it means that the total amount that could have been paid to the inspector as an additional inspection unit cost until 1/10 of the total project schedule was taken is up to 1 million won.

On the other hand, after calculating the possible expenditure of the inspection cost reserve cost, the additional inspection unit price is adjusted by comparing the calculated inspection cost reserve cost expenditure with the inspection cost reserve expenditure (S230). In other words, it is determined whether to adjust the additional inspection unit price by comparing the total additional inspection unit cost that has already been paid to the inspector with the total additional inspection unit cost that could have been paid to the inspector when the project progress rate is deferred.

On the other hand, in an embodiment of the present invention, the step of adjusting the additional inspection unit price by comparing the inspection cost preliminary cost expenditure amount and the inspection cost preliminary cost expenditure amount is, when the inspection cost preliminary cost expenditure is less than the inspection cost preliminary expenditure available amount , It is possible to adjust the additional inspection unit price as an amount available for expenditure of the inspection cost preliminary cost relative to the number of payment processing times.

Referring to FIG. 9, since the expenditure of the preliminary expenses for the inspection up to the current cycle is 500,000 won, which is less than 1,000,000 won of the possible expenditure of the preliminary expenses for the inspection, the additional inspection unit price can be adjusted. In this case, according to an embodiment of the present invention, the additional inspection unit price may be adjusted to 2,000 won, which is 1,000,000 (possible amount of preliminary expenditure for inspection cost)/500 (number of payment processing).

In an embodiment of the present invention, the step of adjusting the additional inspection unit price by comparing the inspection cost preliminary cost expenditure amount and the inspection cost preliminary cost expenditure amount, the ratio of the inspection cost preliminary expenditure amount to the inspection cost reserve cost expenditure available amount If the ratio is less than the reference ratio, the additional inspection unit price may be adjusted by the possible amount of the inspection cost preliminary expenditure relative to the number of payment processing times.

Specifically, even if the additional inspection cost is increased based on the number of additional inspection payments counted up to the current cycle, if the situation in which the second inspection unit must be paid to the inspector in the process of a future project occurs more frequently, the additional inspection unit price is adjusted. This can cause difficulties in managing the inspection cost budget in the future. Therefore, if the difference between the possible expenditure of the inspection cost and the preliminary expenditure of the inspection cost is not large, the additional inspection unit price is not adjusted. To this end, the additional inspection unit price can be adjusted only when the ratio of the inspection cost and the preliminary expenditure to the possible expenditure of the inspection cost is less than a predetermined standard ratio.

In addition, as an embodiment of the present invention, the step of periodically adjusting the additional inspection unit cost may further include calculating a progress rate of the project based on the total number of inspection passes and inputs relative to the total number of tasks of the project. have.

Specifically, the progress of the project may be calculated as the time taken to the current cycle among the entire schedule of the project. However, the project's schedule may be terminated early if the project's sponsor has collected a target amount of final product. Therefore, the progress of the project can be calculated based on the total number of inspection passes and the number of inputs compared to the total number of work in the project.

For example, assuming that the total number of work is 20,000, if the number of inputs that have passed the inspection until the current cycle is 2,000, the project progress rate will be calculated as 10%, which is 2,000/20,000X100.

Meanwhile, as an embodiment of the present invention, although not clearly shown in the drawings, the first inspection unit price is composed of the basic inspection unit price, the second inspection unit price is composed of the basic inspection unit price and the additional inspection unit price, and the reviewer of the above-described project The appropriate inspection cost method for performance involves calculating the time from the time when the work for the first worker is canceled to the time when the work is requested to be reassigned to the second worker as the waiting time for the payment of the inspection cost. In addition, the additional inspection unit price may be calculated based on a waiting time for payment of the inspection cost.

Specifically, after the payment of the inspection fee to the inspector is canceled due to the cancellation of the work on the first worker, the time taken to request the job execution by reassigning the reassignment job to the second worker (waiting time for the payment of inspection fee) is calculated. do. In addition, when the inspector passes the inspection after performing the inspection of the work result of the second worker, the additional inspection unit of the second inspection unit paid is calculated in proportion to the waiting time for the payment of the inspection cost. That is, the inspector compensates for the penalty of loss of expected profits received due to the first worker's failure to start rework by the amount of time waiting for the inspection cost to be paid later.

Meanwhile, as an embodiment of the present invention, the above-described additional inspection unit price may be calculated using Equation 1 below.

[Equation 1]

Additional inspection unit price = Waiting time for payment of inspection fee X Unit cost X predetermined ratio

For example, assume that the unit cost is 100 won and the predetermined ratio is 0.1. At this time, if the time taken for the assignment to the first worker to be canceled and the reassignment work to be reassigned to the second worker is 48 hours, the waiting time for the inspector to pay the inspection fee is calculated as 48 hours. In addition, the additional inspection unit price will be calculated at KRW480. At this time, assuming that the basic inspection unit price is 500 won, the inspector will be paid a cost of 980 won as the second inspection unit price.

Meanwhile, in the above description, steps S110 to S210 may be further divided into additional steps or may be combined into fewer steps, according to an embodiment of the present invention. In addition, some steps may be omitted as necessary, or the order between steps may be changed. In addition, even if other contents are omitted, the contents of FIG. 14 to be described later may also be applied to the apparatus for paying an appropriate inspection cost for the inspection performance of the inspector of the crowdsourcing-based project of Figs. 1 to 13.

Hereinafter, an apparatus for paying an appropriate inspection cost for an inspector's inspection performance of a crowdsourcing-based project according to an embodiment of the present invention will be described with reference to FIG. 14.

14 is a diagram for explaining an apparatus for paying an appropriate inspection cost for an inspector's inspection performance of a crowdsourcing-based project according to an embodiment of the present invention.

Referring to FIG. 14, a method 300 (hereinafter, referred to as an apparatus) for paying an appropriate inspection cost for an inspector's inspection performance of a crowdsourcing-based project includes a communication module 310, a memory 320, and a processor 330.

The communication module 310 transmits a crowdsourcing-based work for one project to a plurality of workers 32 to request a task to be performed, and receives work results from the plurality of workers 32. The communication module 310 transmits the work results received from the plurality of workers 32 to the plurality of inspectors 34 to request inspection, and receives the inspection results from the plurality of inspectors 34. At this time, for the rejected inspection result, a corresponding rework is transmitted to the worker 32 to request rework, and the rework result is received from the worker 32.

The memory 320 stores a program for paying an appropriate inspection cost for the inspection performance of the inspector of the project based on the data received from the communication module 310.

The processor 330 executes a program stored in the memory 320. As the processor 330 executes the program stored in the memory 320, the processor 330 may perform the method of paying an appropriate inspection cost for the inspection performance of the inspector of the crowdsourcing-based project described with reference to FIGS. 3 to 13.

The device 300 described with reference to FIG. 14 may be provided as a component of the above-described server.

The method for paying an appropriate inspection cost for an inspector's inspection performance of a crowdsourcing-based project according to an embodiment of the present invention described above is implemented as a program (or application) to be executed in combination with a computer, which is hardware, and is computer-readable. It can be stored on a recording medium.

The above-described program includes C, C++, JAVA, Ruby, which can be read by a processor (CPU) of the computer through the device interface of the computer, in order for the computer to read the program and execute the methods implemented as a program. It may include a code (Code) coded in a computer language such as machine language. Such code may include a functional code related to a function defining necessary functions for executing the methods, and a control code related to an execution procedure necessary for the processor of the computer to execute the functions according to a predetermined procedure. can do. In addition, these codes may further include additional information required for the processor of the computer to execute the functions or code related to a memory reference to which location (address address) of the internal or external memory of the computer should be referenced. have. In addition, when the processor of the computer needs to communicate with any other computer or server in a remote in order to execute the functions, the code is It may further include a communication-related code for whether to communicate, what kind of information or media to be transmitted and received during communication.

The stored medium is not a medium that stores data for a short moment, such as a register, cache, memory, etc., but a medium that stores data semi-permanently and can be read by a device. Specifically, examples of the storage medium include, but are not limited to, ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, and the like. That is, the program may be stored in various recording media on various servers to which the computer can access, or on various recording media on the user's computer. In addition, the medium may be distributed over a computer system connected through a network, and computer-readable codes may be stored in a distributed manner.

The above description of the present invention is for illustrative purposes only, and those of ordinary skill in the art to which the present invention pertains will be able to understand that other specific forms can be easily modified without changing the technical spirit or essential features of the present invention. will be. Therefore, it should be understood that the embodiments described above are illustrative and non-limiting in all respects. For example, each component described as a single type may be implemented in a distributed manner, and similarly, components described as being distributed may also be implemented in a combined form.

The scope of the present invention is indicated by the claims to be described later rather than the detailed description, and all changes or modified forms derived from the meaning and scope of the claims and their equivalent concepts should be construed as being included in the scope of the present invention. do.

10: requester
20: service provider
30: public
32: worker
34: inspector

Claims (10)

As a method performed by a computer,
Allocating a plurality of tasks of a crowdsourcing-based project (hereinafter, "project") to a plurality of workers and requesting the task to be performed;
Receiving a plurality of job results from the plurality of workers;
Assigning the plurality of work results to a plurality of inspectors and requesting to perform the inspection;
Receiving a plurality of inspection results for the plurality of work results from the plurality of inspectors as passing or rejecting the inspection;
Paying an inspection fee to an inspector who inputs the inspection result as an inspection pass;
Requesting to perform rework to a worker who inputs a work result inputted as a rejection of the inspection result; And
If the worker requested to perform the rework (hereinafter, "the first worker") does not start performing the rework within the rework limit period, canceling the assignment of the task requiring the rework performance to the first worker Including,
Assigning a plurality of tasks of the project to a plurality of workers and requesting the task to be performed,
Reassigning the task (hereinafter, "reassignment task") for which the assignment to the first worker has been canceled to a second worker other than the first worker among the plurality of workers, and requesting the task to be performed,
Allocating the plurality of work results to a plurality of inspectors and requesting to perform the inspection,
Including requesting to perform the inspection by assigning the work result input from the second worker to the inspector who assigned the work result input from the first worker,
The step of paying an inspection fee to an inspector who inputs the inspection result as inspection pass,
In relation to the work result of the general work other than the reassignment work, the inspection cost of the first inspection unit price is paid to the inspector, and
In relation to the work result of the reassignment operation, an inspection fee of a second inspection unit price greater than the first inspection unit price is paid to the inspector, and
The first inspection unit price is composed of the basic inspection unit price,
The second inspection unit price is composed of the basic inspection unit price and the additional inspection unit price,
Further comprising the step of periodically adjusting the additional inspection unit price,
The step of periodically adjusting the additional inspection unit price,
Counting the number of times of payment processing of the inspection cost of the second inspection unit price until a current period,
Calculating a preliminary expenditure for inspection cost based on the number of payment processing times and the additional inspection unit price,
Calculating a possible amount of expenditure for the inspection cost reserve based on the progress rate of the project and the total amount of the reserve cost for inspection;
Comprising the step of adjusting the additional inspection unit price by comparing the inspection cost preliminary cost expenditure amount and the inspection cost preliminary cost expenditure possible amount,
A method of paying an appropriate inspection fee for the inspection performance of the inspector of a crowdsourcing-based project. The method of claim 1,
The step of receiving a plurality of work results from the plurality of workers,
Including receiving a rework result from the first worker who started performing the rework within the rework limited period,
The step of paying an inspection fee to an inspector who inputs the inspection result as inspection pass,
In relation to the rework result as a work result of the general work, processing to pay the inspection cost of the first inspection unit price to the inspector,
A method of paying an appropriate inspection fee for the inspection performance of the inspector of a crowdsourcing-based project. The method of claim 1,
Deferring the payment of the inspection fee to the inspector who entered the inspection result as a rejection,
A method of paying an appropriate inspection fee for the inspection performance of the inspector of a crowdsourcing-based project. delete The method of claim 1,
The step of adjusting the additional inspection unit price by comparing the inspection cost preliminary cost expenditure and the inspection cost preliminary expenditure available amount,
If the expenditure of the above inspection cost reserve is less than the possible expenditure of the above inspection cost reserve,
Adjusting the additional inspection unit price with the available amount of the inspection cost preliminary expenditure relative to the number of payment processing times,
A method of paying an appropriate inspection fee for the inspection performance of the inspector of a crowdsourcing-based project. The method of claim 5,
The step of adjusting the additional inspection unit price by comparing the inspection cost preliminary cost expenditure and the inspection cost preliminary expenditure available amount,
If the ratio of the inspection cost reserve cost expenditure to the possible expenditure of the inspection cost reserve cost is less than the standard ratio, adjusting the additional inspection unit price to the possible expenditure of the inspection cost reserve cost relative to the number of payment processing times,
A method of paying an appropriate inspection fee for the inspection performance of the inspector of a crowdsourcing-based project. The method of claim 1,
The step of periodically adjusting the additional inspection unit price,
Further comprising the step of calculating a progress rate of the project based on the total number of inspection pass inputs relative to the total number of work of the project,
A method of paying an appropriate inspection fee for the inspection performance of the inspector of a crowdsourcing-based project. The method of claim 1,
The first inspection unit price is composed of the basic inspection unit price,
The second inspection unit price is composed of a basic inspection unit price and an additional inspection unit price,
Regarding the reassignment operation, the step of calculating a time from a time when the work for the first worker is canceled to a time when the work is requested to be reassigned to the second worker to perform the work is calculated as a waiting time for payment of the inspection cost,
The additional inspection unit price is calculated based on the waiting time for payment of the inspection fee,
A method of paying an appropriate inspection fee for the inspection performance of the inspector of a crowdsourcing-based project. The method of claim 8,
The additional inspection unit price is calculated using the following formula,
Additional inspection unit price = Waiting time for payment of inspection fee X Unit cost X predetermined ratio
A method of paying an appropriate inspection fee for the inspection performance of the inspector of a crowdsourcing-based project. In combination with a computer, a computer-readable recording medium in order to execute the method of paying the appropriate inspection cost for the inspection performance of the inspector of the crowdsourcing-based project of any one of paragraphs 1 to 3 and 5 to 9 A computer program stored on it.

Images