KR102183838B1 - Method for adjusting the unit cost of project work using verification work of crowdsourcing based projects for artificial intelligence training data generation - Google Patents

Abstract

The present invention provides a method for adjusting the unit cost of a project job using a job for verification of a crowdsourcing-based project for generating artificial intelligence learning data which adjusts the job unit cost of a project in accordance with the evaluated competence of workers. The method comprises: a step of assigning a plurality of jobs to a plurality of workers to request the workers to perform the jobs; a step of receiving a plurality of job results from the plurality of workers; a step of assigning the plurality of job results to a plurality of inspectors to request the inspectors to inspect the job results; a step of receiving inspection pass or rejection as a plurality of inspection results for the plurality of job results from the plurality of inspectors; a step of extracting inspection-passing job results satisfying a prescribed condition to generate a job pool for verification; a step of assigning jobs for verification included in the job pool for verification to the plurality of workers to request the workers to perform the jobs; a step of receiving job results for verification from the plurality of workers; a step of calculating average job performing time (hereafter, first job performing time) for each job for verification of the plurality of workers; a step of comparing the first job performing time and job performing time (hereafter, second job performing time) recorded for each job for verification before calculation of the first job performing time; and a step of adjusting project job unit costs for the plurality of workers by using results of the comparison.

Description

{METHOD FOR ADJUSTING THE UNIT COST OF PROJECT WORK USING VERIFICATION WORK OF CROWDSOURCING BASED PROJECTS FOR ARTIFICIAL INTELLIGENCE TRAINING DATA GENERATION}

The present invention relates to a method for adjusting project work cost using a crowdsourcing-based project verification task for generating artificial intelligence learning data.

Recently, more and more companies collect and process large amounts of data on a crowdsourcing basis that engages the general public in some process of corporate activities. In other words, by opening a project and allowing the general public, that is, workers to participate in the project, necessary information is collected through the work results completed by the workers.

At this time, in order to collect more reliable information, the company assigns the work result completed by the worker to the inspector to perform the inspection work.

Specifically, when one project is opened, a plurality of tasks are assigned to each of the plurality of workers. Each worker performs a plurality of tasks assigned to it, and provides the task results. Thereafter, a plurality of inspection tasks are assigned to each of the plurality of inspectors, and each inspector performs a plurality of inspection tasks assigned to them.

On the other hand, as the project progresses, workers become more proficient in the tasks belonging to the project.

However, the project work cost does not take into account the proficiency level of these workers, and the unit price at the time of opening the project and the unit price at the second half of the project are set equally.

That is, in most cases, the initial unit cost of a project is set according to the initial worker who is not skilled, and based on these, the standard wage per unit time is set to be paid. In this case, the profit that far exceeds the standard wage per unit time in the latter half of the project. There will be many workers who get it.

This is a result of incorrectly setting the initial work cost of the project, and it is a great burden to the client. Therefore, it is necessary to change the project work cost considering that the workers become more proficient as the project progresses.

Unexamined Patent Publication No. 10-2014-0095956, 2014.08.04.

The problem to be solved by the present invention is to create a verification task during the progress of the project and provide it to the workers, check the work execution time for the verification task, and adjust the work unit cost of the project according to the proficiency of the evaluated workers. It is to provide a method of controlling the cost of project work using the verification work of crowdsourcing-based projects.

However, the problem to be solved by the present invention is not limited to the problem as described above, and other problems may exist.

In order to solve the above-described problems, a method for adjusting project work cost using a work for verifying a crowdsourcing-based project according to an aspect of the present invention is to perform a plurality of tasks of a crowdsourcing-based project (hereinafter, "project") to a plurality of workers. Assigning and requesting to perform a task; Receiving a plurality of job results from the plurality of workers; Assigning the plurality of work results to a plurality of inspectors and requesting to perform inspection; Receiving an inspection pass or rejection as a plurality of inspection results for the plurality of work results from the plurality of inspectors; Generating a work pool for verification by extracting a work result that has passed an inspection that satisfies a predetermined condition from among a plurality of inspection results for the plurality of work results during a predetermined time period; Allocating a verification task included in the verification task pool to the plurality of workers and requesting the task to be performed; Receiving verification results from the plurality of workers; Calculating an average task execution time (hereinafter, "first task execution time") for each verification task of the plurality of workers; Comparing the first task execution time with the task execution time (hereinafter, "second task execution time) recorded for each of the verification tasks before calculation of the first task execution time; And the comparison result And adjusting the project work unit cost for the plurality of workers by using the method, wherein the comparing of the first work execution time and the second work execution time includes a plurality of verification tasks of the plurality of workers. Comprising the step of calculating each deviation between the first task execution time and the second task execution time for, and adjusting the project work unit cost for the plurality of workers using the comparison result, the calculated deviation And adjusting the project work unit cost for the plurality of workers based on the average value of.

In some embodiments of the present invention, the predetermined condition may be a condition that satisfies a predetermined ranking based on the inspection time required by the inspector.

In some embodiments of the present invention, the step of generating the work pool for verification may include: requesting re-review from one or more other inspectors different from the inspector who inspected the inspection-passed work result; And adding the work result that has passed the re-check by the other inspector to the work pool for verification.

In some embodiments of the present invention, the step of allocating the verification task to a plurality of workers and requesting the task to be performed may be performed from a point in time after the task result for tasks of a predetermined ratio or more among all tasks included in the project is submitted. The verification work may be assigned to a plurality of workers and requested to perform the work.

In some embodiments of the present invention, the step of calculating the first task execution time for each verification task of the plurality of workers includes n number of tasks performed among the plurality of workers (where n is a natural number of 1 or more) Less than the operator's first task execution time for the verification task can be excluded.

Some embodiments of the present invention include calculating a rejection rate for a verification job included in the verification job pool based on the plurality of inspection results, and a verification job in which the calculated rejection rate is greater than or equal to a preset rejection rate It may further include the step of excluding the first task execution time for.

In some embodiments of the present invention, the step of excluding the first task execution time for the verification task in which the rejection rate is equal to or greater than a preset rejection rate may include a deviation between the first task execution time and the second task execution time. If the deviation is greater than or equal to the set deviation, the first task execution time may be excluded.

In some embodiments of the present invention, the step of adjusting the project work unit cost for the plurality of workers using the comparison result, the total average value of the first work execution time is the total average value of the second work execution time Reducing the work cost of the project by a ratio corresponding to the average value of each deviation when it has a shorter time; And when the total average value of the first work execution time has a longer time than the total average value of the second work execution time, increasing the work unit cost of the project by a ratio corresponding to the average value of each deviation. It may include.

In some embodiments of the present invention, after the project work cost for the plurality of workers is adjusted, the second work execution time recorded for the verification work of the verification pool is applied to each verification work of the plurality of workers. It may further include updating to the first task execution time, which is the average task execution time for.

A computer program according to another aspect of the present invention for solving the above-described problem is combined with a computer that is hardware to execute a method of adjusting the project work unit cost using the work for verification of the crowdsourcing-based project, and is stored in a computer-readable recording medium. Is saved.

Other specific details of the present invention are included in the detailed description and drawings.

According to the present invention, the proficiency level of workers changes as the project progresses, and if the initially calculated project work unit price is inappropriate from the current point in time, it is possible to adjust the work unit cost of the project in consideration of the proficiency level of the changed workers. have.

In particular, it is possible not only to adjust the initially calculated project unit cost, but also to determine whether the proficiency of workers has improved from the previous point based on the current point of time even after the project unit cost is adjusted, and adjust the project unit cost based on this.

The effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the following description.

1 is a conceptual diagram of a crowdsourcing service according to an embodiment of the present invention.
2 is a flowchart illustrating a crowdsourcing-based project progress process according to an embodiment of the present invention.
3 is a flowchart of a method for adjusting a project work cost using a work for verification of a crowdsourcing-based project according to an embodiment of the present invention.
4 and 5 are views for explaining contents of generating a verification task.
The verification work is created based on the actual work performed by the multiple workers of the project.
6 is a view for explaining the calculation and comparison of the first and second task execution times for the verification task.
7 is a diagram for explaining content of updating a second task execution time.
8 is a view for explaining a project work unit cost adjustment device according to an embodiment of the present invention.

Advantages and features of the present invention, and a method of achieving them will become apparent with reference to the embodiments described below in detail together with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but may be implemented in a variety of different forms, only the present embodiments are intended to complete the disclosure of the present invention, It is provided to fully inform the technician of the scope of the present invention, and the present invention is only defined by the scope of the claims.

The terms used in the present specification are for describing exemplary embodiments and are not intended to limit the present invention. In this specification, the singular form also includes the plural form unless specifically stated in the phrase. As used in the specification, “comprises” and/or “comprising” do not exclude the presence or addition of one or more other elements other than the mentioned elements. Throughout the specification, the same reference numerals refer to the same elements, and “and/or” includes each and all combinations of one or more of the mentioned elements. Although "first", "second", and the like are used to describe various elements, it goes without saying that these elements are not limited by these terms. These terms are only used to distinguish one component from another component. Therefore, it goes without saying that the first component mentioned below may be the second component within the technical idea of the present invention.

Unless otherwise defined, all terms (including technical and scientific terms) used in the present specification may be used as meanings that can be commonly understood by those of ordinary skill in the art to which the present invention belongs. In addition, terms defined in a commonly used dictionary are not interpreted ideally or excessively unless explicitly defined specifically.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a conceptual diagram of a crowdsourcing service according to an embodiment of the present invention.

Referring to FIG. 1, a crowdsourcing service is performed by being composed of a client 10, a service provider 20, and a public 30.

The sponsor 10 refers to a company or individual who requests a crowdsourcing-based project (hereinafter, referred to as a project).

The requester 10 requests a project for the purpose of collecting source data or annotating data for generating artificial intelligence learning data. The data generated through the project can be used as learning data for arbitrary machine learning such as supervised learning, unsupervised learning, and reinforcement learning. Collection of source data refers to collecting raw data such as recorded voice collection and photo collection. Data annotation refers to inputting relevant annotation data into source data such as text, photos, and videos. For example, the data annotation may include finding an entity in a given fingerprint or finding a similar sentence, but is not limited thereto. Meanwhile, the types of the above-described projects are only one embodiment, and various projects may be handled in the present invention according to the design of the client.

The service provider 20 refers to a company that provides crowdsourcing services.

When the service provider 20 receives a request for a project for a product or service from the client 10, the service provider 20 allocates a task for the project to the general public 30 and receives the work result from the public 30. Thereafter, the final product extracted based on the work result is provided to the client 10.

At this time, the service provider 20 provides a crowdsourcing service to the client 10 and the public 30 through a crowdsourcing platform (hereinafter, the platform). That is, when the service provider 20 receives a request for a project from the client 10, the service provider 20 opens the project on the platform. Thereafter, when the work result for the open project is received from the public 30, the project is terminated on the platform, and the final product may be extracted and provided to the client 10.

The public 30 refers to the general public participating in the project open on the platform. Here, the public 30 may participate in a project opened on the platform through an application or website provided by the service provider 20.

The public 30 is composed of a worker 32 and an inspector 34.

The worker 32 decides to participate in a specific project among a plurality of projects open on the platform. Thereafter, the operator 32 performs an operation such as collection of source data or data annotation, and transmits it to the platform.

The inspector 34 decides to participate in a specific project among a plurality of projects open on the platform. Thereafter, the inspector 34 inspects the result of the work performed by the operator 32. As a result of performing the inspection, the inspector 34 may perform inspection pass processing or rejection processing, and input a rejection reason during rejection processing. In the case of passing the inspection, it is not necessary to rework and re-examination due to this, so passing inspection has the same meaning as completion of inspection.

2 is a flowchart illustrating a crowdsourcing-based project progress process according to an embodiment of the present invention.

First, the requester 10 requests one or more projects to the service provider 20 (S11).

Thereafter, the service provider 20 opens the requested project on the platform (S12). At this time, the service provider 20 may determine the grade in consideration of the difficulty of the project before opening the project. That is, depending on the difficulty level, it is possible to determine which level or higher public 30 to expose the project. Accordingly, it is possible to increase the reliability of the work result of the project.

Thereafter, the service provider 20 requests the work by assigning the work to the workers 32 of the corresponding level or higher according to the level of the project (S13).

Thereafter, the worker 32 performs the assigned task (S14). In this case, the worker 32 may input the reason for the inability to work without performing the work for a work in which the work itself is impossible for some reason.

Thereafter, the service provider 20 receives the work result from the worker 32 (S15), and assigns the work result to the inspector 34 to request the inspection (S16).

Similarly, according to an embodiment of the present invention, only a suitable project may be exposed to the inspector 34 out of the total projects being performed according to the level set according to the difficulty of the project or the qualification requirements of the inspector 34.

Thereafter, the inspector 34 performs the assigned inspection (S17). At this time, the inspector 34 determines that the inspection is completed when it is determined that the work has been properly performed, and if it is determined that the inspection operation is wrong, the inspector 34 rejects it. During rejection processing, the inspector 34 inputs the rejection reason for what reason the job was judged to be wrong.

Thereafter, the service provider 20 receives the inspection result from the inspector 34 (S18).

When the inspection result is the inspection completion, the service provider 20 uses the work result as valid data, and based on this, extracts the final product at the end of the project.

When the inspection result is rejection processing, the service provider 20 may internally perform the inspection again, or assign the work to the worker 32 again to perform the rework. When reworking, re-examination by an inspector is required.

Thereafter, when the project period ends or sufficient valid data is secured, the service provider 20 terminates the project (S19), calculates a final result based on the secured valid data, and provides it to the client 10 ( S20).

At this time, before the end of the project, the service provider 20 evaluates the performance results of the worker 32 and the inspector 34, calculates the work cost and the inspection cost according to the evaluation, to the worker 32 and the inspector 34. give.

In FIGS. 1 and 2, it is simply expressed as a requester 10, a service provider 20, an operator 32, and an inspector 34, but these are smartphones, tablets, PDAs, laptops, desktops, operated by each participant. It means a computer device such as a server or a telecommunication device.

Hereinafter, a method of adjusting a project work unit cost using a work for verifying a crowdsourcing-based project according to an embodiment of the present invention will be described with reference to FIGS. 3 to 7.

3 is a flowchart of a method for adjusting a project work cost using a work for verification of a crowdsourcing-based project according to an embodiment of the present invention.

Meanwhile, the steps shown in FIG. 3 may be understood as being performed by a platform server (hereinafter, referred to as a server) operated by the service provider 20, but are not limited thereto.

In addition, a plurality of workers 32 or a plurality of inspectors 34 perform work using a predetermined terminal device. The terminal device of the operator 32 or the inspector 34 may be a computer device or a telecommunication device such as a smartphone, tablet, PDA, laptop, desktop, etc., but is not limited thereto.

First, referring to FIG. 3, the server assigns a plurality of tasks of a crowdsourcing-based project (hereinafter referred to as a project) to a plurality of workers 32 and requests the task to be performed (S105), and a plurality of workers 32 A plurality of work results are inputted from (S110).

Here, one project may contain multiple jobs, and the plurality of jobs may be grouped into a plurality of tasks according to their type, or each job may be a different type of job or the same job. , But is not limited thereto.

Next, the server assigns the plurality of work results to the plurality of inspectors 34 to request the inspection performance (S115), and receives a plurality of inspection results for the plurality of job results from the plurality of inspectors 34 (S120). ). At this time, the inspector 34 may input the inspection pass or rejection as a plurality of inspection results.

Next, the server extracts a job result that has passed the inspection that satisfies a predetermined condition from among the plurality of inspection results for the plurality of job results, and generates a job pool for verification (S125).

For reference, in the description of the present invention, a task for verifying the operator 32 is referred to as a'verification operation', and the result of the operation is referred to as a'verification operation result'. To be distinguished from this, a task that the worker 32 actually performs the task and should be used as a project result is referred to as a'practical-only work', and the work result is referred to as a'practical-only work result'.

In an embodiment, the server may extract, from among a plurality of inspection results for a plurality of job results during a predetermined period, a job result that has passed the inspection that satisfies a predetermined condition and includes it in the verification job pool.

This is to verify the plurality of workers 32 who will perform the work in the second time period (t1) by using the work result of the first worker who performed the work in the first time period (t0), and similarly, the second time This is a structure in which a plurality of workers 32 who will perform the work in the third time period t2 are verified using the work result of the second worker who performed the work in the section t1.

Hereinafter, the verification operation will be described in detail with reference to FIGS. 4 and 5.

4 and 5 are views for explaining contents of generating a verification task.

The verification work is generated based on the practical work actually performed by the plurality of workers 32 of the project.

First, referring to FIG. 4, when a work result (P1) for a practical work performed by a plurality of workers 32 is input, the practical work result (P1) is assigned to the inspector 34, and the inspector 34 ) Performs an inspection on the actual work result (P1).

Then, when the inspection by the inspector 34 on the practical work result P1 is completed, the server extracts the inspection result P2 from among the plurality of inspection results. In an embodiment of the present invention, since the verification task is generated based on the verification pass, that is, the initial verification pass or re-review pass, the verification task is determined as the correct answer as the verification pass or rejection, and whether or not the correct answer is entered Through this, the operator 32 is verified.

Then, the server extracts the inspection-passed job result P3 that satisfies a predetermined condition among the inspection-passed job results P2 and generates a verification job pool. In this case, as an embodiment of the predetermined condition, a predetermined ranking based on the inspection time required by the inspector 34 may be satisfied.

That is, the server sorts the actual work results P2 that have passed the inspection in the order in which the time required for the inspection is long, and extracts only the practical work results P3 corresponding to a predetermined priority among them.

Here, the long time spent for inspection means that the inspection is more likely to be faithfully performed compared to other inspectors 34, and this means that the probability of errors in the inspection results is low. It is desirable to extract the exclusive work result (P3) and include it in the work pool for verification.

In this case, the predetermined ranking may be set in advance by the server, the administrator, or the requester 10, and the predetermined ranking may be extracted not only in units of number but also in units of time.

As described above, the inspection result P3 that satisfies the predetermined condition may be immediately included in the verification job pool.

However, since there is a possibility that the inspector 34 who has inspected the actual work result may also make an error, another inspector 34 re-examines the inspected work result in order to provide a more verified work to the operator 32. After performing the re-examination, it is possible to include the results of the practical work that has passed the re-examination in the work pool for verification.

That is, if the server extracts the work result (P3) that has passed the inspection and satisfies a predetermined condition among the work results (P2) that have passed the inspection for the plurality of work results (P1), the work result (P3) is inspected. It is possible to request a re-examination from one or more inspectors 34 different from the inspector 34 (P4), and a work result (P5) that has passed the inspection by the inspector who has re-examined may be generated as a verification operation (P6).

Referring to FIG. 5, when a result of the practical use that satisfies a predetermined priority is extracted, the result of the practical use is subjected to an additional inspection process by different inspectors 34, respectively.

For example, in FIG. 4, when the first inspector passes the inspection, three'practical work results 1 to 3 (P3) according to a predetermined ranking. 'Is extracted, and the three practical work results (P3) are re-inspected by the second and third inspectors different from the first inspector (P4). In this case, in the example of FIG. 5, it is shown that the re-examination is performed by two inspectors, but is not limited thereto.

As a result of re-inspection by the 2nd and 3rd inspectors, if the 2nd inspector passed the inspection and only the'practical work result 1'passed by the third inspector, Only the'practical work result 1 (P5)' that has been inspected by all the inspectors 34 may be included in the verification work pool as the verification work (P6).

After the verification work pool is generated, the verification work pool may be updated by checking whether the condition is satisfied in real time or at predetermined intervals with respect to the verification result of the actual work result.

On the other hand, the task is not performed for the actual task, there is no actual task that satisfies a predetermined condition, or all the verification tasks included in the verification task pool are assigned to the worker 32 for further verification. There may be conditions in which the dragon task cannot be assigned. In this case, an embodiment of the present invention may add a verification task prepared in advance by the client 10 to the verification pool, and the time to be added is not limited to a specific period of the project.

Referring back to FIG. 3, as the verification task pool is created, the server assigns verification tasks included in the verification task pool to a plurality of workers 32 to request the task performance (S130), and a plurality of workers 32 ) Receives the verification result from (S135).

In this case, the verification work may be assigned to the plurality of workers 32 from a point in time after a work result for a task having a predetermined ratio or more of all tasks included in the project is submitted.

In other words, since proficient workers (32) rarely appear at the beginning of the project, pre-allocation of the verification work may be a waste of resources, so the verification work from the time when the work results corresponding to the mid-to-late part of the project are submitted. It is desirable that the allocation of

However, the timing of assignment of verification tasks is not necessarily limited thereto, and of course, they may be assigned from the beginning of the project to periodically check the proficiency level of the worker pool.

As the server receives the verification work results from the plurality of workers 32, the average work execution time of the workers 32 who have performed the verification work as shown in FIG. 6, and the verification work are recorded. By calculating and comparing the work execution time, it is possible to adjust the project work unit cost for the plurality of workers 32.

6 is a view for explaining the calculation and comparison of the first and second task execution times for the verification task.

The server calculates an average work execution time (hereinafter, referred to as "first work execution time") for each verification task of the plurality of workers 32 (S140).

In this case, the average work execution time is an average of the work execution time calculated based on the start time of performing the verification work assigned to each worker 32 and the time when the work execution is completed and the work result is input.

In one embodiment, the first task execution time for the verification task of the worker 32 whose number of tasks is less than n (where n is a natural number of 1 or more) among the plurality of workers 32 may be excluded.

For example, a worker 32 with fewer than n tasks performed may be a new worker unfamiliar with performing the task, and if too many new workers' data is considered together, it will be a problem in evaluating the current skilled worker pool. It can be ruled out.

Next, the server compares the first task execution time with the task execution time (hereinafter referred to as "second task execution time") recorded for each verification task before the calculation of the first task execution time (S145 ), by using the comparison result to adjust the project work unit cost for the plurality of workers 32 (S150).

Here, the second task execution time is a time recorded for the verification task before the first task execution time is calculated, and the task execution time day of the worker 32 who performed the practical task corresponding to the verification task I can.

In addition, when the second task execution time is updated to the first task execution time, the second task execution time may be an average task execution time obtained by performing the verification task results of the plurality of workers 32.

The server calculates deviations between the first task execution time and the second task execution time for a plurality of verification tasks of the plurality of workers 32 and compares the first and second task execution times. And, based on the calculated average value of each deviation, the project cost for the plurality of workers 32 is adjusted.

In an embodiment, the server may calculate an overall average value of the execution time of the first task for the plurality of verification tasks and the overall average value of the execution time of the second task for the plurality of verification tasks.

In addition, when the total average value of the first work execution time has a shorter time than the total average value of the second work execution time, the work unit cost of the project may be reduced by a ratio corresponding to the average value of each deviation.

For example, in FIG. 6, the total average value of the first task execution time for the verification task IDs 1001 to 1003 is 88/3 seconds ((30s+40s+18s)/3), and the total of the second task execution time The average value is 125/3 seconds ((40s+60s+25s)/3), and the overall average value of the first task execution time has a shorter time than the overall average value of the second task execution time. In this case, the server can reduce the work cost of the project by 12.3%, a ratio corresponding to the average value (-37/3=-12.3) of each deviation (-10, -20, -7).

In contrast, when the total average value of the first work execution time has a longer time than the total average value of the second work execution time, the work unit cost of the project may be increased by a ratio corresponding to the average value of each deviation.

In addition, in the present invention, there may be various embodiments of adjusting the work unit cost of a project based on the average value of each deviation of the first and second work execution times, and in addition to applying a ratio to the average value of the deviation, You can also adjust the project unit cost by classifying and applying the average value by grade (-11~-20=-10%, -21~30=-20%, etc.).

On the other hand, when the difficulty of the verification task is too high, the deviation for each worker 32 may be too large in the task execution time as well as the task execution result. When these results are used, a result different from the proficiency trend of the workers 32 is reflected when adjusting the project work cost. To prevent this, an embodiment of the present invention has too high difficulty in adjusting the project work cost. Data on verification work can be excluded.

In other words, the server calculates the rejection rate for the verification job included in the verification job pool based on the results of multiple inspections, and then, in the case of the verification job where the calculated rejection rate is higher than the preset rejection rate, it is for verification with high difficulty. By judging as a task, the time to perform the first task can be excluded.

For example, if the rejection rate for one verification task is a multi-allocation method in which one practical task is assigned to a plurality of workers 32, worker A passes the initial inspection, and worker B is returned once and then inspected. If passed, worker C is rejected twice and passed inspection, the rejection rate can be calculated as (total of 3 rejections/3 people) = 100%. Alternatively, if only the results of the initial inspection are considered (total 2 times/three persons) = 66.7% can be calculated.

In this case, when the deviation between the first task execution time and the second task execution time is greater than or equal to a preset deviation, the server may exclude the first task execution time.

That is, even if the rejection rate for the verification work is high, the work execution time of the worker 32 who has previously performed the actual work and the average work execution time of the workers 32 who have performed the current verification work When there is little deviation, it can be seen that there is little deviation in proficiency among all the workers 32. On the other hand, in the opposite case, since there is a high possibility that a large deviation exists among the workers 32 who have currently performed the verification work, in this case, the first work execution time can be excluded.

After the work unit cost of the project is adjusted, the server may exclude previously used data from the index in order to grasp the trends of the workers 32 based on the current standard.

7 is a diagram for explaining content of updating a second task execution time.

After the project unit cost for the plurality of workers 32 is adjusted, the server calculates the second work execution time recorded for the verification work of the verification pool as an average work for each verification work of the plurality of workers 32 It can be updated to the execution time of the first task.

When the project unit cost is adjusted for the first time, the second task execution time of the worker 32 who performed the practical task corresponding to the verification task is excluded, and the second task execution time is a plurality of the verification tasks. It is updated with the first task execution time of the workers 32.

After the project unit cost is adjusted again, the second task execution time recorded as the average task execution time of the plurality of workers 32 is excluded, and the first task execution time, which is the average task execution time of another plurality of workers 32 The time is updated to the second task execution time.

For example, if there are 100 workers, and verification tasks such as ID 1001 are assigned to 50 people in a random order, and the project unit cost increases from 1,000 won to 1,500 won based on the data, 50 The second task execution time is updated as the average task execution time.

After the project was further progressed, verification tasks such as ID 1001 were assigned to the remaining 50 workers in a random order, and as the workers 32 became more proficient in the latter part of the project, the project unit cost was from 1,500 won to 2,000. In the case of rising to a circle, the average work execution time for the previous 50 people is excluded, and the second work execution time is updated with the average work execution time for the current 50 people.

Through such an update process, it is possible to more accurately determine whether the current workers 32 are more proficient than the existing ones.

Meanwhile, in the above description, steps S11 to S150 may be further divided into additional steps or may be combined into fewer steps according to an embodiment of the present invention. In addition, some steps may be omitted as necessary, and the order between steps may be changed. In addition, even if other contents are omitted, the contents of FIG. 8 to be described later can also be applied to the method of adjusting the project work unit cost using the work for verification of the crowdsourcing-based project of FIGS. 1 to 7.

Hereinafter, a description will be given of a project work cost control device 200 (hereinafter referred to as a project work cost control device) using a work for verifying a crowdsourcing-based project according to an embodiment of the present invention.

8 is a view for explaining a project work unit cost adjustment device 200 according to an embodiment of the present invention.

Referring to FIG. 8, the apparatus 200 for adjusting the unit cost of a project according to the present invention includes a communication module 210, a memory 220, and a processor 230.

The communication module 210 transmits a crowdsourcing-based work for one project to a plurality of workers 32 to request a task to be performed, and receives a work result from the plurality of workers 32. Further, the work results received from the plurality of workers 32 are transmitted to the plurality of inspectors 34 to request inspection, and the inspection results are received from the plurality of inspectors 34. In addition, the communication module 210 allocates a verification task to a plurality of workers 32 to request the task to be performed, and receives a task result for the verification task from the plurality of workers 32.

The memory 220 stores a program for adjusting the project work cost of the plurality of workers 32 based on the work execution time for the verification work result.

As the processor 230 executes the program stored in the memory 220, after generating the verification task pool, assigns the verification task to the plurality of workers 32, and the average task execution time for the verification task result And, by comparing the work execution time recorded in the verification work, the work unit cost of the project of the workers 32 is adjusted.

The method for controlling project work cost using the work for verifying a crowdsourcing-based project according to an embodiment of the present invention described above is implemented as a program (or application) to be executed in combination with a computer that is hardware and stored in a medium. I can.

The above-described program includes C, C++, JAVA, Ruby, which can be read by a processor (CPU) of the computer through the device interface of the computer, in order for the computer to read the program and execute the methods implemented as a program. It may include a code (Code) coded in a computer language such as machine language. Such code may include a functional code related to a function defining necessary functions for executing the methods, and a control code related to an execution procedure necessary for the processor of the computer to execute the functions according to a predetermined procedure. can do. In addition, such code may further include additional information required for the processor of the computer to execute the functions or code related to a memory reference to which location (address address) of the internal or external memory of the computer should be referenced. have. In addition, when the processor of the computer needs to communicate with any other computer or server in the remote in order to execute the functions, the code uses the communication module of the computer to determine how It may further include a communication-related code for whether to communicate, what kind of information or media should be transmitted and received during communication.

The stored medium is not a medium that stores data for a short moment, such as a register, cache, memory, etc., but a medium that stores data semi-permanently and can be read by a device. Specifically, examples of the storage medium include, but are not limited to, ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, and the like. That is, the program may be stored in various recording media on various servers to which the computer can access, or on various recording media on the user's computer. In addition, the medium may be distributed over a computer system connected through a network, and computer-readable codes may be stored in a distributed manner.

The above description of the present invention is for illustrative purposes only, and those of ordinary skill in the art to which the present invention pertains will be able to understand that other specific forms can be easily modified without changing the technical spirit or essential features of the present invention will be. Therefore, it should be understood that the embodiments described above are illustrative in all respects and not limiting. For example, each component described as a single type may be implemented in a distributed manner, and similarly, components described as being distributed may also be implemented in a combined form.

The scope of the present invention is indicated by the claims to be described later rather than the detailed description, and all changes or modified forms derived from the meaning and scope of the claims and their equivalent concepts should be interpreted as being included in the scope of the present invention. do.

10: Client
20: service provider
30: public
32: worker
34: inspector
200: Project work unit cost control device
210: communication module
220: memory
230: processor

Claims (10)

As a method performed by a computer,
Assigning a plurality of practical tasks of a crowdsourcing-based project (hereinafter, "project") to a plurality of workers and requesting to perform the task;
Receiving a plurality of practical work results from the plurality of workers;
Allocating the plurality of practical work results to a plurality of inspectors and requesting to perform inspection;
Receiving an inspection pass or rejection as a plurality of inspection results for the plurality of practical work results from the plurality of inspectors;
Based on a plurality of inspection results of the plurality of practical work results during a first time period, a plurality of verification jobs is generated using a practical job corresponding to a practical job result that satisfies a predetermined condition. step;
Requesting to perform a task by assigning the plurality of verification tasks to a plurality of workers who perform tasks by receiving the plurality of practical tasks during a second time section after the first time section;
Receiving work results for the plurality of verification tasks from the plurality of workers;
Calculating an average work execution time (hereinafter, "first work execution time") of the plurality of workers in relation to each verification task;
Comparing the first task execution time for each verification task and the task execution time recorded for each practical task corresponding to each verification task (hereinafter, "second task execution time); And
Including the step of adjusting the project work unit cost for the plurality of workers by using the comparison result,
Comparing the first task execution time for each verification task and the task execution time recorded for each practical task corresponding to each verification task (hereinafter, "second task execution time),
Comprising the step of calculating a deviation between the first task execution time and the second task execution time for each of the plurality of verification tasks,
Adjusting the project work unit cost for the plurality of workers using the comparison result,
Including the step of adjusting the project work unit cost for the plurality of workers based on the average value of the deviation calculated for each of the plurality of verification tasks,
A method of adjusting the cost of project work using the verification work of crowdsourced projects. The method of claim 1,
The predetermined conditions are:
It is a condition that satisfies a predetermined ranking based on the inspection execution time of the inspector,
A method of controlling the cost of project work using the verification work of crowdsourced projects. The method of claim 2,
Based on a plurality of inspection results of the plurality of practical work results during a first time period, a plurality of verification jobs is generated using a practical job corresponding to a practical job result that satisfies a predetermined condition. The steps are,
Requesting re-examination from at least one other inspector different from the inspector who inspects the results of the practical use that satisfy the predetermined condition; And
Comprising the step of generating the plurality of verification tasks using a practical job corresponding to a result of a practical job that satisfies the predetermined condition and has been re-examined by the other inspector,
A method of controlling the cost of project work using the verification work of crowdsourced projects. The method of claim 1,
The step of requesting to perform a task by assigning the plurality of verification tasks to a plurality of workers who are assigned the plurality of practical tasks and perform tasks during a second time interval after the first time interval,
Allocating the verification work to a plurality of workers from the time when the results of the practical work for the practical work of more than a predetermined percentage of the total work included in the project are submitted to request the execution of the work,
A method of controlling the cost of project work using the verification work of crowdsourced projects. The method of claim 1,
The step of calculating the average work execution time (hereinafter, "first work execution time") of the plurality of workers in relation to each of the verification tasks,
The number of practical tasks among the plurality of workers is calculated by excluding and calculating the task execution time of workers with less than n (however, n is a natural number of 1 or more).
A method of controlling the cost of project work using the verification work of crowdsourced projects. The method of claim 1,
The step of calculating a deviation between the first task execution time and the second task execution time for each of the plurality of verification tasks,
Calculating rejection rates for the plurality of verification tasks based on the plurality of inspection results,
The calculated rejection rate is to exclude and calculate the first task execution time for the verification task that is more than a preset rejection rate,
A method of controlling the cost of project work using the verification work of crowdsourced projects. The method of claim 6,
Excluding and calculating the first task execution time for the verification task in which the calculated rejection rate is greater than or equal to a preset rejection rate,
Performing the first task execution time for the verification task in which the calculated rejection rate is greater than or equal to a preset rejection rate and the second task for practical tasks corresponding to the verification task in which the calculated rejection rate is greater than or equal to a preset rejection rate When the deviation of the time is greater than or equal to a preset deviation, the calculated rejection rate excludes and calculates the first task execution time for the verification task whose rejection rate is greater than or equal to the preset rejection rate,
A method of controlling the cost of project work using the verification work of crowdsourced projects. The method of claim 1,
Adjusting the project work unit cost for the plurality of workers using the comparison result,
When the overall average value of the first task execution time for each verification task has a shorter time than the overall average value of the second task execution time for each practical task corresponding to each verification task, the calculation Reducing the work unit cost of the project for the plurality of workers by a ratio corresponding to the average value of the deviation; And
When the total average value of the first task execution time for each verification task is longer than the overall average value of the second task execution time for each practical task corresponding to each verification task, the calculation Comprising the step of increasing the work unit cost of the project for the plurality of workers by a ratio corresponding to the average value of the deviation,
A method of controlling the cost of project work using the verification work of crowdsourced projects. The method of claim 1,
After the project work cost for the plurality of workers is adjusted, the second work execution time for each practical work corresponding to each verification work is calculated in relation to each verification work, and the average work performed by the plurality of workers Further comprising the step of updating the first task execution time, which is a time,
A method of controlling the cost of project work using the verification work of crowdsourced projects. A computer program combined with a computer and stored in a computer-readable recording medium to execute the method of adjusting the project work unit cost using the work for verifying the crowdsourcing-based project according to any one of claims 1 to 9.

Images