KR102183837B1 - Method for paying work fee based on hourly wage for each workers using verification work of crowdsourcing based projects for artificial intelligence training data generation - Google Patents

Abstract

Provided is a method for paying a work fee based on an hourly wage for each worker using a verification work of a crowdsourcing-based project for generating artificial intelligence training data to stably manage project expenses. According to the present invention, the method comprises the following steps of: assigning a plurality of works to a plurality of workers to request execution of the work; receiving a plurality of work results from the plurality of workers; assigning the plurality of work results to a plurality of inspectors to request execution of inspection; receiving an inspection pass or rejection from a plurality of inspectors as a plurality of inspection results for the plurality of work results; using the work results to select (i) at least one first worker, who receives a work fee by a unit wage per work execution case and (ii) at least one second worker, who receives the work fee by an hourly wage per time, among the plurality of workers; and paying the work fee to the first worker based on the unit wage per work execution case and paying the work fee to the second worker by the hourly wage per unit time.

Description

METHOD FOR PAYING WORK FEE BASED ON HOURLY WAGE FOR EACH WORKERS USING VERIFICATION WORK OF CROWDSOURCING BASED PROJECTS FOR ARTIFICIAL INTELLIGENCE TRAINING DATA GENERATION}

The present invention relates to a method for paying work cost based on an hourly wage system for each worker using a work for verification of a crowdsourcing based project for generating artificial intelligence learning data.

Recently, more and more companies collect and process large amounts of data on a crowdsourcing basis that engages the general public in some process of corporate activities. In other words, by opening a project and allowing the general public, that is, workers to participate in the project, necessary information is collected through the work results completed by the workers.

At this time, in order to collect more reliable information, the company assigns the work result completed by the worker to the inspector to perform the inspection work.

Specifically, when one project is opened, a plurality of tasks are assigned to each of the plurality of workers. Each worker performs a plurality of tasks assigned to it, and provides the task results. Thereafter, a plurality of inspection tasks are assigned to each of the plurality of inspectors, and each inspector performs a plurality of inspection tasks assigned to them.

On the other hand, as the project progresses, workers become more proficient in the tasks belonging to the project. However, currently, the cost of work for workers is paid as a unit wage per job execution set according to the inexperienced workers at the beginning of the project, without taking into account the proficiency of the workers according to the progress of the project.

If work costs are paid throughout the project at the unit wage set in this way, as the latter part of the project, more workers earn profits that far exceed the standard wage per unit time.

Therefore, the project sponsors want to select skilled workers and convert them into an hourly wage system, but there is no clear standard for this.

Unexamined Patent Publication No. 10-2014-0095956, 2014.08.04.

The problem to be solved by the present invention is to create a verification task during the progress of the project and provide it to the workers, and by checking the work execution time for the verification task, the worker who has become proficient compared to the average of all workers is paid per hour. It is to provide a method of paying work costs based on an hourly wage system for each worker using the verification work of a crowdsourcing-based project in which work costs are paid by selecting workers to be paid, and the remaining workers are paid in unit wages per work performed.

However, the problem to be solved by the present invention is not limited to the problem as described above, and other problems may exist.

According to an aspect of the present invention for solving the above-described problem, a method of paying work cost based on an hourly wage system for each worker using a work for verification of a crowdsourcing-based project is performed by performing a plurality of tasks of a crowdsourcing-based project (hereinafter, "project"). Assigning to a plurality of workers and requesting to perform a task; Receiving a plurality of job results from the plurality of workers; Assigning the plurality of work results to a plurality of inspectors and requesting to perform inspection; Receiving an inspection pass or rejection as a plurality of inspection results for the plurality of work results from the plurality of inspectors; Using the work result, among the plurality of workers, (i) at least one first worker who will pay the work cost in unit wage per job execution and (ii) at least one second worker who will pay the work cost in unit wage per hour. Selecting a worker; And paying a work cost to the first worker based on a unit wage per job execution, and paying the work cost to the second worker based on a unit wage per hour. At this time, the plurality of tasks requesting to perform the task includes a plurality of verification tasks generated from a result of a task that has passed an inspection that satisfies a predetermined condition among a plurality of inspection results for the plurality of task results, and the task Using the result, the step of selecting the first worker and the second worker may include calculating an individual task execution time (hereinafter, "first task execution time") for each verification task of the plurality of workers; Calculating an average task execution time (hereinafter, "second task execution time") for each verification task of the plurality of workers; Selecting the second worker among the plurality of workers based on a comparison result of the first and second task execution times; And selecting the remaining workers other than the second worker among the plurality of workers as the first worker.

In some embodiments of the present invention, the step of selecting a second worker from among the plurality of workers based on the comparison result of the first and second task execution times, for each verification provided to any individual worker among the plurality of workers Sorting the tasks in chronological order; Calculating a deviation of the execution time of the first and second tasks for each of the verification tasks; And selecting at least one individual worker in which the calculated trend for each deviation falls within a predetermined range as a second worker.

In some embodiments of the present invention, the step of selecting at least one individual worker in which the calculated trend for each deviation falls within a predetermined range as a second worker, wherein the calculated trend for each deviation is a predetermined value And at least one individual worker included in the time range may be selected as the second worker.

In some embodiments of the present invention, the step of selecting a first worker and a second worker by using the work result includes: sorting each verification task provided to any individual worker among the plurality of workers in chronological order; Calculating an average correct answer rate based on the correct answer rate and the correct answer rate of the plurality of workers for each of the plurality of verification tasks; Setting a difficulty level for each of the verification tasks based on the average correct answer rate; And selecting, as the second worker, a worker who satisfies at least one of a predetermined number of times and a time condition for inputting a correct answer to the verification task having a difficulty of a certain level or higher among the plurality of workers.

In some embodiments of the present invention, the step of selecting a worker who satisfies at least one of a predetermined number of times and a time condition as the second worker by inputting a correct answer to the verification task having a certain difficulty level or more among the plurality of workers is , According to the comparison result of the first and second task execution times, it may be performed when the number of preset minimum second workers is not selected.

In some embodiments of the present invention, the predetermined condition may be a condition that satisfies a predetermined ranking based on the inspection time required by the inspector.

In some embodiments of the present invention, the plurality of verification tasks is performed by extracting some of the work results that have passed re-review by one or more other inspectors different from the inspector who has inspected the inspection-passed work results corresponding to the verification tasks. Can be created by

In some embodiments of the present invention, the step of selecting the first worker and the second worker using the work result is after the time when the work result for work of a predetermined ratio or more of the total work included in the project is submitted. From, the first and second workers can be selected.

In some embodiments of the present invention, the step of calculating the second task execution time for each verification task of the plurality of workers includes n number of tasks performed among the plurality of workers (where n is a natural number of 1 or more) Less than the operator's second task execution time for the verification task can be excluded.

A computer program according to another aspect of the present invention for solving the above-described problems is combined with a computer as hardware to execute a method of paying an hourly wage system based on each worker using the work for verification of the crowdsourcing-based project, and computer reading It is stored on a recordable medium.

Other specific details of the present invention are included in the detailed description and drawings.

According to the present invention described above, by converting proficient workers who perform more tasks over a certain period of time than other workers to an hourly wage system and paying work costs, the project cost can be managed more stably by reducing the overall project estimate from the perspective of the project client. The advantage is that you can do it.

In addition, in terms of workers participating in the project, it is possible to receive work costs more stably compared to receiving work costs in terms of the number of work performed. In particular, work costs are not paid when returning work results, and work costs are paid only for work results that have passed inspection, but it is the result of receiving work costs for reworking after returning. In the case of conversion, high satisfaction from not only the client but also the worker can be expected.

In addition, since the number of workers participating in the project is quite large, it may be difficult to receive task assignments due to competition as the project reaches the mid-to-late end, unless it is a multi-allocation method. As a result of stably receiving task assignments, the satisfaction of workers can be improved.

The effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the following description.

1 is a conceptual diagram of a crowdsourcing service according to an embodiment of the present invention.
2 is a flowchart illustrating a crowdsourcing-based project progress process according to an embodiment of the present invention.
3 is a flowchart of a method of paying an hourly wage system-based work cost for each worker using a work for verification of a crowdsourcing-based project according to an embodiment of the present invention.
Hereinafter, the verification operation will be described in detail with reference to FIGS. 4 and 5.
6 is a flowchart of a process of selecting the first and second workers using the verification work result.
7 is a diagram for explaining a process of selecting a second worker.
8 is a view for explaining the content of selecting a second worker based on the difficulty of the verification work.
9 is a view for explaining an hourly wage system-based work cost payment device for each worker according to an embodiment of the present invention.

Advantages and features of the present invention, and a method of achieving them will become apparent with reference to the embodiments described below in detail together with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but may be implemented in a variety of different forms, only the present embodiments are intended to complete the disclosure of the present invention, It is provided to fully inform the technician of the scope of the present invention, and the present invention is only defined by the scope of the claims.

The terms used in the present specification are for describing exemplary embodiments and are not intended to limit the present invention. In this specification, the singular form also includes the plural form unless specifically stated in the phrase. As used in the specification, “comprises” and/or “comprising” do not exclude the presence or addition of one or more other elements other than the mentioned elements. Throughout the specification, the same reference numerals refer to the same elements, and “and/or” includes each and all combinations of one or more of the mentioned elements. Although "first", "second", and the like are used to describe various elements, it goes without saying that these elements are not limited by these terms. These terms are only used to distinguish one component from another component. Therefore, it goes without saying that the first component mentioned below may be the second component within the technical idea of the present invention.

Unless otherwise defined, all terms (including technical and scientific terms) used in the present specification may be used as meanings that can be commonly understood by those of ordinary skill in the art to which the present invention belongs. In addition, terms defined in a commonly used dictionary are not interpreted ideally or excessively unless explicitly defined specifically.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a conceptual diagram of a crowdsourcing service according to an embodiment of the present invention.

Referring to FIG. 1, a crowdsourcing service is performed by being composed of a client 10, a service provider 20, and a public 30.

The sponsor 10 refers to a company or individual who requests a crowdsourcing-based project (hereinafter, referred to as a project).

The requester 10 requests a project for the purpose of collecting source data or annotating data for generating artificial intelligence learning data. The data generated through the project can be used as learning data for arbitrary machine learning such as supervised learning, unsupervised learning, and reinforcement learning. Collection of source data refers to collecting raw data such as recorded voice collection and photo collection. Data annotation refers to inputting relevant annotation data into source data such as text, photos, and videos. For example, the data annotation may include finding an entity in a given fingerprint or finding a similar sentence, but is not limited thereto. Meanwhile, the types of the above-described projects are only one embodiment, and various projects may be handled in the present invention according to the design of the client.

The service provider 20 refers to a company that provides crowdsourcing services.

When the service provider 20 receives a request for a project for a product or service from the client 10, the service provider 20 allocates a task for the project to the general public 30 and receives the work result from the public 30. Thereafter, the final product extracted based on the work result is provided to the client 10.

At this time, the service provider 20 provides a crowdsourcing service to the client 10 and the public 30 through a crowdsourcing platform (hereinafter, the platform). That is, when the service provider 20 receives a request for a project from the client 10, the service provider 20 opens the project on the platform. Thereafter, when the work result for the open project is received from the public 30, the project is terminated on the platform, and the final product may be extracted and provided to the client 10.

The public 30 refers to the general public participating in the project open on the platform. Here, the public 30 may participate in a project opened on the platform through an application or website provided by the service provider 20.

The public 30 is composed of a worker 32 and an inspector 34.

The worker 32 decides to participate in a specific project among a plurality of projects open on the platform. Thereafter, the operator 32 performs an operation such as collection of source data or data annotation, and transmits it to the platform.

The inspector 34 decides to participate in a specific project among a plurality of projects open on the platform. Thereafter, the inspector 34 inspects the result of the work performed by the operator 32. As a result of performing the inspection, the inspector 34 may perform inspection pass processing or rejection processing, and input a rejection reason during rejection processing. In the case of passing the inspection, it is not necessary to rework and re-examination due to this, so passing inspection has the same meaning as completion of inspection.

2 is a flowchart illustrating a crowdsourcing-based project progress process according to an embodiment of the present invention.

First, the requester 10 requests one or more projects to the service provider 20 (S11).

Thereafter, the service provider 20 opens the requested project on the platform (S12). At this time, the service provider 20 may determine the grade in consideration of the difficulty of the project before opening the project. That is, depending on the difficulty level, it is possible to determine which level or higher public 30 to expose the project. Accordingly, it is possible to increase the reliability of the work result of the project.

Thereafter, the service provider 20 requests the work by assigning the work to the workers 32 of the corresponding level or higher according to the level of the project (S13).

Thereafter, the worker 32 performs the assigned task (S14). In this case, the worker 32 may input the reason for the inability to work without performing the work for a work in which the work itself is impossible for some reason.

Thereafter, the service provider 20 receives the work result from the worker 32 (S15), and assigns the work result to the inspector 34 to request the inspection (S16).

Similarly, according to an embodiment of the present invention, only a suitable project may be exposed to the inspector 34 out of the total projects being performed according to the level set according to the difficulty of the project or the qualification requirements of the inspector 34.

Thereafter, the inspector 34 performs the assigned inspection (S17). At this time, the inspector 34 determines that the inspection is completed when it is determined that the work has been properly performed, and if it is determined that the inspection operation is wrong, the inspector 34 rejects it. During rejection processing, the inspector 34 inputs the rejection reason for what reason the job was judged to be wrong.

Thereafter, the service provider 20 receives the inspection result from the inspector 34 (S18).

When the inspection result is the inspection completion, the service provider 20 uses the work result as valid data, and based on this, extracts the final product at the end of the project.

When the inspection result is rejection processing, the service provider 20 may internally perform the inspection again, or assign the work to the worker 32 again to perform the rework. When reworking, re-examination by an inspector is required.

Thereafter, when the project period ends or sufficient valid data is secured, the service provider 20 terminates the project (S19), calculates a final result based on the secured valid data, and provides it to the client 10 ( S20).

At this time, before the end of the project, the service provider 20 evaluates the performance results of the worker 32 and the inspector 34, calculates the work cost and the inspection cost according to the evaluation, to the worker 32 and the inspector 34. give.

In FIGS. 1 and 2, it is simply expressed as a requester 10, a service provider 20, an operator 32, and an inspector 34, but these are smartphones, tablets, PDAs, laptops, desktops, operated by each participant. It means a computer device such as a server or a telecommunication device.

Hereinafter, with reference to FIGS. 3 to 8, a method of paying an hourly wage based work cost for each worker 32 using a work for verifying a crowdsourcing-based project according to an embodiment of the present invention will be described.

FIG. 3 is a flowchart of a method of paying an hourly wage system based on a work cost for each worker 32 using a work for verification of a crowdsourcing-based project according to an embodiment of the present invention.

Meanwhile, the steps shown in FIG. 3 may be understood as being performed by a platform server (hereinafter, referred to as a server) operated by the service provider 20, but are not limited thereto.

In addition, a plurality of workers 32 or a plurality of inspectors 34 perform work using a predetermined terminal device. The terminal device of the operator 32 or the inspector 34 may be a computer device or a telecommunication device such as a smartphone, tablet, PDA, laptop, desktop, etc., but is not limited thereto.

First, referring to FIG. 3, the server assigns a plurality of tasks of a crowdsourcing-based project (hereinafter referred to as a project) to a plurality of workers 32 and requests the task to be performed (S110), and a plurality of workers 32 A plurality of work results are inputted from (S120).

Here, one project may contain multiple jobs, and the plurality of jobs may be grouped into a plurality of tasks according to their type, or each job may be a different type of job or the same job. , But is not limited thereto.

Next, the server assigns the plurality of work results to the plurality of inspectors 34 to request the inspection performance (S130), and receives the plurality of inspection results for the plurality of job results from the plurality of inspectors 34 (S140). ). At this time, the inspector 34 may input the inspection pass or rejection as a plurality of inspection results.

On the other hand, in an embodiment of the present invention, in allocating a plurality of tasks to a plurality of workers 32, a verification task is provided along with a practical task so that each worker 32 cannot recognize it.

That is, when the server assigns the verification task to the plurality of workers 32, the actual task is assigned together, and the worker 32 assigned the verification task cannot distinguish this from the actual task. The server receives the work results for the practical work (hereinafter referred to as "real-only work results") and the work results for the verification work (hereinafter, referred to as "verification work results") from the worker 32. And, it is possible to select a skilled worker 32 based on the time when the verification work result is performed.

For reference, in the description of the present invention, the task for verifying the skilled worker 32 is referred to as a'verification task', and to be distinguished from this, the worker 32 actually performs the task and a task that should be used as a result of the project Let's call it'practical work'.

In the present invention, the process of generating the verification task will be described in detail as follows.

The server extracts an inspection result that satisfies a predetermined condition from among a plurality of inspection results for a plurality of job results, and generates a verification job pool.

In an embodiment, the server may extract, from among a plurality of inspection results for a plurality of job results during a predetermined period, a job result that has passed the inspection that satisfies a predetermined condition and includes it in the verification job pool.

This is to verify the plurality of workers 32 who will perform the work in the second time period (t1) by using the work result of the first worker who performed the work in the first time period (t0), and similarly, the second time This is a structure in which a plurality of workers 32 who will perform the work in the third time period t2 are verified using the work result of the second worker who performed the work in the section t1.

Hereinafter, the verification operation will be described in detail with reference to FIGS. 4 and 5.

4 and 5 are views for explaining contents of generating a verification task.

The verification work is generated based on the practical work actually performed by the plurality of workers 32 of the project.

First, referring to FIG. 4, when a work result (P1) for a practical work performed by a plurality of workers 32 is input, the practical work result (P1) is assigned to the inspector 34, and the inspector 34 ) Performs an inspection on the actual work result (P1).

Then, when the inspection by the inspector 34 on the practical work result P1 is completed, the server extracts the inspection result P2 from among the plurality of inspection results. In one embodiment of the present invention, since the verification task is created based on the verification pass, that is, the first verification pass or re-review pass, the verification task is determined as the correct answer as the verification pass or reject, and whether or not such a correct answer is entered. Through this, the operator 32 is verified.

Then, the server extracts the inspection-passed job result P3 that satisfies a predetermined condition among the inspection-passed job results P2 and generates a verification job pool. In this case, as an embodiment of the predetermined condition, a predetermined ranking based on the inspection time required by the inspector 34 may be satisfied.

That is, the server sorts the actual work results P2 that have passed the inspection in the order in which the time required for the inspection is long, and extracts only the practical work results P3 corresponding to a predetermined priority among them.

Here, the long time spent for inspection means that the inspection is more likely to be faithfully performed compared to other inspectors 34, and this means that the probability of errors in the inspection results is low. It is desirable to extract the exclusive work result (P3) and include it in the work pool for verification.

In this case, the predetermined ranking may be set in advance by the server, the administrator, or the requester 10, and the predetermined ranking may be extracted not only in units of number but also in units of time.

As described above, the inspection result P3 that satisfies the predetermined condition may be immediately included in the verification job pool.

However, since there is a possibility that the inspector 34 who has inspected the actual work result may also make an error, another inspector 34 re-examines the inspected work result in order to provide a more verified work to the operator 32. After performing the re-examination, it is possible to include the results of the practical work that has passed the re-examination in the work pool for verification.

That is, if the server extracts the work result (P3) that has passed the inspection and satisfies a predetermined condition among the work results (P2) that have passed the inspection for the plurality of work results (P1), the work result (P3) is inspected. Request for re-examination from one or more inspectors 34 different from the inspector 34 (P4), and the results of the practical work (P5) that have been inspected by the inspector 34 who have re-examined are converted to the verification operation (P6). Can be generated.

Referring to FIG. 5, when a result of the practical use that satisfies a predetermined priority is extracted, the result of the practical use is subjected to an additional inspection process by different inspectors 34, respectively.

For example, in FIG. 4, when the first inspector passes the inspection, three'practical work results 1 to 3 (P3) according to a predetermined ranking. 'Is extracted, and the three practical work results (P3) are re-inspected by the second and third inspectors different from the first inspector (P4). In this case, in the example of FIG. 5, it is shown that the re-examination is performed by two inspectors, but is not limited thereto.

As a result of re-inspection by the 2nd and 3rd inspectors, if the 2nd inspector passed the inspection and only the'practical work result 1'passed by the third inspector, It is possible to create a verification job (P6) using only the'Practical work result 1 (P5)' that has been inspected by all inspectors and then include it in the verification job pool.

After the verification work pool is generated, the verification work pool may be updated by checking whether the condition is satisfied in real time or at predetermined intervals with respect to the verification result of the actual work result.

On the other hand, the task is not performed for the actual task, there is no actual task that satisfies a predetermined condition, or all the verification tasks included in the verification task pool are assigned to the worker 32 for further verification. There may be conditions in which the dragon task cannot be assigned. In this case, an embodiment of the present invention may add a verification task prepared in advance by the client 10 to the verification pool, and the time to be added is not limited to a specific period of the project.

After the verification work is generated as described above, the verification work is assigned to the plurality of workers 32 together with the actual work, and the server receives the verification work result together with the actual work result from the worker 32.

In one embodiment, the server allocates a task for verification to each worker 32 based on the'real-only task-worker' matching data. For example, when the worker A and the actual task a are matched, the server may allocate the verification task a generated by the actual task a to the plurality of workers 32 other than the worker A.

In this case, the verification work may be assigned to a plurality of workers 32 from the time when a work result for a work of a predetermined ratio or more of the total work included in the project is submitted, and request the work to be performed.

In other words, since proficient workers (32) rarely appear at the beginning of the project, pre-allocation of the verification work may be a waste of resources, so the verification work from the time when the work results corresponding to the mid-to-late part of the project are submitted. It is desirable that the allocation of

However, the timing of assignment of verification tasks is not necessarily limited thereto, and of course, they may be assigned from the beginning of the project to periodically check the proficiency level of the worker pool.

As the server receives the verification work results from the plurality of workers 32, the server uses the verification work results to at least one first worker who will pay the work cost as a unit wage per job execution among the plurality of workers 32 Wow, at least one second worker who will pay the work cost in unit wage per hour is selected (S150).

Then, the server pays the work cost to the first worker in unit wage per job execution, and pays the work cost to the second worker in unit wage per hour (S160).

6 is a flowchart of a process of selecting the first and second workers using the verification work result.

The server performs individual work execution time (hereinafter referred to as "first work execution time") and average work execution time (hereinafter referred to as "second work execution time") for each verification task of the plurality of workers 32 Is calculated (S151, S152).

Here, the individual task execution time refers to the task execution time for each worker calculated based on the start time of performing the verification task assigned to each worker 32 and the time when task performance is completed and the task result is input, The average work execution time is an average of work execution time for each worker 32 based on each verification work.

In this case, the second task execution time for the verification task of the worker 32 whose number of tasks is less than n (where n is a natural number of 1 or more) among the plurality of workers 32 may be excluded.

For example, a worker 32 with fewer than n tasks performed may be a new worker who is unfamiliar with performing a task, and if too many new workers' data is considered together, the average task execution time of the workers 32 is too much. Since it becomes longer, the time to perform the work for a specific worker can be evaluated shorter than the current proficiency level, so the data of new workers can be excluded.

Next, the server compares the first task execution time and the second task execution time, selects a second worker among the plurality of workers 32 based on the comparison result (S153), and selects the second worker among the plurality of workers 32 2 The remaining workers excluding the workers are selected as the first workers (S154).

7 is a diagram for explaining a process of selecting a second worker.

In order to select the second worker, the server first sorts each verification task provided to any individual worker among the plurality of workers 32 in chronological order.

The plurality of verification tasks are not assigned to the workers 32 in the order of their assigned IDs, but are extracted in a random order from the verification task pool and assigned to the plurality of workers 32. Therefore, in order to select the second workers who have become familiar with the project over time, it is necessary to first arrange the verification tasks assigned to each of the workers 32 in chronological order.

Next, deviations of the first and second task execution times for each verification task are calculated, and at least one individual worker whose trend for each calculated deviation falls within a predetermined range is selected as a second worker.

Referring to FIG. 7, the first task execution time of worker A for the verification tasks of ID 1001 sorted in chronological order is 25 seconds, and the second task execution time, which is the average of all workers 32, was calculated as 18 seconds. , The deviation was calculated as 7 seconds. This means that worker A performed the work later than the average of all workers 32.

As time passes, the deviation of the first and second work damage times with respect to the average of worker A and all workers 32 gradually decreases to 7 seconds, 6 seconds, -2 seconds, -2 seconds, and -1 seconds. At some point, it can be seen that worker A is performing work faster than the average of all workers 32, which means that worker A is more familiar with the project overall than all workers 32. do.

Therefore, the server checks whether the deviation of the first and second work execution times between the worker A and the entire workers 32 falls within a predetermined range, and the second worker who will pay the work cost to the worker A at the hourly unit wage. After selecting as, work costs are paid.

In this case, the server may select at least one individual worker whose trend for each calculated deviation falls within a predetermined value and time range as the second worker.

That is, it is checked whether the trend for each deviation calculated as shown in FIG. 7 is included in a predetermined numerical range, for example, a trend range of a negative value, but the time range corresponding to the mid-late half of the project rather than the beginning of the project Or, by checking whether the trend for the deviation is included in a time range such as whether the deviation of the negative value is continuously maintained for a predetermined period of time, an individual worker who satisfies this may be selected as a second worker.

On the contrary, if the trend for each deviation calculated by individual workers is out of a predetermined numerical range, that is, if it has a positive value rather than a negative value, it is regarded as a worker who is less familiar with the project than the average of all workers 32. I can. In addition, even if a certain numerical range is satisfied, if it is not included in the certain period range, for example, the deviation of the negative value is maintained only at the beginning of the project, or the deviation of the negative value appears irregularly even in the middle of the project. If the period range is not satisfied, it is determined as a worker who is less familiar with the project and selected as the first worker.

On the other hand, in an embodiment of the present invention, after evaluating the degree of difficulty for a verification task, a worker who performs a verification task having a high degree of difficulty better than other workers may be selected as the second worker.

Such a method of selecting a second worker may be additionally implemented when the number of second workers is not selected as much as a preset minimum number of second workers by a method based on a comparison result of the first and second work execution times. , Is not necessarily limited thereto.

8 is a view for explaining the content of selecting a second worker based on the difficulty of the verification work.

Specifically, the server sorts each verification task provided to any individual worker among the plurality of workers 34 in chronological order, and then calculates the correct answer rate of the plurality of workers 34 for each of the sorted plurality of verification tasks.

For example, the correct answer rates of the plurality of workers 34 for each of the ID 1001 to ID 1005 verification tasks were calculated as 75%, 70%, 50%, 30%, and 20%.

Here, the correct answer rate for the plurality of workers 32 is calculated based on the number of correct answers compared to the number of exposures of the verification work.

Thereafter, the server calculates an average correct answer rate for a plurality of verification tasks based on the correct answer rate. If the verification work for each ID is equally exposed to the workers 32, the average correct answer rate for the verification work of IDs 1001 to 1005 for 1000 workers is (75%+70%+50%+30%) +20%)/5.

Then, the server sets the difficulty level for the verification task based on the average correct answer rate.

In the above example, the average correct answer rate may be calculated as 49%, and the level of difficulty for each grade divided into a plurality of sections such as high and medium difficulty can be set by classifying a predetermined section based on 49%. For example, the correct answer rate of 0 to 29% may be set to higher difficulty, 30% to 69% of difficulty, and 70% to 100% may be set to lower difficulty.

Then, the server selects, as a second worker, a worker who satisfies at least one of a predetermined number of times and a time condition in which the correct answer input for the verification task having a certain difficulty level or more among the plurality of workers 32 is satisfied.

Referring to the example of FIG. 8, among a plurality of workers 32, a worker whose number of correct answers to a verification task having a difficulty level is input more than two times is selected as a second worker, or together or separately, having a difficulty level. When the correct answer input to the verification task is inputted two or more times in a row or after a predetermined period of time, an operator may be selected as a second operator.

Through this, if the second worker is not sufficiently selected only by using the work execution time, or if the second worker is more strictly selected through additional criteria along with the method of using the work execution time, the difficulty of the verification work. The operator 32, whose accuracy and proficiency is guaranteed, may be selected as a second worker through a selection method in consideration of.

In addition, in an embodiment of the present invention, the time to allocate the task for verification may be assigned from the time when the task result for tasks of a predetermined ratio or more is submitted, and the first and second workers are selected together or separately. The timing of doing this may also proceed from the time when the work result for the work of a predetermined ratio or more is submitted.

In other words, selecting the first and second workers is for the workers 32 who are familiar with performing the work in the project, and the time when the workers 32 familiar with the work of the project appear is likely to be in the mid-late and late stages of the project, It is preferable that the process of selecting the first and second workers proceeds after a predetermined time.

Meanwhile, in the above description, steps S11 to S160 may be further divided into additional steps or may be combined into fewer steps according to an embodiment of the present invention. In addition, some steps may be omitted as necessary, and the order between steps may be changed. In addition, even if other contents are omitted, the contents of FIG. 9 to be described later can also be applied to a method of paying work cost based on an hourly wage system for each worker using the work for verification of the crowdsourcing-based project of FIGS. 1 to 8.

Hereinafter, a device for paying work cost based on an hourly pay system for each worker using a work for verifying a crowdsourcing-based project according to an embodiment of the present invention (200, hereinafter referred to as a device for paying work cost based on an hourly pay system for each worker) will be described. .

9 is a view for explaining an hourly wage system-based work cost payment device 200 for each worker according to an embodiment of the present invention.

Referring to FIG. 9, the apparatus 200 for paying work cost based on an hourly wage for each worker includes a communication module 210, a memory 220, and a processor 230.

The communication module 210 transmits a crowdsourcing-based work for one project to a plurality of workers 32 to request a task to be performed, and receives a work result from the plurality of workers 32. Further, the work results received from the plurality of workers 32 are transmitted to the plurality of inspectors 34 to request inspection, and the inspection results are received from the plurality of inspectors 34. In addition, the communication module 210 allocates a verification task to a plurality of workers 32 to request the task to be performed, and receives a task result for the verification task from the plurality of workers 32.

The memory 220 stores a program for selecting the worker 32 who will pay the work cost in unit wage per job execution and unit wage per hour among the plurality of workers 32 based on the job execution time for the verification job. .

As the processor 230 executes the program stored in the memory 220, a first task execution time, which is an individual task execution time for each verification task of the plurality of workers 32, and a second task execution time, which is an average task execution time. Calculates the work execution time, selects a second worker who will pay the work cost by hourly unit wage among the plurality of workers 32 based on the comparison result of the first and second work execution times, and the rest except for the second worker The worker 32 is selected as the first worker who will pay the work cost in unit wages per job execution, and pays the work cost according to each payment method.

In the above-described method for paying an hourly wage system based on a worker using the work for verification of a crowdsourcing-based project according to an embodiment of the present invention described above, the method is implemented as a program (or application) to be executed in combination with a computer, which is hardware. Can be stored on media.

The above-described program includes C, C++, JAVA, Ruby, which can be read by a processor (CPU) of the computer through the device interface of the computer, in order for the computer to read the program and execute the methods implemented as a program. It may include a code (Code) coded in a computer language such as machine language. Such code may include a functional code related to a function defining necessary functions for executing the methods, and a control code related to an execution procedure necessary for the processor of the computer to execute the functions according to a predetermined procedure. can do. In addition, such code may further include additional information required for the processor of the computer to execute the functions or code related to a memory reference to which location (address address) of the internal or external memory of the computer should be referenced. have. In addition, when the processor of the computer needs to communicate with any other computer or server in the remote in order to execute the functions, the code uses the communication module of the computer to determine how It may further include a communication-related code for whether to communicate, what kind of information or media should be transmitted and received during communication.

The stored medium is not a medium that stores data for a short moment, such as a register, cache, memory, etc., but a medium that stores data semi-permanently and can be read by a device. Specifically, examples of the storage medium include, but are not limited to, ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, and the like. That is, the program may be stored in various recording media on various servers to which the computer can access, or on various recording media on the user's computer. In addition, the medium may be distributed over a computer system connected through a network, and computer-readable codes may be stored in a distributed manner.

The above description of the present invention is for illustrative purposes only, and those of ordinary skill in the art to which the present invention pertains will be able to understand that other specific forms can be easily modified without changing the technical spirit or essential features of the present invention will be. Therefore, it should be understood that the embodiments described above are illustrative in all respects and not limiting. For example, each component described as a single type may be implemented in a distributed manner, and similarly, components described as being distributed may also be implemented in a combined form.

The scope of the present invention is indicated by the claims to be described later rather than the detailed description, and all changes or modified forms derived from the meaning and scope of the claims and their equivalent concepts should be interpreted as being included in the scope of the present invention. do.

10: Client
20: service provider
30: public
32: worker
34: inspector
200: Method of paying work costs based on the hourly wage system for each worker
210: communication module
220: memory
230: processor

Claims (13)

As a method performed by a computer,
Allocating a plurality of tasks of a crowdsourcing-based project (hereinafter, "project") to a plurality of workers and requesting the task to be performed;
Receiving a plurality of job results from the plurality of workers;
Assigning the plurality of work results to a plurality of inspectors and requesting to perform inspection;
Receiving an inspection pass or rejection as a plurality of inspection results for the plurality of work results from the plurality of inspectors;
Using the work result, among the plurality of workers, (i) at least one first worker who will pay the work cost in unit wage per job execution and (ii) at least one second worker who will pay the work cost in unit wage per hour. Selecting a worker; And
Including the step of paying a work cost to the first worker based on a unit wage per job performance, and paying the work cost to the second worker based on a unit wage per hour,
The plurality of tasks for requesting to perform the task includes a plurality of verification tasks generated from a result of an inspection passed that satisfies a predetermined condition among a plurality of inspection results for the plurality of operation results,
Using the work result, the step of selecting a first worker and a second worker,
Calculating an individual task execution time (hereinafter, "first task execution time") for each verification task of the plurality of workers;
Calculating an average task execution time (hereinafter, "second task execution time") for each verification task of the plurality of workers;
Selecting the second worker among the plurality of workers based on a comparison result of the first and second task execution times; And
Including the step of selecting the remaining workers other than the second worker among the plurality of workers as the first worker,
The step of selecting the second worker among the plurality of workers based on the comparison result of the first and second task execution times,
Arranging each verification task provided to any individual worker among the plurality of workers in chronological order;
Calculating a deviation of the execution time of the first and second tasks for each of the verification tasks; And
Including the step of selecting at least one individual worker whose trend for each of the calculated deviations falls within a predetermined range as a second worker,
A method of paying work costs based on the hourly wage system for each worker using the work for verification of crowdsourcing-based projects. The method of claim 1,
The step of selecting at least one individual worker in which the calculated trend for each deviation falls within a predetermined range as a second worker,
To select at least one individual worker whose trend for each deviation is calculated in a predetermined value and time range as the second worker,
A method of paying work costs based on the hourly wage system for each worker using the work for verification of crowdsourcing-based projects. The method of claim 1,
The predetermined conditions are:
It is a condition that satisfies a predetermined ranking based on the inspection time required by the inspector,
A method of paying work costs based on the hourly wage system for each worker using the work for verification of crowdsourcing-based projects. The method of claim 3,
The plurality of verification operations,
It is generated by extracting some of the work results that have passed re-review by at least one other inspector different from the inspector who has inspected the inspection-passed work result corresponding to the verification work,
A method of paying work costs based on the hourly wage system for each worker using the work for verification of crowdsourcing-based projects. The method of claim 1,
Using the work result, the step of selecting a first worker and a second worker,
To select the first and second workers from a point in time after a job result for a job of a predetermined ratio or more among all jobs included in the project is submitted,
A method of paying work costs based on the hourly wage system for each worker using the work for verification of crowdsourcing-based projects. The method of claim 1,
The step of calculating the second task execution time for each verification task of the plurality of workers,
To exclude the second task execution time for the verification task of the worker whose number of tasks is less than n (where n is a natural number of 1 or more) among the plurality of workers,
A method of paying work costs based on the hourly wage system for each worker using the work for verification of crowdsourcing-based projects. As a method performed by a computer,
Allocating a plurality of tasks of a crowdsourcing-based project (hereinafter, "project") to a plurality of workers and requesting the task to be performed;
Receiving a plurality of job results from the plurality of workers;
Assigning the plurality of work results to a plurality of inspectors and requesting to perform inspection;
Receiving an inspection pass or rejection as a plurality of inspection results for the plurality of work results from the plurality of inspectors;
Using the work result, among the plurality of workers, (i) at least one first worker who will pay the work cost in unit wage per job execution and (ii) at least one second worker who will pay the work cost in unit wage per hour. Selecting a worker; And
Including the step of paying a work cost to the first worker based on a unit wage per job performance, and paying the work cost to the second worker based on a unit wage per hour,
The plurality of tasks for requesting to perform the task includes a plurality of verification tasks generated from a result of an inspection passed that satisfies a predetermined condition among a plurality of inspection results for the plurality of operation results,
Using the work result, the step of selecting a first worker and a second worker,
Calculating an individual task execution time (hereinafter, "first task execution time") for each verification task of the plurality of workers;
Calculating an average task execution time (hereinafter, "second task execution time") for each verification task of the plurality of workers;
Selecting the second worker among the plurality of workers based on a comparison result of the first and second task execution times; And
Including the step of selecting the remaining workers other than the second worker among the plurality of workers as the first worker,
Using the work result, the step of selecting a first worker and a second worker,
Arranging each verification task provided to any individual worker among the plurality of workers in chronological order;
Calculating an average correct answer rate based on the correct answer rate and the correct answer rate of the plurality of workers for each of the plurality of verification tasks;
Setting a difficulty level for each of the verification tasks based on the average correct answer rate; And
The step of selecting a worker who satisfies at least one of a predetermined number of times and a time condition for inputting a correct answer to the verification task having a difficulty of a certain degree or more among the plurality of workers as the second worker,
A method of paying work costs based on the hourly wage system for each worker using the work for verification of crowdsourcing-based projects. The method of claim 7,
The step of selecting a worker who satisfies at least one of a predetermined number of times and a time condition for inputting a correct answer to the verification task having a difficulty of a certain level or higher among the plurality of workers as the second worker,
According to the comparison result of the first and second task execution times, it is performed when the number of preset minimum second workers is not selected,
A method of paying work costs based on the hourly wage system for each worker using the work for verification of crowdsourcing-based projects. The method of claim 7,
The predetermined conditions are:
It is a condition that satisfies a predetermined ranking based on the inspection time required by the inspector,
A method of paying work costs based on the hourly wage system for each worker using the work for verification of crowdsourcing-based projects. The method of claim 9,
The plurality of verification operations,
It is generated by extracting some of the work results that have passed re-review by at least one other inspector different from the inspector who has inspected the inspection-passed work result corresponding to the verification work,
A method of paying work costs based on the hourly wage system for each worker using the work for verification of crowdsourcing-based projects. The method of claim 7,
Using the work result, the step of selecting a first worker and a second worker,
To select the first and second workers from a point in time after a job result for a job of a predetermined ratio or more among all jobs included in the project is submitted,
A method of paying work costs based on the hourly wage system for each worker using the work for verification of crowdsourcing-based projects. The method of claim 7,
The step of calculating the second task execution time for each verification task of the plurality of workers,
To exclude the second task execution time for the verification task of the worker whose number of tasks is less than n (where n is a natural number of 1 or more) among the plurality of workers,
A method of paying work costs based on the hourly wage system for each worker using the work for verification of crowdsourcing-based projects. A computer program combined with a computer and stored in a computer-readable recording medium to execute a method for paying work cost based on an hourly wage for each worker using the work for verifying a crowdsourcing-based project according to any one of claims 1 to 12.

Images