KR102164844B1 - Method for evaluating workers using the unit of work difficulty of crowdsourcing based projects for artificial intelligence training data generation - Google Patents

Abstract

Provided is a method for evaluating a worker using a difficulty level of a unit work of a crowdsourcing-based project for generating artificial intelligence learning data to correctly estimate the worker. According to the present invention, the method comprises the following steps of: assigning a plurality of works of a crowdsourcing-based project (hereinafter, referred to as a project) to a plurality of workers to request execution of the works; receiving a plurality of work results or a work pass request from the plurality of workers; assigning the plurality of work results to a plurality of inspectors to request execution of inspection; receiving inspection pass or reject, as a plurality of inspection results of the plurality of work results, from the plurality of inspectors; and evaluating and scoring the plurality of workers when the project is finished. The step of evaluating and scoring the plurality of works comprises the following steps of: assigning a difficulty level of a work unit based on the number of inputs of the work pass request with respect to each work of the project; assigning a work score based on the difficulty level of the work unit with respect to each work; and summing the work scores of one or more works, which are worked by the worker and complete the inspection, to score each worker.

Description

{METHOD FOR EVALUATING WORKERS USING THE UNIT OF WORK DIFFICULTY OF CROWDSOURCING BASED PROJECTS FOR ARTIFICIAL INTELLIGENCE TRAINING DATA GENERATION}

The present invention relates to a method for evaluating workers using the difficulty of a work unit of a crowdsourcing-based project for generating artificial intelligence learning data.

Recently, more and more companies collect and process large amounts of data on a crowdsourcing basis that engages the general public in some process of corporate activities. In other words, by opening a project and allowing the general public, that is, workers to participate in the project, necessary information is collected through the work results completed by the workers.

At this time, in order to collect more reliable information, the company assigns the work result completed by the worker to the inspector to perform the inspection work.

Specifically, when one project is opened, a plurality of tasks are assigned to each of the plurality of workers. Each worker performs a plurality of tasks assigned to it, and provides the task results. Thereafter, a plurality of inspection tasks are assigned to each of the plurality of inspectors, and each inspector performs a plurality of inspection tasks assigned to them.

On the other hand, even if the tasks are included in the same project, the rejection rate of each task varies greatly depending on the difficulty level. Therefore, even if the same number of tasks are inspected and completed, the operator who performed 100 easy tasks and the worker who did 100 difficult tasks must be evaluated differently.

However, in the conventional case, if the number of completed inspections and the average rejection rate are the same, there is a problem that they are evaluated as workers of the same level, and a method for more accurately evaluating workers based on the difficulty of each task is required.

Unexamined Patent Publication No. 10-2014-0095956, 2014.08.04.

The task to be solved by the present invention is to assign a work unit difficulty based on the number of work pass requests for each work, and assign a work score according to the work unit difficulty, and then evaluate the workers through scoring for each worker. It is to provide a method of evaluating workers by utilizing the difficulty of work units of crowdsourcing-based projects.

However, the problem to be solved by the present invention is not limited to the problem as described above, and other problems may exist.

In order to solve the above-described problems, a method for evaluating a worker using the difficulty of a work unit of a crowdsourcing-based project according to an aspect of the present invention is performed by assigning a plurality of tasks of a crowdsourcing-based project (hereinafter, a project) to a plurality of workers. Requesting execution, receiving a plurality of work results or work pass requests from the plurality of workers, assigning the plurality of work results to a plurality of inspectors and requesting to perform inspection, the plurality of inspectors And receiving a test pass or rejection as a plurality of test results for the plurality of work results from, and when the project is finished, evaluating and scoring the plurality of workers. In this case, the step of evaluating and scoring the plurality of workers includes assigning a work unit difficulty level to each task of the project based on the number of times the work pass request is input, and the work unit for each task And assigning a task score based on the difficulty level, and scoring each worker by summing the task scores of one or more tasks performed by each worker and passed the inspection.

In some embodiments of the present invention, the step of assigning a plurality of tasks of the project to a plurality of workers and requesting the task to be performed comprises assigning the same task to a plurality of different workers to request the task to be performed, and each of the projects In the step of assigning a work unit difficulty level to the work of the project based on the number of times the work pass request is input, the work unit difficulty level is determined based on the number of times the work pass request is input and the rejection rate for each work of the project. Can be given.

In some embodiments of the present invention, the step of assigning a task score for each task based on the task unit difficulty level comprises dividing the task unit difficulty level into a predetermined number of sections, and a plurality of tasks belonging to the same section. It may include assigning the same task score to the unit difficulty level, and assigning the task score to each task based on the task difficulty level and the section.

Some embodiments of the present invention may further include determining a pool of workers of other projects whose similarity to the project is greater than or equal to a reference similarity, based on the scoring results of the plurality of workers of the project.

In some embodiments of the present invention, the step of assigning a plurality of tasks of the project to a plurality of workers and requesting the task to be performed, the task for which the work pass request is input, withdraws the assignment, and reassigns the task to another worker to perform the task. Can be requested.

In some embodiments of the present invention, the step of requesting task execution by assigning a plurality of tasks of the project to a plurality of workers, the task in which the number of times the work pass request is input is set to a high difficulty task, and , Assignment to workers and requests to perform tasks may be suspended for a predetermined period.

In some embodiments of the present invention, the step of requesting task performance by assigning a plurality of tasks of the project to a plurality of workers includes, when the remaining tasks other than the high difficulty task among the plurality of tasks of the project are completed, the The high-difficulty task may be reassigned to a worker of an excellent grade among the plurality of workers to request the task to be performed.

In some embodiments of the present invention, the worker of the excellent grade may be selected based on the rejection rate of each worker among the plurality of workers.

In some embodiments of the present invention, the worker of the excellent grade is that the number of times the work pass request is input among the plurality of workers is less than the reference number, and the reference number -n times (where n is a natural number of 1 or more) or more. It may be selected based on the rejection rate of each worker among one or more workers who have passed the inspection.

A computer program according to another aspect of the present invention for solving the above-described problem is combined with a computer as hardware to execute a worker evaluation method utilizing the work unit difficulty of the crowdsourcing-based project, and stored in a computer-readable recording medium. do.

Other specific details of the present invention are included in the detailed description and drawings.

According to the present invention described above, unlike the prior art that simply evaluates the ability of a worker by considering only the number of completed inspections and the average rejection rate, a task unit difficulty level is assigned to each task, and a more difficult task through an evaluation based on the difficulty level. It is possible to make a higher evaluation to the operator who performed the operation.

In particular, the difficulty level of the work unit is automatically calculated based on the number of times the work pass is requested in the work process of the actual workers, rather than the difficulty level set unilaterally by the client or a third party. There is an advantage that a more accurate operator's evaluation can be made.

In addition, a task that has been requested more than a certain number of work passes is set as a high-difficulty task, the assignment is temporarily suspended, and then, after all tasks of relatively low-difficulty tasks are completed, they are reassigned to high-level workers to perform. It has the advantage of solving the problem of delayed project schedule.

The effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the following description.

1 is a conceptual diagram of a crowdsourcing service according to an embodiment of the present invention.
2 is a flowchart illustrating a crowdsourcing-based project progress process according to an embodiment of the present invention.
3 is a flowchart of a method for evaluating a worker using a work unit difficulty of a crowdsourcing-based project according to an embodiment of the present invention.
4 is a flowchart illustrating a process of evaluating and scoring a plurality of workers.
5 is a diagram illustrating an example of evaluating and scoring a plurality of workers.
6 is a view for explaining an example of assigning a work unit difficulty level in consideration of a rejection rate of each work assigned to each worker and a work pass request.
7 is a diagram illustrating an example of inputting a work result and a work path request for each worker.
8 is a view for explaining a worker evaluation apparatus according to an embodiment of the present invention.

Advantages and features of the present invention, and a method of achieving them will become apparent with reference to the embodiments described below in detail together with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but may be implemented in a variety of different forms, only the present embodiments are intended to complete the disclosure of the present invention, It is provided to fully inform the technician of the scope of the present invention, and the present invention is only defined by the scope of the claims.

The terms used in the present specification are for describing exemplary embodiments and are not intended to limit the present invention. In this specification, the singular form also includes the plural form unless specifically stated in the phrase. As used in the specification, “comprises” and/or “comprising” do not exclude the presence or addition of one or more other elements other than the mentioned elements. Throughout the specification, the same reference numerals refer to the same elements, and “and/or” includes each and all combinations of one or more of the mentioned elements. Although "first", "second", and the like are used to describe various elements, it goes without saying that these elements are not limited by these terms. These terms are only used to distinguish one component from another component. Therefore, it goes without saying that the first component mentioned below may be the second component within the technical idea of the present invention.

Unless otherwise defined, all terms (including technical and scientific terms) used in the present specification may be used as meanings that can be commonly understood by those of ordinary skill in the art to which the present invention belongs. In addition, terms defined in a commonly used dictionary are not interpreted ideally or excessively unless explicitly defined specifically.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a conceptual diagram of a crowdsourcing service according to an embodiment of the present invention.

Referring to FIG. 1, a crowdsourcing service is performed by being composed of a client 10, a service provider 20, and a public 30.

The sponsor 10 refers to a company or individual who requests a crowdsourcing-based project (hereinafter, referred to as a project).

The requester 10 requests a project for the purpose of collecting source data or annotating data for generating artificial intelligence learning data. The data generated through the project can be used as learning data for arbitrary machine learning such as supervised learning, unsupervised learning, and reinforcement learning. Collection of source data refers to collecting raw data such as recorded voice collection and photo collection. Data annotation refers to inputting relevant annotation data into source data such as text, photos, and videos. For example, the data annotation may include finding an entity in a given fingerprint or finding a similar sentence, but is not limited thereto. Meanwhile, the types of the above-described projects are only one embodiment, and various projects may be handled in the present invention according to the design of the client.

The service provider 20 refers to a company that provides crowdsourcing services.

When the service provider 20 receives a request for a project for a product or service from the client 10, the service provider 20 allocates a task for the project to the general public 30 and receives the work result from the public 30. Thereafter, the final product extracted based on the work result is provided to the client 10.

At this time, the service provider 20 provides a crowdsourcing service to the client 10 and the public 30 through a crowdsourcing platform (hereinafter, the platform). That is, when the service provider 20 receives a request for a project from the client 10, the service provider 20 opens the project on the platform. Thereafter, when the work result for the open project is received from the public 30, the project is terminated on the platform, and the final product may be extracted and provided to the client 10.

The public 30 refers to the general public participating in the project open on the platform. Here, the public 30 may participate in a project opened on the platform through an application or website provided by the service provider 20.

The public 30 is composed of a worker 32 and an inspector 34.

The worker 32 decides to participate in a specific project among a plurality of projects open on the platform. Thereafter, the operator 32 performs an operation such as collection of source data or data annotation, and transmits it to the platform.

The inspector 34 decides to participate in a specific project among a plurality of projects open on the platform. Thereafter, the inspector 34 inspects the result of the work performed by the operator 32. As a result of performing the inspection, the inspector 34 may perform inspection pass processing or rejection processing, and input a rejection reason during rejection processing. In the case of passing the inspection, it is not necessary to rework and re-examination due to this, so passing inspection has the same meaning as completion of inspection.

2 is a flowchart illustrating a crowdsourcing-based project progress process according to an embodiment of the present invention.

First, the requester 10 requests one or more projects to the service provider 20 (S11).

Thereafter, the service provider 20 opens the requested project on the platform (S12). At this time, the service provider 20 may determine the grade in consideration of the difficulty of the project before opening the project. That is, depending on the difficulty level, it is possible to determine which level or higher public 30 to expose the project. Accordingly, it is possible to increase the reliability of the work result of the project.

Thereafter, the service provider 20 requests the work by assigning the work to the workers 32 of the corresponding level or higher according to the level of the project (S13).

Thereafter, the worker 32 performs the assigned task (S14). In this case, the worker 32 may input the reason for the inability to work without performing the work for a work in which the work itself is impossible for some reason.

Thereafter, the service provider 20 receives the work result from the worker 32 (S15), and assigns the work result to the inspector 34 to request the inspection (S16).

Similarly, according to an embodiment of the present invention, only a suitable project may be exposed to the inspector 34 out of the total projects being performed according to the level set according to the difficulty of the project or the qualification requirements of the inspector 34.

Thereafter, the inspector 34 performs the assigned inspection (S17). At this time, if the inspector 34 determines that the work has been properly performed, the inspector 34 decides to complete the inspection, and if it determines that the inspection operation is wrong, the inspector 34 rejects it. During rejection processing, the inspector 34 inputs the rejection reason for what reason the job was judged to be wrong.

Thereafter, the service provider 20 receives the inspection result from the inspector 34 (S18).

When the inspection result is the inspection completion, the service provider 20 uses the work result as valid data, and based on this, extracts the final product at the end of the project.

When the inspection result is rejection processing, the service provider 20 may internally perform the inspection again, or assign the work to the worker 32 again to perform the rework. When reworking, re-examination by an inspector is required.

Thereafter, when the project period ends or sufficient valid data is secured, the service provider 20 terminates the project (S19), calculates a final result based on the secured valid data, and provides it to the client 10 ( S20).

At this time, before the end of the project, the service provider 20 evaluates the performance results of the worker 32 and the inspector 34, calculates the work cost and the inspection cost according to the evaluation, to the worker 32 and the inspector 34. give.

In FIGS. 1 and 2, it is simply expressed as a requester 10, a service provider 20, an operator 32, and an inspector 34, but these are smartphones, tablets, PDAs, laptops, desktops, operated by each participant. It means a computer device such as a server or a telecommunication device.

Hereinafter, a method for evaluating a worker using a work unit difficulty of a crowdsourcing-based project according to an embodiment of the present invention will be described with reference to FIGS. 3 to 7.

3 is a flowchart of a method for evaluating a worker 32 using a work unit difficulty of a crowdsourcing-based project according to an embodiment of the present invention. 4 is a flowchart illustrating a process of evaluating and scoring a plurality of workers 32. 5 is a diagram illustrating an example of evaluating and scoring a plurality of workers 32. 6 is a view for explaining an example of assigning a work unit difficulty level in consideration of a rejection rate of each work assigned to each worker 32 and a work pass request. 7 is a diagram for explaining an example of inputting a work result and a work path request for each worker 32.

Meanwhile, the steps shown in FIG. 3 may be understood as being performed by a platform server (hereinafter, referred to as a server) operated by the service provider 20, but are not limited thereto.

In addition, a plurality of workers 32 or a plurality of inspectors 34 perform work using a predetermined terminal device. The terminal device of the operator 32 or the inspector 34 may be a computer device or a telecommunication device such as a smartphone, tablet, PDA, laptop, desktop, etc., but is not limited thereto.

Referring to FIG. 3, first, the server assigns a plurality of tasks of a crowdsourcing-based project (hereinafter referred to as a project) to a plurality of workers 32 and requests the task to be performed (S110), and a plurality of workers 32 A plurality of work results or work pass requests are received from (S120).

Here, one project may contain multiple jobs, and the plurality of jobs may be grouped into a plurality of tasks according to their type, or each job may be a different type of job or the same job. , But is not limited thereto.

An embodiment of the present invention not only receives the work result from the worker 32, but also requests a work pass to pass a task that is difficult for the worker 32 to perform the task because the work is impossible or the difficulty of the worker 32 is high. Can be input.

In the conventional case, when the worker 32 is assigned one task, it has to continue to perform the assigned task until the inspection passes. Therefore, when the operator 32 is difficult to perform the task due to the high degree of difficulty, the worker 32 has to spend a lot of time and perform the task without being assigned another task, which in turn reduces the fatigue of the worker 32 While increasing, it acted as a problem of losing the opportunity to be paid for work by performing other work at the same time, and also acted as a problem of delaying the schedule of the entire project.

To this end, according to an embodiment of the present invention, even if the worker 32 is assigned a task having a high degree of difficulty, it is possible to prevent the task assigned through the task pass request from being performed.

That is, according to an embodiment of the present invention, when the worker 32 is assigned a task, the task is completed and the task result is input, or if the task is difficult to perform due to a high degree of difficulty, the task pass request may be input.

Next, the server allocates a plurality of work results to the inspector 34 to request the inspection performance (S130), and inputs the inspection pass or rejection as a plurality of inspection results for the plurality of work results from the plurality of inspectors 32 Receive (S140).

Thereafter, when the project is finished, the server evaluates and scores the plurality of workers 32.

4 and 5, in order to evaluate and score a plurality of workers 32, the server first assigns a work unit difficulty level based on the number of times a work pass request is input for each work of a project (S151). ).

As a simple example, for a job with a job ID of '1001', a job pass request is entered once, and a job with a job ID of '1002' if a job pass request is entered twice, and the server works for job ID '1001'. A unit difficulty level of '1' may be assigned, and a higher work unit difficulty level '2' than ID '1001' may be assigned to the ID '1002' task.

Then, the server assigns a task score to each task based on the task unit difficulty (S152).

In the example above, the server gives a job score of 1 for the ID '1001' task with a work unit difficulty of '1', and the work score for the ID '1002' task with a work unit difficulty of '2'. 2 points can be awarded.

Thereafter, the server scores each worker 32 by summing up the work scores of one or more tasks that each worker 32 performs the task and has passed the inspection (S153).

In the example of FIG. 5, as a result of summing the work scores of each worker 32, worker A is scored as 46 points and worker B is scored as 53 points.

Meanwhile, the above-described example is only the simplest example, and in the embodiment of the present invention, a work unit difficulty level may be assigned according to the number of work pass requests in various ways, and a work score may be assigned based on this.

As an embodiment of assigning a work score, the server extracts the minimum number of work pass requests and the maximum number of work pass requests for one project, and classifies and ranks between them for each section to give the work score.

In another embodiment, the server may assign a work unit difficulty level by considering the rejection rate together with the number of times the work pass request is input, and assign a work score based on this.

Specifically, the server may allocate a plurality of tasks to a plurality of workers 32 and, when requesting to perform a task, may request to perform the task by assigning the same task to a plurality of different workers 32.

In addition, the server may assign a work unit difficulty level to each task of the project based on the number of times the work pass request is input and the rejection rate, and assign a work score through this.

At this time, the rejection rate may be calculated by counting only the first rejection case with respect to the work result of the worker 32. For example, if a specific task is assigned to five workers, and the results of three workers among them are initially rejected by the inspector, the rejection rate of the task may be calculated as 60%.

Accordingly, in the example shown in FIG. 5, in the case of the ID '1001' task, the work pass request is one time, in the case of the ID '1002' task, the work pass request is two times, but the rejection rate is 60% and 30%, respectively. In each case, the difficulty level of each work unit can be equally assigned to 0.6.

In this case, the work unit difficulty may be calculated based on a result of multiplying the number of work pass requests and the rejection rate, and the simplest example may be a result value expressed as a product of these.

On the other hand, the task assignment of the multiple assignment method, in which the same task is assigned to a plurality of different workers 32, requires subjective evaluation to determine the emotional state, or the data of collecting conversation data sets in a given situation. It is used when it is necessary to secure diversity.

Tasks requiring such a multi-allocation method are not all assigned the same task to all workers 32 participating in the project, and are duplicated and assigned to a plurality of workers 32 among all workers 32 for one task. will be. Therefore, when any one of the workers 32 inputs a request for a work pass, the work may be withdrawn and reassigned to another worker 32 who has not been assigned the work.

As another example, the present invention may give a work unit difficulty level by considering the rejection rate for each work assigned to each worker and the number of work pass requests, not the rejection rate of work units having the same ID.

Referring to FIG. 6, when worker A and worker B perform a task with respect to a task ID '1001' in which one work pass request is input, the same work unit difficulty is given when only the work pass request is considered, but the rejection rate is Considering that, even for the same task, the task unit difficulty may be given differently.

After the work unit difficulty level is assigned in this way, the server assigns work points to each work. At this time, the server divides the work unit difficulty level into a predetermined number of sections, and for a plurality of work unit difficulties belonging to the same section. The same task points can be assigned.

In addition, for each task, a task score can be assigned based on task difficulty and section.

For example, the server divides the work unit difficulty into a predetermined number of sections, such as dividing it into a first section when the work unit difficulty level is 1 or 2, and dividing it into a second section when the work unit difficulty level is 3 or 4 , For the same section, the same work points can be assigned.

This is because the deviation of the work unit difficulty is also large due to the large deviation in the number of work pass requests, and if a work score is assigned for each work unit difficulty, the work score deviation between a plurality of tasks may be very large. For example, 10 When the difficulty of each work unit is calculated, it can be divided into five sections and a task score can be assigned according to the section.

Meanwhile, the server may determine a pool of workers of other projects whose similarity to the project is greater than or equal to the reference similarity based on the scoring results of the plurality of workers 32 of the project.

That is, in an embodiment of the present invention, in determining the worker pool of another similar project, the scoring result of the worker 32 in the current project may be utilized. For example, if 100 workers are required and the project sponsor 20 wants to determine the worker pool by using the scoring results of workers of similar projects, 100 people will be selected according to the scoring ranking of workers of similar projects. I can.

Here, the similar project may be specified as a similar project by the client, or if it has a degree of agreement greater than or equal to a threshold through image analysis and summary analysis between projects, it may be specified as a similar project.

On the other hand, when the worker 32 makes a work pass request, the server may request to perform the work by withdrawing the assignment for the work for which the work path request has been input, and reassigning the work to another worker 32. In this case, the server may prevent the same task from being assigned to the worker 32 who has entered the work pass request again. In this case, it goes without saying that the other worker 32 can also input a work pass request when it is difficult to perform the corresponding work.

In addition, when receiving a work pass request for a job reassigned by another worker 32, the server determines whether or not the number of times the work pass request is input for the job is equal to or greater than the reference number. In addition, if the determination result is greater than the reference number, the task is set as a high-difficulty task, and the assignment to the worker 32 and the request to perform the task are stopped for a predetermined period.

That is, in the case of high-difficulty tasks, even if they are continuously reassigned to other workers 32, there is a high possibility that the assigned workers 32 will continue to request a work pass, which in turn acts as a factor delaying the schedule of the project. During the period, the assignment to the worker 32 and the request to perform the task are stopped.

In contrast, when the number of times the work pass request is input is less than the reference number, the server withdraws the assignment for the work and reassigns it to another worker 32 to request the work to be performed.

In this case, the predetermined reference number set in an embodiment of the present invention may be determined according to the difficulty of the project.

That is, basically, if the difficulty of the project is high, the predetermined number of criteria decreases. Conversely, if the difficulty of the project is high, the predetermined number of criteria can be increased.

For example, if a project is of low difficulty, it can be estimated as a high-difficulty task only when many workers (32) request a work pass, but if a project is of high difficulty, even if a small number of workers (32) request a work pass, it is already a high-difficulty task. It can be estimated as

Therefore, even if there are only two requests for a work pass in a high-difficulty project, the task can be estimated as a high-difficulty task.

Referring to the example of FIG. 7, the server for jobs in which the number of times the work pass request is input is less than the standard number of times, that is, jobs such as 2, 5, 6, and 7 with only one work pass request. When requesting a work pass, the assignment is withdrawn and reassigned to another worker 32 to request the work again.

However, for tasks such as 3, 9, 10, etc., which have made two work pass requests, which is a predetermined number of times, are not withdrawn or reassigned, the assignment to the worker 32 and the request to perform the task are suspended for a predetermined period. Let it.

For high-difficulty tasks in the suspended state as described above, the server performs the high-difficulty task when the other tasks excluding the high-difficulty task among the plurality of tasks included in the project are completed, the high-level task is performed by a worker with an excellent grade among the plurality of workers 32. You can request to perform work by reassigning to

That is, according to an embodiment of the present invention, for tasks classified as high-difficulty tasks, task execution is stopped, and when all tasks of relatively low-difficulty tasks are completed, the reassignment of high-difficulty tasks can be started at that time. At this time, the server may be able to reassign the high-difficulty tasks to a plurality of workers 32, but extracts the workers 32 of excellent grade, which are workers 32 more likely to perform the task, and the corresponding worker 32 High-difficulty work can be reassigned to the class.

On the other hand, in the case of a task set as a high-difficulty task, it is carried out at the end of the project, and it is a task that was treated as difficult to perform by most of the workers (32) in the project. Can be given.

In one embodiment, the worker 32 of an excellent grade may be selected based on the rejection rate of the worker 32 from among the plurality of workers 32.

That is, the server may arrange the rejection rates of each of the workers 32 in descending order, and select workers 32 corresponding to a predetermined higher grade as workers 32 of the excellent grade. Since the operator 32 with a lower rejection rate is more likely to perform a high-difficulty task, an embodiment of the present invention calculates the rejection rate based on the inspection result when the other tasks except for the high-difficulty task are completed, and is excellent through it. It is possible to select a grade of worker 32

In addition, in the selection of the worker 32 of the excellent grade as an embodiment, in addition to considering the rejection rate, the number of times the work pass request is input among the plurality of workers 32 is less than the reference number, and the reference number-n times (only , n is a natural number of 1 or more) or more may be selected from one or more workers 32 who have passed the inspection.

For example, if 10 workers are first selected based on the rejection rate among the plurality of workers 32, the reference number is 2, and n is set to 1, the work pass request is input among the above 10 workers. The number of times is less than two, and the worker 32 who has passed the work for which the request for one or more work passes has been input may be selected as the worker 32 of an excellent grade.

That is, one embodiment of the present invention is not simply considering the rejection rate when selecting the workers 32 of the excellent grade, but a high degree of understanding of the project, so that the work pass request is lower than the reference number, and at the same time, other workers 32 ) Have higher understanding of the project than those of other workers 32, so it is possible to select the workers 32 who have passed the work for which the work path request by the other workers 32 is input as the workers 32 of excellent grade.

In another embodiment, the present invention is the number of times the number of times the work pass request is input among the plurality of workers 32 when selecting the worker 32 of the excellent grade is less than the reference number, Among the workers 32, it may be selected based on the rejection rate of each worker 32 and the speed of performing work.

In other words, since the point in time for reassigning the high-difficulty task corresponds to the end of the project, the worker 32 who has a low rejection rate and who can complete the task more quickly among those with a higher understanding than other workers 32 is ranked as excellent. It can be selected as the worker 32.

At this time, the speed of task execution can be calculated as the'number of tasks performed/project participation period'.

Alternatively, for each task assigned to each worker 32, the time when the task started and the time when the task was completed and the task result was input may be recorded, and an average task execution speed obtained by the average of these may be applied. In this case, the time at which the task starts to be performed may be the time at which the worker 32 is assigned the task, and the time at which the task result is input refers to the time at which the task result is submitted through the online workspace. On the other hand, in calculating the work execution speed, the rework may not be added up because it takes additional time due to the inspection by the inspector 34.

Meanwhile, an embodiment of the present invention has been described for the workers 32, but is not limited thereto. That is, according to an embodiment of the present invention, it is possible to request an inspection pass even for the inspectors 34, and the inspector may be evaluated based on this.

That is, when a plurality of work results are assigned to the inspectors 34 to request the inspection, the server may receive an inspection result or an inspection pass request from the plurality of inspectors 34.

When the project is finished, the server can evaluate and score a plurality of inspectors 34, and specifically, for each task of the project, the degree of difficulty per inspection is given based on the number of times the inspection pass request is input. , For each inspection case, an inspection score is given based on the inspection unit difficulty, and each inspector 34 is scored by summing the inspection scores for one or more inspection results for which each inspector 34 has completed the inspection. I can.

In addition, when there is a request for an inspection pass, the server may withdraw the inspection request for the work result assigned to the corresponding inspector 34, and reassign it to another inspector 34 to request the inspection performance.

According to this embodiment, the degree of understanding of the inspector 34 for a task with high difficulty is low, reducing to some extent simply inputting the result of the task to be returned by passing the inspection, and is assigned to the inspector 34 with higher understanding. The advantage is that the quality of the entire project can be improved further by doing so.

In addition, in the process of determining the pool of reviewers for similar projects, reviewers may be selected using the scoring results of the reviewers 34.

Meanwhile, in the above description, steps S110 to S153 may be further divided into additional steps or may be combined into fewer steps, according to an embodiment of the present invention. In addition, some steps may be omitted as necessary, and the order between steps may be changed. In addition, even if other contents are omitted, the contents of FIG. 8 to be described later can also be applied to a method of evaluating a worker using the difficulty of a work unit of a crowdsourcing-based project of FIGS.

Hereinafter, a description will be given of a worker evaluation apparatus 200 (hereinafter referred to as a worker evaluation apparatus) utilizing the work unit difficulty of a crowdsourcing-based project according to an embodiment of the present invention.

8 is a view for explaining a worker evaluation apparatus 200 according to an embodiment of the present invention.

Referring to FIG. 8, the operator evaluation apparatus 200 includes a communication module 210, a memory 220, and a processor 230.

The communication module 210 transmits a crowdsourcing-based work for one project to a plurality of workers 32 to request a task to be performed, and receives a work result from the plurality of workers 32. Further, the work results received from the plurality of workers 32 are transmitted to the plurality of inspectors 34 to request inspection, and the inspection results are received from the plurality of inspectors 34. In this case, the communication module 210 may receive a work pass request as well as a work result from the plurality of workers 32.

A program for evaluating and scoring a plurality of workers is stored in the memory 220.

The processor 230 executes a program stored in the memory 220. As the processor 230 executes the program stored in the memory 220, the task unit difficulty is assigned to each task of the project based on the number of times the work path request is input, and the task unit difficulty level is applied to each task. Based on the task score, each worker 32 performs a task and scores each worker by summing the task scores of one or more tasks that have passed the inspection.

The worker evaluation method using the work unit difficulty of a crowdsourcing-based project according to an embodiment of the present invention described above may be implemented as a program (or application) to be executed in combination with a computer, which is hardware, and stored in a medium. have.

The above-described program includes C, C++, JAVA, Ruby, which can be read by a processor (CPU) of the computer through the device interface of the computer, in order for the computer to read the program and execute the methods implemented as a program. It may include a code (Code) coded in a computer language such as machine language. Such code may include a functional code related to a function defining necessary functions for executing the methods, and a control code related to an execution procedure necessary for the processor of the computer to execute the functions according to a predetermined procedure. can do. In addition, such code may further include additional information required for the processor of the computer to execute the functions or code related to a memory reference to which location (address address) of the internal or external memory of the computer should be referenced. have. In addition, when the processor of the computer needs to communicate with any other computer or server in the remote in order to execute the functions, the code uses the communication module of the computer to determine how It may further include a communication-related code for whether to communicate, what kind of information or media should be transmitted and received during communication.

The stored medium is not a medium that stores data for a short moment, such as a register, cache, memory, etc., but a medium that stores data semi-permanently and can be read by a device. Specifically, examples of the storage medium include, but are not limited to, ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, and the like. That is, the program may be stored in various recording media on various servers to which the computer can access, or on various recording media on the user's computer. In addition, the medium may be distributed over a computer system connected through a network, and computer-readable codes may be stored in a distributed manner.

The above description of the present invention is for illustrative purposes only, and those of ordinary skill in the art to which the present invention pertains will be able to understand that it can be easily modified into other specific forms without changing the technical spirit or essential features of the present invention. will be. Therefore, it should be understood that the embodiments described above are illustrative in all respects and not limiting. For example, each component described as a single type may be implemented in a distributed manner, and similarly, components described as being distributed may also be implemented in a combined form.

The scope of the present invention is indicated by the claims to be described later rather than the detailed description, and all changes or modified forms derived from the meaning and scope of the claims and their equivalent concepts should be interpreted as being included in the scope of the present invention. do.

10: Client
20: service provider
30: public
32: worker
34: inspector
200: operator evaluation device
210: communication module
220: memory
230: processor

Claims (10)

As a method performed by a computer,
Assigning a plurality of tasks of a crowdsourcing-based project (hereinafter, referred to as a project) to a plurality of workers and requesting the task to be performed;
Receiving a plurality of work results or work pass requests from the plurality of workers;
Assigning the plurality of work results to a plurality of inspectors and requesting to perform inspection;
Receiving an inspection pass or rejection as a plurality of inspection results for the plurality of work results from the plurality of inspectors; And
When the project is completed, the step of evaluating and scoring the plurality of workers,
The step of evaluating and scoring the plurality of workers,
Assigning a work unit difficulty level to each task of the project based on the number of times the work pass request is input;
Assigning a task score to each task based on the task unit difficulty; And
Each worker performing a task and summing the work scores of one or more tasks that have passed the inspection and scoring each worker,
Assigning a plurality of tasks of the project to a plurality of workers and requesting to perform the task,
Assigning the same task to multiple different workers and requesting the task to be performed,
The step of assigning a work unit difficulty level based on the number of times the work pass request is input for each work of the project,
For each task of the project, assigning a work unit difficulty level based on the number of times the work pass request is input and a rejection rate,
Worker evaluation method using work unit difficulty of crowdsourced projects. delete The method of claim 1,
Assigning a work score based on the work unit difficulty for each of the tasks,
Dividing the work unit difficulty level into a predetermined number of sections, and assigning the same work score to a plurality of work unit difficulty levels belonging to the same section;
Including the step of assigning the task score based on the task unit difficulty and the section for each task,
Worker evaluation method using work unit difficulty of crowdsourced projects. The method of claim 1,
Based on the scoring results of the plurality of workers of the project, further comprising the step of determining a pool of workers of other projects whose similarity to the project is greater than or equal to a reference similarity,
Worker evaluation method using work unit difficulty of crowdsourced projects. The method of claim 1,
Assigning a plurality of tasks of the project to a plurality of workers and requesting to perform the task,
The task in which the request for the work pass is entered is withdrawn from the assignment and reassigned to another worker to request the task to be performed,
Worker evaluation method using work unit difficulty of crowdsourced projects. The method of claim 5,
Assigning a plurality of tasks of the project to a plurality of workers and requesting to perform the task,
A task in which the number of times the work pass request is input is greater than or equal to a predetermined reference number is set as a high-difficulty task, and the assignment to the worker and the request to perform the task are suspended for a predetermined period,
Worker evaluation method using work unit difficulty of crowdsourced projects. The method of claim 6,
Assigning a plurality of tasks of the project to a plurality of workers and requesting to perform the task,
When the remaining tasks other than the high-difficulty task among the plurality of tasks of the project are completed, reassigning the high-difficulty task to a worker of an excellent grade among the plurality of workers to request the task to be performed,
Worker evaluation method using work unit difficulty of crowdsourced projects. The method of claim 7,
Workers of the excellent grade,
Selected based on the rejection rate of each worker among the plurality of workers,
Worker evaluation method using work unit difficulty of crowdsourced projects. The method of claim 8,
Workers of the excellent grade,
Among the plurality of workers, the number of times the work pass request is input is less than the reference number and selected based on the rejection rate of each worker among one or more workers who have passed the inspection and passed n times (where n is a natural number of 1 or more). felled,
Worker evaluation method using work unit difficulty of crowdsourced projects. A computer program combined with a computer and stored in a computer-readable recording medium to execute an operator evaluation method utilizing the work unit difficulty of the crowdsourcing-based project of any one of claims 1, 3 to 9.

Images