KR102195961B1 - Method for applying the number of work limits per workers in a crowdsourcing based projects for artificial intelligence training data generation - Google Patents

Abstract

Provided is a method for applying the number of work restrictions for each worker in crowdsourcing-based projects. The method comprises the steps of: presetting the number of initial work restrictions and the maximum number of work restrictions for one worker of a project; assigning a plurality of works of the project to the plurality of workers and requesting work performance; receiving a plurality of work results from the plurality of workers; allocating the plurality of work results to a plurality of inspectors and requesting inspection performance; receiving a plurality of inspection results for the plurality of work results from the plurality of inspectors as passing or rejecting the inspection; and increasing and updating the maximum number of work restrictions for each worker, based on the inspection result after the number of initial work restrictions.

Description

{METHOD FOR APPLYING THE NUMBER OF WORK LIMITS PER WORKERS IN A CROWDSOURCING BASED PROJECTS FOR ARTIFICIAL INTELLIGENCE TRAINING DATA GENERATION}

The present invention relates to a method of applying a limit number of work per worker in a crowdsourcing-based project.

Recently, more and more companies collect and process large amounts of data on a crowdsourcing basis that engages the general public in some process of corporate activities. In other words, by opening a project and allowing the general public, that is, workers to participate in the project, necessary information is collected through the work results completed by the workers.

At this time, in order to collect more reliable information, the company assigns the work result completed by the worker to the inspector to perform the inspection work.

Specifically, when one project is opened, a plurality of tasks are assigned to each of the plurality of workers. Each worker performs a plurality of tasks assigned to it, and provides the task results. Thereafter, a plurality of inspection tasks are assigned to each of the plurality of inspectors, and each inspector performs a plurality of inspection tasks assigned to them.

On the other hand, in the current project, it is set to limit the number of tasks that a worker can perform in the project, regardless of the worker.

In this case, it is appropriate for workers who cannot perform the work well to be limited by the number of work restrictions, but workers who do well work quickly and at the same time provide high-quality work results. There is a problem that is subject to restrictions by.

Therefore, it is necessary to differentially set the number of work restrictions set equally for each worker to suit the work capability of the worker.

Unexamined Patent Publication No. 10-2014-0095956, 2014.08.04.

The problem to be solved by the present invention is to calculate a project matching value for evaluating the ability of workers using the average working time for each functional element in the previous project and the number of jobs using the functional element, and based on the project matching value, Provides a method of applying the number of work limits for each worker in a crowdsourcing-based project that can be updated by differentially increasing the maximum number of work limits for each worker based on the project's inspection results.

However, the problem to be solved by the present invention is not limited to the problem as described above, and other problems may exist.

In order to solve the above-described problem, the method of applying the number of work restrictions per worker in a crowdsourcing-based project according to an aspect of the present invention is the initial number and maximum number of work restrictions for one worker in a crowdsourcing-based project (hereinafter, "project"). Presetting the number of work restrictions; Allocating a plurality of tasks of the project to a plurality of workers and requesting the task to be performed; Receiving a plurality of job results from the plurality of workers; Assigning the plurality of work results to a plurality of inspectors and requesting to perform inspection; Receiving a plurality of inspection results for the plurality of work results from the plurality of inspectors as passing or rejecting the inspection; And increasing and updating the maximum number of job restrictions for each worker, based on a result of the inspection after the initial number of job restrictions. In this case, the step of pre-setting the number of initial work limit and the maximum work limit for one worker of the project may include: identifying a functional element of the project; Identifying a functional element in the previous project in which the worker participated, based on the history of the worker's participation in the previous project; Calculating an average working time for each functional element for the functional element included in the project among the functional elements included in the previous project; Calculating the project matching value for the worker based on the average working time for each functional element and the number of jobs performed by the worker using each functional element in the previous project; And among the plurality of workers, a first initial limit number of work is preset for workers below a predetermined level according to the project matching value, and a predetermined percentage of workers exceeding the predetermined level is greater than the first initial work limit number. It includes the step of presetting the number of reduced second initial work restrictions, the functional element is determined based on a work tool (Tool) for executing the project, the work tool is provided in the project, and the workers It is a tool used to perform the required task.

In some embodiments of the present invention, the step of requesting task performance by assigning a plurality of tasks of the project to a plurality of workers includes counting the number of current tasks performed by the worker and determining whether it matches the preset initial task limit number. Step to do; Stopping the assignment of additional tasks to the worker when the current number of tasks performed matches the preset number of initial task restrictions; Calculating the number of inspection passes by the operator based on the inspection result for the job result corresponding to the number of currently performed jobs by the worker; And resuming the assignment of a plurality of tasks to the worker when the number of passes of the inspection matches the number of initially restricted tasks.

Some embodiments of the present invention may further include resetting the number of job restrictions corresponding to the number of initial job restrictions, based on a result of the inspection after the initial number of job restrictions. In this case, the step of resetting the number of job restrictions corresponding to the number of initial job restrictions includes: calculating a rejection score corresponding to the number of rejected job results among the inspection results after the initial job restriction number for the plurality of workers. ; And when the rejection score corresponds to a preset condition, resetting the number of job restrictions corresponding to the initial job restriction number from a corresponding point in time, wherein the preset condition corresponds to the inspection result at each preset cycle. It may be any one of a case in which the rejection score is increased by a certain percentage or more compared to the previous period, and when the rejection score corresponding to the inspection result accumulated after the initial limit number of tasks is a predetermined score or more.

In some embodiments of the present invention, the step of calculating a rejection score corresponding to the number of rejected operation results among the inspection results for the plurality of workers after the initial limit number of operations may include: When it exceeds the set proficiency point quantity, the rejection score may be calculated by applying a weight based on the average rejection rate of the plurality of workers.

In some embodiments of the present invention, the step of calculating a rejection score corresponding to the number of rejected operation results among the inspection results for the plurality of workers after the initial limit number of operations, the rejection rate of the worker is the average rejection When the rejection rate is greater than or equal to 1, a weight of a first value greater than 1 is applied, and when the rejection rate of the worker is less than the average rejection rate, a weight of a second value smaller than 1 may be applied to calculate the rejection score.

In some embodiments of the present invention, the step of requesting work by assigning a plurality of tasks of the project to a plurality of workers, the first cumulative work performed by the worker from the point when the number of work restrictions is reset to the present Counting the number of executions and determining whether it matches the reset number of job restrictions; Stopping the assignment of additional tasks to the worker when the number of the first cumulative task execution matches the reset task limit number; Calculating the number of inspection passes of the worker based on the inspection result of the job result corresponding to the number of the first cumulative job execution by the worker; And resuming the assignment of a plurality of tasks to the worker when the number of inspection passes matches the number of task restrictions.

In some embodiments of the present invention, the step of allocating a plurality of tasks of the project to a plurality of workers and requesting the task to be performed comprises counting the total number of tasks performed by the worker from the first limit to the present time, Calculating the number of accumulated tasks; Determining whether the number of executions of the second cumulative work matches the maximum number of work restrictions; And stopping the assignment of additional tasks to the worker when the number of the second cumulative work execution matches the maximum work limit number.

In some embodiments of the present invention, the step of increasing and updating the maximum number of job restrictions for each worker, based on the inspection result after the initial number of job restrictions, corresponds to the preset maximum number of job restrictions of the worker. Calculating the number of first inspection passes of the worker based on the inspection result for the job result; And updating and increasing the maximum number of work restrictions of the worker in correspondence with the number of passes of the initial inspection.

In some embodiments of the present invention, the step of increasing and updating the maximum number of job restrictions for each worker, based on the inspection result after the initial number of job restrictions, includes the worker according to the update of the maximum number of job restrictions. Calculating the number of additional tasks performed by counting the total number of additional tasks performed; Calculating the number of initial inspection passes of the operator based on the inspection result for the job result corresponding to the number of additional jobs performed by the operator; And it may include the step of increasing the updated maximum number of work restrictions by the operator corresponding to the number of passes for the first inspection and updating again.

A computer program according to another aspect of the present invention for solving the above-described problems is combined with a computer as hardware to execute a method of applying the number of work limits per worker in the crowdsourcing-based project, and is stored in a computer-readable recording medium. .

Other specific details of the present invention are included in the detailed description and drawings.

According to the present invention described above, the initially set number of work limits can be set differently in consideration of the worker's performance in the previous project. That is, for a worker who can perform a task more proficiently, a corresponding limit is set lower than that of other workers, so that more tasks can be assigned and performed.

In addition, even if the initial limit number of work is passed, the number of work limits is set again according to the rejection score considering the rejection rate of the worker, thereby preventing the quality and progress of the project from deteriorating due to unskilled workers. have.

In addition, by setting and updating the maximum number of work limits, workers with a high rejection rate who do not perform well are prevented from performing more than the limit, and at the same time, additional work is assigned to workers who perform well. It has the advantage of being able to improve the quality and progress of the project at the same time by allowing it to be received.

The effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the following description.

1 is a conceptual diagram of a crowdsourcing service according to an embodiment of the present invention.
2 is a flowchart illustrating a crowdsourcing-based project progress process according to an embodiment of the present invention.
3 is a view for explaining the number of work restrictions applied to an embodiment of the present invention.
4 is a flowchart of a method of applying the number of work restrictions for each worker according to an embodiment of the present invention.
5 is a flow chart for explaining contents of presetting the number of initial limit jobs.
6 is an exemplary view for explaining the participation history of the previous project.
7 is an exemplary view for explaining the content of calculating the work time for each functional element a worker participates.
8 is a flow chart for explaining the content to which the initial number of work restrictions is applied.
9 is a diagram for explaining an example in which the number of initial job restrictions and the number of job restrictions are applied.
10 is a flow chart for explaining contents of task assignment to a plurality of workers in relation to the maximum number of work restrictions.
11 is a diagram illustrating an example of updating by increasing the maximum number of work restrictions.
12 is a view for explaining an apparatus for applying the maximum number of work restrictions per worker according to an embodiment of the present invention.

Advantages and features of the present invention, and a method of achieving them will become apparent with reference to the embodiments described below in detail together with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but may be implemented in a variety of different forms, only the present embodiments are intended to complete the disclosure of the present invention, It is provided to fully inform the technician of the scope of the present invention, and the present invention is only defined by the scope of the claims.

The terms used in the present specification are for describing exemplary embodiments and are not intended to limit the present invention. In this specification, the singular form also includes the plural form unless specifically stated in the phrase. As used in the specification, “comprises” and/or “comprising” do not exclude the presence or addition of one or more other elements other than the mentioned elements. Throughout the specification, the same reference numerals refer to the same elements, and “and/or” includes each and all combinations of one or more of the mentioned elements. Although "first", "second", and the like are used to describe various elements, it goes without saying that these elements are not limited by these terms. These terms are only used to distinguish one component from another component. Therefore, it goes without saying that the first component mentioned below may be the second component within the technical idea of the present invention.

Unless otherwise defined, all terms (including technical and scientific terms) used in the present specification may be used as meanings that can be commonly understood by those of ordinary skill in the art to which the present invention belongs. In addition, terms defined in a commonly used dictionary are not interpreted ideally or excessively unless explicitly defined specifically.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a conceptual diagram of a crowdsourcing service according to an embodiment of the present invention.

Referring to FIG. 1, a crowdsourcing service is performed by being composed of a client 10, a service provider 20, and a public 30.

The sponsor 10 refers to a company or individual who requests a crowdsourcing-based project (hereinafter, referred to as a project).

The requester 10 requests a project for the purpose of collecting source data or annotating data for generating artificial intelligence learning data. The data generated through the project can be used as learning data for arbitrary machine learning such as supervised learning, unsupervised learning, and reinforcement learning. Collection of source data refers to collecting raw data such as recorded voice collection and photo collection. Data annotation refers to inputting relevant annotation data into source data such as text, photos, and videos. For example, the data annotation may include finding an entity in a given fingerprint or finding a similar sentence, but is not limited thereto. Meanwhile, the types of the above-described projects are only one embodiment, and various projects may be handled in the present invention according to the design of the client.

The service provider 20 refers to a company that provides crowdsourcing services.

When the service provider 20 receives a request for a project for a product or service from the client 10, the service provider 20 allocates a task for the project to the general public 30 and receives the work result from the public 30. Thereafter, the final product extracted based on the work result is provided to the client 10.

At this time, the service provider 20 provides a crowdsourcing service to the client 10 and the public 30 through a crowdsourcing platform (hereinafter, the platform). That is, when the service provider 20 receives a request for a project from the client 10, the service provider 20 opens the project on the platform. Thereafter, when the work result for the open project is received from the public 30, the project is terminated on the platform, and the final product may be extracted and provided to the client 10.

The public 30 refers to the general public participating in the project open on the platform. Here, the public 30 may participate in a project opened on the platform through an application or website provided by the service provider 20.

The public 30 is composed of a worker 32 and an inspector 34.

The worker 32 decides to participate in a specific project among a plurality of projects open on the platform. Thereafter, the operator 32 performs an operation such as collection of source data or data annotation, and transmits it to the platform.

The inspector 34 decides to participate in a specific project among a plurality of projects open on the platform. Thereafter, the inspector 34 inspects the result of the work performed by the operator 32. As a result of performing the inspection, the inspector 34 may perform inspection pass processing or rejection processing, and input a rejection reason during rejection processing. In the case of passing the inspection, it is not necessary to rework and re-examination due to this, so passing inspection has the same meaning as completion of inspection.

2 is a flowchart illustrating a crowdsourcing-based project progress process according to an embodiment of the present invention.

First, the requester 10 requests one or more projects to the service provider 20 (S11).

Thereafter, the service provider 20 opens the requested project on the platform (S12). At this time, the service provider 20 may determine the grade in consideration of the difficulty of the project before opening the project. That is, depending on the difficulty level, it is possible to determine which level or higher public 30 to expose the project. Accordingly, it is possible to increase the reliability of the work result of the project.

Thereafter, the service provider 20 requests the work by assigning the work to the workers 32 of the corresponding level or higher according to the level of the project (S13).

Thereafter, the worker 32 performs the assigned task (S14). In this case, the worker 32 may input the reason for the inability to work without performing the work for a work in which the work itself is impossible for some reason.

Thereafter, the service provider 20 receives the work result from the worker 32 (S15), and assigns the work result to the inspector 34 to request the inspection (S16).

Similarly, according to an embodiment of the present invention, only a suitable project may be exposed to the inspector 34 out of the total projects being performed according to the level set according to the difficulty of the project or the qualification requirements of the inspector 34.

Thereafter, the inspector 34 performs the assigned inspection (S17). At this time, the inspector 34 decides to complete the inspection when it is determined that the work has been properly performed, and if it is determined that the inspection work is wrong, it rejects the inspection. During rejection processing, the inspector 34 inputs the rejection reason for what reason the job was judged to be wrong.

Thereafter, the service provider 20 receives the inspection result from the inspector 34 (S18).

When the inspection result is the inspection completion, the service provider 20 uses the work result as valid data, and based on this, extracts the final product at the end of the project.

When the inspection result is rejection processing, the service provider 20 may internally perform the inspection again, or assign the work to the worker 32 again to perform the rework. When reworking, re-examination by an inspector is required.

Thereafter, when the project period ends or sufficient valid data is secured, the service provider 20 terminates the project (S19), calculates a final result based on the secured valid data, and provides it to the client 10 ( S20).

At this time, before the end of the project, the service provider 20 evaluates the performance results of the worker 32 and the inspector 34, calculates the work cost and the inspection cost according to the evaluation, to the worker 32 and the inspector 34. give.

In FIGS. 1 and 2, it is simply expressed as a requester 10, a service provider 20, an operator 32, and an inspector 34, but these are smartphones, tablets, PDAs, laptops, desktops, operated by each participant. It means a computer device such as a server or a telecommunication device.

3 is a view for explaining the number of work restrictions applied to an embodiment of the present invention.

According to an embodiment of the present invention, a safety device called the number of work restrictions is set because workers 32 whose quality is not guaranteed as a result of work may enter due to the nature of a project called crowdsourcing.

At this time, in the present invention, the number of initial work restrictions and the maximum number of work restrictions are set as the number of work restrictions.

Specifically, when the project is opened, the workers 32 are assigned a new task to perform the task (S21), and the new task is assigned as much as the preset initial task limit number to perform the task (S22).

Thereafter, the workers 32 can be assigned a new task only when all the work results corresponding to the initial limit number have been inspected and passed (S23-Y), and if there is a rejected work result, the new task assignment is stopped (S23-N, S24), new work can be assigned only when all rework is completed.

Thereafter, when the number of tasks assigned to the worker 32 reaches the maximum task limit number (S27), the task assignment is stopped (S28), and the workers 32 cannot work beyond the maximum task limit number.

Here, the number of initial task restrictions is a kind of workers (32) who can initially receive tasks as much as the set number of tasks after the project is opened, and receive additional tasks only when all task results for the initially assigned task have been inspected. Is the limit. In particular, the initial limit number of work is to ensure the quality of the work results of the workers 32 by ensuring that additional work can be assigned only after all inspections are completed for a predetermined number, and the inspector 34 within a limited number of cases. It also includes the purpose of educating the worker 32.

In addition, the maximum number of work limits set in the present invention means the number of work that each worker 32 can perform to the maximum within one project. For example, if the preset maximum number of work limits is 50, in principle, each worker 32 cannot perform more than 50 work.

However, not all workers 32 are assigned 50 tasks equally and at the same time. In other words, when the first task is completed, the workers 32 request and perform the next task, and when the second task is completed, the next task is requested and performed. Therefore, the task is performed and assigned according to each situation. During the same period, one worker may reach 50, while another worker may be assigned only 10 or 30 tasks during the same period to complete.

This maximum number of work limits is set to limit a specific worker from making excessively defective work results, and in addition to this, it is also set when it is desired to collect work results performed by various workers.

On the other hand, the initial limit number of work is, the remaining work of the project remains, the worker 32, which limits the work performance of the worker 32 even though the maximum number of work limit set for one worker 32 has not been reached. It is the means that most greatly limits the autonomy of a person.

Therefore, if the initial number of work restrictions is not resolved quickly, unlike the maximum number of work restrictions, it may cause a problem of the departure of the worker 32 who wants to perform the work hard and reducing the motivation of the excellent worker.

Above all, due to the nature of the crowdsourcing project that is inspected in the order of submission, if a worker (32) who has passed the initial limit number of jobs submits a lot of new jobs, the worker who is expected to be an excellent worker submits to pass the initial limit number of jobs. For the case, the inspection can be very late, which in turn leads to a situation where the optimal worker for the project cannot perform the work.

In this way, when the same initial limit number of work is set for all workers 32 without considering the work ability of the worker 32, the above problem can be reduced even a little by setting a larger number of the initial limit number of work. . In addition, when a worker with below average performance passes the initial limit number, it takes a relatively long time to inspect the work result of the worker, which takes more time for other workers to pass the initial limit number. It makes you need.

In this regard, according to an exemplary embodiment of the present invention, the problem caused by the above-described initial number of restrictions can be solved by setting the number of initial restrictions on the worker 32 according to the job proficiency of the worker 32.

In addition, an embodiment of the present invention is to be updated by differentially increasing the maximum number of work limits for each worker 32 in consideration of the inspection results for the workers 32, to the worker 32 who performs a better job. It can give you the opportunity to do more.

Hereinafter, a method of applying the number of work restrictions per worker 32 in a crowdsourcing-based project according to an embodiment of the present invention (hereinafter, a method of applying the number of work restrictions per worker) will be described with reference to FIGS. 4 to 11. .

4 is a flowchart of a method of applying the number of work restrictions per worker 32 according to an embodiment of the present invention. 5 is a flow chart for explaining contents of presetting the number of initial limit jobs. 6 is an exemplary view for explaining the participation history of the previous project. 7 is an exemplary view for explaining the content of calculating the work time for each functional element in which the worker 32 participated.

Meanwhile, the steps shown in FIG. 4 may be understood as being performed by a platform server (hereinafter, referred to as a server) operated by the service provider 20, but are not limited thereto.

First, the server presets the number of initial work restrictions and the maximum number of work restrictions for one worker 32 of a crowdsourcing-based project (hereinafter, referred to as a project) (S110).

Specifically, before opening the project, the server identifies functional elements of the project (S111). In this case, the functional element is determined based on a work tool for executing the project, and the work tool is provided in the project, and may be a tool used by the worker 32 to perform the work required by the project. In other words, the server identifies what work tools are used to perform the work of the project scheduled to be open, and determines functional elements based on this.

For example, if you need a bounding box tool for bounding work, an input box tool for transfer work, and a make step tool for sorting work to perform a project , The project can be identified as including a bounding functional element, a transcription functional element, and a classification functional element.

Thereafter, the server identifies functional elements in the previous project in which the worker 32 participated, based on the history of the worker 32's participation in the previous project (S112). That is, the server identifies functional elements included in each of the previous projects in which the worker 32 participated.

For example, referring to FIG. 6, worker A has participated in previous projects A, B, and C. The server identifies the functional elements of the previous project in which worker A participated, and as a result, the previous project A contains the functional elements a, b, c, and the previous project B contains the functional elements b, d, e, f, and Project C is identified as containing functional elements a, d, and g.

Then, the server calculates the average working time for each functional element for the functional element included in the project to be opened among the functional elements included in the previous project (S113). That is, when the previous project includes the same functional element as the functional element identified in the project to be opened, the average working time of the workers 32 using the functional element in the previous project is calculated.

For example, referring to FIG. 6, for functional elements a, b, and d of a project scheduled to be open, worker A worked with functional elements a and b in the previous project A, and functional elements b and d in the previous project B. It can be seen that the work was performed using the function elements a and d in the previous project C.

For this, the server calculates the average working time of 16.5 seconds and 24 seconds using functional elements a and b in the previous project A, and the average working time of 15 seconds and 12 seconds using functional elements b and d in the previous project B, respectively. In the previous project C, the average working time of 13.5 seconds and 28 seconds using functional elements a and d can be calculated, respectively.

Meanwhile, according to an embodiment, the average working time for each functional element may be calculated using work information of a previous project of the worker 32. At this time, the work information of the previous project may include at least one of the total work time for each functional element in the previous project, the total work number, the average work time, the number of rejections, the work time of the rejected task, and the work time considering the number of rejections. I can.

In this case, the average working time for each functional element may be calculated in consideration of the difficulty of a task using the corresponding functional element, and the difficulty of the task may be determined based on the number of times the task was rejected.

Here, the total working time for each functional element is calculated by recording the start time and end time of use for a specific functional element in the previous project, and the time that a plurality of workers 32 participating in the previous project used the functional element. Means the summed total time. In addition, the total number of jobs means the number of final job results provided by the plurality of workers 32.

In one embodiment, the average work time means a value obtained by dividing the total work time by the total number of work, and the rejection work time means a value obtained by multiplying the rejection number and the average work time. In addition, the working hours considering the number of rejections means the sum of the total working hours and the rejected working hours.

Accordingly, the average work time for each functional element in an embodiment of the present invention may be calculated by dividing the work time considering the number of rejections by the total number of work.

Referring to FIG. 7, in the previous project A, when a plurality of workers 32 used the functional element a (for example, a bounding box tool), the total working time is 1500 seconds, and the total number of jobs is 100, the average working time Is calculated as 15 seconds. At this time, since the number of rejections is 10, the working time of the rejected task is calculated as 150 seconds, which is the average work time multiplied by the number of rejections.

Accordingly, in the previous project A, the total working time using the functional element a becomes 1650 seconds, and if this is divided by the total number of operations 100, the average working time for the functional element a is calculated as 16.5 seconds.

According to this method, the average working hours for functional element b and functional element c are calculated as 24 seconds and 62.5 seconds, respectively.

Referring again to FIG. 5, the server next, based on the average working time for each functional element and the number of jobs performed by the worker 32 using each functional element in the previous project, the project for the worker 32 A matching value is calculated (S114).

At this time, the project matching value is a numerical value indicating how well the worker 32 performed the work in the previous project with respect to the functional element included in the project to be opened, and the corresponding worker 32 indicates the work ability suitable for the project to be opened. You can judge whether you have it.

In one embodiment, the server calculates the nth working time by multiplying the average working time of the nth functional element (n is a natural number) and the number of jobs performed by the operator 32 using the nth functional element, and By summing all the time, a matching value for the project scheduled to be opened by the worker 32 may be calculated.

At this time, even if a plurality of functional elements among the n functional elements are based on the same work tool, the previous project including the plurality of functional elements may be different, and in this case, when calculating the matching value of the project to be opened, You can also apply weights.

6, for functional elements a, b, and d of the project to be opened, worker A used functional elements a, b in the previous project A, functional elements b, d in the previous project B, and transferred In Project C, I worked with functional elements a and d. At this time, the functional elements used in the previous projects A to C are referred to as first to sixth functional elements, respectively, and a project matching value for worker A may be calculated.

Specifically, when the first to sixth working hours for each of the first to sixth functional elements are 165 seconds, 120 seconds, 90 seconds, 24 seconds, 108 seconds, 84 seconds, the project matching value for worker A is It is calculated as 591 points, the sum of these.

On the other hand, in an embodiment of the present invention, when the number of tasks performed by the worker 32 using a specific functional element in a previous project is less than or equal to the average number of tasks of the functional element, the value of the number of tasks performed using the functional element is By setting it to 0, it is possible to reduce the influence of the working time on the corresponding functional element. That is, when the number of tasks completed by the worker 32 using a specific functional element is less than the average number of tasks completed by the plurality of workers 32 using the functional element, the weight value for the functional element Is set to 0, and can be excluded from the matching value.

For example, among the first to sixth functional elements, if the number of tasks performed by the operator A using the first functional element is 10, and the average number of tasks of a plurality of workers for the first functional element is 13, As the weight value for the first functional element is set to 0, the project matching value is calculated as 426 points instead of 591 points.

In another embodiment applied separately or in conjunction with the above-described embodiment, when the workers 32 are listed based on the number of tasks performed using a specific functional element, the priority of the corresponding worker 32 is a predetermined priority. For the following functional elements, by setting the value of the number of jobs performed by using the functional element to 0, it is possible to reduce the degree of influence of the working time for the corresponding functional element.

For example, when the pre-determined priority is the middle priority, if the ranking based on the number of tasks for the first functional element of worker A among the first to sixth functional elements is 40 out of 60, the first functional element As the weight value for is set to 0, the project matching value is calculated as 426 points instead of 591 points.

When the project matching value is calculated in this way, the server pre-sets the first number of initial work restrictions for workers below a predetermined level according to the project matching value among the plurality of workers 32, and the number of workers exceeding the predetermined level in advance. 1 The second initial limit number of work, which is reduced by a predetermined ratio from the number of first limit work, is preset (S115).

For example, if the number of the first initial task limit is set to 10, the number of the second initial task limit, which is 30% lower than 10, can be set for workers whose project matching value is in the top 30%, and the remaining 70% For the workers of, the first limit of 10 can be set.

In this case, a certain ratio applied to workers of a predetermined grade or higher may be differentially applied by a ratio according to a higher grade. In other words, for the top 30%, the second initial limit number is applied, for the top 10%, a 30% reduction ratio, for the top 10-20%, a reduction ratio of 20%, and the top 20-30%. For, a reduction rate of 10% can be applied differentially.

As described above, an embodiment of the present invention allows the number of initial work limits to be set differently for each worker 32 according to the project matching value, so that the limit is quickly resolved for excellent workers so that they can more actively participate in the project. It can be induced, and training and inspection can be performed by limiting the number of new workers or workers with low performance so that the quality of the work result can be guaranteed.

When the initial limit number of tasks is set for the workers 32, the server assigns a plurality of tasks to the plurality of workers 32 and requests the task to be performed (S120), and a plurality of task results from the plurality of workers 32 It receives the input (S130).

Then, the server assigns the plurality of work results to the plurality of inspectors 34 to request the inspection performance (S140), and passes or rejects the inspection results for the plurality of work results from the plurality of inspectors 34 It is input as (S150).

8 is a flow chart for explaining the content to which the initial number of work restrictions is applied. 9 is a diagram for explaining an example in which the number of initial job restrictions and the number of job restrictions are applied.

Workers (32) are able to assign and perform tasks within the initial limit number of differently set tasks, and when the number of tasks is reached, additional task assignments and tasks cannot be performed. Work can be done.

Specifically, the server counts the number of current tasks performed by the worker 32 to determine whether it matches the preset initial task limit (S121, S122), and if the current task execution number matches the initial task limit (S122-Y) ), stop the assignment of additional tasks to the worker 32 (S123).

Thereafter, the server calculates the number of inspection passes by the operator 32 based on the inspection result for the work result corresponding to the current number of work performed by the operator 32, and when the number of inspection passes coincides with the initial limit number ( S124-Y), it is possible to resume the assignment of a plurality of tasks to the worker 32 by removing the limit on the number of initial task restrictions (S125).

Referring to FIG. 9, when the number of preset initial limit jobs for worker A is 5, worker A cannot continuously perform more than 5 jobs. The server checks whether the number of current job executions of worker A has reached the initial job limit number each time the job result is submitted, and stops job assignment when 5 cases are reached.

Therefore, worker A, whose assignment of additional tasks is suspended, can perform the next task by requesting and assigning the next task only when all five task results have been inspected. That is, if some of the 5 work results are rejected and reworked, worker A cannot assign additional work even if some work passes the inspection.

If the inspection is passed for all five work results, worker A can freely be assigned and performed work within the maximum number of work limits described later.

In an embodiment of the present invention, when the worker 32 performs many tasks in a state that is not familiar with the work, the rejection rate for the work result is high, and the fatigue of the inspector 34 and the corresponding worker 32 increases. However, this eventually delays the schedule of the project, and the above problem can be solved by limiting the number of initial work restrictions.

In addition, in the case of the worker 32 whose rejection rate is still high despite the setting of the initial number of job restrictions, the number of job restrictions to be described later may be reset or the maximum number of job restrictions may be applied.

In an embodiment, the server may reset the number of job restrictions corresponding to the initial number of job restrictions based on a result of the inspection after the initial number of job restrictions for the worker 32 is released.

That is, even if the server still does not submit the work results as much as the maximum number of work limits to the worker 32, which has a high rejection rate, it is possible to re-set the sanctions such as the initial limit number of work to the worker 32.

To this end, the server calculates a rejection score corresponding to the number of rejected operation results among the inspection results for the plurality of workers 32 after the initial limit number of operations. In addition, when the calculated rejection score corresponds to a preset condition, the number of job restrictions corresponding to the initial number of job restrictions from that point in time may be reset.

Here, the preset condition is when the rejection score corresponding to the inspection result at each preset cycle has increased by a certain percentage or more compared to the previous period, or when the rejection score corresponding to the inspection result accumulated after the initial limit number of work is at least a certain score. Can be

When the number of job restrictions is reset as described above, the server counts the number of the first cumulative job executions performed by the operator 32 from the time of reset to the present, as in the case where the initial job limitation number is applied, It is determined whether they match, and if they match, the assignment of additional tasks to the worker 32 is stopped.

And the server calculates the number of inspection passes of the worker 32 based on the inspection result for the work result corresponding to the number of the first cumulative work performed by the operator 32, and when the number of inspection passes matches the number of work restrictions, Assignment of a plurality of tasks to the worker 32 is resumed.

Referring back to the example of FIG. 9, the worker A is allotted a task to perform a task and submits a task result after all five cases, which are the first task restrictions, have been inspected and passed.

At this time, the server calculates the rejection score for every 5 cases, which is the preset job result submission cycle for worker A, and when the rejection score in the current cycle increases by, for example, 30% or more compared to the previous cycle, the job after the current cycle You can reset the number of work limits for. According to this limitation, worker A can assign additional work only after all five work results have been inspected and passed.

As another example, when worker A continues to receive and execute a total of 15 tasks after the initial limit number of tasks, and if the rejection score for the results of 10 tasks exceeds a certain score, the assignment of additional tasks is stopped, and 5 tasks are performed. Additional tasks can be assigned only when all the results of the 11th to 15th tasks, which are limited to the number of cases, have been inspected and passed.

Meanwhile, the rejection score in an embodiment of the present invention is calculated based on the number of rejected task results among tasks performed by the worker 32.

Specifically, the server may calculate a rejection score based on whether the number of current tasks performed by the worker 32 exceeds a preset number of proficiency points.

Here, the number of proficiency points is a reference value at which the project is opened and the worker 32 performs a task to some extent, and then begins to become proficient in performing the task, and may be set in advance by the server.

The server can derive the number of proficiency points through a pilot project. The server selects the workers 32 based on a project similar to a project currently in progress among projects that have been carried out previously. For example, the server evenly selects high-level proficient workers, intermediate proficient workers, and low-level proficient workers based on the proficiency measured through a previous similar project. The selected high, middle, and low proficiency level workers participate in the pilot project and perform tasks.

At this time, the server may select the number of workers 32 participating in the pilot project equal to the number participating in the formal project, or may select only a predetermined ratio of the number of workers 32 participating in the formal project. .

The server may set the number of work at the point when the average review completion rate of the pilot project by the selected workers 32 sharply increases and the average rejection rate sharply decreases as the number of proficiency points.

The number of proficiency points may be set in advance by the project manager or the requester 10, not the server.

When the number of current tasks performed by the worker 32 is less than a preset number of proficiency points, the server may calculate a rejection score without applying a weight.

Because individual differences and differences in experience are different for each worker 32, the variation in initial work ability is relatively large. In addition, in the case of the beginning of the work, the work may be performed incorrectly depending on the interpretation of the work guide.

Therefore, the server calculates a rejection score without applying a weight when the number of current tasks is less than the number of proficiency points. For example, if the number of current tasks performed by the specific worker 32 at the beginning of the project is 30 and the number of proficiency points is 50, the server may calculate a rejection score without applying a weight.

In this case, the server may calculate the rejection score by multiplying the rejection unit price by the rejection work quantity when the current number of tasks performed by the worker 32 is less than the number of proficiency points.

For example, if the number of rejected tasks is 20 out of 30 currently performed by a specific worker and the rejection unit price is 20 won, the rejection score of the worker is calculated as 400 points.

Alternatively, the server may calculate a rejection score by applying a weight based on the average rejection rate of the plurality of workers 32 when the number of current tasks performed by the worker 32 exceeds the number of proficiency points. Here, the average rejection rate can be calculated for each number of jobs through the pilot project.

In addition, the server has a weight of a first value (for example, 2) greater than 1 if the worker 32's rejection rate is greater than or equal to the average rejection rate when the number of current tasks performed by the worker 32 exceeds the proficiency point quantity. And, if the rejection rate of the worker 32 is less than the average rejection rate, the rejection score may be calculated by applying a weight of a second value less than 1.

For example, when the number of current tasks performed by the worker 32 exceeds the number at the time of proficiency, and the rejection rate of the worker 32 is higher than the average rejection rate, the number of rejections greater than the average rejection quantity is twice as weighted. Is applied to calculate the rejection score.

Specifically, referring to Equation 1 below, the server does not apply a weight with respect to the quantity corresponding to the average rejected quantity among the total rejected quantity of the worker 32, and the first in relation to the remaining quantity exceeding the average rejected quantity. By applying the weight of the value, the rejection score can be calculated.

[Equation 1]

Average rejected quantity x rejected unit price + (rejected quantity-average rejected quantity) x rejected unit price x 2

For example, if a worker's total rejection quantity for 70 jobs is 50, the average rejection quantity for 70 jobs is 35, and the rejection unit price is 10 won, the rejection score is 35×10+(50-35) It can be calculated as ×10×2 = 650 points.

In contrast, when the rejection rate of the worker 32 is less than the average rejection rate, the server calculates a rejection score by applying a weight of the second value (for example, 0.5 times) to the rejection quantity less than the average rejection quantity.

Specifically, referring to Equation 2 below, the server applies the weight of the second value with respect to the quantity that is less than the average rejected quantity among the total rejected quantity of the worker 32, and does not apply the weight with respect to the remaining quantity. Companion score can be calculated.

[Equation 2]

(Average rejected quantity-rejected quantity) × rejected unit price × 0.5 + (rejected quantity-(average rejected quantity-rejected quantity)) × rejected unit price

For example, if a worker's total rejection quantity for 70 jobs is 20, the average rejection quantity for 70 jobs is 35, and the rejection unit price is 10 won, the rejection score is (35-20)×10× It can be calculated as 0.5+(20-(35-20))×10=125 points.

Such a rejection score may be calculated by accumulating work results and inspection results performed after the initial limit number of work by the worker 32, and when the rejection score is more than a certain score, the number of work restrictions may be reset. In addition, the rejection score may be calculated for each preset job result submission cycle, and when the rejection score in the current cycle increases by more than a certain percentage compared to the previous cycle, the number of job restrictions may be reset. The method of resetting the number of work restrictions according to the method of calculating the rejection score may be applied individually or simultaneously.

Referring to FIG. 4 again, the server increases and updates the maximum number of job restrictions for each worker 32 based on the inspection result after the initial number of job restrictions (S160).

At this time, in the above-described embodiment, when the number of job restrictions is reset after the initial number of job restrictions is released, the server determines the maximum number of job restrictions for each worker 32 based on the inspection result after the reset number of job restrictions. It can be updated by increasing it.

10 is a flow chart for explaining the content of task assignment to a plurality of workers 32 in relation to the maximum number of work restrictions. 11 is a diagram illustrating an example of updating by increasing the maximum number of work restrictions.

The server counts the total number of tasks performed by the operator 32 from the first task limit to the present to calculate the second cumulative task execution number (S126), and whether the second cumulative task execution number matches the maximum task limit number. Whether or not it is determined (S127), and if they match, the assignment of additional tasks to the worker 32 is stopped (S128).

As shown in the example of FIG. 11, when the maximum number of tasks is 50, the server counts and calculates the total number of tasks performed by the worker 32 so far. When the cumulative number of tasks reaches 50, the worker ( 32) stop assigning additional tasks.

That is, the worker 32 can receive only a maximum of 50 tasks for one project, and it is not possible to assign additional tasks any more. However, in an embodiment of the present invention, it is possible to exceptionally allow the assignment of additional tasks to the worker 32 who is performing the task well without causing a rejection. Hereinafter, the maximum number of task restrictions for each worker 32 is increased. Explain in detail what is being asked.

The server calculates the number of initial inspection passes of the operator 32 based on the inspection result for the job result corresponding to the preset maximum number of job restrictions of the operator 32, and corresponds to the number of initial inspection passes. It can be updated by increasing the maximum number of work limits.

Here, the number of first inspection passes refers to the number of inspections passed at one time without rejection as a result of inspection for the initial work result. In other words, it means the number of cases passed without rejection of the work result.

In this regard, referring to FIG. 11, when the worker 32 reaches the maximum number of work limits of 50, it is a principle that additional work assignments are not possible, but as an exception, the inspection result for the maximum number of work limits of 50, If the number of initial inspection passes is 40, the maximum number of work restrictions may be increased by 40, which is the number of initial inspection passes, and may be updated to a total of 90 cases.

The process of updating the maximum number of work limits may be repeated. That is, when the worker 32 additionally performs work as the maximum number of work limits is updated, the total number of additional work is counted to calculate the number of additional work, and corresponds to the number of additional work performed by the worker 32 Based on the inspection result for the result of the work to be performed, the number of initial inspection passes by the worker 32 is calculated. In addition, it is possible to update again by increasing the maximum number of work restrictions of the worker 32 corresponding to the number of first inspection passes.

Referring to FIG. 11, it is a principle that the worker 32 cannot perform a task in excess of 90 tasks as the maximum number of tasks is primarily increased and updated to 90 tasks. However, as a result of the inspection result of the 40 work results additionally assigned by the worker 32, if the number of initial inspection passes is 35, the maximum number of work restrictions increases again by 35, which is the initial inspection pass, and can be updated to a total of 125 cases. have.

This process of updating the maximum number of tasks can be repeated until all tasks in the project have been assigned.

Meanwhile, in an embodiment of the present invention, while the maximum number of job restrictions is updated, the aforementioned limitation on the number of job restrictions may not be applied. That is, in an exemplary embodiment of the present invention, when the worker 32 additionally performs a task according to the update of the maximum number of task restrictions, the suspension of the additional task assignment related to the number of task restrictions may not be applied.

This is when the number of remaining work in the project is not large, and the maximum number of work limits increases based on the number of initial inspection passes that have not been rejected, which is a trusted worker who understands and performs the work well. Since it can be seen as (32), the limit on the number of work restrictions may not apply in this case.

The update of the maximum number of work limits can be made each time the first number of inspection passes occurs. Alternatively, when the number of accumulated work performed by the worker 32 reaches the maximum work limit, the maximum work limit number may be updated.

In another embodiment, the server calculates the number of inspection passes of the operator 32 based on the inspection result for the job result corresponding to the preset maximum number of job restrictions of the operator 32, and the number of inspection passes is a preset maximum. If it matches the number of work restrictions, it may be updated by increasing the maximum number of work restrictions of the worker 32 corresponding to the number of initial inspection passes.

Referring to the example of FIG. 11, the operator 32, which is set to 50 pre-set maximum number of work limits, completed and submitted all 50 work, and at this time, the test result of the work result of the worker 32 was 45. In the case where the rework remains due to the rejection of the pass and 5 cases, the maximum number of work limits may be restricted so that the maximum number of work limits is not updated to the corresponding worker 32 until all 5 rework are inspected and passed.

That is, the embodiment of the present invention can increase the maximum number of job restrictions when all of the preset maximum number of job restrictions have passed the inspection. This is because when the initial inspection result is rejected and rework is necessary, it is preferable to prioritize the rework rather than allow the worker 32 to be assigned an additional task.

In another embodiment, the server calculates the number of initial inspection passes of the worker 32 based on the inspection result for the job result corresponding to the preset maximum number of job restrictions of the worker 32, and the worker 32 It may be updated by increasing the maximum number of work restrictions of the worker 32 corresponding to the number of first inspection passes among the predetermined number of work performed.

In this case, the predetermined number may be set as a predetermined ratio of the preset maximum number of work restrictions. For example, when the predetermined number is set to 10, which is 20% of the maximum number of work limits, the number of first inspection passes among the 10 most recently performed by the worker 32 is calculated, and if the initial inspection passes is 8, 8 cases The maximum number of work limits can be increased as much as possible.

Accordingly, one embodiment of the present invention does not collectively increase the number of initial inspection, which is 40 cases, based on the total number of pre-set maximum number of work restrictions, which is 50, but the number of cases that have passed the initial inspection among the 10 most recently performed tasks. As the number of persons can be increased by as many as 8, there is an advantage of being able to update by increasing the maximum number of work limits depending on whether the work has been successfully performed recently by reflecting the recentness rather than the update based on the total number of work.

On the other hand, according to the embodiment, the present invention not only increases and updates the maximum number of job restrictions, but also decreases and updates the maximum number of job restrictions.

In another embodiment, the server calculates the number of initial inspection passes and the number of re-examination passes of the operator 32 based on the inspection result for the job result corresponding to the preset maximum number of job restrictions of the operator 32, and It can be updated by increasing the maximum number of work restrictions of the worker 32 corresponding to the ratio of the number of inspection passes and the number of re-review passes.

Here, the number of re-inspection passes refers to the number of cases other than the number of initial inspection passes, such as the number of cases that were rejected as a result of the initial work, and passed as a result of the rework result.

In another embodiment, the server calculates the number of initial inspection passes of the operator 32 based on the inspection result corresponding to the preset maximum number of work restrictions of the operator 32, and limits the preset maximum work of the operator 32 Based on the number of jobs and the number of jobs required, the average job required per job can be calculated. In addition, it is possible to update by increasing the maximum number of work restrictions of the worker 32 corresponding to the number of initial inspection passes and the average work required time.

That is, in an embodiment of the present invention, the number of first inspection passes means that the quality that has never been rejected is high, so in order to further utilize this point, the average work required time is calculated and the maximum work speed is applied to the worker 32 It can further increase the number of work restrictions.

For example, the average time required for each worker 32 may be calculated, sorted in descending order, and then classified by grade. Accordingly, it is possible to update the maximum number of work restrictions by increasing the number of higher-level workers whose average time required for work is shorter than that of other workers by a predetermined ratio than the number of initial inspection passes. For grade workers, the maximum number of work limits can be updated by increasing the number of cases that are a predetermined percentage less than the number of first inspection passes. On the other hand, the maximum number of work limits can be updated by increasing only the number of first inspection passes for ordinary grade workers whose average work time is similar to those of other workers. In this case, it goes without saying that the class to be distinguished may be further subdivided according to the embodiment.

Here, the time required for the task may be calculated by calculating the time when the task is started for each task and the time when the task is completed and the task result is input, and summing the deviations thereof.

At this time, the time when each task begins to perform tasks refers to the time when the online workspace is accessed, and the online workspace refers to a space where online tasks such as a web page or application page, or a specific URL provided by the server can be performed. it means.

Meanwhile, in the above description, steps S11 to S160 may be further divided into additional steps or may be combined into fewer steps according to an embodiment of the present invention. In addition, some steps may be omitted as necessary, and the order between steps may be changed. In addition, even if other contents are omitted, the contents of FIG. 12 to be described later may also be applied to the method of applying the number of work restrictions for each worker 32 of FIGS.

Hereinafter, an apparatus 200 for applying the number of job restrictions per worker 32 (hereinafter, a device for applying the number of job restrictions for each worker) in a crowdsourcing-based project according to an embodiment of the present invention will be described.

12 is a view for explaining an apparatus for applying the number of work restrictions for each worker according to an embodiment of the present invention.

Referring to FIG. 12, the apparatus 200 for applying the number of work restrictions for each worker according to the present invention includes a communication module 210, a memory 220, and a processor 230.

The communication module 210 transmits a plurality of tasks of the project to the plurality of workers 32 to request the task to be performed, and receives the work results from the plurality of workers 32. In addition, a plurality of work results are assigned to a plurality of inspectors 34 to request inspection, and the inspection results are inputted from the plurality of inspectors 34.

In the memory 220, a program for presetting and applying the initial number of limiting jobs and the maximum number of limiting jobs for one worker is stored.

The processor 230 executes a program stored in the memory 220. As the processor 230 executes the program stored in the memory 220, it identifies the functional elements of the project, and based on the history of the worker 32's participation in the previous project, the functional elements in the previous project in which the worker participated. After identification, the average working time for each functional element for the functional element included in the project to be opened among the functional elements included in the previous project is calculated.

And, based on the average working time for each functional element and the number of tasks performed by the worker 32 using each functional element in the previous project, after calculating the project matching value for the worker 32, the project matching value Based on the preset number of the first initial work limit, the project matching value exceeds a predetermined level, the worker 32 presets the second initial limit number of work, which is reduced by a certain percentage from the first initial limit number.

In addition, the processor 230 increases and updates the maximum number of job restrictions for each worker 32 based on the inspection result after the initial number of job restrictions.

The apparatus 200 for applying the number of work restrictions for each worker described with reference to FIG. 12 may be provided as a component of the above-described server.

In the crowdsourcing-based project according to an embodiment of the present invention described above, the method of applying the number of work limits per worker may be implemented as a program (or application) to be executed in combination with a computer, which is hardware, and stored in a medium. .

The above-described program includes C, C++, JAVA, Ruby, which can be read by a processor (CPU) of the computer through the device interface of the computer, in order for the computer to read the program and execute the methods implemented as a program. It may include a code (Code) coded in a computer language such as machine language. Such code may include a functional code related to a function defining necessary functions for executing the methods, and a control code related to an execution procedure necessary for the processor of the computer to execute the functions according to a predetermined procedure. can do. In addition, such code may further include additional information required for the processor of the computer to execute the functions or code related to a memory reference to which location (address address) of the internal or external memory of the computer should be referenced. have. In addition, when the processor of the computer needs to communicate with any other computer or server in the remote in order to execute the functions, the code uses the communication module of the computer to determine how It may further include a communication-related code for whether to communicate or what information or media to transmit and receive during communication.

The stored medium is not a medium that stores data for a short moment, such as a register, cache, memory, etc., but a medium that stores data semi-permanently and can be read by a device. Specifically, examples of the storage medium include, but are not limited to, ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, and the like. That is, the program may be stored in various recording media on various servers to which the computer can access, or on various recording media on the user's computer. In addition, the medium may be distributed over a computer system connected through a network, and computer-readable codes may be stored in a distributed manner.

The above description of the present invention is for illustrative purposes only, and those of ordinary skill in the art to which the present invention pertains will be able to understand that it can be easily modified into other specific forms without changing the technical spirit or essential features of the present invention. will be. Therefore, it should be understood that the embodiments described above are illustrative in all respects and not limiting. For example, each component described as a single type may be implemented in a distributed manner, and similarly, components described as being distributed may also be implemented in a combined form.

The scope of the present invention is indicated by the claims to be described later rather than the detailed description, and all changes or modified forms derived from the meaning and scope of the claims and their equivalent concepts should be interpreted as being included in the scope of the present invention. do.

10: Client
20: service provider
30: public
32: worker
34: inspector
200: Device for applying the number of work restrictions per worker
210: communication module
220: memory
230: processor

Claims (10)

As a method performed by a computer,
Pre-setting the number of initial work restrictions and the maximum number of work restrictions per worker of a crowdsourcing-based project (hereinafter, “project”);
Allocating a plurality of tasks of the project to a plurality of workers and requesting the task to be performed;
Receiving a plurality of work results from the plurality of workers;
Assigning the plurality of work results to a plurality of inspectors and requesting to perform inspection;
Receiving a plurality of inspection results for the plurality of work results from the plurality of inspectors as passing or rejecting the inspection; And
After all the inspections for a plurality of jobs corresponding to the initial number of job restrictions have been passed, based on the inspection results for a plurality of jobs performed between the maximum number of job restrictions and the first number of job restrictions, each Including the step of increasing and updating the maximum number of work restrictions for each worker,
The step of presetting the number of initial work restrictions and the maximum number of work restrictions for one worker of the project,
Identifying functional elements of the project;
Identifying a functional element in the previous project in which the worker participated, based on the history of the worker's participation in the previous project;
Calculating an average working time for each functional element for the functional element included in the project among the functional elements included in the previous project;
Calculating a project matching value, which is a numerical value of the worker's performance in the previous project, based on the average working time for each functional element and the number of tasks performed by the worker using each functional element in the previous project. ; And
Among the plurality of workers, a first initial limit number is preset for workers below a predetermined level according to the project matching value, and decreases by a certain percentage from the first initial limit number for workers exceeding the predetermined level. Pre-setting the number of the second initial operation limit,
Assigning a plurality of tasks of the project to a plurality of workers and requesting to perform the task,
Counting the number of current tasks performed by the worker and determining whether it matches the preset initial task limit number;
Stopping the assignment of additional tasks to the worker when the current number of tasks performed matches the preset number of initial task restrictions;
Calculating the number of inspection passes by the operator based on the inspection result for the job result corresponding to the number of currently performed jobs by the worker; And
If the number of passes of the inspection matches the number of initial work restrictions, resuming the assignment of a plurality of tasks to the worker,
The functional element is determined based on a tool for executing the project,
The work tool is provided in the project and is a tool used by workers to perform the work required by the project,
How to apply the limit of work per worker in crowdsourced projects.
delete The method of claim 1,
Resetting the number of job restrictions corresponding to the number of initial job restrictions, based on the inspection result after the initial number of job restrictions,
Resetting the number of job restrictions corresponding to the number of initial job restrictions,
Calculating a rejection score corresponding to the number of rejected operation results among the inspection results after the initial operation restriction number for the plurality of workers; And
If the rejection score corresponds to a preset condition, resetting the number of job restrictions corresponding to the initial number of job restrictions from the point in time,
The preset condition is, when the rejection score corresponding to the inspection result for each preset period has increased by a certain percentage or more compared to the previous period, and when the rejection score corresponding to the inspection result accumulated after the initial limit number of tasks is a predetermined score or more. Which is any one of,
How to apply the limit of work per worker in crowdsourcing-based projects.
The method of claim 3,
The step of calculating a rejection score corresponding to the number of rejected operation results among the inspection results after the initial operation restriction number for the plurality of workers,
When the number of current tasks performed by the worker exceeds the preset proficiency point quantity, calculating the rejection score by applying a weight based on the average rejection rate of the plurality of workers,
How to apply the limit of work per worker in crowdsourcing-based projects.
The method of claim 4,
The step of calculating a rejection score corresponding to the number of rejected operation results among the inspection results after the initial operation restriction number for the plurality of workers,
If the worker's rejection rate is more than the average rejection rate, a weight of a first value greater than 1 is applied, and when the worker's rejection rate is less than the average rejection rate, a weight of a second value smaller than 1 is applied, and the rejection To calculate the score,
How to apply the limit of work per worker in crowdsourcing-based projects.
The method of claim 3,
Assigning a plurality of tasks of the project to a plurality of workers and requesting to perform the task,
Counting the number of first cumulative tasks performed by the worker from the time when the number of task restrictions is reset to the present time, and determining whether the number of task restrictions matches the reset number of tasks;
Stopping the assignment of additional tasks to the worker when the number of the first cumulative task execution matches the reset task limit number;
Calculating the number of inspection passes of the worker based on the inspection result of the job result corresponding to the number of the first cumulative job execution by the worker; And
When the number of passes of the inspection matches the number of work restrictions, resuming the assignment of a plurality of tasks to the worker,
How to apply the limit of work per worker in crowdsourcing-based projects.
The method of claim 1,
Assigning a plurality of tasks of the project to a plurality of workers and requesting to perform the task,
Calculating a second cumulative number of tasks performed by counting the total number of tasks performed by the operator from the first task limit number to the present;
Determining whether the number of executions of the second cumulative work matches the maximum number of work restrictions; And
If the number of execution of the second cumulative task matches the maximum number of task restrictions, stopping the assignment of additional tasks to the worker,
How to apply the limit of work per worker in crowdsourcing-based projects.
The method of claim 7,
The step of increasing and updating the maximum number of job restrictions for each worker based on the inspection result after the initial number of job restrictions,
Calculating the number of initial inspection passes by the operator based on the inspection result for the job result corresponding to the preset maximum number of job restrictions; And
Comprising the step of increasing and updating the maximum number of work restrictions of the operator in correspondence with the number of passes of the initial inspection,
How to apply the limit of work per worker in crowdsourcing-based projects.
The method of claim 8,
The step of increasing and updating the maximum number of job restrictions for each worker based on the inspection result after the initial number of job restrictions,
Calculating the number of additional tasks performed by counting the total number of additional tasks performed by the worker according to the update of the maximum task limit number;
Calculating the number of initial inspection passes of the operator based on the inspection result for the job result corresponding to the number of additional jobs performed by the operator; And
Comprising the step of re-update by increasing the number of the updated maximum work limit number of the worker corresponding to the number of passing the initial inspection,
How to apply the limit of work per worker in crowdsourcing-based projects.
A computer program combined with a computer and stored in a computer-readable recording medium to execute the method of applying the number of work restrictions per worker in the crowdsourcing-based project of any one of claims 1 and 3 to 9.

Images