KR102195958B1 - Method for applying the maximum number of work limits per workers based on reliability in a multi-assignment crowdsourcing based projects for artificial intelligence training data generation - Google Patents

Abstract

Provided is a method for applying a limit on the maximum number of tasks for each worker based on reliability in a multi-assignment crowdsourcing-based project. The method comprises the steps of: previously setting a limit on the maximum number of tasks for each worker for a multi-assignment project; assigning multiple tasks of the project to multiple workers and requesting the tasks to be performed; receiving multiple task results from the multiple workers; calculating the reliability of each of the multiple workers based on the multiple task results; and updating the limit on the maximum number of tasks for each worker based on the reliability.

Description

METHOD FOR APPLYING THE MAXIMUM NUMBER OF WORK LIMITS PER WORKERS BASED ON RELIABILITY IN A MULTI-ASSIGNMENT CROWDSOURCING BASED in a multi-allocated crowdsourcing-based project for generating artificial intelligence learning data PROJECTS FOR ARTIFICIAL INTELLIGENCE TRAINING DATA GENERATION}

The present invention relates to a method for applying the maximum number of work limits per worker based on reliability in a multi-allocation crowdsourcing-based project.

Recently, more and more companies collect and process large amounts of data on a crowdsourcing basis that engages the general public in some process of corporate activities. In other words, by opening a project and allowing the general public, that is, workers to participate in the project, necessary information is collected through the work results completed by the workers.

At this time, some of the crowdsourcing-based projects collect necessary data through a multi-allocation method in which the same task is duplicated and assigned to multiple workers to perform tasks, and multiple task results for the same task are collected.

In such a multi-allocation project, it is desirable to allocate more tasks to workers with excellent or reliable work ability, since the production of valid data depends only on the work results of the workers.

However, in the current project, the maximum number of tasks that a worker can perform in the project is set to be limited, regardless of the worker. In this case, workers who perform well are not particularly problematic, but workers who do not perform well need to be reduced from the maximum number of work limits.

In a project where an inspection is performed by an inspector, workers who cannot be returned based on the number of rejection of the work result of the worker can be selected, but in a multi-allocation project, there is no way to evaluate the ability of the workers because the inspector does not participate. It is a situation. Therefore, there is a need for a method to select workers with high reliability while performing tasks better and to allocate more tasks to those workers.

Unexamined Patent Publication No. 10-2014-0095956, 2014.08.04.

The problem to be solved by the present invention is to calculate the reliability of workers based on the work results of a plurality of workers, and to the reliability in a crowdsourcing-based project of a multi-allocation method that can adjust the maximum number of work limits of workers based on the reliability. It is to provide a method of applying the maximum number of work limits per worker based on the basis.

However, the problem to be solved by the present invention is not limited to the problem as described above, and other problems may exist.

The method of applying the maximum number of tasks per worker based on reliability in a crowdsourcing-based project of a multi-allocation scheme according to an aspect of the present invention for solving the above-described problem is a multi-allocation scheme in which one task is redundantly assigned to a plurality of workers. Presetting a maximum number of work limits per worker of a crowdsourcing-based project (hereinafter, "project"); Allocating a plurality of tasks of the project to a plurality of workers and requesting the task to be performed; Receiving a plurality of job results from the plurality of workers; Calculating the reliability of each of the plurality of workers based on the plurality of work results; And updating the maximum number of work restrictions for each worker based on the reliability. In this case, calculating the reliability of each of the plurality of workers based on the plurality of work results may include: calculating the number of workers who input work results corresponding to a plurality of opinions among the work results of the plurality of workers; And calculating the reliability of the worker based on the number of inputting work results corresponding to the plurality of opinions relative to the total number of performing the multi-allocated work of the worker, and the reliability of each worker based on the reliability. In the step of updating the maximum number of work restrictions, among the reliability of the plurality of workers divided into a plurality of sections, the maximum number of work restrictions of workers having a reliability less than or equal to the first section is reduced to correspond to the reliability, and the second The maximum number of work restrictions of workers having a reliability exceeding the second section, which is higher than the first section, is increased to correspond to the weighted reliability.

In some embodiments of the present invention, the step of requesting a task by assigning the plurality of tasks to a plurality of workers may include calculating the cumulative number of tasks performed by counting the total number of tasks performed by the worker to date; Determining whether the number of cumulative work executions matches the maximum work limit number; And stopping the assignment of additional tasks to the worker when the cumulative task execution number matches the maximum task limit number.

In some embodiments of the present invention, the step of calculating the reliability of each of the plurality of workers based on the plurality of work results may include, from the total number of the multiple-allocated tasks of the workers, from the work results of the plurality of workers. Results of work that do not have multiple opinions can be excluded.

Some embodiments of the present invention further include the step of determining the type of one or more functional elements included in the project, wherein the step of calculating the reliability of each of the plurality of workers based on the results of the plurality of tasks comprises: Regarding the project of the allocation method, the reliability of each of the plurality of workers is calculated for each type of functional element, the functional element is determined based on a work tool for executing the project, and the work tool is a project It is provided in and may be a tool used to perform the work required by the worker's project.

In some embodiments of the present invention, the step of calculating the reliability of each of the plurality of workers based on the results of the plurality of tasks may include, from the total number of cases of performing the multiple-allocated tasks of the workers including the one or more functional elements, It is possible to exclude a work result in which a plurality of opinions do not occur from the work results of the plurality of workers.

In some embodiments of the present invention, the step of calculating the reliability of each of a plurality of workers based on the results of the plurality of tasks includes N tasks that have been multi-allocated to the plurality of workers and evaluated for multiple opinions (N is Whenever a natural number of 1 or more) is accumulated, the reliability for each worker corresponding to the N tasks may be calculated.

In some embodiments of the present invention, the cumulative number of the N cases may be set to gradually decrease based on the progress of the project.

In some embodiments of the present invention, the updating of the maximum number of work restrictions for each worker based on the reliability may include, in the reliability, the maximum number of work restrictions of workers having a reliability less than or equal to the first section among the plurality of sections. If the number of current tasks performed by the worker is greater than the maximum number of tasks to be reduced, the maximum number of task restrictions may be reduced by the number of current tasks performed.

In some embodiments of the present invention, the updating of the maximum number of work restrictions for each worker based on the reliability may include, among the reliability of the plurality of workers divided into the plurality of sections, the first section is exceeded, and It is possible to maintain the maximum number of work restrictions of workers having a reliability less than or equal to the second section as the previous maximum number of work restrictions.

A computer program according to another aspect of the present invention for solving the above-described problem is combined with a computer as hardware to execute a method of applying the maximum number of work limits per worker based on reliability in the crowdsourcing-based project of the multi-allocation method, It is stored on a computer-readable recording medium.

Other specific details of the present invention are included in the detailed description and drawings.

According to the present invention described above, there is an advantage that it is possible to evaluate the work ability of workers even for a multi-allocation type project, and the maximum number of work limits of workers can be adjusted based on this evaluation.

That is, by adjusting the maximum number of work limits, it is possible to increase the quality and speed of the project by preventing workers with relatively low work capabilities from performing more than the limit number of tasks.

Conversely, for workers with high reliability who perform tasks well, by increasing and updating the maximum number of task restrictions corresponding to the reliability and weight, more tasks can be assigned and performed.

In other words, if workers who perform well are assigned additional tasks, the number of tasks assigned to workers with low reliability is relatively reduced, and the quality and speed of the project can be further improved as it goes to the end of the project. .

In addition, by making the evaluation cycle for reliability more frequent as it goes to the second half of the project, it does not block the influx of new workers even in the early as well as mid-late of the project, and at the same time induces highly reliable workers to perform more tasks can do.

The effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the following description.

1 is a conceptual diagram of a crowdsourcing service according to an embodiment of the present invention.
2 is a flowchart illustrating a crowdsourcing-based project progress process according to an embodiment of the present invention.
3 is a flow chart of a method of applying the maximum number of work restrictions per worker.
4 is an exemplary view for explaining the work of a multi-allocation method project.
5 is a flowchart illustrating a process of requesting a task to be performed by assigning a task to a plurality of workers.
6 is another exemplary diagram for explaining the work of a multi-allocation method project.
7 is a diagram for explaining a method of calculating a worker's reliability according to an embodiment of the present invention.
8 is a diagram showing an example of a work result of a worker who has performed a task of a multi-allocation method project including a tagging function element.
9 is a diagram illustrating an example for explaining the content of updating the maximum number of work restrictions.
10 is a view for explaining an apparatus for applying the maximum number of work restrictions per worker according to an embodiment of the present invention.

Advantages and features of the present invention, and a method of achieving them will become apparent with reference to the embodiments described below in detail together with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but may be implemented in a variety of different forms, only the present embodiments are intended to complete the disclosure of the present invention, It is provided to fully inform the technician of the scope of the present invention, and the present invention is only defined by the scope of the claims.

The terms used in the present specification are for describing exemplary embodiments and are not intended to limit the present invention. In this specification, the singular form also includes the plural form unless specifically stated in the phrase. As used in the specification, “comprises” and/or “comprising” do not exclude the presence or addition of one or more other elements other than the mentioned elements. Throughout the specification, the same reference numerals refer to the same elements, and “and/or” includes each and all combinations of one or more of the mentioned elements. Although "first", "second", and the like are used to describe various elements, it goes without saying that these elements are not limited by these terms. These terms are only used to distinguish one component from another component. Therefore, it goes without saying that the first component mentioned below may be the second component within the technical idea of the present invention.

Unless otherwise defined, all terms (including technical and scientific terms) used in the present specification may be used as meanings that can be commonly understood by those of ordinary skill in the art to which the present invention belongs. In addition, terms defined in a commonly used dictionary are not interpreted ideally or excessively unless explicitly defined specifically.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a conceptual diagram of a crowdsourcing service according to an embodiment of the present invention.

Referring to FIG. 1, a crowdsourcing service is performed by being composed of a client 10, a service provider 20, and a public 30.

The sponsor 10 refers to a company or individual who requests a crowdsourcing-based project (hereinafter, referred to as a project).

The requester 10 requests a project for the purpose of collecting source data or annotating data for generating artificial intelligence learning data. The data generated through the project can be used as learning data for arbitrary machine learning such as supervised learning, unsupervised learning, and reinforcement learning. Collection of source data refers to collecting raw data such as recorded voice collection and photo collection. Data annotation refers to inputting relevant annotation data into source data such as text, photos, and videos. For example, the data annotation may include finding an entity in a given fingerprint or finding a similar sentence, but is not limited thereto. Meanwhile, the types of the above-described projects are only one embodiment, and various projects may be handled in the present invention according to the design of the client.

The service provider 20 refers to a company that provides crowdsourcing services.

When the service provider 20 receives a request for a project for a product or service from the client 10, the service provider 20 allocates a task for the project to the general public 30 and receives the work result from the public 30. Thereafter, the final product extracted based on the work result is provided to the client 10.

At this time, the service provider 20 provides a crowdsourcing service to the client 10 and the public 30 through a crowdsourcing platform (hereinafter, the platform). That is, when the service provider 20 receives a request for a project from the client 10, the service provider 20 opens the project on the platform. Thereafter, when the work result for the open project is received from the public 30, the project is terminated on the platform, and the final product may be extracted and provided to the client 10.

The public 30 refers to the general public participating in the project open on the platform. Here, the public 30 may participate in a project opened on the platform through an application or website provided by the service provider 20.

The public 30 is composed of a worker 32 and an inspector 34.

The worker 32 decides to participate in a specific project among a plurality of projects open on the platform. Thereafter, the operator 32 performs an operation such as collection of source data or data annotation, and transmits it to the platform.

The inspector 34 decides to participate in a specific project among a plurality of projects open on the platform. Thereafter, the inspector 34 inspects the result of the work performed by the operator 32. As a result of performing the inspection, the inspector 34 may perform inspection pass processing or rejection processing, and input a rejection reason during rejection processing. In the case of passing the inspection, it is not necessary to rework and re-examination due to this, so passing inspection has the same meaning as completion of inspection.

2 is a flowchart illustrating a crowdsourcing-based project progress process according to an embodiment of the present invention.

First, the requester 10 requests one or more projects to the service provider 20 (S11).

Thereafter, the service provider 20 opens the requested project on the platform (S12). At this time, the service provider 20 may determine the grade in consideration of the difficulty of the project before opening the project. That is, depending on the difficulty level, it is possible to determine which level or higher public 30 to expose the project. Accordingly, it is possible to increase the reliability of the work result of the project.

Thereafter, the service provider 20 requests the work by assigning the work to the workers 32 of the corresponding level or higher according to the level of the project (S13).

Thereafter, the worker 32 performs the assigned task (S14). In this case, the worker 32 may input the reason for the inability to work without performing the work for a work in which the work itself is impossible for some reason.

Thereafter, the service provider 20 receives the work result from the worker 32 (S15), and assigns the work result to the inspector 34 to request the inspection (S16).

Similarly, according to an embodiment of the present invention, only a suitable project may be exposed to the inspector 34 out of the total projects being performed according to the level set according to the difficulty of the project or the qualification requirements of the inspector 34.

Thereafter, the inspector 34 performs the assigned inspection (S17). At this time, the inspector 34 decides to complete the inspection when it is determined that the work has been properly performed, and if it is determined that the inspection work is wrong, it rejects the inspection. During rejection processing, the inspector 34 inputs the rejection reason for what reason the job was judged to be wrong.

Thereafter, the service provider 20 receives the inspection result from the inspector 34 (S18).

When the inspection result is the inspection completion, the service provider 20 uses the work result as valid data, and based on this, extracts the final product at the end of the project.

When the inspection result is rejection processing, the service provider 20 may internally perform the inspection again, or assign the work to the worker 32 again to perform the rework. When reworking, re-examination by an inspector is required.

Thereafter, when the project period ends or sufficient valid data is secured, the service provider 20 terminates the project (S19), calculates a final result based on the secured valid data, and provides it to the client 10 ( S20).

At this time, before the end of the project, the service provider 20 evaluates the performance results of the worker 32 and the inspector 34, calculates the work cost and the inspection cost according to the evaluation, to the worker 32 and the inspector 34. give.

In FIGS. 1 and 2, it is simply expressed as a requester 10, a service provider 20, an operator 32, and an inspector 34, but these are smartphones, tablets, PDAs, laptops, desktops, operated by each participant. It means a computer device such as a server or a telecommunication device.

Hereinafter, a method of applying the maximum number of work limits per worker 32 based on reliability in a multi-allocation crowdsourcing-based project according to an embodiment of the present invention with reference to FIGS. 3 to 9 (hereinafter, the maximum work limit per worker How to apply the number of cases).

3 is a flow chart of a method of applying the maximum number of work restrictions per worker.

Meanwhile, the steps shown in FIG. 3 may be understood as being performed by a platform server (hereinafter, referred to as a server) operated by the service provider 20, but are not limited thereto.

First, the server presets the maximum number of work limits per worker 32 of a multi-allocation crowdsourcing-based project (hereinafter, referred to as a project) (S110).

4 is an exemplary diagram for explaining the work of a multi-allocation method project.

A project according to an embodiment of the present invention proceeds in a multi-allocation method in which one task is redundantly assigned to a plurality of workers. In this case, the number of workers 32 performing a task for each task is set equally for all tasks of the project. For example, if a specific task is duplicated to three workers, another task is also assigned to three workers.

In addition, in the multi-allocation method, since multiple opinions of the workers 32 are submitted as meaningful results, the number of workers 32 who are assigned the same task for this purpose may be an odd number, but is not limited thereto. In addition, each worker 32 may be randomly assigned to an arbitrary task of the project.

As an example, the multi-allocation project in the present invention may be a project for selecting an answer to a question that requires subjective evaluation.

Referring to FIG. 4, a worker 32 assigned a task for sentence-level emotion labeling should read the text of a presented sentence and select an answer by evaluating the atmosphere felt. At this time, the plurality of workers 32 who have been assigned the task in duplicate will select whether the atmosphere of the text is negative, neutral, or positive based on each subjective judgment.

There is no fixed answer for the type of work presented in FIG. 4. That is, this is because the multi-allocation task requires that a specific answer be selected as a task result based on the subjective judgment of the individual performing the task. At this time, in an embodiment of the present invention, only when an answer corresponding to a plurality of opinions selected by a plurality of workers 32 who have multiple assignments of work occurs, the answer is classified as a valid work result, and the project sponsor ( 10).

In other words, in a multi-allocation project in which an answer to a question that requires subjective evaluation is selected, it is not the correct answer to the answer selected by an individual worker, but the occurrence of an answer corresponding to the opinions selected by multiple workers is a valid task. Treated as a result. Due to this characteristic, in a multi-allocation project, the inspector 34 in charge of inspection may be excluded from the project.

On the other hand, in an embodiment of the present invention, the maximum number of work limits is preset for each worker. Here, the maximum number of work limits means the number of work that each worker 32 can perform to the maximum within one project. For example, if the preset maximum number of work limits is 50, in principle, each worker 32 cannot perform more than 50 work.

However, not all workers 32 are assigned 50 tasks equally and at the same time. In other words, when the first task is completed, the workers 32 request and perform the next task, and when the second task is completed, the next task is requested and performed. Therefore, the task is performed and assigned according to each situation. During the same period, one worker may reach 50, while another worker may be assigned only 10 or 30 tasks during the same period to complete.

On the other hand, one project may contain a plurality of jobs, and the plurality of jobs may be grouped into a plurality of tasks according to the type, or each job may be a different type of job or the same job. It is not limited thereto.

In addition, the plurality of workers 32 perform work using a predetermined terminal device. The terminal device of the worker 32 may be a computer device or a telecommunication device such as a smart phone, tablet, PDA, laptop, desktop, etc., but is not limited thereto.

Next, the server assigns a plurality of tasks of the project to the plurality of workers 32 to request the task to be performed (S120), and receives a plurality of task results from the plurality of workers 32 (S130).

5 is a flowchart illustrating a process of requesting a task to be performed by assigning a task to a plurality of workers 32.

According to an embodiment of the present invention, when requesting a task by assigning a plurality of tasks of a project to the worker 32, the total number of tasks performed by the worker 32 to date is counted to calculate the cumulative number of tasks performed. (S121), it is determined whether the number of cumulative work executions matches the maximum number of work executions (S122), and if they do match, the assignment of additional tasks to the worker 32 is stopped (S123).

That is, when the maximum number of work limits is 50, the server counts and calculates the total number of tasks performed by the worker 32 so far. When the cumulative number of tasks reaches 50, the server adds additional Stop assigning tasks.

In other words, the worker 32 can only be assigned a maximum of 50 tasks for one project, and it is not possible to assign additional tasks any more.

Next, the server calculates the reliability of each of the plurality of workers 32 based on the results of the plurality of jobs (S140).

In an embodiment of the present invention, reliability is how much effective data was produced as a result of the work by participating in a multi-allocation project in which the worker 32 selects an answer to a question that requires subjective evaluation. It is an indicator for evaluation. That is, the reliability has a different meaning from the rate of correct answers that evaluates whether the worker 32 has selected the correct answer or wrote the correct answer.

The reliability of the operator 32 may be calculated as follows.

The server calculates the number of the workers 32 who have entered the work results corresponding to multiple opinions among the work results of the multiple workers 32, and multi-allocated jobs of the workers 32. The reliability of the worker 32 is calculated based on the number of inputs of work results corresponding to multiple opinions compared to the total number of performing operations.

6 is another exemplary diagram for explaining the work of a multi-allocation method project. 7 is a diagram for explaining a method of calculating a worker's reliability according to an embodiment of the present invention.

For example, workers 32 respond to a given conversation based on their own subjective judgment as they are assigned a task as shown in FIG. 6. In this case, the workers 32 who are assigned the multi-allocation method as shown in FIG. 6 will select'O' or'X' after evaluating whether B's answer is an unpleasant answer according to each subjective judgment.

Referring to FIG. 7, five workers participated in a project performing such a multi-allocation method, and in this case, the multi-allocation method is that three workers are assigned the same task to perform.

Workers who performed the first task'Problem 1'were Worker 1, Worker 2, and Worker 3. At this time, among the answers to'Question 1', the worker who selected the answer O is worker 1 and the worker 3, and the worker who selected the answer X is one worker 2.

Therefore, the answer corresponding to the majority opinion is the O answer, and when calculating the reliability of each worker 1 and worker 3 who entered the O answer, the work result is included in the number of inputted work results corresponding to the majority opinion.

Of the five tasks, the total number of cases performed by worker 1 is 3 (Problem 1 to Problem 3). And the number of worker 1 who entered the work result corresponding to the majority opinion is 2 cases (problem 1, problem 2). Therefore, the reliability of worker 1 is calculated to be about 66.67%.

If the reliability of other workers is calculated in this way, it is calculated that worker 2 is about 33.33%, worker 3 and worker 5 are about 66.67%, and worker 4 is 100%.

On the other hand, according to an embodiment of the present invention, in the total number of tasks performed by the workers 32 that are multi-allocated, work results in which multiple opinions do not occur may be excluded from the work results of the plurality of workers 32.

Specifically, if an answer corresponding to multiple opinions among answers selected by the operator 32 assigned to the multi-allocated task does not occur, the task result for the task is not classified as valid data. Therefore, the task is not summed up when calculating the total number of tasks performed by the worker 32 multi-allocated. In addition, tasks for which no answer corresponding to the majority opinion has occurred are reassigned to the other plurality of workers 32.

For example, if a multi-allocation task for performing sentence-level emotion labeling in FIG. 4 was assigned to each of three workers, and three workers each selected'negative','neutral', and'positive' answers, multiple There is no response corresponding to the opinion. Therefore, the work is not included in the total number of cases for calculating the reliability of each of the three workers.

Meanwhile, in an embodiment of the present invention, the reliability of the worker 32 may be calculated according to the type of functional element included in the project. This means that each worker 32 has a preferred or familiar type of functional element, and when the workers 32 participate in a project that includes the type of functional element, the ability of the workers 32 can be better exhibited, and the work result. This is because the quality of the product is also excellent.

FIG. 8 is a diagram showing an example of a work result of a worker 32 who has performed a work of a multi-allocation method project including a tagging function element.

The server may determine the types of one or more functional elements included in the project, and may calculate the reliability for each of the plurality of workers 32 for each type of functional element in relation to a multi-allocation project.

In this case, the functional element is determined based on a work tool for executing the project, and the work tool is provided in the project and may be a tool used by a worker to perform a task required by the project.

For example, if you need a bounding box tool for bounding work, an input box tool for transfer work, and a make step tool for sorting work to perform a project , The project can be classified as including a bounding functional element, a transcription functional element, and a classification functional element.

The multi-allocation task shown in FIG. 8 is a task including a tagging function element for selecting text judged as a noun phrase from a sentence presented based on the subjective judgment of the worker 32. Since such tagging tasks also require subjective judgment of the workers 32, it is possible to redundantly allocate to a plurality of workers in a multi-allocation method.

In order to perform a task including a tagging function element, the workers 32 may use a tagging box tool. At this time, the functional element included in the project based on the tagging box tool is determined as a tagging functional element.

Even in an embodiment that considers these functional elements, when calculating the reliability of each of the plurality of workers 32, from the total number of performing multiple-allocated tasks of the workers 32 including one or more functional elements, a plurality of workers 32 ), work results that do not have multiple opinions can be excluded.

For example, the work results of each of the three workers who performed the work shown in FIG. 8 are all different. In the given sentence, worker 1 tagged'night' and'chaeyeon' as noun phrases, worker 2 tagged'night','chaeyeon', and'jayagi' as noun phrases, and worker 3 tagged'night' as a noun phrase I did.

At this time, whether or not a work result corresponding to the majority opinion is generated should be determined based on the overall work result for the corresponding work as whether or not there is a work result in which the opinions of the workers 32 coincide. According to this, all three workers tagged'night' as a noun phrase, but the opinions of the three workers do not agree on the texts'Chaeyeon' and'Jayagi', so the work result corresponding to the majority opinion does not occur. Did. Therefore, the work results including the corresponding functional element are excluded from the total number of cases for calculating the reliability of the worker 32.

On the other hand, when there are multiple types of functional elements of the project, reliability can be calculated for each type of the plurality of functional elements. For example, if the functional elements included in one job are tagging and bounding functional elements, the corresponding job results are included when calculating the number of job results for calculating the reliability of each of the tagging functional elements and the bounding functional elements.

Referring back to FIG. 3, as the reliability of the worker 32 is calculated as described above, the server updates the maximum number of work restrictions for each worker 32 based on the reliability (S150).

9 is a view showing an example for explaining the content of updating the maximum number of work restrictions.

The server divides each reliability of the plurality of workers 32 into a plurality of sections. In addition, the maximum number of work restrictions of workers having a reliability less than or equal to the first section is reduced to correspond to the reliability.

In contrast, the server increases the maximum number of work restrictions of the worker 32 having a reliability exceeding the second interval higher than the first interval to correspond to the weighted reliability.

For example, when the reference value for separating the first section and the second section is 70%, the server may classify the reliability into a first section having a reliability of 70% or less and a second section exceeding 70%.

Further, the reference values for setting the first section and the second section may be different from each other. For example, a reference value for setting the first section may be 40%, and a reference value for setting the second section may be 70%. Accordingly, the server may classify the reliability into a first interval having a reliability of 40% or less, a second interval exceeding 70%, and a third interval exceeding 40% and less than 70%. In this case, the server may maintain the maximum number of job restrictions of the worker 32 having the reliability of the third section, which is an intermediate grade, as the previous maximum number of job restrictions.

For example, if the maximum number of work limits for the plurality of workers 32 is set to 50, the worker A can no longer allocate additional work when the maximum work limit of 50 is reached.

At this time, when the calculated reliability for worker A is 80% and exceeds the second section (70%~), which is higher than the first section (0~70%), the weight set in advance to 80% of the reliability of worker A is 50 By adding %, you can increase the maximum number of work limits to 65, which is 130% of the total.

Reliability for such workers 32 is multi-allocated to a number of workers 32 and corresponds to N tasks whenever N tasks (N is a natural number of 1 or more) that have been evaluated for multiple opinions are accumulated. It is possible to calculate the reliability for each worker 32. At this time, the cumulative number of N cases is set to gradually decrease based on the progress of the project, and the N data used in the previous reliability evaluation are not included in the next reliability calculation target, N cases.

For example, in FIG. 9, the first N cases were set to 10, and worker A currently performed 15 jobs. Of the 15 work results of worker A, a number of workers submitted work results, and as a total of 10 jobs that had been evaluated for multiple opinions were accumulated, the reliability of 10 cases was calculated as 80%, and the maximum number of jobs was restricted. It increased to a total of 65 cases.

After that, as the project progressed, N cases for worker A were set to decrease from 10 cases to 8 cases. Worker A has performed 25 tasks to date, and among them, a total of 18 tasks that have been evaluated for multiple opinions have been accumulated, and the server calculated the reliability of worker A for eight tasks excluding the ten tasks used in calculating the previous reliability. Recalculate. If the reliability of worker A's current cumulative number of 8 cases is rapidly lowered and is calculated as 30%, the maximum number of work restrictions of worker A is set to be reduced to about 20 cases, 30% of 65 cases.

At this time, the server reduces the maximum number of work restrictions of the worker 32 having a reliability less than or equal to the first section among the plurality of sections to correspond to the reliability, but the number of current tasks performed by the worker 32 is less than the maximum number of work restrictions to be hidden. If there are more, the maximum number of work restrictions may be reduced by the number of current work performed by the worker 32.

That is, in the example of FIG. 9, as the reliability of worker A is calculated as 30%, the maximum number of work limits should be reduced to 20, but because worker A has already performed 25 tasks, the maximum number of work limits for worker A Is set to 25.

On the other hand, each worker 32 has different work execution times and capabilities, and the cumulative number of tasks that have been evaluated for multiple opinions by the other workers 32 are also calculated differently, so that each worker 32 ), N cases for calculating reliability may be different from each other. That is, N cases are currently being applied to 8 cases for worker A, while N cases are currently applied to 4 cases for worker B who has completed more work than worker A.

However, since the cumulative number of N cases is set based on the progress of the project, worker B has reduced N cases in the order of 8, 6, and 4, whereas worker A has N cases according to the current project progress rate. It will be set to 4 immediately.

This process of updating the maximum number of tasks can be repeated until all tasks in the project have been assigned.

Meanwhile, in the above description, steps S11 to S150 may be further divided into additional steps or may be combined into fewer steps, according to an embodiment of the present invention. In addition, some steps may be omitted as necessary, and the order between steps may be changed. In addition, even if other contents are omitted, the contents of FIG. 10 to be described later may also be applied to the method of applying the maximum number of work restrictions per worker 32 of FIGS. 1 to 9.

Hereinafter, a device for applying the maximum number of work restrictions per worker 32 based on reliability in a multi-allocation type crowdsourcing-based project according to an embodiment of the present invention (200, hereinafter, a device for applying the maximum number of work limits per worker) Do it.

10 is a view for explaining an apparatus 200 for applying the maximum number of work restrictions per worker according to an embodiment of the present invention.

Referring to FIG. 10, the apparatus 200 for applying the maximum number of tasks for each worker according to the present invention includes a communication module 210, a memory 220, and a processor 230.

The communication module 210 transmits a multi-allocation method to a plurality of workers 32 to request a task to be performed, and receives a work result from the plurality of workers 32.

The memory 220 stores a program for updating the maximum number of work limits per worker 32 of the project based on the reliability of each worker.

The processor 230 executes a program stored in the memory 220. As the processor 230 executes the program stored in the memory 220, the number of workers who input the work results corresponding to the plurality of opinions among the work results of the plurality of workers 32 is calculated, and The reliability of the worker 32 is calculated based on the number of inputting work results corresponding to multiple opinions compared to the total number of multi-allocated tasks.

And, the processor 230 reduces the maximum number of work restrictions of workers having a reliability less than or equal to the first section among the reliability of the plurality of workers 32 divided into a plurality of sections to correspond to the reliability, and The maximum number of work restrictions of the worker 32 having a reliability exceeding the high second interval is increased to correspond to the weighted reliability.

The apparatus 200 for applying the maximum number of work limits for each worker described with reference to FIG. 10 may be provided as a component of the above-described server.

The method for applying the maximum number of work limits per worker based on reliability in a multi-allocation crowdsourcing-based project according to an embodiment of the present invention described above is a program (or application) to be executed in combination with a computer that is hardware. It can be implemented and stored in a medium.

The above-described program includes C, C++, JAVA, Ruby, which can be read by a processor (CPU) of the computer through the device interface of the computer, in order for the computer to read the program and execute the methods implemented as a program. It may include a code (Code) coded in a computer language such as machine language. Such code may include a functional code related to a function defining necessary functions for executing the methods, and a control code related to an execution procedure necessary for the processor of the computer to execute the functions according to a predetermined procedure. can do. In addition, such code may further include additional information required for the processor of the computer to execute the functions or code related to a memory reference to which location (address address) of the internal or external memory of the computer should be referenced. have. In addition, when the processor of the computer needs to communicate with any other computer or server in the remote in order to execute the functions, the code uses the communication module of the computer to determine how It may further include a communication-related code for whether to communicate or what information or media to transmit and receive during communication.

The stored medium is not a medium that stores data for a short moment, such as a register, cache, memory, etc., but a medium that stores data semi-permanently and can be read by a device. Specifically, examples of the storage medium include, but are not limited to, ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, and the like. That is, the program may be stored in various recording media on various servers to which the computer can access, or on various recording media on the user's computer. In addition, the medium may be distributed over a computer system connected through a network, and computer-readable codes may be stored in a distributed manner.

The above description of the present invention is for illustrative purposes only, and those of ordinary skill in the art to which the present invention pertains will be able to understand that it can be easily modified into other specific forms without changing the technical spirit or essential features of the present invention. will be. Therefore, it should be understood that the embodiments described above are illustrative in all respects and not limiting. For example, each component described as a single type may be implemented in a distributed manner, and similarly, components described as being distributed may also be implemented in a combined form.

The scope of the present invention is indicated by the claims to be described later rather than the detailed description, and all changes or modified forms derived from the meaning and scope of the claims and their equivalent concepts should be interpreted as being included in the scope of the present invention. do.

10: Client
20: service provider
30: public
32: worker
34: inspector
200: Device for applying the maximum number of work limit per worker
210: communication module
220: memory
230: processor

Claims (10)

As a method performed by a computer,
Pre-setting the maximum number of tasks per worker for a crowdsourcing-based project (hereinafter, referred to as “project”) of a multi-allocation scheme in which one task is duplicated to a plurality of workers;
Allocating a plurality of tasks of the project to a plurality of workers and requesting the task to be performed;
Receiving a plurality of work results from the plurality of workers;
Calculating the reliability of each of the plurality of workers based on the plurality of work results; And
Including the step of updating the maximum number of work restrictions for each worker based on the reliability,
Calculating the reliability of each of the plurality of workers based on the plurality of work results,
Calculating the number of workers who input work results corresponding to multiple opinions among the work results of the plurality of workers; And
Comprising the step of calculating the reliability of the worker based on the number of inputting work results corresponding to the plurality of opinions relative to the total number of performing the multi-allocated work of the worker,
In the total number of cases in which the multi-allocated work of the worker is performed, work results in which multiple opinions do not occur from the work results of the plurality of workers are excluded,
The step of updating the maximum number of work restrictions for each worker based on the reliability,
Among the reliability of the plurality of workers divided into a plurality of sections, the maximum number of work restrictions of workers having a reliability less than or equal to the first section is reduced to correspond to the reliability, and exceeds the second section higher than the first section. To increase the maximum number of work restrictions of workers with reliability to correspond to the weighted reliability,
How to apply the maximum number of tasks per worker based on reliability in a multi-allocation crowdsourcing-based project.
The method of claim 1,
Assigning the plurality of tasks to a plurality of workers and requesting the task to be performed,
Calculating a cumulative number of tasks performed by counting the total number of tasks performed by the operator to date;
Determining whether the number of cumulative work executions matches the maximum work limit number; And
If the cumulative number of job executions matches the maximum number of job restrictions, stopping the assignment of additional jobs to the worker,
How to apply the maximum number of tasks per worker based on reliability in a multi-allocation crowdsourcing-based project.
delete The method of claim 1,
Further comprising determining the type of one or more functional elements included in the project,
Calculating the reliability of each of the plurality of workers based on the plurality of work results,
In relation to the project of the multi-allocation method, the reliability of each of the plurality of workers is calculated for each type of functional element,
The functional element is determined based on a work tool for performing a project, and the work tool is provided in a project and is a tool used by a worker to perform a task required by the project,
How to apply the maximum number of tasks per worker based on reliability in a multi-allocation crowdsourcing-based project.
The method of claim 4,
Calculating the reliability of each of the plurality of workers based on the plurality of work results,
In the total number of cases of performing the multi-allocated tasks of the workers including the one or more functional elements, the work results in which multiple opinions do not occur from the work results of the plurality of workers are excluded,
How to apply the maximum number of tasks per worker based on reliability in a multi-allocation crowdsourcing-based project.
The method of claim 1,
Calculating the reliability of each of the plurality of workers based on the plurality of work results,
Whenever N tasks (N is a natural number of 1 or more) that have been multi-allocated to the plurality of workers and evaluated for multiple opinions are accumulated, the reliability for each worker corresponding to the N tasks is calculated,
How to apply the maximum number of tasks per worker based on reliability in a multi-allocation crowdsourcing-based project.
The method of claim 6,
The cumulative number of N cases is set to gradually decrease based on the progress of the project,
How to apply the maximum number of tasks per worker based on reliability in a multi-allocation crowdsourcing-based project.
The method of claim 1,
The step of updating the maximum number of work restrictions for each worker based on the reliability,
Reducing the maximum number of work restrictions of workers having a reliability equal to or less than the first section among the plurality of sections to correspond to the reliability,
When the number of current tasks performed by the worker is greater than the maximum number of tasks to be reduced, reducing the maximum number of tasks limited by the number of current tasks performed,
How to apply the maximum number of tasks per worker based on reliability in a multi-allocation crowdsourcing-based project.
The method of claim 1,
The step of updating the maximum number of work restrictions for each worker based on the reliability,
Of the reliability of the plurality of workers divided into the plurality of sections, maintaining the maximum number of work restrictions of workers having a reliability of exceeding the first section and lower than the second section as the previous maximum number of work restrictions,
How to apply the maximum number of tasks per worker based on reliability in a multi-allocation crowdsourcing-based project.
In order to execute the method of applying the maximum number of work limits per worker based on the reliability in the crowdsourcing-based project of any one of paragraphs 1, 2 and 4 to 9 in combination with a computer, A computer program stored on a computer-readable recording medium.

Images