KR102244697B1 - Project curation method considering worker’s tendency of crowdsourcing based projects for artificial intelligence training data generation - Google Patents

Abstract

The present invention relates to a project curation method considering a tendency of a worker of a crowdsourcing-based project for artificial intelligence training data generation. The method includes: a step of requesting work execution by assigning works of a crowdsourcing-based project (hereinafter, project) to workers; a step of receiving the input of work results from the workers; a step of requesting inspection execution by assigning the work results to inspectors; a step of receiving the input of inspection results regarding the work results as inspection pass or return from the inspectors; a step of extracting a project attribute of the project; a step of classifying the project as a heavy project or a light project based on the project attribute of the project; and a step of curating the project to a heavy worker group corresponding to the heavy project or a light worker group corresponding to the light project based on the classification result.

Description

Project curation method considering worker disposition of crowdsourcing-based project for generating artificial intelligence learning data {PROJECT CURATION METHOD CONSIDERING WORKER'S TENDENCY OF CROWDSOURCING BASED PROJECTS FOR ARTIFICIAL INTELLIGENCE TRAINING DATA GENERATION}

The present invention relates to a project curation method in consideration of worker propensity for crowdsourcing-based projects for generating artificial intelligence learning data.

Recently, more and more companies collect and process large amounts of data on a crowdsourcing basis that engages the general public in some process of corporate activities. In other words, by opening a project and allowing the general public, that is, workers to participate in the project, necessary information is collected through the results of work completed by the workers.

When one project is opened, multiple tasks are assigned to each of the multiple workers. Each worker performs a plurality of tasks assigned to it, and provides the task results.

At this time, if a worker with little work experience or a new worker participates in a project with a high level of difficulty to perform the task, the worker feels difficult to perform the task and has a bad impression on the crowdsourcing platform. This may result in a decrease in the worker's intention to revisit the crowdsourcing platform.

In addition, if a worker with a small number of visits to the crowdsourcing platform participates in a project with a large number of remaining work or a short deadline, it is difficult to expect the appropriate work speed of the worker. Work bottlenecks may also be caused. Eventually, this may lead to delays in the project.

Therefore, it is necessary to induce appropriate workers to participate in the project by curating the project to workers with a corresponding propensity in consideration of the properties of the project.

Unexamined Patent Publication No. 10-2014-0095956, 2014.08.04.

The problem to be solved by the present invention is to provide a project curation method in consideration of the worker tendencies of crowdsourcing-based projects for generating artificial intelligence learning data.

The problems to be solved by the present invention are not limited to the problems mentioned above, and other problems that are not mentioned will be clearly understood by those skilled in the art from the following description.

The project curation method in consideration of the worker tendency of the crowdsourcing-based project according to one aspect of the present invention for solving the above-described problems is a method performed by a computer, and includes a plurality of crowdsourcing-based projects (hereinafter, "projects"). Assigning a task to a plurality of workers and requesting to perform a task, receiving a plurality of work results from the plurality of workers, assigning the plurality of work results to a plurality of inspectors and requesting to perform inspection, the plurality of Receiving a plurality of inspection results for the plurality of work results as inspection pass or rejection from an inspector of, extracting the project properties of the project, and converting the project to a heavy project or a light project based on the project properties of the project And curating the project to a heavy worker group corresponding to the heavy project or a light worker group corresponding to the light project based on the classification result, and extracting a project property of the project. The steps include calculating an average time required to perform a task per one task of the project, calculating the number of remaining tasks in the project, and an average task execution time per task as a project property of the project, and And extracting the number of remaining jobs.

In some embodiments of the present invention, the method comprises a plurality of workers of a crowdsourcing platform (hereinafter, "platform"), based on the average platform stay period and the average platform revisit period of each worker, the heavy worker group or the It may further include the step of classifying the light worker group.

In some embodiments of the present invention, the step of classifying the plurality of workers of the platform into the heavy worker group or the light worker group includes: Further based on time, it may be classified into the heavy worker group or the light worker group.

In some embodiments of the present invention, the step of calculating the number of remaining tasks of the project includes calculating the number of remaining tasks of the project and the number of tasks completed during the previous unit period, and the one The step of extracting the average task execution time per task and the number of remaining tasks may include, as a project property of the project, the average task execution time per task and the task that has been completed during the previous unit period compared to the remaining number of tasks. The number of cases can be extracted.

In some embodiments of the present invention, the step of extracting the project attribute of the project of the method further comprises extracting the rejection rate of the project, and the average task per task is performed as the project attribute of the project. In the step of extracting the required time and the number of remaining tasks, the average task execution time per one task, the remaining number of tasks, and the rejection rate may be extracted as project properties of the project.

In some embodiments of the present invention, the step of extracting the project property of the project of the method further comprises extracting the expected work revenue per unit period of the project, and the one work as the project property of the project In the step of extracting the average task execution time per task and the number of remaining tasks, the average task execution time per task, the remaining task count, and the estimated work revenue per unit period can be extracted as project properties of the project. have.

In some embodiments of the present invention, the step of extracting the project property of the project of the method further comprises extracting the rework limit time of the project, and the average per task as the project property of the project In the step of extracting the time required to perform a task and the number of remaining tasks, the average time required to perform a task per task, the number of remaining tasks, and the limit time for rework may be extracted as project properties of the project.

In some embodiments of the present invention, the method comprises the step of extracting the functional element of the project and the work device corresponding to the functional element of the project among a plurality of workers of the platform, based on the holding work device input by each worker. It further comprises the step of curating the project to the worker who has the project, the functional element is determined based on a work tool (Tool) for performing the project, the work tool is provided in the project, and the workers request the project It may be a tool used to perform a task to be performed.

In some embodiments of the present invention, the method may further include providing incentive points to workers who participated in the curated project and performed tasks for a predetermined period of time.

A computer program according to another aspect of the present invention for solving the above-described problem executes a project curation method in consideration of the worker tendency of the crowdsourcing-based project by combining with a computer as hardware, and stored in a computer-readable recording medium. I can.

Other specific details of the present invention are included in the detailed description and drawings.

According to the present invention, among a plurality of open and ongoing projects, a project with high difficulty that takes a long time to perform a task or a project with a large number of remaining tasks is classified as a heavy project, and the corresponding project of the heavy worker. Can induce the participation of people. Therefore, it is possible to increase the quality of the work result of the project, and to prevent the delay of the project so that the project can be completed by the deadline.

On the other hand, by classifying a project with easy difficulty or a project with a small number of remaining work as a light project and encouraging the corresponding light worker to participate in the project, it leaves a good impression on the crowdsourcing-based project on the light worker, and ultimately It is possible to prevent light workers from leaving the project and induce revisiting of the platform.

In other words, by curating an appropriate project for each worker, taking into account the propensity of the workers of the crowdsourcing platform, appropriate workers are involved according to the difficulty of the project so that the client who requested the project can receive a quality final product. , By instilling a good impression on the crowdsourcing platform to workers, it is possible to encourage workers to visit the platform continuously and participate in the project.

The effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the following description.

1 is a conceptual diagram of a crowdsourcing service according to an embodiment of the present invention.
2 is a flowchart illustrating a crowdsourcing-based project progress process according to an embodiment of the present invention.
FIG. 3 is a flowchart schematically illustrating a project curation method in consideration of worker propensity for a crowdsourcing-based project according to an embodiment of the present invention.
4 is a flowchart schematically illustrating a method of extracting an attribute of a crowdsourcing-based project according to an embodiment of the present invention.
FIG. 5 is a diagram for describing a method of calculating an average time required to perform a task per task of a project.
6 is a diagram for explaining a method of calculating a period of stay and a revisit period of a worker in a crowdsourcing platform.
FIG. 7 is a diagram for explaining a method of calculating a worker's stay period and revisit period when a predetermined reference value for determining the stay period of the crowdsourcing platform is set.
8 is a diagram showing an average platform stay period and an average platform revisit period of a plurality of operators of the platform.
9 is a diagram illustrating a method of classifying a worker into a heavy worker group or a light worker group based on the average platform stay period compared to the average platform revisit period of the worker.
10 is a diagram for explaining classifying all workers into a heavy worker group or a light worker group.
11 is a diagram for explaining classifying some of all workers into a heavy worker group or a light worker group.
12 is an exemplary diagram illustrating a method of curating a heavy project to a heavy worker group using a recommended badge according to an embodiment of the present invention.
13 is an exemplary view showing a method for curating a heavy worker group by dividing a heavy project into separate recommended tasks according to an embodiment of the present invention.
14 is a diagram illustrating an example of a work screen of a project including a bounding functional element.
15 is a diagram for explaining a project curation apparatus in consideration of worker propensity for a crowdsourcing-based project according to an embodiment of the present invention.

Advantages and features of the present invention, and a method of achieving them will become apparent with reference to the embodiments described below in detail together with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but may be implemented in various different forms, and only these embodiments are intended to complete the disclosure of the present invention, and to those skilled in the art to which the present invention pertains. It is provided to fully inform the scope of the present invention, and the present invention is only defined by the scope of the claims.

The terms used in the present specification are for describing exemplary embodiments and are not intended to limit the present invention. In this specification, the singular form also includes the plural form unless specifically stated in the phrase. As used herein, “comprises” and/or “comprising” do not exclude the presence or addition of one or more other elements other than the mentioned elements. Throughout the specification, the same reference numerals refer to the same elements, and "and/or" includes each and all combinations of one or more of the mentioned elements. Although "first", "second", and the like are used to describe various elements, it goes without saying that these elements are not limited by these terms. These terms are only used to distinguish one component from another component. Therefore, it goes without saying that the first component mentioned below may be the second component within the technical idea of the present invention.

Unless otherwise defined, all terms (including technical and scientific terms) used in the present specification may be used with meanings that can be commonly understood by those of ordinary skill in the art to which the present invention belongs. In addition, terms defined in a commonly used dictionary are not interpreted ideally or excessively unless explicitly defined specifically.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a conceptual diagram of a crowdsourcing service according to an embodiment of the present invention.

Referring to FIG. 1, the crowdsourcing service is performed by being composed of a client 10, a service provider 20, and a public 30.

The sponsor 10 refers to a company or individual who requests a crowdsourcing-based project (hereinafter, referred to as a project).

The client 10 requests a project for the purpose of collecting source data or annotating data for generating artificial intelligence learning data. The data generated through the project can be used as learning data for arbitrary machine learning such as supervised learning, unsupervised learning, and reinforcement learning. Collection of source data means collecting raw data such as recorded voice collection and photo collection. Data annotation refers to inputting relevant annotation data into source data such as text, photos, and videos. For example, the data annotation may include finding an entity in a given fingerprint or finding a similar sentence, but is not limited thereto. Meanwhile, the types of the above-described projects are only one embodiment, and various projects may be handled in the present invention according to the design of the client.

The service provider 20 refers to a company that provides crowdsourcing services.

When the service provider 20 receives a request for a project for a product or service from the client 10, the service provider 20 allocates a task for the project to the general public 30 and receives the work result from the public 30. Thereafter, the final product extracted based on the work result is provided to the requester 10.

At this time, the service provider 20 provides a crowdsourcing service to the client 10 and the public 30 through a crowdsourcing platform (hereinafter, the platform). That is, when the service provider 20 receives a request for a project from the client 10, the service provider 20 opens the project on the platform. Thereafter, when the work result for the open project is provided by the public 30, the project is terminated on the platform, and the final product may be extracted and provided to the client 10.

The public 30 refers to the general public participating in the project open on the platform. Here, the public 30 may participate in a project opened on the platform through an application or website provided by the service provider 20.

The public 30 is composed of a worker 32 and an inspector 34.

The worker 32 decides to participate in a specific project among a plurality of projects open on the platform. Thereafter, the operator 32 performs an operation such as collection of source data or data annotation, and transmits it to the platform.

The inspector 34 decides to participate in a specific project among a plurality of projects opened on the platform. Thereafter, the inspector 34 inspects the result of the work performed by the operator 32. As a result of performing the inspection, the inspector 34 may perform inspection pass processing or rejection processing, and input a rejection reason during rejection processing. In the case of passing the inspection, rework and the resulting re-inspection are not required, so passing the inspection has the same meaning as the completion of the inspection.

2 is a flowchart illustrating a crowdsourcing-based project progress process according to an embodiment of the present invention.

First, the client 10 requests one or more projects to the service provider 20 (S11).

Thereafter, the service provider 20 opens the requested project on the platform (S12). At this time, the service provider 20 may determine the grade in consideration of the difficulty of the project before opening the project. That is, depending on the difficulty level, it is possible to determine which level or higher public 30 to expose the project. Accordingly, it is possible to increase the reliability of the work result of the project.

Thereafter, the service provider 20 requests the work by assigning the work to the workers 32 of the corresponding level or higher according to the class of the project (S13).

Thereafter, the worker 32 performs the assigned task (S14). In this case, the worker 32 may input the reason for the inability to work without performing the work for a work in which the work itself is impossible for some reason.

Thereafter, the service provider 20 receives the work result from the worker 32 (S15), and assigns the work result to the inspector 34 to request the inspection (S16).

Thereafter, the inspector 34 performs the assigned inspection (S17). At this time, the inspector 34 decides to pass the inspection when it is determined that the work has been properly performed, and if it is determined that the inspection work is wrong, the inspector 34 rejects it. During rejection processing, the inspector 34 inputs the rejection reason for what reason the job was judged to be wrong.

Thereafter, the service provider 20 receives the inspection result from the inspector 34 (S18).

If the inspection result passes the inspection, the service provider 20 uses the work result as valid data, and based on this, extracts the final product at the end of the project.

If the result of the inspection is rejection processing, the service provider 20 may internally perform the inspection again, or assign the work to the worker 32 again to perform the rework. When reworking, re-examination by an inspector is required.

Thereafter, when the project period ends or sufficient valid data is secured, the service provider 20 terminates the project (S19), calculates a final result based on the secured valid data, and provides it to the client 10 ( S20).

At this time, before the end of the project, the service provider 20 evaluates the performance results of the worker 32 and the inspector 34, calculates the work cost and the inspection cost according to the evaluation, to the worker 32 and the inspector 34. give.

In Figures 1 and 2, it is simply expressed as a requester 10, a service provider 20, an operator 32, and an inspector 34, but these are smartphones, tablets, PDAs, laptops, desktops operated by each participant. It means a computer device such as a server or a telecommunication device.

Hereinafter, a project curation method in consideration of worker propensity of a crowdsourcing-based project according to an embodiment of the present invention will be described with reference to FIGS. 3 to 14. Details of overlapping content related to the crowdsourcing service configuration and the crowdsourcing-based project progress process described with reference to FIGS. 1 and 2 will be omitted.

Meanwhile, the steps illustrated in FIGS. 3 and 4 may be understood as being performed by a platform server (hereinafter, referred to as a server) operated by the service provider 20, but are not limited thereto.

FIG. 3 is a flowchart schematically illustrating a project curation method in consideration of worker propensity for a crowdsourcing-based project according to an embodiment of the present invention.

First, referring to FIG. 3, a plurality of tasks of a project are assigned to a plurality of workers to request the task to be performed (S110), and a plurality of task results are input from a plurality of workers (S120). Then, a plurality of work results are assigned to a plurality of inspectors to request inspection (S130), and then, a plurality of inspection results for the plurality of work results are input from the plurality of inspectors as passing or rejecting the inspection (S140).

After step S140, the project properties of the project are extracted (S150). Then, the project is classified as a heavy project or a light project based on the project properties extracted in step S150 (S160).

Here, a heavy project corresponds to a heavy worker, and refers to a project that is determined to be suitable for a heavy worker to perform a task. On the other hand, a light project corresponds to a light worker, and refers to a project that is determined to be suitable for a light worker to perform a task. The heavy worker and the light worker will be described later in detail with reference to FIGS. 6 to 11.

Whether a project is suitable for heavy workers or light workers is ultimately based on the properties of the project. Specifically, after opening the project, the project properties are extracted based on the work performance data (Data) of the workers participating in the project and the inspection performance data (Data) of the inspector, and based on this, whether the project is a heavy project or light It is to determine if it is a project.

A project attribute, which is a criterion for determining whether a project corresponds to a heavy project or a light project, may be extracted as described later.

4 is a flowchart schematically illustrating a method of extracting properties of a crowdsourcing-based project according to an embodiment of the present invention, and FIG. 5 is for explaining a method of calculating an average task execution time per one task of a project. It is a drawing.

Referring to FIG. 4, in an embodiment of the present invention, step S150 includes a step of calculating the average task execution time per one task of a project (S151), a step of calculating the number of remaining tasks of the project (S152), and a project. And extracting the average task execution time per task and the number of remaining tasks as project properties of (S153).

First, referring to FIG. 5, a step (S151) of calculating an average task execution time per one task of a project will be described.

The average time required to perform work per task of a project is calculated based on the time required to perform work and the total number of tasks of all workers participating in the project.

Referring to FIG. 5, when three workers participated in project A to perform tasks, worker 1 performed 15 tasks for 6 hours, and worker 2 performed 16 tasks for 8 hours. And it can be seen that worker 3 performed 20 tasks in 6 hours. At this time, the average time required to perform a task per task in Project A is about 0.391 hours, divided by the total number of work hours, which is 20 hours, by the total number of tasks, which is 51.

In step S151, the average task execution time per task is calculated, and then in step S152, the number of remaining tasks for the project is calculated. The number of remaining work in the project refers to the number of remaining work in the project until the deadline from the time when the properties of the project are extracted.

On the other hand, the time point for extracting the properties of the project may be determined as follows.

The properties of the project can be extracted at the point in time when a preset period has elapsed after the project is opened or when the value of the number of remaining jobs compared to the total number of jobs is less than or equal to the preset value (that is, when the completion rate of the project is more than a specific value). . However, it is not limited thereto.

In step S153, the average task execution time per task calculated in steps S151 and S152 and the number of remaining tasks are extracted as properties of the project.

On the other hand, according to an embodiment of the present invention, step S152 calculates the number of remaining tasks of the project and the number of tasks completed during the previous unit period, and step S153 is the average task execution per one task as a project property of the project. It is possible to extract the number of tasks completed in the previous unit period compared to the required time and the number of remaining tasks.

For example, assuming that the unit period is 1 day, the time of extracting the properties of the project is May 11, and the deadline of the project is May 15, the task will be performed from May 11 to May 15. Calculate the number of remaining tasks to be done and the number of tasks completed during May 10 (unit period immediately before May 11). In addition, based on the calculated result, the number of tasks completed during the immediately preceding unit period compared to the remaining number of tasks may be extracted.

In step S153, as the project property of the project, the number of tasks completed during the immediately preceding unit period compared to the number of remaining tasks as well as the average task execution time per task may be extracted.

In addition to the above-described average task execution time per task and the number of remaining tasks, properties of the project may be extracted in various ways. This will be described in detail after the step S170 is described.

Referring to FIG. 3 again, based on the properties of the project extracted in step S150, the project is classified as a heavy project or a light project in step S160.

The method of classifying the project in step S160 is as follows.

For example, the average task execution time per task extracted as a project property and the number of remaining tasks are compared with each preset reference value, and if the value is greater than or equal to each of the above reference values, it is classified as a heavy project. If it is less than each of the reference values, it may be classified as a light project.

Specifically, the project may be classified as a heavy project when the average time required to perform a task per task is greater than or equal to the first reference value and the number of remaining tasks is greater than or equal to the second reference value. Alternatively, the project may be classified as a light project when the average time required to perform a task per task is less than the third reference value and the number of remaining tasks is less than the fourth reference value. In other words, a project with a relatively high value of the average time required to perform a task per task and the number of remaining tasks is classified as a heavy project, and a project with a low value is classified as a light project.

In this case, the reference value for classification of the heavy project and the reference value for classification of the light project may be the same or different. When the reference values are different from each other, the reference value for classification of the heavy project may be set to a higher value than the reference value for classification of the light project.

In step S170, the project is curated to a heavy worker group corresponding to a heavy project or a light worker group corresponding to a light project based on the classification result in step S160.

Here, a heavy worker means a worker who has a long stay on the platform, has a large number of visits, or is frequent. In other words, a heavy worker refers to a worker who frequently visits the platform and, once visited, has a tendency to stay and perform work for a relatively long time.

On the other hand, a light worker means a worker who has a small residence time on the platform, a small number of visits, or a rare worker. That is, a light worker refers to a worker who visits the platform occasionally, and once visits the platform, has a tendency to stay for a relatively short time and perform work. On the other hand, new workers will have similar tendencies to light workers until they become accustomed to crowdsourcing projects, so they can be treated substantially the same as light workers.

The method of classifying workers into heavy worker groups or light worker groups is as follows.

6 is a diagram for explaining a method of calculating the period of stay and revisit period of the worker in the crowdsourcing platform, and FIG. 7 is a period of stay of the worker when a predetermined reference value for determining the period of stay of the crowdsourcing platform is set And a diagram for explaining a method of calculating a revisit period.

Although not clearly shown, as an embodiment of the present invention, the project curation method in consideration of the worker tendencies of a crowdsourcing-based project includes a plurality of workers of the platform, during the average platform stay period and the average platform revisit period of each worker. On the basis of, the step of classifying into the heavy worker group or the light worker group (not shown) may be further included. In an embodiment, the worker classification step may be performed simultaneously in parallel with the above-described project classification step, but is not limited thereto.

Here, the average length of stay on the platform refers to the average period of time required to leave the platform after accessing the platform, and the average period of revisiting the platform refers to the average period of time taken to access the platform again after leaving the platform. it means.

At this time, the average platform stay period and the average platform revisit period may be calculated as the average platform stay period during the recent unit period and the average platform revisit period during the latest unit period. However, it is not limited thereto.

The method of calculating the period of staying on the platform and the period of revisiting the platform is as follows.

For example, referring to FIG. 6, assuming that the unit period is 4 days and the average platform stay is calculated as the number of days, worker 1 stays for 4 hours on May 6 and then leaves the platform for 2 hours. Stayed on the platform for a while. And on May 9th, I reconnected to the platform and stayed for 5 hours.

At this time, since the average duration of the worker's stay on the platform is calculated as the number of days, the duration of stay on the platform is 2 days for the unit period of worker 1 as of May 10, regardless of the number of visits by worker 1 on May 6th.

Meanwhile, Worker 2 accessed the platform on May 6, May 7, and May 9, respectively, and stayed for 6 hours, 4 hours, and 8 hours. Therefore, as of May 10, the period of stay on the platform for Worker 2's unit period is 3 days.

And as of May 10, Worker 1's revisit period is 2 days until May 9, when he left the platform and revisited again on May 6, and as of May 10, Worker 2 The revisit period of May 7 is 1 day, which is the period taken until May 9, after leaving the platform and revisiting after leaving the platform.

On the other hand, in an embodiment of the present invention, the period of stay of the worker may be calculated as the period of stay only when the worker stays for a time greater than or equal to a preset reference value. This is to avoid judging as the length of stay on the platform when a worker simply visits the platform without participating in a specific project to perform work.

For example, referring to FIG. 7, assuming that the preset value is 1 hour, the 7th of May, when worker 3 stays on the platform for 30 minutes, is not included in the period of stay on the platform. Therefore, at this time, as of May 10, the average duration of worker 3's stay on the platform corresponds to May 6 and May 9, which is 2 days.

And as of May 10, the revisit period of Worker 3 corresponds to 1 day from May 7, when Worker 3 stays on the platform for 30 minutes, until May 9, when Worker 3 revisits and stays for 4 hours. .

In this way, by calculating the length of stay of the worker according to whether the worker participated in the project and performed the actual work, the heavy worker group that induces to participate in the project with a relatively long time required to perform the task and the number of remaining tasks is further carefully selected and classified. can do.

On the other hand, in order to calculate the average platform stay period and the average platform revisit period, a unit period of the last 1 week or 1 month may be used, but is not limited thereto.

Hereinafter, a method of classifying a plurality of workers of the crowdsourcing platform into a heavy worker group or a light worker group based on the average platform stay period and the average platform revisit period of each worker will be described.

8 is a diagram showing the average platform stay period and the average platform revisit period of a plurality of workers of the platform, and FIG. 9 is a heavy worker group or a light worker group based on the average platform stay period compared to the average platform revisit period of the workers. It is a diagram showing a classification method.

FIG. 10 is a diagram for explaining classifying all workers into a heavy worker group or a light worker group, and FIG. 11 is a diagram illustrating classifying some of all workers into a heavy worker group or a light worker group.

According to an embodiment of the present invention, all workers on the platform are sorted in descending order based on the average platform stay period compared to the average platform revisit period of a plurality of workers, and workers with a preset reference value or more are assigned to a heavy worker group and a preset criterion. Workers below the value can be classified into light worker groups.

8 and 9, assuming that the reference value of the average platform stay period compared to the average platform revisit period for classifying heavy and light workers is 1, the average platform stay period and the average platform revisit period of each worker As shown in FIG. 9, workers 1, 5, and 6 with a value of the average platform stay period of 1 or more compared to the average revisit period of the platform are classified into the heavy worker group. In addition, workers 2, 3 and 4 whose average platform stay period is less than 1 compared to the average revisit period are classified into the light worker group.

Referring to FIG. 10, when a heavy worker and a light worker are classified by the same reference value as described above, some of the entire workers are classified into a heavy worker group and the remaining workers are classified into a light worker group.

Meanwhile, according to an embodiment of the present invention, a reference value used for selecting a heavy worker group and a reference value used for selecting a light worker group may be different as the preset reference value. That is, the fifth reference value may be used to select a heavy worker group, and a sixth reference value smaller than the fifth reference value may be used to select the light worker group.

In this case, referring to FIG. 11, some of all workers are classified into a heavy worker group and some of the remaining workers are classified into a light worker group. Accordingly, there may be an operator who does not belong to either the heavy operator group or the light operator group.

On the other hand, in an embodiment of the present invention, the step of classifying the plurality of workers of the platform into the heavy worker group or the light worker group includes: Further based on the available time to perform a task, it may be classified into the heavy worker group or the light worker group.

In the above embodiment, a plurality of workers is classified into a heavy worker group or a light worker group based on profile information input when each worker joins (or registers) as a member of the crowdsourcing platform.

For example, assuming that the unit period is 1 week, it is classified as a heavy worker group or a light worker group based on the available days of the week that the worker can perform the work on and the time available for the work on the days of the week that can be performed. can do. In this case, the more time the worker can perform the work, the heavier the worker group, and the less time the worker can perform the work, the light worker group may be classified.

12 is an exemplary diagram illustrating a method of curating a heavy project to a heavy worker group using a recommended badge according to an embodiment of the present invention.

13 is an exemplary view showing a method of dividing a heavy project into separate recommended tasks and curating a heavy worker group according to an embodiment of the present invention.

Referring to FIG. 12, in a crowdsourcing platform, a heavy project may be curated by including an image object such as a recommendation badge that induces participation of a heavy worker group corresponding to the heavy project in an object representing the corresponding project.

Alternatively, referring to FIG. 13, on the screen of the crowdsourcing platform of the heavy worker group, a project classified as a heavy project may be displayed in a list of recommended tasks separated from possible tasks to be curated to a heavy worker group. Meanwhile, in the case of the light worker group, projects classified as light projects will be displayed on the recommended work list on the platform screen of the light worker group and will be curated by the light worker group.

According to the above-described method, it is possible to recommend a project suitable for the worker to perform according to the worker's disposition to the worker who has connected to the crowdsourcing platform, and induce the worker to participate in the project. In particular, since light workers visit the platform less often and cannot accurately predict the date of visit, when light workers connect to the platform, curate light projects to light workers, and when light workers participate in light projects. , You will be able to leave a good impression on the platform, and ultimately induce workers to revisit the platform.

On the other hand, as an embodiment of the present invention, the project curation method in consideration of worker tendencies of crowdsourcing-based projects may further include providing incentive points to workers who participated in the curated project and performed tasks for a predetermined period of time. have.

Through this, it is possible to actively induce heavy workers' participation in heavy projects that take a long time to perform one task and the number of remaining tasks is sufficient.

In addition, by actively inducing light workers to participate in the light project so that they can gain proper work experience even during intermittent visits, and above all, prevent light workers from leaving the platform by giving them a good impression of the crowdsourcing platform. can do.

In the following, as described above, a method of extracting the properties of a project, which is a standard for classifying a project into a heavy project or a light project, will be described in detail.

In an embodiment of the present invention, step S150 further includes a step of extracting a rejection rate of the project (not shown), and step S153 includes the average task execution time per task as a project property of the project, and the In addition to the number of remaining jobs, the rejection rate can be further extracted.

The rejection rate can be calculated as the number of rejections relative to the total number of tasks that have passed the inspection of the project based on the time when the properties of the project are extracted, or can be calculated as the average number of rejections per task that has passed the inspection. However, it is not limited thereto.

In this case, the number of rejections may be calculated in consideration of only the initial rejection of the task, or may be calculated including re-rejection, but is not limited thereto.

The high rejection rate of a project means that the project is difficult to work with. Therefore, if a light worker participates in a project with a high rejection rate, it may cause repetitive rejection of the result of repetitive work due to a higher degree of difficulty, which can delay the progress of the project. In particular, rework is required when the work result is returned, but the project may be delayed due to the belated revisit of the light worker. In addition, light workers who have difficulty in performing tasks may have a bad impression of the crowdsourcing platform, which may eventually lead to workers leaving the platform.

Therefore, by extracting the rejection rate as an attribute of the project, and categorizing and curating heavy projects or light projects by additionally considering them, the above problems can be prevented by having an appropriate work experience according to the capabilities of the worker.

Specifically, if the average task execution time per task is greater than or equal to the first standard value, the number of remaining tasks is greater than or equal to the second standard value, and the rejection rate is equal to or greater than the seventh standard value, the project is classified as a heavy project. I can. Alternatively, the project can be classified as a light project if the average time required to perform an operation per task is less than the third standard value, the number of remaining tasks is less than the fourth standard value, and the rejection rate is less than the eighth standard value. have.

In this case, the seventh reference value and the eighth reference value may be the same or different. The seventh reference value may be set to a higher value than the eighth reference value.

In addition, according to another embodiment of the present invention, step S150 further includes the step of extracting the expected work revenue per unit period of the project, and step S153 is the average task execution time per task as a project property of the project. And, in addition to the number of remaining jobs, the expected work revenue per unit period may be further extracted.

The project can be classified as a heavy project based on extracting the projected profit per unit period, together with the average time required to perform work and the number of remaining work, as a property of the project.

Specifically, projects in which the projected work income per unit period is higher than the preset reference value are classified as heavy projects and curated to corresponding heavy workers. This will lead to an influx of heavy worker groups to heavy projects that can generate high expected work returns. In other words, through a project with a high expected work income, it will be possible to provide sufficient motivation and rewards to heavy workers who tend to perform a relatively large number of tasks.

In this case, the expected work profit per unit period may be calculated based on the average work execution time and work unit cost. For example, if the unit period is 3 hours, the average time required to perform the work is 5 minutes, and the unit cost is 100 won, the estimated 5 profit per unit period of 3 hours is calculated as 3,600 won.

In addition, as another embodiment of the present invention, step S150 further includes extracting the rework limit time of the project, and step S153 includes the average task execution time per task as a project property of the project, and In addition to the remaining number of jobs, the rework limit time may be further extracted.

In the case of a rejected task, if the worker assigned the rejected task does not start rework within the rework time limit, the assignment is canceled and assigned to another worker. This is to prevent project-wide bottlenecks caused by delays in rework.

If a light worker who stays on the platform for a short period of time and takes a long time to revisit, participate in a project with a short rework time limit, it is difficult to expect to perform rework within the rework time limit, so the rework time limit is limited. It is recommended that short projects be classified as heavy projects and curated by a group of heavy workers. To this end, projects whose rework time limit is shorter than the preset reference value are classified as heavy projects and curated to corresponding heavy workers.

On the other hand, the method of curating the above-described project may be based on the work equipment possessed by the worker. In this regard, it will be described below.

14 is a diagram illustrating an example of a work screen of a project including a bounding functional element.

According to an embodiment of the present invention, a project curation method in consideration of the worker tendencies of a crowdsourcing-based project is based on the step of extracting the functional elements of the project (not shown) and the holding work devices input by each worker, It may further include curating the project to a worker who has a work device corresponding to the functional element of the project among the workers of the project (not shown).

This embodiment curates the project by considering the worker's work equipment in addition to the worker's propensity as described above.

In this case, the functional element is determined based on a work tool for executing a project, and the work tool is provided in the project and may be a tool used by workers to perform a task required by the project.

For example, if you need a Bounding Box Tool for bounding work, an Input Box Tool for transfer work, and a Make Step Tool for sorting work to perform a project. , The project can be classified as including a bounding functional element, a transcription functional element, and a classification functional element.

Specifically, referring to FIG. 14, a worker who participates in the project presented in FIG. 14 performs a bounding operation for selecting a specific object, a pig, from the presented image. The operator may use a bounding box tool to perform the bounding operation. That is, the bounding functional elements included in the project are determined based on the bounding box tool.

At this time, the project in FIG. 14 will be more appropriate in terms of work speed or quality of work for a worker having related equipment such as a drawing tablet or a drawing pad due to the nature of the bounding work.

Accordingly, by extracting the functional elements included in the project and curating the project to a worker who has a working device corresponding to the functional element, the work quality of the project can be improved and the operator's work convenience can be promoted.

In this case, the worker's work device may be based on the information on the work device inputted when the worker joins the crowdsourcing platform, but is not limited thereto.

Meanwhile, in the above description, steps S110 to S170 and S151 to S153 may be further divided into additional steps or may be combined into fewer steps according to an embodiment of the present invention. In addition, some steps may be omitted as necessary, or the order between steps may be changed. In addition, even if other contents are omitted, the contents of FIG. 15 to be described later may be applied to the project curation method in consideration of the worker tendency of the crowdsourcing-based project of FIGS. 3 to 14.

Hereinafter, with reference to FIG. 15, a project curation apparatus in consideration of a worker tendency of a crowdsourcing-based project according to an embodiment of the present invention will be described.

15 is a diagram for explaining a project curation apparatus in consideration of a worker tendency of a crowdsourcing-based project according to an embodiment of the present invention.

Referring to FIG. 15, a project curation device 200 (hereinafter, referred to as a device) in consideration of worker propensity for a crowdsourcing-based project includes a communication module 210, a memory 220, and a processor 230.

The communication module 210 transmits a crowdsourcing-based work for one project to a plurality of workers 32 to request a task to be performed, and receives a work result from the plurality of workers 32. The communication module 210 transmits the work results received from the plurality of workers 32 to the plurality of inspectors 34 to request inspection, and receives the inspection results from the plurality of inspectors 34. At this time, for the rejected inspection result, a corresponding rework is transmitted to the worker 32 to request rework, and the rework result is received from the worker 32.

The memory 220 stores a program for classifying a project into a heavy project or a light project based on data received from the communication module 210 and classifying workers into a heavy worker group or a light worker group.

The processor 230 executes a program stored in the memory 220. As the processor 230 executes the program stored in the memory 220, the processor 230 may perform the project curation method in consideration of the worker tendency of the crowdsourcing-based project described with reference to FIGS. 3 to 14.

The device 200 described with reference to FIG. 15 may be provided as a component of the above-described server.

The project curation method in consideration of the worker tendency of a crowdsourcing-based project according to an embodiment of the present invention described above is implemented as a program (or application) to be executed in combination with a computer, which is hardware, and is stored in a computer-readable recording medium. Can be saved.

The above-described program includes C, C++, JAVA, Ruby, which can be read by a processor (CPU) of the computer through the device interface of the computer, in order for the computer to read the program and execute the methods implemented as a program. It may include a code (Code) coded in a computer language such as machine language. Such code may include a functional code related to a function defining necessary functions for executing the methods, and a control code related to an execution procedure necessary for the processor of the computer to execute the functions according to a predetermined procedure. can do. In addition, these codes may further include additional information required for the processor of the computer to execute the functions or code related to a memory reference to which location (address address) of the internal or external memory of the computer should be referenced. have. In addition, when the processor of the computer needs to communicate with any other computer or server in a remote in order to execute the functions, the code is It may further include a communication-related code for whether to communicate, what kind of information or media to be transmitted and received during communication.

The stored medium is not a medium that stores data for a short moment, such as a register, cache, memory, etc., but a medium that stores data semi-permanently and can be read by a device. Specifically, examples of the storage medium include, but are not limited to, ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, and the like. That is, the program may be stored in various recording media on various servers to which the computer can access, or on various recording media on the user's computer. In addition, the medium may be distributed over a computer system connected through a network, and computer-readable codes may be stored in a distributed manner.

The above description of the present invention is for illustrative purposes only, and those of ordinary skill in the art to which the present invention pertains will be able to understand that other specific forms can be easily modified without changing the technical spirit or essential features of the present invention. will be. Therefore, it should be understood that the embodiments described above are illustrative and non-limiting in all respects. For example, each component described as a single type may be implemented in a distributed manner, and similarly, components described as being distributed may also be implemented in a combined form.

The scope of the present invention is indicated by the claims to be described later rather than the detailed description, and all changes or modified forms derived from the meaning and scope of the claims and their equivalent concepts should be construed as being included in the scope of the present invention. do.

10: requester
20: service provider
30: public
32: worker
34: inspector

Claims (10)

As a method performed by a computer,
Allocating a plurality of tasks of a crowdsourcing-based project (hereinafter, "project") to a plurality of workers and requesting the task to be performed;
Receiving a plurality of work results from the plurality of workers;
Assigning the plurality of work results to a plurality of inspectors and requesting to perform the inspection;
Receiving a plurality of inspection results for the plurality of work results from the plurality of inspectors as passing or rejecting the inspection;
Extracting project properties of the project;
Classifying the project as a heavy project or a light project based on the project properties of the project; And
Curating the project to a heavy worker group corresponding to the heavy project or a light worker group corresponding to the light project based on the classification result,
Extracting the project properties of the project,
Calculating an average task execution time per task of the project, and
Calculating the number of remaining tasks for the project;
Including the step of extracting the average task execution time per task and the number of remaining tasks as project properties of the project,
Classifying the project as a heavy project or a light project,
Classifying the project as a heavy project if the average task execution time per task is greater than or equal to a first reference value and the number of remaining tasks is greater than or equal to a second reference value; and
If the average task execution time per task is less than a third reference value and the number of remaining tasks is less than a fourth reference value, classifying the project as a light project,
The first reference value is greater than or equal to the third reference value, the second reference value is greater than or equal to the fourth reference value,
Classifying a plurality of workers of a crowdsourcing platform (hereinafter, "platform") into the heavy worker group or the light worker group based on the average platform stay period and the average platform revisit period of each worker; further includes doing,
A project curation method that considers the worker tendencies of crowdsourcing-based projects. delete The method of claim 1,
Classifying the plurality of workers of the platform into the heavy worker group or the light worker group,
Classifying the plurality of workers of the platform into the heavy worker group or the light worker group further based on the available time to perform work per unit period input by each worker,
A project curation method that considers the worker tendencies of crowdsourcing-based projects. The method of claim 1,
The step of calculating the number of remaining tasks for the project,
Calculate the number of remaining tasks in the above project and the number of tasks completed in the previous unit period,
The step of extracting the average task execution time per task and the number of remaining tasks as project properties of the project,
Extracting, as a project property of the project, the average task execution time per task and the number of tasks completed during the immediately preceding unit period compared to the number of remaining tasks,
A project curation method that considers the worker tendencies of crowdsourcing-based projects. The method of claim 1,
Extracting the project properties of the project,
Further comprising the step of extracting the rejection rate of the project,
The step of extracting the average task execution time per task and the number of remaining tasks as project properties of the project,
Extracting the average task execution time per task, the number of remaining tasks, and the rejection rate as project properties of the project,
A project curation method that considers the worker tendencies of crowdsourcing-based projects. The method of claim 1,
Extracting the project properties of the project,
Further comprising the step of extracting the expected work revenue per unit period of the project,
The step of extracting the average task execution time per task and the number of remaining tasks as project properties of the project,
Extracting the average task execution time per one task, the number of remaining tasks, and the expected work revenue per unit period as project properties of the project,
A project curation method that considers the worker tendencies of crowdsourcing-based projects. The method of claim 1,
Extracting the project properties of the project,
Further comprising the step of extracting the rework time limit of the project,
The step of extracting the average task execution time per task and the number of remaining tasks as project properties of the project,
Extracting the average task execution time per one task, the number of remaining tasks, and the rework limit time as project properties of the project,
A project curation method that considers the worker tendencies of crowdsourcing-based projects. The method of claim 1,
Extracting functional elements of the project; And
Curating the project to a worker who has a work device corresponding to a functional element of the project among a plurality of workers of the platform, based on the holding work device input by each worker,
The functional element is determined based on a tool for executing a project, and the work tool is provided in the project, and is a tool used by workers to perform tasks required by the project,
A project curation method that considers the worker tendencies of crowdsourcing-based projects. The method of claim 1,
Further comprising the step of providing incentive points to workers who participated in the curated project and performed work for a predetermined period of time,
A project curation method that considers the worker tendencies of crowdsourcing-based projects. A computer program stored in a computer-readable recording medium in order to perform the project curation method in consideration of the worker tendency of the crowdsourcing-based project of any one of claims 1, 3 to 9 in combination with a computer.

Images