KR102205811B1 - Method for setting minimum work time using work time of each functional elements of crowdsourcing based project for artificial intelligence training data generation - Google Patents

Abstract

Provided is a method for setting a minimum work time using work time of each functional element of a crowdsourcing-based project for generating artificial intelligence training data, which is efficient in management of a project schedule. According to the present invention, the method comprises the following steps: identifying a plurality of functional elements included in an open-scheduled crowdsourcing-based project (hereinafter, referred to as a project); extracting previous project history information including corresponding functional elements from each functional element; calculating a work time for each functional element based on the previous project history information; summing the work time for each functional element to estimate a work time (hereinafter, referred to as an estimated work time) per one job of the project; setting a first minimum work time of the project based on the expected work time; assigning a task of the project to a worker and requesting execution of the task; and activating a work result input interface after the first minimum work time has elapsed to receive work results from the worker.

Description

How to set minimum working time using working hours for each functional element of a crowdsourcing-based project for generating artificial intelligence learning data {METHOD FOR SETTING MINIMUM WORK TIME USING WORK TIME OF EACH FUNCTIONAL ELEMENTS OF CROWDSOURCING BASED PROJECT FOR ARTIFICIAL INTELLIGENCE TRAINING DATA GENERATION}

The present invention relates to a method for setting a minimum working time using working hours for each functional element of a crowdsourcing-based project for generating artificial intelligence learning data.

Recently, more and more companies collect and process large amounts of data on a crowdsourcing basis that engages the general public in some process of corporate activities. In other words, by opening a project and allowing the general public, that is, workers to participate in the project, necessary information is collected through the work results completed by the workers.

At this time, in order to collect more reliable information, the company assigns the work result completed by the worker to the inspector to perform the inspection work.

Specifically, when one project is opened, a plurality of tasks are assigned to each of the plurality of workers. Each worker performs a plurality of tasks assigned to it, and provides the task results. Thereafter, a plurality of inspection tasks are assigned to each of the plurality of inspectors, and each inspector performs a plurality of inspection tasks assigned to them.

However, since there is an unusual work progress by some workers, a minimum working time is set in the project to prevent this. In other words, when workers participate in the project and perform work, the minimum amount of time is spent on the project, thereby preventing some workers from performing the work in an emergency.

Currently, pilots are conducted before the project is opened to determine how many tasks a plurality of workers perform during the same time period, and the minimum work time is set by measuring the work time per work.

However, this method has a problem that it is difficult to accurately measure time in a way that does not consider each worker's capabilities. In addition, there is a problem that additional costs are incurred by conducting a pilot or the like before the project is opened.

Unexamined Patent Publication No. 10-2014-0095956, 2014.08.04.

The problem to be solved by the present invention is to provide a method for setting a minimum working time using working hours for each functional element of a crowdsourcing-based project for generating artificial intelligence learning data.

However, the problem to be solved by the present invention is not limited to the problem as described above, and other problems may exist.

In order to solve the above-described problem, a method for setting the minimum working time using working hours for each functional element of a crowdsourcing-based project according to an aspect of the present invention is to include a plurality of open crowdsourcing-based projects (hereinafter, projects). Identifying functional elements, extracting previous project history information including the corresponding functional element for each functional element, calculating working hours per functional element based on the previous project history information, respectively Estimating a work time per work (hereinafter, an expected work time) of the project by summing work hours per functional element of, setting a first minimum work time of the project based on the expected work time, the Assigning a task of a project to a worker and requesting the task to be performed; and activating a task result input interface after the first minimum working time has elapsed, and receiving a task result from the worker, and the first of the project The step of setting the minimum working time is to set a time corresponding to the first ratio of the expected working time as the first minimum working time of the project, and the functional element is a work tool for executing the project. It is determined based on the work tool, which is provided in the project and is a tool used by workers to perform work required by the project.

In some embodiments of the present invention, the previous project history information includes the total work time, the total number of work, the average work time, the number of rejections, the work time of the rejected work in the previous project. And it may include at least one of the work time in consideration of the number of times.

In some embodiments of the present invention, the total working time may be generated by recording a start time and an end time of use of a work tool corresponding to a corresponding functional element in a previous project.

In some embodiments of the present invention, the method comprises: estimating a time required for the project (hereinafter, an expected time required) based on the projected working time, comparing the projected time required with the target time required. According to the step and the comparison result, the step of adjusting the first minimum working time of the project may be further included.

In some embodiments of the present invention, the step of adjusting the first minimum working time of the project according to the comparison result, if the estimated time required is shorter than the target time required, corresponds to the first ratio of the expected work time The first minimum working time of the project is maintained as the required time, and the first ratio may be set based on the number of possible work per unit time and a standard wage per unit time.

In some embodiments of the present invention, adjusting the first minimum working time of the project according to the comparison result, if the estimated time required is longer than the target time required, corresponds to a second ratio of the target time required It is possible to adjust the first minimum working time of the project as the time required.

In some embodiments of the present invention, the step of estimating the time required for the project based on the projected work time (hereinafter, the projected time required) may be performed by multiplying the projected target work quantity by the projected work time. You can estimate the duration.

In some embodiments of the present invention, the method comprises: assigning the work result to an inspector and requesting to perform inspection, receiving an inspection result for the work result from the inspector as passing or rejecting inspection, and the inspection result Requesting the operator to perform rework for the job result (hereinafter referred to as the rejected job result) received as rejection, calculating an object-based rejection rate for the rejected job result, based on the object-based rejection rate And setting a second minimum working time of the project, and activating a work result input interface after the second minimum working time has elapsed, and receiving a rework result from the worker, and the inspection result In the step of receiving the input, the result of the inspection of the job result is received as inspection pass or rejection for each object, but if the number of rejected objects among one or more objects included in the job is at least one, the job result is rejected as a whole, and the object The base rejection rate may be calculated based on the number of rejected objects among one or more objects included in the task.

In some embodiments of the present invention, the step of setting the second minimum working time of the project based on the object-based rejection rate comprises multiplying the first minimum working time by the object-based rejection rate to obtain the second minimum working time. Can be set.

In some embodiments of the present invention, the first minimum working time of the first task and the second task of the project having the same work unit cost is set equally, and by different object-based rejection rates, the first task and the second task The second minimum working time of the second operation may be set differently from each other.

A computer program according to another aspect of the present invention for solving the above-described problems is combined with a computer as hardware to execute a method for setting the minimum working time using working hours for each functional element of the crowdsourcing-based project, and computer-readable It is stored on the recording medium.

Other specific details of the present invention are included in the detailed description and drawings.

According to the present invention described above, since the minimum working time is set by using previous project history information for each functional element of the project, it is not necessary to conduct a pilot or the like before the project is opened, which is efficient in terms of time and cost.

In addition, the minimum working time is effective for project schedule management by adjusting according to the target duration of the project.

When performing rework due to rejection processing as a result of the inspection, the object-based rejection rate of rejected tasks is not applied equally, that is, the minimum work time applied at the time of initial operation is not applied equally. It is possible to reduce unnecessary waiting time for workers when performing rework by calculating and setting a new minimum working time by applying an object-based rejection rate. In conclusion, the overall progress of the project can be improved by shortening the rework time due to rejection processing.

The effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the following description.

1 is a conceptual diagram of a crowdsourcing service according to an embodiment of the present invention.
2 is a flowchart illustrating a crowdsourcing-based project progress process according to an embodiment of the present invention.
3 is a flowchart of a method of setting a first minimum working time using working hours for each functional element of a crowdsourcing-based project according to an embodiment of the present invention.
4 is an exemplary diagram for explaining extraction of a previous project in which each functional tool applied to an open project according to an embodiment of the present invention is used.
5 is an exemplary view for explaining calculating the working time per each functional element according to an embodiment of the present invention.
6 is a flowchart of a method of adjusting a first minimum working time according to a target duration of a crowdsourcing-based project according to an embodiment of the present invention.
7 is a flowchart of a method of setting a second minimum working time for a crowdsourcing-based project according to an embodiment of the present invention.
8 is an exemplary diagram for explaining an inspection result for each object according to an embodiment of the present invention.
9 is an exemplary diagram for explaining calculation of an object rejection rate according to an embodiment of the present invention.
10 is a view for explaining a device for setting a minimum working time using working hours for each functional element of a crowdsourcing-based project according to an embodiment of the present invention.

Advantages and features of the present invention, and a method of achieving them will become apparent with reference to the embodiments described below in detail together with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but may be implemented in a variety of different forms, only the present embodiments are intended to complete the disclosure of the present invention, It is provided to fully inform the technician of the scope of the present invention, and the present invention is only defined by the scope of the claims.

The terms used in the present specification are for describing exemplary embodiments and are not intended to limit the present invention. In this specification, the singular form also includes the plural form unless specifically stated in the phrase. As used in the specification, “comprises” and/or “comprising” do not exclude the presence or addition of one or more other elements other than the mentioned elements. Throughout the specification, the same reference numerals refer to the same elements, and “and/or” includes each and all combinations of one or more of the mentioned elements. Although "first", "second", and the like are used to describe various elements, it goes without saying that these elements are not limited by these terms. These terms are only used to distinguish one component from another component. Therefore, it goes without saying that the first component mentioned below may be the second component within the technical idea of the present invention.

Unless otherwise defined, all terms (including technical and scientific terms) used in the present specification may be used as meanings that can be commonly understood by those of ordinary skill in the art to which the present invention belongs. In addition, terms defined in a commonly used dictionary are not interpreted ideally or excessively unless explicitly defined specifically.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a conceptual diagram of a crowdsourcing service according to an embodiment of the present invention.

Referring to FIG. 1, a crowdsourcing service is performed by being composed of a client 10, a service provider 20, and a public 30.

The sponsor 10 refers to a company or individual who requests a crowdsourcing-based project (hereinafter, referred to as a project).

The requester 10 requests a project for the purpose of collecting source data or annotating data for generating artificial intelligence learning data. The data generated through the project can be used as learning data for arbitrary machine learning such as supervised learning, unsupervised learning, and reinforcement learning. Collection of source data refers to collecting raw data such as recorded voice collection and photo collection. Data annotation refers to inputting relevant annotation data into source data such as text, photos, and videos. For example, the data annotation may include finding an entity in a given fingerprint or finding a similar sentence, but is not limited thereto. Meanwhile, the types of the above-described projects are only one embodiment, and various projects may be handled in the present invention according to the design of the client.

The service provider 20 refers to a company that provides crowdsourcing services.

When the service provider 20 receives a request for a project for a product or service from the client 10, the service provider 20 allocates a task for the project to the general public 30 and receives the work result from the public 30. Thereafter, the final product extracted based on the work result is provided to the client 10.

At this time, the service provider 20 provides a crowdsourcing service to the client 10 and the public 30 through a crowdsourcing platform (hereinafter, the platform). That is, when the service provider 20 receives a request for a project from the client 10, the service provider 20 opens the project on the platform. Thereafter, when the work result for the open project is received from the public 30, the project is terminated on the platform, and the final product may be extracted and provided to the client 10.

The public 30 refers to the general public participating in the project open on the platform. Here, the public 30 may participate in a project opened on the platform through an application or website provided by the service provider 20.

The public 30 is composed of a worker 32 and an inspector 34.

The worker 32 decides to participate in a specific project among a plurality of projects open on the platform. Thereafter, the operator 32 performs an operation such as collection of source data or data annotation, and transmits it to the platform.

The inspector 34 decides to participate in a specific project among a plurality of projects open on the platform. Thereafter, the inspector 34 inspects the result of the work performed by the operator 32. As a result of performing the inspection, the inspector 34 may perform inspection pass processing or rejection processing, and input a rejection reason during rejection processing. In the case of passing the inspection, it is not necessary to rework and re-inspect the result.

2 is a flowchart illustrating a crowdsourcing-based project progress process according to an embodiment of the present invention.

First, the requester 10 requests one or more projects to the service provider 20 (S11).

Thereafter, the service provider 20 opens the requested project on the platform (S12). At this time, the service provider 20 may determine the grade in consideration of the difficulty of the project before opening the project. That is, depending on the difficulty level, it is possible to determine which level or higher public 30 to expose the project. Accordingly, it is possible to increase the reliability of the work result of the project.

Thereafter, the service provider 20 requests the work by assigning the work to the workers 32 of the corresponding level or higher according to the level of the project (S13).

Thereafter, the worker 32 performs the assigned task (S14). In this case, the worker 32 may input the reason for the inability to work without performing the work for a work in which the work itself is impossible for some reason.

Thereafter, the service provider 20 receives the work result from the worker 32 (S15), and assigns the work result to the inspector 34 to request the inspection (S16).

Thereafter, the inspector 34 performs the assigned inspection (S17). At this time, the inspector 34 decides to pass the inspection when it is determined that the work has been properly performed, and if it is determined that the inspection work is wrong, the inspector 34 rejects it. During rejection processing, the inspector 34 inputs the rejection reason for what reason the job was judged to be wrong.

Thereafter, the service provider 20 receives the inspection result from the inspector 34 (S18).

If the inspection result passes the inspection, the service provider 20 uses the work result as valid data, and based on this, extracts the final product at the end of the project.

When the inspection result is rejection processing, the service provider 20 may internally perform the inspection again, or assign the work to the worker 32 again to perform the rework. When reworking, re-examination by an inspector is required

Thereafter, when the project period ends or sufficient valid data is secured, the service provider 20 terminates the project (S19), calculates a final result based on the secured valid data, and provides it to the client 10 ( S20).

At this time, before the end of the project, the service provider 20 evaluates the performance results of the worker 32 and the inspector 34, calculates the work cost and the inspection cost according to the evaluation, to the worker 32 and the inspector 34. give.

In FIGS. 1 and 2, it is simply expressed as a requester 10, a service provider 20, an operator 32, and an inspector 34, but these are smartphones, tablets, PDAs, laptops, desktops, operated by each participant. It means a computer device such as a server or a telecommunication device.

On the other hand, in the current system, the minimum time required to perform one task is set in order to prevent an emergency operation. In other words, the work result input interface is activated only after the minimum work time has elapsed. For example, if the minimum work time is 1 minute, the work result input interface is activated only after 1 minute has elapsed, so the operator can wait 1 minute before completing the work (enter work result). As a result, it was possible to prevent a situation in which fraudulent workers for the purpose of money did not perform the work properly and entered the work result immediately.

However, in order to set the minimum working time, a pilot must be conducted before opening the project. Specifically, in order to set the minimum work time, it is necessary to select the workers to be put into the pilot and measure the work time per one work by grasping the number of work performed by each of the selected workers during the same time period. There is a problem that it is inefficient in terms of cost.

In order to solve this problem, an embodiment of the present invention estimates the work time per one task by using the previous project history information for each functional element of the project, and sets the first minimum work time based on this. .

Hereinafter, a first minimum working time according to an embodiment of the present invention will be described with reference to FIGS. 3 to 5.

3 is a flowchart of a method of setting a first minimum working time using working hours for each functional element of a crowdsourcing-based project according to an embodiment of the present invention. 4 is an exemplary diagram for explaining extraction of a previous project in which each functional tool applied to an open project according to an embodiment of the present invention is used. 5 is an exemplary view for explaining calculating the working time per each functional element according to an embodiment of the present invention.

Meanwhile, the steps shown in FIG. 3 may be understood as being performed by a platform server (hereinafter, referred to as a server) operated by the service provider 20, but are not limited thereto.

In addition, a plurality of workers 32 or a plurality of inspectors 34 perform work using a predetermined terminal device. The terminal device of the operator 32 or the inspector 34 may be a computer device or a telecommunication device such as a smartphone, tablet, PDA, laptop, desktop, etc., but is not limited thereto.

First, referring to FIG. 3, the server identifies a plurality of functional elements included in an open crowdsourcing-based project (hereinafter, referred to as a project) (S110).

Here, the functional elements of the project are determined based on a work tool for executing the project. In addition, the work tool is provided in the project and refers to a tool used by the workers 32 to perform the work required by the project.

For example, the Bounding Box Tool for bounding work, the Input Box Tool for transfer work, and the Make Step Tool for sorting work to perform tasks on the project. If necessary, the work of the project can be classified as including a bounding function element, a transcription function element, and a classification function element to correspond to each work tool.

Next, for each functional element, the server extracts previous project history information including the corresponding functional element (S120).

When the project to be opened includes functional element 1, functional element 2, and functional element 3, projects in which functional element 1, functional element 2, or functional element 3 have been used are extracted from the previously performed projects. After that, the history information of previous projects is extracted for each functional element.

Referring to FIG. 4, when a project scheduled to be open includes functional element 1, functional element 2, and functional element 3, at least one of functional element 1, functional element 2, and functional element 3 among previously performed projects is included. Previous projects A, B, C and D were extracted. Specifically, previous project A included functional element 1, functional element 2, and functional element 3, previous project B included functional element 2, functional element 3, and functional element 4, and previous project C included functional element 1, function It included element 2 and functional element 4, and the previous project D included functional element 1, functional element 3, and functional element 4.

Accordingly, for functional element 1 of an open project, the history information of previous project A, previous project C, and previous project D is extracted, and for functional element 2, the history of previous project A, previous project B, and previous project C. Information is extracted, and for functional element 3, history information of previous project A, previous project B, and previous project D is extracted.

Next, the server calculates a working time per functional element based on the previous project history information (S130).

Here, the previous project history information is one of the total working hours, the total number of tasks, the average working time, the number of rejected tasks, the working hours of the rejected tasks, and the number of working hours in which the work tools corresponding to the functional elements were used in the previous project. Contains at least one.

In other words, the working time of the functional element is information on the accumulated time that the work tool corresponding to the functional element was used by the plurality of workers 32 until the previous project was opened and terminated, and the previous project was opened and completed until the end. It is generated based on the number of work results, the average work time accordingly, the number of work that was returned during work, the rework time for the rejected work, and the work time considering the rework time.

At this time, the total working time is generated by recording the start time and end time of use of a specific functional tool in the previous project, and a plurality of workers 32 participating in the previous project used the tool. This can mean the total time combined. In addition, the total number of jobs may mean a final work result provided by the plurality of workers 32. In addition, the number of rejections refers to the number of times that the work (work result) submitted by the worker 32 during the project is rejected by the inspector 34, and at this time, the worker 32 rework the rejected task I will submit it again. At this time, the rejected work is not included in the total number of jobs, and if one work is rejected several times, it is included as one job in the total number of jobs.

According to an embodiment, the average working time (sec) means a value obtained by dividing the total working time (sec) by the total number of tasks, and the working time (sec) of the rejected task is multiplied by the number of rejections and the average working time (sec). It means a value, and the work time (sec) considering the number of rejections means the sum of the total work time (sec) and the work time (sec) of the rejected work.

Accordingly, the working time per functional tool (sec) may be calculated by dividing the working time (sec) considering the number of rejections by the total number of tasks.

At this time, if the number of previous projects in which a specific function tool was used is n (n is a natural number), the working time of the specific function tool is the sum of the work hours taking into account n number of rejections, and the total number of operations. It can be calculated by sharing.

In the example of FIG. 5, it is assumed that the project to be opened includes a bounding functional element, a transcription functional element, and a classification functional element. The server corresponds to the working time of the bounding box tool corresponding to the bounding function element, the working time of the input box tool corresponding to the transfer function element, and the classification function element based on the previous project history information. The working time of the Make Step Tool can be calculated.

Referring to FIG. 5A, history information of a previous project A, a previous project C, and a previous project D may be extracted for a bounding box tool.

In previous project A, the total working time using the bounding box tool was 1500sec. When the total number of jobs is 100, the average work time is 15sec. to be. In addition, since the number of rejections is 10, the operation time of the rejected operation is 150sec, which is the product of the average operation time and the rejection number. Becomes. Accordingly, in the previous project A, the total working time using the bounding box tool (that is, working time considering the number of rejections) was 1650 sec. Becomes.

Also, in the previous project C, the total working time using the bounding box tool was 800sec. When the total number of jobs is 50, the average work time is 16sec. to be. In addition, since the number of rejections is 25, the operation time of the rejected operation is 400sec, which is the product of the average operation time and the rejection number. Becomes. Accordingly, in the previous project C, the total working time using the bounding box tool (that is, working time considering the number of rejections) was 1200 sec. Becomes.

Also, in the previous project D, the total working time using the bounding box tool was 600sec. When the total number of jobs is 12, the average work time is 50sec. to be. In addition, since the number of rejections is 3, the operation time of the rejected operation is 150sec, which is the product of the average operation time and the rejection number. Becomes. Accordingly, in the previous project D, the total working time using the bounding box tool (that is, working time considering the number of rejections) was 750 sec. Becomes.

Accordingly, the average time to perform one task using the bounding box tool is the working time considering the number of rejections of the previous project A, the working time considering the number of rejections of the previous project C, and the working time considering the number of rejections of the previous project D. Divided by the sum of the total number of tasks in the previous project A, the total number of tasks in the previous project C, and the total number of tasks in the previous project D, i.e. (1650+1200+750)/(100+50+12) ) = 22.22sec. to be.

Referring to FIG. 5B, history information of a previous project A, a previous project B, and a previous project C may be extracted with respect to an input box tool.

In the previous project A, the total working time using the input box tool was 350sec. And, when the total number of jobs is 100, the average work time is 3.5sec. to be. In addition, since the number of rejections is 20, the operation time of the rejected operation is 70sec, which is the product of the average operation time and the rejection number. Becomes. Accordingly, in the previous project A, the total working time using the input box tool (that is, working time considering the number of rejections) was 420 sec. Becomes.

Also, in the previous project B, the total working time using the input box tool was 180sec. When the total number of jobs is 30, the average work time is 6sec. to be. In addition, since the number of rejections is 20, the operation time of the rejected operation is 120sec, which is the product of the average operation time and the rejection number. Becomes. Accordingly, the total working time using the input box tool in the previous project B (that is, working time considering the number of rejections) is 300 sec. Becomes.

Also, in the previous project C, the total working time using the input box tool was 100sec. And, when the total number of jobs is 20, the average work time is 5sec. to be. In addition, since the number of rejections is 10, the operation time of the rejected operation is 50sec, which is the product of the average operation time and the rejection number. Becomes. Accordingly, the total working time using the bounding box tool in the previous project C (that is, working time considering the number of rejections) was 150 sec. Becomes.

Accordingly, the average time to perform one task using the input box tool is the working time considering the number of rejections of the previous project A, the working time considering the number of rejections of the previous project B, and the working time considering the number of rejections of the previous project C. Divided by the sum of the sum of the total number of tasks in the previous project A, the total number of tasks in the previous project B, and the total number of tasks in the previous project C, i.e. (420+300+150)/(100+30+20) ) = 5.8sec. to be.

Referring to FIG. 5C, history information of a previous project A, a previous project B, and a previous project D may be extracted with respect to the Make Step Tool.

In previous project A, the total working time using the make step tool was 500sec. And, when the total number of jobs is 100, the average work time is 5sec. to be. In addition, since the number of rejections is 20, the working time of the rejected job is 100sec, which is the product of the average work time and the number of rejections. Becomes. Accordingly, the total working time (that is, working time considering the number of rejections) using the make step tool in the previous project A was 600 sec. Becomes.

Also, in the previous project B, the total working time using the make step tool was 480sec. When the total number of jobs is 60, the average work time is 8sec. to be. In addition, since the number of rejections is 15, the operation time of the rejected operation is 120sec, which is the product of the average operation time and the rejection number. Becomes. Accordingly, the total working time (that is, working time considering the number of rejections) using the make step tool in the previous project B was 600 sec. Becomes.

Also, in the previous project D, the total working time using the make step tool was 320sec. When the total number of jobs is 40, the average work time is 8sec. to be. In addition, since the number of rejections is 10, the operation time of the rejected task is 80sec, which is the product of the average operation time and the rejection number. Becomes. Accordingly, the total working time (that is, working time considering the number of rejections) using the make step tool in the previous project D was 400sec. Becomes.

Accordingly, the average time to perform one task using the make step tool is the working time considering the number of rejections of the previous project A, the working time considering the number of rejections of the previous project B, and the working time considering the number of rejections of the previous project D. Divided by the sum of the total number of tasks in the previous project A, the total number of tasks in the previous project B, and the total number of tasks in the previous project D, i.e. (600+600+400)/(100+60+40) ) = 8sec. to be.

Next, referring again to FIG. 3, the server estimates the work time per work (hereinafter, the expected work time) of the project by summing the work time per functional element (S140).

In other words, by adding the average working time of each functional tool generated based on the previous project history information for each functional element, the expected working time to perform the work of the open project can be measured.

As described above, assuming that the project to be opened includes a bounding functional element, a transfer functional element, and a classification functional element, the average time to perform one task using the bounding box tool is 22.22 sec. And the average time to perform one task using each input box tool is 5.8sec. And the average time to perform one task using the make step tool is 8sec. Therefore, the expected work time per one task of the open project is 22.22+5.8+8 = 36sec. Becomes.

Next, the server sets the first minimum working time of the project based on the expected working time (S150).

The server sets the time corresponding to the first ratio of the expected working time as the first minimum working time of the project. The first ratio is set based on the number of possible tasks per unit time and a standard wage per unit time. Here, the unit time may be 1 hour, but is not limited now.

Specifically, according to the following [Equation 1], the work unit cost is determined by calculating the number of possible work per unit time based on the expected work time. That is, when the worker 32 performs the maximum number of possible tasks per unit time, the work unit cost should be determined so that it cannot earn work costs that exceed the standard wage per unit time.

[Equation 1]

Unit of work = standard wage per unit hour/(3600 sec/estimated work time)

If the minimum hourly wage of 8,350 won is applied as the standard wage per unit hour, and the estimated working time is 180 seconds (3 minutes), the number of possible work per hour is calculated as 3600 seconds/180 seconds = 20 cases. Accordingly, the unit cost of work can be determined as 8,350 won/20 cases = 417.5 won, about 417 won. That is, even if the worker 32 completes the work by filling all 20 tasks per unit time during an hour, 8,340 won lower than the minimum hourly wage will be paid as work cost.

According to the following [Equation 2], the first minimum working time is set based on the determined work unit cost. That is, the first minimum work time may be determined based on the number of possible work per unit time that can earn a predetermined multiple of the standard wage per unit time. If the minimum hourly wage is 8,350 won per unit hour, and 1.5 times the predetermined multiple is applied, it is determined based on the number of work per unit hour that can earn 12,525 won.

[Equation 2]

1st minimum working time = 3600 seconds/((standard wage per unit time * constant rate)/work unit price)

If the work unit price is determined to be 417 won, the first minimum work time in the example above is about 3600 seconds/(12,525 won/417 won)=120 seconds, which is about 2 minutes.

Next, the server assigns the task of the project to the worker 32 and requests the task to be performed (S160).

In one embodiment of the present invention, each task is configured to include one or more objects.

An object is a unit of work that composes a work, and refers to a unit of work that can determine whether to pass or reject inspection within a work. For example, in the case of completing a plurality of sentences according to a scenario, one sentence may be one object, and in the case of annotating an area that satisfies a specific condition in a specific image, one area that satisfies the condition is one It may be an object of.

Next, after the first minimum work time has elapsed, the server activates the work result input interface and receives the work result from the worker 32 (S170).

According to an embodiment of the present invention, an interface for receiving a work result may be activated only after the first minimum work time has elapsed. That is, in order for the worker 32 to transmit the work result to the server, the first minimum work time must elapse.

For example, the interface for inputting the work result is separate from the interface provided to the operator 32 to perform work such as collection of source data or data annotation, and completes the input of the work result or moves on to the next work. It may be an interface to go to.

The first minimum working time set as described above functions to prevent unintentional work progress by some workers, but may act as a factor in which the project schedule cannot be met. That is, since the first minimum working time is set based on the working time per functional element without considering the schedule of the project, there may be a situation where the target number of work cannot be achieved by the predetermined schedule.

Therefore, it is necessary to set the first minimum working time in consideration of not only the working time for each functional element, but also the schedule of the project.

Hereinafter, with reference to FIG. 6, a method of adjusting the first minimum working time set by using the working time for each functional element in consideration of the target duration of the project will be described.

6 is a flowchart of a method of adjusting a first minimum working time according to a target duration of a crowdsourcing-based project according to an embodiment of the present invention.

Meanwhile, the steps shown in FIG. 6 may be understood as being performed by a platform server (hereinafter, referred to as a server) operated by the service provider 20, but are not limited thereto.

Referring to FIG. 6, the server estimates a period (hereinafter, an expected period) required for the project based on the expected work time (S210).

Specifically, the server estimates the estimated time required by multiplying the project's target work quantity by the estimated work time.

For example, if the project's target number of work is 1,000 and the estimated work time is 180 seconds (3 minutes), the estimated time required may be estimated as 1,000 cases x 180 seconds = 50 hours (3 days 2 hours).

Next, the server compares the expected duration of the project and the target duration (S220), and adjusts the first minimum working time of the project according to the comparison result (S230).

As a result of the comparison, if the estimated duration is shorter than the target duration, the first minimum working time of the project is maintained at a time corresponding to the first ratio of the estimated working time.

When the expected lead time is shorter than the target lead time, it is usually the case that the difficulty of the project is low. Because the difficulty of the project is low, the time required per task (estimated working time) is short, so the project can proceed with ease. Therefore, there is no need to adjust the first minimum working time set to a time corresponding to the first ratio of the expected working time.

In contrast, as a result of the comparison, if the estimated duration is longer than the target duration, the first minimum working time of the project is adjusted to a time corresponding to the second ratio of the target duration.

When the expected lead time is longer than the target lead time, it is usually the case that the difficulty of the project is high. Due to the high difficulty of the project, the time required per task (estimated working time) is long, making it difficult to meet the schedule of the project or it may be overrun. Therefore, it is necessary to adjust the first minimum working time of the project to a time corresponding to the second ratio of the target duration.

The second ratio is a value smaller than the first ratio, and may be set, for example, in decimal units.

By adjusting the first minimum working time to a time corresponding to the second ratio of the target required period, that is, shortening the first minimum working time, workers 32 having excellent work performance are not limited to the first minimum working time. It enables you to perform tasks without your ability, and thereby achieves the target number of tasks by a set schedule.

When the first minimum working time is appropriately adjusted based on the target required period according to steps S210 to S230, the server assigns the task of the project to the worker 32 and requests the task to be performed (S160), and the first minimum working time After this has elapsed, the work result input interface is activated to receive the work result from the operator 32 (S170).

That is, if the first minimum working time set based on the working time for each functional element does not affect the project schedule, the first minimum working time is applied when performing the work of the plurality of workers 32, but the first If the minimum working time affects the schedule of the project, the server adjusts the first minimum working time based on the target duration of the project, and the first minimum working time adjusted when performing the work of the plurality of workers 32 is To be applied.

As described above, when the plurality of workers 32 participate in the project and perform work, the first minimum working time set or adjusted is applied.

However, in a project that includes many objects in one job, if the job result is returned because of only a few objects, the worker has to wait for the minimum working time unnecessarily in the process of reworking. There is a problem that the overall work speed is slowed down.

For example, if there are 30 objects per source and the first minimum work time is set to 5 minutes, and the work result is rejected by one object, it takes less than 1 minute to rework one object. It does not take, but you have to wait 5 minutes due to the first minimum working time setting.

Therefore, it is necessary to separately apply the minimum rework time (hereinafter, the second minimum work time) for the rejected work.

Hereinafter, a second minimum working time according to an embodiment of the present invention will be described with reference to FIGS. 7 to 9.

7 is a flowchart of a method of setting a second minimum working time for a crowdsourcing-based project according to an embodiment of the present invention. 8 is an exemplary diagram for explaining an inspection result for each object according to an embodiment of the present invention. 9 is an exemplary diagram for explaining calculation of an object rejection rate according to an embodiment of the present invention.

Meanwhile, the steps shown in FIG. 7 may be understood as being performed by a platform server (hereinafter, referred to as a server) operated by the service provider 20, but are not limited thereto.

First, referring to FIG. 7, the server assigns the work result to the inspector 34 to request the inspection performance (S310), and receives the inspection result for the work result from the inspector 34 as passing or rejecting the inspection (S320). .

Specifically, the server receives the inspection result of each work result from each inspector 34 as inspection pass or reject by object.

In this case, when the inspection result of at least one object among the plurality of objects of one work result is received as a rejection, one operation result is entirely rejected. That is, only when the inspection results of a plurality of objects of one work result are all inputted as the inspection pass, one work result is processed as a whole through the inspection pass.

Referring to FIG. 8, when one job is configured to include 4 objects, the job result A in which the inspection result for one of the 4 objects is input as rejection is processed as a rejection. Although not shown in FIG. 8, when the inspection results for two of the four objects are entered as rejection, and the inspection results for three of the four objects are entered as rejection, the inspection results for four objects Even if all are entered as rejected, the result A is rejected as a whole. On the other hand, the work result B, in which all the inspection results for 4 objects are inputted as inspection passes, is processed as a whole.

Next, the server requests the operator to perform rework on the job result (hereinafter referred to as the rejected job result) received as a rejection of the inspection result (S330). In this case, the server does not apply the same first minimum working time applied at the time of the initial work, but applies a second minimum working time different from the first minimum working time when reworking. It will be described in detail below.

Next, the server calculates an object-based rejection rate with respect to the rejected job result (S340).

Here, the object-based rejection rate is a rejection rate based on the number of rejected objects among one or more objects included in one job, and is different from the rejection rate, that is, the rejection rate measured in units of workers. Usually, the rejection rate is 30% when one worker performs 10 tasks and 3 of them are rejected.

In other words, the server calculates the ratio of the number of objects included in each work result to the number of objects included in each work result as a rejected object as a rejection rate based on the object of each work result. .

Referring to FIG. 9, when one job includes 4 objects, job result A is calculated as 25%, and job result B is 3 of 4 because 1 object was rejected. Since 2 objects were rejected, the object-based rejection rate is calculated as 75%.

Next, referring again to FIG. 8, the server sets the second minimum working time of the project based on the object-based rejection rate (S350).

Specifically, the second minimum working time is set by multiplying the first minimum working time by the object-based rejection rate.

For example, when the first minimum working time is 120 seconds and the object-based rejection rate is 0.25, the second minimum working time may be set to 30 seconds.

In other words, the operator 32 had to wait 120 seconds (3 minutes) in the past to input the rework result of modifying only one object out of four, but object-based rejection based on the actual object unit returned. As the rate is applied, only about 30 seconds, which is the time required for actual work, needs to be allotted, which has the advantage of reducing the overall work time.

On the other hand, if the work unit price for all tasks belonging to one project is the same, the first minimum work time is the same for all tasks, but the second minimum work time is object-based rejection rate because object-based rejection rate is applied. If these are different, they may be set differently for each task.

In contrast, when the work unit cost for each work belonging to one project is different, the first and second minimum work hours may be set differently for each work.

Next, the server activates the work result input interface after the second minimum work time has elapsed and receives the rework result from the worker (S360).

Like the first minimum working time, according to an embodiment of the present invention, the interface for receiving the rework result may be activated only after the second minimum working time has elapsed. That is, in order for the worker 32 to transmit the rework result to the server, the second minimum working time must elapse.

Meanwhile, in the above description, steps S11 to S360 may be further divided into additional steps or may be combined into fewer steps, according to an embodiment of the present invention. In addition, some steps may be omitted as necessary, and the order between steps may be changed. In addition, even if other contents are omitted, the contents of FIG. 10 to be described later can also be applied to the method of setting the minimum working time using working hours for each functional element of the crowdsourcing-based project of FIGS.

Hereinafter, an apparatus 400 for setting a minimum working time using working hours for each functional element of a crowdsourcing-based project according to an embodiment of the present invention will be described with reference to FIG. 10.

10 is a view for explaining a device for setting a minimum working time using working hours for each functional element of a crowdsourcing-based project according to an embodiment of the present invention.

Referring to FIG. 10, the minimum work time setting device 400 (hereinafter, the minimum work time setting device) utilizing work time for each functional element of a crowdsourcing-based project includes a communication module 410, a memory 420, and a processor ( 430).

The communication module 410 transmits a crowdsourcing-based work for one project to a plurality of workers 32 to request a task to be performed, and receives work results from the plurality of workers 32. The work results received from the plurality of workers 32 are transmitted to the plurality of inspectors 34 to request inspection performance, and the inspection results are received from the plurality of inspectors 34.

The communication module 410 requests the operator 32 to perform rework for a job result (hereinafter referred to as a rejected job result) received as a rejection of the inspection result, and receives the rework result from the operator 32.

In the memory 420, a program for setting the minimum working time of the project using the working time for each functional element based on the data received from the communication module 210, and adjusting the minimum working time based on the target duration of the project Is saved.

The processor 430 executes a program stored in the memory 420. As the processor 430 executes the program stored in the memory 420, it extracts previous project history information for a plurality of functional elements included in the project, and works hours per each functional element based on the previous project history information. Is calculated, the work time per work (hereinafter, estimated work time) of the project is estimated by summing the work time per functional element, and the first minimum work time of the project is set based on the expected work time.

In addition, the processor 430 estimates the time required for the project based on the projected work time (hereinafter, the projected time required), compares the projected time required and the target time required, Adjust the minimum working time.

In addition, the processor 430 calculates an object-based rejection rate with respect to the rejected work result, and sets a second minimum working time of the project based on the object-based rejection rate.

The minimum working time setting apparatus 400 described with reference to FIG. 10 may be provided as a component of the server described above.

The method for setting the minimum working time using working time for each functional element of a crowdsourcing-based project according to an embodiment of the present invention described above is implemented as a program (or application) in order to be executed in combination with a computer, which is hardware. Can be stored in.

The above-described program includes C, C++, JAVA, Ruby, which can be read by a processor (CPU) of the computer through the device interface of the computer, in order for the computer to read the program and execute the methods implemented as a program. It may include a code (Code) coded in a computer language such as machine language. Such code may include a functional code related to a function defining necessary functions for executing the methods, and a control code related to an execution procedure necessary for the processor of the computer to execute the functions according to a predetermined procedure. can do. In addition, such code may further include additional information required for the processor of the computer to execute the functions or code related to a memory reference to which location (address address) of the internal or external memory of the computer should be referenced. have. In addition, when the processor of the computer needs to communicate with any other computer or server in the remote in order to execute the functions, the code uses the communication module of the computer to determine how It may further include a communication-related code for whether to communicate or what information or media to transmit and receive during communication.

The stored medium is not a medium that stores data for a short moment, such as a register, cache, memory, etc., but a medium that stores data semi-permanently and can be read by a device. Specifically, examples of the storage medium include, but are not limited to, ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, and the like. That is, the program may be stored in various recording media on various servers to which the computer can access, or on various recording media on the user's computer. Further, the medium may be distributed over a computer system connected through a network, and computer-readable codes may be stored in a distributed manner.

The above description of the present invention is for illustrative purposes only, and those of ordinary skill in the art to which the present invention pertains will be able to understand that it can be easily modified into other specific forms without changing the technical spirit or essential features of the present invention. will be. Therefore, it should be understood that the embodiments described above are illustrative in all respects and not limiting. For example, each component described as a single type may be implemented in a distributed manner, and similarly, components described as being distributed may also be implemented in a combined form.

The scope of the present invention is indicated by the claims to be described later rather than the detailed description, and all changes or modified forms derived from the meaning and scope of the claims and their equivalent concepts should be interpreted as being included in the scope of the present invention do.

10: Client
20: service provider
30: public
32: worker
34: inspector
400: minimum working time setting device
410: communication module
420: memory
430: processor

Claims (11)

As a method performed by a crowdsourcing platform server (hereinafter, server),
Determining, by the server, a plurality of functional elements included in a crowdsourcing based project (hereinafter, referred to as a project) to be opened;
Extracting, by the server, previous project history information including the corresponding functional element for each functional element;
Calculating, by the server, working hours per functional element based on the previous project history information;
Estimating, by the server, a work time per work (hereinafter, an expected work time) of the project by summing work hours per functional element;
Setting, by the server, a first minimum work limit time of the project based on the expected work time;
Assigning, by the server, a task of the project to a worker and requesting the worker terminal to perform the task; And
And receiving, by the server, activating a work result input interface after the first minimum work limit time elapses, and receiving a work result from the worker terminal,
The step of setting, by the server, a first minimum work limit time of the project,
The time corresponding to the first ratio of the expected work time is set as the first minimum work time limit of the project,
The functional element is determined based on a work tool for performing a project, and the work tool is provided in the project and is a tool used by workers to perform work required by the project,
Assigning, by the server, the work result to an inspector and requesting the inspector terminal to perform inspection;
Receiving, by the server, an inspection result for the work result from the inspector terminal as passing or rejecting the inspection;
Requesting, by the server, to perform rework to a worker terminal for a job result (hereinafter referred to as a rejected job result) received as a rejection of the inspection result;
Calculating, by the server, an object-based rejection rate for the rejected job result;
Setting, by the server, a second minimum work limit time of the project based on the object-based rejection rate; And
The server further comprises the step of receiving a rework result from the worker terminal by activating a work result input interface after the second minimum work limit time elapses,
The step of receiving, by the server, the inspection result,
The result of the inspection of the job result is received as inspection pass or rejection for each object, but if the number of rejected objects among one or more objects included in the job is at least one, the job result is rejected as a whole,
The object-based rejection rate is calculated based on the number of rejected objects among one or more objects included in the job,
The step of setting, by the server, a second minimum work limit time of the project based on the object-based rejection rate,
To set the second minimum work limit time by multiplying the first minimum work limit time by the object-based rejection rate,
A method of setting minimum working hours using working hours for each functional element of a crowdsourcing-based project. The method of claim 1,
The previous project history information,
Including at least one of the total work time, the total number of work, the average work time, the number of rejections, the work time of the rejected work, and the work time taking into account the number of rejections in the previous project in which the work tool corresponding to the functional element was used,
A method of setting minimum working hours using working hours for each functional element of a crowdsourcing-based project. The method of claim 2,
The total working time is generated by recording the use start time and end time of the work tool corresponding to the functional element in the previous project,
A method of setting minimum working hours using working hours for each functional element of a crowdsourcing-based project. The method of claim 1,
Estimating, by the server, a period required for the project (hereinafter, an expected required period) based on the expected working time;
Comparing, by the server, an expected duration and a target duration of the project; And
The server further comprising, according to the comparison result, adjusting the first minimum work time limit of the project,
A method of setting minimum working hours using working hours for each functional element of a crowdsourcing-based project. The method of claim 4,
The step of adjusting, by the server, the first minimum work limit time of the project according to the comparison result,
If the estimated time required is shorter than the target time required, maintaining the first minimum work limit time of the project at a time corresponding to a first ratio of the expected work time,
The first ratio is set based on the number of possible tasks per unit time and a standard wage per unit time,
A method of setting minimum working hours using working hours for each functional element of a crowdsourcing-based project. The method of claim 4,
The step of adjusting, by the server, the first minimum work limit time of the project according to the comparison result,
If the estimated duration is longer than the target duration, adjusting the first minimum work limit time of the project to a time corresponding to a second ratio of the target duration,
A method of setting minimum working hours using working hours for each functional element of a crowdsourcing-based project. The method of claim 4,
The step of estimating, by the server, a period required for the project (hereinafter, an expected required period) based on the expected working time,
Estimating the estimated time required by multiplying the target work quantity of the project by the estimated work time,
A method of setting minimum working hours using working hours for each functional element of a crowdsourcing-based project. delete delete The method of claim 1,
The first minimum work time limit of the first task and the second task of the project having the same work unit cost are set to be the same,
According to different object-based rejection rates, the second minimum task time limit of the first task and the second task are set differently,
A method of setting minimum working hours using working hours for each functional element of a crowdsourcing-based project. A computer stored in a computer-readable recording medium in order to perform a method of setting the minimum working time using working hours for each functional element of a crowdsourcing-based project in any one of paragraphs 1 to 7, in combination with a computer. program.

Images