KR102156583B1 - Method for evaluating sincerity of worker using actual working time of crowdsourcing based projects for artificial intelligence training data generation - Google Patents

Abstract

Provided is a method of evaluating sincerity of workers using actual working time of crowd sourcing-based projects for artificial intelligence (AI) training data generation. The method comprises the steps of: allocating a plurality of works of the crowd sourcing-based projects to a plurality of workers and requesting the workers to perform the works; receiving an input of a plurality of work results from the plurality of workers; measuring the work possessing time of each work based on the point of time when each work is started and the point of time when each work is completed and the work results are inputted for each work allocated to each worker; measuring an actual time spent for each work based on the point of time when each worker accessed an online work space for performing each work; and evaluating the sincerity of each worker based on the work possessing time and the actual time spent for each work. The method of evaluating sincerity of workers using actual working time of crowd sourcing-based projects for AI training data generation is able to improve the processing speed of the whole projects.

Description

{METHOD FOR EVALUATING SINCERITY OF WORKER USING ACTUAL WORKING TIME OF CROWDSOURCING BASED PROJECTS FOR ARTIFICIAL INTELLIGENCE TRAINING DATA GENERATION}

The present invention relates to a method for evaluating the integrity of a worker using actual working hours of a crowdsourcing-based project for generating artificial intelligence learning data.

Recently, more and more companies collect and process large amounts of data on a crowdsourcing basis that engages the general public in some process of corporate activities. In other words, by opening a project and allowing the general public, that is, workers to participate in the project, necessary information is collected through the work results completed by the workers.

At this time, in order to collect more reliable information, the company assigns the work result completed by the worker to the inspector to perform the inspection work.

Specifically, when one project is opened, a plurality of tasks are assigned to each of the plurality of workers. Each worker performs a plurality of tasks assigned to it, and provides the task results. Thereafter, a plurality of inspection tasks are assigned to each of the plurality of inspectors, and each inspector performs a plurality of inspection tasks assigned to them.

On the other hand, when a job is assigned to a worker, the assigned job is occupied by a specific worker until the specific worker submits the job. In this case, if the worker assigned the job does not quickly submit the job, the job cannot be done by other workers.

In order to solve the problem of the worker environment, which is difficult to control in the online environment, it is necessary to reassign the job after evaluation for the worker who only occupies the job and does not submit the job quickly.

Unexamined Patent Publication No. 10-2014-0095956, 2014.08.04.

The problem to be solved by the present invention is to measure the job occupancy time and the actual time required for each job to evaluate the sincerity of the worker, and to withdraw the work data during or scheduled work of the unfaithful worker to manage the work of the unfaithful worker. It is to provide a method of evaluating the integrity of workers using the actual working hours of crowdsourcing-based projects.

However, the problem to be solved by the present invention is not limited to the problem as described above, and other problems may exist.

In order to solve the above-described problems, the method for evaluating the integrity of a worker using the actual working time of a crowdsourcing-based project according to an aspect of the present invention is to allocate a plurality of tasks of a crowdsourcing-based project (hereinafter, a project) to a plurality of workers. Requesting to perform a task, receiving a plurality of task results from the plurality of workers, and inputting the task result by starting the task execution time and completing the task execution for each task assigned to each worker Measuring the occupied time for each task based on one time, measuring the actual time required for each task based on the time each worker accesses the online workspace for performing the task for each task And evaluating the sincerity of each worker based on the occupancy time and the actual required time.

In some embodiments of the present invention, the step of evaluating the sincerity of each worker based on the occupied time and the actual required time may be performed for each job of each worker based on the occupied time and the actual required time. The sincerity level is evaluated, and the sincerity level of each worker can be evaluated by summing and averaging the level of integrity for each task.

Some embodiments of the present invention may further include the step of withdrawing one or more tasks assigned to an unfaithful worker whose sincerity is less than or equal to a predetermined first reference value among the plurality of workers and reassigning it to another worker.

In some embodiments of the present invention, the step of reassigning one or more tasks assigned to the unfaithful worker and reassigning them to another worker may include at least one task in which the unfaithful worker has started performing the task but has not completed the task. Can be reassigned to the other worker.

In some embodiments of the present invention, the step of reassigning one or more tasks assigned to the insincere worker and reassigning it to another worker, the insincere worker re-assigning one or more tasks that have not started performing the task to the other worker. Can be assigned.

In some embodiments of the present invention, the step of reassigning one or more tasks assigned to the unfaithful worker and reassigning them to other workers may be initiated when the plurality of tasks of the project are all assigned to the plurality of workers. I can.

Some embodiments of the present invention may further include limiting the work participation of the unfaithful worker when the number of the one or more tasks that have been withdrawn from the assignment is greater than or equal to a predetermined limit.

Some embodiments of the present invention may further include increasing the task assignment limit of a sincere worker whose sincerity is equal to or greater than a predetermined second reference value among the plurality of workers by a predetermined multiple of a general worker.

In some embodiments of the present invention, the step of evaluating the sincerity of each worker based on the occupied time and the actual required time includes the number of tasks that each worker has completed performing the task at least a predetermined minimum number. At this point, you can evaluate the integrity of each worker.

A computer program according to another aspect of the present invention for solving the above-described problems is combined with a computer as hardware to execute a method of evaluating the integrity of the operator using the actual working time of the project, and is stored in a computer-readable recording medium. .

Other specific details of the present invention are included in the detailed description and drawings.

According to the present invention described above, it is possible to increase the speed of progress of the entire project by limiting the withdrawal and participation in work of the unscrupulous worker by evaluating the sincerity of the worker, and preventing intentional work congestion.

In addition, it is possible to allow a general worker or a sincere worker to rearrange the withdrawn work of an unfaithful worker, so that other workers are given more work opportunities.

The effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the following description.

1 is a conceptual diagram of a crowdsourcing service according to an embodiment of the present invention.
2 is a flowchart illustrating a crowdsourcing-based project progress process according to an embodiment of the present invention.
3 is a flowchart of a method for evaluating the integrity of a worker using actual working hours of a crowdsourcing-based project according to an embodiment of the present invention.
4 is a diagram for explaining occupation time for each job and actual time required for each job.
5 is a view for explaining the integrity of each task.
6 is a view for explaining the content of evaluating the sincerity of the worker based on the sincerity of each job.
7 is a view for explaining an apparatus for evaluating the integrity of a worker using actual working hours of a crowdsourcing-based project according to an embodiment of the present invention.

Advantages and features of the present invention, and a method of achieving them will become apparent with reference to the embodiments described below in detail together with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but may be implemented in a variety of different forms, only the present embodiments are intended to complete the disclosure of the present invention, It is provided to fully inform the technician of the scope of the present invention, and the present invention is only defined by the scope of the claims.

The terms used in the present specification are for describing exemplary embodiments and are not intended to limit the present invention. In this specification, the singular form also includes the plural form unless specifically stated in the phrase. As used in the specification, “comprises” and/or “comprising” do not exclude the presence or addition of one or more other elements other than the mentioned elements. Throughout the specification, the same reference numerals refer to the same elements, and “and/or” includes each and all combinations of one or more of the mentioned elements. Although "first", "second", and the like are used to describe various elements, it goes without saying that these elements are not limited by these terms. These terms are only used to distinguish one component from another component. Therefore, it goes without saying that the first component mentioned below may be the second component within the technical idea of the present invention.

Unless otherwise defined, all terms (including technical and scientific terms) used in the present specification may be used as meanings that can be commonly understood by those of ordinary skill in the art to which the present invention belongs. In addition, terms defined in a commonly used dictionary are not interpreted ideally or excessively unless explicitly defined specifically.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a conceptual diagram of a crowdsourcing service according to an embodiment of the present invention.

Referring to FIG. 1, a crowdsourcing service is performed by being composed of a client 10, a service provider 20, and a public 30.

The sponsor 10 refers to a company or individual who requests a crowdsourcing-based project (hereinafter, referred to as a project).

The requester 10 requests a project for the purpose of collecting source data or annotating data for generating artificial intelligence learning data. The data generated through the project can be used as learning data for arbitrary machine learning such as supervised learning, unsupervised learning, and reinforcement learning. Collection of source data refers to collecting raw data such as recorded voice collection and photo collection. Data annotation refers to inputting relevant annotation data into source data such as text, photos, and videos. For example, the data annotation may include finding an entity in a given fingerprint or finding a similar sentence, but is not limited thereto. Meanwhile, the types of the above-described projects are only one embodiment, and various projects may be handled in the present invention according to the design of the client.

The service provider 20 refers to a company that provides crowdsourcing services.

When the service provider 20 receives a request for a project for a product or service from the client 10, the service provider 20 allocates a task for the project to the general public 30 and receives the work result from the public 30. Thereafter, the final product extracted based on the work result is provided to the client 10.

At this time, the service provider 20 provides a crowdsourcing service to the client 10 and the public 30 through a crowdsourcing platform (hereinafter, the platform). That is, when the service provider 20 receives a request for a project from the client 10, the service provider 20 opens the project on the platform. Thereafter, when the work result for the open project is received from the public 30, the project is terminated on the platform, and the final product may be extracted and provided to the client 10.

The public 30 refers to the general public participating in the project open on the platform. Here, the public 30 may participate in a project opened on the platform through an application or website provided by the service provider 20.

The public 30 is composed of a worker 32 and an inspector 34.

The worker 32 decides to participate in a specific project among a plurality of projects open on the platform. Thereafter, the operator 32 performs an operation such as collection of source data or data annotation, and transmits it to the platform.

The inspector 34 decides to participate in a specific project among a plurality of projects open on the platform. Thereafter, the inspector 34 inspects the result of the work performed by the operator 32. As a result of performing the inspection, the inspector 34 may perform inspection pass processing or rejection processing, and input a rejection reason during rejection processing. In the case of passing the inspection, it is not necessary to rework and re-examination due to this, so passing inspection has the same meaning as completion of inspection.

2 is a flowchart illustrating a crowdsourcing-based project progress process according to an embodiment of the present invention.

First, the requester 10 requests one or more projects to the service provider 20 (S11).

Thereafter, the service provider 20 opens the requested project on the platform (S12). At this time, the service provider 20 may determine the grade in consideration of the difficulty of the project before opening the project. That is, depending on the difficulty level, it is possible to determine which level or higher public 30 to expose the project. Accordingly, it is possible to increase the reliability of the work result of the project.

Thereafter, the service provider 20 requests the work by assigning the work to the workers 32 of the corresponding level or higher according to the level of the project (S13).

Thereafter, the worker 32 performs the assigned task (S14). In this case, the worker 32 may input the reason for the inability to work without performing the work for a work in which the work itself is impossible for some reason.

Thereafter, the service provider 20 receives the work result from the worker 32 (S15), and assigns the work result to the inspector 34 to request the inspection (S16).

Similarly, according to an embodiment of the present invention, only a suitable project may be exposed to the inspector 34 out of the total projects being performed according to the level set according to the difficulty of the project or the qualification requirements of the inspector 34.

Thereafter, the inspector 34 performs the assigned inspection (S17). At this time, the inspector 34 determines that the inspection is completed when it is determined that the work has been properly performed, and if it is determined that the inspection operation is wrong, the inspector 34 rejects it. During rejection processing, the inspector 34 inputs the rejection reason for what reason the job was judged to be wrong.

Thereafter, the service provider 20 receives the inspection result from the inspector 34 (S18).

When the inspection result is the inspection completion, the service provider 20 uses the work result as valid data, and based on this, extracts the final product at the end of the project.

When the inspection result is rejection processing, the service provider 20 may internally perform the inspection again, or assign the work to the worker 32 again to perform the rework. When reworking, re-examination by an inspector is required.

Thereafter, when the project period ends or sufficient valid data is secured, the service provider 20 terminates the project (S19), calculates a final result based on the secured valid data, and provides it to the client 10 ( S20).

At this time, before the end of the project, the service provider 20 evaluates the performance results of the worker 32 and the inspector 34, calculates the work cost and the inspection cost according to the evaluation, to the worker 32 and the inspector 34. give.

In FIGS. 1 and 2, it is simply expressed as a requester 10, a service provider 20, an operator 32, and an inspector 34, but these are smartphones, tablets, PDAs, laptops, desktops, operated by each participant. It means a computer device such as a server or a telecommunication device.

Meanwhile, in allocating tasks in the project to the workers 32, the tasks assigned to the specific workers 32 are bound to the specific workers 32 until the corresponding workers 32 submit the work. In this case, if the worker 32 to whom the job is assigned does not quickly submit the job, the job does not proceed, as well as a problem that other workers cannot perform the job instead.

In order to solve the problem of such a remote uncontrolled worker environment, an embodiment of the present invention measures the time required for the actual work of the workers 32 and evaluates the integrity of each worker 32 based on this. In addition, by allowing the work of inconscientious workers to be reassigned to other workers, it is possible to prevent intentional congestion and speed up the progress of the entire project.

Hereinafter, a method for evaluating the integrity of the worker 32 using actual working time of a crowdsourcing-based project according to an embodiment of the present invention will be described with reference to FIGS. 3 to 6.

3 is a flowchart of a method for evaluating the integrity of the worker 32 using the actual working time of a crowdsourcing-based project according to an embodiment of the present invention. 4 is a diagram for explaining occupation time for each job and actual time required for each job.

Meanwhile, the steps shown in FIG. 3 may be understood as being performed by a platform server (hereinafter, referred to as a server) operated by the service provider 20, but are not limited thereto.

In addition, a plurality of workers 32 or a plurality of inspectors 34 perform work using a predetermined terminal device. The terminal device of the operator 32 or the inspector 34 may be a computer device or a telecommunication device such as a smartphone, tablet, PDA, laptop, desktop, etc., but is not limited thereto.

First, referring to FIG. 3, the server assigns a plurality of tasks of a crowdsourcing-based project (hereinafter referred to as a project) to a plurality of workers 32 and requests the task to be performed (S110), and a plurality of workers 32 A plurality of work results are inputted from (S120).

Here, one project may contain multiple jobs, and the plurality of jobs may be grouped into a plurality of tasks according to their type, or each job may be a different type of job or the same job. , But is not limited thereto.

Next, the occupied time for each task is measured based on the time when the task starts to be performed for each task assigned to each worker 32 and the time when the task is completed and the task result is input (S130).

Here, the occupied time for each job means the time at which job data is allocated. Specifically, the time to which the work data is allocated refers to the time when the worker 32 started performing the work and the time when the work result was input, and the time when the work started was assigned by the worker 32 according to the embodiment. It may be the time it was received. In this embodiment, when the worker 32 requests the next task after completing any one task, the next task is assigned and the task can be started.

As an exception, when a plurality of tasks are assigned as a unit, the time when the task starts to be performed may not be recorded as the time at which the task is assigned. That is, when a plurality of tasks are assigned at the same time, if the worker 32 starts any one of the plurality of tasks, the other tasks are in a waiting state for commencement, so that the assigned time cannot be recorded as the time when the task starts to be performed. do. Therefore, in the above case, the time occupied by each task is measured by individually checking the starting time of the task.

At this time, in the case of the above exceptional example, the time when one task result was entered and the time when the next task was started is checked among the multiple tasks assigned, and the start time between tasks, that is, one task is completed and the next task starts. It goes without saying that the sincerity of the worker 32 whose time until the following is too delayed may be managed separately or together with the sincerity to be described later.

Next, the actual time required for each task is measured based on the time when each worker 32 accesses the online workspace for performing the task for each task (S140).

Here, the online work space means a space in which online work such as a web page or a page on an application or a specific URL provided by a server can be performed.

At this time, in order to more accurately measure the actual time required, an embodiment of the present invention may measure the response of the worker 32. For example, in order to accurately measure the actual time required even when the worker 32 is eating or doing personal business while accessing the online workspace, in the case of a PC, movement of a mouse or keystroke is periodically performed. Can be measured. In the case of mobile, it is possible to check whether a camera is used, an acceleration/gyro sensor is used, a touch input, or whether an application for performing a task is activated.

Referring to FIG. 4, the occupied time for each task and the actual time required for each task are as follows.

First, a specific worker 32 started performing a certain task at 17:35:00 on December 11, and stopped for 12 minutes after performing the task for 30 seconds. After that, I reconnected to the online workspace at 17:47:30, performed it for 20 seconds, and then stopped for 12 minutes. Finally, after accessing the online workspace again at 17:59:50 time and executing it for 10 seconds, the work result was entered at 18:00:00 time.

In this case, the total time that the worker 32 occupied one work is 1500 seconds, but the actual work time is only 60 seconds.

In order to minimize the discrepancy between the time occupied by a task and the actual work time as described above, an embodiment of the present invention measures the occupation time (1500 seconds) for each task and the actual time required (60 seconds) for each task. After, it is characterized by using it.

Next, the integrity of each worker 32 is evaluated based on the occupied time and the actual required time (S150).

In one embodiment, the integrity of each worker 32 may be evaluated when the number of tasks that each worker 32 has completed is equal to or greater than a predetermined minimum number.

In other words, when evaluating the sincerity of the worker 32 targeting too few cases, it is the same as a relative penalty is given until a delay of 1-2 times due to the circumstances of the worker 32. Since a problem of lowering motivation occurs, and the reliability of the measured integrity of the worker 32 is lowered, the sincerity of the worker 32 is evaluated from a point when a certain number or more is satisfied.

5 is a view for explaining the integrity of each task. 6 is a view for explaining the content of evaluating the sincerity of the worker 32 based on the sincerity of each job.

In one embodiment, in order to evaluate the integrity of each worker 32, first, based on the occupied time and actual required time, the sincerity of each worker 32 for each job may be evaluated. In this case, the integrity of each task may be evaluated as a value obtained by dividing the actual work time for each task by the occupied time. After the sincerity of each job is evaluated, the sincerity of each worker 32 can be evaluated by summing and averaging the integrity of each job.

Referring to FIG. 5, the operator 32 started performing work at 18:01 on December 11, and when submitting the work result at 18:02, the occupation time and actual time required are the same as 60 seconds, respectively. Do. In this case, the integrity of the work is evaluated as 1.

That is, the maximum value of the job-specific integrity is 1, and the maximum value of the worker's sincerity calculated by summing and averaging them is also 1.

According to this method, the sincerity for each job of the worker 32 shown in FIG. 6 is calculated as 1/60 for the ID 1001 job, and for ID 1002 and ID 1003 jobs. Each job's integrity is calculated as 1, and in the case of ID 1004, the job's integrity is calculated as 1/1329.

In addition, by summing and averaging each job's sincerity for four cases, the sincerity of the worker 32 is calculated as 0.5049.

On the other hand, in an embodiment of the present invention, when the occupancy time and actual required time for each task are calculated, the step of evaluating the integrity of each task is omitted, and the occupation time and actual required time for each task are reduced. It goes without saying that it is also possible to evaluate the integrity of each worker 32 by summation and evaluation.

When the sincerity of the worker 32 is evaluated, afterwards, an embodiment of the present invention allocates and withdraws one or more tasks assigned to an unfaithful worker whose sincerity is less than or equal to a predetermined first reference value among the plurality of workers 32. Tasks that have been withdrawn can be reassigned.

In this case, the predetermined first reference value, which is a comparison criterion for the level of integrity, may be a specific priority when the levels of integrity of the plurality of workers 32 are sorted in descending order. Alternatively, it may be a specific value. For example, the first reference value may be an average value of the sincerity of the workers 32 participating in the project. That is, workers 32 who have a degree of conscientiousness less than or equal to a specific ranking or less than a specific value are selected as unfaithful workers.

In an embodiment of the present invention, when selected as an unfaithful worker, a task assigned to an unfaithful worker may be withdrawn and then assigned to another worker or a sincere worker.

In one embodiment, one or more tasks in which an inconscientious worker has begun to perform a task but have not completed the task may be reassigned to another worker. In other words, after the unfaithful worker starts performing the task, it is possible to take out the occupied task and assign it to another worker or a sincere worker.

Apart from or in conjunction with the above embodiment, one or more tasks for which the unfaithful worker has not started performing the task may be reassigned to another worker. This means that the unfaithful worker removes a task that has not started performing the task and reassigns it to another worker or a sincere worker.

That is, according to an embodiment of the present invention, an ongoing work or a scheduled work of an unfaithful worker may be reassigned to another worker or a sincere worker.

In one embodiment, the reassignment of the withdrawal task of an insincere worker may be initiated when a plurality of tasks of the project are all assigned to the plurality of workers 32. If there is a task that has not been assigned yet, it is better for general workers to receive a new task and carry out the task rather than reassigning and proceeding the withdrawn task of the unfaithful worker, and it is desirable for the unfaithful worker to continue the task. to be.

In another embodiment, it is preferable to allocate the withdrawn work of the insincere worker in consideration of a point in time when there are no tasks left to be performed by another worker or a sincere worker. This is to determine in consideration of the work status of other workers or sincere workers.

For example, if workers A, B, and C are assigned 10 each for a project that needs to perform a total of 30 tasks, and unfaithful worker C is delaying 3 cases, it is decided to reassign 3 cases. When all tasks of A and B have been completed or are almost completed, one or two cases may be reassigned. On the contrary, when the work of workers A and B is not completed, if the work of the unfaithful worker is forcibly assigned, the general workers can feel the burden.

Accordingly, according to an embodiment of the present invention, it is preferable to withdraw and reassign the work of the insincere worker at the time when the completion of the project is delayed by the insincere worker.

Of course, as soon as the project starts, it is possible to determine the integrity of the worker 32 and assign the work of the unfaithful worker to the sincerity worker or general worker.

On the other hand, according to an embodiment of the present invention, the number of withdrawal of a task in which a task is being performed or a task that has not started performing the task may be determined according to the sincerity of an unfaithful worker.

For example, worker A, who is determined to be an insincere worker, withdraws one job being performed, and worker B, who is less insincerity than worker A, may withdraw the two jobs being performed. These criteria may be set in advance by the administrator.

Alternatively, the number of withdrawal of work may be determined so as to be differentiated for each unfaithful worker according to a time reference value that considers the actual time required compared to the occupation time of the unfaithful worker. That is, in the case of an unfaithful worker whose occupation time is longer compared to the actual required time, the number of withdrawn work may be greater than that of other unfaithful workers.

Alternatively, it may be determined by the server in consideration of the completion status of other workers. As a simple example, if there are two workers who have completed all of the work, the two tasks of the unscrupulous worker can be withdrawn and reassigned.

In an embodiment, if the number of one or more tasks that have been withdrawn from assignment of an unfaithful worker is equal to or greater than a predetermined limit, the unfaithful worker's participation in the task may be restricted. Here, the withdrawn task may include one or more of a task being performed and a task scheduled to be performed by an unfaithful worker.

In one embodiment of the present invention, the work of a sincerity worker whose work integrity is equal to or greater than a predetermined second reference value among the plurality of workers 32, similar to the way that a kind of penalty is given to an unfaithful worker based on the sincerity of the worker 32 It is also possible to increase the allocation limit by a predetermined multiple of the average worker.

That is, one embodiment of the present invention can be managed by dividing the worker 32 into a sincerity worker, a general worker, and an insincere worker according to the work integrity, and in the case of an unscrupulous worker, it is possible to withdraw an ongoing task or a scheduled task They are reassigned to workers or conscientious workers. In this case, the sincere worker may be provided with a kind of incentive to receive more tasks by a predetermined multiple than that of a general worker. At this time, the sincere worker may select and receive the amount of work possible within the allocation limit increased by a predetermined multiple.

Meanwhile, in the above description, steps S11 to S150 may be further divided into additional steps or may be combined into fewer steps according to an embodiment of the present invention. In addition, some steps may be omitted as necessary, and the order between steps may be changed. In addition, even if other contents are omitted, the contents of FIG. 7 to be described later can also be applied to the method of evaluating the integrity of the worker 32 using actual working hours of the crowdsourcing-based project of FIGS. 1 to 6.

In addition, although the present invention has described the evaluation of the integrity of the worker 32, it is also possible to evaluate the integrity of the inspector 34 according to an embodiment. In this case, the occupation time for each job is replaced by the occupation time for each inspection, and the actual time required for each job is replaced by the actual time required for each inspection.In the case of an unfaithful inspector, the cases during inspection or waiting for inspection are withdrawn to other general inspectors or integrity inspectors. Can be reassigned.

Hereinafter, the apparatus 100 for evaluating the integrity of the worker 32 using the actual working time of a crowdsourcing-based project according to an embodiment of the present invention will be described.

7 is a diagram for explaining the apparatus 100 for evaluating the integrity of the worker 32 using the actual working time of a crowdsourcing-based project according to an embodiment of the present invention.

Referring to FIG. 7, the apparatus 100 for evaluating the integrity of the worker 32 using the actual working time of the crowdsourcing-based project according to the above 100 includes a communication module 110, a memory 120, and a processor 130.

The communication module 110 transmits a crowdsourcing-based work for one project to a plurality of workers 32 to request a task to be performed, and receives work results from the plurality of workers 32. Further, the work results received from the plurality of workers 32 are transmitted to the plurality of inspectors 34 to request inspection, and the inspection results are received from the plurality of inspectors 34.

The memory 120 stores a program for measuring the actual working time of the crowdsourcing-based project based on the data received from the communication module 110 and evaluating the integrity of the worker 32 based thereon.

The processor 130 executes a program stored in the memory 120. As the processor 130 executes the program stored in the memory 120, each task is performed based on the time when the task starts to be performed and the time when the task is completed and the task result is input for each task assigned to each worker. The occupancy time of each job is measured, and the actual time required for each job is measured based on the time each worker 32 accesses the online work space for performing the job for each job, and the measured occupancy time and actual consumption The integrity of each worker 32 is evaluated based on time.

The apparatus 100 for evaluating the integrity of the worker 32 using the actual working time of the crowdsourcing-based project described with reference to FIG. 7 may be provided as a component of the above-described server.

The method for evaluating the integrity of the worker 32 using the actual working time of a crowdsourcing-based project according to an embodiment of the present invention described above is implemented as a program (or application) to be executed in combination with a computer that is hardware. Can be stored on media.

The above-described program includes C, C++, JAVA, Ruby, which can be read by a processor (CPU) of the computer through the device interface of the computer, in order for the computer to read the program and execute the methods implemented as a program. It may include a code (Code) coded in a computer language such as machine language. Such code may include a functional code related to a function defining necessary functions for executing the methods, and a control code related to an execution procedure necessary for the processor of the computer to execute the functions according to a predetermined procedure. can do. In addition, these codes may further include additional information required for the processor of the computer to execute the functions, or code related to a memory reference to which location (address address) of the internal or external memory of the computer should be referenced. have. In addition, when the processor of the computer needs to communicate with any other computer or server in the remote in order to execute the functions, the code uses the communication module of the computer to determine how It may further include a communication-related code for whether to communicate, what kind of information or media should be transmitted and received during communication.

The stored medium is not a medium that stores data for a short moment, such as a register, cache, memory, etc., but a medium that stores data semi-permanently and can be read by a device. Specifically, examples of the storage medium include, but are not limited to, ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, and the like. That is, the program may be stored in various recording media on various servers to which the computer can access, or on various recording media on the user's computer. In addition, the medium may be distributed over a computer system connected through a network, and computer-readable codes may be stored in a distributed manner.

The above description of the present invention is for illustrative purposes only, and those of ordinary skill in the art to which the present invention pertains will be able to understand that it can be easily modified into other specific forms without changing the technical spirit or essential features of the present invention. will be. Therefore, it should be understood that the embodiments described above are illustrative in all respects and not limiting. For example, each component described as a single type may be implemented in a distributed manner, and similarly, components described as being distributed may also be implemented in a combined form.

The scope of the present invention is indicated by the claims to be described later rather than the detailed description, and all changes or modified forms derived from the meaning and scope of the claims and their equivalent concepts should be interpreted as being included in the scope of the present invention. do.

10: Client
20: service provider
30: public
32: worker
34: inspector
100: worker's integrity evaluation device
110: communication module
120: memory
130: processor

Claims (10)

As a method performed by a computer,
Assigning a plurality of tasks of a crowdsourcing-based project (hereinafter, referred to as a project) to a plurality of workers and requesting the task to be performed;
Receiving a plurality of job results from the plurality of workers;
Measuring a time occupied by each task based on a time when the task was started and a time when the task was completed and the task result was input for each task assigned to each worker;
Measuring an actual time required for each task based on a time each worker accesses the online workspace for performing the task for each task;
Evaluating the integrity of each worker based on the ratio of the actual time required to the occupied time; And
Reassigning one or more tasks assigned to an unfaithful worker whose integrity is equal to or less than a predetermined first reference value among the plurality of workers and reassigning it to another worker,
The step of withdrawing one or more tasks assigned to the unfaithful worker and reassigning it to another worker,
The insincere worker starts performing the task but reassigns one or more tasks that have not completed the task performance to the other worker, or
Reassigning one or more tasks for which the insincere worker has not started performing the task to the other worker,
A method of evaluating the integrity of workers using actual working hours of crowdsourcing-based projects. The method of claim 1,
The step of evaluating the integrity of each worker based on the ratio of the actual time required to the occupied time,
Evaluating the sincerity of each worker based on the ratio of the actual time required to the occupied time, summing and averaging the sincerity of each worker to evaluate the sincerity of each worker,
A method of evaluating the integrity of workers using actual working hours of crowdsourcing-based projects. delete The method of claim 1,
The step of withdrawing one or more tasks assigned to the unfaithful worker and reassigning it to another worker,
To determine the number of one or more tasks to withdraw the assignment according to the sincerity of the insincere worker,
A method of evaluating the integrity of workers using actual working hours of crowdsourcing-based projects. The method of claim 1,
The step of withdrawing one or more tasks assigned to the unfaithful worker and reassigning it to another worker,
Determining the number of one or more tasks to withdraw the assignment according to a ratio of the actual required time to the occupied time of the insincere worker,
A method of evaluating the integrity of workers using actual working hours of crowdsourcing-based projects. The method of claim 1,
The step of withdrawing one or more tasks assigned to the unfaithful worker and reassigning it to another worker,
Started when the plurality of tasks of the project are all assigned to the plurality of workers,
A method of evaluating the integrity of workers using actual working hours of crowdsourcing-based projects. The method of claim 1,
If the number of the assigned withdrawn one or more tasks is greater than or equal to a predetermined limit, limiting the work participation of the insincere worker further comprising,
A method of evaluating the integrity of workers using actual working hours of crowdsourcing-based projects. The method of claim 1,
Increasing the task assignment limit of the sincerity worker whose sincerity is equal to or greater than a predetermined second reference value among the plurality of workers by a predetermined multiple of the general worker,
A method of evaluating the integrity of workers using actual working hours of crowdsourcing-based projects. The method of claim 1,
The step of evaluating the integrity of each worker based on the ratio of the actual time required to the occupied time,
Evaluating the sincerity of each worker at a point in time when the number of tasks each worker has completed performing the task is more than a predetermined minimum number,
A method of evaluating the integrity of workers using actual working hours of crowdsourcing-based projects. In order to perform the method of evaluating the integrity of the worker using the actual working time of the crowdsourcing-based project of any of paragraphs 1, 2, 6 to 9 in combination with a computer, Stored computer programs.

Images