KR102195955B1 - Method for reassigning work having history of return of crowdsourcing based project for artificial intelligence training data generation - Google Patents

Abstract

The present invention relates to a method for reassigning a work with a return history in a crowdsourcing-based project for artificial intelligence learning data generation. The method includes: a step of assigning works of a crowdsourcing-based project (hereinafter, ″project″) to workers and making a work execution request; a step of receiving the input of work results from the workers; a step of assigning the work results to inspectors and making an inspection execution request; and a step of receiving the input of inspection results with respect to the work results from the inspectors as inspection pass or return. The step of assigning the works to the workers and making the work execution request includes: a step of making a re-work execution request in relation to the worker that has input a work result (hereinafter, first work result) in which the inspection result of return is input; and a step of reassigning a work (hereinafter, target work) satisfying a predetermined condition with the first work result input to another worker and making a work execution request.

Description

{METHOD FOR REASSIGNING WORK HAVING HISTORY OF RETURN OF CROWDSOURCING BASED PROJECT FOR ARTIFICIAL INTELLIGENCE TRAINING DATA GENERATION}

The present invention relates to a method for rearranging a task with a history of rejection of a crowdsourcing-based project for generating artificial intelligence learning data.

Recently, more and more companies collect and process large amounts of data on a crowdsourcing basis that engages the general public in some process of corporate activities. In other words, by opening a project and allowing the general public, that is, workers to participate in the project, necessary information is collected through the work results completed by the workers.

At this time, in order to collect more reliable information, the company assigns the work result completed by the worker to the inspector to perform the inspection work.

Specifically, when one project is opened, a plurality of tasks are assigned to each of the plurality of workers. Each worker performs a plurality of tasks assigned to it, and provides the task results. Thereafter, a plurality of inspection tasks are assigned to each of the plurality of inspectors, and each inspector performs a plurality of inspection tasks assigned to them.

The work performed by the worker but returned as a result of the inspection may be newly assigned to another worker due to various reasons, such as the worker's giving up or not performing rework.

Most of these tasks are tasks of high difficulty, so even if they are newly assigned to other workers, they are likely to be rejected again. Accordingly, in such a task, a situation in which a worker assigned a task abandons the task or does not perform rework may be repeatedly assigned to another worker.

As a result, the assignment-task-review process may be repeated continuously for the same task, resulting in a delay in the project schedule, and the inspector's fatigue increases because the inspector must repeatedly perform the inspection for the same task. Problems can arise.

Unexamined Patent Publication No. 10-2014-0095956, 2014.08.04.

The problem to be solved by the present invention is to provide a method for reassigning a task with a history of rejection of a crowdsourcing-based project for generating artificial intelligence learning data.

However, the problem to be solved by the present invention is not limited to the problem as described above, and other problems may exist.

In order to solve the above-described problem, a method of reassigning a task with a rejection history of a crowdsourcing-based project according to an aspect of the present invention is to assign a plurality of tasks of a crowdsourcing-based project (hereinafter, a project) to a plurality of workers. Requesting execution, receiving a plurality of work results from the plurality of workers, assigning the plurality of work results to a plurality of inspectors and requesting to perform inspection, and the plurality of work results from the plurality of inspectors Including the step of receiving input of the inspection result for the inspection pass or rejection, and the step of assigning the plurality of tasks to a plurality of workers and requesting to perform the work, the task result (hereinafter, referred to as the first Requesting rework to the operator who has entered the work result) and reassigning the work (hereinafter referred to as the target work) that satisfies a predetermined condition among the work receiving the first work result to another worker and requesting to perform the work Including the step of, and receiving a plurality of work results from the plurality of workers, the step of receiving a rework result from a worker who has been requested to perform the rework, and reassigning the target work to another worker The step of requesting to perform a task is to provide a result of the first task to the other worker and request to perform a task for the target task.

In some embodiments of the present invention, the step of reassigning the target task to another worker and requesting the task to be performed may include transparently displaying the first task result on the background of the online task execution screen for the target task by the other worker. It may overlap and output to request execution of a task for the target task.

In some embodiments of the present invention, the step of requesting a task to be performed by reassigning the target task to another worker includes a reason for rejection of the first task result and the first task result input from the inspector to the other worker. It may be to provide a request to perform a task for the target task.

In some embodiments of the present invention, the step of receiving a plurality of work results from the plurality of workers includes receiving a work result (hereinafter, a second work result) for the target work from the other worker, and the target Measuring the similarity of the first work result and the second work result for a work, and if the similarity is greater than or equal to a predetermined reference value, providing a reconfirmation request message for the second work result to the other worker. Can include.

In some embodiments of the present invention, the step of measuring the similarity between the first task result and the second task result with respect to the target task includes an Intersection over Union (IoU) between the first task result and the second task result. It may be to measure the similarity by using.

In some embodiments of the present invention, the step of receiving a plurality of work results from the plurality of workers includes receiving a work result (hereinafter, a second work result) for the target work from the other worker, and the target The step of determining whether the first task result and the second task result for the task are the same, and the step of requesting to perform the inspection by assigning the plurality of task results to a plurality of inspectors comprises: the first task result and Only when the results of the second work are different from each other, the step of assigning the second work result to an inspector and requesting to perform the inspection may be included.

In some embodiments of the present invention, the step of requesting to perform a task by reassigning a task (hereinafter, referred to as a target task) that satisfies a predetermined condition among tasks received with the first task result to another worker, wherein the task result is rejected. The task may be requested by assigning a target task whose number of times is greater than or equal to a predetermined reference number to another worker.

In some embodiments of the present invention, the step of requesting to perform a task by reassigning a task (hereinafter, referred to as a target task) that satisfies a predetermined condition among tasks received with the first task result to another worker, to perform the rework The task may be requested to perform the task by assigning a target task, which has received a request for abandonment of rework rather than the rework result, from the requested worker to another worker.

In some embodiments of the present invention, the step of requesting to perform a task by reassigning a task (hereinafter, referred to as a target task) that satisfies a predetermined condition among tasks received with the first task result to another worker, to perform the rework The requested worker may request to perform the task by assigning a target task that has not started to perform the rework within the rework limit period to another worker.

A computer program according to another aspect of the present invention for solving the above-described problem is combined with a computer that is hardware to execute a method of reassigning a task having a rejection history of the crowdsourcing-based project, and is stored in a computer-readable recording medium. .

Other specific details of the present invention are included in the detailed description and drawings.

According to the present invention described above, if the second work result input by the new worker is similar or the same as the first work result input by the previous worker, the submission of the second work result itself is impossible or the second work to the new worker. By providing a reconfirmation request message for the result, it is possible to prevent a new worker from repeating the same mistakes that the previous worker has done.

In addition, when reassigning a job with a rejection history, not only a job guide but also rejection history information (the first job result and the rejection reason for the first job result) is provided, so that new workers who have been reassigned the job should be careful when performing the job. Make it possible to recognize the matter in advance.

Due to this, the quality of work can be improved, and the same process can not be repeated so that the project can be carried out efficiently.

The effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the following description.

1 is a conceptual diagram of a crowdsourcing service according to an embodiment of the present invention.
2 is a flowchart illustrating a crowdsourcing-based project progress process according to an embodiment of the present invention.
3 is a flow chart for explaining a cycle of work and inspection of a crowdsourcing-based project according to an embodiment of the present invention.
4 is a flowchart of a method for reassigning a task with a rejection history of a crowdsourcing-based project according to an embodiment of the present invention.
5 is an exemplary view of a work screen on which a rejection history is displayed according to an embodiment of the present invention.
6 is a flowchart of a method for requesting reconfirmation through determination of similarity according to an embodiment of the present invention.
7 is an exemplary diagram for describing IoU according to an embodiment of the present invention.
8 and 9 are exemplary diagrams for explaining measurement of IoU values between a bounding box of a previous work result and a bounding box of a rework result according to an embodiment of the present invention.
10 is a flowchart of a method for requesting to perform an inspection through determination of whether the same is the same according to an embodiment of the present invention.
11 is a diagram for explaining a device for reassignment of a task having a rejection history of a crowdsourcing-based project according to an embodiment of the present invention.

Advantages and features of the present invention, and a method of achieving them will become apparent with reference to the embodiments described below in detail together with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but may be implemented in a variety of different forms, only the present embodiments are intended to complete the disclosure of the present invention, It is provided to fully inform the technician of the scope of the present invention, and the present invention is only defined by the scope of the claims.

The terms used in the present specification are for describing exemplary embodiments and are not intended to limit the present invention. In this specification, the singular form also includes the plural form unless specifically stated in the phrase. As used in the specification, “comprises” and/or “comprising” do not exclude the presence or addition of one or more other elements other than the mentioned elements. Throughout the specification, the same reference numerals refer to the same elements, and “and/or” includes each and all combinations of one or more of the mentioned elements. Although "first", "second", and the like are used to describe various elements, it goes without saying that these elements are not limited by these terms. These terms are only used to distinguish one component from another component. Therefore, it goes without saying that the first component mentioned below may be the second component within the technical idea of the present invention.

Unless otherwise defined, all terms (including technical and scientific terms) used in the present specification may be used as meanings that can be commonly understood by those of ordinary skill in the art to which the present invention belongs. In addition, terms defined in a commonly used dictionary are not interpreted ideally or excessively unless explicitly defined specifically.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a conceptual diagram of a crowdsourcing service according to an embodiment of the present invention.

Referring to FIG. 1, a crowdsourcing service is performed by being composed of a client 10, a service provider 20, and a public 30.

The sponsor 10 refers to a company or individual who requests a crowdsourcing-based project (hereinafter, referred to as a project).

The requester 10 requests a project for the purpose of collecting source data or annotating data for generating artificial intelligence learning data. The data generated through the project can be used as learning data for arbitrary machine learning such as supervised learning, unsupervised learning, and reinforcement learning. Collection of source data refers to collecting raw data such as recorded voice collection and photo collection. Data annotation refers to inputting relevant annotation data into source data such as text, photos, and videos. For example, the data annotation may include finding an entity in a given fingerprint or finding a similar sentence, but is not limited thereto. Meanwhile, the types of the above-described projects are only one embodiment, and various projects may be handled in the present invention according to the design of the client.

The service provider 20 refers to a company that provides crowdsourcing services.

When the service provider 20 receives a request for a project for a product or service from the client 10, the service provider 20 allocates a task for the project to the general public 30 and receives the work result from the public 30. Thereafter, the final product extracted based on the work result is provided to the client 10.

At this time, the service provider 20 provides a crowdsourcing service to the client 10 and the public 30 through a crowdsourcing platform (hereinafter, the platform). That is, when the service provider 20 receives a request for a project from the client 10, the service provider 20 opens the project on the platform. Thereafter, when the work result for the open project is received from the public 30, the project is terminated on the platform, and the final product may be extracted and provided to the client 10.

The public 30 refers to the general public participating in the project open on the platform. Here, the public 30 may participate in a project opened on the platform through an application or website provided by the service provider 20.

The public 30 is composed of a worker 32 and an inspector 34.

The worker 32 decides to participate in a specific project among a plurality of projects open on the platform. Thereafter, the operator 32 performs an operation such as collection of source data or data annotation, and transmits it to the platform.

The inspector 34 decides to participate in a specific project among a plurality of projects open on the platform. Thereafter, the inspector 34 inspects the result of the work performed by the operator 32. As a result of performing the inspection, the inspector 34 may perform inspection pass processing or rejection processing, and input a rejection reason during rejection processing. In the case of passing the inspection, it is not necessary to rework and re-examination due to this, so passing inspection has the same meaning as completion of inspection.

2 is a flowchart illustrating a crowdsourcing-based project progress process according to an embodiment of the present invention.

First, the requester 10 requests one or more projects to the service provider 20 (S11).

Thereafter, the service provider 20 opens the requested project on the platform (S12). At this time, the service provider 20 may determine the grade in consideration of the difficulty of the project before opening the project. That is, depending on the difficulty level, it is possible to determine which level or higher public 30 to expose the project. Accordingly, it is possible to increase the reliability of the work result of the project.

Thereafter, the service provider 20 requests the work by assigning the work to the workers 32 of the corresponding level or higher according to the level of the project (S13).

Thereafter, the worker 32 performs the assigned task (S14). In this case, the worker 32 may input the reason for the inability to work without performing the work for a work in which the work itself is impossible for some reason.

Thereafter, the service provider 20 receives the work result from the worker 32 (S15), and assigns the work result to the inspector 34 to request the inspection (S16).

Thereafter, the inspector 34 performs the assigned inspection (S17). At this time, the inspector 34 decides to pass the inspection when it is determined that the work has been properly performed, and if it is determined that the inspection work is wrong, the inspector 34 rejects it. During rejection processing, the inspector 34 inputs the rejection reason for what reason the job was judged to be wrong.

Thereafter, the service provider 20 receives the inspection result from the inspector 34 (S18).

If the inspection result passes the inspection, the service provider 20 uses the work result as valid data, and based on this, extracts the final product at the end of the project.

When the inspection result is rejection processing, the service provider 20 may internally perform the inspection again, or assign the work to the worker 32 again to perform the rework. When reworking, re-examination by an inspector is required.

Thereafter, when the project period ends or sufficient valid data is secured, the service provider 20 terminates the project (S19), calculates a final result based on the secured valid data, and provides it to the client 10 ( S20).

At this time, before the end of the project, the service provider 20 evaluates the performance results of the worker 32 and the inspector 34, calculates the work cost and the inspection cost according to the evaluation, to the worker 32 and the inspector 34. give.

In FIGS. 1 and 2, it is simply expressed as a requester 10, a service provider 20, an operator 32, and an inspector 34, but these are smartphones, tablets, PDAs, laptops, desktops, operated by each participant. It means a computer device such as a server or a telecommunication device.

As described above, the project proceeds by circulating the steps S13 to S18, that is, a series of processes of work-inspection until the project is closed after being opened. Hereinafter, with reference to FIG. 3, a method in which the project proceeds while the process of work-inspection is cycled will be described in detail.

3 is a flow chart for explaining a cycle of work and inspection of a crowdsourcing-based project according to an embodiment of the present invention.

Meanwhile, the steps shown in FIG. 3 may be understood as being performed by a platform server (hereinafter, referred to as a server) operated by the service provider 20, but are not limited thereto.

In addition, a plurality of workers 32 or a plurality of inspectors 34 perform work using a predetermined terminal device. The terminal device of the operator 32 or the inspector 34 may be a computer device or a telecommunication device such as a smartphone, tablet, PDA, laptop, desktop, etc., but is not limited thereto.

Referring to FIG. 3, the server assigns a plurality of tasks of a crowdsourcing-based project (hereinafter, a project) to a plurality of workers 32 and requests the task to be performed (S110), and a plurality of tasks from the plurality of workers 32 Receives the result (S120), assigns a plurality of work results to a plurality of inspectors 34 to request inspection performance (S130), and passes the inspection results for a plurality of work results from the plurality of inspectors 34 or It is input as a return (S140).

Here, the plurality of workers 32 and the plurality of inspectors 34 refer to all workers and inspectors participating in the same project.

The server performs each task assignment and inspection assignment as each of the plurality of workers 32 and the plurality of inspectors 34 enters the project.

For example, if the project has a total of 100 tasks and a total of 3 workers (32) and a total of 3 inspectors (34) participate in the project, when worker A enters the project for the first time, the first task is transferred to worker A. When worker B enters the second time, the second task is assigned to worker B, and when worker C enters the third time, the third task is assigned to worker C. After that, when each worker performs the assigned task and inputs the task result, the next task is assigned to each worker. For example, when worker B performs a task first, the server assigns the fourth task to worker B, then when worker C performs the task, the server assigns the fifth task to worker C, and then worker A. When the server performs the task, the server assigns the sixth task to worker A.

The same goes for inspection. When the work result is input from each worker 32, the server allocates the first work result to the inspector A who enters the project first (worker B performs the second work and enters the first), and the inspector B who enters the second Assigns the second task result to (worker C performs the third task and inputs the second time), and assigns the third task result (worker A performs the first task and enters the third time) to inspector C who enters the third time.

In this way, for a total of 100 tasks of the project, the server requests work and inspection from a plurality of workers 32 and a plurality of inspectors 34 in the order in which they entered the project, and receives the work results and inspection results from each. will be.

At this time, the work that passed the inspection as a result of the inspection is used as valid data for the final product, but the rejected work goes through a rework and re-review process again. That is, the rejected operation is continuously repeated steps S110 to S140 until the inspection passes.

That is, when the initial inspection result is received as a rejection for the initial job result, the job is reworked multiple times by the worker 32 who first performed the job until the inspection passes. If the rework is repeated many times, the worker 32 loses the motivation for the work and does not perform the rework anymore or requests to give up work at all.

When the worker 32 does not perform rework for a predetermined period of time or requests to abandon the work, the server withdraws the work assigned to the worker 32 and reassigns the work to another worker 32 newly.

In the conventional case, when a task whose assignment has been withdrawn to another worker 32 is newly reassigned, the previous task information is only provided to the other worker 32 despite the previous work history, that is, a rejection history for the task. The work was reassigned without providing. That is, the other worker 32 performs the work in a zero base even though the rejection history remains. In most cases, tasks whose assignment has been withdrawn are highly difficult, so even if another worker 32 performs the task, it is highly likely to be rejected again, and eventually, it is highly likely to be reassigned to another worker 32 again. As a result, the assignment-task-review process for the same task is repeated continuously, and this is a high difficulty, so the schedule of the project is delayed even though many tasks have been completed by passing the inspection due to the few tasks that are continuously reassigned. It worked as a problem.

In order to solve this problem, an embodiment of the present invention provides the rejected operation results together and rearranges the job when reassigning a job with a rejection history, so that the newly reassigned worker 32 works efficiently. To be able to perform.

Hereinafter, with reference to FIGS. 4 to 10, targeting the job result received as a rejection of the inspection result according to an embodiment of the present invention, it is determined whether each job in which each job result is input satisfies a predetermined condition. , A method of performing reassignment for a task that satisfies a predetermined condition will be described.

4 is a flowchart of a method for reassigning a task having a rejection history of a crowdsourcing-based project according to an embodiment of the present invention. 5 is an exemplary view of a work screen on which a rejection history is displayed according to an embodiment of the present invention. 6 is a flowchart of a method for requesting reconfirmation through determination of similarity according to an embodiment of the present invention. 7 is an exemplary diagram for describing IoU according to an embodiment of the present invention. 8 and 9 are exemplary diagrams for explaining measurement of IoU values between a bounding box of a previous work result and a bounding box of a rework result according to an embodiment of the present invention. 10 is a flowchart of a method for requesting to perform an inspection through determination of whether the same is the same according to an embodiment of the present invention.

Meanwhile, the steps shown in FIGS. 4, 6 and 10 may be understood as being performed by a platform server (hereinafter, referred to as a server) operated by the service provider 20, but are not limited thereto.

First, referring to FIG. 4, the server requests rework to the worker who inputs the job result (hereinafter, referred to as the first job result) received as a rejection of the inspection result (S111), and rework from the worker who is requested to perform the rework. The operation result is input (S121).

Here, the first work result may be an initial work result or a rework result. As described above, this is because the rework is repeatedly performed by the worker 32 who initially performed the corresponding task until the work whose initial work result was rejected is passed through the inspection.

For example, assume that worker A is assigned task A and performs a task. When worker A's initial job result for job A is rejected, the server requests worker A to perform the first rework on job A. In this case, the first work result means the initial work result rejected for work A.

Thereafter, when worker A performs the first rework on job A, but the result of the first rework is also rejected, the server requests worker A to perform the second rework on job A. In this case, the first work result means the result of the first rework that was rejected for work A.

Thereafter, when worker A performs the second rework on job A, but the second rework result is also rejected, the server requests worker A to perform the third rework on job A. In this case, the first work result refers to the result of the second rework that was rejected for work A.

In a state in which rework execution is requested for the job in which the first job result is input, the server determines whether to delete the job from the worker 32 (withdrawal of assignment) and reassign it to another worker 32. Since the situation in which the worker 32 performs rework for a specific task is repeated or the worker 32 may not want to perform the rework, it is determined to delete and reassign a specific task in consideration of these factors. .

Specifically, the server reassigns a job that satisfies a predetermined condition (hereinafter, a target job) among jobs receiving the first job result to another worker and requests the job to be performed (S112).

The server does not reassign the job to another worker just because it is rejected, but extracts only the jobs that meet a predetermined condition from the jobs that have received the first job result, that is, jobs that have been rejected and requested to perform rework. It is to reassign to (32).

In one embodiment, the server requests the other worker 32 to perform the task by assigning a target task whose number of times the task result is rejected is greater than or equal to a predetermined reference number.

That is, as described above, the first work result may be the rejected initial work result, or the rejected primary rework result, or the rejected secondary rework result. The number of times the job result was rejected is different for the received job.

For example, if the first job result entered for a specific job is the rejected 4th rework result, and 5th rework is requested, the number of rejections for a specific job is 5 times (rejection of the initial job result, 1 It can be calculated as the rejection of the result of the primary rework, the rejection of the secondary rework result, the rejection of the third rework result, and the rejection of the fourth rework result).

In this case, if the predetermined number of times is 4, since the number of rejections of the specific job is 5, the server determines that the specific job satisfies a predetermined condition and extracts the specific job as a target job. Thereafter, the server deletes the target task from the worker 32 (withdraws the assignment for performing the fifth rework) and reassigns it to another worker 32 to request the task to be performed.

That the situation in which the worker 32 continuously performs rework for a specific task is repeated means that the worker 32 has poor understanding and ability to perform the task in order to perform the task. Accordingly, when the number of rejections for a specific task exceeds a predetermined reference number, the specific task is reassigned to the other worker 32 so that the other worker 32 performs it, so that the project can proceed efficiently.

In another embodiment, the server assigns a target task, which has received a request for abandoning rework rather than a rework result, from the worker 32 who is requested to perform the rework, to another worker 32 and requests the task to be performed.

The worker 32 who has been requested to perform the rework may determine that it is impossible to perform the rework by himself and request abandonment of the rework.

For example, if the first job result input for a specific job is the rejected initial job result, and when the primary rework is requested, the worker 32 who is requested to perform the primary rework is based on the initial job performance experience. Therefore, you can predict that you will not be able to properly perform the first rework yourself. That's why you can ask yourself to give up rework to avoid wasting time and labor.

When a request to give up rework is input for a specific job, the server determines that the specific job satisfies a predetermined condition and extracts the specific job as a target job. Thereafter, the server deletes the target task from the worker 32 (withdraws the assignment for performing rework) and reassigns it to another worker 32 to request the task to be performed.

When the worker 32 requests to give up rework for a specific task, it means that the worker 32 has poor understanding and ability to perform the task in order to perform the task. Accordingly, when a request for giving up rework is input for a specific task, the specific task is reassigned to the other worker 32 so that the other worker 32 performs it, so that the project can proceed efficiently.

In another embodiment, the server assigns a target task for which the worker 32 requested to perform the rework has not started performing the rework within the rework limit period to another worker 32 and requests the task to be performed.

The worker 32 requested to perform the rework must start performing the rework within the rework limit period. Here, the rework limit period is preset by the server, but is not limited thereto and may be changed according to the project progress.

For example, if the limit for rework is 1 day, if the worker 32 does not start performing rework even after 1 day after receiving a request to perform rework for a specific job, the worker 32 is currently reworked. It is highly likely that this is a situation where you cannot perform.

The server determines that the specific job satisfies a predetermined condition and extracts the specific job as a target job if the rework is not started within the rework limit time limit. Thereafter, the server deletes the target task from the worker 32 (withdraws the assignment for performing rework) and reassigns it to another worker 32 to request the task to be performed.

The fact that the worker 32 does not start performing the rework within the rework limit period for a specific task means that the worker 32 does not have time, etc., to perform the corresponding task. Therefore, if the rework performance for a specific job is not started within the rework limit period, the server does not wait for the worker 32 to perform the rework, but reassigns the specific job to the other worker 32 and the other worker 32 The project can be carried out efficiently by letting it run.

At this time, when reassigning the target task, the server provides the first task result to another worker 32 and requests the task to be performed on the target task.

In an embodiment, the server may request to perform a task for the target task by transparently overlapping and outputting the first task result on the background of an online task execution screen for the target task of another worker 32.

Here, the online task execution screen is a connection screen of a space where online tasks such as a web page or a page on an application or a specific URL provided by a server can be performed, and means a screen displayed to the worker 32 for performing the task.

5A is an example of a bounding operation screen. Here, the bounding operation refers to an operation in which the operator 32 designates one or more objects existing on the source image using a bounding box.

Referring to FIG. 5A, a task result (bounding box) that was rejected by a previous worker 32 is also displayed on the target task, so that the other worker 32 can perform the bounding task while taking this into account. Due to this, the other worker 32 may not repeat the same mistake as the previous worker 32.

5B is an example of a text tagging work screen. Here, the text tagging operation refers to an operation in which the operator 32 tags one or more words in text existing on the source image to designate an area (similar to designating using a bounding box).

Referring to FIG. 5B, a task result (tagged area) that was rejected by a previous worker 32 is displayed in the target task, so that another worker 32 can perform text tagging while taking this into account. Due to this, the other worker 32 may not repeat the same mistake as the previous worker 32.

5C is an example of an alternative work (O/X work) screen. Here, the alternative operation means an operation in which the operator 32 selects an answer with O or X for one or more questions existing on the source image.

Referring to FIG. 5C, a task result (selected answer) that was rejected by the previous worker 32 is displayed together with the target task, so that another worker 32 can perform an alternative task while taking this into account. Due to this, the other worker 32 may not repeat the same mistake as the previous worker 32.

In another embodiment, the server may request another worker 32 to perform a task on the target task by providing the first task result and the reason for rejection of the first task result input from the inspector 34 together.

That is, the server transparently overlaps and outputs the first task result on the background of the online task execution screen for the target task, and displays the reason for rejection of the first task result, so that the other worker 32 By making it possible to accurately check the reason for rejection (which part of the previous worker did wrong work), the other worker 32 can be prevented from repeating the same mistakes as the previous worker 32.

Depending on the embodiment, when there are multiple first work results (work results inputted in return for the inspection results) for the target work, the server transparently overlaps all the first work results on the background of the online work execution screen. And output, or may be output by transparently overlapping only the most recent first job result among the plurality of first job results on the background of the online job execution screen.

Referring to FIG. 4 again, the server receives a job result (hereinafter, a second job result) for a target job from another worker 32 (S122).

When receiving the second job result for the target job from the other worker 32, the server compares the second job result for the target job and the first job result that is the previous job history of the target job, and the other worker 32 You can determine whether or not the job has been properly performed.

In an embodiment, for the bounding task or the text tagging task type, it may be determined whether another worker 32 appropriately performed the task by measuring the similarity between the first task result and the second task result.

Referring to FIG. 6, when receiving a second job result in step S122, the server measures the similarity between the first job result and the second job result for the target job (S123).

In this case, the degree of similarity may be measured using Intersection over Union (IoU) between the first and second results.

Here, the IoU value is a value measured according to the degree of overlap between the bounding box (or tagged area) input as a result of rework and the bounding box (or tagged area) input as a previous work result. Hereinafter, for convenience of description, both the bounding box and the tagged area are expressed as bounding boxes.

Referring to FIG. 7, an IoU value means a ratio of an intersection area to a union area of two bounding boxes. Accordingly, the larger the degree of overlap of the bounding box is, the closer to 1 is measured, and the smaller the degree of overlap is, the closer to 0 is measured.

According to an embodiment, when one bounding box is input as a first task result, an IoU value with one bounding box input as a first task result is measured for each bounding box input as a second task result. I can.

Referring to Figure 8a, when one bounding box (BBOX1) is input in the first operation result and one bounding box (BBOX1') is input in the second operation result, the server depends on the degree of overlap between BBOX1 and BBOX1'. The measured IoU 11' value is measured as the IoU value of the bounding box (BBOX1') of the second work result.

Referring to FIG. 8B, when one bounding box (BBOX1) is input in the first operation result and two bounding boxes (BBOX1' and BBOX2') are input in the second operation result, the server overlaps BBOX1 and BBOX1'. The IoU 11' value measured according to the degree is measured as the IoU value of the bounding box (BBOX1') of the second work result, and the IoU 12' value measured according to the degree of overlap between BBOX1 and BBOX2' is bound to the second work result. It is measured by the IoU value of the box (BBOX2`). At this time, since BBOX1 and BBOX1' match and BBOX1 and BBOX2' do not match, the IoU 12' value is measured as 0 or less than the IoU 11' value.

According to another embodiment, when a plurality of bounding boxes are input as a first job result, IoU values with all bounding boxes input as a first job result are measured for each of the bounding boxes input as a second job result, The largest IoU value among the measurement results may be measured as the IoU value of each bounding box input as the second work result.

Referring to FIG. 9A, when two bounding boxes (BBOX1 and BBOX2) are input in the first operation result and one bounding box (BBOX1') is input in the second operation result, the server overlaps between BBOX1 and BBOX1'. According to the IoU 11' value is measured, and the IoU 21' value is measured according to the degree of overlap between BBOX2 and BBOX1'. At this time, since BBOX1 and BBOX1' match and BBOX2 and BBOX1' do not match, the IoU 21' value is measured as 0 or less than the IoU 11' value. Accordingly, the server measures the IoU 11' value, which is the larger of the IoU 11' value and the IoU 21' value, as the IoU value of the bounding box BBOX1′ of the second job result.

Referring to FIG. 9B, when two bounding boxes (BBOX1 and BBOX2) are input in the first operation result and two bounding boxes (BBOX1' and BBOX2') are input in the second operation result, the server is BBOX1 and BBOX1'. The IoU 11' value is measured according to the degree of overlap between the liver, and the IoU 21' value is measured according to the degree of overlap between BBOX2 and BBOX1'. At this time, since BBOX1 and BBOX1' match and BBOX2 and BBOX1' do not match, the IoU 21' value is measured as 0 or less than the IoU 11' value. Accordingly, the server measures the IoU 11' value, which is the larger of the IoU 11' value and the IoU 21' value, as the IoU value of the bounding box BBOX1′ of the second job result. In addition, the server measures the value of IoU 12' according to the degree of overlap between BBOX1 and BBOX2', and measures the value of IoU 22' according to the degree of overlap between BBOX2 and BBOX2'. At this time, since BBOX1 and BBOX2' do not match and BBOX2 and BBOX2' match, the IoU 12' value is measured as 0 or less than the IoU 22' value. Accordingly, the server measures the IoU 22' value, which is the larger of the IoU 12' value and the IoU 22' value, as the IoU value of the bounding box BBOX2′ of the second job result.

Referring back to FIG. 6, if the degree of similarity is greater than or equal to a predetermined reference value, the server provides a reconfirmation request message for the second work result to another worker (S124).

As described above, the IoU value is measured close to 1 if the degree of overlap between the two bounding boxes is large, and close to 0 if the degree of overlap is small. That is, when the IoU value is measured close to 1, it means that there is little change in the area of the bounding box input to the second work result compared to the area of the bounding box input to the first work result.

Therefore, when comparing the IoU value with the reference value, the server determines that the area of the bounding box has changed when the IoU value is less than the reference value, and determines that the area of the bounding box has not changed when the IoU value is greater than the reference value.

For example, when the reference value is 0.7 and the IoU value of the bounding box of the second work result is measured as 0.5, it is determined that the area of the bounding box has changed, and the IoU value of the bounding box of the second work result is 0.9. If measured, it is determined that the area of the bounding box has not changed.

That is, if it is determined that the area of the bounding box has changed, the server determines that the other worker 32 has performed the job differently from the first job result input by the previous worker 32, and the area of the bounding box has not changed. If it is determined that, the server determines that the other worker 32 has performed the task similar to the first task result input by the previous worker 32.

At this time, when there are a plurality of bounding boxes of the second operation result, the IoU value of each of the plurality of bounding boxes is compared with a reference value, and if the IoU value of at least one bounding box is less than the reference value, the server It is determined that the task was performed differently from the first task result input by the previous worker 32.

For example, when there are 3 bounding boxes of the second job result, each of the IoU values is 0.5, 0.7, and 0.8, and the reference value is 0.7, one of the 3 IoU values is less than the reference value, so the server It may be determined that (32) performed the task differently from the first task result input by the previous worker (32).

Accordingly, if the degree of similarity is less than the predetermined reference value, the server determines that the second work result has been properly worked and requests the inspector 34 to perform the inspection.

In contrast, if the degree of similarity is greater than or equal to a predetermined reference value, the server determines that the second work result has been inappropriately worked (there is a high probability of being rejected because the work is similar to the first work result) Provides a reconfirmation request message for the second operation result. For example, the server says, “It is judged to be similar to the case where the result of the operation you entered is rejected. Please check the rejection history (result of the first operation or reason for rejection) for the job.” Reconfirmation request message such as, etc. can be provided.

When the other worker 32 receives the reconfirmation request message, it checks the rejection history displayed on the work screen again and submits the corrected or unmodified second work result. After reconfirmation, the server allocates the submitted second work result to the inspector 34 to request the inspection. At this time, even if the second work result has not been modified, it may be assigned to the inspector 34 so that the inspection proceeds.

In another embodiment, for the alternative type of work or other types of work, it may be determined whether another worker appropriately performed the work by determining whether the first work result and the second work result are the same.

Referring to FIG. 10, when receiving a second job result in step S122, the server determines whether the first job result and the second job result for the target job are the same (S125), and the first job result and the second job Only when the results are different, the second task result is assigned to the inspector and the inspection is requested (S131).

Since the alternative task is the task of determining one answer from two views, the other worker 32 transparently overlaps the background of the online task execution screen and selects a different answer from the displayed first task result to perform the task. Just do it.

Accordingly, the server determines that the second work result has been properly worked only when the second work result is different from the first work result and requests the inspector 34 to perform the check.

Alternatively, if the second job result is the same as the first job result, the server may disable the submission of the second job result itself. This is because there is a high possibility that the other worker 32 who has entered the second work result has performed the work unfaithfully without checking the first work result, which is the rejection history. In this case, the server may provide a guide message to the other worker 32 to perform the task appropriately. For example, it is the same as when the result of the operation you entered is rejected. You can provide a message such as "The result of the work cannot be submitted."

On the other hand, the server may allocate a target task in which reassignment to another worker 32 is repeated a predetermined number of times or more to excellent workers of the project. That is, a target task having difficulty in performing a task by several workers 32 is assigned to a worker with excellent work performance so that the target task can be quickly passed through the inspection.

Here, the excellent worker may be selected based on the rejection rate of the worker 32 from among the plurality of workers 32.

That is, when the server assigns a plurality of work results to a plurality of inspectors 34 to request inspection, and receives a plurality of inspection results for a plurality of work results from the plurality of inspectors 34 as passing or rejecting the inspection, Based on the result, the rejection rate of each worker 32 may be calculated.

And, by arranging the rejection rate of each worker 32 in descending order, the workers 32 corresponding to a predetermined higher grade are selected as excellent workers 32, and a target task whose reassignment is repeated a predetermined number of times or more is assigned. This is because the lower the rejection rate, the higher the possibility that the worker 32 can perform the target task of high difficulty.

Alternatively, the server may reassign to another worker 32 by presenting a recommended work area for a target task in which reassignment to another worker 32 is repeated a predetermined number of times or more. That is, by using the rejection history of all the workers 32 who have been assigned the target task as learning data, the recommended work area is extracted, and the recommended work area is presented to the other workers 32 who are assigned the target work next. It is to ensure that the work passes through the inspection quickly.

Meanwhile, in the above description, steps S11 to S131 may be further divided into additional steps or may be combined into fewer steps, according to an embodiment of the present invention. In addition, some steps may be omitted as necessary, and the order between steps may be changed. In addition, even if other contents are omitted, the contents of FIG. 11 to be described later may also be applied to a method of reassigning a task having a rejection history of the crowdsourcing-based project of FIGS. 1 to 10.

Hereinafter, an apparatus 200 for reassigning a task having a rejection history of a crowdsourcing-based project according to an embodiment of the present invention will be described with reference to FIG. 11.

11 is a diagram for explaining a device for reassigning a job with a rejection history of a crowdsourcing-based project according to an embodiment of the present invention.

Referring to FIG. 11, a task reassignment device 200 (hereinafter, a task reassignment device) having a rejection history of a crowdsourcing-based project includes a communication module 210, a memory 220, and a processor 230.

The communication module 210 transmits a crowdsourcing-based work for one project to a plurality of workers 32 to request a task to be performed, and receives a work result from the plurality of workers 32. The work results received from the plurality of workers 32 are transmitted to the plurality of inspectors 34 to request inspection performance, and the inspection results are received from the plurality of inspectors 34.

The communication module 210 requests the operator 32 who has input the job result (hereinafter, referred to as the first job result) received as a rejection of the inspection result to perform rework, and rework from the worker 32 requested to perform the rework. Receive work results.

The communication module 210 reassigns a task (hereinafter referred to as a target task) that satisfies a predetermined condition among tasks receiving the first task result to another worker 32 and requests the other worker 32 to perform the task. Together with the results of the first work. The communication module 210 receives a task result (hereinafter, referred to as a second task result) for the target task from another worker 32.

The memory 220 stores a program for reassigning a job with a rejection history based on data received from the communication module 210.

The processor 230 executes a program stored in the memory 220. As the processor 230 executes the program stored in the memory 220, it extracts a target job that satisfies a predetermined condition from among jobs receiving the first job result.

Specifically, the processor 230 may extract a job whose rejection number is equal to or greater than a predetermined reference number as a target job. In addition, the processor 230 may extract, as a target job, a job that has received a request for abandoning rework rather than a result of the rework from the worker 32 who is requested to perform the rework. In addition, the processor 230 may extract, as a target task, a task that the worker 32 requested to perform the rework has not initiated to perform the rework within the rework limit period.

The processor 230 measures the similarity between the first task result and the second task result with respect to the target task, and determines whether the similarity is greater than or equal to a predetermined reference value. When the similarity is greater than or equal to a predetermined reference value, the communication module 210 provides a reconfirmation request message for the second work result to the other worker 32.

Alternatively, the processor 230 determines whether the first task result and the second task result for the target task are the same, and only when the first task result and the second task result are different from each other, the communication module 210 2 Assign the work results to the inspector 34 to request the inspection.

The job reassignment apparatus 200 described with reference to FIG. 11 may be provided as a component of the above-described server.

The method for reassigning a task having a rejection history of a crowdsourcing-based project according to an embodiment of the present invention described above may be implemented as a program (or application) to be executed in combination with a computer, which is hardware, and stored in a medium. .

The above-described program includes C, C++, JAVA, Ruby, which can be read by a processor (CPU) of the computer through the device interface of the computer, in order for the computer to read the program and execute the methods implemented as a program. It may include a code (Code) coded in a computer language such as machine language. Such code may include a functional code related to a function defining necessary functions for executing the methods, and a control code related to an execution procedure necessary for the processor of the computer to execute the functions according to a predetermined procedure. can do. In addition, such code may further include additional information required for the processor of the computer to execute the functions or code related to a memory reference to which location (address address) of the internal or external memory of the computer should be referenced. have. In addition, when the processor of the computer needs to communicate with any other computer or server in the remote in order to execute the functions, the code uses the communication module of the computer to determine how It may further include a communication-related code for whether to communicate or what information or media to transmit and receive during communication.

The stored medium is not a medium that stores data for a short moment, such as a register, cache, memory, etc., but a medium that stores data semi-permanently and can be read by a device. Specifically, examples of the storage medium include, but are not limited to, ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, and the like. That is, the program may be stored in various recording media on various servers to which the computer can access, or on various recording media on the user's computer. In addition, the medium may be distributed over a computer system connected through a network, and computer-readable codes may be stored in a distributed manner.

The above description of the present invention is for illustrative purposes only, and those of ordinary skill in the art to which the present invention pertains will be able to understand that it can be easily modified into other specific forms without changing the technical spirit or essential features of the present invention. will be. Therefore, it should be understood that the embodiments described above are illustrative in all respects and not limiting. For example, each component described as a single type may be implemented in a distributed manner, and similarly, components described as being distributed may also be implemented in a combined form.

The scope of the present invention is indicated by the claims to be described later rather than the detailed description, and all changes or modified forms derived from the meaning and scope of the claims and their equivalent concepts should be interpreted as being included in the scope of the present invention. do.

10: Client
20: service provider
30: public
32: worker
34: inspector
200: work reassignment device
210: communication module
220: memory
230: processor

Claims (10)

As a method performed by a computer,
Assigning a plurality of tasks of a crowdsourcing-based project (hereinafter, referred to as a project) to a plurality of workers and requesting the task to be performed;
Receiving a plurality of work results from the plurality of workers;
Assigning the plurality of work results to a plurality of inspectors and requesting to perform inspection; And
Including the step of receiving an inspection result of the plurality of work results from the plurality of inspectors as inspection pass or reject,
Assigning the plurality of tasks to a plurality of workers and requesting the task to be performed,
A step of requesting the operator to perform rework to the worker who has input the work result (hereinafter, referred to as the first work result) received as a rejection of the inspection result; and
Reassigning a task (hereinafter referred to as a target task) that satisfies a predetermined condition among tasks receiving the first task result to another worker and requesting the task to be performed,
Reassigning the target task to another worker and requesting the task to be performed,
The first task result is transparently overlapped and output on the background of the online task execution screen for the target task by the other worker, and the reason for rejection of the first task result received from the inspector is provided to the other worker. To request the execution of the task for the target task,
The step of requesting to perform a task by reassigning a task (hereinafter, referred to as a target task) that satisfies a predetermined condition among tasks receiving the first task result to another worker,
The target task whose number of times the work result was rejected is more than a predetermined number of times, the target task that received a rework abandon request other than the rework result from the worker who was requested to perform the rework, or the worker who was requested to perform the rework Reassigning the target task that has not started performing the rework within the rework limit period to another worker to request the task to be performed,
A method of reassigning work that has a history of rejection of a crowdsourced project. delete delete The method of claim 1,
The step of receiving a plurality of work results from the plurality of workers,
Receiving a work result (hereinafter, a second work result) for the target work from the other worker; and
Measuring a similarity between the first task result and the second task result for the target task,
If the similarity is greater than or equal to a predetermined reference value, providing a reconfirmation request message for the second work result to the other worker,
A method of reassigning work that has a history of rejection of a crowdsourced project. The method of claim 4,
Measuring the similarity between the first task result and the second task result for the target task,
To measure the similarity between the first work result and the second work result by using IoU (Intersection over Union),
A method of reassigning work that has a history of rejection of a crowdsourced project. The method of claim 1,
The step of receiving a plurality of work results from the plurality of workers,
Receiving a work result (hereinafter, a second work result) for the target work from the other worker; and
Determining whether the first task result and the second task result for the target task are the same,
Assigning the plurality of work results to a plurality of inspectors and requesting to perform the inspection,
Including the step of requesting to perform the inspection by assigning the second work result to an inspector only when the first operation result and the second operation result are different from each other,
A method of reassigning work that has a history of rejection of a crowdsourced project. delete delete delete A computer program stored in a computer-readable recording medium in order to perform a method of reassignment of a task having a rejection history of the crowdsourcing-based project of any one of claims 1, 4 to 6 in combination with a computer.

Images