KR102205810B1 - Method for automatically returning rework result of crowdsourcing based project for artificial intelligence training data generation - Google Patents

Abstract

Provided is a method for automatically returning rework results of a crowdsourcing-based project for artificial intelligence training data generation. The method comprises the steps of: assigning a plurality of works of a crowdsourcing-based project (hereinafter referred to as a project) to a plurality of workers and requesting to perform the works; receiving a plurality of work results from the workers; assigning the work results to a plurality of inspectors and requesting to perform inspection; receiving, from the inspectors, a plurality of inspection results of the work results as inspection pass or return; requesting a worker who inputs a previous work result to perform rework with respect to a work result (hereinafter referred to as the previous work result) for which an inspection result is input as return; receiving, from the worker, a rework result obtained by correcting the previous work result; determining whether to correct the previous work result by comparing the rework result and the previous work result; and if it is determined that the previous work result is not corrected, automatically determining an inspection result of the rework result as return.

Description

{METHOD FOR AUTOMATICALLY RETURNING REWORK RESULT OF CROWDSOURCING BASED PROJECT FOR ARTIFICIAL INTELLIGENCE TRAINING DATA GENERATION} for generating artificial intelligence learning data

The present invention relates to a method for automatically rejecting rework results of a crowdsourcing-based project for generating artificial intelligence learning data.

Recently, more and more companies collect and process large amounts of data on a crowdsourcing basis that engages the general public in some process of corporate activities. In other words, by opening a project and allowing the general public, that is, workers to participate in the project, necessary information is collected through the work results completed by the workers.

At this time, in order to collect more reliable information, the company assigns the work result completed by the worker to the inspector to perform the inspection work.

Specifically, when one project is opened, a plurality of tasks are assigned to each of the plurality of workers. Each worker performs a plurality of tasks assigned to it, and provides the task results. Thereafter, a plurality of inspection tasks are assigned to each of the plurality of inspectors, and each inspector performs a plurality of inspection tasks assigned to them.

In addition, the work result returned as a result of the inspection is reworked by the operator, and the rework result is re-checked by the inspector.If the operator provides the rework result without modifying the rejected work result when performing the rework, the inspector is rejected. A situation arises in which re-examination of the same rework result as the previously performed work result is required. This acts as a problem that wastes time and labor of the inspector.

In particular, in the case of a project that includes multiple detailed tasks in one task, the inspector must perform the inspection for each detailed task result, so even though the rework result has not been modified compared to the rejected task result, the inspector has multiple tasks. Since detailed work results must be inspected one by one, the problem of wasting the inspector's time and labor can lead to a serious level, and may cause an inspection bottleneck.

Unexamined Patent Publication No. 10-2014-0095956, 2014.08.04.

The problem to be solved by the present invention is to provide a method for automatically rejecting the rework results of a crowdsourcing-based project for generating artificial intelligence learning data.

However, the problem to be solved by the present invention is not limited to the problem as described above, and other problems may exist.

According to an aspect of the present invention for solving the above-described problems, a method for automatically rejecting the rework result of a crowdsourcing-based project is performed by assigning a plurality of tasks of a crowdsourcing-based project (hereinafter, a project) to a plurality of workers. Requesting, receiving a plurality of work results from the plurality of workers, assigning the plurality of work results to a plurality of inspectors to request inspection, and the plurality of work results from the plurality of inspectors Receiving a plurality of inspection results as passing or rejecting the inspection, requesting rework to the worker who has entered the previous work result for the work result (hereinafter, referred to as previous work result) in which the inspection result has been entered as rejection, Receiving a rework result of correcting the previous work result from the worker, comparing the rework result and the previous work result, determining whether to modify the previous work result, and not modifying the previous work result If it is determined that it is not, the step of automatically determining the result of the rework result as a rejection, and each task of the project is designated using a bounding box for one or more target objects existing on the source image. And requesting input of class information corresponding to the bounding box, and determining whether to modify the previous work result by comparing the rework result and the previous work result, the area or number of the bounding box It is determined whether the previous work result is modified according to a change or a change in class information corresponding to the bounding box.

In some embodiments of the present invention, the step of determining whether to modify the previous work result by comparing the rework result and the previous work result, is input as the bounding box input as the rework result and the previous work result. Measuring an Intersection over Union (IoU) value of the bounding box, determining whether an area of the bounding box has changed by comparing the IoU value with a reference value, and whether the area of the bounding box has changed It may include determining whether to modify the previous work result according to the determination result.

In some embodiments of the present invention, the measuring of the IoU value comprises, when a plurality of bounding boxes are input as the previous work result, each of the bounding boxes input as the rework result is input as the previous work result. IoU values with all the bounding boxes may be measured, and the largest IoU value among the measurement results may be measured as the IoU value of each bounding box input as a result of the rework.

In some embodiments of the present invention, the reference value may be set or changeable by the project manager or the inspector.

In some embodiments of the present invention, the step of comparing the rework result and the previous work result to determine whether to modify the previous work result, the number of bounding boxes input as the rework result is the previous work result Determining whether or not the number of bounding boxes has been added or decreased, and determining whether to modify the previous work result according to a determination result of whether the number of bounding boxes has been added or decreased. can do.

In some embodiments of the present invention, the step of determining whether to modify the previous work result by comparing the rework result and the previous work result includes class information corresponding to the first bounding box input as the rework result And determining whether the class information corresponding to the first bounding box inputted as the result of the previous operation is identical, and determining whether to modify the result of the previous operation according to the determination result of whether the class information is identical It may include the step of.

In some embodiments of the present invention, the step of comparing the rework result and the previous work result to determine whether to modify the previous work result, the rework result and the previous work result, respectively, predictive data and actual measurement data And generating an error matrix corresponding to the comparison result of the rework result and the previous work result, determining whether to modify the previous work result based on the error matrix, and TP of the error matrix (True Positive) value indicates the number of bounding boxes in which the area or class information has not been changed, TN (True Negative) value indicates that the number of bounding boxes is 0 in both the rework result and the previous work result, and FN( False Negative) value indicates the sum of the number of bounding boxes whose area has been changed, the number of deleted bounding boxes, and the number of bounding boxes whose class information is changed, and the FP (False Positive) value indicates the number of added bounding boxes. can do.

In some embodiments of the present invention, the method further comprises, if it is determined that the previous work result has been modified, allocating the rework result to an inspector who has performed the inspection of the previous work result and requesting to perform the retest In the step of requesting to perform the re-examination, information on a bounding box and an added bounding box in which area or class information has been changed may be provided to request the inspector to perform re-examination.

In some embodiments of the present invention, the method may further include the step of restricting the execution of the work for a predetermined period for a worker whose number of times the rework result is automatically determined as rejection is equal to or greater than the reference number. .

A computer program according to another aspect of the present invention for solving the above-described problems is combined with a computer that is hardware to execute an automatic return method of the rework result of the crowdsourcing-based project, and is stored in a computer-readable recording medium.

Other specific details of the present invention are included in the detailed description and drawings.

According to the present invention described above, it is possible to detect an illegal worker who is not faithfully performing the rework by determining whether to modify the previous work result.

In addition, by automatically returning the same rework result as the previous work result, it is possible to save the inspector's time and labor.

In addition, when a rework result that has been revised compared to the previous work result is requested to be re-examined to the inspector, information related to the revised matter can be provided together so that the inspector can conveniently perform the re-review.

The effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the following description.

1 is a conceptual diagram of a crowdsourcing service according to an embodiment of the present invention.
2 is a flowchart illustrating a crowdsourcing-based project progress process according to an embodiment of the present invention.
3 is a flowchart of a method for automatically rejecting a rework result of a crowdsourcing-based project according to an embodiment of the present invention.
4 is an exemplary view for explaining the work of a project according to an embodiment of the present invention.
5 is an example of step S170 of FIG. 3.
6 is an exemplary diagram for describing IoU according to an embodiment of the present invention.
7 and 8 are exemplary diagrams for explaining measurement of an IoU value between a bounding box of a previous work result and a bounding box of a rework result according to an embodiment of the present invention.
9 is another example of step S170 of FIG. 3.
10 is another example of step S170 of FIG. 3.
11 is an exemplary view for explaining a comparison between a previous work result and a rework result according to an embodiment of the present invention.
12 is a diagram for explaining an apparatus for automatically rejecting a rework result of a crowdsourcing-based project according to an embodiment of the present invention.

Advantages and features of the present invention, and a method of achieving them will become apparent with reference to the embodiments described below in detail together with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but may be implemented in a variety of different forms, only the present embodiments are intended to complete the disclosure of the present invention, It is provided to fully inform the technician of the scope of the present invention, and the present invention is only defined by the scope of the claims.

The terms used in the present specification are for describing exemplary embodiments and are not intended to limit the present invention. In this specification, the singular form also includes the plural form unless specifically stated in the phrase. As used in the specification, “comprises” and/or “comprising” do not exclude the presence or addition of one or more other elements other than the mentioned elements. Throughout the specification, the same reference numerals refer to the same elements, and “and/or” includes each and all combinations of one or more of the mentioned elements. Although "first", "second", and the like are used to describe various elements, it goes without saying that these elements are not limited by these terms. These terms are only used to distinguish one component from another component. Therefore, it goes without saying that the first component mentioned below may be the second component within the technical idea of the present invention.

Unless otherwise defined, all terms (including technical and scientific terms) used in the present specification may be used as meanings that can be commonly understood by those of ordinary skill in the art to which the present invention belongs. In addition, terms defined in a commonly used dictionary are not interpreted ideally or excessively unless explicitly defined specifically.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a conceptual diagram of a crowdsourcing service according to an embodiment of the present invention.

Referring to FIG. 1, a crowdsourcing service is performed by being composed of a client 10, a service provider 20, and a public 30.

The sponsor 10 refers to a company or individual who requests a crowdsourcing-based project (hereinafter, referred to as a project).

The requester 10 requests a project for the purpose of collecting source data or annotating data for generating artificial intelligence learning data. The data generated through the project can be used as learning data for arbitrary machine learning such as supervised learning, unsupervised learning, and reinforcement learning. Collection of source data refers to collecting raw data such as recorded voice collection and photo collection. Data annotation refers to inputting relevant annotation data into source data such as text, photos, and videos. For example, the data annotation may include finding an entity in a given fingerprint or finding a similar sentence, but is not limited thereto. Meanwhile, the types of the above-described projects are only one embodiment, and various projects may be handled in the present invention according to the design of the client.

The service provider 20 refers to a company that provides crowdsourcing services.

When the service provider 20 receives a request for a project for a product or service from the client 10, the service provider 20 allocates a task for the project to the general public 30 and receives the work result from the public 30. Thereafter, the final product extracted based on the work result is provided to the client 10.

At this time, the service provider 20 provides a crowdsourcing service to the client 10 and the public 30 through a crowdsourcing platform (hereinafter, the platform). That is, when the service provider 20 receives a request for a project from the client 10, the service provider 20 opens the project on the platform. Thereafter, when the work result for the open project is received from the public 30, the project is terminated on the platform, and the final product may be extracted and provided to the client 10.

The public 30 refers to the general public participating in the project open on the platform. Here, the public 30 may participate in a project opened on the platform through an application or website provided by the service provider 20.

The public 30 is composed of a worker 32 and an inspector 34.

The worker 32 decides to participate in a specific project among a plurality of projects open on the platform. Thereafter, the operator 32 performs an operation such as collection of source data or data annotation, and transmits it to the platform.

The inspector 34 decides to participate in a specific project among a plurality of projects open on the platform. Thereafter, the inspector 34 inspects the result of the work performed by the operator 32. As a result of performing the inspection, the inspector 34 may perform inspection pass processing or rejection processing, and input a rejection reason during rejection processing. In the case of passing the inspection, it is not necessary to rework and re-inspect the result.

2 is a flowchart illustrating a crowdsourcing-based project progress process according to an embodiment of the present invention.

First, the requester 10 requests one or more projects to the service provider 20 (S11).

Thereafter, the service provider 20 opens the requested project on the platform (S12). At this time, the service provider 20 may determine the grade in consideration of the difficulty of the project before opening the project. That is, depending on the difficulty level, it is possible to determine which level or higher public 30 to expose the project. Accordingly, it is possible to increase the reliability of the work result of the project.

Thereafter, the service provider 20 requests the work by assigning the work to the workers 32 of the corresponding level or higher according to the level of the project (S13).

Thereafter, the worker 32 performs the assigned task (S14). In this case, the worker 32 may input the reason for the inability to work without performing the work for a work in which the work itself is impossible for some reason.

Thereafter, the service provider 20 receives the work result from the worker 32 (S15), and assigns the work result to the inspector 34 to request the inspection (S16).

Thereafter, the inspector 34 performs the assigned inspection (S17). At this time, the inspector 34 decides to pass the inspection when it is determined that the work has been properly performed, and if it is determined that the inspection work is wrong, the inspector 34 rejects it. During rejection processing, the inspector 34 inputs the rejection reason for what reason the job was judged to be wrong.

Thereafter, the service provider 20 receives the inspection result from the inspector 34 (S18).

If the inspection result passes the inspection, the service provider 20 uses the work result as valid data, and based on this, extracts the final product at the end of the project.

When the inspection result is rejection processing, the service provider 20 may internally perform the inspection again, or assign the work to the worker 32 again to perform the rework. When reworking, re-examination by an inspector is required

Thereafter, when the project period ends or sufficient valid data is secured, the service provider 20 terminates the project (S19), calculates a final result based on the secured valid data, and provides it to the client 10 ( S20).

At this time, before the end of the project, the service provider 20 evaluates the performance results of the worker 32 and the inspector 34, calculates the work cost and the inspection cost according to the evaluation, to the worker 32 and the inspector 34. give.

In FIGS. 1 and 2, it is simply expressed as a requester 10, a service provider 20, an operator 32, and an inspector 34, but these are smartphones, tablets, PDAs, laptops, desktops, operated by each participant. It means a computer device such as a server or a telecommunication device.

In the conventional case, whether or not the operator inputs the rework result by modifying the previous work result when performing the rework could be determined only through inspection. In other words, if the operator does not modify the previous work result and enters the rework result in the same way as the previous work result, the inspector has to inspect the same work result twice, which reduces inspection efficiency and wastes inspector resources. Led to problems.

In order to solve this problem, an embodiment of the present invention determines whether to modify the previous work result through comparison of the rework result and the previous work result, and if the previous work result is not modified, the rework result is sent to the inspector. Allows automatic rejection without inspection.

Hereinafter, with reference to FIGS. 3 to 11, a method of determining whether to modify a previous work result through comparison of a rework result and a previous work result according to an embodiment of the present invention and a method of automatically rejecting the rework result will be described. do.

3 is a flowchart of a method for automatically rejecting a rework result of a crowdsourcing-based project according to an embodiment of the present invention. 4 is an exemplary view for explaining the work of a project according to an embodiment of the present invention. 5 is an example of step S170 of FIG. 3. 6 is an exemplary diagram for describing IoU according to an embodiment of the present invention. 7 and 8 are exemplary diagrams for explaining measurement of an IoU value between a bounding box of a previous work result and a bounding box of a rework result according to an embodiment of the present invention. 9 is another example of step S170 of FIG. 3. 10 is another example of step S170 of FIG. 3. 11 is an exemplary view for explaining a comparison between a previous work result and a rework result according to an embodiment of the present invention.

Meanwhile, the steps illustrated in FIGS. 3, 5, 9 and 10 may be understood as being performed by a platform server (hereinafter, referred to as a server) operated by the service provider 20, but are not limited thereto. .

In addition, a plurality of workers 32 or a plurality of inspectors 34 perform work using a predetermined terminal device. The terminal device of the operator 32 or the inspector 34 may be a computer device or a telecommunication device such as a smartphone, tablet, PDA, laptop, desktop, etc., but is not limited thereto.

First, referring to FIG. 3, the server assigns a plurality of tasks of a crowdsourcing-based project (hereinafter, a project) to a plurality of workers 32 to request the task performance (S110), and a plurality of tasks from the plurality of workers 32 Receiving work results (S120), assigning a plurality of work results to a plurality of inspectors 34 to request inspection performance (S130), and receiving a plurality of inspection results for a plurality of work results from the plurality of inspectors 34 It is inputted as passing or rejecting the inspection (S140).

Here, the plurality of workers 32 and the plurality of inspectors 34 refer to all workers and inspectors participating in the same project.

Since the steps S110 to S140 overlap with the steps S13 to S18 described above with reference to FIG. 2, a detailed description will be omitted.

However, in this specification, each task of the project includes a bounding task and a labeling task. That is, each task of the project is to designate one or more target objects existing on the source image using a bounding box and request input of class information corresponding to the bounding box.

Referring to FIG. 4, if two target objects TO1 and TO2 exist in the source image, each worker 32 performs a bounding operation by designating each target object using a bounding box, and each bound Labeling is performed by inputting class information on the target object (for example, if the target object is a cat, class information is input as'cat').

Next, referring to FIG. 3 again, the server requests the operator who has entered the previous work result to perform rework for the work result (hereinafter, the previous work result) inputted as rejection of the inspection result (S150), and The rework result obtained by correcting the previous work result is received from (S160).

As described in step S18 of FIG. 2, the work result inputted as rejection is reworked by the operator 32 who inputs the work result. On the other hand, the work result inputted through the inspection result is used as valid data for the final product.

The server requests rework for the previous work result while providing the reason for rejection input by the inspector 34 to the worker 32 for the work result (hereinafter, the previous work result) inputted as rejection of the inspection result. . The worker 32 must check the reason for rejection and correctly rework the results of previous work performed by the operator. That is, the operator 32 must correct the previous work result and input the rework result.

However, in the case of fraudulent workers, rework may not be performed properly. In other words, it is possible to input the same rework result as the previous work result without modifying the previous work result. Therefore, it is necessary to determine whether the operator inputs the rework result by modifying the previous work result when performing the rework.

Next, the server compares the rework result and the previous work result, and determines whether to modify the previous work result (S170).

Specifically, the server determines whether to modify the previous work result according to a change in the area or number of bounding boxes or change in class information corresponding to the bounding box.

1) Determine whether to modify the previous work result according to the change of the bounding box area

Referring to FIG. 5, the server measures IoU (Intersection over Union) values of the bounding box input as a result of rework and the bounding box input as a previous work result (S171).

Here, the IoU value is a value measured according to the degree of overlap between the bounding box input as a rework result and the bounding box input as a previous work result.

Referring to FIG. 6, an IoU value means a ratio of an intersection area to a union area of two bounding boxes. Therefore, the larger the degree of overlap of the bounding box is, the closer to 1 is measured, and the smaller the degree of overlap is, the closer to 0 is measured.

According to an embodiment, when one bounding box is input as a result of a previous work, an IoU value with one bounding box input as a result of a previous work may be measured for each bounding box input as a result of rework.

Referring to Figure 7a, when one bounding box (BBOX1) is input from the previous work result and one bounding box (BBOX1') is input from the rework result, the server is measured according to the degree of overlap between BBOX1 and BBOX1'. Measure the IoU 11' value as the IoU value of the bounding box (BBOX1') of the rework result.

Referring to FIG. 7B, when one bounding box (BBOX1) is input from the previous work result and two bounding boxes (BBOX1' and BBOX2') are input from the rework result, the server is at the degree of overlap between BBOX1 and BBOX1'. The measured IoU 11' value is measured as the IoU value of the bounding box (BBOX1') of the rework result, and the IoU 12' value measured according to the degree of overlap between BBOX1 and BBOX2' is the bounding box (BBOX2') of the rework result. Measured by the IoU value of ). At this time, because BBOX1 and BBOX1' match and BBOX1 and BBOX2' do not match (because BBOX2' is a bounding box added from the rework result), the value of IoU 12' is measured as 0 or less than the value of IoU 11'. It is measured as

According to another embodiment, when a plurality of bounding boxes are input as a result of a previous operation, IoU values with all bounding boxes input as a result of the previous operation are measured for each of the bounding boxes input as a result of rework, and among the measurement results The largest IoU value can be measured as the IoU value of each bounding box entered as a result of rework.

Referring to FIG. 8A, when two bounding boxes (BBOX1 and BBOX2) are input from the previous work result and one bounding box (BBOX1') is input from the rework result, the server depends on the degree of overlap between BBOX1 and BBOX1'. Measure the value of IoU 11' and measure the value of IoU 21' according to the degree of overlap between BBOX2 and BBOX1'. At this time, because BBOX1 and BBOX1' match and BBOX2 and BBOX1' do not match (because BBOX2 is a bounding box deleted from the rework result), the value of IoU 21' is measured as 0 or less than the value of IoU 11'. Will be measured. Accordingly, the server measures the IoU 11' value, which is the larger of the IoU 11' value and the IoU 21' value, as the IoU value of the bounding box (BBOX1') of the rework result.

8B, when two bounding boxes (BBOX1 and BBOX2) are input from the previous work result and two bounding boxes (BBOX1' and BBOX2') are input from the rework result, the server overlaps between BBOX1 and BBOX1'. Measure the value of IoU 11' according to the degree, and measure the value of IoU 21' according to the degree of overlap between BBOX2 and BBOX1'. At this time, since BBOX1 and BBOX1' match and BBOX2 and BBOX1' do not match, the IoU 21' value is measured as 0 or less than the IoU 11' value. Accordingly, the server measures the IoU 11' value, which is the larger of the IoU 11' value and the IoU 21' value, as the IoU value of the bounding box (BBOX1') of the rework result. In addition, the server measures the value of IoU 12' according to the degree of overlap between BBOX1 and BBOX2', and measures the value of IoU 22' according to the degree of overlap between BBOX2 and BBOX2'. At this time, since BBOX1 and BBOX2' do not match and BBOX2 and BBOX2' match, the IoU 12' value is measured as 0 or a value smaller than the IoU 22' value. Therefore, the server measures the IoU 22' value, which is the larger of the IoU 12' value and the IoU 22' value, as the IoU value of the bounding box (BBOX2') of the rework result.

Next, the server compares the IoU value with the reference value and determines whether the area of the bounding box has changed (S172).

As described above, the IoU value is measured close to 1 if the degree of overlap between the two bounding boxes is large, and close to 0 if the degree of overlap is small. In other words, when the IoU value is measured close to 1, it means that there is little change in the area of the bounding box input in the rework result when compared with the area of the bounding box input in the previous work result.

Therefore, when comparing the IoU value with the reference value, the server determines that the area of the bounding box has changed when the IoU value is less than the reference value, and determines that the area of the bounding box has not changed when the IoU value is greater than the reference value.

For example, when the reference value is 0.7, if the IoU value of the bounding box of the rework result is measured as 0.5, it is determined that the area of the bounding box has changed, and if the IoU value of the bounding box of the rework result is measured as 0.9 It is determined that the area of the bounding box has not changed.

Next, the server determines whether to modify the previous work result according to the determination result of whether the area of the bounding box has changed (S173).

That is, if it is determined that the area of the bounding box has changed, the server determines that the previous work result has been modified, and if it determines that the area of the bounding box has not changed, the server determines that the previous work result has not been modified.

At this time, if there are multiple bounding boxes of the rework result, the IoU value of each of the plurality of bounding boxes is compared with the reference value, and if the IoU value of at least one bounding box is less than the reference value, the server indicates that the previous work result has been modified. Judge.

For example, if the rework result has 3 bounding boxes, each of the IoU values is 0.5, 0.7, and 0.8, and the reference value is 0.7, then 1 out of 3 IoU values are less than the reference value, so the server will It can be determined that is modified.

According to an embodiment, the reference value can be set or changed by the manager or inspector of the project.

In the case of a project that requires fine modification, the operator 32 modifies the area of the bounding box of the previous work result by a size corresponding to only a few pixels and inputs the rework result.At this time, the IoU value is measured close to 1. It is judged to be more than the reference value. That is, even though the worker 32 has correctly performed the rework, there may be a situation in which it is determined that the previous work result has not been modified.

In order to prevent such a situation from occurring, the manager or inspector can appropriately set or change the reference value according to the project characteristics.

2) Determine whether to modify the previous work result according to the change in the number of bounding boxes

Referring to FIG. 9, the server determines whether the number of bounding boxes input as a result of rework is added or decreased compared to the number of bounding boxes input as a result of a previous work (S174).

That is, if the number of bounding boxes input as a result of the rework is larger than the number of bounding boxes input as a result of the previous work, it is determined that the bounding box is added from the rework result. If the number of bounding boxes input as a result of the rework is less than the number of bounding boxes input as a result of the previous work, it is determined that the bounding box is reduced from the rework result.

For example, if the number of bounding boxes entered as a result of rework is 3 and the number of bounding boxes entered as a result of previous work is 2, the server determines that the bounding box has been added from the rework result, and the rework result If the number of bounding boxes input to is 3 and the number of bounding boxes input as a result of the previous operation is 4, the server determines that the bounding box has been deleted from the rework result.

Next, the server determines whether to modify the previous work result according to the determination result of whether the number of bounding boxes has been added or decreased (S175).

That is, if there is a change in the number of areas of the bounding box (i.e., it is determined that it is added or decreased), the server determines that the previous operation result has been modified, and if there is no change in the number of areas of the bounding box (i.e. If it is determined that it has not been reduced), the server determines that the previous work result has not been modified.

3) Determine whether to modify the previous work result according to the change of class information corresponding to the bounding box

Referring to FIG. 10, the server determines whether the class information corresponding to the first bounding box input as a result of the rework and the class information corresponding to the first bounding box input as the previous work result match (S176).

For example, if the class information corresponding to the first bounding box input as a result of rework is'aaa' and the class information corresponding to the first bounding box input as a result of the previous work is'aaa', the server returns the class information. If it is determined to match and the class information corresponding to the first bounding box input as a result of rework is'bbb' and the class information corresponding to the first bounding box input as a result of the previous work is'aaa', the server is It is determined that the information does not match.

Next, the server determines whether to modify the previous work result according to the determination result of whether the class information matches (S177).

That is, if it is determined that the class information does not match, the server determines that the previous work result has been modified, and if it is determined that the class information matches, the server determines that the previous work result has not been modified.

In this case, when there are a plurality of bounding boxes, each class information corresponding to the bounding box is compared, and if it is determined that at least one class information does not match, the server determines that the previous work result has been modified.

For example, class information corresponding to the first bounding box and the second bounding box input as a result of rework is'aaa' and'bbb', respectively, and the first bounding box and the second bounding box input as a result of the previous work If the class information corresponding to is'aaa' and'ccc', respectively, the class information corresponding to the first bounding box is matched, but the class information corresponding to the second bounding box does not match. It can be judged as.

According to an embodiment, the server may generate an error matrix based on a result of comparing the rework result and the previous work result, and determine whether to modify the previous work result based on the error matrix.

Specifically, the server sets the rework result and the previous work result as predicted data and actual measurement data, respectively, and generates an error matrix corresponding to the comparison result of the rework result and the previous work result to determine whether to modify the previous work result. I can.

That is, the server assumes the previous work result as the correct answer, and when comparing the rework result with the previous work result, it can determine whether or not to modify the previous work result by grasping what error has occurred in the rework result compared to the correct answer. .

Here, the error matrix includes a TP (True Positive) value, a TN (True Negative) value, a FN (False Negative) value, and an FP (False Positive) value.

The TP value indicates the number of bounding boxes in which the area or class information has not been changed. For example, if there are 3 bounding boxes input in the rework result, and all 3 bounding boxes have not changed the area or class information, the TP value is 3. Alternatively, if there are 3 bounding boxes entered in the rework result, the area or class information of the two bounding boxes has not been changed, and the area or class information of the remaining one bounding box is changed, the TP value is 2.

The TN value indicates that the number of bounding boxes is 0 in both the rework result and the previous work result. In the present invention, this value is meaningless to determine whether the rework result is corrected through the error matrix.

The FN value indicates the sum of the number of bounding boxes with changed area, the number of deleted bounding boxes, and the number of bounding boxes with changed class information. For example, when there are 3 bounding boxes input in the rework result, and the area of one bounding box among the 3 bounding boxes is changed, the FN value becomes 1. Alternatively, when there are three bounding boxes input in the rework result, and the area and class information of one bounding box among the three bounding boxes is changed, the FN value becomes 2. Alternatively, if there are 4 bounding boxes entered in the previous work result, but there are 3 bounding boxes entered in the rework result, the FN value is 1. Alternatively, there are 4 bounding boxes entered in the previous work result, but there are 3 bounding boxes entered in the rework result, the area and class information of 1 of the 3 bounding boxes has changed, and 1 of the other 2 bounding boxes has also been changed. When the area is changed, the FN value becomes 4.

The FP value indicates the number of bounding boxes added. For example, if there are 4 bounding boxes entered in the previous work result, but 5 bounding boxes entered in the rework result, the FP value will be 1.

Therefore, regardless of the TP value, when the FN value or the FP value is 1 or more, the server may determine that the rework result has changed compared to the previous work result, and may determine that the previous work result has been modified.

In contrast, when the FN value and the FP value are 0, and the TP value is 1 or more, the server may determine that the rework result has not changed compared to the previous work result, and thus the previous work result has not been modified.

Referring to FIG. 11A, two target objects TO1 and TO2 exist in a source image. In the previous work, only the target object (TO2) was assigned a bounding box and input class information. Thereafter, when reworking, the bounding box designation and class information input were also performed for the target object TO1, and the class information for the target object TO2 was changed from'cat' to'dog'.

Accordingly, when comparing the previous work result and the rework result for each of the two target objects TO1 and TO2, the bounding box is added from the rework result for the target object TO1, so the FP value becomes 1. In addition, for the target object TO2, there is no change in the bounding box area (IoU value is greater than or equal to the reference value), but the FN value is 1 because the class information has changed.

The error matrix of the rework result is generated as TP value of 0, TN value of 0, FP value of 1, and FN value of 1. Therefore, regardless of the TP value, since the FN value and the FP value are 1 or more, the server determines that the previous work result has been modified.

Referring to FIG. 11B, two target objects TO1 and TO2 exist in the source image. During the previous work, the target object (TO1) and the target object (TO2) were assigned a bounding box and input class information. After that, there was no change to the bounding box and class information that were specified and entered in the previous work when reworked.

Therefore, when comparing the previous work result and the rework result for each of the two target objects (TO1 and TO2), there is no change in the area of the bounding box for the target object (TO1) and the target object (TO2) (IoU value More than this reference value) Since the class information has not changed, that is, the previous work result and the rework result are the same for each of the two target objects (TO1 and TO2), the TP value becomes 2, and the number of bounding boxes is added or Since it has not been decreased, the FN value and FP value become 0.

The error matrix of the rework result is generated with TP of 2, TN of 0, FP of 0, and FN of 0. Therefore, since the FN value and the FP value are 0, and the TP value is 1 or more, the server determines that the previous work result has not been modified.

Next, when it is determined that the previous work result has not been modified, the server automatically determines the rework result to be rejected (S180).

That is, if it is determined that the previous work result has not been modified, the server determines that the worker 32 has performed illegal rework and automatically returns the rework result.

According to an embodiment, the server restricts the job execution for a predetermined period of time to a worker whose number of times the inspection result of the rework result is automatically determined as rejection is equal to or greater than the reference number.

For example, if the reference number of times is 5, if there are 5 or more rework results that are automatically rejected among a plurality of rework results input by a specific worker, the specific worker is determined as an illegal worker and for a predetermined period, for example 1 Penalties can be offered to make it impossible to perform tasks for a week.

Although not clearly shown in FIG. 3, the automatic return method of the rework result of a crowdsourcing-based project is, when it is determined that the previous work result has been modified, the rework result is assigned to the inspector who has inspected the previous work result and re-reviewed. It may further include the step of requesting the execution.

In this case, the server provides information on the bounding box and the added bounding box in which the area or class information has been changed, and requests the inspector to perform re-examination. That is, the server stores information on changes in the rework result by comparing the rework result with the previous work result in step S170. Thereafter, when it is determined that the previous work result has been modified, and when the inspector requests to perform re-review of the rework result, information on the change in the rework result is provided together, so that the inspector can change any part of the rework result. It is possible to quickly grasp whether there was any, so that the re-examination can be performed more easily.

Meanwhile, in the above description, steps S11 to S177 may be further divided into additional steps or may be combined into fewer steps, according to an embodiment of the present invention. In addition, some steps may be omitted as necessary, and the order between steps may be changed. In addition, even if other contents are omitted, the contents of FIG. 12 to be described later can also be applied to the method of automatically rejecting the rework result of the crowdsourcing-based project of Figs.

Hereinafter, with reference to FIG. 12, an automatic rejection device 200 for a rework result of a crowdsourcing-based project according to an embodiment of the present invention will be described.

12 is a diagram for explaining an apparatus for automatically returning a rework result of a crowdsourcing-based project according to an embodiment of the present invention.

Referring to FIG. 12, an automatic revocation device 200 (hereinafter, an automatic revocation device) of a rework result of a crowdsourcing-based project includes a communication module 210, a memory 220, and a processor 230.

The communication module 210 transmits a crowdsourcing-based work for one project to a plurality of workers 32 to request a task to be performed, and receives a work result from the plurality of workers 32. The work results received from the plurality of workers 32 are transmitted to the plurality of inspectors 34 to request inspection performance, and the inspection results are received from the plurality of inspectors 34.

The communication module 210 transmits the job result rejected as a result of the inspection (hereinafter, the previous job result) to the worker 32 who performed the previous job result to request rework, and the worker who performed the previous job result ( 32), the rework result is received. The rework result received from the operator 32 who performed the previous work result is sent to the inspector 34 who has inspected the previous work result to request the re-inspection, and the inspector 34 who inspected the previous work result The retest result is received from.

The memory 220 stores a program for automatically returning the input rework result in the same way as the previous work result based on the data received from the communication module 210.

The processor 230 executes a program stored in the memory 220. As the processor 230 executes the program stored in the memory 220, it compares the rework result with the previous work result to determine whether to modify the previous work result, and if it is determined that the previous work result has not been modified, that is, If it is determined that the rework result and the previous work result are the same, the inspection result for the rework result is automatically determined as rejection.

In this case, the processor 230 may determine whether to modify the previous work result according to a change in the area or number of bounding boxes or change in class information corresponding to the bounding box.

The automatic companion device 200 described with reference to FIG. 12 may be provided as a component of the above-described server.

The method for automatically returning the rework result of a crowdsourcing-based project according to an embodiment of the present invention described above may be implemented as a program (or application) to be executed in combination with a computer, which is hardware, and stored in a medium.

The above-described program includes C, C++, JAVA, Ruby, which can be read by a processor (CPU) of the computer through the device interface of the computer, in order for the computer to read the program and execute the methods implemented as a program. It may include a code (Code) coded in a computer language such as machine language. Such code may include a functional code related to a function defining necessary functions for executing the methods, and a control code related to an execution procedure necessary for the processor of the computer to execute the functions according to a predetermined procedure. can do. In addition, such code may further include additional information required for the processor of the computer to execute the functions or code related to a memory reference to which location (address address) of the internal or external memory of the computer should be referenced. have. In addition, when the processor of the computer needs to communicate with any other computer or server in the remote in order to execute the functions, the code uses the communication module of the computer to determine how It may further include a communication-related code for whether to communicate or what information or media to transmit and receive during communication.

The stored medium is not a medium that stores data for a short moment, such as a register, cache, memory, etc., but a medium that stores data semi-permanently and can be read by a device. Specifically, examples of the storage medium include, but are not limited to, ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, and the like. That is, the program may be stored in various recording media on various servers to which the computer can access, or on various recording media on the user's computer. Further, the medium may be distributed over a computer system connected through a network, and computer-readable codes may be stored in a distributed manner.

The above description of the present invention is for illustrative purposes only, and those of ordinary skill in the art to which the present invention pertains will be able to understand that it can be easily modified into other specific forms without changing the technical spirit or essential features of the present invention. will be. Therefore, it should be understood that the embodiments described above are illustrative in all respects and not limiting. For example, each component described as a single type may be implemented in a distributed manner, and similarly, components described as being distributed may also be implemented in a combined form.

The scope of the present invention is indicated by the claims to be described later rather than the detailed description, and all changes or modified forms derived from the meaning and scope of the claims and their equivalent concepts should be interpreted as being included in the scope of the present invention do.

10: Client
20: service provider
30: public
32: worker
34: inspector
200: automatic companion device
210: communication module
220: memory
230: processor

Claims (10)

As a method performed by a crowdsourcing platform server (hereinafter, server),
The server allocating a plurality of tasks of a crowdsourcing-based project (hereinafter, referred to as a project) to a plurality of workers and requesting the task to be performed by a plurality of worker terminals;
Receiving, by the server, a plurality of job results from the plurality of worker terminals;
Assigning, by the server, the plurality of work results to a plurality of inspectors and requesting a plurality of inspector terminals to perform inspection;
Receiving, by the server, a plurality of inspection results for the plurality of work results from the plurality of inspector terminals as passing or rejecting the inspection;
Requesting, by the server, for a job result (hereinafter, referred to as a previous job result) inputted as a rejection of the inspection result, to a worker terminal receiving the previous job result to perform rework;
Receiving, by the server, a rework result obtained by modifying the previous work result from the worker terminal;
Determining, by the server, whether to modify the previous work result by comparing the rework result and the previous work result; And
If the server determines that the previous work result has not been modified, automatically determining a rejection result of the inspection result for the rework result,
Each task in the above project,
Designating using a bounding box for one or more target objects existing on the source image and requesting input of class information corresponding to the bounding box,
The step of the server comparing the rework result and the previous work result and determining whether to modify the previous work result,
It is determined whether or not the previous work result is modified according to a change in the area or number of the bounding boxes or class information corresponding to the bounding box,
The step of the server comparing the rework result and the previous work result and determining whether to modify the previous work result,
Measuring, by the server, an Intersection over Union (IoU) value of the bounding box input as a result of the rework and the bounding box input as a result of the previous work,
The server comparing the IoU value with a reference value, and determining whether the area of the bounding box has changed;
And determining, by the server, whether to modify the previous work result according to a determination result of whether the area of the bounding box has changed,
The step of measuring the IoU value by the server,
When multiple bounding boxes are entered as the result of the previous operation,
For each of the bounding boxes input as the rework result, IoU values with all the bounding boxes input as the previous work result are measured,
To measure the largest IoU value among the measurement results as the IoU value of each bounding box input as the rework result,
A method of automatically rejecting rework results for crowdsourced projects. delete delete The method of claim 1,
The reference value is received from the manager terminal or inspector terminal of the project Which can be set or changed,
A method of automatically rejecting rework results for crowdsourced projects. The method of claim 1,
The step of the server comparing the rework result and the previous work result and determining whether to modify the previous work result,
Determining, by the server, whether the number of bounding boxes input as a result of the rework is added or decreased compared to the number of bounding boxes input as a result of the previous operation;
In accordance with a result of determining whether the number of bounding boxes has been added or decreased by the server, determining whether to modify the previous work result,
A method of automatically rejecting rework results for crowdsourced projects. The method of claim 1,
The step of the server comparing the rework result and the previous work result and determining whether to modify the previous work result,
Determining, by the server, whether the class information corresponding to the first bounding box input as a result of the rework and the class information corresponding to the first bounding box input as the previous operation result match,
The server comprising the step of determining whether to modify the previous work result according to a determination result of whether the class information matches,
A method of automatically rejecting rework results for crowdsourced projects. The method of claim 1,
The step of the server comparing the rework result and the previous work result and determining whether to modify the previous work result,
Set the rework result and the previous work result as predicted data and actual measured data, respectively,
Generating an error matrix corresponding to the comparison result of the rework result and the previous work result,
Based on the error matrix, it is determined whether or not the previous work result is modified,
The TP (True Positive) value of the error matrix indicates the number of bounding boxes in which area or class information has not been changed, and the TN (True Negative) value is the number of bounding boxes in both the rework result and the previous work result. 0, and the FN (False Negative) value indicates the sum of the number of bounding boxes with changed area, the number of deleted bounding boxes, and the number of bounding boxes with the class information changed, and a false positive (FP) value is added. Indicating the number of bounding boxes
A method of automatically rejecting rework results for crowdsourced projects. The method of claim 1,
If the server determines that the previous work result has been modified, allocating the rework result to an inspector who has performed the inspection of the previous work result, and requesting the inspector terminal to perform the re-review,
The step of requesting the server to perform the re-review,
By providing information on the bounding box and the added bounding box in which the area or class information has been changed, and requesting the inspector terminal to perform re-examination,
A method of automatically rejecting rework results for crowdsourced projects. The method of claim 1,
The server further comprising the step of limiting the performance of the work by the worker terminal for a predetermined period for a worker whose number of times the inspection result of the rework result is automatically determined as rejection is equal to or greater than a reference number,
A method of automatically rejecting rework results for crowdsourced projects. A computer program stored in a computer-readable recording medium in order to perform the method of automatically returning the rework result of the crowdsourcing-based project according to any one of claims 1, 4 to 9 in combination with a computer.

Images