KR101887415B1 - Program and method for checking data labeling product - Google Patents

Abstract

Disclosed is a method for checking a data labeling operation which comprises the steps of: determining a cross-checking level; determining the number of workers to distribute one data according to the determined cross-checking level; distributing first data to the determined number of workers; obtaining labeling information for the first data submitted by each worker; comparing the labeling information for the first data obtained from each worker; and requesting a rework operation when all of the labeling information for the first data obtained from each worker are not matched.

Description

TECHNICAL FIELD [0001] The present invention relates to a data labeling method,

The present invention relates to a method and program for validating data labeling operations.

In machine learning, what is more important than learning methods and algorithms is data for learning. Therefore, securing a lot of high quality learning data is very important in the field of machine learning.

In order to obtain learning data, generally, raw data and labeling information for raw data are required. Since labeling work must be performed by a person, he or she may have a work force to carry out labeling work where he / she studies or develops machine learning.

However, labeling is often done by anyone without expert knowledge. In order to reduce labor costs, a platform has been emerged in which data is distributed to a large number of workers over the Internet, and payment is made according to labeling work. For example, there is the mechanical Turk of amazon.

However, in the case of the existing platform, there is a problem that the reliability of the learning data can not be guaranteed because there is no process of confirming whether the worker has faithfully performed the labeling. Therefore, it is required to develop a technology that can increase the reliability of learning data while maintaining the advantages of the existing platform.

A problem to be solved by the present invention is to provide a data labeling operation acceptance method and a program.

The problems to be solved by the present invention are not limited to the above-mentioned problems, and other problems which are not mentioned can be clearly understood by those skilled in the art from the following description.

According to an aspect of the present invention, there is provided a method of validating data labeling operations, the method comprising: determining a cross-validation level; determining a number of workers to distribute one data according to the determined cross- Distributing first data to the determined number of workers, obtaining labeling information on the first data submitted by each of the workers, labeling information on the first data obtained from each of the workers, And requesting rework if the labeling information for the first data obtained from each of the workers is not determined to be all the same.

In addition, the step of requesting the re-work may include the steps of acquiring a verification result for the labeling information, determining an operator to request re-work according to the verification result, and requesting the re-work to the determined operator can do.

The acquiring step may include acquiring information for selecting unsuitable labeling information among the labeling information for the first data obtained from each of the workers, And requesting an operator who has submitted the labeling information selected as unsuitable labeling information among the workers to rework.

The step of acquiring the verification result may include acquiring the reason why the improper labeling information is not suitable, and the step of requesting the re-work may include transmitting the obtained reason to the worker who has submitted the inappropriate labeling information The method comprising the steps of:

The method includes the steps of acquiring learning data including inspection results for labeling information and labeling information obtained from operators, and learning a model capable of outputting inspection results for labeling information using the learning data can do.

The acquiring of the verification result may include inputting labeling information for the first data to the learned model and obtaining the verification result from the learned model.

The step of requesting rework may include requesting all of the workers to rework until it is determined that all of the labeling information for the first data obtained from each of the workers is identical have.

In addition, when it is determined that all of the labeling information for the first data obtained from each of the workers matches, payment may be provided to each of the workers.

The step of comparing the labeling information may further include dividing each of the obtained labeling information into a plurality of partial labeling information according to a predetermined criterion and comparing the divided partial labeling information with each other, The step of requesting the task may include requesting reworking of the partial labeling information in which the comparison results are determined to be inconsistent.

Also, a computer program stored in a computer-readable recording medium in combination with a computer, which is hardware, is provided so as to perform the data labeling job verification method according to the disclosed embodiments.

Other specific details of the invention are included in the detailed description and drawings.

According to the disclosed embodiments, it is possible to provide a method for inspecting a data labeling job that can increase the reliability of learning data while minimizing the inspection process by an actual inspector through cross validation.

The effects of the present invention are not limited to the above-mentioned effects, and other effects not mentioned can be clearly understood by those skilled in the art from the following description.

FIG. 1 is a conceptual diagram illustrating a data labeling operation acceptance system according to an embodiment.
2 is a flowchart illustrating a method of validating a data labeling operation according to an exemplary embodiment.
3 is a view showing an example of performing labeling on an image including an object.
FIG. 4 is a diagram illustrating an example of performing labeling on an image including text.
5 is a diagram showing an example of dividing the labeling information.
6 is a view showing a maintenance payment method according to an embodiment.

BRIEF DESCRIPTION OF THE DRAWINGS The advantages and features of the present invention, and the manner of achieving them, will be apparent from and elucidated with reference to the embodiments described hereinafter in conjunction with the accompanying drawings. It should be understood, however, that the invention is not limited to the disclosed embodiments, but may be embodied in many different forms and should not be construed as limited to the embodiments set forth herein, Is provided to fully convey the scope of the present invention to a technician, and the present invention is only defined by the scope of the claims.

The terminology used herein is for the purpose of illustrating embodiments and is not intended to be limiting of the present invention. In the present specification, the singular form includes plural forms unless otherwise specified in the specification. The terms " comprises "and / or" comprising "used in the specification do not exclude the presence or addition of one or more other elements in addition to the stated element. Like reference numerals refer to like elements throughout the specification and "and / or" include each and every combination of one or more of the elements mentioned. Although "first "," second "and the like are used to describe various components, it is needless to say that these components are not limited by these terms. These terms are used only to distinguish one component from another. Therefore, it goes without saying that the first component mentioned below may be the second component within the technical scope of the present invention.

Unless defined otherwise, all terms (including technical and scientific terms) used herein may be used in a sense that is commonly understood by one of ordinary skill in the art to which this invention belongs. In addition, commonly used predefined terms are not ideally or excessively interpreted unless explicitly defined otherwise.

As used herein, the term "part" or "module" refers to a hardware component, such as a software, FPGA, or ASIC, and a "component" or "module" performs certain roles. However, "part" or " module " is not meant to be limited to software or hardware. A "module " or " module " may be configured to reside on an addressable storage medium and configured to play back one or more processors. Thus, by way of example, "a" or " module " is intended to encompass all types of elements, such as software components, object oriented software components, class components and task components, Microcode, circuitry, data, databases, data structures, tables, arrays, and variables, as used herein. Or " modules " may be combined with a smaller number of components and "parts " or " modules " Can be further separated.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

FIG. 1 is a conceptual diagram illustrating a data labeling operation acceptance system according to an embodiment.

Referring to FIG. 1, a data labeling job validation system includes a server 100, a plurality of worker terminals 10 to 30, and one or more acceptor terminals 50.

The server 100 and the terminals 10 to 30 and 50 shown in FIG. 1 refer to a computing device including at least one processor.

In this specification, the term "operator" means an individual who performs labeling of data provided from the server 100 and receives a predetermined fee. For example, the server 100 may provide an operator with an image containing a specific object or text, an operator may specify an object or a range of text included in the image, Perform the labeling operation that you enter. The result of the operator performing the labeling operation is transmitted to the server 100.

As used herein, the term "inspector" means an individual who performs inspection of the results of labeling uploaded by workers. According to the embodiment, the "inspector" is not a person but can also be understood to mean a learned model, a program, a terminal, and the like capable of performing inspection automatically.

If there is an improper labeling result uploaded by the operator, the inspector rejects the inappropriate labeling result and enters the reason for the rejection. The server 100 transmits the labeling result to the operator who has performed the improper labeling, and requests rework.

When the labeling result performed by the operator passes inspection of the inspector, the server 100 compensates the operator for the labeling operation. For example, the server 100 can pay a predetermined point to the worker. In one embodiment, accumulated points above a certain level may be exchanged for cash and deposited to the operator. For example, an operator can deposit points accumulated in excess of one thousand points with cash of equal value.

In one embodiment, the server 100 can perform reliable verification of all labeling results, thereby obtaining highly reliable learning data.

In machine learning, reliable learning materials are needed for rapid learning and rapid growth of models. According to the disclosed embodiment, the server 100 can increase the efficiency of the machine learning by providing reliable learning data.

However, in the case of performing inspection for all labeling results, a bottleneck of learning data generation may occur in the inspection process according to the number of inspectors, and development of a method for improving the bottleneck phenomenon is required.

Therefore, according to the disclosed embodiment, the server 100 allocates the same data to a plurality of workers, and if the labeling results of a plurality of workers are all the same, the verification can be omitted and the reliability can be maintained while reducing the inspection count. However, when the same data is allocated to a plurality of workers, a plurality of compensations must be paid for one piece of data, so that the cost may increase.

On the other hand, the more the same data is assigned to more workers, the lower the likelihood of the result being wrong even though all the labeling results of all workers are identical. Therefore, the more the cost is allocated and the same data is allocated to more workers, the higher the reliability of the generated learning data becomes.

2 is a flowchart illustrating a method of validating a data labeling operation according to an exemplary embodiment.

Referring to FIG. 2, the method of validating a data labeling operation according to the disclosed embodiment includes steps that are processed in a time-series manner in the server 100 shown in FIG. Therefore, even when the method of FIG. 2 is omitted, the description of the server 100 of FIG. 1 can be applied to the method of FIG.

In step S110, the server 100 determines the cross validation level of the data labeling operation. For example, the cross-check level may refer to the number of workers who will assign the same data.

The cross-validation level of the data labeling operation is proportional to the cost of generating the training data, since the compensation to be paid for the same data increases proportionally as the cross-validation level increases.

Also, the higher the level of cross validation, the higher the reliability of the learning data. That is, as the cost of generating the learning data increases, the reliability of the learning data can be increased.

In one embodiment, when the server 100 is requested to generate learning data, it provides information on the reliability of the learning data according to the cost, and determines the level of cross validation according to the request of the requester.

In step S120, the server 100 determines the number of workers to distribute one data according to the cross validation level determined in step S110.

For example, if the cross validation level determined in step S110 is 1, one data may be distributed to only one worker, and if two, one data may be distributed to two workers. Likewise, if the cross validation level determined in step S110 is three, one data can be distributed to three workers.

In step S130, the server 100 distributes the data to the determined number of workers in step S120.

In one embodiment, when the cross validation level is three, three workers are classified as one team, and a plurality of data can be distributed to three workers belonging to one team.

In another embodiment, if the cross validation level is 3, three workers may be arbitrarily selected for each data to distribute the data.

In one embodiment, distributing data to an operator may be to transfer data to each operator's terminal 10-30. In another embodiment, distributing the data to the worker may be to assign each data to an account of each worker, and each worker may obtain data assigned to each worker ' s account via the network.

In step S140, the server 100 acquires the labeling information submitted by each of the workers with respect to the data distributed in step S130.

In one embodiment, obtaining labeling information may be to obtain labeled data. In another embodiment, obtaining labeling information may be obtaining region selection information (e.g., coordinate information) for selecting a particular region of distributed data and information about an object or text included in the selected region .

Referring to FIG. 3, an example of performing labeling on an image including an object is shown.

The image 200 shown in FIG. 3 includes at least one object. The operator can select or select a specific area 210 of the image 200 and information 220 describing the objects contained in the selected area 210. [

In one embodiment, a plurality of objects may be included in one image 200, and an operator may perform labeling of one or more objects included in the image 200.

3, the labeling information includes information on the coordinates of the specific area 210 and information on the objects included in the specific area 210 (for example, information indicating that the object included in the specific area 210 is a bed ).

Referring to FIG. 4, an example of performing labeling on an image including text is shown.

The image 300 shown in FIG. 4 includes at least one text. The operator can select a specific area 310 of the image 300 and input the text 320 contained in the selected area 310. [

In one embodiment, a plurality of texts may be included in one image 300, and an operator may perform labeling of one or more texts contained in the image 300.

In the embodiment shown in FIG. 4, the labeling information includes the coordinates of the specific area 310 and the text contained in the specific area 310.

In step S150, the server 100 compares the labeling information on the data obtained from each of the operators in step S140.

For example, the server 100 compares coordinates included in the labeling information obtained in step S140, and determines whether or not each coordinate is within a predetermined error range with respect to each other. Since the selection of regions may differ slightly from person to person, a predetermined error range should be taken into account in comparing the selected regions.

In addition, the server 100 compares information on each object or text included in the labeling information obtained in step S140. The server 100 determines whether or not the information about each object or the text is matched or recognized.

For example, when different workers input one object with a different name, the server 100 may determine that the labeling information matches when the identities of the respective names are recognized. However, the server 100 may determine a representative name for each object and generate learning data using the determined representative name.

In step S160, as a result of the comparison in step S150, the server 100 requests rework if it is not determined that all of the labeling information obtained from each of the workers matches.

In one embodiment, if the server 100 does not determine that all of the labeling information obtained from each of the workers is identical, the server 100 may request verification from the inspector and obtain the verification result.

The server 100 may obtain the inspection result for the labeling information and determine the worker to request the rework according to the inspection result obtained. The server 100 can request re-work to the determined worker.

In one embodiment, the verification results include information for selecting unsuitable labeling information among the labeling information for the data obtained from each of the operators.

For example, the inspector may select unsuitable labeling information among the labeling information that do not coincide with each other, and the server 100 may obtain the result of selecting the inspector.

The server 100 can request re-work to an operator who has submitted the selected labeling information as unsuitable labeling information among the workers. When the worker submits the reworked labeling information, the server 100 again compares the labeling information of the other workers with the reworked labeling information, and if so, causes the work to end, and if not, requests rework .

In one embodiment, the verification result includes the reason why the non-compliant labeling information is unsuitable. For example, the verification result may include the evaluator's evaluation information, such as an incorrect selection of the area in the labeling information, or an incorrectly entered object or text contained in the area.

When the server 100 requests rework, the server 100 may transmit the reason obtained from the inspector to the submitter who has submitted the improper labeling information.

In one embodiment, the server 100 may obtain a model that can automatically perform validation using machine learning.

For example, the server 100 acquires the learning data using the labeling information obtained from the workers and the inspection result for each labeling information obtained in the process of performing the data labeling job inspection method according to the disclosed embodiment can do.

For example, the server 100 may acquire learning data including the suitability of each labeling information and the reason why each labeling information is inappropriate for improper labeling information.

The server 100 can learn a model for inputting labeling information and outputting the inspection result using the learning data.

In one embodiment, the inspector in accordance with the disclosed embodiment may refer to a model that is learned and used by the server 100, rather than a person. In this case, verification can be performed automatically by the learned model.

For example, the server 100 may input the labeling information into the learned model and obtain the inspection result (e.g., suitability, reason, etc., for the labeling information) inputted as the output thereof .

In one embodiment, the server 100 may request re-work from the worker's power source if all of the comparison results in step S150 do not match, without a separate verification job. In this case, the worker can review the results of his or her work, submit it again if it thinks there is no problem, and submit it again if a problem is found.

In one embodiment, the server 100 may request re-work to all workers until the labeling information of all the workers matches, so that the reliability of the learning data may be increased without performing the inspection process.

In one embodiment, the server 100 may divide each of the obtained labeling information into a plurality of partial labeling information according to a predetermined criterion, and compare the divided partial labeling information with each other.

The server 100 may request the re-work of the partial labeling information that the comparison results are determined to be inconsistent.

Referring to FIG. 5, an example of dividing the labeling information is shown.

In an embodiment, the service according to the disclosed embodiment can be used to label the meter reading of the gas meter.

An image 400 of the gas meter is distributed to one or more operators and the operator designates the specific area 410 in the image 400 and inputs the numbers contained in the area 410 to generate labeling information.

In one embodiment, if the object or text contained in the image 400 is skewed, the operator can specify the area in a skewed shape and add information about the orientation of the object or text to the labeling information .

For example, if the numerical portion of the gas meter shown in Fig. 5 is tilted about 30 degrees counterclockwise, the operator may mark the area of the numerical portion, enter the degree of tilting of the numerical portion at an approximate angle , You can select a specific orientation, or you can display the direction in which the number portion is tilted by drawing an arrow with a drag-and-drop operation along the tilted direction of the number portion.

In one embodiment, data provided to an operator for labeling may include pre-entered labeling guides. For example, there may be an appropriate labeling method depending on the type of data. The server 100 can provide a labeling guide including an explanation of the labeling method or labeling method according to the type of data or a labeling reference to the worker so as to assist the worker to submit a more accurate result.

The server 100 obtains the labeling information 420 in which the operator entered the number included in the area 410. [

In one embodiment, the server 100 may divide the labeling information 420 into a plurality of partial labeling information 421 through 428. For example, the server 100 may divide the numbers included in the labeling information 420 into different labeling information.

The server 100 performs cross validation for each of the divided partial labeling information 421 to 428. [ For example, in the labeling information submitted by a plurality of workers, the first number 421 may be compared with each other, and the second number 422 to the eighth number 428 may be compared with each other.

The server 100 carries out the inspection by the inspector only for the partial labeling information in which the results submitted by all the workers are out of the 8 numbers included in each of the divided partial labeling information 421 to 428, Can be requested.

Accordingly, the server 100 can increase the passage probability of cross validation and reduce the amount of rework. For example, if cross validation of the entire labeling information 420 is performed, rework of all information should be requested even if only one digit is wrong. On the other hand, if the cross validation is performed for each of the divided partial labeling information 421 to 428, it is possible to request rework after checking only the wrong number, which is efficient.

In one embodiment, the divided partial labeling information may include information about each selected number, as well as the selected region and the direction in which the region is tilted. For example, the server 100 performs cross validation on each of the numbers included in the labeling information, the coordinates of the selected area, and the degree of the inclination of the area, and according to the result, You can proceed with a request for review and reconsideration.

In one embodiment, the server 100 may dispense the divided points for each of the divided labels, once the cross validation or validation for each of the divided labeling information is complete. In another embodiment, the server 100 may pay points to the operator once cross validation or validation of all of the partitioned labeling information is complete.

6 is a view showing a maintenance payment method according to an embodiment.

Step S510 is a step in which the operator performs a labeling operation on data in a state where data is allocated to the operator.

When the job is completed, the operator submits the labeling information to the server 100 and acquires the corresponding point (step S520).

In step S520, the server 100 performs cross validation. The server 100 proceeds to check for the labeling information that has not passed the cross validation (i.e., not all match) (step S530).

As a result of the inspection, the dispensed point is determined for the operator who submitted the appropriate labeling information, and the state is switched to the state where the operator can pay for the point (step S540).

That is, the point is paid to the worker in accordance with the completion of the work in step S520, but the worker can not receive the compensation for the point until the point is confirmed through the inspection.

As a result of the inspection, the operator who submitted the inappropriate labeling information requests the re-work, and the operator performs the labeling operation again (step S510).

When the operator performs the labeling operation again, the operator determines whether the point is confirmed through cross validation with appropriate labeling information or validation by the inspector.

The steps of a method or algorithm described in connection with the embodiments of the present invention may be embodied directly in hardware, in software modules executed in hardware, or in a combination of both. The software module may be a random access memory (RAM), a read only memory (ROM), an erasable programmable ROM (EPROM), an electrically erasable programmable ROM (EEPROM), a flash memory, a hard disk, a removable disk, a CD- May reside in any form of computer readable recording medium known in the art to which the invention pertains.

While the present invention has been described in connection with what is presently considered to be practical exemplary embodiments, it is to be understood that the invention is not limited to the disclosed embodiments, but, on the contrary, You will understand. Therefore, it should be understood that the above-described embodiments are illustrative in all aspects and not restrictive.

10: worker terminal
20: worker terminal
30: worker terminal
50: Inspector terminal
100: Server

Claims (10)

Determining a cross-validation level of the server;
The server determining the number of workers to distribute one data according to the determined cross-check level;
Distributing the first data to the determined number of worker terminals;
Obtaining labeling information for the first data submitted by each of the worker terminals by the server;
Comparing the labeling information for the first data obtained from each of the operator terminals with the server; And
Requesting re-work if the server does not determine that all of the labeling information for the first data obtained from each of the worker terminals matches; / RTI >
Wherein the first data comprises:
Wherein the image data is image data including a specific object or text. The method according to claim 1,
Wherein the step of requesting re-
Obtaining a verification result for the labeling information from the inspector terminal by issuing a verification request for the labeling information to the inspector terminal;
Determining an operator terminal to request rework according to the verification result; And
Requesting the determined worker terminal to rework; Wherein the data labeling operation verification method comprises: 3. The method of claim 2,
The step of acquiring the verification result includes:
Acquiring, from the acceptor terminal, information for selecting unsuitable labeling information among the labeling information for the first data obtained from each of the operator terminals; Lt; / RTI >
Wherein the step of requesting re-
Requesting a worker terminal that has submitted labeling information selected as unsuitable labeling information among the worker terminals to rework; Wherein the data labeling operation verification method comprises: The method of claim 3,
The step of acquiring the verification result includes:
Obtaining a reason for the unsuitable labeling information being unsuitable from the inspector terminal; Lt; / RTI >
Wherein the step of requesting re-
Transmitting the obtained reason to the worker terminal that has submitted the inappropriate labeling information; Further comprising the step of validating the data labeling operation. 3. The method of claim 2,
Obtaining learning data including inspection results for labeling information and labeling information obtained from worker terminals; And
Learning a model capable of outputting a verification result for labeling information using the learning data; Wherein the data labeling operation verification method comprises: 6. The method of claim 5,
The step of acquiring the verification result includes:
Inputting labeling information for the first data into the learned model; And
Obtaining the acceptance result from the learned model; Wherein the data labeling operation verification method comprises: The method according to claim 1,
Wherein the step of requesting re-
Requesting all of the worker terminals to rework until it is determined that all of the labeling information for the first data obtained from each of the worker terminals are identical; Wherein the data labeling operation verification method comprises: The method according to claim 1,
Paying a fee to each of the worker terminals when it is determined by the server that the labeling information for the first data obtained from each of the worker terminals is identical; Further comprising the step of validating the data labeling operation. The method according to claim 1,
Wherein comparing the labeling information comprises:
Dividing each of the obtained labeling information into a plurality of partial labeling information according to a predetermined criterion; And
Comparing the divided partial labeling information, respectively; Lt; / RTI >
Wherein the step of requesting re-
Requesting rework of partial labeling information that is determined to be incompatible with the comparison results; Wherein the data labeling operation verification method comprises: A computer program stored in a computer-readable recording medium in combination with a computer which is hardware.

Images