KR102321007B1 - Method and apparatus for determining whether to introduce machine learning models for labeling tasks according characteristics of crowdsourcing based on projects - Google Patents

Abstract

The present invention relates to a method and apparatus for determining whether to introduce machine learning models for labeling tasks according to characteristics of crowdsourcing-based projects, which reduces time loss and cost loss required for a project. According to one embodiment of the present invention, the method comprises the following steps of: calculating a matching score between a project and at least one pre-trained machine learning model when the project for a labeling task is requested; checking whether a pre-trained machine learning model for the labeling task exists among the at least one pre-trained machine learning model on the basis of the calculated matching score; determining whether to perform automatic labeling or manual labeling according to whether the pre-trained machine learning model for the labeling task exists on the basis of a checking result; and performing automatic labeling on the basis of the checked pre-trained machine learning model when the pre-trained machine learning model for the labeling task exists on the basis of the checking result, and comparing a first cost required when manually performing the labeling task with a cost when automatically performing the labeling task by using the pre-trained machine learning model or custom model, which has the calculated matching score within a preset threshold range, and determining whether to perform automatic labeling or manual labeling according to a comparison result when the pre-trained machine learning model for the labeling task does not exist on the basis of the checking result.

Description

Method and device for determining whether to introduce machine learning models for labeling tasks according to the characteristics of crowdsourcing-based projects

The present invention relates to a method and apparatus for determining whether to introduce a machine learning model for a labeling task according to the characteristics of a crowdsourcing-based project, and more specifically, to an operator whether or not to automatically process the labeling task based on an artificial intelligence-based machine learning model. It relates to a method and apparatus for deciding whether to introduce a machine learning model for labeling work according to the characteristics of a crowdsourcing-based project, which allows the user to decide whether to process it manually or not.

More and more companies are collecting and processing large amounts of data based on crowdsourcing that engages the general public in some processes of business activities.

Crowdsourcing is a compound word of "crowd" and "outsourcing".

This crowdsourcing is a structure suitable for the gig economy, which has the meaning of fixed-term work traded in the digital marketplace, and is optimized for on demand, in which companies provide services immediately to the needs of consumers. may be provided.

Through this, if the public is involved in the company's activities, new ideas and practical opinions can be heard from the company's point of view, and the public can be paid for participating in feedback, and RPA technology based on cloud and AI, cognitive and Analysis) combined with Digital Workforce can be used to provide high-quality project results to consumers.

In addition, using crowdsourcing has the advantage of lowering development costs and obtaining potential customers if the public participates directly and brings out the desired results rather than entrusting it to an external professional to make or service a product.

Recently, crowdsourcing is used for data labeling services such as data labeling automation, distributed processing design, online non-face-to-face management, and human resource management (HR Tech). It is being actively used in industries that require learning data.

Specifically, when one project is opened, a plurality of tasks are assigned to each of a plurality of workers. Each worker performs a plurality of assigned tasks and provides task results. Thereafter, a plurality of inspection tasks for the work results are assigned to each of the plurality of inspectors, and each inspector performs the assigned plurality of inspection tasks.

However, when labeling of work data is manually performed through a plurality of workers, each task is assigned to a plurality of workers, and the performance results are collected from each worker. In this case, the amount of data is huge. The more time-efficient, the less cost-effective.

Therefore, depending on whether there exists a pre-trained machine learning model to process the entire job data that needs to perform the labeling task, manual labeling by the operator taking into account the cost, or automatic labeling through fine-tuning or model development Decision-making technology needs to be developed.

Korean Patent Application Laid-Open No. 10-2020-0065191 (Registration date: May 29, 2020)

The present invention has been proposed to solve the above problems, and manual labeling by the operator is performed in consideration of the cost depending on whether there is a pre-trained machine learning model for processing the entire job data that needs to perform the labeling operation. A machine learning model for labeling tasks according to the characteristics of crowdsourcing-based projects that can reduce time and cost losses for the project by deciding to perform or perform automatic labeling through fine tuning or model development To provide a method and apparatus for determining whether to introduce.

The problems to be solved by the present invention are not limited to the problems mentioned above, and other problems not mentioned will be clearly understood by those skilled in the art from the following description.

The method for determining whether to introduce a machine learning model for labeling work according to the characteristics of the crowdsourcing-based project performed by the device according to an embodiment of the present invention for solving the above-described problems is, when a project for the labeling task is requested, calculating a matching score between the project and at least one pre-trained machine learning model; checking whether a pre-trained machine learning model for the labeling task exists among the at least one pre-trained machine learning model based on the calculated matching score; and determining whether automatic labeling or manual labeling according to the confirmation result, whether a pre-trained machine learning model for the labeling task exists, and the confirmation result, pre-trained machine learning for the labeling task When a model exists, automatic labeling is performed based on the confirmed pre-trained machine learning model, and when a pre-trained machine learning model for the labeling task does not exist, the labeling operation is performed manually Comparing the first cost required and the second cost required for automatically performing the labeling operation using a pre-trained machine learning model or a custom model in which the calculated matching score is within a preset threshold range, and the comparison It may further include the step of determining whether automatic labeling or manual labeling according to the result.

On the other hand, the apparatus for determining whether to introduce a machine learning model for labeling work according to the characteristics of a crowdsourcing-based project according to an embodiment of the present invention includes a request server for requesting the project and a terminal of at least one worker put into the project and a communication communication module for performing communication; a storage module for storing at least one process necessary for determining whether to introduce a machine learning model for the labeling task; and a control module for controlling an operation for determining whether to introduce the machine learning model based on the at least one process, wherein the control module is, when a project for the labeling operation is requested, the project and at least one calculates a matching score between pre-trained machine learning models of Determine whether automatic labeling or manual labeling is determined according to whether a pre-trained machine learning model for the labeling task exists as a result of the confirmation, and as a result of the confirmation, a pre-trained machine for the labeling task If a learning model exists, automatic labeling is performed based on the identified pre-trained machine learning model, and if there is no pre-trained machine learning model for the labeling task, the labeling operation is performed manually. Comparing the first cost required when performing the labeling operation and the second cost of automatically performing the labeling operation using a pre-trained machine learning model or a custom model in which the calculated matching score is within a preset threshold range, It can be controlled to determine whether automatic labeling or manual labeling is performed according to the comparison result.

Other specific details of the invention are included in the detailed description and drawings.

According to the present invention, manual labeling is performed by an operator in consideration of the cost depending on whether there is a pre-trained machine learning model for processing the entire job data that needs to perform the labeling operation, or automatic labeling through fine tuning or model development to reduce the time loss and cost loss required for the project.

Effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the following description.

1 is a block diagram illustrating a network structure of a system for determining whether to introduce a machine learning model for a labeling operation according to the characteristics of a crowdsourcing-based project according to an embodiment of the present invention.
2 is a block diagram showing the configuration of an apparatus for determining whether to introduce a machine learning model for a labeling operation according to the characteristics of a crowdsourcing-based project according to an embodiment of the present invention.
3 is a flowchart illustrating a method of determining whether to introduce a machine learning model for a labeling operation according to the characteristics of a crowdsourcing-based project according to an embodiment of the present invention.
4 is a flowchart illustrating a method for calculating a second cost in a decision apparatus according to an embodiment of the present invention.
5 is a view for explaining an example of calculating a first cost and a second cost in the determination device according to an embodiment of the present invention.

Advantages and features of the present invention and methods of achieving them will become apparent with reference to the embodiments described below in detail in conjunction with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but may be implemented in various different forms, and only these embodiments allow the disclosure of the present invention to be complete, and those of ordinary skill in the art to which the present invention pertains. It is provided to fully understand the scope of the present invention to those skilled in the art, and the present invention is only defined by the scope of the claims.

The terminology used herein is for the purpose of describing the embodiments and is not intended to limit the present invention. As used herein, the singular also includes the plural unless specifically stated otherwise in the phrase. As used herein, “comprises” and/or “comprising” does not exclude the presence or addition of one or more other components in addition to the stated components. Like reference numerals refer to like elements throughout, and "and/or" includes each and every combination of one or more of the recited elements. Although "first", "second", etc. are used to describe various elements, these elements are not limited by these terms, of course. These terms are only used to distinguish one component from another. Accordingly, it goes without saying that the first component mentioned below may be the second component within the spirit of the present invention.

Unless otherwise defined, all terms (including technical and scientific terms) used herein will have the meaning commonly understood by those of ordinary skill in the art to which this invention belongs. In addition, terms defined in a commonly used dictionary are not to be interpreted ideally or excessively unless specifically defined explicitly.

Spatially relative terms "below", "beneath", "lower", "above", "upper", etc. It can be used to easily describe the correlation between a component and other components. A spatially relative term should be understood as a term that includes different directions of components during use or operation in addition to the directions shown in the drawings. For example, when a component shown in the drawing is turned over, a component described as “beneath” or “beneath” of another component may be placed “above” of the other component. can Accordingly, the exemplary term “below” may include both directions below and above. Components may also be oriented in other orientations, and thus spatially relative terms may be interpreted according to orientation.

As used herein, the term “unit” or “module” refers to a hardware component such as software, FPGA, or ASIC, and “unit” or “module” performs certain roles. However, "part" or "module" is not meant to be limited to software or hardware. A “part” or “module” may be configured to reside on an addressable storage medium and may be configured to reproduce one or more processors. Thus, by way of example, “part” or “module” refers to components such as software components, object-oriented software components, class components and task components, processes, functions, properties, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuitry, data, databases, data structures, tables, arrays and variables. Components and functionality provided within “parts” or “modules” may be combined into a smaller number of components and “parts” or “modules” or additional components and “parts” or “modules”. can be further separated.

Unless otherwise defined, all terms (including technical and scientific terms) used herein will have the meaning commonly understood by those of ordinary skill in the art to which this invention belongs. In addition, terms defined in a commonly used dictionary are not to be interpreted ideally or excessively unless specifically defined explicitly.

In the present specification, a project is performed based on crowd sourcing that engages the public in some process of corporate activity, and may further include a task as a set of sub-actions.

In the present specification, the evaluation function and/or evaluation score may refer to a method for evaluating a project performance result (work result, result, work, etc.) and an output score derived through evaluation.

A crowdsourcing service-based project that creates and provides a request by a client with a large number of people as workers, and provides a reward corresponding to the work to the workers who were put in to create the request, is a project where the client can In the case of a labeling request, a plurality of participants are selected and put in for the labeling work. Due to the nature of the work data, if the amount of work data is vast, the compensation cost for the worker is inevitably increased in proportion to the amount of work data, which increases the cost burden. get bigger

The present invention automatically processes the labeling of the job data based on the cost and pre-trained machine learning model when manually processing the labeling for the job data by at least one operator (the first processing method) (the second processing method) processing method), compares the cost, and selects a method that can reduce costs among the two methods to determine to perform labeling on the work data. Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. However, for convenience of explanation, it is limited to a worker, and this worker may be a worker performing labeling on work data, as well as other subjects corresponding to human resources.

1 is a block diagram illustrating a network structure of a system for determining whether to introduce a machine learning model for a labeling operation according to the characteristics of a crowdsourcing-based project according to an embodiment of the present invention.

Referring to FIG. 1, the system for determining whether to introduce a machine learning model for a labeling operation is a device for determining whether to introduce a machine learning model for a labeling operation (hereinafter referred to as a 'determining device') 100, a request server 200, and a worker The terminal 300 is configured and performed, and in this case, the operator terminal 300 may be at least one.

The determination device 100 represents a device or terminal of a company (company) for providing a crowdsourcing service, and before opening a requested (open) project on a platform, at least one worker is put in to manually label It decides whether to process by performing (manual labeling) or automatically performing labeling based on a pre-trained machine learning model (automatic labeling). At this time, the determination device 100 considers the cost of performing labeling through each of the two processing methods to determine whether manual labeling or automatic labeling, and determines to process the labeling in a way that requires less cost.

That is, the determination device 100 calculates the cost (first cost) required when the job data is processed by manual labeling and the cost (second cost) required when the job data is processed by automatic labeling, respectively, When the first cost is greater than the second cost, it is determined to perform manual labeling, and when the first cost is less than or equal to the second cost, it is determined to perform automatic labeling.

On the other hand, when the determination device 100 receives a request for a project for labeling work by receiving work request information from the request server 200, based on the work request information and at least one pre-learned machine learning model, Check whether there is a pre-trained machine learning model that can be used for the labeling task of the project, and if there is a pre-trained machine learning model, use the pre-trained machine learning model to automatically label the job data , and if there is no pre-trained machine learning model, at least one worker is selected and requested to perform manual labeling on the job data.

Here, the determination device 100 extracts a label class from the work request information to check whether there is a pre-trained machine learning model that can be used for the labeling task of the project, and the pre-trained machine learning model that holds Extract the label class from each specification document. Thereafter, the determination device 100 calculates a matching score based on the similarity between the two label classes, and whether to use the pre-trained machine learning model as it is, fine-tune it, or learn a custom model based on the matching score. decide whether to use Here, the similarity can be confirmed by vectorizing each label class to the word level and calculating the cosine similarity between the obtained vectors.

For example, when a pre-trained machine learning model with a matching score equal to or greater than a preset threshold range is detected, the corresponding labeling task is automatically performed using the detected pre-trained machine learning model. In addition, when a pre-trained machine learning model whose matching score is within a preset threshold range is detected, the detected pre-trained machine learning model is fine-tuned to automatically perform the corresponding labeling task. In addition, when a pre-trained machine learning model whose matching score is less than or equal to a preset threshold range is detected, the corresponding labeling task is automatically performed using a custom model. Here, each threshold value and threshold range may be set or changed by a user or an administrator, and the numerical values are not limited.

The request server 200 refers to a company or individual requesting a project, that is, a device of a client.

The client requests a project for the purpose of collecting source data or data annotation for the generation of artificial intelligence learning data through the request server 200 . The data generated through the project can be used as learning data for arbitrary machine learning such as supervised learning, unsupervised learning, and reinforcement learning. The collection of source data means the collection of raw data such as recorded voice collection and photo collection. Data annotation refers to inputting relevant annotation data into source data such as text, photos, and videos. For example, data annotation may include, but is not limited to, finding an entity in a given fingerprint, finding a similar sentence, and the like. On the other hand, the type of the above-mentioned project is only one embodiment, and various projects may be handled in the present invention according to the design of the client.

The request server 200 requests at least one project (project A, project B, ?? The work request information including at least one of the completion deadline, requester and worker requirements (request condition) is transmitted together.

In this case, the job data may be an image such as a photo or a moving picture, or text such as a document, and the job request information may be a standardized image document, an electronic document, or an unstructured text document. Here, the work type may be a work instruction or a work guide for what kind of work should be performed.

In addition, the work types are collection, bounding, segmentation, sentiment, OCR (Optical Character Recognition), landmark, labeling, attribute classification, tagging, section extraction, characteristic evaluation. etc. However, in the present invention, the description will be limited to the case where the operation form is labeled.

The worker terminal 300 refers to a terminal of the general public selected as a worker to be put into an open project opened on the platform.

The selected worker performs the job requested by the determination device 100 using the worker terminal 300 , that is, a predetermined terminal. Here, the operator terminal 300 may be configured in plurality, and the number is not limited. Each worker terminal 300 may participate in (inject) an open project through an application or website provided by the determination device 100 .

When a work request for an open project on a platform is received from the determining device 100 , the first worker terminal 301 and the second worker terminal 302 may determine and notify whether to participate in the work for the project.

When participating in a job, the first worker terminal 301 and the second worker terminal 302 perform the requested job according to the job request, and transmit the performance result, that is, the job result on the platform.

On the other hand, the worker terminal 300 is a mobile terminal, a mobile phone, a smart phone (smart phone), a laptop computer (laptop computer), a desktop computer (desktop computer), a digital broadcasting terminal, PDA (personal digital assistants), PMP (portable multimedia), respectively player), navigation, slate PC, tablet PC, ultrabook, wearable device, for example, watch-type terminal (smartwatch), glass-type terminal (smart glass), It may be a computer device such as a head mounted display (HMD) or a telecommunication device, but is not limited thereto.

2 is a block diagram showing the configuration of an apparatus for determining whether to introduce a machine learning model for a labeling operation according to the characteristics of a crowdsourcing-based project according to an embodiment of the present invention.

Referring to FIG. 2 , the determining device 100 may include a communication module 101 , a storage module 103 , and a control module 105 .

The communication module 101 transmits and receives at least one piece of information or data with the request server 200 and at least one worker terminal 300 to perform (complete) at least one project requested by the request server 200 . . In addition, the communication module 101 may perform communication with the request server 200 and at least one operator terminal 300, and other devices, and transmits a wireless signal in a communication network according to wireless Internet technologies. to transmit and receive.

As wireless Internet technologies, for example, WLAN (Wireless LAN), Wi-Fi (Wireless-Fidelity), Wi-Fi (Wireless Fidelity) Direct, DLNA (Digital Living Network Alliance), WiBro (Wireless Broadband), WiMAX (World Interoperability for Microwave Access), High Speed Downlink Packet Access (HSDPA), High Speed Uplink Packet Access (HSUPA), Long Term Evolution (LTE), Long Term Evolution-Advanced (LTE-A), etc., and the condition setting device 100 ) transmits and receives data according to at least one wireless Internet technology to the extent that it includes Internet technologies not listed above.

As for short range communication, Bluetooth™, RFID (Radio Frequency Identification), Infrared Data Association (IrDA), UWB (Ultra Wideband), ZigBee, NFC (Near Field Communication), Wi - At least one of Wireless-Fidelity (Fi), Wi-Fi Direct, and Wireless Universal Serial Bus (USB) technologies may be used to support short-distance communication. It is possible to support wireless communication between the determining device 100 and the requesting server 200, the determining device 100, and the at least one worker terminal 300 for wireless area networks. In this case, the local area network may be a local area network (Wireless Personal Area Networks).

The storage module 103 stores information on an open project and information on at least one previously performed completed project, and stores at least one process necessary for determining whether to introduce a machine learning model for labeling work.

The control module 105 controls an operation for determining whether to introduce a machine learning model based on at least one process stored in the storage module 103 .

Specifically, when a project for labeling work is requested from the request server 200, the control module 105 calculates a matching score between the project and at least one pre-trained machine learning model possessed, and the Based on the calculated matching score, it is checked (detected) whether there is a pre-trained machine learning model that can be used for the requested labeling task among at least one pre-trained machine learning model, and as a result of the check, the Whether to label job data automatically or manually depends on whether there are pre-trained machine learning models available.

As a result of the check, if there is a pre-trained machine learning model for the labeling task, the control module 105 performs automatic labeling based on the checked pre-trained machine learning model.

On the other hand, as a result of the check, if there is no pre-trained machine learning model for the labeling task, the control module 105 controls the first cost required for manually performing the labeling task and the pre-trained machine learning model or custom Comparing the second cost required for automatically performing the labeling operation using the model, it is determined whether automatic labeling or manual labeling is performed according to the comparison result. Here, the pre-trained machine learning model considered to calculate the second cost is a pre-trained machine learning model in which the previously calculated matching score is within a preset threshold range.

At this time, when the first cost is greater than the second cost, the control module 105 selects at least one operator and assigns a labeling task to perform manual labeling. On the other hand, when the first cost is less than or equal to the second cost, the control module 105 performs an automatic labeling operation using a pre-trained model or a custom model, but the previously calculated matching score is within a preset threshold range If a pre-trained model exists, automatic labeling is performed using the pre-trained model. Allows automatic labeling to be performed.

3 is a flowchart illustrating a method of determining whether to introduce a machine learning model for a labeling operation according to the characteristics of a crowdsourcing-based project according to an embodiment of the present invention.

Determination device 100 receives a request for a project for labeling work as it receives work request information from the request server 200 (S201), and can be used to process the labeling task before opening the project on the platform. In order to check whether a pre-trained machine learning model exists, a matching score between the project and at least one pre-trained model is calculated ( S203 ).

Next, the determination device 100 checks whether there is a pre-trained machine learning model for the labeling task based on the calculated matching score, that is, detects the pre-trained machine learning model (S205), and the check As a result, when the pre-trained machine learning model is detected, the determination apparatus 100 performs automatic labeling using the detected pre-trained machine learning model ( S207 ). In this case, the detected pre-trained machine learning model represents a pre-trained machine learning model that can be applied as it is for the labeling task, and the matching score may completely match.

In addition, as a result of the confirmation, when the pre-trained machine learning model is not detected, the determination device 100 automatically handles the first cost and the labeling operation required when the labeling operation is manually processed (manual labeling) Calculate the second cost required in the case (automatic labeling) (S209), and as a result of the comparison, if the first cost is greater than the second cost, select at least one operator and request to manually perform the labeling operation ( S211).

On the other hand, as a result of the comparison, if the first cost is less than or equal to the second cost, the determining device 100 checks the matching score calculated in step S203 ( S213 ), and the calculated matching score is within a preset threshold range. If there is a pre-trained model, automatic labeling is performed based on the pre-trained model (S215), or if there is only a pre-trained model whose matching scores are all below a preset threshold range, automatic modeling is performed based on the custom model do (S217).

4 is a flowchart illustrating a method for calculating a second cost in a decision apparatus according to an embodiment of the present invention.

Referring to FIG. 4 , the determination apparatus 100 repeatedly performs a calculation operation of calculating a cost and its accuracy (job accuracy) by using a preset number of training data from among all job data.

First, the determination device 100 performs learning by using a preset number of training data among all job data based on a pre-trained machine learning model or a custom model, The cost incurred thereby is calculated by adding up the cost of using and inspecting the work for the number of learning data and the cost of inspecting the number of work data in the data pool (S301). Here, the number of work data in the data pool is a value obtained by subtracting the number of training data from the total number of work data, which means the number of remaining data excluding the training data. That is, the number of working data in the data pool represents the number of unlearned data. Since the accuracy must be calculated using the work data of this data pool, inspection is required, and therefore the inspection cost for the number of work data in the data pool must be added up in step S301.

Next, the accuracy is calculated through the work data inspection of the data pool (S303), and the calculated accuracy is applied for the calculation operation in the next loop.

After step S303, the number of training data and the number of working data in the data pool are compared. As a result of the comparison, if the number of training data is smaller than the number of working data in the data pool, the calculation operation of steps S301 to S305 is repeated. However, in the loop performed after the first performed loop, the cost is calculated by reflecting the accuracy in the number of work data in the data pool when step S301 is performed. That is, the correctly processed data is removed from the data pool and then used in the next loop.

Therefore, as the accuracy calculated in step S303 is higher, the number of loops to be repeatedly performed is reduced. In addition, the accuracy of steps S301 to S305 is kept the same or improved according to iterative learning.

The calculation operation of steps S301 to S303 is repeatedly performed until the number of work data in the data pool becomes smaller than the number of training data. And after summing the inspection cost (S307), the calculation operation is no longer repeated and is summed together with all the calculated values calculated by repeating the previous n times.

Accordingly, it is possible to calculate the second cost incurred when automatic labeling is performed by learning or developing a pre-trained model or a custom model.

5 is a view for explaining an example of calculating a first cost and a second cost in the determination device according to an embodiment of the present invention.

With reference to FIG. 5 , the first cost and the second cost required when the total number of work data is 1000, the work cost is 10 won, the inspection cost is 1 won, and the number of learning data is set to 200 are calculated, respectively.

First, the first cost can be calculated by adding up the work cost and the inspection cost for the total number of work data, and can be expressed as in Equation 1 below.

According to Equation 1, the first cost is calculated as 11,000 won.

On the other hand, the second cost repeats the calculation operation by summing the work cost and inspection cost for the number of learning data and the inspection cost for the number of work data in the data pool, and all of the calculated values as a result of the repeated execution It can be obtained by summing, and can be expressed as in Equation 2 below.

However, since the loops performed after the first loop must be used except for the accurately worked data from the work data of the data pool, the accuracy is reflected in the number of work data in the data pool and used. 5, the accuracy is limited to 20% for convenience of explanation, whereby the number of working data in the data pool in loop 2 is 440, and the number of working data in the data pool in loop 3 is 152, As the number of data becomes smaller than the number of training data, the number of working data in the data pool in the last loop 4 becomes 152.

According to Equation 2, the second cost is calculated as 9,664 won.

As a result of the calculation, since the first cost is 11,000 won and the second cost is 9,664 won, it can be determined that less cost is required when automatic labeling is performed.

Therefore, it is decided to proceed with the labeling operation using a pre-trained machine learning model or a custom model (ML model).

The steps of a method or algorithm described in relation to an embodiment of the present invention may be implemented directly in hardware, as a software module executed by hardware, or by a combination thereof. A software module may contain random access memory (RAM), read only memory (ROM), erasable programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory, hard disk, removable disk, CD-ROM, or It may reside in any type of computer-readable recording medium well known in the art to which the present invention pertains.

As mentioned above, although embodiments of the present invention have been described with reference to the accompanying drawings, those skilled in the art to which the present invention pertains can realize that the present invention can be embodied in other specific forms without changing its technical spirit or essential features. you will be able to understand Therefore, it should be understood that the embodiments described above are illustrative in all respects and not restrictive.

100: decision device 200: request server
300: operator terminal 301: first operator terminal
302: second worker terminal 110: communication module
130: storage module 150: control module

Claims (17)

In the method of determining whether to introduce a machine learning model for the labeling operation of a crowdsourcing-based project performed by a device for determining whether to introduce a machine learning model for the labeling operation of the crowdsourcing-based project,
calculating a matching score between the project and at least one pre-trained machine learning model when a project for the labeling task is commissioned;
checking whether a pre-trained machine learning model for the labeling task exists among the at least one pre-trained machine learning model based on the calculated matching score; and
As a result of the confirmation, determining whether automatic labeling or manual labeling is performed according to whether a pre-trained machine learning model for the labeling task exists,
As a result of the above confirmation,
If there is a pre-trained machine learning model for the labeling task, performing automatic labeling based on the confirmed pre-trained machine learning model,
When a pre-trained machine learning model for the labeling task does not exist, the first cost required for manually performing the labeling task and the pre-trained machine learning in which the calculated matching score is within a preset threshold range Comparing the second cost required for automatically performing the labeling operation using a model or a custom model, and further comprising the step of determining whether automatic labeling or manual labeling is performed according to the comparison result,
As a result of the above comparison,
When the first cost is greater than the second cost, manual labeling is performed by assigning the labeling task to at least one operator,
When the first cost is less than or equal to the second cost, a pre-trained machine learning model in which the calculated matching score is within a preset threshold range among the at least one pre-trained machine learning model is used, or the custom It performs automatic labeling work using the model,
The first cost is obtained by adding up the work cost and the inspection cost for the total number of work data to be performed the labeling work,
The second required cost is the sum of the work cost and inspection cost for the number of learning data to be learned among all the work data for which the labeling operation needs to be performed, and the inspection cost for the number of work data in the data pool After repeatedly performing the calculation operation to obtain the data pool until the number of work data in the data pool becomes smaller than the number of learning data, the obtained respective calculated values are summed and obtained,
It is defined as one loop that the calculation operation is performed once,
When performing the iteration, it is characterized in that the ratio of the data accurately worked through inspection among the work data of the data pool is reflected as accuracy in the loops performed after the first loop,
How to decide whether to introduce machine learning models for labeling tasks.
According to claim 1,
The matching score is
It is characterized in that it is calculated based on the similarity between the label class extracted from the work request information for the project and the label class extracted through the specification document of each of the at least one pre-trained machine learning model,
How to decide whether to introduce machine learning models for labeling tasks.
3. The method of claim 2,
The similarity is
Characterized in that each of the label classes is vectorized at the word level, and the cosine similarity between the obtained vectors is calculated,
How to decide whether to introduce machine learning models for labeling tasks.
delete delete According to claim 1,
When the number of work data in the data pool is smaller than the number of training data, the calculation operation using only the number of work data in the data pool is finally stopped,
How to decide whether to introduce machine learning models for labeling tasks.
According to claim 1,
The number of learning data is determined as a preset number,
The calculation operation using only the number of work data in the data pool is characterized in that the work cost and the inspection cost for the number of work data in the data pool are summed,
How to decide whether to introduce machine learning models for labeling tasks.
8. The method of claim 7,
The number of work data in the data pool is,
Characterized in that the calculation is performed by reflecting the accuracy according to the operation performed immediately before in the number of remaining data except for the number of learning data from the total number of work data,
How to decide whether to introduce machine learning models for labeling tasks.
In the device for determining whether to introduce a machine learning model for the labeling task of a crowdsourcing-based project,
a communication module for performing communication with a request server for requesting the project and a worker terminal, which is a terminal of at least one worker put into the project;
a storage module for storing information on the project and information on at least one previously performed completed project, and storing at least one process necessary to determine whether to introduce a machine learning model for the labeling operation; and
A control module for controlling an operation for determining whether to introduce the machine learning model based on the at least one process,
The control module is
When a project for the labeling task is requested, a matching score between the project and at least one pre-trained machine learning model is calculated, and the at least one pre-trained machine learning model is calculated based on the calculated matching score. Check whether there is a pre-trained machine learning model for the labeling task among them, and determine whether automatic labeling or manual labeling according to whether there is a pre-trained machine learning model for the labeling task as a result of the confirmation and
As a result of the above confirmation,
If there is a pre-trained machine learning model for the labeling task, performing automatic labeling based on the confirmed pre-trained machine learning model,
When a pre-trained machine learning model for the labeling task does not exist, the first cost required for manually performing the labeling task and the pre-trained machine learning in which the calculated matching score is within a preset threshold range Comparing the second cost required when automatically performing the labeling operation using a model or a custom model, and controlling whether to automatically label or manually label according to the comparison result,
As a result of the above comparison,
When the first cost is greater than the second cost, manual labeling is performed by assigning the labeling task to at least one operator,
When the first cost is less than or equal to the second cost, a pre-trained machine learning model in which the calculated matching score is within a preset threshold range among the at least one pre-trained machine learning model is used, or the custom It performs automatic labeling work using the model,
The first cost is obtained by adding up the work cost and the inspection cost for the total number of work data to be performed the labeling work,
The second required cost is the sum of the work cost and inspection cost for the number of learning data to be learned among all the work data for which the labeling operation needs to be performed, and the inspection cost for the number of work data in the data pool After repeatedly performing the calculation operation to obtain the data pool until the number of work data in the data pool becomes smaller than the number of learning data, the obtained respective calculated values are summed and obtained,
It is defined as one loop that the calculation operation is performed once,
When performing the iteration, it is characterized in that the ratio of the data accurately worked through inspection among the work data of the data pool is reflected as accuracy in the loops performed after the first loop,
A device for deciding whether to introduce machine learning models for labeling tasks.
10. The method of claim 9,
The matching score is
It is characterized in that it is calculated based on the similarity between the label class extracted from the work request information for the project and the label class extracted through the specification document of each of the at least one pre-trained machine learning model,
A device for deciding whether to introduce machine learning models for labeling tasks.
11. The method of claim 10,
The similarity is
Characterized in that each of the label classes is vectorized at the word level, and the cosine similarity between the obtained vectors is calculated,
A device for deciding whether to introduce machine learning models for labeling tasks.
delete delete 10. The method of claim 9,
When the number of work data in the data pool becomes smaller than the number of learning data, the calculation operation using only the number of work data in the data pool is finally stopped,
How to decide whether to introduce machine learning models for labeling tasks.
10. The method of claim 9,
The number of learning data is determined as a preset number,
The calculation operation using only the number of work data in the data pool is characterized in that the work cost and the inspection cost for the number of work data in the data pool are summed,
A device for deciding whether to introduce machine learning models for labeling tasks.
16. The method of claim 15,
The number of work data in the data pool is,
Characterized in that the calculation is performed by reflecting the accuracy according to the operation performed immediately before in the number of remaining data except for the number of learning data from the total number of work data,
A device for deciding whether to introduce machine learning models for labeling tasks.
A computer-readable medium for executing a method for determining whether to introduce a machine learning model for a labeling operation of a crowdsourcing-based project according to any one of claims 1 to 3, 6 to 8, in combination with a computer stored in a computer program.

Images