KR102480575B1 - Method and apparatus for cloud outsorcing task distribution by using artificial intelligent - Google Patents

Abstract

In the crowd outsourcing task distribution method using artificial intelligence according to an embodiment of the present invention, a task data generation step in which a processor controlled by an artificial intelligence module sets a work target area of work target data requiring work to generate work data ; Transmission step of the communication unit transmitting the work data to some worker terminals among a plurality of worker terminals connected to the Internet; and a receiving and learning step of learning the artificial intelligence module based on the completion data when the communication unit receives the completion data in which the work for the work data is completed from the plurality of worker terminals. and repeating the receiving step when transmitting to a predetermined number of terminals among the plurality of worker terminals or until the job for the work target data is completed.

Description

Crowd outsourcing task distribution method and apparatus using artificial intelligence {METHOD AND APPARATUS FOR CLOUD OUTSORCING TASK DISTRIBUTION BY USING ARTIFICIAL INTELLIGENT}

The present invention relates to a method and apparatus for crowd outsourcing job distribution using artificial intelligence. More specifically, it relates to a crowd outsourcing job distribution method and apparatus using artificial intelligence capable of efficient pre-processing and distribution of outsourcing jobs and collection of work results.

Recently, with the development of artificial intelligence (AI) technology, research on artificial intelligence that recognizes specific objects included in digital content such as images, voices, and texts has been actively conducted. In particular, artificial intelligence technology that recognizes a specific object in a digital image created through a digital camera and determines the situation of the area where it was filmed is used in various industries, such as being applied to surveillance cameras and self-driving cars.

In order to recognize a specific object in digital content using artificial intelligence, artificial intelligence must be trained in advance using various learning data including the object to be recognized.

The learning data may be provided by displaying the outline, area, etc. of an object existing in the digital content so that artificial intelligence can distinguish between an object and a non-object in the digital content. However, since there is no data in which objects and non-objects are distinguished, the task of displaying the outline and area of an object in digital content must be performed by a person.

Since artificial intelligence improves recognition accuracy as it learns a lot of diverse training data, it is important to learn as much training data as possible. Recently, a crowd outsourcing method of generating learning data has been used in which a number of workers are recruited through the Internet to easily generate learning data and request a task of displaying an outline or region of an object in digital content.

At this time, in order to secure reliability, it is preferable to generate learning data by collecting a plurality of results of work completed for one judgment. Thus, work data has been collected and used by distributing the same work to a plurality of workers for the same digital content. However, when the work target is the entire digital content area, it is inefficient because the amount of work is excessively large. Therefore, there is a need for a method of reducing and distributing the amount of work.

In addition, each worker may differently determine the area including the object, and a problem may occur in that undesirable learning data is generated because the workers perform the work insincerely. In this case, there is an inconvenience in that the inspector has to individually inspect whether the learning data generation task has been completed properly, and there is a problem in that artificial intelligence trained based on undesirable learning data does not properly recognize objects in real situations. may occur.

Therefore, it is required to develop a management technology capable of managing crowd outsourcing method work so that artificial intelligence learning can be efficiently performed.

Republic of Korea Patent Registration No. 10-1887415 (2017.11.21.)

The present invention provides a crowd outsourcing job distribution method and apparatus using artificial intelligence that pre-processes a work target to generate and distribute work data so that work objects can be minimized.

As a technical means for achieving the above-mentioned technical problem, the crowd outsourcing job distribution method using artificial intelligence according to an embodiment of the present invention is a work target area of work target data where a processor controlled by an artificial intelligence module needs work. Job data generation step of generating job data by setting; Transmission step of the communication unit transmitting the work data to some worker terminals among a plurality of worker terminals connected to the Internet; And a receiving and learning step of learning the artificial intelligence module based on the completion data when the communication unit receives the completion data in which the work for the work data is completed from the plurality of worker terminals, and the work data The generating step, the transmitting step, and the receiving and learning steps include repeating processes when transmitting to a predetermined number of terminals among the plurality of worker terminals or until the work on the work target data is completed.

When the receiving and learning step is performed for the first time, the work data generating step may stop repeating until the completion data is received by a preset amount.

The work data generating step generates a worked region from the completion data for each of the plurality of worker terminals as a mask for each of the plurality of worker terminals, generates a trimap by merging the masks for each of the plurality of worker terminals, and includes the trimap. It is possible to designate a region equal to or higher than a preset merged concentration as the work target region.

In the transmitting step, the some worker terminals may be terminals according to a predetermined order of excellent worker terminals.

In the receiving and learning step, the work data is inspected before the artificial intelligence module is trained, and in the transmission step, the some operator terminals are terminals whose accuracy of the operator evaluated in the process of inspecting the work data is equal to or greater than a preset value. can

In the receiving and learning step, a mask for each of the plurality of worker terminals is generated from the completion data for each of the plurality of worker terminals, and a similarity between the entire mask for each of the plurality of worker terminals and the mask of the target worker terminal is calculated to determine the accuracy. can decide

In the receiving and learning step, the similarity between the plurality of masks for each worker terminal and the mask of the target worker terminal is determined by merging the plurality of masks for each worker terminal to generate a trimap, and determining the similarity with the mask of the target worker terminal. As a criterion, the accuracy can be determined.

In the receiving and learning step, learning may be performed based on the completion data whenever a preset number of completion data is received from the communication unit.

In the receiving and learning step, learning may be performed based on correct answer data generated after inspecting the completion data.

As a technical means for achieving the above technical problem, the crowd outsourcing job distribution device using artificial intelligence according to an embodiment of the present invention transmits work data to a plurality of worker terminals, and receives completion data from the plurality of worker terminals. A communication unit for receiving, generating the work data to be transmitted to the plurality of worker terminals, and controlling the operation of a processor configured to perform an operation of verifying the completion data received from the plurality of worker terminals, and controlling the operation of the processor to complete the completion The artificial intelligence module may include a memory in which an artificial intelligence module learned as data is stored, and the artificial intelligence module may generate the work data in which only a part of each data requiring work is limited to a work target area.

According to an embodiment of the present invention, completion data that is a learning target can be reflected in artificial intelligence learning in real time, and thus, the efficiency of a learning data generation task performed by a plurality of workers can be improved.

According to an embodiment of the present invention, work data may be generated and distributed by varying the difficulty of work, and thus the efficiency of work distribution to a plurality of workers may be improved.

According to an embodiment of the present invention, it is possible to generate and distribute work data with reduced work difficulty in real time, thereby improving the work convenience of a plurality of workers.

1 is a block diagram showing a crowd outsourcing work distribution device using artificial intelligence according to an embodiment of the present invention.
Figure 2 is a flow chart for explaining a job management method using a crowd outsourcing job distribution device using artificial intelligence according to an embodiment of the present invention.
FIG. 3 is a flowchart illustrating a method of processing data in the crowd outsourcing job distribution device using artificial intelligence of FIG. 2 .
4 is a flow chart illustrating a process of generating and distributing work data in the crowd outsourcing work distribution method using artificial intelligence of FIGS. 2 and 3 .
5 is a graph showing work load reduction according to an embodiment of the present invention.
6 to 11 are exemplary diagrams for explaining an example of determining accuracy, propensity, and task difficulty for each worker in the crowd outsourcing task distribution method using artificial intelligence of FIG. 2 .
12 is an exemplary diagram for explaining an example of determining a reflection rate for each worker in the crowd outsourcing job distribution method using artificial intelligence of FIG. 2 .

Hereinafter, embodiments of the present invention will be described in detail so that those skilled in the art can easily practice the present invention with reference to the accompanying drawings. However, the present invention may be embodied in many different forms and is not limited to the embodiments described herein. And in order to clearly explain the present invention in the drawings, parts irrelevant to the description are omitted, and similar reference numerals are attached to similar parts throughout the specification.

Various embodiments of the present document may be implemented as software (eg, a program) including instructions stored in a storage medium readable by a machine (eg, a computer). A device is a device capable of calling a stored command from a storage medium and operating according to the called command, and may include an electronic device (eg, a server) according to the disclosed embodiments. An instruction may include code generated or executed by a compiler or interpreter. The device-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-temporary' only means that the storage medium does not contain a signal and is tangible, but does not distinguish whether data is stored semi-permanently or temporarily in the storage medium.

According to one embodiment, the method according to various embodiments disclosed in this document may be included and provided in a computer program product. Computer program products may be traded between sellers and buyers as commodities. The computer program product may be distributed in the form of a device-readable storage medium (eg compact disc read only memory (CD-ROM)) or online through an application store (eg Play Store™). In the case of online distribution, at least part of the computer program product may be temporarily stored or temporarily created in a storage medium such as a memory of a manufacturer's server, an application store server, or an intermediary server.

Each component (eg, module or program) according to various embodiments may be composed of a single object or a plurality of entities, and some of the sub-components may be omitted, or other sub-components may be various. It may be further included in the embodiment. Alternatively or additionally, some components (eg, modules or programs) may be integrated into one entity and perform the same or similar functions performed by each corresponding component prior to integration. According to various embodiments, operations performed by modules, programs, or other components may be executed sequentially, in parallel, repetitively, or heuristically, or at least some operations may be executed in a different order, may be omitted, or other operations may be added. can

In this specification, "transmitting, receiving, collecting, extracting, uploading, or updating information or data" means directly sending, receiving, collecting, extracting, uploading, or updating information or data, or indirectly transmitting through another relay server, It includes the meaning of receiving, collecting, extracting, uploading or updating.

Hereinafter, the present invention will be described in detail with reference to the accompanying drawings.

The present invention relates to a crowd outsourcing job distribution method and apparatus using artificial intelligence that can be used to perform a crowd outsourcing job management function using artificial intelligence. The crowd outsourcing task distribution method is performed by the same device as the crowd outsourcing task management device. In other words, the crowd outsourcing job distribution device may further perform a crowd outsourcing job management function by performing a function of additionally processing completion data. Hereinafter, in this specification, the crowd outsourcing job management device is collectively referred to as a crowd outsourcing job distribution device, and the device is described as being capable of further performing a crowd outsourcing job management function.

First of all, for convenience of understanding, the overall operation of the crowd outsourcing job management function is first introduced, and as an initial job, how the crowd outsourcing job distribution method handles the job while performing the corresponding function will be described later with reference to FIGS. 3 to 5 do.

1 is a block diagram showing a crowd outsourcing work distribution device using artificial intelligence according to an embodiment of the present invention.

Referring to FIG. 1 , a crowd outsourcing task distribution device 400 according to an embodiment of the present invention refers to a device of a system that manages data preprocessing tasks that generate learning data for learning an artificial intelligence module. Specifically, the crowd outsourcing job distribution device 400 transmits job data to a plurality of worker terminals (100, 200, 300), and the completion data for the job from the plurality of worker terminals (100, 200, 300) If received, it may be a device that inspects part of the completed data to generate desirable learning data, transmits the remaining complete data that is difficult to inspect to the inspector terminal 500, and manages the learning data generation task so that a professional inspector can inspect it. there is.

The crowd outsourcing job distribution device 400 may be any type or combination of types of computing devices, such as network servers, web servers, file servers, supercomputers, desktop computers, and the like.

The crowd outsourcing job distribution device 400 includes a communication unit 410, a processor 420, and a memory 430.

The communication unit 410 transmits work data to a plurality of worker terminals 100, 200, and 300, and receives completion data from the plurality of worker terminals 100, 200, and 300 when the work for the work data is completed. is configured to At this time, the work data may be sequentially transmitted by groups by grouping some of the plurality of worker terminals 100, 200, and 300, and completion data may also be sequentially received according to the order of completion. Also, transmission and reception can be performed simultaneously.

The communication unit 410 is connected to the Internet and may be composed of a network adapter capable of transmitting and receiving data to and from the operator terminals 100, 200, 300 and the inspector terminal 500 through the Internet. The communication unit 410 may be configured to be connected to the Internet through a wired (LAN or WAN) or wireless (Bluetooth, WiFi, Infrared data association (IrDA) method.

The processor 420 generates work data to be transmitted to the plurality of worker terminals 100, 200, and 300, and performs an operation of inspecting the completed data received from the plurality of worker terminals 100, 200, and 300. can be configured. At this time, the work data to be transmitted may be pre-processed data in which a work target area requiring work is set in data (digital contents, etc.) requiring work by utilizing an artificial intelligence module. In this case, since the work area of the plurality of worker terminals 100, 200, and 300 is limited, the work can be performed more easily than working on the entire digital content. At this time, since it is the artificial intelligence module that performs the preprocessing, the higher the level of the artificial intelligence module, the more precisely processed work data can be generated. Generation of the work data only needs to be completed before the work data transmission of the communication unit 410, and the communication unit 410 can sequentially transmit the work data to a plurality of worker terminals 100, 200, and 300, so that completion You can create and send work data several times between receiving data and training the artificial intelligence module. In other words, first work data is generated and sent to a portion of the plurality of worker terminals 100, 200, and 300, and a predetermined amount of completion data is received from the plurality of worker terminals 100, 200, and 300. After that, it can be learned and the next work data can be generated and transmitted.

The processor 420 is a dedicated processor (eg, an embedded processor) for performing the above-described arithmetic operations, or a general-purpose processor capable of performing corresponding operations by executing one or more software programs stored in a memory device. (eg, CPU, graphical processing units (GPUs), single-core processors, multi-core processors, application specific integrated circuits (ASICs), etc.), or communication processors (CPs). .

The memory 430 is a device in which an artificial intelligence module that controls the operation of the processor 420 is stored, and is, for example, a solid state drive (SSD), a hard disk drive, or a card-type memory (SD or XD). memory, etc.), RAM (random access memory, RAM), SRAM (static random access memory), ROM (read-only memory, ROM), EEPROM (electrically erasable programmable read-only memory), PROM (programmable read-only memory) , a storage medium of at least one type among a magnetic memory, a magnetic disk, and an optical disk, and various storage media capable of storing identifiable physical location information therefor. 1, memory 430 is shown as being located within crowd outsourcing job distribution device 400, in some embodiments, memory 430 is a separate device physically separate from crowd outsourcing job distribution device 400. may consist of In this case, the memory 430 and the crowd outsourcing job distribution device 400 may be connected through a network.

The artificial intelligence module is a set of instructions for controlling the operation of the crowd outsourcing job distribution device 400, and may include computer- or machine-executable instructions described in any appropriate programming language to perform the various functions described above. there is.

The artificial intelligence module creates work data by setting the work target area of the work target data, and when the work data for which the work on the work data is completed is received from the plurality of worker terminals 100, 200, and 300, the plurality of workers Determine the accuracy, propensity, and difficulty of each task, determine the reflection rate for each worker based on the accuracy and difficulty of each worker, and check the completion data for each multiple worker based on the reflection rate. is configured to determine Operations of the artificial intelligence module will be described in more detail with reference to FIGS. 2 to 12 .

Meanwhile, the worker terminals 100 , 200 , and 300 are terminals used by workers, and may be implemented as a computing device for generating completion data by processing work data. In addition, the inspector terminal 500 is a terminal used by the inspector and may be implemented as a computing device for performing inspection on completed data requiring professional inspection among complete data. For example, the operator terminals 100, 200, and 300 and the inspector terminal 500 may include a smartphone, a tablet PC, a mobile phone, a video phone, an e-book reader, a desktop PC, a laptop PC, a netbook computer, a workstation, and a server. , a PDA, a PMP, an MP3 player, a mobile medical device, a camera, or a wearable device.

Figure 2 is a flow chart for explaining a job management method using a crowd outsourcing job distribution device using artificial intelligence according to an embodiment of the present invention, Figure 3 is a data from the crowd outsourcing job distribution device using artificial intelligence of FIG. It is a flow chart to explain the processing method.

Referring to FIGS. 2 and 3 , first, the crowd outsourcing job distribution device creates work data by setting a work target area of work target data (S210).

"Work target data" refers to original data of work data to be transmitted to the worker terminal 100, and the work target data may be composed of various types of data such as digital images, texts, and sounds. The crowd outsourcing job distribution device 400 generates job data by pre-processing work target data using an artificial intelligence module.

The crowd outsourcing job distribution method using artificial intelligence includes a process of generating job data by setting a work target area of work target data requiring work by a processor controlled by an artificial intelligence module (S211), and generating work data by a communication unit (S211). A transmission step (S212) of transmitting to some worker terminals among a plurality of worker terminals connected to the Internet, and when the completion data for which the work for the work data is completed is received from the plurality of worker terminals by the communication unit, the completion data and a receiving and learning step (S213) of learning the artificial intelligence module based on, and the work data generation step, transmission step, and receiving and learning step are performed when transmitting to a predetermined number of terminals among a plurality of worker terminals or when This is repeated until the operation on the target data is completed.

Here, the quality of the subsequent work data generated by the artificial intelligence module is determined according to the quality of the initial completion data received after the work data is first generated and transmitted.

Therefore, if work data is generated after training the artificial intelligence module on too little completion data, poor work data will be generated if the quality of the initial completion data is very poor. In order to avoid this, it is preferable to generate follow-up data after a predetermined amount of completion data is received when the receiving and learning steps are first generated.

In addition, the same work contents performed by a plurality of workers are highly likely to be worked the same in the next work. Therefore, as a method of generating work data, the worked area in the received completed data is generated as a mask for each worker terminal, a trimap is generated by merging the masks for each worker terminal, and a trimap is generated in an area included in the trimap. When a predetermined merged concentration or higher is designated as the work target area, the same work content performed by a plurality of workers can be specified, and work data indicating areas with high probability to be worked and areas without it can be generated.

Since the higher the level of the initial training data, the higher the performance of the artificial intelligence module, so the better the worker who generates the initial training data, the better. In addition, since the job data is received first, the possibility of submitting the completion data first is high, so it is preferable to sequentially transmit the job data according to the excellence level of the excellent worker in the transmission step.

Here, the method of selecting the excellent worker will vary, but when the present invention receives the completion data, if the inspection of the completion data is performed separately from the distribution of the work data, the accuracy will be calculated in the inspection process, so the excellence based on accuracy figure can be judged.

The accuracy calculated in the inspection process can be based on the similarity with the work of all other workers, and if the work area is created as a mask and a Trimap is produced, the accuracy can be determined based on the similarity with the Trimap. The mask and trimap for determining the above accuracy will be described later together with the verification step.

Learning for the artificial intelligence module may be performed on the corresponding completion data whenever a preset number of completion data is received.

However, in order to prevent completion data of poor quality from being initially learned, completion data may be inspected in advance and then learned based on generated correct answer data. A more detailed description will be given later with reference to FIG. 4 .

Although the method and apparatus according to the present invention refer to the operation S210, the subsequent inspection step (S30) and the inspection reference data forming step (S55) are performed to describe the work data and amount of work generated by the method and device according to the present invention in more detail. explain

In the flow chart of FIG. 3, it is shown as if the work (S50) below the inspection is performed after the generation and distribution (S210) of work data, but the generation and distribution (S210) of work data progresses as the reception of completion data progresses. Since it proceeds sequentially, the following inspection work may be performed even before generating and distributing the remaining work data for the previously received completed data.

When the completion data is received, the completion data may be inspected (S30). The step of inspecting the completed data (S30) is the step of reconstructing the completed data to reconstruct the work data into data that can be superimposed according to the type of work data (S34) and analyzing the completed data to determine the accuracy of the operator and the work inclination of the operator , and a step of determining the propensity of each worker to evaluate the difficulty of work (S220). Specific examples corresponding to this will be described later with reference to FIGS. 6 to 11 .

A step ( S55 ) of forming work inspection standard data by evaluating the ability of each worker based on the propensity evaluation data and completion data for each worker may be additionally performed. The step of forming workpiece inspection standard data (S55) is a step of generating inspection criteria to be used for inspection of a workpiece after the workpiece inspection step. Since the accuracy of the work was calculated in S30, it is possible to directly generate the inspection standard data by reflecting it as a weight (S30). It is preferable to additionally perform a filtering step (S230) for excluding work and a step (S240) for determining the reflection rate for each worker to evaluate and utilize the ability of each worker in advance to generate inspection standard data according to the weight (S250). ( S55) A specific example of this step will be described later with reference to FIG. 12 .

Figure 4 is to explain in detail the crowd outsourcing job distribution method (S210) using artificial intelligence according to an embodiment of the present invention, until the work data with reduced workload is regenerated by repeating S210 in Figure 3 twice. It is a flow chart showing the data processing of

The artificial intelligence module of the crowd outsourcing job distribution device sets a work area for the first work target data and generates first work data (S214).

As mentioned above, work target data is digital content that includes objects to be learned by an artificial intelligence module, and may be composed of various types of data such as images, texts, and sounds.

For convenience of explanation, a case in which work target data is composed of image data will be described as an example.

The artificial intelligence module first randomly selects first work target data from among all work target data and sets the work target area where the object to be learned by the artificial intelligence module is located to be distinguished from other areas in the first work target data. 1 Create work target data. For example, if the object to be learned is "dog", the artificial intelligence module may generate first task data by displaying the boundary of the area where "dog" is located in the first task data including "dog". can

Thereafter, the generated first work data is transmitted to some worker terminals among a plurality of worker terminals connected to the Internet. At this time, since some worker terminals are assigned work data first, the initial learning data of the artificial intelligence module is generated. The initial performance of the AI module is important because the AI module that learned this will be used to produce all work data in the future. Therefore, the quality of the initial learning data that determines the initial performance of the AI module is also important. In order to increase the quality of the initial learning material, it is advantageous that some worker terminals that first receive the first work data are workers with excellent capabilities. This excellent worker may be determined based on any criteria such as the worker's career and field of study. However, since the crowd outsourcing job distribution device using artificial intelligence evaluates the accuracy of each terminal for each task, the accuracy of the worker evaluated by the crowd outsourcing job distribution device using artificial intelligence in the process of reviewing this job or existing job data is the standard. As a result, a worker whose accuracy exceeds a certain level can be designated as an excellent worker.

Then, the crowd outsourcing work distribution device receives the first completion data for the first work data from the worker terminal 100 (S215).

The first completion data refers to data in which the operator has completed a task for the first task data through the operator terminal 100 . For example, when the object to be learned by the artificial intelligence module is "dog", the task for the first task data may be a task of displaying the outline of "dog". The operator may generate first completion data by precisely editing the boundary line in the first work data in which the boundary line of the area where the "dog" is located is schematically displayed to fit the contour of the "dog".

When the first completion data is received from the worker terminal, the crowd outsourcing job distribution device learns the artificial intelligence module based on the first completion data (S216).

In the first work data, the boundary line for "dog" is roughly displayed, but in the first work data performed by a human, the boundary line for "dog" may be more precisely displayed.

When a plurality of first completion data is received, the accuracy of a specific worker terminal among them is calculated in the process of learning the first completion data to the artificial intelligence module. The process of inspecting and evaluating the completed data and learning the artificial intelligence module will be described later.

The artificial intelligence module is learned based on the first completion data, and when using the artificial intelligence module for which learning has been completed, a work area for "dog" can be set more accurately and precisely for other images including "dog". can

Thereafter, the crowd outsourcing job distribution device sets a work area for the second work target data based on the learned artificial intelligence module to generate second work data (S217).

The second work target data may be work target data selected from among work target data other than the first work target data. For example, if the work target data is composed of 100 image data including “dog”, the first work target data may be composed of 10 images including “dog”. The artificial intelligence module generates first work data by displaying an area including “dog” in the 10 pieces of first work object data, and first completion data in which the work for the 10 first work object data is completed from the worker terminal. When is received, it may be learned based on the first completion data. The artificial intelligence module that has completed learning may more precisely display the area including “dog” for the second task data consisting of 10 other images, and the second task data is generated through the artificial intelligence module. Thereafter, when the second work data is transmitted to a plurality of worker terminals and the second completion data in which the work for the second work data is completed is received from the worker terminal, the artificial intelligence module is learned based on the second completion data. In the same way, the third work data is generated by setting an area including “dog” for the third work object data composed of another 10 images through the artificial intelligence module. In this case, the learned artificial intelligence module may display a region including “dog” with respect to the third work object data more accurately than when processing the second work object data. In the same way, the AI module is periodically learned until the work on the work target data is completed, and the precision of the work data generated through the AI module can be improved as the learning continues.

In other words, the steps of the artificial intelligence module generating/transmitting work data from the work target data and the step of receiving completion data and learning the artificial intelligence module may be repeatedly performed. This step may be repeated until all tasks for the work target data are distributed, and in the case of duplicate work for the same work target data, it will be repeated until work data is generated and sent to a preset number of worker terminals. may be By combining the creation of work data with the learning of the AI module, the more work data created later, the more learned AI modules are used to create work data. As a result, it has the effect of reducing the amount of work.

In some embodiments, learning of the artificial intelligence module may be performed based on first correct answer data that has been verified, not based on first completion data. That is, the artificial intelligence module may be learned based on the first correct answer data that has been confirmed by completing the inspection of the first completion data prior to processing the second work target data, and through the learned artificial intelligence module, the second The work target data may be processed to generate second work data. In this case, the first correct answer data may correspond to data that is finally inferred as a correct answer after verification is completed according to a verification method for complete data described later.

Briefly examine the process of calculating accuracy in the inspection and learning process. If a plurality of work results can be estimated as the correct answer, masks for each worker terminal are generated from the completion data for each worker terminal, and the degree of similarity between the entire mask for each worker terminal and the mask of the target worker terminal is calculated to achieve accuracy. can decide

Here, the degree of similarity between all masks for each of the plurality of worker terminals and the mask of the target worker terminal is equal to the average value of similarity between each mask of each of the plurality of worker terminals and the mask of the target worker terminal. In other words, since the similarity between the entire mask of each worker terminal and the mask of the target worker terminal corresponds to the linear sum of the similarity between each of the plurality of masks of each worker terminal and the mask of the target worker terminal, the mask for each of the plurality of worker terminals is prepared in advance. Trimap generated by merging can be replaced by calculating the similarity with the mask of the target worker device.

If the task of measuring the accuracy can be performed with the target data for which the correct answer has already been determined before the task, a more reliable accuracy can be determined.

5 is a graph showing the amount of work reduced according to an embodiment of the present invention.

If the work data is created and distributed once and then the work is performed, the work load will all be 100% regardless of the progress of the project. As described above, if work data is generated after training the artificial intelligence module with too little completion data, the artificial intelligence module may be distorted by low-level completion data. Therefore, it is desirable to collect results as much as the basic minimum amount of inspection work (Q) and generate follow-up work data after the artificial intelligence module learns. However, since the present invention learns completion data whenever a task progresses, the workload of the generated task data decreases as the project progresses.

Referring back to FIGS. 2 and 3 , the crowd outsourcing job distribution device may inspect the completed data when the job completion data for the job data is received from the worker terminal (S30).

First, complete data is reconstructed into data that can be superimposed (S34). For example, sound or text is one-dimensional data, and images are two-dimensional data. It is preferable that each data can collect the results of work by a plurality of workers. Accordingly, one-dimensional data can be reconstructed into a vector and two-dimensional data can be reconstructed into a matrix so that an amount of the same size can be overlapped.

Then, the accuracy, inclination, and difficulty of the work for each of the plurality of workers are determined (S220).

Crowd outsourcing job distribution device 400 according to an embodiment of the present invention uses an artificial intelligence module to determine the job accuracy, propensity, and difficulty of the job for a plurality of workers, and based on this, to determine other jobs. It is characterized by inspecting the completion data of workers and managing the work performance of workers. Hereinafter, with reference to FIGS. 8 to 9 , a method of determining task accuracy, propensity, and difficulty for each of a plurality of workers will be described in detail.

First, the crowd outsourcing job distribution device 400 generates a plurality of masks for each worker when completion data is received from the worker terminal 100 .

Referring to FIG. 6 , the plurality of masks OM1 , OM2 , and OM3 for each worker may be a mask displaying a region for a specific object in a work target. For example, when a work target corresponds to a digital image and is a task of defining a specific object in the digital image, a plurality of workers define the specific object by displaying the outline of the specific object in the digital image according to their own standards. can do. In this case, the completion data received from the worker terminal 100 may be data for a digital image in which the contour of a specific object is displayed, and the plurality of worker-specific masks OM1, OM2, and OM3 are areas defined by the contour of the specific object may be a mask for

Then, the artificial intelligence module merges a plurality of worker-specific masks (OM1, OM2, OM3) to create a Trimap.

Since a plurality of workers display the outline of a specific object in a digital image according to their respective standards, the masks OM1, OM2, and OM3 of each worker have different shapes. The artificial intelligence module creates a Trimap (TM) by overlapping a plurality of worker-specific masks (OM1, OM2, OM3). Specifically, one merged mask is generated by summing pixel values of corresponding pixels of the plurality of masks OM1 , OM2 , and OM3 for each operator.

Here, the pixels of the masks OM1 , OM2 , and OM3 for each operator mean elements of the masks OM1 , OM2 , and OM3 . That is, the mask for each operator may be composed of n*n elements, and it may be understood that a pixel refers to each element. Also, a pixel value means a data value of each element. For example, when a plurality of workers set a specific area to define a specific object, the data value of elements corresponding to the corresponding area may be defined as 1, and the data of elements corresponding to the remaining areas except for the corresponding area The value can be defined as 0. In this case, the pixel value of the corresponding region is determined to be 1, and the pixel values of the remaining regions are determined to be 0.

A pixel value at a pixel where the display area of the first operator's mask OM1, the display area of the second operator's mask OM2, and the display area of the third operator's mask OM3 all overlap is a pixel value of the first operator's mask. It can be expressed as the normalized sum of the pixel value of (OM1), the pixel value of the mask (OM2) of the second operator, and the pixel value of the mask (OM3) of the third operator. For example, if the pixel value of the mask display area is defined as 1 and the pixel value of the non-display area is defined as 0, the mask OM1 of the first operator, the mask OM2 of the second operator, and the mask OM2 of the third operator In pixels where the display areas of the mask OM3 all overlap, the sum of pixel values is defined as 3, which can be normalized by dividing it by Opacity defined as 1/n (where n = the number of masks). Therefore, the sum of the pixel values can be normalized to 3*1/3 = 1. Each pixel value of the mask integrated in the same way can be calculated as a normalized sum of corresponding pixel values of the first operator's mask OM1, the second operator's mask OM2, and the third operator's mask OM3. there is.

In FIG. 6, an example in which Trimap is formed in the form of a two-dimensional matrix is shown, but Trimap may be formed in the form of a one-dimensional vector according to the form of complete data. For example, if the completion data corresponds to voice data, a mask for each operator may be generated for a region where a voice of a specific object is displayed, and a trimap in the form of a one-dimensional vector may be generated by merging the masks for each operator.

Meanwhile, the aforementioned Trimap is generated based on completion data for each of a plurality of workers generated for the same job data. That is, in the case of a task of defining a specific object within the same digital image, a trimap may be generated by receiving completion data for each operator for the same digital image and overlapping a plurality of masks for each operator.

Thereafter, the artificial intelligence module determines the work tendency, work accuracy, and difficulty of the work of the target worker based on the mask and Trimap of the target worker among the plurality of workers.

Specifically, the work inclination of the target worker is determined based on the following [Equation 1].

Here, an Opacity Mask Score (OMS) is an indicator for identifying work inclinations for each worker, and may be an index indicating the degree of agreement between a specific work result of a worker and other work results of other workers. In [Equation 1], TM(t) is a pixel value of Trimap for a specific task (ie, completed data) (t), and BM(t, i) is a specific task (t) of the target worker (i) means the pixel value of the mask for That is, OMS means the average of pixel values of the trimap of the selected area of the mask for the specific job (t) of the target worker (i).

In the case of a worker with a high OMS, since most of the pixels with high pixel values are selected in Trimap, the region selected by the worker overlaps with the region selected by other workers. Accordingly, it can be seen that a worker with a high OMS tends to under-display the outline of a specific object compared to other workers.

On the other hand, in the case of a worker with a low OMS, since many pixels having low pixel values are selected in Trimap, the region selected by the corresponding worker may correspond to regions not selected by other workers. Accordingly, it can be seen that a worker with a low OMS tends to display the outline of a specific object excessively compared to other workers.

7(a) shows a mask 711 of a worker with low OMS and a mask 713 of Trimap, and (b) shows a mask 715 of a worker with high OMS and a mask 713 of Trimap. For convenience of explanation, only the boundary of each mask is indicated in FIG. 9 .

As shown in (a) of FIG. 7, it can be seen that the mask 711 of the worker with low OMS is located inside the Trimap mask 713, and through this, the worker with low OMS defines the specific object to another It can be seen that they are under-represented compared to the workers.

On the other hand, as shown in (b) of FIG. 7, it can be seen that the mask 715 of the operator with high OMS is located outside the Trimap mask 713, and through this, the operator with high OMS defines the definition of a specific object. It can be seen that they display excessively compared to other workers.

In addition, the artificial intelligence module determines the work accuracy of each of the plurality of workers based on the following [Equation 2].

Here, TWS (Task-Worker Similarity) is an index representing the task accuracy of a worker, and means a cosine similarity between the task results of a specific worker and the task results of the rest of the workers. In [Equation 2], V _t means a 1-dimensional vector of Trimap of a specific task (t), and V _t,i is a 1-dimensional vector of a worker mask of a specific task (t) of a specific worker (i) it means.

8 is a diagram for explaining an example of a TWS calculation process. Referring to FIG. 10, Trimap (TM) in the form of a two-dimensional matrix for a specific task (t) can be 1-dimensional vectorized (V _t ) by applying a Flatten function, and a specific operator (i ) can also be one-dimensional vectorized (V _t,i ) by applying the Flatten function. Then, by calculating V _t -V _t,i , a one-dimensional vector for the work results of the workers other than the specific worker (i) is calculated, and the cosine between V _t -V _t,i and V _t,i By calculating the similarity, the cosine similarity between the specific worker (i) and the work results of the other workers except the specific worker (i) is calculated.

9 is a diagram showing TWS calculation results according to the work result of a specific operator. Referring to FIG. 9 , as a result of calculating the TWS based on [Equation 2] after converting the mask of a specific worker into a 1D vector and converting the Trimap into a 1D vector, the TWS of the specific worker was calculated as 0.936. Through this, it can be seen that the work result 911 of a specific worker has a similarity of about 0.936 compared to the result 913 of Trimap. When the work result 911 of a specific worker has a lower similarity to the result 913 of Trimap, the TWS value may be lowered.

In some embodiments, the artificial intelligence module may further calculate a trimap score (TMS) based on the following [Equation 3] in order to improve the intuitiveness of the operator's TWS.

Here, TMS(t,i) is the Trimap score of a specific worker (i) for a specific task (t), minimizing the variance of the part that matches the work results of other workers among the specific worker (i)'s work results Therefore, it can be described as an index that maximizes the accuracy index for the work result of a specific worker (i). In [Equation 3], TM(t) means a pixel value of Trimap for a specific job t, and BM(t,i) is a job mask of a specific worker i for a specific job t represents the pixel value of On the other hand, n means the number of pixels whose pixel value exceeds 0 in Trimap.

10 is a diagram showing a calculation process of TMS(t,i) of a specific worker i for a specific job t. Referring to FIG. 10, when the task result for a specific task (t) is expressed as a mask consisting of 3*3 = 9 pixels, the sum of │TM(t)-BM(t-i)│ is shown in FIG. As such, it is calculated as 2, and when TMS(t,i) is calculated according to Equation 3 above, TMS(t,i) is calculated as 0.77.

On the other hand, the artificial intelligence module determines the task difficulty for a specific task from the deviation of a plurality of workers' masks for the specific task.

Specifically, the task difficulty for a specific task may be determined based on Equation 4 below.

Here, ICS(t) is an index representing task difficulty for a specific task t, and can be described as an indicator of how clear and unambiguous the task is. In [Equation 4], TM(t) means a pixel value of Trimap for a specific task t, and n means the number of effective pixels whose pixel value exceeds 0. The clearer the task, the more similar the work results of the workers are, and since the overlapping area of the task masks is wide, ICS(t) in [Equation 4] can be calculated larger.

11 are diagrams showing ICS(t) calculation results for actual works. As shown in FIG. 11, it can be seen that the ICS is measured high for an image in which the shape of an object is simple and can be relatively clearly defined.

Referring back to FIG. 2 , the crowd outsourcing job distribution device determines whether a job whose reflection rate can be calculated for the target jobs of the completed data based on the accuracy and job difficulty of each of a plurality of workers (S230).

Here, the voting power can be described as the weight or contribution rate of workers who contributed to the trimap formation of a specific task. Crowd outsourcing job distribution device according to an embodiment of the present invention provides a high reflection rate to a worker who has accurately performed a specific job in the process of determining the verification standard data (ie, correct answer data) necessary for the inspection of a specific job. It is configured to improve the reliability of the reference data.

On the other hand, when a specific task itself is ambiguous and the level of difficulty is excessively high, there may not be a worker who has sufficiently accurately performed the task. In addition, when a plurality of workers who performed a specific task did not faithfully perform the task, only the completion data of defective workers for that task exists, so when a reflection rate is given to this and the inspection standard data is selected, reliability is improved. Low acceptance criteria data may be selected.

Thus, the crowd outsourcing job distribution device according to an embodiment of the present invention may first filter whether a corresponding job is a job suitable for calculating a reflection ratio before calculating a reflection ratio for each worker through an artificial intelligence module.

For example, the artificial intelligence module examines whether the maximum value of TWS of completion data received from a plurality of workers exceeds the accuracy threshold and the ICS exceeds the difficulty threshold. The TWS may be calculated based on [Equation 2], and it may be determined whether the highest maximum value among the TWS values for the task to be determined exceeds the accuracy threshold. As mentioned above, since TWS is an indicator of the worker's work accuracy, a high maximum value of TWS may mean that excellent workers who performed the work with high accuracy are included among the workers who performed this work. .

In addition, ICS of complete data can be calculated based on [Equation 4] above. As mentioned above, since the higher the ICS, the lower the difficulty of the task, so that the ICS for a specific task exceeds the difficulty threshold may mean that the specific task has an appropriate difficulty level.

If the maximum TWS of the completion data for a specific task is less than the accuracy threshold and the ICS for the specific task is less than the difficulty threshold, the difficulty of the task itself is too high and there is no completion data of excellent workers with high accuracy. Therefore, it can be judged as an ambiguous task in which calculation of the reflection rate is impossible. In this case, the crowd outsourcing job inspection apparatus 400 transmits completion data for the ambiguous job to the inspector terminal 500, and expert inspection for the ambiguous job is performed through the inspector terminal 500 (S255).

On the other hand, if the maximum TWS of the completion data for a specific task exceeds the accuracy threshold and the ICS for the specific task exceeds the difficulty threshold, the task includes the completion data of excellent workers, and the difficulty of the task itself is Since it is at an appropriate level, it can be seen that it has the qualifications to select excellent workers.

Accordingly, the crowd outsourcing job distribution device determines that the corresponding job is eligible for reflection rate calculation, and determines the reflection rate for each worker based on the accuracy and task difficulty of each worker (S240).

Specifically, the crowd outsourcing job distribution device 400 according to an embodiment of the present invention, prior to calculating the reflection rate for each worker through the artificial intelligence module, complete data of excellent workers whose accuracy is equal to or greater than a preset threshold value among a plurality of workers extract

Specifically, the artificial intelligence module calculates TWS for each worker with respect to tasks that satisfy the above conditions, and determines whether the TWS exceeds a first threshold value. The higher the TWS, the higher the work result of the operator and the work result of other workers, so it is highly likely to be close to the correct answer data. Accordingly, the artificial intelligence module filters only the completion data of workers whose work accuracy is greater than or equal to a preset first threshold. Completion data of workers below the first threshold corresponds to work results with low accuracy. On the other hand, as can be seen from the TWS definition of [Equation 2], the TWS for each plurality of workers is defined as the cosine similarity of Trimap generated by overlapping the completion data of a specific worker and Trimap, that is, the completion data of a plurality of workers, If there are bad workers whose accuracy is too low, the accuracy of Trimap itself may be lowered, and it can be seen that it has the effect of lowering the TWS for well-intentioned workers for the work.

Therefore, the artificial intelligence module of the crowd outsourcing job distribution device according to an embodiment of the present invention removes completed data having a TWS of less than a first threshold value among the completed data of a plurality of workers, and completes data having a TWS equal to or greater than the first threshold value. Trimap is generated again by selecting only the selected ones, and based on the regenerated Trimap, TWS for a plurality of workers for a specific job is calculated again.

Thereafter, the artificial intelligence module examines whether the TWS calculated again based on the completion data of the selected plurality of workers exceeds the second threshold, and determines the completion data of specific workers having TWS exceeding the second threshold. It is selected as the completion data of excellent workers.

The accuracy threshold, difficulty threshold, first threshold, and second threshold may be determined in various ways according to required accuracy of learning data. For example, in the case of artificial intelligence applied to self-driving cars, since it is necessary to accurately recognize objects, learning based on accurate learning data may be required. In this case, by setting the above-mentioned threshold value high, it is possible to improve the accuracy of the learning data.

Thereafter, the artificial intelligence module calculates the similarity between the correct answer data for the quality check task and the completion data of the excellent worker for the quality check task to determine the intersection over union (IOU) of the excellent worker.

Specifically, the artificial intelligence module may select completion data for a quality check task among tasks performed by excellent workers. Here, the quality check task refers to a task in which correct answer data (Ground Truth) is secured in advance to determine the reflection rate for each excellent worker, and the IOU for each excellent worker is calculated for the quality check task for which correct answer data exists The star reflection rate can be determined.

IOU means a value obtained by dividing the area of the intersection area of the two areas A and B by the value of the sum area, and means a Jaccard index defined by [Equation 5] below.

In the case of excellent workers, since the work accuracy is high, the IOU between the completion data and the correct answer data for the quality check task may appear high.

Then, the artificial intelligence module calculates the quality score of the excellent worker from the ratio of the IOU of the excellent worker to the task difficulty for the quality check task.

Specifically, the quality score may be calculated through [Equation 6] below.

Here, QScore means the quality score of a particular good worker for a quality check task (QcT). Meanwhile, in [Equation 6], QcT _w,i denotes completion data for the latest i-th quality check task QcT of a particular excellent worker w. In addition, QcT _gt,i means correct answer data for the latest i-th quality check task (QcT).

As can be seen from [Equation 6] above, the artificial intelligence module can give a higher quality score to an excellent worker w having excellent accuracy for difficult tasks.

Then, the artificial intelligence module determines the reflection rate of excellent workers based on the quality score.

Specifically, the artificial intelligence module determines the reflection rate of a specific excellent worker by applying a weight according to the period to the quality score of the specific excellent worker. For example, as shown in [Equation 7], by applying a weight (DecayWeight) for a period to the quality score (QScore) of a specific excellent worker and dividing the number of quality check tasks (QcT) to be applied to the reflection rate calculation , the influence score (IP) of a specific excellent worker can be calculated.

Here, IP refers to an index indicating the degree of influence of a specific excellent worker that is reflected in determining inspection criterion data (ie, correct answer data) for a specific task to be inspected. QcT Range means the number of quality check tasks (QcT) to be applied to IP calculation, and the higher the QcT Range, the higher the reliability of IP. DecayWeight is a weight for a period, and may be a factor having a larger value as it is closer to the current point in time, and is defined by Equation 8 below.

Here, Weight means weight, and t means QcT Range. i means an integer from 1 to t-1. Meanwhile, a specific value of the weight may be variously determined according to task difficulty, type, and the like.

For example, the process of calculating the influence score (IP) of a particular excellent worker (i) may be described as follows. If the impact score (IP) is calculated based on the five quality check tasks (QcT), first, the artificial intelligence module determines the five quality check tasks (QcT) based on the completion data of a particular excellent worker (i). After calculating the IOU and the task difficulty (ICS) of the quality check task (QcT), it is possible to calculate the quality score (QScore) for three excellent workers based on [Equation 6]. Specifically, the IOU for the five quality check tasks (QcT) may be calculated based on [Equation 5] above, and the ICS may be calculated based on [Equation 4] above. An example of calculating IOU and ICS for 5 quality check tasks (QcT) of a certain excellent worker (i) is shown in [Table 1].

division IOU ICS QcT1 0.9 0.8 QcT2 0.8 0.9 QcT3 0.8 0.7 QcT4 0.9 0.9 QcT5 0.7 0.8

In this case, according to [Equation 6] above, the quality score (Qscore) of the specific excellent worker (i) is calculated as follows.

QScore = [1.12, 0.88, 1.14, 1.00, 0.87]

In addition, when the weight is 0.9, the weight (DecayWeight) according to the period of the five quality check tasks (QcT) can be calculated as follows according to [Equation 8].

DecayWeight(0.9, 5)= 0.9 ⁱ = [1.0, 0.9, 0.81, 0.73, 0.65]

Therefore, the influence score (IP) for a particular excellent worker (i) can be calculated as follows.

Then, the artificial intelligence module normalizes the influence points (IP) so that the sum of the influence points (IP) of all excellent workers is 1 to calculate the voting power (VP).

For example, if there are 3 excellent workers and the influence scores (IP) of the 3 excellent workers are as shown in [Table 2], the sum of the influence scores (IP) of the 3 excellent workers is 1 Normalized reflection factor (VP) can be calculated as shown in Table 3 below.

Worker w1 w2 w3 Impact Point One 0.9 0.8

Worker w1 w2 w3 Voting Power 1/2.7 0.9/2.7 0.8/2.7

Referring back to FIG. 2 , the artificial intelligence module determines inspection criterion data for inspection of completed data for each of a plurality of workers based on the calculated reflection rate (S240).

For example, the inspection criterion data (ie, inspection criterion trimap) may be determined by applying the reflection factor (VP) to the opacity applied when forming the trimap.

12 is a diagram for explaining an example in which reflection rates for three excellent workers are calculated and inspection standard data is determined based on the reflection rates.

Referring to FIG. 12 , reflection rates for three excellent workers w1 , w2 , and w3 are calculated as shown in FIG. 12 , and inspection standard data is determined based on the reflection rates.

Determination of the inspection criterion data may be determined based on the reflection rate of a particular excellent worker selected according to the type of work result.

Specifically, if the variable form of the completion data is a categorical variable, the method of taking the average value of the completion data cannot be applied, so the completion with the highest reflection rate of the best worker with the highest reflection rate among excellent workers Determine the data as the inspection standard data.

On the other hand, if the variable form of the complete data is a continuous variable, it can be seen that a statistical approach to the complete data is possible. In this case, the artificial intelligence module determines the inspection standard data by applying an inference model suitable for the completion data of excellent workers.

For example, as shown in FIG. 12, Opacity masks (OM1, OM2, OM3) are generated for each worker by applying the reflection rate of excellent workers as Opacity to the completion data of each worker, and by overlapping them, Trimap (TM ), and by selecting only pixels having a pixel value exceeding a specific threshold value in the formed Trimap (TM), the inspection reference data can be determined.

However, the method of determining the inspection standard data is not limited to this, and the artificial intelligence module selects various statistical values such as the average value and median value of the completed data to which the reflection rate of excellent workers is applied, or the alpha-matting algorithm. It can be applied to determine the inspection standard data.

In some embodiments, the artificial intelligence module may apply a selective inference model without applying a statistical inference model even if the complete data is in the form of a continuous variable. That is, even if the completion data has a continuous data value between 0 and 1, if it is necessary to determine the correct answer data as 0 or 1 due to the nature of the task, a selective inference model is applied to select the completion data of the best worker with the highest reflection rate. It can be set as answer data.

Then, the crowd outsourcing job distribution device 400 according to an embodiment of the present invention adjusts the existing reflection rate (VP) for each worker based on the calculated reflection rate (VP) for each worker. That is, the reflection rate (VP) can be adjusted to increase for workers who are determined to have many correct data among complete data through inspection, and the reflection rate (VP) can be adjusted to decrease for workers who are determined to have few correct data among complete data. there is.

Crowd outsourcing job distribution device 400 according to an embodiment of the present invention inspects the completed data of the workers based on the determined job data, and selects the completed data processed as the correct answer as the learning data. Since artificial intelligence can be learned based on verified learning data, the learning efficiency of artificial intelligence can be increased and the performance of artificial intelligence can be advanced.

As described above, the crowd outsourcing job distribution device 400 using artificial intelligence according to an embodiment of the present invention can distribute the job of generating learning data for artificial intelligence learning to unspecified workers through the Internet. . In particular, by pre-processing the work target data through an artificial intelligence module and transmitting the pre-processed work data to unspecified workers, the work of the workers can be performed more easily. In other words, since the artificial intelligence module can generate work data by performing preprocessing to set the work target area in work target data and transmit the generated work data to a plurality of workers, workers can more easily create learning data. can be performed.

In addition, since the artificial intelligence module is configured to learn first completion data in which work on some first job data is completed in the process of generating job data, it can be gradually learned in the process of generating job data, and the first job After distributing the data, in the process of generating the second and third work data, a work area may be set in a more advanced way, and thus, highly pre-processed work data may be generated and distributed to workers. Therefore, the workload of a plurality of workers can be further reduced, and ultimately, the task of generating learning data itself can be configured to be performed by an artificial intelligence module.

In addition, the crowd outsourcing task distribution device 400 using artificial intelligence according to an embodiment of the present invention calculates the task difficulty and the accuracy of the workers who performed the task, the task difficulty is sufficiently low, and the accuracy of the plurality of workers is calculated. If is high enough, it is configured to perform inspection through an artificial intelligence module for that task. In addition, when the difficulty of the task is too high or the accuracy of the plurality of workers is too low, and the inspection through the artificial intelligence module is unclear, it is configured to perform an expert inspection on the task. In other words, experts only need to inspect some ambiguous tasks, and since most of the inspections are performed through an artificial intelligence module, the workload of experts can be sufficiently reduced and the efficiency of crowd outsourcing work can be improved.

In this case, the crowd outsourcing job distribution device 400 using artificial intelligence according to an embodiment of the present invention determines the accuracy, propensity, and difficulty of a task for each of a plurality of workers, and determines the reflection rate based on the accuracy and difficulty for each of a plurality of workers. And configured to determine the correct answer data for the completion data inspection for each of a plurality of workers based on the reflection rate. That is, a high reflection rate is given to workers with excellent accuracy, and completion data of excellent workers is reflected at a high rate in determining correct answer data. Therefore, it is possible to minimize the problem of lowering the reliability of correct answer data due to the work results of a plurality of workers with low accuracy, and the reliability of inspection by the artificial intelligence module can be further improved.

In particular, the crowd outsourcing job distribution device 400 using artificial intelligence according to an embodiment of the present invention applies a weight according to a period in determining a reflection rate for each of a plurality of workers. Even an excellent worker may have poor recent work accuracy due to various factors. Since the crowd outsourcing job distribution device 400 according to an embodiment of the present invention is configured to determine the reflection rate by applying a weight according to a period to the quality score of each worker, the job result of an excellent worker whose job accuracy has recently decreased is the correct answer. It is possible to prevent data from being influenced, and through this, reliability of correct answer data can be improved.

The above description of the present invention is for illustrative purposes, and those skilled in the art can understand that it can be easily modified into other specific forms without changing the technical spirit or essential features of the present invention. will be. Therefore, the embodiments described above should be understood as illustrative in all respects and not limiting. For example, each component described as a single type may be implemented in a distributed manner, and similarly, components described as distributed may be implemented in a combined form.

The scope of the present invention is indicated by the following claims rather than the detailed description above, and all changes or modifications derived from the meaning and scope of the claims and equivalent concepts should be construed as being included in the scope of the present invention. do.

100, 200, 300: worker terminal
400: crowd outsourcing task management device
500: inspector terminal
410: communication department
420: processor
430: memory

Claims (10)

An operation of generating first task data by setting a first boundary line of an area where an object is located in first task data requiring a task of displaying an object in a digital image by a processor controlled by an artificial intelligence module as a task area. data generation step;
A transmission step of transmitting the first work data by a communication unit to some worker terminals among a plurality of worker terminals connected to the Internet;
When the communication unit receives completion data in which the task of changing the first boundary line to the second boundary line in the first work data is completed from the plurality of worker terminals, displaying the object in a digital image based on the completion data A receiving and learning step of training the artificial intelligence module to generate second work data by setting a third boundary of a region where the object is located in second work target data requiring work as a work target region,
The work data generating step, the transmitting step, and the receiving and learning step are repeated when transmitting to a predetermined number of terminals among the plurality of worker terminals or until the work on the work target data is completed,
The work data generation step,
An area worked from the completed data for each of the plurality of worker terminals is generated as a mask for each of the plurality of worker terminals, a trimap is generated by merging the masks for each of the plurality of worker terminals, and a preset merging is performed in the region included in the trimap. Designating the concentration or higher as the work target area,
Crowd outsourcing work distribution method using artificial intelligence. According to claim 1,
The work data generation step,
After the receiving and learning step is performed for the first time, starting repetition after the completion data is received by a preset amount,
Crowd outsourcing work distribution method using artificial intelligence. delete According to claim 1,
In the transmission step,
The some worker terminals are terminals according to the order of the predetermined excellent worker terminals,
Crowd outsourcing work distribution method using artificial intelligence. According to claim 1,
In the receiving and learning step,
Examining the first work data before learning the artificial intelligence module;
In the transmission step,
The some operator terminals are terminals whose accuracy of the operator evaluated in the process of inspecting the first work data is greater than or equal to a preset value,
Crowd outsourcing work distribution method using artificial intelligence. According to claim 5,
In the receiving and learning step,
Creating a mask for each of the plurality of worker terminals from the completion data for each of the plurality of worker terminals, and calculating a similarity between the entire mask for each of the plurality of worker terminals and the mask of the target worker terminal to determine the accuracy,
Crowd outsourcing work distribution method using artificial intelligence. According to claim 6,
In the receiving and learning step,
The degree of similarity between the plurality of masks for each worker terminal and the mask of the target worker terminal is determined by merging the plurality of masks for each worker terminal to generate a trimap, and determining the accuracy based on the similarity with the mask of the target worker terminal. doing,
Crowd outsourcing work distribution method using artificial intelligence. According to claim 1,
In the receiving and learning step,
Learning based on the completion data whenever a preset number of the completion data is received from the communication unit,
Crowd outsourcing work distribution method using artificial intelligence. According to claim 8,
In the receiving and learning step,
Learning based on correct answer data generated after inspecting the completion data,
Crowd outsourcing work distribution method using artificial intelligence. delete

Images