KR20210105775A - Method and apparatus for cloud outsorcing task checking by using artificial intelligent - Google Patents

Abstract

A cloud outsourcing task checking method using artificial intelligence according to an embodiment of the present invention includes a data receiving step of allowing a communication part to receive completion data for a learning data forming task from a plurality of worker terminals through the Internet, a data reconstruction step of allowing a processor controlled by an artificial intelligence module to reconstruct the completion data into superimposable data, and a tendency determining step of allowing the processor controlled by the artificial intelligence module to analyze the reconstructed data for each of the workers and determining at least one of the propensity of the workers, accuracy, and difficulty of the learning data forming task.

Description

Inspection method and device for crowd outsourcing work using artificial intelligence {METHOD AND APPARATUS FOR CLOUD OUTSORCING TASK CHECKING BY USING ARTIFICIAL INTELLIGENT}

The present invention relates to a method and apparatus for inspecting crowd outsourcing tasks using artificial intelligence. More specifically, it relates to a method and apparatus for crowd-outsourcing work inspection using artificial intelligence that performs evaluation and selection of workers and work results based on the outsourcing work results.

Recently, with the development of artificial intelligence (AI) technology, research on artificial intelligence for recognizing specific objects included in digital content such as images, voices, and texts is being actively conducted. In particular, artificial intelligence technology that recognizes a specific object within a digital image generated by a digital camera and determines the situation of a photographed area is being used in various industrial fields, such as applied to surveillance cameras and autonomous vehicles.

In order to recognize a specific object in digital content using artificial intelligence, it is necessary to train artificial intelligence in advance using various learning data including the object to be recognized.

The learning data may be provided by displaying an outline, a region, etc. of an object existing in the digital content so that the artificial intelligence can distinguish between an object and a non-object in the digital content. However, since there is no data in which objects and non-objects are distinguished, an operation of displaying an outline, an area, etc. of an object in digital content is inevitably performed by a human.

As artificial intelligence learns a lot of diverse and learning data, the recognition accuracy improves, so it is important to learn as much learning data as possible. Recently, a crowd-outsourcing method of generating learning data has been used in which a plurality of workers are recruited through the Internet to easily generate the learning data and the task of displaying the outline, area, etc. of an object in digital content is requested.

At this time, each worker may determine the area including the object differently, and the worker may work insincerely, thereby generating undesirable learning data. In this case, there is the inconvenience that the inspector has to individually inspect whether the learning data creation task has been properly completed, and the problem that the artificial intelligence learned based on undesirable learning data does not properly recognize the object in the real situation can occur.

Accordingly, it is required to develop a management technology that can inspect the work results of the crowd-outsourcing method so that AI learning can be performed efficiently.

Republic of Korea Patent Registration No. 10-1887415 (2017.11.21.)

The present invention provides an apparatus and method for inspecting a crowd outsourcing operation using artificial intelligence designed so that outsourced learning data of excellent quality can be selected or inspected and learned.

As a technical means for achieving the above-described technical problem, the method of inspecting a crowd outsourcing operation using artificial intelligence according to an embodiment of the present invention, the communication unit receives the completed data for the learning data formation operation from a plurality of worker terminals through the Internet receiving data; a data reconstruction step in which the processor controlled by the artificial intelligence module reconstructs the completed data into superimposable data; and a propensity determining step in which the processor controlled by the artificial intelligence module analyzes the reconstructed data for each of a plurality of workers and determines at least one of the propensity, accuracy, and difficulty of the learning data formation task for each of the plurality of workers. .

In the data reconstruction step, depending on the type of the completed data, one-dimensional data may be reconstructed into a vector, and in the case of two-dimensional data, the entire surface may be reconstructed into a matrix having a divided surface.

The tendency determining step is a mask generating step of generating a mask for each of the plurality of worker terminals with a matrix having an element corresponding to the divided surface from the reconstruction data for each of the plurality of operator terminals, and analyzing the mask to analyze the plurality of workers It may be a propensity analysis step of determining at least one of star propensity, accuracy, and difficulty of the learning data forming task.

The tendency analysis step may determine the difficulty of the learning data forming task for the task from the deviation between the masks for each of the plurality of worker terminals for the task.

The propensity determining step may further include a Trimap generating step of generating a Trimap by merging the masks for each of the plurality of worker terminals between the mask generating step and the propensity analyzing step.

The method of merging the masks for each of the plurality of worker terminals in the Trimap generation step may be a method of indicating an average of the values of the elements of the masks for each of the plurality of operator terminals in the Trimap.

The tendency determining step may determine the accuracy of the target worker terminal by calculating a similarity between the mask of the target worker terminal and the Trimap among the plurality of worker terminals.

The propensity determining step is a method of calculating the similarity, using a cosine similarity calculated for each element with respect to the entire mask of the target worker terminal, and determining the difference between elements as one variable to be substituted for the cosine similarity. using, as another variable, the value of the element indicated by the recognition displayed on the mask of the target operator terminal, and the difference for each element is the value of the element displayed on the Trimap and the value of the element displayed on the mask of the target operator terminal it could be a car

The tendency determining step is a method of calculating the degree of similarity, wherein the sum of differences for each element is specified as a complement of 1 of a value obtained by dividing the number of elements with a value greater than 0 in the Trimap, and the difference for each element is the element displayed in the Trimap. It may be a difference between the value of and the value of the element displayed on the mask of the target worker terminal.

The tendency determining step may determine the working tendency of the target worker terminal by calculating an average of elements in the Trimap corresponding to the selected element among the masks of the target worker terminal.

The tendency determining step may determine an Opacity Mask Score (OMS) value obtained by dividing the sum of values of elements corresponding to the mask of the target worker terminal by the number of selected elements in the Trimap as the work tendency.

The tendency determining step is a method of calculating the work difficulty, and may be determined as a value obtained by dividing the sum of the values of the elements displayed in the Trimap by the number of the elements in which the value of the element is greater than zero.

As a technical means for achieving the above-described technical problem, the crowd outsourcing work inspection apparatus using artificial intelligence according to an embodiment of the present invention is a communication unit for receiving the work completion data for the work data from a plurality of worker terminals, the A processor configured to perform an operation for inspecting the completed data received from a plurality of worker terminals and a memory in which an artificial intelligence module for controlling an operation operation of the processor is stored, wherein the artificial intelligence module analyzes two or more of the completed data Thus, it is possible to determine at least one or more of the accuracy of the operator, the tendency of the operator, and the difficulty of the operation.

According to an embodiment of the present invention, the propensity or accuracy of the worker is calculated from the completed data, so that the quality of the crowd outsourcing work can be measured more accurately.

According to an embodiment of the present invention, the task difficulty of the learning data is calculated, and this is reflected in the quality determination of the completed data, so that most of the inspection is performed through the artificial intelligence module, the work load of experts can be sufficiently reduced, The efficiency of crowd outsourcing operations can be improved.

1 is a block diagram showing an apparatus for inspecting a crowd outsourcing operation using artificial intelligence according to an embodiment of the present invention.
2 is a flowchart illustrating a job management method using a crowd outsourcing job inspection device using artificial intelligence according to an embodiment of the present invention.
Figure 3 is a flowchart for explaining a method of processing data in the crowd outsourcing operation inspection device using artificial intelligence of Figure 2;
4 is a flowchart for explaining a process of generating and distributing job data in the method of inspecting crowd outsourcing jobs using artificial intelligence of FIGS. 2 and 3 .
5 is a flowchart illustrating a determination step, such as a propensity for each worker, according to an embodiment of the present invention.
6 to 11 are exemplary views for explaining an example of determining the accuracy, propensity, and work difficulty for each worker in the crowd outsourcing work inspection method using artificial intelligence of FIG. 2 .
FIG. 12 is an exemplary diagram for explaining an example of determining a reflection rate for each worker in the method of inspecting crowd outsourcing work using artificial intelligence of FIG. 2 .

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those of ordinary skill in the art can easily implement them. However, the present invention may be embodied in many different forms and is not limited to the embodiments described herein. And in order to clearly explain the present invention in the drawings, parts irrelevant to the description are omitted, and similar reference numerals are attached to similar parts throughout the specification.

Various embodiments of the present document may be implemented as software (eg, a program) including instructions stored in a machine-readable storage medium (eg, a computer). A device is a device capable of calling a stored command from a storage medium and operating according to the called command, and may include an electronic device (eg, a server) according to the disclosed embodiments. Instructions may include code generated or executed by a compiler or interpreter. The device-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' means that the storage medium does not include a signal and is tangible, and does not distinguish that data is semi-permanently or temporarily stored in the storage medium.

According to an embodiment, the method according to various embodiments disclosed in this document may be included and provided in a computer program product. Computer program products may be traded between sellers and buyers as commodities. The computer program product may be distributed in the form of a machine-readable storage medium (eg, compact disc read only memory (CD-ROM)) or online through an application store (eg, Play Store™). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily created in a storage medium such as a memory of a server of a manufacturer, a server of an application store, or a mediation server.

Each of the components (eg, a module or a program) according to various embodiments may be composed of a singular or a plurality of entities, and some sub-components of the aforementioned sub-components may be omitted, or other sub-components may be various. It may be further included in the embodiment. Alternatively or additionally, some components (eg, a module or a program) may be integrated into a single entity to perform the same or similar functions performed by each corresponding component prior to integration. According to various embodiments, operations performed by a module, program, or other component may be sequentially, parallel, repetitively or heuristically executed, or at least some operations may be executed in a different order, omitted, or other operations may be added. can

As used herein, the meaning of “sending, receiving, collecting, extracting, uploading or updating information or data” means directly sending, receiving, collecting, extracting, uploading or updating information or data, or indirectly transmitting, through another relay server; includes the meaning of receiving, collecting, extracting, uploading or updating.

Hereinafter, the present invention will be described in detail with reference to the accompanying drawings.

The present invention is a method and apparatus for inspecting a crowd outsourcing task, and may be utilized in a device for managing a crowd outsourcing task using artificial intelligence. In this specification, while schematically explaining the entire crowd outsourcing operation management apparatus and method to show the actual application of the present invention, the inspection process will be described in detail.

The crowd outsourcing job management method is a step of creating and distributing crowd outsourcing job data to workers, collecting the job results, inspecting the collected work, and learning the inspected results to AI according to its value. can be divided. Crowd-outsourcing work inspection method is the second step among them. Since the crowd-outsourced job review device uses the same processor as the crowd-outsourced job management device, they may refer to substantially the same entity. In this specification, it is collectively referred to as a crowd outsourcing work inspection device, and the device is described as being capable of adding a crowd outsourcing work management function.

1 is a block diagram showing an apparatus for inspecting a crowd outsourcing operation using artificial intelligence according to an embodiment of the present invention.

Referring to FIG. 1 , an apparatus 400 for inspecting a crowd outsourcing task according to an embodiment of the present invention may refer to a device of a system for managing data preprocessing tasks that generate learning data for learning an artificial intelligence module. Specifically, the crowd outsourcing work inspection apparatus 400 generates desirable learning data by inspecting a part of the completed data when the completion data for the task is received from a plurality of worker terminals 100 , 200 , and 300 , It may be a device that manages the learning data generation task so that the inspection of the professional inspector is performed by transmitting the remaining completed data that is difficult to the inspector terminal 500 . However, since the crowd outsourcing operation inspection apparatus 400 can perform inspection operations targeting only completed operation data, the present data pre-processing operation and data transmission function are not required.

The crowd outsourced job review device 400 may be any type or combination of types of computing devices, such as a network server, a web server, a file server, a supercomputer, a desktop computer, and the like.

The crowd outsourcing operation inspection apparatus 400 includes a communication unit 410 , a processor 420 , and a memory 430 .

The communication unit 410 may transmit the job data to the plurality of worker terminals 100 , 200 , and 300 , and when the job on the job data is completed, the completion data from the plurality of worker terminals 100 , 200 , 300 . is configured to receive In this case, the job data may be sequentially transmitted by forming a group by a portion of the plurality of worker terminals 100 , 200 , and 300 , and the completion data may also be sequentially received according to the order in which they are completed. In addition, transmission and reception may be performed simultaneously. However, when the crowd outsourcing operation inspection apparatus 400 performs only the inspection operation, there is no need to transmit data to the plurality of operator terminals 100 , 200 , and 300 .

The communication unit 410 is connected to the Internet and may be configured with a network adapter capable of transmitting and receiving data to and from the operator terminals 100 , 200 , 300 and the inspector terminal 500 through the Internet. The communication unit 410 may be configured to be connected to the Internet through a wired (LAN or WAN) or wireless (Bluetooth, WiFi, Infrared data association (IrDA) method).

The processor 420 may generate work data to be transmitted to the plurality of worker terminals 100 , 200 , and 300 , and perform an operation of inspecting the completed data received from the plurality of worker terminals 100 , 200 and 300 . configured to perform

The processor 420 is a dedicated processor (eg, an embedded processor) for performing the above-described arithmetic operation, or a general-purpose processor capable of performing the operations by executing one or more software programs stored in the memory device. (eg, CPU, graphic processing units (GPU), single-core processor, multi-core processor, application specific integrated circuits (ASIC), etc.), or a communication processor (CP). .

The memory 430 is a device in which an artificial intelligence module for controlling the arithmetic operation of the processor 420 is stored, for example, a solid state drive (SSD), a hard disk drive, or a card type memory (SD or XD). memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM) , a storage medium of at least one type of a magnetic memory, a magnetic disk, and an optical disk, and various storage media capable of storing identifiable physical location information thereof. In FIG. 1 , the memory 430 is illustrated as being located within the crowd outsourcing operation inspection device 400 , but in some embodiments, the memory 430 is a separate device physically separate from the crowd outsourcing operation inspection device 400 . can be composed of In this case, the memory 430 and the crowd outsourcing operation inspection device 400 may be connected through a network.

The artificial intelligence module is a set of instructions for controlling the operation of the crowd-outsourced work inspection device 400, and may include computer- or machine-executable instructions that are described in any suitable programming language and perform the various functions described above. have.

When the communication unit receives the completed data on which the work for the work data is completed from the plurality of worker terminals 100, 200, and 300 through the Internet, the processor controlled by the artificial intelligence module reconstructs the completed data into superimposable data .

In addition, the processor controlled by the artificial intelligence module determines the accuracy, propensity and work difficulty for the work for each worker, and determines the reflection rate for each worker based on the accuracy and difficulty level for the plurality of workers, and to the reflection rate. It is configured to determine the inspection reference data for the inspection of the completed data by a plurality of workers based on it. Operations of the artificial intelligence module will be described in more detail with reference to FIGS. 2 to 12 .

Meanwhile, the worker terminals 100 , 200 , and 300 are terminals used by workers, and may be implemented as a computing device for processing work data to generate completion data. In addition, the inspector terminal 500 is a terminal used by the inspector, and may be implemented as a computing device for performing inspection on completed data requiring professional inspection among completed data. For example, the worker terminals 100 , 200 , 300 and the inspector terminal 500 are a smartphone, a tablet PC, a mobile phone, a video phone, an e-book reader, a desktop PC, a laptop PC, a netbook computer, a workstation, a server , a PDA, a PMP, an MP3 player, a mobile medical device, a camera, or a wearable device.

Figure 2 is a flowchart for explaining a job management method using the crowd outsourcing job inspection device using artificial intelligence according to an embodiment of the present invention, Figure 3 is the data from the crowd outsourcing job inspection device using the artificial intelligence of Figure 2 It is a flowchart for explaining the processing method.

Referring to FIGS. 2 and 3 , first, the crowd outsourcing inspection apparatus may set a work target area of the work target data to generate the job data ( S210 ).

“Work target data” means original data of work data to be transmitted to the operator terminal 100 , and the work target data may be composed of various types of data such as digital images, texts, and sounds. The crowd outsourcing job inspection device 400 generates job data by pre-processing the job target data using an artificial intelligence module.

The crowd outsourcing work distribution method using artificial intelligence is a work data generation step (S211) in which the processor controlled by the artificial intelligence module sets the work target area of the work target data that requires work to generate the job data (S211), and the communication unit includes the job data A transmitting step (S212) of transmitting a plurality of worker terminals connected to the Internet to some of the worker terminals and the communication unit. and a receiving and learning step (S213) of learning the artificial intelligence module based on This is repeated until the operation on the target data is completed. A specific example corresponding to this will be described later with reference to FIG. 4 .

When the completion data is received, the completion data can be inspected. The step of inspecting the completed data (S30) is a step of reconstructing the completed data that reconstructs the work data into data that can be overlapped according to the type of the work data (S34) and analyzing the completed data to determine the accuracy of the worker, the worker's work propensity , and a step (S220) of determining the propensity for each worker to evaluate the work difficulty, etc. A specific example corresponding to this will be described later with reference to FIGS. 5 to 11 .

A step (S55) of forming work inspection reference data by evaluating the ability of each worker based on the worker-specific propensity evaluation data and the completed data may be additionally performed. The step of forming the work inspection standard data (S55) is a step of generating an inspection standard to be used for the inspection of the work after the work inspection step. Since the accuracy of the work is calculated in S30, it is possible to directly generate the inspection standard data by reflecting it as a weight (S30), but depending on the difficulty of the work data and the accuracy of the work, there may be tasks that are inappropriate to mechanically generate the inspection standards. It is preferable to additionally perform a filtering step (S230) to exclude a job and a reflection rate determination step (S240) for each worker to evaluate and utilize the capabilities of each worker in advance to generate the inspection reference data according to the weight (S250). ( S55) A specific example of this step will be described later with reference to FIG. 12 .

4 is a flowchart illustrating a process in which job data corresponding to S210 of FIGS. 2 and 3 is generated and distributed.

Referring to Figure 4, the crowd outsourcing work distribution method using artificial intelligence is a work data generation step in which the processor controlled by the artificial intelligence module sets the work target area of the work target data that requires work to generate the work data, the communication unit A transmitting step of transmitting the work data to some of the worker terminals connected to the Internet, and the communication unit when the completed data on which the work on the work data is completed is received from the plurality of worker terminals to the communication unit, the completed data When transmitting a job data generation and distribution step comprising a receiving and learning step of learning the artificial intelligence module based on Repeat until Hereinafter, each step will be described in detail with an example.

The artificial intelligence module of the crowd-outsourcing work inspection device sets a work area for the first work target data to generate the first work data (S214).

As mentioned above, the work target data is digital content including an object to be learned by the artificial intelligence module, and may be composed of various types of data such as images, texts, and sounds.

For convenience of explanation, a case in which the work target data is composed of image data will be exemplified below.

The artificial intelligence module first randomly selects the first work target data from all work target data, and sets the work target area where the object to be learned by the artificial intelligence module is located in the first work target data to be distinguished from other areas, 1 Create work target data. For example, if the object to be learned is a “dog”, the artificial intelligence module generates the first job data by marking the boundary of the area where the “dog” is located in the first job target data including the “dog”. can

In this case, a method for the AI module to recognize the boundary of the area where the “dog” is located may use Trimap, which will be described later. In other words, as data for learning the artificial intelligence module, the area used for learning from the completed data for each worker terminal is generated as a mask for each worker terminal, and a Trimap is generated by merging the masks for each worker terminal. In the area included in this Trimap, a part marked as an area used for learning with a frequency higher than a preset number can be designated as the work target area.

Thereafter, the generated first job data is transmitted to some worker terminals among a plurality of worker terminals connected to the Internet. At this time, since some worker terminals are given priority to work data, the initial learning material of the artificial intelligence module is generated. Since the AI module will be used for all future work data production, the initial performance of the AI module is important, and the quality of the initial learning material that determines the initial performance of the AI module is also important. Therefore, in order to improve the quality of the initial learning material, it is advantageous that some of the worker terminals receiving the first work data are worker terminals having excellent capabilities. This excellent worker terminal may be determined by any criteria, such as the worker's career and major field. However, since the crowd outsourcing inspection management device using artificial intelligence evaluates the accuracy for each job of each terminal, the crowd outsourcing job inspection device using artificial intelligence evaluates the worker's accuracy in the process of inspecting the job data. A worker can be designated as a good worker terminal.

Then, the crowd outsourcing work inspection device receives the first completed data for the first work data from the worker terminal 100 (S215).

The first completed data means data in which the worker completes the work for the first work data through the worker terminal 100 . For example, when the object to be learned by the artificial intelligence module is a “dog”, the operation on the first job data may be an operation of displaying the outline of the “dog”. The operator may generate the first completed data by precisely editing the boundary line in the first work data in which the boundary line of the area where the “dog” is located is schematically displayed to fit the “dog” outline.

When the first completed data is received from the worker terminal, the crowd outsourcing task inspection device learns the artificial intelligence module based on the first completed data (S216).

Although the boundary line for “dog” is schematically displayed in the first work data, the boundary line for “dog” will be more precisely displayed in the first completed data worked by a human.

When a plurality of first completed data is received, the accuracy of a specific worker terminal among them is calculated in the process of learning the first completed data to the artificial intelligence module. The process of inspecting and evaluating the completed data and training the AI module will be described later.

The artificial intelligence module is learned based on the first completed data, and when the learned artificial intelligence module is used, the work area for "dog" will be set more accurately and precisely with respect to other images including "dog". can

Thereafter, the crowd outsourcing work inspection device sets a work area for the second work target data based on the learned artificial intelligence module to generate the second work data (S217).

The second work target data may be selected work target data among the remaining work target data except for the first work target data. For example, when the work target data consists of 100 image data including “dogs”, the first work target data may consist of 10 images including “dogs”. The artificial intelligence module generates the first work data by displaying the area including the “dog” in the 10 first work target data, and the first completed data in which the work on the 10 first work target data is completed from the worker terminal When is received, it may be learned based on the first completed data. The learned AI module can more precisely display the area including “dog” for the second task data composed of 10 other images, and the second task data is generated through the artificial intelligence module. Thereafter, when the second job data is transmitted to a plurality of worker terminals, and the second completed data in which the work for the second work data is completed is received from the worker terminal, the artificial intelligence module is learned based on the second completed data. In the same way, the third work data is generated by setting the area including “dog” for the third work target data composed of another 10 images through the artificial intelligence module. In this case, the learned artificial intelligence module may display a region including “dog” for the third work target data more precisely than when processing the second work target data. In the same way, the AI module is periodically trained until the job on the target data is completed, and the precision of the job data generated through the AI module can be improved as the learning is repeated.

In other words, the step of generating/transmitting the job data from the job target data by the AI module and the step of receiving the completion data and learning the AI module may be repeatedly performed. This step may be repeated until all the tasks for the task data are distributed, and in the case of a duplicate task for the same task data, it will be repeatedly performed until the task data is generated and sent to a preset number of worker terminals. may be

In some embodiments, the learning of the artificial intelligence module may be performed based on the first correct answer data that has been checked, not based on the first completed data. That is, the artificial intelligence module may be learned based on the first correct answer data determined by completing the inspection of the first completed data prior to processing the second work target data, and through the learned artificial intelligence module, the second The work target data may be processed to generate second work data. In this case, the first correct answer data may correspond to the data inferred to be the final correct answer by completing the inspection according to the inspection method of the completed data to be described later.

The above content is about the process of generating and distributing work data and is not directly related to the present invention, but has been described as a reference because it is a function that the crowd outsourcing work inspection device using artificial intelligence of the present invention can do together.

Referring back to FIG. 3 , the crowd outsourcing work distribution apparatus may inspect the completed data ( S30 ) when the completed data on which the work on the work data is completed is received from the worker terminal.

The completed data verification step (S30) will be described with reference to FIG. 5 showing the detailed steps of the completed data verification step (S30).

First, the completed data is reconstructed into superimposable data (S34). For example, sound or text is one-dimensional data, and images and the like are two-dimensional data. Each data is preferably the result of a plurality of workers can be collected. Accordingly, to enable overlap for quantities of the same size, one-dimensional data may be reconstructed as a vector and two-dimensional data may be reconstructed as a matrix.

In particular, when describing two-dimensional data in more detail, in the case of image data, it may be a matrix in which an image is divided into m X n surfaces. Also, depending on the type of 2D data, it can be converted into a 2D matrix of m X n (where m and n are integers greater than or equal to 1) corresponding to the corresponding surface 1:1. The value of this matrix may have 1 if included to distinguish whether the corresponding face is included or not, and 0 if not included.

After reconfiguration, at least one of accuracy, tendency, and work difficulty for a plurality of worker-specific tasks is determined (S220).

Crowd outsourcing task inspection apparatus 400 according to an embodiment of the present invention determines at least one of the difficulty of the task accuracy, propensity, and learning data formation task for a plurality of workers using an artificial intelligence module, and Based on this, there is a characteristic of inspecting the completion data of workers for other tasks. Hereinafter, with reference to FIGS. 6 to 11 , a method of determining the work accuracy, propensity, and difficulty for a work for each of a plurality of workers will be described in detail.

A step of generating a plurality of masks for each worker ( S223 ) will be described with reference to FIG. 6 .

The plurality of worker-specific masks OM1 , OM2 , and OM3 may be masks in which a corresponding mark is written on a mattress in a work target in which an area for a specific object is marked. For example, if the work target corresponds to a digital image and is a job of defining a specific object in the digital image, a plurality of workers define the specific object by displaying the outline of the specific object in the digital image according to their respective criteria. can do. In this case, the completion data received from the operator terminal 100 may be data for a digital image in which the outline of a specific object is displayed. If the data reconstructed as described above is a matrix in which each element is matched 1:1 with a face divided into m X n faces, a plurality of worker-specific masks (OM1, OM2, OM3) is the shape of a specific object in the completed data. An element value corresponding to the divided surface in which the area defined by is equal to or greater than a preset ratio may be 1, and an element value corresponding to the divided surface having a preset ratio or less may be a matrix in which 0 is described.

Thereafter, the artificial intelligence module generates a Trimap by merging the plurality of worker masks (OM1, OM2, OM3) (S226).

Since the plurality of workers mark the outline of a specific object in the digital image according to their respective criteria, the result data and masks OM1 , OM2 , and OM3 of each worker have different shapes. The AI module creates a Trimap(TM) by merging a plurality of worker-specific masks (OM1, OM2, OM3). Specifically, one merged mask is generated by summing element values of corresponding elements of the masks OM1 , OM2 , and OM3 for each worker. In this case, the element value in the element on the surface where the display area of the mask OM1 of the first operator overlaps the display area of the mask OM2 of the second operator and the display area of the mask OM3 of the third operator overlap It may be expressed as a normalized sum of the element value of the mask OM1 of the first operator, the element value of the mask OM2 of the second operator, and the element value of the mask OM3 of the third operator. For example, if the element value of the mask display area is defined as 1 and the element value of the non-display area is 0, the mask OM1 of the first operator, the mask OM2 of the second operator, and the third operator In the element corresponding to the surface where all the display areas of the mask OM3 overlap, the sum of the element values is defined as 3, which can be normalized by multiplying it by the Opacity defined as 1/n (however, n = the number of masks). have. Accordingly, the sum of the element values can be normalized to 3*1/3 = 1. In the same way, each element value of the integrated mask may be calculated as a normalized sum of the corresponding element values of the mask OM1 of the first operator, the mask OM2 of the second operator, and the mask OM3 of the third operator. have. Therefore, Trimap can be expressed as an average matrix obtained by dividing the sum of each mask expressed as a matrix by the number of masks.

Although an example in which the Trimap is formed in the form of a two-dimensional matrix is illustrated in FIG. 6 , the Trimap may be formed in the form of a one-dimensional vector according to the form of completed data. For example, if the completion data corresponds to voice data, a mask for each worker may be generated for an area in which the voice of a specific object is displayed, and a one-dimensional vector form of a Trimap may be generated by merging the masks for each worker.

On the other hand, the above-described Trimap is generated based on the completion data for each worker generated for the same job data. That is, in the case of a task of defining a specific object within the same digital image, a Trimap may be generated by receiving a plurality of worker-specific completion data for the same digital image and overlapping a plurality of worker-specific masks.

Thereafter, the artificial intelligence module determines the work tendency, work accuracy, and difficulty of the work of the target worker based on the mask and the trimap of the target worker among the plurality of workers (S229).

Specifically, the work tendency of the target worker is determined based on the following [Equation 1].

Here, the Opacity Mask Score (OMS) is an index for identifying the work propensity for each worker, and may be an index indicating the degree of agreement between the worker's specific work result and the work results of the rest of the workers. In [Equation 1], TM(t) is the element value of the Trimap for a specific task (ie, completed data) (t), and BM(t, i) is the specific task (t) of the target worker (i). It means the element value of the mask for That is, OMS means the average of element values for the trimap of the selected area of the mask for a specific task (t) of the target operator (i).

In the case of an operator with high OMS, most of the elements with high element values are selected in the Trimap, so the area selected by the operator overlaps the area selected by other operators. Accordingly, it can be seen that workers with high OMS tend to under-display the outline of a specific object compared to other workers.

On the other hand, in the case of an operator with a low OMS, since many elements having a low element value are selected in the Trimap, the area selected by the operator may correspond to an area not selected by other operators. Accordingly, it can be seen that workers with low OMS tend to over-display the outline of a specific object compared to other workers.

7(a) shows the operator's mask 711 and Trimap's mask 713 with low OMS, and (b) shows the operator's mask 715 and Trimap's mask 713 with high OMS. For convenience of explanation, only elements corresponding to the boundary lines of each mask are indicated by solid lines in FIG. 5 .

As shown in (a) of Figure 7, it can be seen that the operator's mask 711 with a low OMS is located inside the Trimap mask 713, and through this, the operator with a low OMS can change the definition of a specific object. It can be seen that the display is understated compared to the workers.

On the other hand, as shown in (b) of FIG. 7 , it can be confirmed that the mask 715 of the operator with high OMS is located outside the Trimap mask 713 , and through this, the operator with high OMS defines a specific object It can be seen that displayed excessively compared to other workers.

In addition, the artificial intelligence module may determine the work accuracy for each of the plurality of workers based on the similarity with the work results of other workers. Since the work results of other workers are integrated into Trimap, the actual work accuracy of multiple workers is evaluated based on the similarity to Trimap.

Specifically, the work accuracy for a plurality of workers is measured based on the following [Equation 2].

Here, TWS (Task-Worker Similarity) is an index indicating the work accuracy of the worker, and means the cosine similarity between the work result of a specific worker and the work result of the other workers. In [Equation 2], V _t means a one-dimensional vector of the Trimap of a specific task (t), and V _t,i is a one-dimensional vector of a worker mask of a specific task (t) of a specific operator (i) it means.

In other words, the similarity for calculating the accuracy uses the cosine similarity calculated for each element with respect to the entire mask of the target worker terminal, and uses the element-specific difference as one variable to be substituted for the cosine similarity, As another variable, the value of the element indicated by the recognition indicated on the mask of the target operator terminal is used.

8 is a diagram for explaining an example of a TWS calculation process. Referring to FIG. 8 , Trimap(TM) in the form of a two-dimensional matrix for a specific task (t) _{can be one-dimensional vectorized (V t} ) by applying a Flatten function, and a specific operator (i) for a specific task (t) ) of the worker mask (OM) can also be one-dimensional vectorized (V _t,i ) by applying the Flatten function. Thereafter, _{by calculating V t} -V _t,i , a one-dimensional vector for the work results of the remaining workers except for the specific worker (i) is calculated, and the cosine between _{V t} -V _t,i and V _t,i By calculating the similarity, the cosine similarity between the work results of the specific worker (i) and the work results of the workers other than the specific worker (i) is calculated.

9 is a view showing a TWS calculation result according to the work result of a specific operator. Referring to FIG. 9 , after one-dimensional vectorization of the mask of a specific operator and one-dimensional vectorization of the Trimap, the TWS was calculated based on Equation 2 above. As a result, the TWS of the specific operator was calculated to be 0.936. Through this, it can be seen that the work result 911 of a specific operator has a similarity of about 0.936 compared to the result 913 of the Trimap. When the similarity of the work result 911 of a specific operator is lower than that of the Trimap result 913, the TWS value may be lowered.

Cosine similarity is useful for comparing the magnitude of similarity in order, but has a drawback in that it cannot intuitively indicate the quantitative difference.

In some embodiments, in order to improve the intuitiveness of the TWS of the operator, the artificial intelligence module uses element-specific differences instead of cosine similarity for the real work area excluding elements that no one has marked as a recognition area, which is an imaginary number of the work. indicators can be used.

In this case, a Trimap score (TMS) based on the following [Equation 3] may be used instead of the TWS.

Here, TMS(t,i) is a Trimap score of a specific worker (i) for a specific job (t), and minimizes the variance for a part consistent with the job results of other workers among the job results of the specific worker (i) Thus, it can be described as an index that maximizes the accuracy index for the work result of a specific worker (i). In [Equation 3], TM(t) means the element value of the Trimap for a specific job (t), and BM(t,i) is the job mask of a specific worker (i) for a specific job (t) means the element value of On the other hand, n means the number of elements whose element values exceed 0 in the Trimap.

10 is a diagram illustrating a calculation process of TMS(t,i) of a specific worker (i) for a specific job (t). Referring to FIG. 10 , when the operation result for a specific operation t is expressed as a mask composed of 3*3 = 9 elements, the sum of |TM(t)-BM(ti)│ is shown in FIG. As shown, it is calculated as 2, and when TMS(t,i) is calculated according to Equation 3 above, TMS(t,i) is calculated as 0.77.

On the other hand, the artificial intelligence module determines the task difficulty for a particular task from the deviation of a plurality of worker masks for the particular task.

A high task difficulty means that there are many difficulties in judgment, and a lot of difficulty in judgment means that various judgments may appear for the same task. Therefore, the large variation in the mask for each worker means that the work difficulty is high. The deviation between the masks for a plurality of workers may be calculated by calculating the area of each mask to obtain the deviation.

If the determination value (1 or 0) for a specific element appears evenly in the masks for each worker, it may be determined that the determination of the corresponding element is difficult, and if the number of these elements is large, the work difficulty may be considered high. . If the values of each element are determined to be various, the average value of the corresponding element will converge to 0.5. Since the TM has already calculated the average value of each element in the masks for each worker, it can be calculated using this.

Specifically, the task difficulty for a specific task is determined by dividing the sum of the values of the elements displayed in the Trimap by the number of elements whose values are greater than 0. It may be determined based on [Equation 4], which is expressed as an equation.

Here, ICS(t) is an index indicating the difficulty of the task for a specific task t, and may be described as an index indicating how clear the task is without being ambiguous. In Equation 4, TM(t) denotes an element value of Trimap for a specific operation t, and n denotes the number of effective elements whose element value exceeds 0. The area that all workers can distinguish (element value is 0) corresponds to an imaginary number in the amount of work, so the measurement was taken as an example. It is free to be specific. As the work becomes clearer, the work results of the workers will be similar, and since the overlapping area of the work masks is wide, ICS(t) in Equation 4 can be calculated to be large.

It may be substituted for simply obtaining the normal distribution for each element of each Trimap and calculating based on the frequency or amount of an element whose value is close to the middle (0.5 in this embodiment). In this case, unlike the equation, the ICS value has a value proportional to the difficulty.

11 is a diagram illustrating ICS(t) calculation results for actual tasks. As shown in FIG. 11 , it can be seen that the ICS is highly measured for an image in which the shape of an object is simple and can be relatively clearly defined.

After the step (S30) of inspecting the completed data, by utilizing the accuracy and work difficulty calculated as a result of the inspection, it is possible to form the work inspection reference data (S55).

Referring back to FIGS. 2 and 3 , the crowd outsourcing task inspection device determines whether a reflection rate is a workable task for the target tasks of the completed data based on the accuracy and task difficulty for a plurality of workers (S230), and the reflection rate is impossible to calculate If it's a job, filter it.

Here, the reflection rate (voting power) may be described as a weight or a contribution rate of the operator who contributed to the formation of the Trimap of a specific task. Crowd outsourcing work inspection apparatus according to an embodiment of the present invention provides a high reflection rate to a worker who accurately performed a specific task in the process of determining the inspection reference data (ie, correct answer data) necessary for the inspection for a specific task It is configured to improve the reliability of the reference data.

On the other hand, if the specific task itself is ambiguous and the difficulty level is too high, there may not be a worker who performed the task accurately enough. In addition, if a plurality of workers who performed a specific job did not all faithfully perform the job, only the completion data of the bad workers exists for the job. Low acceptance reference data may be selected.

Accordingly, the device for inspecting the crowd outsourcing work according to an embodiment of the present invention may first filter whether the corresponding task is a suitable task for calculating the reflection rate before calculating the reflection rate for each worker through the artificial intelligence module.

For example, the artificial intelligence module examines whether the maximum value of the TWS of the completed data received from the plurality of workers exceeds the accuracy threshold, and the ICS exceeds the difficulty threshold. TWS may be calculated based on [Equation 2], and it may be determined whether the highest maximum value among TWS values for the task to be determined exceeds the accuracy threshold. As mentioned earlier, since TWS is an indicator of the work accuracy of workers, a high maximum value of TWS may mean that among the workers who performed this work, excellent workers who performed the work with high accuracy are included. .

Also, the ICS of the completed data may be calculated based on Equation 4 above. As mentioned earlier, the higher the ICS, the lower the difficulty of the task, so that the ICS for a particular task exceeds the difficulty threshold may mean that the particular task has an appropriate difficulty.

If the maximum TWS of the completed data for a specific task is less than the accuracy threshold and the ICS for a particular task is less than the difficulty threshold, the difficulty of the task itself is too high, and there is no completion data of an excellent worker with high accuracy. Therefore, it may be judged as an ambiguous operation that cannot calculate the reflection rate. In this case, the crowd outsourcing work inspection apparatus 400 transmits the completion data for the ambiguous task to the inspector terminal 500, and the expert inspection for the ambiguous task is performed through the inspector terminal 500 (S255).

On the other hand, if the maximum TWS of the completed data for a specific job exceeds the accuracy threshold and the ICS for a specific job exceeds the difficulty threshold, the job includes the completion data of the best worker, and the difficulty of the job itself is Since it is an appropriate level, it can be considered that it has the target qualifications to select excellent workers.

Accordingly, the crowd outsourcing work inspection device determines that the target for the calculation of the reflection rate for the corresponding task is eligible, and determines the reflection rate for each of the plurality of workers based on the accuracy and the difficulty of the plurality of workers (S240).

Specifically, the crowd outsourcing work inspection apparatus 400 according to an embodiment of the present invention, prior to calculating the reflection rate for each worker through the artificial intelligence module, complete data of excellent workers whose accuracy is greater than or equal to a preset threshold value among a plurality of workers extract

Specifically, the artificial intelligence module calculates TWS for a plurality of workers for tasks satisfying the above-described conditions, and determines whether the TWS exceeds a first threshold. The higher the TWS, the higher the worker's work results can be seen to be consistent with the work results of other workers, so it is more likely to be close to the correct answer data. Accordingly, the artificial intelligence module filters only the completion data of workers whose work accuracy is greater than or equal to a preset first threshold. Completion data of workers less than the first threshold value corresponds to work results with low accuracy. On the other hand, as can be seen from the TWS definition of [Equation 2], the TWS for a plurality of workers is defined as the cosine similarity of the Trimap generated by overlapping the completion data of a specific operator and the Trimap, that is, the completion data of a plurality of workers, If there are bad workers with too low accuracy, the accuracy of Trimap itself may be lowered, and it can be seen that it has the effect of lowering the TWS for workers with good intentions for the work.

Therefore, the artificial intelligence module of the device for inspecting crowd outsourcing work according to an embodiment of the present invention removes the completion data in which the TWS is less than the first threshold value among the completion data of the plurality of workers, and the completion data in which the TWS is equal to or greater than the first threshold value A Trimap is generated again by selecting only a few, and the TWS for a plurality of workers for a specific job is recalculated based on the re-created Trimap.

Thereafter, the artificial intelligence module examines whether the TWS recalculated based on the completion data of the selected plurality of workers exceeds the second threshold value, and the completion data of specific workers having a TWS exceeding the second threshold value Selected as completion data of excellent workers.

The accuracy threshold, the difficulty threshold, the first threshold, and the second threshold may be variously determined according to the required accuracy of the learning data. For example, in the case of artificial intelligence applied to an autonomous vehicle, since it is necessary to accurately recognize an object, learning based on accurate learning data may be required. In this case, by setting the above-described threshold to be high, the accuracy of the training data can be improved.

Thereafter, the artificial intelligence module determines the IOU (Intersection over Union) of the excellent worker by calculating the similarity between the correct answer data for the quality check job and the completed data of the excellent worker for the quality check job.

Specifically, the artificial intelligence module may select the completion data for the quality check task among the tasks of the excellent worker. Here, the quality check task refers to a task in which the correct answer data (ground truth) is secured in advance to determine the reflection rate for each excellent worker. A star reflection rate may be determined.

IOU means a value obtained by dividing the area of the intersection of the two areas A and B by the value of the sum area, and means a Jaccard index defined by Equation 5 below.

In the case of excellent workers, since the work accuracy is high, the IOU between the completion data and the correct answer data for the quality check task may appear high.

Then, the artificial intelligence module calculates the quality score of the excellent worker from the ratio of the IOU of the excellent worker to the task difficulty for the quality check task.

Specifically, the quality score may be calculated through the following [Equation 6].

Here, QScore means the quality score of a specific good worker for a quality check task (QcT). On the other hand, in [Equation 6], QcT _w,i means completion data for the latest i-th quality check task (QcT) of a specific excellent worker (w). In addition, QcT _gt,i means correct answer data for the latest i-th quality check task (QcT).

As can be seen from [Equation 6], the artificial intelligence module can give a higher quality score to an excellent worker w having excellent accuracy for a difficult task (low ICS).

Thereafter, the AI module determines the reflection rate of excellent workers based on the quality score.

Specifically, the AI module determines the reflection rate of a specific excellent worker by applying a weight according to the period to the quality score of a specific excellent worker. For example, as shown in Equation 7 below, by applying a weight (DecayWeight) for a period to the quality score (QScore) of a specific excellent worker, and dividing the number of quality check tasks (QcT) to be applied to the reflection rate calculation , the influence score (IP) of a specific excellent worker can be calculated.

Here, IP means an index indicating the degree of influence of a specific excellent worker reflected in determining the inspection standard data (ie, correct answer data) for a specific task to be inspected. QcT Range means the number of quality check tasks (QcT) to be applied to IP calculation, and the higher the QcT Range, the higher the IP reliability. DecayWeight is a weight for a period, and may be an element having a larger value as it is closer to the current time point, and is defined by the following [Equation 8].

Here, Weight means weight, and t means QcT Range. i means an integer from 1 to t-1. On the other hand, the specific value of the weight may be variously determined according to the task difficulty, type, and the like.

For example, the process of calculating the influence score (IP) of a specific excellent worker (i) may be described as follows. If the influence score (IP) is calculated based on the five quality check tasks (QcT), first, the artificial intelligence module determines the five quality check tasks (QcT) based on the completion data of a specific excellent worker (i). After calculating the IOU and calculating the task difficulty (ICS) of the quality check task (QcT), it is possible to calculate a quality score (QScore) for three excellent workers based on Equation 6 above. Specifically, the IOU for the five quality check tasks (QcT) may be calculated based on [Equation 5], and the ICS may be calculated based on the [Equation 4]. An example of the calculation of IOU and ICS for 5 quality check tasks (QcT) of a specific excellent worker (i) is shown in [Table 1] below.

division IOU ICS QcT1 0.9 0.8 QcT2 0.8 0.9 QcT3 0.8 0.7 QcT4 0.9 0.9 QcT5 0.7 0.8

In this case, according to the above [Equation 6], the quality score (Qscore) of the specific excellent worker (i) is calculated as follows. QScore = [1.12, 0.88, 1.14, 1.00, 0.87]

In addition, when the weight is 0.9, the weight (DecayWeight) according to the period of the five quality check tasks (QcT) may be calculated as follows according to Equation (8).

DecayWeight(0.9, 5)= 0.9 ⁱ = [1.0, 0.9, 0.81, 0.73, 0.65]

Therefore, the influence score (IP) for a specific excellent worker (i) can be calculated as follows.

Then, the artificial intelligence module calculates the reflection rate (Voting Power; VP) by normalizing the influence score (IP) so that the sum of the influence scores (IP) of all excellent workers becomes 1.

For example, if there are 3 excellent workers and the influence score (IP) for the 3 excellent workers is as shown in [Table 2] below, so that the sum of the influence points (IP) for the 3 excellent workers is 1 By normalization, the reflection rate (VP) can be calculated as shown in [Table 3] below.

Worker w1 w2 w3 Impact Point One 0.9 0.8

Worker w1 w2 w3 Voting Power 1/2.7 0.9/2.7 0.8/2.7

Referring again to FIG. 2 , the artificial intelligence module determines the inspection reference data for inspection of the completed data for each worker based on the calculated reflection rate (S250). For example, by applying the reflection rate (VP) to the Opacity applied when forming the Trimap, it is possible to determine the inspection standard data (ie, the inspection standard Trimap).

12 is a view for explaining an example in which the reflection rate for three excellent workers is calculated, and the inspection reference data is determined based on the reflection rate.

Referring to FIG. 12 , the reflection rates for the three excellent workers w1, w2, and w3 are calculated as shown in FIG. 12 , respectively, and inspection reference data is determined based on the reflection rates.

Determination of the inspection reference data may be determined based on the reflection rate of a specific excellent worker selected according to the type of work result.

Specifically, if the variable type of the completion data is categorical data, the method of taking the average value of the completion data cannot be applied, so the completion with the highest reflection rate of the best worker with the highest reflection rate among the excellent workers Determine the data as the inspection reference data.

On the other hand, when the variable form of the completed data is continuous data, it can be considered that statistical access to the completed data is possible. In this case, the AI module determines the inspection standard data by applying an inference model suitable for the completion data of excellent workers.

For example, as shown in FIG. 12 , by applying the reflection rate of excellent workers as Opacity to the completion data of each worker, Opacity masks (OM1, OM2, OM3) for each worker are generated, and by overlapping them, Trimap (TM ) and selecting only elements having element values exceeding a specific threshold in the formed Trimap(TM), it is possible to determine the inspection reference data.

However, the method of determining the inspection standard data is not limited thereto, and the AI module selects various statistical values such as the average value and the median value of the completed data to which the reflection rate of excellent workers is applied, or an alpha-matting algorithm. It can be applied to determine the inspection standard data.

In some embodiments, the artificial intelligence module may apply the selective inference model without applying the statistical inference model, even if the completed data is in the form of a continuous variable. That is, even if the completed data has a continuous data value between 0 and 1, if the correct answer data needs to be determined as 0 or 1 due to the nature of the task, the completion data of the best worker with the highest reflection rate is applied by applying the selective inference model. It can be set as correct answer data.

Then, the crowd outsourcing work inspection apparatus 400 according to an embodiment of the present invention adjusts the existing reflection rate (VP) for each worker based on the calculated reflection rate (VP) for each worker. That is, the reflection rate (VP) is adjusted to increase for workers who are determined to have a lot of correct data among the completed data through inspection, and the reflection rate (VP) can be adjusted to decrease for workers who are determined to have less correct data among the completed data. have.

Crowd outsourcing job inspection apparatus 400 according to an embodiment of the present invention, based on the determined job data to inspect the completed data of the workers, and selects the completed data processed as the correct answer as the learning data. Since artificial intelligence can be learned based on the learning data that has been inspected, the learning efficiency of artificial intelligence can be increased, and the performance of artificial intelligence can be advanced.

As described above, the crowd outsourcing task inspection apparatus 400 using artificial intelligence according to an embodiment of the present invention may distribute the task of generating learning data for artificial intelligence learning to unspecified workers through the Internet. . In particular, by performing pre-processing on the work target data through an artificial intelligence module, and transmitting the pre-processed work data to unspecified workers, the work of the workers can be performed more easily. That is, the artificial intelligence module generates job data by performing pre-processing to set the work target area on the job target data, and can transmit the generated job data to a plurality of workers, so that workers can more easily create learning data can be performed.

In addition, since the artificial intelligence module is configured to learn the first completed data in which the work for some first work data is completed in the process of generating the work data, it can be gradually learned in the process of generating the work data, the first work After distributing the data, in the process of generating the second and third work data, the work area may be set in a more advanced way, whereby highly pre-processed work data may be generated and distributed to the workers. Accordingly, the workload of the plurality of workers may be further reduced, and ultimately, the creation of the learning data itself may be configured to be performed by the artificial intelligence module.

In addition, the crowd outsourcing task inspection apparatus 400 using artificial intelligence according to an embodiment of the present invention calculates the task difficulty and the accuracy of the workers who performed the task, the task difficulty is sufficiently low, and the accuracy of a plurality of workers If is high enough, it is configured to perform inspection through the AI module for the corresponding task. In addition, when the task difficulty is too high or the accuracy of a plurality of workers is too low, the inspection through the artificial intelligence module is unclear, it is configured to perform the expert inspection for the task. In other words, experts only need to perform inspections for some ambiguous tasks, and most inspections are performed through an artificial intelligence module, so the work load of experts can be sufficiently reduced and the efficiency of crowd outsourcing work can be improved.

In this case, the crowd outsourcing task inspection apparatus 400 using artificial intelligence according to an embodiment of the present invention determines the accuracy, propensity, and work difficulty for each of a plurality of workers, and reflects the rate of reflection based on the accuracy and difficulty of each of the plurality of workers. It is configured to determine the correct answer data for a plurality of workers-specific completed data inspection based on the reflection rate. That is, a high reflection rate is given to a worker with excellent accuracy, and in determining correct answer data, the completion data of an excellent worker is reflected at a high rate. Therefore, it is possible to minimize the problem that the reliability of the correct answer data is lowered due to the work results of a large number of workers with low accuracy, and the reliability of the inspection by the artificial intelligence module can be further improved.

In particular, the crowd outsourcing operation inspection apparatus 400 using artificial intelligence according to an embodiment of the present invention applies a weight according to a period in determining a reflection rate for each of a plurality of workers. Even a good worker may have lower accuracy in recent work due to various factors. Since the crowd outsourcing work inspection apparatus 400 according to an embodiment of the present invention is configured to determine the reflection rate by applying a weight according to the period to the quality score for each worker, the work result of an excellent worker who has recently fallen in work accuracy is the correct answer It can be prevented from affecting the decision of the data, and this can improve the reliability of the correct answer data.

The description of the present invention described above is for illustration, and those of ordinary skill in the art to which the present invention pertains can understand that it can be easily modified into other specific forms without changing the technical spirit or essential features of the present invention. will be. Therefore, it should be understood that the embodiments described above are illustrative in all respects and not restrictive. For example, each component described as a single type may be implemented in a dispersed form, and likewise components described as distributed may be implemented in a combined form.

The scope of the present invention is indicated by the following claims rather than the above detailed description, and all changes or modifications derived from the meaning and scope of the claims and their equivalent concepts should be interpreted as being included in the scope of the present invention. do.

100, 200, 300: operator terminal
400: Crowd outsourcing task management device
500: inspector terminal
410: communication department
420: processor
430: memory

Claims (13)

Data receiving step of the communication unit receiving the completion data for the learning data forming task from a plurality of worker terminals through the Internet;
a data reconstruction step in which the processor controlled by the artificial intelligence module reconstructs the completed data into superimposable data; and
Including a tendency determining step in which the processor controlled by the artificial intelligence module analyzes the reconstructed data for each of a plurality of workers and determines at least one of the plurality of workers' propensity, accuracy, and difficulty of the learning data forming task,
A method of inspecting crowd outsourcing work using artificial intelligence. According to claim 1,
The data reconstruction step is
According to the type of the completed data, in the case of one-dimensional data, it is reconstructed into a vector, and in the case of two-dimensional data, it is reconstructed into a matrix having a divided surface obtained by dividing the entire surface,
A method of inspecting crowd outsourcing work using artificial intelligence. 3. The method of claim 2,
The tendency determination step is,
a mask generating step of generating a mask for each of the plurality of worker terminals with a matrix having an element corresponding to the divided surface from the reconstruction data for each of the plurality of worker terminals; and
Including a tendency analysis step of analyzing the mask to determine at least one of the plurality of workers' propensity, accuracy, and difficulty of the learning data formation task,
A method of inspecting crowd outsourcing work using artificial intelligence. 4. The method of claim 3,
The tendency analysis step is,
Determining the difficulty of the learning data forming task for the task from the deviation between the masks for each of the plurality of worker terminals for the task,
A method of inspecting crowd outsourcing work using artificial intelligence. 4. The method of claim 3,
The tendency determination step is,
Further comprising a Trimap generating step of generating a Trimap by merging the masks for each of the plurality of worker terminals between the mask generating step and the propensity analysis step,
A method of inspecting crowd outsourcing work using artificial intelligence. 6. The method of claim 5,
In the Trimap generation step,
The method of merging the masks for each of the plurality of worker terminals is a method of indicating an average of the values of the elements of the masks for each of the plurality of operator terminals in the Trimap,
A method of inspecting crowd outsourcing work using artificial intelligence. 7. The method of claim 6,
The tendency determination step is,
By calculating the similarity between the mask of the target worker terminal and the Trimap among the plurality of worker terminals, determining the accuracy of the target operator terminal,
A method of inspecting crowd outsourcing work using artificial intelligence. 8. The method of claim 7,
The tendency determination step is,
As a method of calculating the similarity, a cosine similarity calculated for each element is used for the entire mask of the target worker terminal, a difference between elements is used as one variable to be substituted for the cosine similarity, and another variable is used. Using the value of the element indicated by the recognition indicated on the mask of the target operator terminal,
The difference for each element is the difference between the value of the element displayed on the Trimap and the value of the element displayed on the mask of the target operator terminal,
A method of inspecting crowd outsourcing work using artificial intelligence. 8. The method of claim 7,
The tendency determination step is,
As a method of calculating the similarity, the sum of differences for each element is specified as the complement of 1 of a value divided by the number of elements whose value is greater than 0 in the Trimap,
The difference for each element is the difference between the value of the element displayed on the Trimap and the value of the element displayed on the mask of the target operator terminal,
A method of inspecting crowd outsourcing work using artificial intelligence. 6. The method of claim 5,
The tendency determination step is,
By calculating the average of elements in the Trimap corresponding to the selected element among the masks of the target worker terminal, determining the work tendency of the target worker terminal,
A method of inspecting crowd outsourcing work using artificial intelligence. 11. The method of claim 10,
The tendency determination step is,
In the Trimap, an Opacity Mask Score (OMS) value obtained by dividing the sum of the values of elements corresponding to the mask of the target operator terminal by the number of selected elements is determined as the work tendency,
A method of inspecting crowd outsourcing work using artificial intelligence. 6. The method of claim 5,
The tendency determination step is,
A method of calculating the task difficulty, wherein the sum of the values of the elements displayed in the Trimap is determined as a value obtained by dividing the value of the elements by the number of the elements greater than 0,
A method of inspecting crowd outsourcing work using artificial intelligence. A communication unit for receiving the work completion data for the work data from a plurality of worker terminals;
a processor configured to perform an operation of inspecting the completed data received from the plurality of worker terminals; and
and a memory in which an artificial intelligence module for controlling the arithmetic operation of the processor is stored,
The artificial intelligence module is
Analyze the two or more of the completed data to determine at least one of the accuracy of the worker, the worker's tendency, and the difficulty of the job, a crowd outsourcing job inspection device using artificial intelligence.

Images