KR102470856B1 - Method and apparatus for cloud outsorcing task checking by using artificial intelligent - Google Patents

Abstract

Crowd outsourcing work inspection method using artificial intelligence according to an embodiment of the present invention, the data receiving step of receiving the completion data for the learning data formation work from a plurality of worker terminals via the communication unit via the Internet, controlled by an artificial intelligence module A data reconstruction step in which a processor reconstructs the completed data into superimposable data, and a processor controlled by the artificial intelligence module analyzes the reconstructed data for each of a plurality of workers to determine the propensity, accuracy, and learning of each of the plurality of workers. and a propensity determining step of determining at least one difficulty of the data forming task.

Description

Crowd outsourcing work inspection method and device using artificial intelligence {METHOD AND APPARATUS FOR CLOUD OUTSORCING TASK CHECKING BY USING ARTIFICIAL INTELLIGENT}

The present invention relates to a method and apparatus for inspecting a crowd outsourcing work using artificial intelligence. More specifically, it relates to a crowd outsourcing work inspection method and apparatus using artificial intelligence that evaluates and selects workers and work results based on outsourcing work results.

Recently, with the development of artificial intelligence (AI) technology, research on artificial intelligence that recognizes specific objects included in digital content such as images, voices, and texts has been actively conducted. In particular, artificial intelligence technology that recognizes a specific object in a digital image created through a digital camera and determines the situation of the area where it was filmed is used in various industries, such as being applied to surveillance cameras and self-driving cars.

In order to recognize a specific object in digital content using artificial intelligence, artificial intelligence must be trained in advance using various learning data including the object to be recognized.

The learning data may be provided by displaying the outline, area, etc. of an object existing in the digital content so that artificial intelligence can distinguish between an object and a non-object in the digital content. However, since there is no data in which objects and non-objects are distinguished, the task of displaying the outline and area of an object in digital content must be performed by a person.

Since artificial intelligence improves recognition accuracy as it learns a lot of diverse training data, it is important to learn as much training data as possible. Recently, a crowd outsourcing method of generating learning data has been used in which a number of workers are recruited through the Internet to easily generate learning data and request a task of displaying an outline or region of an object in digital content.

At this time, each worker may differently determine the area including the object, and a problem may occur that undesirable learning data is generated because the workers perform the work insincerely. In this case, there is an inconvenience in that the inspector has to individually inspect whether the learning data generation task has been completed properly, and there is a problem in that artificial intelligence trained based on undesirable learning data does not properly recognize objects in real situations. may occur.

Therefore, it is required to develop a management technology that can inspect the work results of the crowd outsourcing method so that artificial intelligence learning can be efficiently performed.

Republic of Korea Patent Registration No. 10-1887415 (2017.11.21.)

The present invention provides a crowd outsourcing work inspection apparatus and method using artificial intelligence designed so that high quality outsourced learning data can be selected or inspected and learned.

As a technical means for achieving the above-described technical problem, in the crowd outsourcing work inspection method using artificial intelligence according to an embodiment of the present invention, the communication unit receives completion data for a learning data formation task from a plurality of worker terminals through the Internet. receiving data to be received; A data reconstruction step of reconstructing the completed data into superimposable data by a processor controlled by an artificial intelligence module; and a propensity determination step of determining, by a processor controlled by the artificial intelligence module, at least one of a propensity for each of the plurality of workers, accuracy, and difficulty of the learning data formation task by analyzing the reconstructed data for each of a plurality of workers. .

In the data reconstruction step, according to the type of the complete data, one-dimensional data may be reconstructed into a vector, and two-dimensional data may be reconstructed into a matrix having a divided surface obtained by dividing the entire surface.

The propensity determining step may include a mask generating step of generating a mask for each of the plurality of worker terminals with a matrix having an element corresponding to the divided plane from the reconstructed data for each of the plurality of worker terminals, and analyzing the mask to determine the plurality of workers A propensity analysis step of determining at least one of star propensity, accuracy, and difficulty of the learning data forming task may be performed.

In the propensity analysis step, difficulty of the learning data forming task for the task may be determined from deviations between masks for each of the plurality of worker terminals for the task.

The propensity determining step may further include a trimap generating step of generating a trimap by merging the masks for each of the plurality of worker terminals between the mask generating step and the propensity analysis step.

The method of merging the masks for each worker terminal in the trimap generation step may be a method of displaying an average of values of the elements of the mask for each worker terminal in the trimap.

The propensity determining step may determine the accuracy of the target worker terminal by calculating a similarity between a mask of the target worker terminal and the Trimap among the plurality of worker terminals.

The propensity determining step is a method of calculating the similarity, using a cosine similarity calculated for each element with respect to the entire mask of the target worker terminal, and using a difference between elements as a variable to be substituted for the cosine similarity. And, as another variable, the value of the element indicated by the recognition displayed on the mask of the target worker terminal is used, and the difference for each element is the difference between the value of the element displayed in the Trimap and the value of the element displayed in the mask of the target worker terminal. it can be a car

The propensity determining step is a method of calculating the degree of similarity, wherein the total sum of differences for each element is specified as a 1's complement of a value divided by the number of elements having a value greater than 0 in the Trimap, and the difference for each element is the element indicated in the Trimap. It may be the difference between the value of and the value of the element displayed on the mask of the target worker terminal.

In the determining of the propensity, the work propensity of the target worker terminal may be determined by calculating an average of elements in the trimap corresponding to the selected element among the masks of the target worker terminal.

In the propensity determining step, an opacity mask score (OMS) value obtained by dividing the sum of values of elements corresponding to masks of the target worker terminal in the Trimap by the number of selected elements may be determined as the work propensity.

The tendency determination step is a method of calculating the task difficulty, and may be determined by dividing the sum of the values of the elements indicated by the Trimap by the number of elements whose values are greater than 0.

As a technical means for achieving the above-described technical problem, the crowd outsourcing work inspection apparatus using artificial intelligence according to an embodiment of the present invention includes a communication unit for receiving completion data on work data from a plurality of worker terminals, the A processor configured to perform an operation for inspecting the completed data received from a plurality of worker terminals and a memory storing an artificial intelligence module for controlling an operation of the processor, wherein the artificial intelligence module analyzes two or more of the completed data Thus, it is possible to determine at least one of the operator's accuracy, the operator's inclination, and the difficulty of the task.

According to one embodiment of the present invention, the propensity or accuracy of the operator is calculated from the completion data, so that the quality of the crowd outsourcing work can be more accurately measured.

According to an embodiment of the present invention, the task difficulty of the learning data is calculated and reflected in the quality determination of the completed data, and most of the inspection is performed through the artificial intelligence module, so the workload of experts can be sufficiently reduced, The efficiency of crowd outsourcing work can be improved.

1 is a block diagram showing a crowd outsourcing work inspection apparatus using artificial intelligence according to an embodiment of the present invention.
Figure 2 is a flow chart for explaining a job management method using a crowd outsourcing job inspection device using artificial intelligence according to an embodiment of the present invention.
FIG. 3 is a flowchart illustrating a method of processing data in the crowd outsourcing work inspection apparatus using artificial intelligence of FIG. 2 .
Figure 4 is a flow chart for explaining the process of generating and distributing work data in the crowd outsourcing work inspection method using artificial intelligence of FIGS. 2 and 3.
5 is a flowchart for explaining a step of determining the propensity of each worker according to an embodiment of the present invention.
6 to 11 are exemplary diagrams for explaining an example of determining the accuracy, propensity, and task difficulty of each worker in the crowd outsourcing work inspection method using artificial intelligence of FIG. 2 .
12 is an exemplary view for explaining an example of determining a reflection rate for each worker in the crowd outsourcing work inspection method using artificial intelligence of FIG. 2 .

Hereinafter, embodiments of the present invention will be described in detail so that those skilled in the art can easily practice the present invention with reference to the accompanying drawings. However, the present invention may be embodied in many different forms and is not limited to the embodiments described herein. And in order to clearly explain the present invention in the drawings, parts irrelevant to the description are omitted, and similar reference numerals are attached to similar parts throughout the specification.

Various embodiments of the present document may be implemented as software (eg, a program) including instructions stored in a storage medium readable by a machine (eg, a computer). A device is a device capable of calling a stored command from a storage medium and operating according to the called command, and may include an electronic device (eg, a server) according to the disclosed embodiments. An instruction may include code generated or executed by a compiler or interpreter. The device-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-temporary' only means that the storage medium does not contain a signal and is tangible, but does not distinguish whether data is stored semi-permanently or temporarily in the storage medium.

According to one embodiment, the method according to various embodiments disclosed in this document may be included and provided in a computer program product. Computer program products may be traded between sellers and buyers as commodities. The computer program product may be distributed in the form of a device-readable storage medium (eg compact disc read only memory (CD-ROM)) or online through an application store (eg Play Store™). In the case of online distribution, at least part of the computer program product may be temporarily stored or temporarily created in a storage medium such as a memory of a manufacturer's server, an application store server, or an intermediary server.

Each component (eg, module or program) according to various embodiments may be composed of a single object or a plurality of entities, and some of the sub-components may be omitted, or other sub-components may be various. It may be further included in the embodiment. Alternatively or additionally, some components (eg, modules or programs) may be integrated into one entity and perform the same or similar functions performed by each corresponding component prior to integration. According to various embodiments, operations performed by modules, programs, or other components may be executed sequentially, in parallel, repetitively, or heuristically, or at least some operations may be executed in a different order, may be omitted, or other operations may be added. can

In this specification, “transmitting, receiving, collecting, extracting, uploading, or updating information or data” means directly sending, receiving, collecting, extracting, uploading, or updating information or data, or indirectly through another relay server; It includes the meaning of receiving, collecting, extracting, uploading or updating.

Hereinafter, the present invention will be described in detail with reference to the accompanying drawings.

The present invention is a crowd outsourcing job inspection method and apparatus, and can be utilized in a crowd outsourcing job management device using artificial intelligence. In this specification, while schematically describing the entire crowd outsourcing task management apparatus and method in order to show the actual application of the present invention, the inspection process will be described in detail.

The crowd outsourcing work management method consists of generating and distributing crowd outsourcing work data to workers, collecting the work results, inspecting the collected works, and learning the inspected results to artificial intelligence according to their values. can be divided and understood. Crowd outsourcing work inspection method corresponds to the second step among them. Since the crowd outsourcing job inspection device uses the same processor as the crowd outsourcing job management device, they may refer to substantially the same entity. In this specification, collectively referred to as a crowd outsourcing work inspection device, it is described that the device can additionally perform a crowd outsourcing work management function.

1 is a block diagram showing a crowd outsourcing work inspection apparatus using artificial intelligence according to an embodiment of the present invention.

Referring to FIG. 1 , an apparatus 400 for inspecting a crowd outsourcing job according to an embodiment of the present invention may refer to a device of a system that manages data preprocessing jobs that generate learning data for learning an artificial intelligence module. Specifically, when the crowd outsourcing job inspection device 400 receives completion data for a job from a plurality of worker terminals 100, 200, and 300, a portion of the completion data is inspected to generate desirable learning data, and the inspection It may be a device that manages the learning data generation task so that the remaining completed data is transmitted to the inspector terminal 500 so that the professional inspector performs the inspection. However, since the crowd outsourcing job inspection device 400 can perform inspection only on completed job data, it is not necessary to have this data pre-processing job and data transmission function.

The crowd outsourced job review device 400 may be any type or combination of types of computing devices, such as network servers, web servers, file servers, supercomputers, desktop computers, and the like.

The crowd outsourcing work inspection device 400 includes a communication unit 410, a processor 420, and a memory 430.

The communication unit 410 may transmit work data to the plurality of worker terminals 100, 200, and 300, and when the work for the work data is completed, the completion data from the plurality of worker terminals 100, 200, and 300. It is configured to receive. At this time, the work data may be sequentially transmitted by grouping parts of the plurality of worker terminals 100, 200, and 300, and completion data may also be received sequentially according to the order of completion. Also, transmission and reception can be performed simultaneously. However, when the crowd outsourcing job inspection device 400 performs only inspection work, it does not need to have a function of transmitting data to a plurality of worker terminals 100, 200, and 300.

The communication unit 410 is connected to the Internet and may be composed of a network adapter capable of transmitting and receiving data to and from the operator terminals 100, 200, 300 and the inspector terminal 500 through the Internet. The communication unit 410 may be configured to be connected to the Internet through a wired (LAN or WAN) or wireless (Bluetooth, WiFi, Infrared data association (IrDA) method.

The processor 420 may generate work data to be transmitted to the plurality of worker terminals 100, 200, and 300, and performs an operation for inspecting the completed data received from the plurality of worker terminals 100, 200, and 300. configured to perform

The processor 420 is a dedicated processor (eg, an embedded processor) for performing the above-described arithmetic operations, or a general-purpose processor capable of performing corresponding operations by executing one or more software programs stored in a memory device. (eg, CPU, graphical processing units (GPUs), single-core processors, multi-core processors, application specific integrated circuits (ASICs), etc.), or communication processors (CPs). .

The memory 430 is a device in which an artificial intelligence module that controls the operation of the processor 420 is stored, and is, for example, a solid state drive (SSD), a hard disk drive, or a card-type memory (SD or XD). memory, etc.), RAM (random access memory, RAM), SRAM (static random access memory), ROM (read-only memory, ROM), EEPROM (electrically erasable programmable read-only memory), PROM (programmable read-only memory) , a storage medium of at least one type among a magnetic memory, a magnetic disk, and an optical disk, and various storage media capable of storing identifiable physical location information therefor. 1, the memory 430 is shown as being located within the crowd outsourcing job review device 400, but in some embodiments, the memory 430 is a separate device physically separate from the crowd outsourced job review device 400. may consist of In this case, the memory 430 and the crowd outsourcing work inspection device 400 may be connected through a network.

The artificial intelligence module is a set of instructions for controlling the operation of the crowd outsourcing job inspection device 400, and may include computer- or machine-executable instructions described in any appropriate programming language to perform the above-described various functions. have.

When the communication unit receives completion data from a plurality of worker terminals 100, 200, and 300 through the Internet, a processor controlled by an artificial intelligence module reconstructs the completion data into data that can be overlapped. .

In addition, the processor controlled by the artificial intelligence module determines the accuracy, propensity, and difficulty of the task for each of the plurality of workers, determines the reflection rate for each of the plurality of workers based on the accuracy and difficulty of the task for each of the plurality of workers, and determines the reflection rate according to the reflection rate. Based on the plurality of workers, it is configured to determine the inspection criterion data for inspection of the completed data. Operations of the artificial intelligence module will be described in more detail with reference to FIGS. 2 to 12 .

Meanwhile, the worker terminals 100 , 200 , and 300 are terminals used by workers, and may be implemented as a computing device for generating completion data by processing work data. In addition, the inspector terminal 500 is a terminal used by the inspector and may be implemented as a computing device for performing inspection on completed data requiring professional inspection among complete data. For example, the operator terminals 100, 200, and 300 and the inspector terminal 500 may include a smartphone, a tablet PC, a mobile phone, a video phone, an e-book reader, a desktop PC, a laptop PC, a netbook computer, a workstation, and a server. , a PDA, a PMP, an MP3 player, a mobile medical device, a camera, or a wearable device.

Figure 2 is a flow chart for explaining a job management method using a crowd outsourcing job inspection apparatus using artificial intelligence according to an embodiment of the present invention, Figure 3 is a data from the crowd outsourcing job inspection apparatus using artificial intelligence of FIG. It is a flow chart to explain the processing method.

Referring to FIGS. 2 and 3 , first, the crowd outsourcing inspection apparatus may generate work data by setting a work target area of work target data (S210).

“Work target data” refers to original data of work data to be transmitted to the worker terminal 100, and the work target data may be composed of various types of data such as digital images, texts, and sounds. The crowd outsourcing job inspection device 400 generates job data by pre-processing the job target data using an artificial intelligence module.

The crowd outsourcing job distribution method using artificial intelligence includes a process of generating job data by setting a work target area of work target data requiring work by a processor controlled by an artificial intelligence module (S211), and generating work data by a communication unit (S211). A transmission step (S212) of transmitting to some worker terminals among a plurality of worker terminals connected to the Internet, and when the completion data for which the work for the work data is completed is received from the plurality of worker terminals by the communication unit, the completion data and a receiving and learning step (S213) of learning the artificial intelligence module based on, and the work data generation step, transmission step, and receiving and learning step are performed when transmitting to a predetermined number of terminals among a plurality of worker terminals or the above This is repeated until the work on the target data is completed. A specific example corresponding to this will be described later with reference to FIG. 4 .

When completion data is received, completion data can be inspected. The step of inspecting the completed data (S30) is the step of reconstructing the completed data to reconstruct the work data into data that can be superimposed according to the type of work data (S34) and analyzing the completed data to determine the accuracy of the operator and the work inclination of the operator , and a step of determining the propensity of each worker to evaluate the difficulty of work (S220). Specific examples corresponding to this will be described later with reference to FIGS. 5 to 11 .

A step ( S55 ) of forming work inspection standard data by evaluating the ability of each worker based on the propensity evaluation data and completion data for each worker may be additionally performed. The step of forming workpiece inspection standard data (S55) is a step of generating inspection criteria to be used for inspection of a workpiece after the workpiece inspection step. Since the accuracy of the work was calculated in S30, it is possible to directly generate the inspection standard data by reflecting it as a weight (S30). It is preferable to additionally perform a filtering step (S230) for excluding work and a step (S240) for determining the reflection rate for each worker to evaluate and utilize the ability of each worker in advance to generate inspection standard data according to the weight (S250). ( S55) A specific example of this step will be described later with reference to FIG. 12 .

4 is a flowchart for explaining a process of generating and distributing work data corresponding to S210 of FIGS. 2 and 3 .

Referring to FIG. 4, the crowd outsourcing task distribution method using artificial intelligence includes a task data generation step in which a processor controlled by an artificial intelligence module sets a work target area of work target data requiring work and generates work data, and a communication unit A transmission step of transmitting the work data to some worker terminals among a plurality of worker terminals connected to the Internet, and when the communication unit receives the completion data for which the work on the work data is completed from the plurality of worker terminals, the completion data When the work data generation and distribution step including the receiving and learning step of learning the artificial intelligence module based on is transmitted to a predetermined number of terminals among the plurality of worker terminals or when the work for the work target data is completed repeat until Hereinafter, each step will be described in detail by way of example.

The artificial intelligence module of the crowd outsourcing job inspection device sets a work area for the first work target data and generates first work data (S214).

As mentioned above, work target data is digital content that includes objects to be learned by an artificial intelligence module, and may be composed of various types of data such as images, texts, and sounds.

For convenience of explanation, a case in which work target data is composed of image data will be described as an example.

The artificial intelligence module first randomly selects first work target data from among all work target data and sets the work target area where the object to be learned by the artificial intelligence module is located to be distinguished from other areas in the first work target data. 1 Create work target data. For example, if the object to be learned is "dog", the artificial intelligence module may generate first task data by displaying the boundary of the area where "dog" is located in the first task data including "dog". can

At this time, a method for recognizing the boundary of the area where the “puppy” is located by the artificial intelligence module can use Trimap, which will be described later. In other words, as data for learning the artificial intelligence module, a region used for learning is generated as a mask for each worker terminal from completion data for each worker terminal, and a trimap is generated by merging the masks for each worker terminal. In the area included in the trimap, a portion marked as an area used for learning with a higher frequency than a preset number may be designated as a work target area.

Thereafter, the generated first work data is transmitted to some worker terminals among a plurality of worker terminals connected to the Internet. At this time, since some worker terminals are assigned work data first, the initial learning data of the artificial intelligence module is generated. Since the artificial intelligence module is used to produce all work data in the future, the initial performance of the artificial intelligence module is important, and the quality of the initial learning data that determines the initial performance of the artificial intelligence module is also important. Therefore, in order to improve the quality of the initial learning data, it is advantageous that some worker terminals that first receive the first work data are worker terminals having excellent capabilities. This excellent worker terminal may be determined based on any criteria such as the worker's career and field of study. However, since the crowd outsourcing inspection management device using artificial intelligence evaluates the accuracy of each terminal for each task, based on the accuracy of the operator evaluated in the process of inspecting the work data, the crowd outsourcing task inspection device using artificial intelligence evaluates the accuracy of a certain level or higher. A worker may be designated as an excellent worker terminal.

Thereafter, the crowd outsourcing job inspection apparatus receives first completion data for the first job data from the worker terminal 100 (S215).

The first completion data refers to data in which the operator has completed a task for the first task data through the operator terminal 100 . For example, if the object to be learned by the artificial intelligence module is "dog", the task for the first task data may be a task of displaying the outline of "dog". The operator may generate first completion data by precisely editing the boundary line in the first work data in which the boundary line of the area where the "dog" is located is schematically displayed to fit the contour of the "dog".

When the first completion data is received from the worker terminal, the crowd outsourcing job inspection apparatus learns the artificial intelligence module based on the first completion data (S216).

In the first work data, the boundary line for “dog” is roughly indicated, but in the first work data performed by a human, the boundary line for “dog” may be more precisely indicated.

When a plurality of first completion data is received, the accuracy of a specific worker terminal among them is calculated in the process of learning the first completion data to the artificial intelligence module. The process of inspecting and evaluating the completed data and learning the artificial intelligence module will be described later.

The artificial intelligence module is learned based on the first completion data, and when using the artificial intelligence module for which learning has been completed, a work area for “dog” can be set more accurately and precisely for other images including “dog”. can

Thereafter, the crowd outsourcing job inspection apparatus sets a work area for the second work target data based on the learned artificial intelligence module to generate second work data (S217).

The second work target data may be work target data selected from among work target data other than the first work target data. For example, if the work target data is composed of 100 image data including “dog”, the first work target data may be composed of 10 images including “dog”. The artificial intelligence module generates first work data by displaying an area including “dog” in the 10 pieces of first work object data, and first completion data in which the work for the 10 pieces of first work object data is completed from the worker terminal. When is received, it may be learned based on the first completion data. The artificial intelligence module that has completed learning can more precisely display the area including the “dog” for the second task target data consisting of 10 other images, and the second task data is generated through the artificial intelligence module. Thereafter, when the second work data is transmitted to a plurality of worker terminals and the second completion data in which the work for the second work data is completed is received from the worker terminal, the artificial intelligence module is learned based on the second completion data. In the same way, third work data is generated by setting an area including “dog” for the third work object data composed of another 10 images through an artificial intelligence module. In this case, the learned artificial intelligence module may display an area including “dog” with respect to the third work object data more accurately than when processing the second work object data. In the same way, the AI module is periodically learned until the work on the work target data is completed, and the precision of the work data generated through the AI module can be improved as the learning continues.

In other words, the steps of the artificial intelligence module generating/transmitting work data from the work target data and the step of receiving completion data and learning the artificial intelligence module may be repeatedly performed. This step may be repeated until all tasks for the work target data are distributed, and in the case of duplicate work for the same work target data, it will be repeated until work data is generated and sent to a preset number of worker terminals. may be

In some embodiments, learning of the artificial intelligence module may be performed based on first correct answer data that has been verified, not based on first completion data. That is, the artificial intelligence module may be learned based on the first correct answer data that has been confirmed by completing the inspection of the first completion data prior to processing the second work target data, and through the learned artificial intelligence module, the second The work target data may be processed to generate second work data. In this case, the first correct answer data may correspond to data that is finally inferred as a correct answer after verification is completed according to a verification method for complete data described later.

Although the above content is related to the process of generating and distributing work data and is not directly related to the present invention, it has been described for reference because it is a function that can be performed together with the crowd outsourcing work inspection device using artificial intelligence of the present invention.

Referring back to FIG. 3 , the crowd outsourcing work distribution device may inspect the completed data when the completed data for the job data is received from the worker terminal (S30).

The complete data verification step (S30) will be described with reference to FIG. 5 showing detailed steps of the complete data verification step (S30).

First, complete data is reconstructed into data that can be superimposed (S34). For example, sound or text is one-dimensional data, and images are two-dimensional data. It is preferable that each data can collect the results of work by a plurality of workers. Accordingly, one-dimensional data can be reconstructed into a vector and two-dimensional data can be reconstructed into a matrix so that an amount of the same size can be overlapped.

Particularly, in more detail about 2D data, in the case of image data, it may be a matrix in which an image is divided into m X n planes. In addition, depending on the type of 2D data, it can be converted to a 2D matrix of m X n (where m and n are integers greater than or equal to 1) corresponding 1:1 to the corresponding surface. The value of this matrix may have 1 when the corresponding face is included to distinguish whether or not a recognition target is included, and 0 when not included.

After reconfiguration, at least one of accuracy, propensity, and difficulty of work for each of a plurality of workers is determined (S220).

The crowd outsourcing job inspection apparatus 400 according to an embodiment of the present invention uses an artificial intelligence module to determine at least one of job accuracy, propensity, and difficulty for a learning data formation job for a plurality of workers, and Based on this, there is a feature of inspecting the completion data of workers for other tasks. Hereinafter, with reference to FIGS. 6 to 11 , a method of determining task accuracy, propensity, and difficulty for each of a plurality of workers will be described in detail.

Referring to FIG. 6, a step of generating a plurality of masks for each operator (S223) will be described.

The plurality of worker-specific masks OM1 , OM2 , and OM3 may be masks in which an area for a specific object is marked on a matrix in a work target. For example, when a work target corresponds to a digital image and is a task of defining a specific object in the digital image, a plurality of workers define the specific object by displaying the outline of the specific object in the digital image according to their own standards. can do. In this case, the completion data received from the worker terminal 100 may be data for a digital image in which the outline of a specific object is displayed. As described above, if the reconstructed data is a matrix in which each element is corresponded 1: 1 to a face divided into m × n faces, a plurality of operator-specific masks (OM1, OM2, OM3) are the shape of a specific object in the completed data It may be a matrix in which an element value corresponding to a divided surface having a region defined as is greater than or equal to a predetermined ratio is 1, and an element value corresponding to a divided surface having a predetermined ratio or less is 0.

Thereafter, the artificial intelligence module merges a plurality of masks (OM1, OM2, and OM3) for each operator to generate a Trimap (S226).

Since a plurality of workers display the outline of a specific object in a digital image according to their respective standards, the resulting data and masks OM1, OM2, and OM3 of each worker have different shapes. The artificial intelligence module creates a Trimap(TM) by merging multiple worker-specific masks (OM1, OM2, OM3). Specifically, one merged mask is created by adding element values of corresponding elements of the plurality of masks OM1 , OM2 , and OM3 for each operator. In this case, the element value of the element of the surface where the display area of the first operator's mask OM1, the display area of the second operator's mask OM2, and the display area of the third operator's mask OM3 all overlap is It may be expressed as a normalized sum of element values of the mask OM1 of the first operator, element values of the mask OM2 of the second operator, and element values of the mask OM3 of the third operator. For example, if the element value of the mask display area is defined as 1 and the element value of the non-display area is defined as 0, the mask OM1 of the first operator, the mask OM2 of the second operator, and the mask OM2 of the third operator The sum of the element values is defined as 3 for the element corresponding to the surface where all the display areas of the mask OM3 overlap. have. Therefore, the sum of the element values can be normalized to 3* 1/3 = 1. Each element value of the mask integrated in the same way can be calculated as a normalized sum of corresponding element values of the first operator's mask OM1, the second operator's mask OM2, and the third operator's mask OM3. have. Therefore, Trimap can be expressed as an average matrix obtained by dividing the sum of each mask represented by the matrix by the number of masks.

In FIG. 6, an example in which Trimap is formed in the form of a two-dimensional matrix is shown, but Trimap may be formed in the form of a one-dimensional vector according to the form of complete data. For example, if the completion data corresponds to voice data, a mask for each operator may be generated for a region where a voice of a specific object is displayed, and a trimap in the form of a one-dimensional vector may be generated by merging the masks for each operator.

Meanwhile, the aforementioned Trimap is generated based on completion data for each of a plurality of workers generated for the same job data. That is, in the case of a task of defining a specific object within the same digital image, a trimap may be generated by receiving completion data for each operator for the same digital image and overlapping a plurality of masks for each operator.

Thereafter, the artificial intelligence module determines the work inclination, work accuracy, and difficulty of the work of the target worker based on the mask and Trimap of the target worker among the plurality of workers (S229).

Specifically, the work inclination of the target worker is determined based on the following [Equation 1].

Here, an Opacity Mask Score (OMS) is an indicator for identifying work inclinations for each worker, and may be an index indicating the degree of agreement between a specific work result of a worker and other work results of other workers. In [Equation 1], TM (t) is the element value of Trimap for a specific task (ie, completed data) (t), and BM (t, i) is a specific task (t) of the target worker (i) Indicates the element value of the mask for That is, OMS means the average element value of Trimap of the selected area of the mask for the specific job t of the target worker i.

In the case of a worker with a high OMS, most of the elements with high element values were selected in Trimap, so it can be seen that the region selected by the worker overlaps with the region selected by other workers. Accordingly, it can be seen that a worker with a high OMS tends to under-display the outline of a specific object compared to other workers.

On the other hand, in the case of a worker with a low OMS, since many elements having low element values are selected in Trimap, the region selected by the corresponding worker may correspond to regions not selected by other workers. Accordingly, it can be seen that a worker with a low OMS tends to display the outline of a specific object excessively compared to other workers.

7(a) shows a mask 711 of a worker with low OMS and a mask 713 of Trimap, and (b) shows a mask 715 of a worker with high OMS and a mask 713 of Trimap. For convenience of explanation, in FIG. 5 , only elements corresponding to the boundaries of each mask are indicated by solid lines.

As shown in (a) of FIG. 7, it can be seen that the mask 711 of the worker with low OMS is located inside the Trimap mask 713, and through this, the worker with low OMS defines the specific object to another It can be seen that they are under-represented compared to the workers.

On the other hand, as shown in (b) of FIG. 7, it can be seen that the mask 715 of the worker with high OMS is located outside the Trimap mask 713, and through this, the worker with high OMS defines the definition of a specific object. It can be seen that they display excessively compared to other workers.

In addition, the artificial intelligence module may determine the accuracy of a plurality of workers' work based on the degree of similarity with work results of other workers. Since the work results of other workers are integrated into Trimap, the actual work accuracy of multiple workers is evaluated based on the degree of similarity with Trimap.

Specifically, the work accuracy for each of the plurality of workers is measured based on the following [Equation 2].

Here, TWS (Task-Worker Similarity) is an index representing the task accuracy of a worker, and means a cosine similarity between the task results of a specific worker and the task results of the rest of the workers. In [Equation 2], V _t means a 1-dimensional vector of Trimap of a specific task (t), and V _t,i is a 1-dimensional vector of a worker mask of a specific task (t) of a specific worker (i) it means.

In other words, the similarity for calculating the accuracy uses the cosine similarity calculated for each element with respect to the entire mask of the target worker terminal, and uses the difference for each element as a variable to be substituted for the cosine similarity, As another variable, the value of the element indicated by the recognition displayed on the mask of the target worker terminal is used.

8 is a diagram for explaining an example of a TWS calculation process. Referring to FIG. 8, Trimap (TM) in the form of a two-dimensional matrix for a specific task (t) can be 1-dimensional vectorized (V _t ) by applying a Flatten function, and a specific operator (i ) can also be one-dimensional vectorized (V _t,i ) by applying the Flatten function. Then, by calculating V _t -V _t,i , a one-dimensional vector for the work results of the workers other than the specific worker (i) is calculated, and the cosine between V _t -V _t,i and V _t,i By calculating the similarity, the cosine similarity between the specific worker (i) and the work results of the other workers except the specific worker (i) is calculated.

9 is a diagram showing TWS calculation results according to the work result of a specific operator. Referring to FIG. 9, as a result of calculating the TWS based on [Equation 2] after converting the mask of a specific worker into a 1D vector and converting the Trimap into a 1D vector, the TWS of the specific worker was calculated as 0.936. Through this, it can be seen that the work result 911 of a specific worker has a similarity of about 0.936 compared to the result 913 of Trimap. When the work result 911 of a specific worker has a lower similarity to the result 913 of Trimap, the TWS value may be lowered.

Cosine similarity is useful for comparing similarities in order, but has a disadvantage of not intuitively representing the quantitative difference.

In some embodiments, in order to improve the intuitiveness of the operator's TWS, the artificial intelligence module uses the difference per element instead of the cosine similarity as it is for the real work area excluding elements that no one has marked as the recognition area, which can be called imaginary numbers of the work. indicators can be used.

In this case, TMS (Trimap score) based on the following [Equation 3] may be used instead of TWS.

Here, TMS(t,i) is the Trimap score of a specific worker (i) for a specific task (t), minimizing the variance of the part that matches the work results of other workers among the specific worker (i)'s work results Therefore, it can be described as an index that maximizes the accuracy index for the work result of a specific worker (i). In [Equation 3], TM(t) means the element value of Trimap for a specific job t, and BM(t,i) is a job mask of a specific worker i for a specific job t represents the elemental value of On the other hand, n means the number of elements whose element value exceeds 0 in Trimap.

10 is a diagram showing a calculation process of TMS(t,i) of a specific worker i for a specific job t. Referring to FIG. 10, when the task result for a specific task (t) is expressed as a mask consisting of 3*3 = 9 elements, the sum of │TM(t)-BM(t-i)│ is shown in FIG. As such, it is calculated as 2, and when TMS(t,i) is calculated according to Equation 3 above, TMS(t,i) is calculated as 0.77.

On the other hand, the artificial intelligence module determines the task difficulty for a specific task from the deviation of a plurality of workers' masks for the specific task.

High task difficulty means that there is a lot of difficulty in judgment, and a lot of difficulty in judgment means that various judgments can appear for the same task. Therefore, the fact that the variation of the mask for each worker is large means that the difficulty of the work is high. The deviation between the masks of a plurality of workers may be calculated by calculating the area of each mask to obtain the deviation, but it is preferable to estimate the following method in order to calculate a more accurate value.

If the judgment value (1 or 0) appears evenly for a specific element in a plurality of masks for each worker, it can be determined that the determination of the corresponding element is difficult, and if the number of such elements is large, the work difficulty can be considered high. . If the value of each element is judged to be various, the average value of the element will converge to 0.5. Since the average value for each element has already been calculated in TM in the masks for each operator, it can be calculated using this.

Specifically, the task difficulty for a specific task is determined by dividing the sum of element values displayed in Trimap by the number of elements whose value is greater than 0. It can be determined based on [Equation 4], which is expressed as an equation.

Here, ICS(t) is an index representing task difficulty for a specific task t, and can be described as an indicator of how clear and unambiguous the task is. In [Equation 4], TM(t) denotes an element value of Trimap for a specific operation t, and n denotes the number of effective elements whose element values exceed 0. Since the area (element value of 0) that all workers can distinguish is an imaginary number in the amount of work, excluding it is exemplified. It is free to be specific. The clearer the task, the more similar the work results of the workers are, and since the overlapping area of the task masks is wide, ICS(t) in [Equation 4] can be calculated larger.

It may be replaced by simply obtaining a normal distribution for each element of each trimap and calculating based on the frequency or amount of an element whose value is close to the middle (0.5 in this embodiment). In this case, unlike the equation, the ICS value has a value proportional to the degree of difficulty.

11 are diagrams showing ICS(t) calculation results for actual works. As shown in FIG. 11, it can be seen that the ICS is measured high for an image in which the shape of an object is simple and can be relatively clearly defined.

After the step of inspecting the completed data (S30), workpiece inspection reference data may be formed (S55) by utilizing the accuracy and work difficulty calculated as a result of the inspection.

Referring back to FIGS. 2 and 3 , the crowd outsourcing job inspection apparatus determines whether a job whose reflection rate can be calculated for the target jobs of the completed data based on the accuracy and task difficulty of a plurality of workers (S230), and the reflection rate cannot be calculated. Filter if it is a task.

Here, the voting power can be described as the weight or contribution rate of workers who contributed to the trimap formation of a specific task. Crowd outsourcing job inspection apparatus according to an embodiment of the present invention provides a high reflection rate to a worker who has accurately performed a specific job in the process of determining the inspection criterion data (ie, correct answer data) necessary for inspection of a specific job. It is configured to improve the reliability of the reference data.

On the other hand, when a specific task itself is ambiguous and the level of difficulty is excessively high, there may not be a worker who has sufficiently accurately performed the task. In addition, when a plurality of workers who performed a specific task did not faithfully perform the task, only the completion data of defective workers for that task exists, so when a reflection rate is given to this and the inspection standard data is selected, reliability is improved. Low acceptance criteria data may be selected.

Therefore, the crowd outsourcing job inspection apparatus according to an embodiment of the present invention may first filter whether or not the corresponding job is suitable for calculating the reflection ratio before calculating the reflection ratio for each worker through the artificial intelligence module.

For example, the artificial intelligence module examines whether the maximum value of TWS of completion data received from a plurality of workers exceeds the accuracy threshold and the ICS exceeds the difficulty threshold. The TWS may be calculated based on [Equation 2], and it may be determined whether the highest maximum value among the TWS values for the task to be determined exceeds the accuracy threshold. As mentioned above, since TWS is an indicator of the worker's work accuracy, a high maximum value of TWS may mean that excellent workers who performed the work with high accuracy are included among the workers who performed this work. .

In addition, ICS of complete data can be calculated based on [Equation 4] above. As mentioned above, since the higher the ICS, the lower the difficulty of the task, so that the ICS for a specific task exceeds the difficulty threshold may mean that the specific task has an appropriate difficulty level.

If the maximum TWS of the completion data for a specific task is less than the accuracy threshold and the ICS for the specific task is less than the difficulty threshold, the difficulty of the task itself is too high and there is no completion data of excellent workers with high accuracy. Therefore, it can be judged as an ambiguous task in which calculation of the reflection rate is impossible. In this case, the crowd outsourcing job inspection apparatus 400 transmits completion data for the ambiguous job to the inspector terminal 500, and expert inspection for the ambiguous job is performed through the inspector terminal 500 (S255).

On the other hand, if the maximum TWS of the completion data for a specific task exceeds the accuracy threshold and the ICS for the specific task exceeds the difficulty threshold, the task includes the completion data of excellent workers, and the difficulty of the task itself is Since it is at an appropriate level, it can be seen that it has the qualifications to select excellent workers.

Accordingly, the crowd outsourcing job inspection apparatus determines that the job is eligible for reflection rate calculation, and determines the reflection rate for each worker based on the accuracy and task difficulty of each worker (S240).

Specifically, the crowd outsourcing job inspection apparatus 400 according to an embodiment of the present invention, prior to calculating the reflection rate for each worker through an artificial intelligence module, complete data of excellent workers whose accuracy is equal to or greater than a preset threshold value among a plurality of workers extract

Specifically, the artificial intelligence module calculates TWS for each worker with respect to tasks that satisfy the above conditions, and determines whether the TWS exceeds a first threshold value. The higher the TWS, the higher the likelihood that the worker's work result will be closer to the correct answer data, since it can be seen that the work result of the worker has a higher degree of concordance with the work result of other workers. Accordingly, the artificial intelligence module filters only completion data of workers whose work accuracy is greater than or equal to a preset first threshold. Completion data of workers less than the first threshold corresponds to work results with low accuracy. On the other hand, as can be seen from the TWS definition of [Equation 2], the TWS for each plurality of workers is defined as the cosine similarity of Trimap generated by overlapping the completion data of a specific worker and Trimap, that is, the completion data of a plurality of workers, If there are bad workers whose accuracy is too low, the accuracy of Trimap itself may be lowered, and it can be seen that it has the effect of lowering the TWS for well-intentioned workers for the work.

Therefore, the artificial intelligence module of the crowd outsourcing job inspection apparatus according to an embodiment of the present invention removes completed data having a TWS less than the first threshold value among the completed data of a plurality of workers, and complete data having a TWS greater than or equal to the first threshold value. Trimap is generated again by selecting only the selected ones, and based on the regenerated Trimap, TWS for a plurality of workers for a specific job is calculated again.

Thereafter, the artificial intelligence module examines whether the TWS calculated again based on the completion data of the selected plurality of workers exceeds the second threshold, and determines the completion data of specific workers having TWS exceeding the second threshold. It is selected as the completion data of excellent workers.

The accuracy threshold, difficulty threshold, first threshold, and second threshold may be determined in various ways according to required accuracy of learning data. For example, in the case of artificial intelligence applied to self-driving cars, since it is necessary to accurately recognize objects, learning based on accurate learning data may be required. In this case, by setting the above-mentioned threshold value high, it is possible to improve the accuracy of the learning data.

Thereafter, the artificial intelligence module calculates the similarity between the correct answer data for the quality check task and the completion data of the excellent worker for the quality check task to determine the intersection over union (IOU) of the excellent worker.

Specifically, the artificial intelligence module may select completion data for a quality check task among tasks performed by excellent workers. Here, the quality check task refers to a task in which correct answer data (Ground Truth) is secured in advance to determine the reflection rate for each excellent worker, and the IOU for each excellent worker is calculated for the quality check task for which correct answer data exists The star reflection rate can be determined.

IOU means a value obtained by dividing the area of the intersection area of the two areas A and B by the value of the sum area, and means a Jaccard index defined by [Equation 5] below.

In the case of excellent workers, since the work accuracy is high, the IOU between the completion data and the correct answer data for the quality check task may appear high.

Then, the artificial intelligence module calculates the quality score of the excellent worker from the ratio of the IOU of the excellent worker to the task difficulty for the quality check task.

Specifically, the quality score may be calculated through [Equation 6] below.

Here, QScore means the quality score of a particular good worker for a quality check task (QcT). Meanwhile, in [Equation 6], QcT _w,i denotes completion data for the latest i-th quality check task QcT of a particular excellent worker w. In addition, QcT _gt,i means correct answer data for the latest i-th quality check task (QcT).

As can be seen from [Equation 6] above, the artificial intelligence module can give a higher quality score to an excellent worker (w) having excellent accuracy for a difficult task (low ICS).

Then, the artificial intelligence module determines the reflection rate of excellent workers based on the quality score.

Specifically, the artificial intelligence module determines the reflection rate of a specific excellent worker by applying a weight according to the period to the quality score of the specific excellent worker. For example, as shown in [Equation 7], by applying a weight (DecayWeight) for a period to the quality score (QScore) of a specific excellent worker and dividing the number of quality check tasks (QcT) to be applied to the reflection rate calculation , the influence score (IP) of a specific excellent worker can be calculated.

Here, IP refers to an index indicating the degree of influence of a specific excellent worker that is reflected in determining inspection criterion data (ie, correct answer data) for a specific task to be inspected. QcT Range means the number of quality check tasks (QcT) to be applied to IP calculation, and the higher the QcT Range, the higher the reliability of IP. DecayWeight is a weight for a period, and may be a factor having a larger value as it is closer to the current point in time, and is defined by Equation 8 below.

Here, Weight means weight, and t means QcT Range. i means an integer from 1 to t-1. Meanwhile, a specific value of the weight may be variously determined according to task difficulty, type, and the like.

For example, the process of calculating the influence score (IP) of a particular excellent worker (i) may be described as follows. If the impact score (IP) is calculated based on the five quality check tasks (QcT), first, the artificial intelligence module determines the five quality check tasks (QcT) based on the completion data of a particular excellent worker (i). After calculating the IOU and the task difficulty (ICS) of the quality check task (QcT), it is possible to calculate the quality score (QScore) for three excellent workers based on [Equation 6]. Specifically, the IOU for the five quality check tasks (QcT) may be calculated based on [Equation 5] above, and the ICS may be calculated based on [Equation 4] above. An example of calculating IOU and ICS for 5 quality check tasks (QcT) of a certain excellent worker (i) is shown in [Table 1].

division IOU ICS QcT1 0.9 0.8 QcT2 0.8 0.9 QcT3 0.8 0.7 QcT4 0.9 0.9 QcT5 0.7 0.8

In this case, according to [Equation 6] above, the quality score (Qscore) of the specific excellent worker (i) is calculated as follows. QScore = [1.12, 0.88, 1.14, 1.00, 0.87]

In addition, when the weight is 0.9, the weight (DecayWeight) according to the period of the five quality check tasks (QcT) can be calculated as follows according to [Equation 8].

DecayWeight(0.9, 5)= 0.9 ⁱ = [1.0, 0.9, 0.81, 0.73, 0.65]

Therefore, the influence score (IP) for a particular excellent worker (i) can be calculated as follows.

Then, the artificial intelligence module normalizes the influence points (IP) so that the sum of the influence points (IP) of all excellent workers is 1 to calculate the voting power (VP).

For example, if there are 3 excellent workers and the influence scores (IP) of the 3 excellent workers are as shown in [Table 2], the sum of the influence scores (IP) of the 3 excellent workers is 1 Normalized reflection factor (VP) can be calculated as shown in Table 3 below.

Worker w1 w2 w3 Impact Point One 0.9 0.8

Worker w1 w2 w3 Voting Power 1/2.7 0.9/2.7 0.8/2.7

Referring back to FIG. 2 , the artificial intelligence module determines inspection criterion data for inspection of completed data for each of a plurality of workers based on the calculated reflection rate (S250). For example, the inspection criterion data (ie, inspection criterion trimap) may be determined by applying the reflection factor (VP) to the opacity applied when forming the trimap.

12 is a diagram for explaining an example in which reflection rates for three excellent workers are calculated and inspection standard data is determined based on the reflection rates.

Referring to FIG. 12 , reflection rates for three excellent workers w1 , w2 , and w3 are calculated as shown in FIG. 12 , and inspection standard data is determined based on the reflection rates.

Determination of the inspection criterion data may be determined based on the reflection rate of a particular excellent worker selected according to the type of work result.

Specifically, when the variable form of the completion data is a categorical variable, the method of taking the average value of the completion data cannot be applied, so the completion with the highest reflection rate of the best worker with the highest reflection rate among the excellent workers Determine the data as the inspection standard data.

On the other hand, if the variable form of the complete data is a continuous variable, it can be seen that a statistical approach to the complete data is possible. In this case, the artificial intelligence module determines the inspection standard data by applying an inference model suitable for the completion data of excellent workers.

For example, as shown in FIG. 12, Opacity masks (OM1, OM2, OM3) are generated for each worker by applying the reflection rate of excellent workers as Opacity to the completion data of each worker, and by overlapping them, Trimap (TM ) and selecting only elements having an element value exceeding a specific threshold value in the formed Trimap (TM), the inspection criterion data may be determined.

However, the method of determining the inspection standard data is not limited to this, and the artificial intelligence module selects various statistical values such as the average value and median value of the completed data to which the reflection rate of excellent workers is applied, or the alpha-matting algorithm. It can be applied to determine the inspection standard data.

In some embodiments, the artificial intelligence module may apply a selective inference model without applying a statistical inference model even if the complete data is in the form of a continuous variable. That is, even if the completion data has a continuous data value between 0 and 1, if it is necessary to determine the correct answer data as 0 or 1 due to the nature of the task, a selective inference model is applied to select the completion data of the best worker with the highest reflection rate. It can be set as answer data.

Thereafter, the crowd outsourcing job inspection apparatus 400 according to an embodiment of the present invention adjusts the existing reflection rate (VP) for each worker based on the calculated reflection rate (VP) for each worker. That is, the reflection rate (VP) can be adjusted to increase for workers who are determined to have many correct data among complete data through inspection, and the reflection rate (VP) can be adjusted to decrease for workers who are determined to have few correct data among complete data. have.

The crowd outsourcing job inspection apparatus 400 according to an embodiment of the present invention inspects workers' completion data based on the determined job data, and selects completion data processed as correct answers as learning data. Since artificial intelligence can be learned based on verified learning data, the learning efficiency of artificial intelligence can be increased and the performance of artificial intelligence can be advanced.

As described above, the crowd outsourcing job inspection apparatus 400 using artificial intelligence according to an embodiment of the present invention can distribute the job of generating learning data for artificial intelligence learning to unspecified workers through the Internet. . In particular, by pre-processing the work target data through an artificial intelligence module and transmitting the pre-processed work data to unspecified workers, the work of the workers can be performed more easily. In other words, since the artificial intelligence module can generate work data by performing preprocessing to set the work target area in work target data and transmit the generated work data to a plurality of workers, workers can more easily create learning data. can be performed.

In addition, since the artificial intelligence module is configured to learn first completion data in which work on some first job data is completed in the process of generating job data, it can be gradually learned in the process of generating job data, and the first job After distributing the data, in the process of generating the second and third work data, a work area may be set in a more advanced way, and thus, highly pre-processed work data may be generated and distributed to workers. Therefore, the workload of a plurality of workers can be further reduced, and ultimately, the task of generating learning data itself can be configured to be performed by an artificial intelligence module.

In addition, the crowd outsourcing job inspection apparatus 400 using artificial intelligence according to an embodiment of the present invention calculates the job difficulty and the accuracy of the workers who performed the job, the job difficulty is sufficiently low, and the accuracy of the plurality of workers If is high enough, it is configured to perform inspection through an artificial intelligence module for that task. In addition, when the difficulty of the task is too high or the accuracy of the plurality of workers is too low, and the inspection through the artificial intelligence module is unclear, it is configured to perform an expert inspection on the task. In other words, experts only need to inspect some ambiguous tasks, and since most of the inspections are performed through an artificial intelligence module, the workload of experts can be sufficiently reduced and the efficiency of crowd outsourcing work can be improved.

In this case, the crowd outsourcing job inspection apparatus 400 using artificial intelligence according to an embodiment of the present invention determines the accuracy, tendency, and difficulty of a plurality of workers, and determines the reflection rate based on the accuracy and difficulty of each worker. And configured to determine the correct answer data for the completion data inspection for each of a plurality of workers based on the reflection rate. That is, a high reflection rate is given to workers with excellent accuracy, and completion data of excellent workers is reflected at a high rate in determining correct answer data. Therefore, it is possible to minimize the problem of lowering the reliability of correct answer data due to the work results of a plurality of workers with low accuracy, and the reliability of inspection by the artificial intelligence module can be further improved.

In particular, the crowd outsourcing job inspection apparatus 400 using artificial intelligence according to an embodiment of the present invention applies a weight according to a period in determining a reflection rate for each of a plurality of workers. Even an excellent worker may have poor recent work accuracy due to various factors. Since the crowd outsourcing work inspection apparatus 400 according to an embodiment of the present invention is configured to determine the reflection rate by applying a weight according to a period to the quality score of each worker, the work result of an excellent worker whose work accuracy has recently decreased is the correct answer. It is possible to prevent data from being influenced, and through this, reliability of correct answer data can be improved.

The above description of the present invention is for illustrative purposes, and those skilled in the art can understand that it can be easily modified into other specific forms without changing the technical spirit or essential features of the present invention. will be. Therefore, the embodiments described above should be understood as illustrative in all respects and not limiting. For example, each component described as a single type may be implemented in a distributed manner, and similarly, components described as distributed may be implemented in a combined form.

The scope of the present invention is indicated by the following claims rather than the detailed description above, and all changes or modifications derived from the meaning and scope of the claims and equivalent concepts should be construed as being included in the scope of the present invention. do.

100, 200, 300: worker terminal
400: crowd outsourcing task management device
500: inspector terminal
410: communication department
420: processor
430: memory

Claims (13)

A data receiving step in which the communication unit receives completion data for a learning data forming task from a plurality of worker terminals via the Internet;
A data reconstruction step of reconstructing, by a processor controlled by an artificial intelligence module, the completed data into data that can be nested; and
The processor controlled by the artificial intelligence module overlaps the reconstructed nestable data, analyzes the overlapped data, or compares and analyzes the overlapped data and one of the completed data for each of a plurality of workers to obtain the plurality of data. Including a propensity determination step of determining at least one of the operator's propensity, accuracy, and difficulty of the learning data formation task;
In the data reconstruction step,
The complete data is two-dimensional data and is reconstructed into a matrix with divided faces obtained by dividing the entire face.
Crowd outsourcing work inspection method using artificial intelligence. delete According to claim 1,
In the propensity determination step,
a mask generating step of generating a mask for each of the plurality of worker terminals with a matrix having an element corresponding to the divided surface from the reconstructed data for each of the plurality of worker terminals; and
Including a propensity analysis step of analyzing the mask to determine at least one of the propensity, accuracy, and difficulty of the learning data forming task for each of the plurality of workers,
Crowd outsourcing work inspection method using artificial intelligence. According to claim 3,
In the propensity analysis step,
Determining the difficulty of the learning data forming task for the task from the deviation between the masks for each of the plurality of worker terminals for the task,
Crowd outsourcing work inspection method using artificial intelligence. According to claim 3,
In the propensity determination step,
Further comprising a Trimap generation step of generating a Trimap by merging the masks for each of the plurality of worker terminals between the mask generation step and the propensity analysis step,
Crowd outsourcing work inspection method using artificial intelligence. According to claim 5,
In the Trimap generation step,
The method of merging the plurality of worker terminal masks is a method of indicating the average of the element values of the plurality of worker terminal masks in the Trimap.
Crowd outsourcing work inspection method using artificial intelligence. According to claim 6,
In the propensity determination step,
Determining the accuracy of the target worker terminal by calculating the similarity between the mask of the target worker terminal and the Trimap among the plurality of worker terminals,
Crowd outsourcing work inspection method using artificial intelligence. According to claim 7,
In the propensity determination step,
As a method of calculating the similarity, the cosine similarity calculated for each element is used for the entire mask of the target worker terminal, and the difference between elements is used as one variable to be substituted for the cosine similarity, and the other variable Use the value of the element indicated by the recognition displayed on the mask of the target worker terminal as
The difference per element is the difference between the value of the element displayed in the Trimap and the value of the element displayed in the mask of the target worker terminal,
Crowd outsourcing work inspection method using artificial intelligence. According to claim 7,
In the propensity determination step,
As a method of calculating the degree of similarity, the sum of the differences for each element is specified as 1's complement of a value divided by the number of elements having a value greater than 0 in the Trimap,
The difference per element is the difference between the value of the element displayed in the Trimap and the value of the element displayed in the mask of the target worker terminal,
Crowd outsourcing work inspection method using artificial intelligence. According to claim 5,
In the propensity determination step,
Determining the work propensity of the target worker terminal by calculating the average of elements in the Trimap corresponding to the selected element among the masks of the target worker terminal,
Crowd outsourcing work inspection method using artificial intelligence. According to claim 10,
In the propensity determination step,
Determining an Opacity Mask Score (OMS) value obtained by dividing the sum of the values of elements corresponding to the mask of the target worker terminal in the Trimap by the number of selected elements as the work propensity,
Crowd outsourcing work inspection method using artificial intelligence. According to claim 5,
In the propensity determination step,
As a method of calculating the task difficulty, determining the sum of the values of the elements displayed in the Trimap by dividing the value of the elements by the number of elements whose values are greater than 0;
Crowd outsourcing work inspection method using artificial intelligence. Communication unit for receiving the completion data from the plurality of worker terminals, the work for the work data is completed;
a processor configured to perform an operation of verifying the completed data received from the plurality of worker terminals; and
A memory in which an artificial intelligence module for controlling an operation of the processor is stored;
The artificial intelligence module,
The completed data is reconstructed into overlappable data, the reconstructed overlappable data is overlaid, and the overlapped data is analyzed, or the overlapped data and one of the completed data are compared and analyzed to determine the accuracy of the operator, the operator's Determine at least one of the inclination and difficulty of the task,
Crowd outsourcing work inspection device using artificial intelligence, in which the completed data is reconstructed into a matrix having a divided surface obtained by dividing the entire surface as two-dimensional data.

Images