KR102232880B1 - Method for evaluating inspector of crowdsourcing based projects for collecting image or video for artificial intelligence training data generation - Google Patents

Abstract

Provided is a method for evaluating an inspector of crowdsourcing-based projects for collecting images or videos to motivate the inspector. According to the present invention, the method comprises the following steps: assigning a plurality of work results of a crowdsourcing-based project to a plurality of inspectors and requesting inspection; receiving a plurality of inspection results for the plurality of work results from the plurality of inspectors as pass or reject; and calculating an inspection proficiency score by evaluating inspection proficiency of each inspector on the basis of the inspection result input by the inspector. A work is to collect and submit images or videos predetermined conditions. The step of requesting to perform the inspection comprises the following steps: extracting one or more previous work results of an operator who entered work results to be inspected; providing the inspector with the work results to be inspected and at least a part of one or more previous work results as a work result whose duplication is checked; and requesting the inspector to perform an inspection on whether the work result to be checked complies with work guidance and whether it is duplicated with the work result whose duplication is checked. The step of calculating the inspection proficiency score calculates the inspection proficiency score by evaluating the inspection proficiency of each inspector on the basis of the quantity of the work results whose duplication is checked.

Description

{METHOD FOR EVALUATING INSPECTOR OF CROWDSOURCING BASED PROJECTS FOR COLLECTING IMAGE OR VIDEO FOR ARTIFICIAL INTELLIGENCE TRAINING DATA GENERATION}

The present invention relates to a crowdsourcing-based project reviewer evaluation method for collecting images or videos for generating artificial intelligence learning data.

Recently, more and more companies collect and process large amounts of data on a crowdsourcing basis that engages the general public in some process of corporate activities. In other words, by opening a project and allowing the general public, that is, workers to participate in the project, necessary information is collected through the results of work completed by the workers.

When one project is opened, multiple tasks are assigned to each of the multiple workers. Each worker performs a plurality of tasks assigned to it, and provides the task results.

If the type of work required by the project is to collect and submit images or videos that meet certain conditions, the bad worker submits the same image or video multiple times as if it were different work results, or that was taken on the same object. Often, images or videos are submitted multiple times as if they were the result of different work on different objects. At this time, the inspector should compare the work result with other work results previously submitted by the same worker and perform an inspection on whether or not to be duplicated, and filter the overlapping work results.

At this time, if a large amount of work results have already been input by the same operator, the inspector must perform an inspection comparing the work results to be inspected with the large amount of work results in order to determine whether there is overlap. Therefore, the more this process is repeated, the higher the fatigue level of the inspector and the willingness to perform the inspection also decreases.

In particular, since the inspection cost received by the inspector is generally calculated according to the number of work results performed, the inspector does not receive adequate compensation corresponding to the effort and time spent to determine whether or not the inspection is duplicated.

Eventually, such a problem may lead to a deterioration in the quality of the review of a project that requires the collection and submission of images or videos that meet certain conditions, and even lower the appraisal's willingness to participate in the project.

Unexamined Patent Publication No. 10-2014-0095956, 2014.08.04.

The problem to be solved by the present invention is to provide a method for evaluating inspectors of a crowdsourcing-based project for collecting images or videos for generating artificial intelligence learning data.

However, the problem to be solved by the present invention is not limited to the problem as described above, and other problems may exist.

In order to solve the above-described problem, the method for evaluating the inspector of a crowdsourcing-based project for collecting images or videos according to an aspect of the present invention is a method performed by a computer, and includes a plurality of crowdsourcing-based projects (hereinafter, "projects"). Allocating the work of the number to a number of workers and requesting to perform the work, receiving a plurality of work results from the plurality of workers, assigning the plurality of work results to a plurality of inspectors and requesting to perform the inspection, the Receiving a plurality of inspection results for the plurality of work results from a plurality of inspectors as passing or rejecting the inspection, and calculating the inspection proficiency score by evaluating the inspection proficiency of each inspector based on the inspection results input by the inspector. Including the step, wherein the operation is to collect and submit an image or video that satisfies a predetermined condition, and the requesting to perform the inspection is the step of extracting one or more previous work results of a worker who has entered the work result to be inspected. , Providing at least some of the inspection target work result and one or more previous work results to the inspector as a duplicate check target work result, and whether the inspection target work result complies with the work guide to the inspector, and the overlapping check target Including the step of requesting to perform an inspection on whether or not it overlaps with the work result, and calculating the inspection proficiency score, the inspection proficiency level of each inspector is further evaluated based on the quantity of the work result subject to the overlapping check. Calculate the score.

In some embodiments of the present invention, the calculating of the inspection proficiency score includes a basic score corresponding to the number of inspection results input by the inspector and an additional score corresponding to the quantity of the duplicate check target task result for each inspection result. By summing the test proficiency score can be calculated.

In some embodiments of the present invention, the step of calculating the test proficiency score may calculate the test proficiency score using the following equation.

Base score = number of inspection results entered by inspector x unit score

Additional score = Quantity of work results subject to duplicate check x unit score x predetermined percentage

In some embodiments of the present invention, in the step of calculating the test proficiency score, the basic score and the test proficiency score may not be calculated when the test result input by the tester is rejected.

In some embodiments of the present invention, the providing of the task result subject to the redundancy check includes extracting a previous task result that has passed the inspection from among the one or more previous task results, and checks at least a part of the previous task result that has passed the inspection. It can be provided as a target work result.

In some embodiments of the present invention, the providing of the duplicate check target task result includes measuring a similarity between the inspection target task result and the one or more previous task results, and the similarity among the one or more previous task results. It may include the step of providing a previous work result that is greater than or equal to a reference value as the duplicate check target work result.

In some embodiments of the present invention, the measuring of the similarity comprises extracting feature maps of the inspection target work result and the one or more previous work results, respectively, and each feature map is n-dimensional (n is a natural number of 2 or more). The similarity can be measured by embedding as a vector of and measuring the Euclidean distance between each vector.

In some embodiments of the present invention, the requesting to perform the inspection comprises: requesting the inspection by limiting the minimum inspection time for each inspection target task result, the minimum inspection time is a basic minimum inspection time and the overlapping check target task It can be determined by summing an additional minimum inspection time corresponding to the quantity of results.

In some embodiments of the present invention, the additional minimum inspection time may be calculated using the following equation.

Additional minimum inspection time = Quantity of work results subject to overlapping checks x Basic minimum inspection time x predetermined ratio

In some embodiments of the present invention, in the step of receiving the inspection result as passing or rejecting the inspection, the inspection result input interface may be activated after the minimum inspection time has elapsed to receive the inspection result from the inspector.

A computer program according to another aspect of the present invention for solving the above-described problem executes a crowdsourcing-based project reviewer evaluation method for collecting the image or video by combining it with a computer that is hardware, and storing it in a computer-readable recording medium. Can be.

Other specific details of the present invention are included in the detailed description and drawings.

According to the present invention described above, an additional score corresponding to the quantity of the duplicate check target task result is given to the inspector who performs the duplicate check inspection, which filters the duplicate task results submitted by the same operator. Through this, it provides a reasonable reward corresponding to the inspector's effort and time spent on the redundant check inspection, and consequently gives the inspector an appropriate motive in performing the redundant check inspection, which requires relatively more effort and time than the general inspection. can do.

In addition, according to the present invention, the previous work results that have passed the inspection among the previous work results of the operator are provided as the work results to be duplicated check, or only work results similar to the work results to be inspected are selected among the previous work results as the work results to be duplicated check. to provide. This is to perform a pre-filtering process prior to the inspector's redundant check inspection, thereby reducing the number of target work results provided to the inspector. As a result, the inspector can reduce the manpower and time required for the redundant check inspection by performing the redundant check inspection based on the result of a small number of redundant check target work. In addition, this effect may lead to an effect of increasing the accuracy of the inspection results and preventing delays in the entire project schedule due to redundant check inspection.

In addition, the minimum inspection time is extended to correspond to the quantity of work results subject to overlapping checks, and inspection results can be entered only after the minimum inspection time has elapsed, preventing the inspector from performing the inspection effortlessly and ultimately, the quality of the inspection results. It can prevent deterioration.

The effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the following description.

1 is a conceptual diagram of a crowdsourcing service according to an embodiment of the present invention.
2 is a flowchart illustrating a crowdsourcing-based project progress process according to an embodiment of the present invention.
3 is a conceptual diagram showing a method of performing an inspection by an inspector of a project for collecting images or videos.
4 is a flowchart schematically showing a method for evaluating a reviewer of a project for collecting images or moving pictures according to an embodiment of the present invention.
5 is a flow chart schematically showing a method of requesting a reviewer of a project for collecting images or moving pictures according to an embodiment of the present invention to perform inspection.
6 is a conceptual diagram illustrating providing a result of a duplicate check object based on a previous work result of a specific worker according to an embodiment of the present invention.
7 is a conceptual diagram illustrating providing a result of a duplicate check object based on a previous work result that has passed the inspection of a specific worker according to an embodiment of the present invention.
FIG. 8 is a diagram for describing a method of calculating an additional minimum inspection time based on the quantity of work results subject to overlapping check according to an embodiment of the present invention.
9 is an exemplary view showing a screen for reviewing a project for collecting images or moving pictures.
10 is a diagram illustrating an apparatus for evaluating a reviewer of a project for collecting images or moving pictures according to an embodiment of the present invention.

Advantages and features of the present invention, and a method of achieving them will become apparent with reference to the embodiments described below in detail together with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but may be implemented in a variety of different forms. It is provided to fully inform the skilled person of the scope of the present invention, and the present invention is only defined by the scope of the claims.

The terms used in the present specification are for describing exemplary embodiments and are not intended to limit the present invention. In this specification, the singular form also includes the plural form unless specifically stated in the phrase. As used herein, “comprises” and/or “comprising” do not exclude the presence or addition of one or more other elements other than the mentioned elements. Throughout the specification, the same reference numerals refer to the same elements, and "and/or" includes each and all combinations of one or more of the mentioned elements. Although "first", "second", and the like are used to describe various elements, it goes without saying that these elements are not limited by these terms. These terms are only used to distinguish one component from another component. Therefore, it goes without saying that the first component mentioned below may be the second component within the technical idea of the present invention.

Unless otherwise defined, all terms (including technical and scientific terms) used in the present specification may be used with meanings that can be commonly understood by those of ordinary skill in the art to which the present invention belongs. In addition, terms defined in a commonly used dictionary are not interpreted ideally or excessively unless explicitly defined specifically.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a conceptual diagram of a crowdsourcing service according to an embodiment of the present invention.

Referring to FIG. 1, the crowdsourcing service is performed by being composed of a client 10, a service provider 20, and a public 30.

The sponsor 10 refers to a company or individual who requests a crowdsourcing-based project (hereinafter, referred to as a project).

The client 10 requests a project for the purpose of collecting source data or annotating data for generating artificial intelligence learning data. The data generated through the project can be used as learning data for arbitrary machine learning such as supervised learning, unsupervised learning, and reinforcement learning. Collection of source data means collecting raw data such as recorded voice collection and photo collection. Data annotation refers to inputting relevant annotation data into source data such as text, photos, and videos. For example, the data annotation may include finding an entity in a given fingerprint or finding a similar sentence, but is not limited thereto. Meanwhile, the types of the above-described projects are only one embodiment, and various projects may be handled in the present invention according to the design of the client.

The service provider 20 refers to a company that provides crowdsourcing services.

When the service provider 20 receives a request for a project for a product or service from the client 10, the service provider 20 allocates a task for the project to the general public 30 and receives the work result from the public 30. Thereafter, the final product extracted based on the work result is provided to the requester 10.

At this time, the service provider 20 provides a crowdsourcing service to the client 10 and the public 30 through a crowdsourcing platform (hereinafter, the platform). That is, when the service provider 20 receives a request for a project from the client 10, the service provider 20 opens the project on the platform. Thereafter, when the work result for the open project is provided by the public 30, the project is terminated on the platform, and the final product may be extracted and provided to the client 10.

The public 30 refers to the general public participating in the project open on the platform. Here, the public 30 may participate in a project opened on the platform through an application or website provided by the service provider 20.

The public 30 is composed of a worker 32 and an inspector 34.

The worker 32 decides to participate in a specific project among a plurality of projects open on the platform. Thereafter, the operator 32 performs an operation such as collection of source data or data annotation, and transmits it to the platform.

The inspector 34 decides to participate in a specific project among a plurality of projects opened on the platform. Thereafter, the inspector 34 inspects the result of the work performed by the operator 32. As a result of performing the inspection, the inspector 34 may perform inspection pass processing or rejection processing, and input a rejection reason during rejection processing. In the case of passing the inspection, rework and the resulting re-inspection are not required, so passing the inspection has the same meaning as the completion of the inspection.

2 is a flowchart illustrating a crowdsourcing-based project progress process according to an embodiment of the present invention.

First, the client 10 requests one or more projects to the service provider 20 (S11).

Thereafter, the service provider 20 opens the requested project on the platform (S12). At this time, the service provider 20 may determine the grade in consideration of the difficulty of the project before opening the project. That is, depending on the difficulty level, it is possible to determine which level or higher public 30 to expose the project. Accordingly, it is possible to increase the reliability of the work result of the project.

Thereafter, the service provider 20 requests the work by assigning the work to the workers 32 of the corresponding level or higher according to the class of the project (S13).

Thereafter, the worker 32 performs the assigned task (S14). In this case, the worker 32 may input the reason for the inability to work without performing the work for a work in which the work itself is impossible for some reason.

Thereafter, the service provider 20 receives the work result from the worker 32 (S15), and assigns the work result to the inspector 34 to request the inspection (S16).

Thereafter, the inspector 34 performs the assigned inspection (S17). At this time, the inspector 34 decides to pass the inspection when it is determined that the work has been properly performed, and if it is determined that the inspection work is wrong, the inspector 34 rejects it. During rejection processing, the inspector 34 inputs the rejection reason for what reason the job was judged to be wrong.

Thereafter, the service provider 20 receives the inspection result from the inspector 34 (S18).

If the inspection result passes the inspection, the service provider 20 uses the work result as valid data, and based on this, extracts the final product at the end of the project.

If the result of the inspection is rejection processing, the service provider 20 may internally perform the inspection again, or assign the work to the worker 32 again to perform the rework. When reworking, re-examination by an inspector is required.

Thereafter, when the project period ends or sufficient valid data is secured, the service provider 20 terminates the project (S19), calculates a final result based on the secured valid data, and provides it to the client 10 ( S20).

At this time, before the end of the project, the service provider 20 evaluates the performance results of the worker 32 and the inspector 34, calculates the work cost and the inspection cost according to the evaluation, to the worker 32 and the inspector 34. give.

In Figures 1 and 2, it is simply expressed as a requester 10, a service provider 20, an operator 32, and an inspector 34, but these are smartphones, tablets, PDAs, laptops, desktops operated by each participant. It means a computer device such as a server or a telecommunication device.

Hereinafter, a method for evaluating an inspector of a crowdsourcing-based project for collecting images or videos according to an embodiment of the present invention will be described with reference to FIGS. 3 to 9. Details of overlapping content related to the crowdsourcing service configuration and the crowdsourcing-based project progress process described with reference to FIGS. 1 and 2 will be omitted.

3 is a conceptual diagram showing a method of performing an inspection by an inspector of a project for collecting images or moving pictures.

Projects for image or video collection require the operator to collect and submit images or videos that meet predetermined conditions according to their type. For example, this is the case of collecting and submitting images of a newborn's face.

A predetermined condition for collecting and submitting the above-described image or video may be as follows. For example, in a project that collects and submits a newborn's face image, the image collected by a worker must be for the newborn's face, and it is collected in a format or method required by the sponsor 10 of the project. Must have been submitted. For example, a form or method may be required, such as that the face of a newborn should be photographed in front of the face of the newborn, or that the face of a newborn should be photographed and collected indoors. This will be judged by the inspector of the project performing an inspection as to whether or not the inspection target work result 41 complies with the work guide.

At this time, in the case of a job of collecting and submitting an image or video, there is an additional problem with whether or not the above-described job guide is observed. That is, the same operator submits the same image or video multiple times as if it were different work results, or submits multiple images or videos for the same object as if they were for different objects. For example, one worker collects and submits multiple facial images of the same newborn baby. This problem is usually caused frequently by poor workers.

Referring to FIG. 3, in this case, the inspector must determine whether or not the contents are duplicated by comparing one or more previous work results submitted by the operator to the work results 41 to be inspected during the inspection process. That is, it is necessary to determine whether the work result is duplicated by comparing the work result 41 to be inspected with the results of one or more previous work input by the same operator, together with whether the work results comply with the work guide. If a large amount of work results are submitted by the same worker, the inspector must determine whether or not there is overlap by comparing the work result 41 to be inspected with the large amount of previous work results. This inspection performance gives a great fatigue and burden because the inspector has to spend a lot of time inspecting the target work result. Above all, because the inspection cost that the inspector receives from the inspection is calculated as the number of the results of the inspection, even if it is determined whether or not the results of a large amount of previous work and the result of the subject to be inspected overlap by spending a long time. You only get the cost of reviewing the results. In other words, they do not receive adequate compensation for the effort and time invested by the inspector. This depresses the inspector's willingness to inspect it, making the inspector perform the inspection in an easy or meticulous way, and may lead to a problem that many inspectors avoid participating in the project for the collection of images or videos.

Therefore, in the case of a project for the collection of images or videos in which it is necessary to determine whether the work result 41 to be inspected overlaps with other work results previously input by the same worker, the inspector consumes it to check the duplicate work results during the inspection process. It is necessary to motivate the inspector by providing a reasonable evaluation criterion for the time and effort. An embodiment of the present invention for this will be described in detail below.

The steps shown in FIGS. 4 and 5 may be understood as being performed by a platform server (hereinafter, referred to as a server) operated by the service provider 20, but are not limited thereto.

4 is a flowchart schematically showing a method for evaluating a reviewer of a project for collecting images or moving pictures according to an embodiment of the present invention.

First, referring to FIG. 4, the server allocates a plurality of tasks of a project to a plurality of workers and requests the task to be performed (S110). Here, the job refers to a job that requests the collection and submission of images or videos that meet a predetermined condition.

After step S110, a plurality of work results are input from a plurality of workers (S120). Step S120 is redundant with respect to the process of the crowdsourcing-based project described with reference to FIGS. 1 and 2 as described above, and thus a detailed description thereof will be omitted.

When a plurality of work results are inputted (S120), the plurality of work results are assigned to a plurality of inspectors, and the inspection is requested to be performed (S130).

At this time, in step S130, the inspector may request the inspector to perform the inspection to determine whether the inspection target work result overlaps with one or more previous work results of the worker who has input the corresponding inspection target work result.

Here, the overlapping with the previous work result means that not only the contents of the work result to be inspected and the previous work result are completely identical, but also that the contents of the image or video or the object to be photographed are the same, so that the contents are substantially the same. For example, in a project that submits images of a newborn's face, this corresponds to submitting a plurality of images collected by photographing a newborn's face from different angles or from different distances. The criteria for redundancy may be different for each project depending on the characteristics of each project, and may be specified in the work guide or inspection guide for each project.

On the other hand, whether or not the previous work result is duplicated is determined based on the same worker. Therefore, even if the work results submitted by different workers match, this does not correspond to the duplicate work results.

5 is a flow chart schematically showing a method of requesting a reviewer of a project for collecting images or moving pictures according to an embodiment of the present invention to perform inspection.

6 is a conceptual diagram illustrating providing a result of a duplicate check object based on a previous work result of a specific worker according to an embodiment of the present invention.

5, in an embodiment of the present invention, step S130 is a step of extracting one or more previous work results of a worker who has inputted a work result to be inspected (S131), the work result and one or more previous work to an inspector Providing at least some of the results as a result of a task to be checked for duplicate (S132), and requesting the inspector to perform a check on whether the result of the task to be inspected complies with a task guide and whether it overlaps with a result of the task to be checked for duplicate check It includes a step (S133).

First, referring to FIG. 5, the server extracts one or more previous work results of the worker who has input the work result 41 to be inspected, which is assigned to the inspector and requested to perform the inspection (S131). As described above, since it is determined based on the same worker whether or not the work result is duplicated, the previous work result of the worker who inputs the work result 41 to be inspected is extracted.

Here, the previous work result refers to a work result in which the test result has already been input because the test is performed prior to the work result 41 to be inspected among the work results input by the same operator. In this case, even if the work result is input later than the work result 41 to be inspected, if the inspection is performed first and the inspection result is already input, it corresponds to the previous work result.

When the previous work result is extracted in step S131, at least a part of the one or more previous work results together with the test target work result is provided to the inspector as the duplicate check target work result 42 (S132).

Referring to FIG. 6, the worker who inputs the work result 41 to be currently inspected is worker A. Therefore, in step S131, the server extracts the previous work result of worker A. As a result of worker A's previous work, there are 1 work result that has passed the inspection and 1 rejected work result. At this time, at least a part of the extracted one inspection passed task result and one rejected task result is provided as a duplicate check target task result 42. Referring to FIG. 6, all extracted two previous task results are duplicated. It can be seen that the check operation result 42 was provided.

Meanwhile, among the previous work results of the operator who entered the work result 41 to be inspected, the work result 42 subject to duplicate check is a part of the work result that is the target of determining whether or not it overlaps with the work result 41 to be inspected. Means. In other words, the inspector may not compare with all previous work results of a specific worker in order to determine whether the work results 41 to be inspected overlap, and among the previous work results, only work results provided as work results 42 to be checked for overlapping are mutually Can be judged by comparison.

As described above, when the same operator submits a large amount of work results, comparing with all previous work results except for the work results to be inspected 41, determining whether or not overlapping increases the inspector's fatigue, lowering the concentration, and eventually This is because it can lead to a result that deteriorates the quality of the inspection result. Therefore, in order to prevent this, it is possible to extract only a part of the previous work results and provide a result to be redundantly checked based on this.

7 is a conceptual diagram illustrating providing a result of a duplicate check object based on a previous work result that has passed the inspection of a specific worker according to an embodiment of the present invention.

According to an embodiment of the present invention, step S132 may extract a previous work result that has passed the inspection among one or more previous work results, and provide at least some of the previous work results that have passed the inspection as the duplicate check target work result.

Only the work results that have passed the inspection as a result of the inspection performance by the inspector may be provided to the client 10 as a final product. Therefore, among the previous work results of the worker, by providing the work results subject to overlapping check based only on the results of the previous work that has passed the inspection, the burden of the checker’s inspection of the overlapping work results is reduced, and the final product provided to the client 10 at the same time You can increase the quality.

Referring to FIG. 7, as for the previous work result of worker A, there are one inspection-passed work result and one rejected work result. At this time, the previous work results that have passed the inspection excluding the rejected work results are provided as the work results subject to inspection and check. In the case of FIG. 7, it can be seen that one work result that has passed the inspection is provided as a work result subject to overlapping check.

In addition, although not clearly shown, as an embodiment of the present invention, step S132 includes measuring the similarity between the inspection target work result and one or more previous work results (not shown), and the similarity among the one or more previous work results as a reference value. A step (not shown) of providing the above previous work result as a duplicate check target work result.

Specifically, the degree of similarity between the inspection target work result and the previous work result is measured as a value, and the previous work result with a high similarity is provided as the duplicate check target work result. By providing the previous work result similar to the work result 41 to be inspected as the work result to be duplicated through a single filter process, it is possible to reduce the burden of the inspector on the inspection of the duplicate work result.

In this case, the previous work result may include not only the work result that has passed the operator's inspection but also the rejected work result. However, it is not limited thereto.

In addition, as an embodiment of the present invention, the degree of similarity between the inspection target job result 41 and the one or more previous job results that have passed the inspection among the previous job results is measured as a value, and the previous job result that has passed the inspection with a high degree of similarity Can be provided as the result of the task subject to duplicate check. That is, by selecting and providing duplicate check targets, the optimal inspection environment is provided to the inspector to inspect the duplicate work results.

Meanwhile, the degree of similarity is determined based on a preset reference value.

In addition, in an embodiment of the present invention, in the step of measuring the similarity (not shown), feature maps of a task result to be inspected and one or more previous task results are extracted, and each feature map is n-dimensional (n). Is embedded as a vector of 2 or more natural numbers), and the Euclidean distance between each vector can be measured to measure the degree of similarity.

Specifically, in a project that requires the collection and submission of images with predetermined conditions, the target work result 41 and the image of one or more previous work results are input to various well-known deep learning models such as Backbone network to map each feature. Extract (Feature Map). In this case, the feature map may have a predetermined dimension. For example, it may have 512 dimensions, but is not limited thereto.

Meanwhile, the extracted feature map is embedded into an n-dimensional vector. Where n is a natural number greater than or equal to 2.

For example, if the extracted feature map is composed of 512-dimensional features, the 512-dimensional feature information is reduced to 3D through the t-SNE (T-Stochastic Neighbor Embedding) method and converted into a 3D vector. Embed. Then, a three-dimensional vector is displayed in a three-dimensional space, and the Euclidean distance to each three-dimensional vector of the previous target work result is measured based on the three-dimensional vector of the target work result 41. The similarity may be measured based on the measured Euclidean distance, and the previous work result of the image having a high similarity may be provided as the work result 42 to be checked for redundancy.

Referring to FIG. 5 again, after providing the task result to be checked for duplicate (S132), the inspector is asked whether the task result 41 to be inspected complies with the task guide and whether it is overlapped with the task result 42 to be checked for overlapping. Request to perform the inspection (S133). In the case of a general project, it is sufficient to perform the inspection only as to whether or not the inspection target work result (41) complies with the work guide. It is also to request to perform the inspection on whether or not it overlaps with the work result (42).

Meanwhile, referring to FIG. 4 again, after requesting a task by assigning a plurality of tasks to a plurality of inspectors, a plurality of inspection results for a plurality of work results are input from the plurality of inspectors as passing or rejecting the inspection (S140).

At this time, the inspector inputs the inspection result as a rejection not only when the inspection target work result 41 does not comply with the work guide, but also when it overlaps with the overlapping check target work result 42.

Referring again to FIG. 4, when the inspector inputs the inspection result (S140), the inspection proficiency score is calculated by evaluating the inspection proficiency of each inspector based on the inspection result input by the inspector (S150).

At this time, in step S150, the inspection proficiency score is calculated by evaluating the inspection proficiency of each inspector further based on the quantity of the duplicate check target task result.

In the case of a general project, the proficiency level of inspection can be calculated based on the number of work results performed by the inspector. In the case of a project for collecting images or videos, the proficiency score for inspection is calculated further based on the number of work results subject to overlapping checks by the inspector. Since the number of work results subject to overlapping checks may be different for each work result, compensation is provided.

Here, the inspection proficiency score is an index that evaluates the inspection ability of the inspector and affects the number of work results assigned to the inspector or granting inspection authority according to the difficulty of the project. Therefore, by calculating the proficiency score for inspection based on the number of work results subject to overlapping checks, a number of inspections with a large number of work results subject to overlapping checks are performed. You can get an opportunity to receive a test and receive more work results than an inspector with a low proficiency score within the same project.

On the other hand, in step S150 according to an embodiment of the present invention, a basic score corresponding to the number of inspection results input by the inspector and an additional score corresponding to the quantity of the duplicate check target task result for each inspection result are summed to the inspection proficiency level. Score can be calculated.

In this case, the basic score and the additional score described above as an embodiment of the present invention may be calculated using the following equation (1).

[Equation 1]

Base score = number of inspection results entered by inspector x unit score

Additional score = Quantity of work results subject to duplicate check x unit score x predetermined percentage

For example, if there are 20 inspection results entered by inspector A of a project with a unit score of 1 point and a predetermined ratio of 0.5, and a total of 30 work results subject to duplicate checks while performing 20 inspections, inspector A The proficiency score of the test is equivalent to 35 points, which is the sum of 20 basic points and 15 additional points.

Meanwhile, according to an embodiment of the present invention, the predetermined ratio may be determined based on the difficulty of the project. At this time, the difficulty of the project may be determined based on the average working time of the workers who participated in the project. This is because the higher the difficulty of the project, the longer the average working time of the workers who participated in the project. Also, when the difficulty of the project is high, the difficulty of the duplicate check is generally high.

Also, the difficulty of a project can be determined based on the rejection rate of the project. Here, the rejection rate may be calculated as an average daily rejection number by calculating the number of rejected tasks during a predetermined unit period, or may be calculated as the number of rejected tasks per worker. However, it is not limited thereto.

Meanwhile, in the step of calculating the test proficiency score according to an embodiment of the present invention, the basic score and the test proficiency score may not be calculated for the test result inputted by the tester that is a rejection. In other words, the basic score and the test proficiency score are calculated based on the number of work results inputted through the test result.

In addition, in the step of calculating the test proficiency score according to another embodiment of the present invention, in performing the test input by the tester as rejection, the additional score for rejection due to non-compliance with the work guide of the target work result (41) May not yield.

Specifically, in the process of inspecting whether the target work result 41 is in compliance with the work guide and whether it is overlapped with the work result 42 to be checked for overlapping, based on the non-compliance of the work guide of the target work result 41, In the case of rejection, additional points are not calculated even if the result of the task subject to duplicate check is provided.

The additional score is a reward for the inspector's time and effort spent in determining whether the inspection target work result and the duplicate check target work result (42) overlap. It does not give extra points.

On the other hand, in order to prevent an inspector of a crowdsourcing project from performing an unscrupulous inspection, a minimum inspection time required to perform inspection of one work result may be set.

For example, if the minimum inspection time is set to 10 seconds, the inspector must perform inspection on one work result for at least 10 seconds. And after the minimum inspection time of 10 seconds has elapsed, the inspector can complete the inspection. In other words, you can enter the inspection result.

In this case, according to an embodiment of the present invention, in step S130, the minimum inspection time is limited for each inspection target task result to request inspection, but the minimum inspection time corresponds to the basic minimum inspection time and the quantity of the duplicate check target task result. It can be determined by summing up the additional minimum inspection time.

Specifically, when 50 duplicate check target task results 42 are provided to the inspector and one duplicate check target task result 41 is provided, the time required to perform the duplicate task result inspection will be different, respectively. In other words, the greater the number of duplicate check target work results provided to the inspector, the more time the inspector has to spend on inspecting the duplicate work results.

If the same minimum inspection time is given irrespective of the number of inspections subject to overlapping check 42, and the inspector who is provided with a large amount of overlapping check subject work results 41 may lose concentration after the minimum inspection time has elapsed, and perform an easy inspection. You might be able to do it. Therefore, in order to prevent such a problem, the minimum inspection time is calculated by calculating the additional minimum inspection time corresponding to the quantity of work results subject to overlapping checks and summing it with the basic minimum inspection time to determine the minimum inspection time.

On the other hand, according to an embodiment of the present invention, the additional minimum inspection time may be calculated using Equation 2 below.

[Equation 2]

Additional minimum inspection time = Quantity of work results subject to overlapping checks x Basic minimum inspection time x predetermined ratio

FIG. 8 is a diagram for describing a method of calculating an additional minimum inspection time based on a quantity of work results subject to overlapping check according to an embodiment of the present invention.

Referring to FIG. 8, when the basic minimum inspection time is 10 seconds, the number of duplicate check target work results is 2, and the predetermined ratio is 0.1, the additional minimum inspection time corresponds to 2 seconds. At this time, the inspector will be able to input the inspection result after the minimum inspection time of 12 seconds, which is the sum of the basic minimum inspection time of 10 seconds and the additional minimum inspection time of 2 seconds, has elapsed.

Meanwhile, the predetermined ratio may be determined according to the difficulty of the project. A detailed description of how to determine the difficulty of a project will be omitted.

Meanwhile, according to an embodiment of the present invention, in the step of receiving the inspection result as passing or rejecting the inspection, the inspection result input interface may be activated after the minimum inspection time has elapsed to receive the inspection result from the inspector.

9 is an exemplary view showing a screen for reviewing a project for collecting images or moving pictures.

Referring to FIG. 9, the inspector must inspect whether the inspection target work result 41 complies with the work guide during the minimum inspection time, and check whether it overlaps with the overlapping check target work result 42. At this time, only after the minimum inspection time has elapsed, the buttons for inspecting and rejecting, an interface for inputting inspection results, are activated, so that the inspector can input the inspection results.

Meanwhile, in the above description, steps S110 to S150 may be further divided into additional steps or may be combined into fewer steps, according to an embodiment of the present invention. In addition, some steps may be omitted as necessary, or the order between steps may be changed. In addition, even if other contents are omitted, the contents of FIG. 10 to be described later may also be applied to the method of evaluating an inspector of a crowdsourcing-based project for collecting images or videos of FIGS. 1 to 9.

Hereinafter, an apparatus for evaluating a reviewer of a project for collecting images or moving pictures according to an embodiment of the present invention will be described with reference to FIG. 10.

10 is a diagram illustrating an apparatus for evaluating a reviewer of a project for collecting images or moving pictures according to an embodiment of the present invention.

Referring to FIG. 10, an inspector evaluation 200 (hereinafter, an apparatus) includes a communication module 210, a memory 220, and a processor 230.

The communication module 210 transmits a crowdsourcing-based work for one project to a plurality of workers 32 to request a task to be performed, and receives work results from the plurality of workers 32. The communication module 210 transmits the work results received from the plurality of workers 32 to the plurality of inspectors 34 to request inspection, and receives the inspection results from the plurality of inspectors 34. At this time, for the rejected inspection result, a corresponding rework is transmitted to the worker 32 to request rework, and the rework result is received from the worker 32.

The memory 220 stores a program for evaluating an inspector regarding the inspection of the duplicate work result of a crowdsourcing-based project based on the data received from the communication module 210.

The processor 230 executes a program stored in the memory 220. As the processor 230 executes the program stored in the memory 220, the method of evaluating the inspector of a crowdsourcing-based project for collecting images or videos described with reference to FIGS. 3 to 9 may be performed.

The device 200 described with reference to FIG. 10 may be provided as a component of the above-described server.

The method for evaluating a reviewer of a crowdsourcing-based project for collecting images or moving pictures according to an embodiment of the present invention described above is implemented as a program (or application) to be executed in combination with a computer, which is hardware, Can be stored in.

The above-described program includes C, C++, JAVA, Ruby, which can be read by the computer's processor (CPU) through the computer's device interface, so that the computer reads the program and executes the methods implemented as a program. It may include a code coded in a computer language such as machine language. Such code may include a functional code related to a function defining necessary functions for executing the methods, and a control code related to an execution procedure necessary for the processor of the computer to execute the functions according to a predetermined procedure. can do. In addition, these codes may further include additional information required for the processor of the computer to execute the functions or code related to a memory reference to which location (address address) of the internal or external memory of the computer should be referenced. have. In addition, when the processor of the computer needs to communicate with any other computer or server in a remote in order to execute the functions, the code is It may further include a communication-related code for whether to communicate, what kind of information or media to be transmitted and received during communication.

The stored medium is not a medium that stores data for a short moment, such as a register, cache, memory, etc., but a medium that stores data semi-permanently and can be read by a device. Specifically, examples of the storage medium include, but are not limited to, ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, and the like. That is, the program may be stored in various recording media on various servers to which the computer can access, or on various recording media on the user's computer. In addition, the medium may be distributed over a computer system connected through a network, and computer-readable codes may be stored in a distributed manner.

The above description of the present invention is for illustrative purposes only, and those of ordinary skill in the art to which the present invention pertains will be able to understand that other specific forms can be easily modified without changing the technical spirit or essential features of the present invention. will be. Therefore, it should be understood that the embodiments described above are illustrative in all respects and not limiting. For example, each component described as a single type may be implemented in a distributed manner, and similarly, components described as being distributed may also be implemented in a combined form.

The scope of the present invention is indicated by the claims to be described later rather than the detailed description, and all changes or modified forms derived from the meaning and scope of the claims and their equivalent concepts should be construed as being included in the scope of the present invention. do.

10: requester
20: service provider
30: public
32: worker
34: inspector
41: Inspection target work result
42: Duplicate check target operation result

Claims (11)

As a method performed by a computer,
Allocating a plurality of tasks of a crowdsourcing-based project (hereinafter, "project") to a plurality of workers and requesting the task to be performed;
Receiving a plurality of job results from the plurality of workers;
Allocating the plurality of work results to a plurality of inspectors and requesting to perform the inspection;
Receiving a plurality of inspection results for the plurality of work results from the plurality of inspectors as passing or rejecting the inspection; And
Comprising the step of calculating a proficiency score by evaluating the proficiency level of each inspector based on the inspection result input by the inspector,
The above task is to collect and submit images or videos that meet certain conditions,
The step of requesting to perform the inspection,
Extracting one or more previous work results of a worker who has input the work results to be inspected;
Providing the inspector with at least a portion of the inspection subject job result and one or more previous job results as a duplicate check subject job result; And
And requesting the inspector to perform an inspection as to whether the inspection target work result complies with the work guide and whether or not the overlapping check target work result overlaps,
The step of calculating the inspection proficiency score may further include evaluating the inspection proficiency level of each inspector based on the quantity of the duplicate check target work results, and calculating the inspection proficiency score,
The step of requesting to perform the inspection,
Request for inspection by limiting the minimum inspection time for each inspection target task result, but the minimum inspection time is determined by summing the basic minimum inspection time and an additional minimum inspection time corresponding to the quantity of the duplicate check target job results,
A method of evaluating inspectors for crowdsourcing-based projects for collecting images or videos. The method of claim 1,
The step of calculating the inspection proficiency score includes adding a basic score corresponding to the number of inspection results input by the inspector and an additional score corresponding to the quantity of the duplicate check target task result for each inspection result to calculate the inspection proficiency score. Calculating,
A method of evaluating inspectors for crowdsourcing-based projects for collecting images or videos. The method of claim 2,
The step of calculating the test proficiency score may include calculating the test proficiency score using the following formula,
Base score = number of inspection results entered by inspector x unit score
Additional score = Quantity of work results subject to duplicate check x unit score x predetermined percentage
A method of evaluating inspectors for crowdsourcing-based projects for collecting images or videos. The method of claim 2,
In the step of calculating the test proficiency score, the basic score and the test proficiency score are not calculated for the test result inputted by the tester that is rejected,
A method of evaluating inspectors for crowdsourcing-based projects for collecting images or videos. The method of claim 1,
The providing of the duplicate check target task result includes extracting a previous task result that has passed the inspection among the one or more previous task results, and provides at least some of the previous task result that has passed the inspection as the duplicate check target task result,
A method of evaluating inspectors for crowdsourcing-based projects for collecting images or videos. The method of claim 1,
Providing the duplicate check target operation result,
Measuring a similarity between the inspection target work result and the one or more previous work results; And
Including the step of providing a previous work result of the one or more previous work results in which the similarity is greater than or equal to a reference value as the duplicate check target work result,
A method of evaluating inspectors for crowdsourcing-based projects for collecting images or videos. The method of claim 6,
In the measuring of the similarity, each feature map of the inspection target work result and the one or more previous work results is extracted, each feature map is embedded into an n-dimensional (n is a natural number of 2 or more) vectors, and each Measuring the similarity by measuring the Euclidean distance between vectors,
A method of evaluating inspectors for crowdsourcing-based projects for collecting images or videos. delete The method of claim 1,
The additional minimum inspection time is calculated using the following formula,
Additional minimum inspection time = Quantity of work results subject to overlapping checks x Basic minimum inspection time x predetermined ratio
A method of evaluating inspectors for crowdsourcing-based projects for collecting images or videos. The method of claim 1,
In the step of receiving the inspection result as passing or rejecting the inspection, the inspection result input interface is activated after the minimum inspection time has elapsed to receive the inspection result from the inspector,
A method of evaluating inspectors for crowdsourcing-based projects for collecting images or videos. It is combined with a computer and stored in a computer-readable recording medium to execute the method for evaluating the inspector of a crowdsourcing-based project for collecting the image or video of any one of claims 1 to 7, 9, and 10. Computer program.

Images