KR102164769B1 - Method for measuring competence using the number of inspection pass of crowdsourcing based project for artificial intelligence training data generation - Google Patents

Abstract

Provided is a method of measuring competence by using an inspection pass of a crowdsourcing-based project for creating artificial intelligence learning data. The method includes the steps of: requesting job fulfillment by assigning a plurality of jobs of a crowdsourcing-based project (hereinafter referred to as ″project″) to a plurality of workers; receiving a plurality of job results from the plurality of workers; requesting inspection fulfillment by assigning the plurality of job results to a plurality of inspectors; receiving an inspection pass request or a plurality of inspection results about the plurality of job results from the plurality of inspectors; and measuring the competence of each of the workers based on the number of inspection pass requests and the inspection results. According to the present invention, the competence of a worker can be precisely measured.

Description

A method for measuring proficiency using the inspection pass of a crowdsourcing-based project for generating artificial intelligence learning data {METHOD FOR MEASURING COMPETENCE USING THE NUMBER OF INSPECTION PASS OF CROWDSOURCING BASED PROJECT FOR ARTIFICIAL INTELLIGENCE TRAINING DATA GENERATION}

The present invention relates to a proficiency measurement method using a crowdsourcing-based project inspection pass for generating artificial intelligence learning data.

Recently, more and more companies collect and process large amounts of data on a crowdsourcing basis that engages the general public in some process of corporate activities. In other words, by opening a project and allowing the general public, that is, workers to participate in the project, necessary information is collected through the work results completed by the workers.

Specifically, when one project is opened, a plurality of tasks are assigned to each of the plurality of workers. Each worker performs a plurality of tasks assigned to it, and provides the task results. Thereafter, a plurality of inspection tasks are assigned to each of the plurality of inspectors, and each inspector performs a plurality of inspection tasks assigned to them.

However, a project contains a number of tasks, and all of these tasks are not composed of the same level of difficulty. Therefore, if the inspector is assigned a task with a high degree of difficulty, the inspector will be able to accurately determine the task result. The inspection that has not been done, or the inspection after long agony. As a result, there is a problem in that the quality of inspection is deteriorated and an inspection bottleneck occurs.

On the other hand, when measuring the proficiency of the worker, information such as the number of tasks performed by the worker is used.At this time, even if the task within a single project, the degree of difficulty may be different for each task. It is difficult to do. This is because, when a worker who performed a task with high difficulty and a worker who performed a task with low difficulty performed the same number of tasks, it is determined that the two workers have the same skill level.

Unexamined Patent Publication No. 10-2014-0095956, 2014.08.04.

The problem to be solved by the present invention is to provide a method for measuring proficiency using a crowdsourcing-based project inspection pass for generating artificial intelligence learning data.

However, the problem to be solved by the present invention is not limited to the problem as described above, and other problems may exist.

The proficiency measurement method using a crowdsourcing-based project inspection pass according to an aspect of the present invention for solving the above-described problems is performed by assigning a plurality of tasks of a crowdsourcing-based project (hereinafter, a project) to a plurality of workers. Requesting execution, receiving a plurality of work results from the plurality of workers, assigning the plurality of work results to a plurality of inspectors and requesting to perform the inspection, and the plurality of work results from the plurality of inspectors Receiving a plurality of inspection results or a request for an inspection pass, and measuring the proficiency level for each worker based on the inspection result and the number of requests for the inspection pass, and the plurality of inspection results In the step of assigning to and requesting to perform the inspection, the task result inputted with the inspection pass request is withdrawn, reassigned to another inspector to request inspection, and the step of measuring the proficiency level of each worker includes: Calculating a score corresponding to a plurality of inspection results for the results of a plurality of work performed by each worker for each worker, and calculating a score corresponding to the number of requests for inspection passes for the results of a plurality of work performed by each worker for each worker. And measuring the proficiency level for each worker using a score corresponding to the inspection result and a score corresponding to the number of requests for the inspection pass.

In some embodiments of the present invention, the calculating a score corresponding to the inspection result for each worker includes calculating a first score corresponding to the number of inspection passes among the plurality of work results performed by each worker, and Calculating a second score corresponding to the number of rejections for the results of a plurality of tasks performed by each worker, and subtracting the second score from the first score to obtain a score corresponding to the inspection result for each worker. It may include the step of calculating.

In some embodiments of the present invention, the step of calculating a score corresponding to the number of requests for the inspection pass for each worker includes: extracting a job result in which the inspection result passes the inspection among the plurality of job results performed by each worker; and And calculating the number of requests for the inspection pass for the work result in which the inspection result passed the inspection, and calculating a score corresponding to the number of requests for the inspection pass for the work result in which the inspection result passed inspection for each worker. Can include.

In some embodiments of the present invention, the proficiency level may be calculated using the following equation.

1st score = number of inspection passes x 1st unit score

2nd score = number of rejections x 2nd unit score

Score corresponding to the inspection result = 1st score-2nd score

Score corresponding to the number of inspection pass requests = number of inspection pass requests x first unit score x predetermined first ratio

Proficiency = Score corresponding to the test result + score corresponding to the number of requests for test pass

In some embodiments of the present invention, the number of rejections may include a cumulative number of rejections related to the same task.

In some embodiments of the present invention, the number of times per inspector can input the inspection pass request may be limited to a predetermined value.

In some embodiments of the present invention, further comprising the step of measuring the proficiency level for each inspector based on the inspection result and the number of requests for the inspection pass, wherein the measuring proficiency level for each inspector comprises: Calculating a score corresponding to a plurality of inspection results for the results of a plurality of inspections performed, and calculating a score corresponding to the number of inspection pass requests for the results of a plurality of work performed by each inspector for each inspector And, for each inspector, measuring the proficiency level using a score corresponding to the inspection result and a score corresponding to the number of requests for the inspection pass.

In some embodiments of the present invention, in the step of measuring the proficiency level for each inspector, the proficiency level for each inspector may be measured by dividing the inspection result for the initial work result and the inspection result for the rework result.

In some embodiments of the present invention, the step of calculating a score corresponding to the inspection result for each inspector may include calculating a score corresponding to the number of inspection passes among a plurality of inspection results performed by each inspector. I can.

In some embodiments of the present invention, the proficiency level may be calculated using the following equation.

Score corresponding to the inspection result = number of inspection passes x 3rd unit score

Score corresponding to the number of inspection pass requests = number of inspection pass requests x third unit score x predetermined second ratio

Proficiency = Score corresponding to the test result + score corresponding to the number of requests for test pass

A computer program according to another aspect of the present invention for solving the above-described problem is combined with a computer that is hardware to execute a proficiency measurement method using the inspection pass of the crowdsourcing-based project, and is stored in a computer-readable recording medium. .

Other specific details of the present invention are included in the detailed description and drawings.

According to the present invention described above, the inspection quality can be improved and the inspection bottleneck can be alleviated by receiving an inspection pass request from the inspector for a work result that is judged not to be inspected by the inspector.

In addition, by giving an additional score to the operator's proficiency based on the number of inspection passes for the work result performed by the operator, the operator's proficiency can be accurately measured.

In addition, it is possible to accurately measure the proficiency level of the inspector by assigning an additional score to the proficiency level of the inspector based on the number of inspection passes for the result of the work performed by the inspector.

The effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the following description.

1 is a conceptual diagram of a crowdsourcing service according to an embodiment of the present invention.
2 is a flowchart illustrating a crowdsourcing-based project progress process according to an embodiment of the present invention.
3 is a flowchart of a method for measuring proficiency of a worker using a crowdsourcing-based project inspection pass according to an embodiment of the present invention.
4 is a flowchart illustrating reassignment of a work result requested for a verification pass according to an embodiment of the present invention.
5 is a flowchart of a specific method of S150 shown in FIG. 3.
6 is an exemplary view for explaining the calculation of the score corresponding to the inspection result of the operator according to an embodiment of the present invention.
7 is an exemplary diagram for explaining the calculation of a score corresponding to the number of requests for a verification pass by an operator according to an embodiment of the present invention.
8 is a flow chart of a method for measuring proficiency of an inspector using an inspection pass of a crowdsourcing-based project according to an embodiment of the present invention.
9 is a flowchart of a detailed method of S160 shown in FIG. 8.
10 is an exemplary view for explaining calculation of a score corresponding to a result of an inspection by an inspector according to an embodiment.
11 is an exemplary diagram for explaining calculation of a score corresponding to the number of requests for an inspection pass by an inspector according to an embodiment of the present invention.
12 is a diagram for explaining a proficiency measuring apparatus using a crowdsourcing-based project inspection pass according to an embodiment of the present invention.

Advantages and features of the present invention, and a method of achieving them will become apparent with reference to the embodiments described below in detail together with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but may be implemented in a variety of different forms, only the present embodiments are intended to complete the disclosure of the present invention, It is provided to fully inform the technician of the scope of the present invention, and the present invention is only defined by the scope of the claims.

The terms used in the present specification are for describing exemplary embodiments and are not intended to limit the present invention. In this specification, the singular form also includes the plural form unless specifically stated in the phrase. As used in the specification, “comprises” and/or “comprising” do not exclude the presence or addition of one or more other elements other than the mentioned elements. Throughout the specification, the same reference numerals refer to the same elements, and “and/or” includes each and all combinations of one or more of the mentioned elements. Although "first", "second", and the like are used to describe various elements, it goes without saying that these elements are not limited by these terms. These terms are only used to distinguish one component from another component. Therefore, it goes without saying that the first component mentioned below may be the second component within the technical idea of the present invention.

Unless otherwise defined, all terms (including technical and scientific terms) used in the present specification may be used as meanings that can be commonly understood by those of ordinary skill in the art to which the present invention belongs. In addition, terms defined in a commonly used dictionary are not interpreted ideally or excessively unless explicitly defined specifically.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a conceptual diagram of a crowdsourcing service according to an embodiment of the present invention.

Referring to FIG. 1, a crowdsourcing service is performed by being composed of a client 10, a service provider 20, and a public 30.

The sponsor 10 refers to a company or individual who requests a crowdsourcing-based project (hereinafter, referred to as a project).

The requester 10 requests a project for the purpose of collecting source data or annotating data for generating artificial intelligence learning data. The data generated through the project can be used as learning data for arbitrary machine learning such as supervised learning, unsupervised learning, and reinforcement learning. Collection of source data refers to collecting raw data such as recorded voice collection and photo collection. Data annotation refers to inputting relevant annotation data into source data such as text, photos, and videos. For example, the data annotation may include finding an entity in a given fingerprint or finding a similar sentence, but is not limited thereto. Meanwhile, the types of the above-described projects are only one embodiment, and various projects may be handled in the present invention according to the design of the client.

The service provider 20 refers to a company that provides crowdsourcing services.

When the service provider 20 receives a request for a project for a product or service from the client 10, the service provider 20 allocates a task for the project to the general public 30 and receives the work result from the public 30. Thereafter, the final product extracted based on the work result is provided to the client 10.

At this time, the service provider 20 provides a crowdsourcing service to the client 10 and the public 30 through a crowdsourcing platform (hereinafter, the platform). That is, when the service provider 20 receives a request for a project from the client 10, the service provider 20 opens the project on the platform. Thereafter, when the work result for the open project is received from the public 30, the project is terminated on the platform, and the final product may be extracted and provided to the client 10.

The public 30 refers to the general public participating in the project open on the platform. Here, the public 30 may participate in a project opened on the platform through an application or website provided by the service provider 20.

The public 30 is composed of a worker 32 and an inspector 34.

The worker 32 decides to participate in a specific project among a plurality of projects open on the platform. Thereafter, the operator 32 performs an operation such as collection of source data or data annotation, and transmits it to the platform.

The inspector 34 decides to participate in a specific project among a plurality of projects open on the platform. Thereafter, the inspector 34 inspects the result of the work performed by the operator 32. As a result of performing the inspection, the inspector 34 may perform inspection pass processing or rejection processing, and input a rejection reason during rejection processing. In the case of passing the inspection, it is not necessary to rework and re-examination due to this, so passing inspection has the same meaning as completion of inspection.

2 is a flowchart illustrating a crowdsourcing-based project progress process according to an embodiment of the present invention.

First, the requester 10 requests one or more projects to the service provider 20 (S11).

Thereafter, the service provider 20 opens the requested project on the platform (S12). At this time, the service provider 20 may determine the grade in consideration of the difficulty of the project before opening the project. That is, depending on the difficulty level, it is possible to determine which level or higher public 30 to expose the project. Accordingly, it is possible to increase the reliability of the work result of the project.

Thereafter, the service provider 20 requests the work by assigning the work to the workers 32 of the corresponding level or higher according to the level of the project (S13).

Thereafter, the worker 32 performs the assigned task (S14). In this case, the worker 32 may input the reason for the inability to work without performing the work for a work in which the work itself is impossible for some reason.

Thereafter, the service provider 20 receives the work result from the worker 32 (S15), and assigns the work result to the inspector 34 to request the inspection (S16).

Thereafter, the inspector 34 performs the assigned inspection (S17). At this time, the inspector 34 decides to pass the inspection when it is determined that the work has been properly performed, and if it is determined that the inspection work is wrong, the inspector 34 rejects it. During rejection processing, the inspector 34 inputs the rejection reason for what reason the job was judged to be wrong.

Thereafter, the service provider 20 receives the inspection result from the inspector 34 (S18).

If the inspection result passes the inspection, the service provider 20 uses the work result as valid data, and based on this, extracts the final product at the end of the project.

When the inspection result is rejection processing, the service provider 20 may internally perform the inspection again, or assign the work to the worker 32 again to perform the rework. When reworking, re-examination by an inspector is required.

Thereafter, when the project period ends or sufficient valid data is secured, the service provider 20 terminates the project (S19), calculates a final result based on the secured valid data, and provides it to the client 10 ( S20).

At this time, before the end of the project, the service provider 20 evaluates the performance results of the worker 32 and the inspector 34, calculates the work cost and the inspection cost according to the evaluation, to the worker 32 and the inspector 34. give.

In FIGS. 1 and 2, it is simply expressed as a requester 10, a service provider 20, an operator 32, and an inspector 34, but these are smartphones, tablets, PDAs, laptops, desktops, operated by each participant. It means a computer device such as a server or a telecommunication device.

Hereinafter, a method of measuring a worker's proficiency will be described with reference to FIGS. 3 to 7.

3 is a flowchart of a method for measuring proficiency of a worker using a crowdsourcing-based project inspection pass according to an embodiment of the present invention. 4 is a flowchart illustrating reassignment of a work result requested for a verification pass according to an embodiment of the present invention. 5 is a flowchart of a specific method of S150 shown in FIG. 3. 6 is an exemplary view for explaining the calculation of the score corresponding to the inspection result of the operator according to an embodiment of the present invention. 7 is an exemplary diagram for explaining the calculation of a score corresponding to the number of requests for a verification pass by an operator according to an embodiment of the present invention.

Meanwhile, the steps shown in FIGS. 3 and 5 may be understood as being performed by a platform server (hereinafter, referred to as a server) operated by the service provider 20, but are not limited thereto.

In addition, a plurality of workers 32 or a plurality of inspectors 34 perform work using a predetermined terminal device. The terminal device of the operator 32 or the inspector 34 may be a computer device or a telecommunication device such as a smartphone, tablet, PDA, laptop, desktop, etc., but is not limited thereto.

First, referring to FIG. 3, the server allocates a plurality of tasks of a crowdsourcing-based project (hereinafter, a project) to a plurality of workers and requests the task to be performed (S110). The server receives a plurality of job results from a plurality of workers (S120). The server allocates a plurality of work results to a plurality of inspectors and requests the inspection to be performed (S130). The server receives a plurality of inspection results or an inspection pass request for a plurality of work results from a plurality of inspectors (S140).

An embodiment of the present invention is characterized in that it is possible not only to receive an inspection result from the inspector 34, but also to receive an input of an inspection pass request for passing a work result that is difficult for the inspector 34 to perform the inspection due to high difficulty of the operation. To do.

In the conventional case, if one task result was assigned, the inspector 34 could not be assigned another task result until the inspection of the assigned task result was completed. Therefore, if the inspector 34 is difficult to perform the inspection due to the high degree of difficulty in the work, the inspector 34 spends a lot of time without being assigned other work results and has to inspect the work result, which in turn is the inspector 34 While increasing the fatigue of the project, it acted as a problem of losing the opportunity to receive the inspection cost by inspecting other work results at the same time, and also acted as a problem of delaying the schedule of the entire project.

In order to solve this problem, an embodiment of the present invention may prevent the inspector 34 from inspecting the task result assigned through the inspection pass request, even if the inspector 34 is assigned the task result of high difficulty.

In step S140, when the server receives the inspection result for the work result from the inspector 34 as inspection pass or rejection, the steps S110 to S140 are the same as steps S13 to S18 described above with reference to FIG. 2. Therefore, a detailed description will be omitted.

In step S140, when the server receives a request for an inspection pass for a work result from the inspector 34, the server withdraws the assignment of the job result inputted with the inspection pass request, and requests the inspection by reassigning it to another inspector.

4, when the server assigns the work result input from worker A to the inspector A, and the inspector A inputs a request for the inspection pass for the work result, the server withdraws the assignment of the work result to the inspector A, and inspector B Reassign to. Inspector B inspects the assigned work result and inputs the inspection result.

That is, each of the plurality of inspectors 34 inspects the task result and provides the inspection result to the server when it is determined that correct inspection is possible for the task result assigned to the person.

However, each of the plurality of inspectors 34 requests an inspection pass for the job result when it is determined that the correct inspection is impossible (high difficulty of the job) with respect to the job result assigned to him/her. For this reason, the high-difficulty work result is inspected by an inspector having an inspection ability corresponding to the difficulty, and the inspection quality is improved.

Referring back to FIG. 3, the server measures the proficiency level for each worker based on the inspection result and the number of inspection pass requests (S150).

Referring to FIG. 5, the server calculates a score corresponding to a plurality of inspection results for a plurality of work results performed by each worker for each worker (S151).

The score corresponding to the inspection result is calculated based on the number of work results that have passed the inspection (hereinafter, the number of inspection passes) and the number of times the work result has been rejected (hereinafter, referred to as rejection number) among the work results of the worker 32.

Specifically, the server calculates a first score corresponding to the number of inspection passes among the results of a plurality of work performed by each worker.

The first score is calculated by multiplying the number of inspection passes by the operator 32 by the first unit score. Here, the first unit score may be a unit cost of work, but is not limited thereto.

The server calculates a second score corresponding to the number of rejections for the results of a plurality of tasks performed by each worker.

The second score is calculated by multiplying the number of rejections of the worker 32 by the second unit score. Here, the second unit score may be a work unit cost or a value corresponding to a predetermined ratio of the work unit cost, but is not limited thereto.

Depending on the embodiment, the second score may be calculated differently according to the number of inspection passes by the worker 32.

The server compares the number of inspection passes by the operator 32 with the number of proficiency points, and if the number of inspection passes is less than the number of proficiency points (i.e., before the number of inspection passes exceeds the proficiency point), a second weight is not applied. Calculate the score.

Here, the number of proficiency points is a reference value at which a project is opened and the worker 32 performs a task to some extent, and then begins to become proficient in performing a task, and is predetermined by the server.

According to individual differences and differences in experience for each worker 32, there may be many differences in the ability at the beginning of work. Also, depending on the interpretation of the work guide at the beginning of the work, the work may be performed incorrectly. Therefore, before the number of proficient times, it is determined that the operator 32 is not yet proficient in the work, and a second score is calculated without applying a weight to the number of rejections.

In contrast, the server compares the number of inspection passes by the operator 32 with the number of proficiency points, and if the number of inspection passes is greater than the proficiency point number (that is, after the number of inspection passes exceeds the proficiency point number), the operator 32 The second score is calculated by applying a weight after judging that is sufficiently good at work.

In this case, the weight may be set differently based on the average rejection rate. The average rejection rate may be set by the server before the project is opened, and may be set to a value calculated through, for example, a pilot project, but is not limited thereto.

The server compares the rejection rate of the worker 32 and the average rejection rate when the number of inspection passes by the worker 32 exceeds the number of proficiency points, and when the rejection rate of the worker 32 is greater than the average rejection rate, the server The second score may be calculated by applying the weight of the first value.

On the other hand, when the rejection rate of the worker 32 is less than the average rejection rate, the server may calculate the second score by applying the weight of the second value.

Here, the first value is set to a value larger than the second value. This is to ensure that a larger weight is applied when the worker 32's rejection rate is greater than the average rejection rate.

In the following, assuming that the worker 32 participates in the project and performs 10 tasks, and the work unit price is 1,000 won, the number of inspection passes and rejections of the worker 32 is calculated, and the calculated number of inspection passes and rejections Using the number of times, the method should be specifically explained to the score corresponding to the inspection result.

The number of inspection passes means the number of work results in which the final inspection result has passed inspection. That is, it includes both the number of work results in which the initial inspection result passed the inspection and the number of work results in which the re-examination result passed the inspection.

Referring to FIG. 6, 8 of the total 10 work results are verified as passing the inspection, and the number of passing the inspection by the worker A is calculated as 8 cases.

At this time, it can be seen that the work results 1, 2, 7, 10 passed the inspection at the time of the initial inspection, but the work results 3 to 6 passed the inspection at the time of re-review after being rejected once or more.

Accordingly, when the first score is the unit cost of work, the first score of the worker A may be calculated as 1,000x8=8,000 points.

Meanwhile, the number of rejections is calculated including the cumulative number of rejections for the same task. For example, if the same job is rejected three times (due to repeated mistakes by the operator), the cumulative number of rejections for that job is calculated as 3.

Each task is assigned a DataID, and since the identity of the tasks can be determined based on whether the data IDs are the same, the server can calculate the cumulative number of rejections for the same task.

Referring to FIG. 6, out of a total of 10 work results, 8 work results were finally passed the inspection, and 2 work results were rejected. However, the number of rejections of the worker 32 A is calculated as eight times, not two. Out of work results 1 to 7 that passed the final inspection, work results 3 to 5 were rejected once each before finally passing the inspection, and work result 6 was rejected twice before finally passing the inspection. In addition, among the rejected work results 8 and 9, since the work result 8 has already been rejected once, the number of rejections of worker A is calculated as 8 times including all the cumulative rejections of each job.

Accordingly, when the second score is a value (eg, half) of a predetermined ratio of the work unit cost, the second score of the worker A may be calculated as 500×8=4,000 points.

Thereafter, the server calculates a score corresponding to the inspection result for each worker by subtracting the second score from the first score.

As described above, when the first score of worker A is calculated as 8,000 points and the second score is 4,000 points, the score corresponding to the inspection result of worker A is calculated as 8,000-4,000=4,000 points.

Referring again to FIG. 5, the server calculates a score corresponding to the number of requests for the inspection pass for the results of a plurality of tasks performed by each worker for each worker (S152).

The score corresponding to the number of requests for the inspection pass is calculated based on the number of requests for the inspection pass for the result of the work passed by the operator 32.

Specifically, the server extracts a work result in which the inspection result passes the inspection from among the plurality of work results performed by each worker.

In other words, the server excludes the result of a job that is rejected and needs rework from among the results of a plurality of jobs performed by the operator, and the result of the work whose initial inspection result passed the inspection and the final inspection result after one or more rejections passed the inspection. Extract.

Referring to FIG. 7, it can be seen that work results 1, 2, 7, and 10 indicate that the initial inspection result passed the inspection, and work results 3 to 6 indicate that the final inspection result passed the inspection after one or more rejections.

Thereafter, the server calculates the number of times the inspection pass is requested for the work result in which the inspection result passes the inspection.

That is, the server calculates the total number of inspection passes requested for the work result in which the inspection result of the worker 32 has passed the inspection. At this time, the number of inspection passes for the job result that was rejected before finally passing the inspection is calculated. Even if it was rejected, the worker who entered the work result (32) performed the work of a high degree of difficulty (so that it would pass the inspection), so for cases that passed the final inspection, the inspection pass for the job result that was rejected before the final inspection passed. It also takes into account the number of times.

Referring to FIG. 7, work results 1 and 2 each passed inspection at the initial inspection after at least one inspection pass. Work result 3 was returned once and passed the inspection. At this time, there was no inspection pass for the rejected work result 3', and there was one inspection pass for the work result 3 reworked by worker A after being rejected. Work result 4 was returned once and passed the inspection. At this time, there was one inspection pass for the rejected work result 4', and there was no inspection pass for the work result 4 reworked by worker A after being rejected. Work result 5 passed the inspection after being rejected once.At this time, there was one inspection pass for the rejected work result 5', and there was one inspection pass for the work result 5 reworked by worker A after returning. Work result 6 passed the inspection after being returned twice.At this time, there was one inspection pass for the work result 6'' that was returned first, and after the first rejection, worker A reworked it, but it was rejected again second time. There was also one inspection pass for', and there was one inspection pass for work result 6, which was reworked by worker A after the second return. As a result of the work, 7 and 10 were inspected at the initial inspection without passing the inspection pass.

Therefore, the number of inspection passes for the work result passed by the operator A is calculated as a total of 10 times.

The server calculates a score corresponding to the number of requests for the inspection pass for the work result in which the inspection result passed the inspection for each worker.

Specifically, a score corresponding to the number of verification pass requests is calculated by multiplying a value obtained by multiplying the first unit score and a predetermined first ratio by the number of verification pass requests.

When the first unit score is the work unit cost, for example, 1,000 won, and the predetermined first ratio is 10%, as described above, if the number of requests for the inspection pass by worker A is calculated as 10, the number corresponding to the number of requests for the inspection pass The score will be calculated as (1,000x0.1)x10=1,000 points. In this case, the predetermined first ratio is set by the server, and although it has been described as 10 percent above, it is not limited thereto.

Referring back to FIG. 5, the server measures the proficiency level for each worker using a score corresponding to the inspection result and the number of requests for the inspection pass (S153).

The worker's proficiency is calculated using the formula below.

-1st score = number of inspection passes x 1st unit score

-2nd score = number of rejections x 2nd unit score

-Score corresponding to the inspection result = 1st score-2nd score

-Score corresponding to the number of inspection pass requests = number of inspection pass requests x first unit score x predetermined first ratio

-Proficiency level = score corresponding to the test result + score corresponding to the number of test pass requests

The method of calculating the first score, the second score, the score corresponding to the test result, and the score corresponding to the number of requests for the test pass is as described above.

Therefore, as described above, if the score corresponding to the inspection result is calculated as 4,000 points, and the score corresponding to the number of requests for the inspection pass is calculated as 1,000 points, the proficiency level of the worker is calculated as 5,000 points.

As described above, by additionally assigning a score regarding the number of inspection passes to the score according to the inspection result for each work result of the worker 32, the worker 32 who worked on the work result passed the inspection receives more points. It is possible to more accurately measure the proficiency of each worker 32 by receiving it.

The reason that the result of the work performed by the worker 32 is given an additional score as much as the number of passes of the inspection is because the worker 32 has performed a task with a difficulty that is difficult for even the inspector 34 to inspect. This is because it is determined that the worker 32 is more proficient than the other workers 32.

Hereinafter, a method of measuring the proficiency level of an inspector will be described with reference to FIGS. 8 to 11.

8 is a flow chart of a method for measuring proficiency of an inspector using an inspection pass of a crowdsourcing-based project according to an embodiment of the present invention. 9 is a flowchart of a detailed method of S160 shown in FIG. 8. 10 is an exemplary view for explaining calculation of a score corresponding to a result of an inspection by an inspector according to an embodiment. 11 is an exemplary diagram for explaining calculation of a score corresponding to the number of requests for an inspection pass by an inspector according to an embodiment of the present invention.

Meanwhile, the steps shown in FIGS. 8 and 9 may be understood as being performed by a platform server (hereinafter, referred to as a server) operated by the service provider 20, but are not limited thereto.

In addition, a plurality of workers 32 or a plurality of inspectors 34 perform work using a predetermined terminal device. The terminal device of the operator 32 or the inspector 34 may be a computer device or a telecommunication device such as a smartphone, tablet, PDA, laptop, desktop, etc., but is not limited thereto.

Referring to FIG. 8, the server assigns a plurality of tasks of a crowdsourcing-based project (hereinafter, a project) to a plurality of workers and requests the task to be performed (S110). The server receives a plurality of job results from a plurality of workers (S120). The server allocates a plurality of work results to a plurality of inspectors and requests the inspection to be performed (S130). The server receives a plurality of inspection results or an inspection pass request for a plurality of work results from a plurality of inspectors (S140). Steps S110 to S140 are overlapped with those described above, so a detailed description thereof will be omitted. That is, in order to measure the proficiency of the operator and the inspector, the same data (that is, the inspector's inspection result and the number of inspection pass requests for the work result) are used.

The server measures the proficiency level for each inspector based on the inspection result and the number of inspection pass requests (S160).

Referring to FIG. 9, the server calculates a score corresponding to a plurality of inspection results for a plurality of work results performed by each inspector for each inspector (S161).

Specifically, the server calculates a score corresponding to the number of inspection passes among the plurality of inspection results performed by each inspector.

The score corresponding to the inspection result may be calculated by multiplying the number of inspection passes by the inspector 34 by the third unit score. Here, the third unit score may be an inspection unit price, but is not limited thereto.

Referring to FIG. 10, when inspector A performed a total of 6 inspections, 5 cases were inspected as passing and one case was inspected as rejection. Therefore, the number of inspection passes by inspector A is calculated as five.

Accordingly, when the third unit score is the inspection unit price, for example, 1,000 won, a score corresponding to the inspection result of inspector A may be calculated as 1,000x5=5,000 points.

Referring back to FIG. 9, the server calculates a score corresponding to the number of requests for the inspection pass for the results of a plurality of tasks performed by each inspector for each inspector (S162).

Specifically, a score corresponding to the number of verification pass requests is calculated by multiplying a value obtained by multiplying the third unit score and a predetermined second ratio by the number of verification pass requests.

The number of inspection passes of the operator 32 is different from that calculated only for the work results that have passed the inspection of the operator 32, and the number of inspection passes of the inspector 34 is calculated for all work results that the inspector 34 performs inspection. .

That is, the number of inspection passes of the inspector 34 is calculated including the number of inspection passes for all work results, whether the inspection result passes or rejects the inspection. If the inspector 34 inspects the work results that have passed the inspection more than once, the inspector 34 has inspected the work results passed by other inspectors regardless of the inspection result (pass or reject) of the work result. Therefore, it is to consider the number of inspection passes for all work results.

In addition, the server divides the inspection result for the initial job result and the inspection result for the rework result, and calculates the number of inspection pass requests for each job result.

In other words, because the inspector for the initial work result and the inspector for the rework result may be different depending on the inspection pass, the inspection result for the initial work result and the inspection result for the rework result are classified and the inspection pass for each work result. By calculating the number of requests, it is possible to accurately measure the proficiency of each inspector 34.

Referring to FIG. 11, when the number of passes of the inspection result for the initial work result and the inspection result for the rework result are separately calculated, the number of inspection passes of the inspector A is calculated as six times.

Inspector A performed the initial inspection on 1, 2, 4'as a result of the initial work. And inspector A performed re-inspection for 3, 4, 5, and 6 as a result of the rework. At this time, inspector A performed both 4 and 4'as a result of the work. This means that the inspector A inspected the initial work result 4'as a rejection, and the rework result 4 reworked by the worker 32 was inspected by passing the inspection.

When the number of inspection passes for the initial work result 1, 2, 4'and rework result 3, 4, 5, and 6 are calculated, the number of inspection passes for the initial work result is 1, and the number of inspection passes for the first work result 2 1, the number of inspection passes for the rework result 3 is 1, the number of inspection passes for the initial work result 4'is 1, the number of inspection passes for the rework result 4 is 0, and the number of inspection passes for the rework result 5 As a result of reworking 1, the number of inspection passes for 6 is calculated as 1, and when this is summed up, the number of inspection passes for inspector A is calculated as 6.

Therefore, if the third unit score is the inspection unit price, for example, 1,000 won, and the second predetermined ratio is 10%, as described above, if the number of inspection pass requests by inspector A is calculated as 6, the number of inspection pass requests The corresponding score will be calculated as (1,000x0.1)x6=600 points. In this case, the predetermined second ratio is set by the server, and although it has been described as 10 percent above, it is not limited thereto.

Referring to FIG. 9 again, the server measures proficiency by using a score corresponding to the test result and the number of requests for the test pass for each tester (S163).

The proficiency level of the inspector is calculated using the formula below.

-Score corresponding to the inspection result = number of inspection passes x 3rd unit score

-Score corresponding to the number of inspection pass requests = number of inspection pass requests x third unit score x predetermined second ratio

-Proficiency level = score corresponding to the test result + score corresponding to the number of test pass requests

The method of calculating the score corresponding to the inspection result and the number of requests for the inspection pass is as described above.

Therefore, as described above, if the score corresponding to the inspection result is calculated as 5,000 points, and the score corresponding to the number of requests for the inspection pass is calculated as 600 points, the proficiency level of the inspector is calculated as 5,600 points.

On the other hand, the number of inspection passes of the first work result 3', the first work result 5', the first work result 6', and the rework result 6'shown in FIG. 11 is based on the proficiency of the inspector 34 who inspected each work result. Will be applied.

In other words, the number of inspection passes of the initial work result 3'is applied to the proficiency level of inspector B, who inspects the initial work result, and the number of inspection passes in the initial work result 5'is applied to the proficiency level of inspector C, who inspects the initial work result. The number of inspection passes in the initial work result 6'' is applied to the proficiency level of inspector C who has inspected the initial work result, and the number of inspection passes in the rework result 6'is applied to the proficiency level of B who has inspected the rework result. It becomes.

As described above, by additionally assigning a score regarding the number of inspection passes to the score according to the inspection result for each work result of the inspector 34, the inspector 34 who inspects the inspection pass work result receives more points. It is possible to more accurately measure the proficiency of each inspector 34 by receiving it.

The reason why the inspector 34 gives the result of the inspection performed by the number of times the inspection passes is given, because the inspector 34 inspects the result of the pass because it is difficult for other inspectors 34 to inspect it. This is because it is judged that the inspector 34 is more proficient than the other inspectors 34 in performing.

According to an exemplary embodiment of the present invention, the number of times the inspection pass request can be input per inspector may be limited to a predetermined value.

If the inspector 34 is able to input the inspection pass request without limitation, the defect inspector 34 who tries to inspect all the inspection passes for even the most difficult work results and only inspects the easy work results may occur. Accordingly, there may be a problem that the project schedule is delayed.

In addition, when the inspectors 34 indiscriminately pass the inspection, a situation in which the proficiency level for a specific worker 34 is abnormally high may occur.

Therefore, in order to solve this problem, the number of passes per inspector is limited.

Meanwhile, in the above description, steps S11 to S163 may be further divided into additional steps or may be combined into fewer steps, according to an embodiment of the present invention. In addition, some steps may be omitted as necessary, and the order between steps may be changed. In addition, even if other contents are omitted, the contents of FIG. 12 to be described later may also be applied to the proficiency measurement method using the inspection pass of the crowdsourcing-based project of FIGS. 1 to 11.

Hereinafter, a proficiency measuring apparatus 200 using a crowdsourcing-based project inspection pass according to an embodiment of the present invention will be described with reference to FIG. 12.

12 is a diagram for explaining a proficiency measuring apparatus using a crowdsourcing-based project inspection pass according to an embodiment of the present invention.

Referring to FIG. 12, the proficiency measuring device 200 (hereinafter, proficiency measuring device) using a crowdsourcing-based project inspection pass includes a communication module 210, a memory 220, and a processor 230.

The communication module 210 transmits a crowdsourcing-based work for one project to a plurality of workers 32 to request a task execution, and receives a plurality of work results from the plurality of workers 32. A plurality of work results received from a plurality of workers 32 are transmitted to a plurality of inspectors 34 to request inspection performance, and a plurality of inspection results or inspection pass requests are received from the plurality of inspectors 34 .

The memory 220 stores a program for measuring the proficiency level of the operator and the proficiency level of the inspector based on the data received from the communication module 210.

The processor 230 executes a program stored in the memory 220. As the processor 230 executes the program stored in the memory 220, the proficiency level of the operator and the inspector may be measured based on the number of inspection passes for the work result.

Specifically, the processor 230 calculates a score corresponding to a plurality of inspection results for a plurality of work results performed by each worker for each worker, and an inspection pass for a plurality of work results performed by each worker for each worker. A score corresponding to the number of requests is calculated, and proficiency of each worker is measured using a score corresponding to the inspection result for each worker and a score corresponding to the number of inspection pass requests.

In addition, the processor 230 calculates a score corresponding to a plurality of inspection results for the results of a plurality of tasks performed by each inspector for each inspector, and inspects the results of a plurality of tasks inspected by each inspector for each inspector. A score corresponding to the number of pass requests is calculated, and the proficiency level for each examiner is measured using a score corresponding to the inspection result and a score corresponding to the number of requests for the inspection pass.

The proficiency measuring apparatus 200 described with reference to FIG. 12 may be provided as a component of the above-described server.

The proficiency measurement method using a crowdsourcing-based project inspection pass according to an embodiment of the present invention described above may be implemented as a program (or application) to be executed in combination with a computer, which is hardware, and stored in a medium. .

The above-described program includes C, C++, JAVA, Ruby, which can be read by a processor (CPU) of the computer through the device interface of the computer, in order for the computer to read the program and execute the methods implemented as a program. It may include a code (Code) coded in a computer language such as machine language. Such code may include a functional code related to a function defining necessary functions for executing the methods, and a control code related to an execution procedure necessary for the processor of the computer to execute the functions according to a predetermined procedure. can do. In addition, such code may further include additional information required for the processor of the computer to execute the functions or code related to a memory reference to which location (address address) of the internal or external memory of the computer should be referenced. have. In addition, when the processor of the computer needs to communicate with any other computer or server in the remote in order to execute the functions, the code uses the communication module of the computer to determine how It may further include a communication-related code for whether to communicate, what kind of information or media should be transmitted and received during communication.

The stored medium is not a medium that stores data for a short moment, such as a register, cache, memory, etc., but a medium that stores data semi-permanently and can be read by a device. Specifically, examples of the storage medium include, but are not limited to, ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, and the like. That is, the program may be stored in various recording media on various servers to which the computer can access, or on various recording media on the user's computer. In addition, the medium may be distributed over a computer system connected through a network, and computer-readable codes may be stored in a distributed manner.

The above description of the present invention is for illustrative purposes only, and those of ordinary skill in the art to which the present invention pertains will be able to understand that it can be easily modified into other specific forms without changing the technical spirit or essential features of the present invention. will be. Therefore, it should be understood that the embodiments described above are illustrative in all respects and not limiting. For example, each component described as a single type may be implemented in a distributed manner, and similarly, components described as being distributed may also be implemented in a combined form.

The scope of the present invention is indicated by the claims to be described later rather than the detailed description, and all changes or modified forms derived from the meaning and scope of the claims and their equivalent concepts should be interpreted as being included in the scope of the present invention. do.

10: Client
20: service provider
30: public
32: worker
34: inspector
200: proficiency measuring device
210: communication module
220: memory
230: processor

Claims (11)

As a method performed by a computer,
Assigning a plurality of tasks of a crowdsourcing-based project (hereinafter, referred to as a project) to a plurality of workers and requesting the task to be performed;
Receiving a plurality of job results from the plurality of workers;
Assigning the plurality of work results to a plurality of inspectors and requesting to perform inspection;
Receiving a plurality of inspection results or an inspection pass request for the plurality of work results from the plurality of inspectors; And
Including the step of measuring the proficiency level for each worker based on the inspection result and the number of requests for the inspection pass,
Assigning the plurality of work results to a plurality of inspectors and requesting to perform the inspection,
The task result for which the above inspection pass request was entered is withdrawn and reassigned to another inspector to request inspection.
The step of measuring the proficiency level for each worker,
For each worker, calculating a score corresponding to a plurality of inspection results for the results of a plurality of tasks performed by each worker, and
Calculating a score corresponding to the number of requests for inspection passes for the results of a plurality of work performed by each worker for each worker, and
Including the step of measuring the proficiency level for each worker using a score corresponding to the inspection result and a score corresponding to the number of requests for the inspection pass,
A method of measuring proficiency using a crowdsourcing-based project inspection pass. The method of claim 1,
The step of calculating a score corresponding to the inspection result for each worker,
Calculating a first score corresponding to the number of inspection passes among the results of a plurality of work performed by each worker, and
Calculating a second score corresponding to the number of rejections for the results of a plurality of tasks performed by each worker, and
Subtracting the second score from the first score, and calculating a score corresponding to the inspection result for each worker,
A method of measuring proficiency using a crowdsourcing-based project inspection pass. The method of claim 2,
The step of calculating a score corresponding to the number of requests for the inspection pass for each worker,
Extracting a work result in which the inspection result passed the inspection from among the plurality of work results performed by each worker, and
Calculating the number of times the inspection pass is requested for the work result in which the inspection result has passed the inspection;
Comprising the step of calculating a score corresponding to the number of requests for the inspection pass for the work result in which the inspection result passed the inspection for each worker,
A method of measuring proficiency using a crowdsourcing-based project inspection pass. The method of claim 3,
The proficiency level is calculated using the following formula,
1st score = number of inspection passes x 1st unit score
2nd score = number of rejections x 2nd unit score
Score corresponding to the inspection result = 1st score-2nd score
Score corresponding to the number of inspection pass requests = number of inspection pass requests x first unit score x predetermined first ratio
Proficiency = Score corresponding to the test result + score corresponding to the number of requests for test pass
A method of measuring proficiency using a crowdsourcing-based project inspection pass. The method of claim 4,
The number of rejections includes a cumulative number of rejections related to the same task,
A method of measuring proficiency using a crowdsourcing-based project inspection pass. The method of claim 1,
The number of times the inspection pass request can be input per one inspector is limited to a predetermined value,
A method of measuring proficiency using a crowdsourcing-based project inspection pass. The method of claim 1,
Further comprising the step of measuring the proficiency level for each inspector based on the inspection result and the number of requests for the inspection pass,
The step of measuring the proficiency level for each inspector,
Calculating a score corresponding to the plurality of inspection results for the results of the plurality of work performed by each inspector for each inspector, and
For each inspector, calculating a score corresponding to the number of inspection pass requests for the results of a plurality of tasks performed by each inspector, and
Including the step of measuring the proficiency level for each examiner using a score corresponding to the test result and a score corresponding to the number of requests for the test pass,
A method of measuring proficiency using a crowdsourcing-based project inspection pass. The method of claim 7,
The step of measuring the proficiency level for each inspector,
It is to measure the proficiency level of each inspector by dividing the inspection result for the initial work result and the inspection result for the rework result.
A method of measuring proficiency using a crowdsourcing-based project inspection pass. The method of claim 7,
The step of calculating a score corresponding to the inspection result for each inspector,
Comprising the step of calculating a score corresponding to the number of inspection passes among a plurality of inspection results performed by each inspector,
A method of measuring proficiency using a crowdsourcing-based project inspection pass. The method of claim 9,
The proficiency level is calculated using the following formula,
Score corresponding to the inspection result = number of inspection passes x 3rd unit score
Score corresponding to the number of inspection pass requests = number of inspection pass requests x third unit score x predetermined second ratio
Proficiency = Score corresponding to the test result + score corresponding to the number of requests for test pass
A method of measuring proficiency using a crowdsourcing-based project inspection pass. A computer program stored in a computer-readable recording medium in order to perform a proficiency measurement method using a crowdsourcing-based project inspection pass according to any one of claims 1 to 10 in combination with a computer.

Images