KR102195963B1 - Method for adjusting inspection order using preferred inspection qauntity of crowdsourcing based project for artificial intelligence training data generation - Google Patents

Abstract

Provided is a method for adjusting inspection order using a preferred inspection amount of a crowdsourcing-based project for generating artificial intelligence training data to efficiently operate and manage the project. According to the present invention, the method comprises the following steps: setting a preferred inspection amount of a crowdsourcing-based project; assigning a plurality of work of the project to a plurality of workers to request execution of the work; receiving a plurality of work results from the plurality of workers; determining inspection order of the plurality of work results; assigning the plurality of work results to a plurality of inspectors in accordance with the determined inspection order to request execution of inspection; and receiving a plurality of inspection results from the plurality of inspectors. The step of determining the inspection order of the plurality of work results comprises a step of sequentially determining the inspection order of the plurality of work results in accordance with input order of the plurality of work results and a step of moving up the inspection order of a work result corresponding to the preferred inspection amount for each worker.

Description

A method of adjusting the inspection order using the priority inspection amount of a crowdsourcing-based project for generating artificial intelligence learning data {METHOD FOR ADJUSTING INSPECTION ORDER USING PREFERRED INSPECTION QAUNTITY OF CROWDSOURCING BASED PROJECT FOR ARTIFICIAL INTELLIGENCE TRAINING DATA GENERATION}

The present invention relates to a method for adjusting the inspection order using a priority inspection amount of a crowdsourcing-based project for generating artificial intelligence learning data.

Recently, more and more companies collect and process large amounts of data on a crowdsourcing basis that engages the general public in some process of corporate activities. In other words, by opening a project and allowing the general public, that is, workers to participate in the project, necessary information is collected through the work results completed by the workers.

At this time, in order to collect more reliable information, the company assigns the work result completed by the worker to the inspector to perform the inspection work.

Specifically, when one project is opened, a plurality of tasks are assigned to each of the plurality of workers. Each worker performs a plurality of tasks assigned to it, and provides the task results. Thereafter, a plurality of inspection tasks are assigned to each of the plurality of inspectors, and each inspector performs a plurality of inspection tasks assigned to them.

Currently, in determining the inspection order of a plurality of work results, it is common to assign work results to inspectors according to the order in which they are input so that work results are sequentially inspected.

The higher the difficulty of the project, the higher the rejection rate for the work results of the workers at the beginning of the project, and the worker learns the work method through the rejection reason entered by the inspector, which gradually decreases the rejection rate. However, when the work results are inspected in the order in which they are input, the worker who started the work late receives the inspection that much later, so that the worker who started the work late cannot determine whether or not he is performing the work correctly.

Accordingly, the worker performing the work in the wrong way does not recognize that she is performing the work incorrectly until the inspection order comes, and continues to perform the work in the wrong way.

In addition, since the inspector must inspect the results of the work performed incorrectly, it is difficult to perform the inspection efficiently, and accordingly, the company may suffer costly losses in project operation.

Unexamined Patent Publication No. 10-2014-0095956, 2014.08.04.

The problem to be solved by the present invention is to provide a method for adjusting the inspection order using the priority inspection amount of a crowdsourcing-based project for generating artificial intelligence learning data.

However, the problem to be solved by the present invention is not limited to the problem as described above, and other problems may exist.

In order to solve the above-described problems, a method for adjusting the inspection order using a priority inspection amount of a crowdsourcing-based project according to an aspect of the present invention includes the steps of setting a priority inspection amount of a crowdsourcing-based project, and performing a plurality of tasks of the project. Assigning to a plurality of workers and requesting to perform a task, receiving a plurality of work results from the plurality of workers, determining an inspection order of the plurality of work results, the plurality of tasks according to the determined inspection order Allocating a result to a plurality of inspectors and requesting to perform the inspection; and receiving a plurality of inspection results from the plurality of inspectors, and determining the inspection order of the plurality of work results, the plurality of tasks And sequentially determining the inspection order of the plurality of work results according to the order in which the results are input, and adjusting the inspection order in advance for the work results corresponding to the priority inspection amount for each worker.

In some embodiments of the present invention, the method further comprises the step of requesting rework by reassigning the rejected task to a corresponding worker when the inspection result is rejected, and providing a rejection reason input from the examiner can do.

In some embodiments of the present invention, the step of adjusting the inspection order in advance for the work result corresponding to the priority inspection amount for each worker is more than the work result of one or more workers who received inspection in excess of the priority inspection amount. The inspection order of the work results of one or more workers who have not been inspected by the priority inspection amount is adjusted in advance, and the work results of two or more workers who have not been inspected by the priority inspection amount are sequentially inspected according to the input order. It can be a decision.

In some embodiments of the present invention, the method may determine and adjust the inspection order in real time whenever a new work result is input.

In some embodiments of the present invention, the method further comprises performing a pilot of the crowdsourcing-based project before the step of setting the priority inspection amount, and setting the priority inspection amount, the Based on the result of the pilot, the priority inspection amount may be set.

In some embodiments of the present invention, the step of performing the pilot of the crowdsourcing-based project includes high-level proficiency workers, intermediate proficiency workers, and lower proficiency workers based on proficiency measured through the crowdsourcing-based project and other similar projects. You can perform a pilot by selecting evenly.

In some embodiments of the present invention, the step of setting the priority inspection amount may set the priority inspection amount based on the average rejection rate of workers participating in the pilot.

In some embodiments of the present invention, in the step of setting the priority inspection amount, the number of work at a time when the reduction rate of the average rejection rate of the workers participating in the pilot becomes greater than the reference value may be set as the priority inspection amount.

In some embodiments of the present invention, the method can be applied only when the difficulty of the crowdsourcing-based project is higher than the reference difficulty level.

A computer program according to another aspect of the present invention for solving the above-described problems is combined with a computer as hardware to execute a method of adjusting the inspection order using the priority inspection amount of the crowdsourcing-based project, and stored in a computer-readable recording medium. do.

Other specific details of the present invention are included in the detailed description and drawings.

According to the present invention described above, by first inspecting the work result corresponding to the inspection amount for each worker, each worker first checks whether he or she is performing the work properly through the inspection result for the work result corresponding to the inspection amount. Can be identified early in the project entry.

First, when the inspection result for the work result corresponding to the inspection amount is rejection, the operator learns the rejection reason input by the inspector, so that the work after the job corresponding to the inspection quantity can be performed in a correct manner.

Accordingly, as the project progresses, the rejection rate of each worker decreases, and each worker is motivated to be able to carry out the work more actively.

In addition, since the inspection completion rate of the inspector increases as the workers properly perform the work, the inspector can also be motivated to perform the inspection more actively.

As workers and inspectors actively carry out work and inspection, companies can perform project operation and management more efficiently.

The effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the following description.

1 is a conceptual diagram of a crowdsourcing service according to an embodiment of the present invention.
2 is a flowchart illustrating a crowdsourcing-based project progress process according to an embodiment of the present invention.
3 is a flowchart of a method for adjusting an inspection order using a priority inspection amount of a crowdsourcing-based project according to an embodiment of the present invention.
4 is a graph for explaining a priority inspection amount set through a pilot according to an embodiment of the present invention.
5 is a flowchart of a method of adjusting an inspection order for a work result corresponding to a priority inspection amount for each worker according to an embodiment of the present invention.
6 to 9 are exemplary views for explaining that an inspection order for a work result corresponding to a priority inspection amount is adjusted for each worker according to an embodiment of the present invention.
10 is a flowchart of a method for measuring proficiency of an operator according to an embodiment of the present invention.
11 is a view for explaining an apparatus for adjusting an inspection order using a priority inspection amount of a crowdsourcing-based project according to an embodiment of the present invention.

Advantages and features of the present invention, and a method of achieving them will become apparent with reference to the embodiments described below in detail together with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but may be implemented in a variety of different forms, only the present embodiments are intended to complete the disclosure of the present invention, It is provided to fully inform the technician of the scope of the present invention, and the present invention is only defined by the scope of the claims.

The terms used in the present specification are for describing exemplary embodiments and are not intended to limit the present invention. In this specification, the singular form also includes the plural form unless specifically stated in the phrase. As used in the specification, “comprises” and/or “comprising” do not exclude the presence or addition of one or more other elements other than the mentioned elements. Throughout the specification, the same reference numerals refer to the same elements, and “and/or” includes each and all combinations of one or more of the mentioned elements. Although "first", "second", and the like are used to describe various elements, it goes without saying that these elements are not limited by these terms. These terms are only used to distinguish one component from another component. Therefore, it goes without saying that the first component mentioned below may be the second component within the technical idea of the present invention.

Unless otherwise defined, all terms (including technical and scientific terms) used in the present specification may be used as meanings that can be commonly understood by those of ordinary skill in the art to which the present invention belongs. In addition, terms defined in a commonly used dictionary are not interpreted ideally or excessively unless explicitly defined specifically.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a conceptual diagram of a crowdsourcing service according to an embodiment of the present invention.

Referring to FIG. 1, a crowdsourcing service is performed by being composed of a client 10, a service provider 20, and a public 30.

The sponsor 10 refers to a company or individual who requests a crowdsourcing-based project (hereinafter, referred to as a project).

The requester 10 requests a project for the purpose of collecting source data or annotating data for generating artificial intelligence learning data. The data generated through the project can be used as learning data for arbitrary machine learning such as supervised learning, unsupervised learning, and reinforcement learning. Collection of source data refers to collecting raw data such as recorded voice collection and photo collection. Data annotation refers to inputting relevant annotation data into source data such as text, photos, and videos. For example, the data annotation may include finding an entity in a given fingerprint or finding a similar sentence, but is not limited thereto. Meanwhile, the types of the above-described projects are only one embodiment, and various projects may be handled in the present invention according to the design of the client.

The service provider 20 refers to a company that provides crowdsourcing services.

When the service provider 20 receives a request for a project for a product or service from the client 10, the service provider 20 allocates a task for the project to the general public 30 and receives the work result from the public 30. Thereafter, the final product extracted based on the work result is provided to the client 10.

At this time, the service provider 20 provides a crowdsourcing service to the client 10 and the public 30 through a crowdsourcing platform (hereinafter, the platform). That is, when the service provider 20 receives a request for a project from the client 10, the service provider 20 opens the project on the platform. Thereafter, when the work result for the open project is received from the public 30, the project is terminated on the platform, and the final product may be extracted and provided to the client 10.

The public 30 refers to the general public participating in the project open on the platform. Here, the public 30 may participate in a project opened on the platform through an application or website provided by the service provider 20.

The public 30 is composed of a worker 32 and an inspector 34.

The worker 32 decides to participate in a specific project among a plurality of projects open on the platform. Thereafter, the operator 32 performs an operation such as collection of source data or data annotation, and transmits it to the platform.

The inspector 34 decides to participate in a specific project among a plurality of projects open on the platform. Thereafter, the inspector 34 inspects the result of the work performed by the operator 32. As a result of performing the inspection, the inspector 34 may perform inspection pass processing or rejection processing, and input a rejection reason during rejection processing. In the case of passing the inspection, it is not necessary to rework and re-examination due to this, so passing inspection has the same meaning as completion of inspection.

2 is a flowchart illustrating a crowdsourcing-based project progress process according to an embodiment of the present invention.

First, the requester 10 requests one or more projects to the service provider 20 (S11).

Thereafter, the service provider 20 opens the requested project on the platform (S12). At this time, the service provider 20 may determine the grade in consideration of the difficulty of the project before opening the project. That is, depending on the difficulty level, it is possible to determine which level or higher public 30 to expose the project. Accordingly, it is possible to increase the reliability of the work result of the project.

Thereafter, the service provider 20 requests the work by assigning the work to the workers 32 of the corresponding level or higher according to the level of the project (S13).

Thereafter, the worker 32 performs the assigned task (S14). In this case, the worker 32 may input the reason for the inability to work without performing the work for a work in which the work itself is impossible for some reason.

Thereafter, the service provider 20 receives the work result from the worker 32 (S15), and assigns the work result to the inspector 34 to request the inspection (S16).

Thereafter, the inspector 34 performs the assigned inspection (S17). At this time, the inspector 34 decides to complete the inspection when it is determined that the work has been properly performed, and if it is determined that the inspection work is wrong, it rejects the inspection. During rejection processing, the inspector 34 inputs the rejection reason for what reason the job was judged to be wrong.

Thereafter, the service provider 20 receives the inspection result from the inspector 34 (S18).

When the inspection result is the inspection completion, the service provider 20 uses the work result as valid data, and based on this, extracts the final product at the end of the project.

When the inspection result is rejection processing, the service provider 20 may internally perform the inspection again, or assign the work to the worker 32 again to perform the rework. When reworking, re-examination by an inspector is required.

Thereafter, when the project period ends or sufficient valid data is secured, the service provider 20 terminates the project (S19), calculates a final result based on the secured valid data, and provides it to the client 10 ( S20).

At this time, before the end of the project, the service provider 20 evaluates the performance results of the worker 32 and the inspector 34, calculates the work cost and the inspection cost according to the evaluation, to the worker 32 and the inspector 34. give.

In FIGS. 1 and 2, it is simply expressed as a requester 10, a service provider 20, an operator 32, and an inspector 34, but these are smartphones, tablets, PDAs, laptops, desktops, operated by each participant. It means a computer device such as a server or a telecommunication device.

Hereinafter, referring to FIGS. 3 to 9, when the difficulty of the project is high, the inspection amount is first set, so that the work result corresponding to the inspection amount among the plurality of work results performed for each worker can be inspected first. Explain about adjusting the order of inspection for work results.

3 is a flowchart of a method for adjusting an inspection order using a priority inspection amount of a crowdsourcing-based project according to an embodiment of the present invention. 4 is a graph for explaining a priority inspection amount set through a pilot according to an embodiment of the present invention. 5 is a flowchart of a method of adjusting an inspection order for a work result corresponding to a priority inspection amount for each worker according to an embodiment of the present invention. 6 to 9 are exemplary views for explaining that an inspection order for a work result corresponding to a priority inspection amount is adjusted for each worker according to an embodiment of the present invention.

Meanwhile, the steps shown in FIGS. 3 and 5 may be understood as being performed by a platform server (hereinafter, referred to as a server) operated by the service provider 20, but are not limited thereto.

In addition, a plurality of workers 32 or a plurality of inspectors 34 perform work using a predetermined terminal device. The terminal device of the operator 32 or the inspector 34 may be a computer device or a telecommunication device such as a smartphone, tablet, PDA, laptop, desktop, etc., but is not limited thereto.

First, referring to FIG. 3, the server sets a priority inspection amount of a crowdsourcing-based project (S110).

When the difficulty of the crowdsourcing-based project is higher than the standard difficulty, the workers 32 are likely to enter the project and perform the work in an incorrect way when performing the initial work.

At this time, the work result of the worker 32 who first entered the project is immediately inspected, but the work result of the worker 32 who entered the project late is inspected after all the work results of the previous workers 32 have been inspected. Therefore, the worker 32 who entered late does not recognize that he or she is performing the work in the wrong way, and continues to perform the work in the wrong way.

As a result, incorrect work results are continuously accumulated, and an inefficient situation occurs in which the inspector must inspect the wrong work results.

Therefore, for a project with a difficulty level higher than the standard difficulty level, the inspection amount is first set, and the work results corresponding to the inspection amount are first inspected for the work results of all the workers 32 participating in the project, so that all workers 32 It is important to recognize your faults early in entering the project and ensure that the subsequent work can be properly performed.

To this end, the server performs a crowdsourcing-based project pilot before opening the project, and sets the priority inspection amount of the formal project based on the pilot result.

Specifically, the server selects the workers 32 based on a project similar to the project to be opened from among crowdsourcing-based projects that have been carried out previously, and participates the selected workers 32 in the pilot.

In other words, the server evenly selects high-level workers, middle-level workers, and low-level workers based on the proficiency measured through other previous similar projects, and pilots the selected high-level workers, middle-level workers, and low-level workers. Participate in A detailed description of the method of measuring the proficiency of the worker will be described later.

The upper proficiency level workers, the middle proficiency level workers, and the lower proficiency level workers selected by the server participate in the pilot and perform the work.

The server first sets the number of jobs at the point when each worker 32 participating in the pilot becomes proficient in the job, that is, when the rejection rate rapidly decreases, as the inspection quantity.

Specifically, the server calculates the average rejection rate by the number of jobs of each worker 32 participating in the pilot, and the number of jobs at the point when the reduction rate of the calculated average rejection rate becomes greater than the reference value (the number of proficiency points) as the first inspection amount. Set.

Referring to FIG. 4, the server may first determine the number of work at a point in time when the average rejection rate of the workers 32 participating in the pilot decreases sharply as the inspection amount.

For example, if each worker 32 needs to perform a total of 100 tasks in the pilot, when the average rejection rate of the workers 32 drops sharply based on the time when the number of tasks is 20, the server first checks. You can set the quantity to 20.

In other words, the server determines that the point at which the average rejection rate drops sharply is the point at which the workers 32 begin to become proficient in performing the task, and the number of work at that point (the number of proficiency points) is the priority inspection amount of the formal project. Set to

Accordingly, the server is highly likely to perform the work in an incorrect manner for the work results performed before each of the workers 32 participating in the formal project becomes proficient in work, and thus, inspects first.

Referring back to FIG. 3, the server allocates a plurality of tasks of a crowdsourcing-based project to a plurality of workers 32 to request task performance (S120), and receives a plurality of task results from the plurality of workers 32. (S130).

Since the steps S120 and S130 overlap with the steps S13 to S15 described above with reference to FIG. 2, detailed descriptions will be omitted.

Subsequently, the server determines the inspection order of the plurality of work results (S140).

Specifically, referring to FIG. 5, the server sequentially determines the inspection order of the plurality of job results according to the order in which the plurality of job results are input (S141), and the job result corresponding to the priority inspection amount for each worker The inspection order is advanced and adjusted (S142). Accordingly, the work result corresponding to the first inspection amount is first inspected despite the sequential inspection order initially determined in S141.

The server assigns data IDs according to the order in which the plurality of work results are input from the plurality of workers 32 to sequentially determine the inspection order.

Referring to FIG. 6(a), the project was opened, and worker A entered the project and performed a total of 30 tasks, and after that, worker B entered the project and performed a total of 20 tasks. In contrast, data IDs 1001 to 1030 are sequentially assigned to the 30 work results of worker A, and data IDs 1031 to 1050 are sequentially assigned to the 20 work results of worker B.

At this time, since worker A and worker B entered the project and performed the work for the first time, they have not yet received an inspection for the work result corresponding to the inspection amount. Therefore, the server must adjust the inspection order in advance for the job result corresponding to the inspection amount among the job results of worker A and worker B.

Referring to FIG. 6(b), when the first inspection amount is set to 10, the server has priority among the job results of worker A after the job results corresponding to the inspection amount (data ID 1001 to 1010) among the job results of worker A. The order of inspection is adjusted by arranging the work result (data ID 1031~1040) corresponding to the inspection quantity.

According to the inspection order adjusted as described above, 10 work results corresponding to the priority inspection amount of worker A are inspected first, and then 10 work results corresponding to the priority inspection amount of worker B are inspected. And, when all the inspection results corresponding to the priority inspection amount of worker A and worker B are inspected, after that, the remaining work results of worker A (data ID 1011 ~ 1030) and the remaining work results of worker B (data ID 1041 ~ 1050) is sequentially inspected according to the input order.

As described above, when two or more workers enter the project at different times and perform a large amount of work at once, only the work result corresponding to the first inspection amount is extracted from the work results input for each worker, and the worker's project First, according to the order of entry, the order of inspection of the work result corresponding to the inspection amount is adjusted.

The server determines and adjusts the order of inspection in real time whenever a new job result is input. In other words, even if a new worker participates in the project in the middle of the project and enters the project, the work result corresponding to the priority inspection amount of the new worker is inspected with priority.

In the situation where the inspection is sequentially performed in the order of the adjusted inspection as shown in Fig. 6(b), and as shown in Fig. 7(a), when worker C enters the project and inputs five work results, the server Determines and adjusts the inspection order by reflecting the newly inputted worker C's five work results in real time.

First, the server sequentially assigns data IDs 1051 to 1055 to worker C's five job results.

According to an embodiment, the server first adjusts the inspection order of the work results of one or more workers who have not been inspected by the inspection amount prior to the work results of one or more workers who have exceeded the inspection amount and received inspection.

That is, the server first adjusts the inspection order of the work result of the worker C who has not received the inspection by the amount of the inspection prior to the work result of the worker A and worker B who have been inspected by exceeding the inspection amount.

The server places the newly inputted worker C's five work results (data IDs 1051 to 1055) behind the work results that have been inspected at the present time.

Referring to FIG. 7, when the work result of data ID 1029 is inspected at the time when worker C newly inputs the work result (data ID 1051 to 1055), the server stores the work result of data ID 1051 to 1055 with data ID 1029. The work results of data ID 1030 and 1041 to 1050 are placed behind the work results of data ID 1055, and the order of inspection is adjusted.

In a situation in which the inspection is sequentially performed in the order of inspection adjusted as shown in Fig. 7(b), as shown in Fig. 8(a), the operator A newly updates the two work results and the operator C five work results. In the case of input, the server determines and adjusts the inspection order by reflecting the newly input two work results of worker A and five work results of worker C in real time.

First, the server sequentially assigns data IDs 1056 to 1062 to two work results of worker A and five work results of worker C.

At this time, worker A is in a state that has already exceeded the first inspection amount and has been inspected, and operator C is in a state that has received inspection only for five work results that are less than the first inspection amount.

Therefore, the server does not adjust the inspection order of the two newly inputted work results (data ID 1056, 1057) by worker A, but only inspects the five work results newly input by worker C (data IDs 1058 to 1062). Adjust the order ahead.

The server places the newly entered worker C's five work results (data IDs 1058 to 1062) behind the work results that have been inspected at the present time.

Referring to FIG. 8, when the work result of data ID 1041 is inspected at the time when worker A and worker C newly input the work result (data ID 1056 to 1062), the server stores the work result of data ID 1058 to 1062. It is placed after the job result of data ID 1041, and the job result of data IDs 1042 to 1050 is placed behind the job result of data ID 1062. At this time, the work results of the two work results (data IDs 1056 and 1057) newly inputted by the worker A are arranged at the end of the inspection order (the order after data ID 1050).

As described above, when work results are input from two or more workers in real time, the server checks whether each worker has received the first inspection amount as much as the inspection amount, and the order of inspection of the work results of the worker who has not received the inspection amount first. Adjust by moving forward. As a result, all workers can receive the first inspection as much as the inspection amount.

After all workers first receive the inspection as much as the inspection amount, the inspection order is determined in the order in which the work results are entered.

According to an embodiment, the server first determines the inspection order in order according to the input order for the work results of two or more workers who have not received inspection as much as the inspection amount.

In the situation where the inspection is sequentially performed in the order of the adjusted inspection as shown in Fig. 6(b), as shown in Fig. 9(a), the worker C and the worker D enter the project and each input 5 work results. In this case, the server determines and adjusts the inspection order by reflecting the newly inputted worker C and worker D's 5 work results in real time.

First, the server sequentially assigns data IDs 1051 to 1060 to each of the five job results of worker C and worker D according to the input order.

Specifically, worker C and worker D enter the project at about the same time, and worker C first inputs two work results, then worker D inputs two work results, and then worker C again three work results. After that, when the worker D inputs three work results, referring to FIG. 9(a), data IDs 1051 and 1052 are assigned to the two work results previously input by the worker C, and the worker D Data IDs 1053 and 1054 are assigned to the two job results entered by. In addition, data IDs 1055 to 1057 are assigned to the three work results input by worker C, and data IDs 1058 to 1060 are assigned to the three work results input by worker D.

At this time, because worker C and worker D entered the project and performed the work for the first time, they have not yet received an inspection for the work result corresponding to the inspection amount.

Therefore, the server first adjusts the inspection order of the work results of workers C and D that have not been inspected by the inspection amount prior to the work results of workers A and B that have been inspected by exceeding the inspection amount.

The server places the newly inputted worker C and worker D's 5 work results (data IDs 1051 to 1060) after the work results that have been inspected at the present time.

Referring to FIG. 9, when the work result of data ID 1029 is inspected at the time when worker C and worker D newly input work results (data IDs 1051 to 1060), the server stores the work results of data IDs 1051 to 1060. It is placed after the work result of data ID 1029, and the work result of data ID 1030 and 1041 to 1050 is placed behind the work result of data ID 1060, and the order of inspection is adjusted.

As described above, when work results are inputted almost at the same time from two or more workers who have not received inspection as much as the inspection amount, the order in which the work results are inputted from each worker, rather than adjusting the inspection order by dividing the work results by worker. Adjust the inspection order as soon as possible.

Referring back to FIG. 3, the server allocates a plurality of work results to a plurality of inspectors 34 according to the inspection order to request inspection (S150), and a plurality of results of a plurality of operations from the plurality of inspectors 34 The inspection result of is received (S160).

Since the steps S150 and S160 are overlapped with the steps S16 to S18 described above with reference to FIG. 2, detailed descriptions will be omitted.

When the result of the inspection of the work result of the worker 32 is rejected, the server reassigns the rejected work to the corresponding worker 32 to request rework, and provides the rejection reason input from the inspector 34 .

The worker (32) knows that his work method is wrong or that he or she is unfamiliar with the work guide through the reason for rejection, and learns the correct work method, or by knowing the work guide, performs subsequent work in the correct way. Can be done with.

As described above, in the present invention, since the inspection proceeds in the order of inspection adjusted according to the inspection amount first, each of the workers 32 can quickly receive feedback on the work result corresponding to the inspection amount. Due to this, the workers 32 can grasp at the beginning of the project entry whether they are performing the work correctly and perform subsequent work.

That is, if the inspection result of one or more of the job results corresponding to the first inspection amount is rejected, the worker 32 rework the rejected job result through the reason for rejection of the inspector 34 for the rejected job result. And, by learning the correct work method through the reason for rejection or familiarizing the work guide, it is possible to perform work in the correct way for work assigned after work corresponding to the first inspection amount. Due to this, the rejection rate of each worker 32 may be gradually decreased.

Hereinafter, a method of measuring the proficiency of the worker 32 will be described with reference to FIG. 10. Here, the proficiency level is measured based on the work results performed by the worker 32 participating in a crowdsourcing-based project.

10 is a flowchart of a method for measuring proficiency of an operator according to an embodiment of the present invention.

Meanwhile, the steps shown in FIG. 10 may be understood as being performed by a platform server (hereinafter, referred to as a server) operated by the service provider 20, but are not limited thereto.

The server calculates the work score of the worker 32 participating in the crowdsourcing-based project (S210).

The task score corresponds to the number of tasks that have passed the inspection among the tasks performed by the worker 32 (hereinafter, the number of passes through the inspection). Specifically, the work score means a value obtained by multiplying the number of inspection passes by the work unit cost.

For example, if the number of inspection passes by a specific worker 32 is 70 and the unit cost of work is 50 won, the worker's work score is calculated as 3,500 points.

Subsequently, the server compares the number of work of the worker 32 and the number of proficiency points calculated through the pilot (S220).

Here, the number of proficiency points may be used to measure the proficiency level of the worker 32 in all types of projects, or may be used to set a priority inspection amount for a project with high difficulty as described above.

The method of calculating the proficiency point quantity is as described above with reference to FIG. 4.

Subsequently, when the number of work of the worker 32 is less than the number of proficiency points, that is, before the number of work of the worker 32 exceeds the number of proficiency points, a rejection score is calculated (S230). The rejection score corresponds to the number of rejected tasks (hereinafter, referred to as rejection) among the tasks performed by the worker 32. A specific method of calculating the rejection score is different for each case, as described later.

There may be many differences in work ability at the beginning of work according to individual differences and differences in experience for each worker 32. Also, depending on the interpretation of the work guide at the beginning of the work, the work may be performed incorrectly.

Therefore, the server calculates the rejection score without applying a weight to the number of jobs that are smaller than the number of proficiency points. For example, at the beginning of the project, if the number of work (quantity that passed inspection) of a specific worker 32 is 70 and the number of proficiency points is 100, the server determines that the number of work does not exceed the proficiency point quantity and does not apply weight The return score is calculated without.

Specifically, the server calculates a rejection score by multiplying the rejection unit price by the rejection work quantity when the worker's work quantity is less than the proficiency time quantity.

For example, if the number of rejected jobs is 40 out of 70 jobs of a specific worker 32 (the number of passing inspections) and the return unit price is 20 won, the rejection score of the corresponding worker 32 is 800 points.

On the other hand, when the number of work of the worker 32 is greater than the number of proficiency points, that is, after the number of work of the worker 32 exceeds the number of proficiency points, the weighted rejection score is calculated. In this case, the weight is determined based on the average rejection rate. The average rejection rate is as described above with reference to FIG. 4.

Hereinafter, when the quantity of work exceeds the quantity at the time of proficiency, a weight is applied to calculate the rejection score.

When the number of jobs of the worker 32 exceeds the number of proficiency points, the server compares the rejection rate of the worker and the average rejection rate calculated through the pilot (S240).

Specifically, when the number of jobs completed by the worker 32 (passed through the inspection) is 70 and the number of proficiency points is 30, the server compares the rejection rate of the worker in the number of 70 jobs and the average rejection rate.

According to an embodiment, when the worker 32 participating in the formal project is a worker with a low grade, the server may perform comparison with only the work results of the lower proficiency level worker participating in the pilot.

That is, the server compares the rejection rate of workers 32 who have little or no experience in performing the task with the average rejection rate of workers with lower proficiency level participating in the pilot, so that the comparison standard can be slightly lowered for workers with low grades There is.

Subsequently, when the worker's rejection rate is greater than the average rejection rate, the server calculates a rejection score by applying the weight of the first value (S250). Here, the first value is a value greater than 1 (for example, 2).

That is, when the rejection rate of the worker 32 is higher than the average rejection rate in the number of jobs exceeding the proficiency point quantity, the server calculates the rejection score by applying a weight twice as much to the rejection quantity larger than the average rejection quantity.

Specifically, referring to [Equation 1] below, the server does not apply a weight with respect to the quantity (b) corresponding to the average rejection quantity (b) of the total rejection quantity (a) of the worker 32, and the average The rejection score may be calculated by applying a weight of the first value (double weight) with respect to the remaining quantity (ab) exceeding the rejection quantity (b).

[Equation 1]

Average rejected quantity x rejected unit price + (rejected quantity-average rejected quantity) x rejected unit price x 2

For example, if the total rejected quantity (a) of the worker 32 for the 70 job quantity is 50, the average rejected quantity (b) for the 70 job quantity is 35, and the rejection unit price is 10 won, the rejection score Can be calculated as 35×10+(50-35)×10×2 = 650 points.

Subsequently, when the rejection rate of the worker 32 is less than the average rejection rate, the server calculates a rejection score by applying the weight of the second value (S260). Here, the second value is a value less than 1 (for example, 0.5).

That is, when the rejection rate of the worker 32 is smaller than the average rejection rate in the number of jobs exceeding the proficiency point quantity, the server calculates the rejection score by applying a weight of 0.5 times to the rejection quantity less than the average rejection quantity.

Specifically, referring to [Equation 2] below, the server is the weight of the second value (0.5 times the weight of the quantity (bc) that is less than the average rejected quantity (b) of the total rejected quantity (c) of the worker) ), it is possible to calculate the rejection score without applying a weight for the remaining quantity (c-(bc)).

[Equation 2]

(Average rejected quantity-rejected quantity) × rejected unit price × 0.5 + (rejected quantity-(average rejected quantity-rejected quantity)) × rejected unit price

For example, if the total rejected quantity (c) of the worker for the quantity of 70 jobs is 20, the average rejected quantity (b) of the quantity of 70 jobs is 35, and the rejection unit price is 10 won, the rejection score is (35 It can be calculated as -20) × 10 × 0.5 + (20-(35-20)) × 10 = 125 points.

Then, the server measures the proficiency of the worker 32 by using the difference between the calculated work score and the rejection score (S270). At this time, it is determined that the greater the difference value (final score), the greater the proficiency of the worker 32.

According to the final score, the server may classify the worker 32 into an upper proficiency worker, a medium proficiency worker, and a lower proficiency level worker.

If 0 is the lowest score and 300 is the highest, if the final score of the worker 32 is between 0 and 100, it can be classified as a lower proficiency worker, and if it is between 101 and 200, a median proficiency worker It can be classified as, and if it is between 201 and 300 points, the highest proficiency can be classified as a worker.

As a result of proficiency measurement, workers 32 classified as high proficiency workers can participate in all projects requiring high proficiency, medium proficiency, or low proficiency. However, as a result of proficiency measurement, the worker 32 classified as a lower proficiency worker can only participate in a project requiring a lower proficiency level.

In addition, the server may give the worker 32 an opportunity for crowdsourcing-based work for other similar projects requiring proficiency of a predetermined level or higher by using the proficiency measurement result of the worker 32.

For example, if the grade of the worker 32 previously classified as a high proficiency worker in a specific project is A, and the level of the worker required in an open scheduled project similar to the specific project is set to B (higher than A), According to the principle, the worker 32 cannot participate in the project to be opened because the level is low, but the worker 32 is classified as a high proficiency level worker in the specific project similar to the project to be opened, so the server is the worker 32 Is determined as a reliable worker, and the corresponding worker 32 can be given an opportunity to participate in the project to be opened.

In addition, as described above, the server may select the worker 32 to participate in the pilot of a project similar to the specific project through the proficiency measurement result measured by the worker 32 previously participating in a specific project.

Meanwhile, in the above description, steps S11 to S270 may be further divided into additional steps or may be combined into fewer steps according to an embodiment of the present invention. In addition, some steps may be omitted as necessary, and the order between steps may be changed. In addition, even if other contents are omitted, the contents of FIG. 11 to be described later can also be applied to the method of adjusting the inspection order using the priority inspection amount of the crowdsourcing-based project of FIGS.

Hereinafter, a description will be given of an apparatus 300 for adjusting the inspection order using the priority inspection amount of a crowdsourcing-based project according to an embodiment of the present invention.

11 is a view for explaining an apparatus for adjusting an inspection order using a priority inspection amount of a crowdsourcing-based project according to an embodiment of the present invention.

Referring to FIG. 11, the apparatus 300 (hereinafter, the apparatus for adjusting the inspection order) for adjusting the inspection order by using the priority inspection amount of the crowdsourcing-based project according to the communication module 310, the memory 320, and the processor 330 ).

The communication module 310 transmits a crowdsourcing-based work for one project to a plurality of workers 32 to request a task to be performed, and receives work results from the plurality of workers 32. Further, the work results received from the plurality of workers 32 are transmitted to the plurality of inspectors 34 to request inspection, and the inspection results are received from the plurality of inspectors 34.

The memory 320 stores a program for adjusting the inspection order using the priority inspection amount of the crowdsourcing-based project based on the data received from the communication module 310.

The processor 330 executes a program stored in the memory 320. As the processor 230 executes the program stored in the memory 320, it stores the job results inputted from each worker in order, and adjusts the inspection order in advance for the job result corresponding to the inspection amount for each worker. . Accordingly, the project can be efficiently progressed by allowing workers who started work late to receive a priority inspection for the work result corresponding to the inspection amount.

The apparatus 300 for adjusting the inspection order described with reference to FIG. 11 may be provided as a component of the above-described server.

The method for adjusting the inspection order using the priority inspection amount of a crowdsourcing-based project according to an embodiment of the present invention described above may be implemented as a program (or application) to be executed in combination with a computer, which is hardware, and stored in a medium. have.

The above-described program includes C, C++, JAVA, Ruby, which can be read by a processor (CPU) of the computer through the device interface of the computer, in order for the computer to read the program and execute the methods implemented as a program. It may include a code (Code) coded in a computer language such as machine language. Such code may include a functional code related to a function defining necessary functions for executing the methods, and a control code related to an execution procedure necessary for the processor of the computer to execute the functions according to a predetermined procedure. can do. In addition, such code may further include additional information required for the processor of the computer to execute the functions or code related to a memory reference to which location (address address) of the internal or external memory of the computer should be referenced. have. In addition, when the processor of the computer needs to communicate with any other computer or server in the remote in order to execute the functions, the code uses the communication module of the computer to determine how It may further include a communication-related code for whether to communicate or what information or media to transmit and receive during communication.

The stored medium is not a medium that stores data for a short moment, such as a register, cache, memory, etc., but a medium that stores data semi-permanently and can be read by a device. Specifically, examples of the storage medium include, but are not limited to, ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, and the like. That is, the program may be stored in various recording media on various servers to which the computer can access, or on various recording media on the user's computer. In addition, the medium may be distributed over a computer system connected through a network, and computer-readable codes may be stored in a distributed manner.

The above description of the present invention is for illustrative purposes only, and those of ordinary skill in the art to which the present invention pertains will be able to understand that it can be easily modified into other specific forms without changing the technical spirit or essential features of the present invention. will be. Therefore, it should be understood that the embodiments described above are illustrative in all respects and not limiting. For example, each component described as a single type may be implemented in a distributed manner, and similarly, components described as being distributed may also be implemented in a combined form.

The scope of the present invention is indicated by the claims to be described later rather than the detailed description, and all changes or modified forms derived from the meaning and scope of the claims and their equivalent concepts should be interpreted as being included in the scope of the present invention. do.

10: Client
20: service provider
30: public
32: worker
34: inspector
300: inspection order adjustment device
310: communication module
320: memory
330: processor

Claims (10)

As a method performed by a computer,
Establishing a priority inspection amount for crowdsourcing-based projects;
Allocating a plurality of tasks of the project to a plurality of workers and requesting the task to be performed;
Receiving a plurality of work results from the plurality of workers;
Determining an inspection order of the plurality of work results;
Allocating the plurality of work results to a plurality of inspectors according to the determined inspection order and requesting to perform inspection;
Receiving a plurality of inspection results from the plurality of inspectors; And
If the result of the inspection is rejected, reassigning the rejected work to a corresponding worker to request rework, and providing a rejection reason input from the inspector,
The step of determining the inspection order of the plurality of work results,
Determining an inspection order of the plurality of work results sequentially according to the order in which the plurality of work results are input, and
Including the step of adjusting the inspection order in advance for the work result corresponding to the priority inspection amount for each worker,
The step of adjusting the inspection order in advance for the work result corresponding to the priority inspection amount for each worker,
Adjusting the inspection order of the work results of one or more workers who have not been inspected by the priority inspection amount more than the work results of one or more workers who have been inspected in excess of the priority inspection amount, and
For the work results of two or more workers who have not been inspected as much as the priority inspection amount, the inspection order is sequentially determined according to the input order,
Whenever new work results are input, the above inspection order is determined and adjusted in real time,
When receiving the work results of one or more workers who have received inspection in excess of the priority inspection amount as the new work result, the inspection order is not adjusted in advance,
A method of adjusting the inspection order using the priority inspection quantity of a crowdsourcing-based project. delete delete delete The method of claim 1,
Prior to the step of setting the first inspection amount, further comprising the step of performing a pilot of the crowdsourcing-based project,
The step of setting the priority inspection amount may include setting the priority inspection amount based on the result of the pilot,
A method of adjusting the inspection order using the priority inspection quantity of a crowdsourcing-based project. The method of claim 5,
The step of piloting the crowdsourcing-based project,
Performing a pilot by equally selecting upper proficiency workers, intermediate proficiency workers, and lower proficiency workers based on proficiency measured through the crowdsourcing-based project and other similar projects,
A method of adjusting the inspection order using the priority inspection quantity of a crowdsourcing-based project. The method of claim 5,
The step of setting the priority inspection amount,
Setting the priority inspection amount based on the average rejection rate of workers participating in the pilot,
A method of adjusting the inspection order using the priority inspection quantity of a crowdsourcing-based project. The method of claim 7,
The step of setting the priority inspection amount,
Setting the number of work at the time when the reduction rate of the average rejection rate of the workers participating in the pilot becomes greater than the reference value as the priority inspection amount,
A method of adjusting the inspection order using the priority inspection quantity of a crowdsourcing-based project. The method of claim 1,
Applied only when the difficulty of the crowdsourcing-based project is higher than the standard difficulty level,
A method of adjusting the inspection order using the priority inspection quantity of a crowdsourcing-based project. A computer program stored in a computer-readable recording medium in order to perform the inspection adjustment method using the priority inspection amount of the crowdsourcing-based project according to any one of claims 1, 5 to 9 in combination with a computer.

Images