KR102169321B1 - Method for adjusting the cost of project inspection using the verification work results of crowdsourcing based projects for artificial intelligence training data generation - Google Patents

Abstract

Provided is a method of controlling a project inspection unit price by using a job result for the verification of a project based on crowdsourcing for creating artificial intelligence learning data. The method includes the following steps of: requesting job fulfillment by assigning a plurality of jobs to a plurality of workers; receiving a plurality of job results from the plurality of workers; requesting inspection fulfillment by assigning the plurality of job results to a plurality of job inspectors; receiving an inspection pass or return as a plurality of inspection results about the plurality of job results from the plurality of inspectors; creating a plurality of job results for verification by extracting an inspection result satisfying a predetermined condition from among a plurality of inspection results about the plurality of job results during a predetermined time section; receiving an inspection result by assigning the plurality of job results for verification to the plurality of inspectors; calculating an average inspection fulfillment time for each of the job results for verification of the plurality of injectors; comparing inspection fulfillment times recorded with respect to the job results for verification; and controlling a project inspection unit price for the plurality of inspectors by using the result of the comparison.

Description

{METHOD FOR ADJUSTING THE COST OF PROJECT INSPECTION USING THE VERIFICATION WORK RESULTS OF CROWDSOURCING BASED PROJECTS FOR ARTIFICIAL INTELLIGENCE TRAINING DATA GENERATION}

The present invention for generating artificial intelligence learning data relates to a method for adjusting a project inspection means using a work result for verification of a crowdsourcing-based project.

Recently, more and more companies collect and process large amounts of data on a crowdsourcing basis that engages the general public in some process of corporate activities. In other words, by opening a project and allowing the general public, that is, workers to participate in the project, necessary information is collected through the work results completed by the workers.

At this time, in order to collect more reliable information, the company assigns the work result completed by the worker to the inspector to perform the inspection work.

Specifically, when one project is opened, a plurality of tasks are assigned to each of the plurality of workers. Each worker performs a plurality of tasks assigned to it, and provides the task results. Thereafter, a plurality of inspection tasks are assigned to each of the plurality of inspectors, and each inspector performs a plurality of inspection tasks assigned to them.

On the other hand, as the project progresses, inspectors become more proficient in inspecting work results. However, the current project inspection fee does not take into account the proficiency level of the inspectors, and the inspection fee at the time of opening the project and the inspection fee at the second half of the project are set the same.

In other words, in most cases, the initial inspection cost of a project is set according to the initial inspector who is not proficient, and based on these, the standard wage per unit time is set to be paid. In this case, as the number of inspectors who are accustomed to performing the inspection increases in the second half of the project, and as the workers are also familiar with the work performance, it becomes easier to perform inspection on the results of the inspectors' work. There will be more inspectors who earn far exceeding profits.

This is a result of incorrectly setting the initial inspection fee for the project, and it is a great burden to the client. Therefore, it is necessary to change the inspection fee for the project considering that the inspectors become more proficient as the project progresses.

Unexamined Patent Publication No. 10-2014-0095956, 2014.08.04.

The problem to be solved by the present invention is to generate the verification work result during the progress of the project and provide it to the inspectors, check the inspection execution time for the verification work result, and determine the inspection cost of the project according to the proficiency of the evaluated inspectors. The control method is provided by the project inspection tool using the verification work results of the crowdsourcing-based project.

However, the problem to be solved by the present invention is not limited to the problem as described above, and other problems may exist.

In order to solve the above-described problem, a method for adjusting a project inspection means using a work result for verification of a crowdsourcing-based project according to an aspect of the present invention is to perform a plurality of tasks of a crowdsourcing-based project (hereinafter, "project"). Assigning to a worker and requesting to perform a task; Receiving a plurality of job results from the plurality of workers; Assigning the plurality of work results to a plurality of inspectors and requesting to perform inspection; Receiving an inspection pass or rejection as a plurality of inspection results for the plurality of work results from the plurality of inspectors; Generating a plurality of verification work results by extracting an inspection result that satisfies a predetermined condition from among a plurality of inspection results for the plurality of work results during a predetermined time period; Allocating the plurality of verification work results to the plurality of inspectors and receiving the inspection results; Calculating an average inspection execution time (hereinafter, "first inspection execution time") for the results of each of the plurality of inspectors; Comparing the first inspection execution time with the inspection execution time (hereinafter referred to as "second inspection execution time") recorded for the verification work result before the calculation of the first inspection execution time; And adjusting project inspection means for the plurality of inspectors by using the comparison result. In this case, the step of comparing the first inspection execution time and the second inspection execution time includes calculating deviations between the first inspection execution time and the second inspection execution time for the plurality of verification work results of the plurality of inspectors. Including the step of, and the step of adjusting the project inspection means price for the plurality of inspectors using the comparison result, adjusting the project inspection means price for the plurality of inspectors based on the calculated average value of each deviation. Includes steps.

In some embodiments of the present invention, the predetermined condition may be a condition that satisfies a predetermined ranking based on the time required for inspection by the plurality of inspectors.

In some embodiments of the present invention, the step of generating the plurality of verification work results includes re-reviewing at least one other examiner different from the examiner who has examined the actual work result corresponding to the inspection result that satisfies the predetermined condition. Requesting; And generating an actual work result having an inspection result that matches the re-examination result by the other inspector as the plurality of verification work results.

In some embodiments of the present invention, the step of generating the plurality of verification work results may include the verification of the plurality of verification results of the first pass, rejection, or re-review by another inspector among the plurality of verification results. Can be produced as a result.

In some embodiments of the present invention, the step of allocating the plurality of verification work results to the plurality of reviewers and receiving the verification results may include a work result of a predetermined ratio or more of the total work included in the project or the work result. From the time point when the results of the inspection for the inspection are submitted, the results of the verification work may be assigned to a plurality of inspectors to request the inspection.

In some embodiments of the present invention, the step of calculating the first inspection execution time for each verification task result of the plurality of inspectors includes n number of inspections among the plurality of inspectors (where n is a natural number of 1 or more). ), the first inspection execution time for the inspection of the verification work result of an inspector less than) may be excluded.

Some embodiments of the present invention may include calculating rejection rates for a plurality of jobs corresponding to the plurality of verification results based on the plurality of inspection results; And excluding a first inspection execution time for a verification task result corresponding to a task whose rejection rate is greater than or equal to a preset rejection rate.

In some embodiments of the present invention, the step of excluding a first inspection execution time for a verification task result corresponding to a task whose rejection rate is greater than or equal to a preset rejection rate includes the first inspection execution time and the second inspection execution time. When the time deviation is greater than or equal to a preset deviation, the first inspection execution time may be excluded.

In some embodiments of the present invention, the step of adjusting the project inspection cost for the plurality of inspectors using the comparison result, the total average value of the first inspection execution time is the total average value of the second inspection execution time Reducing the inspection cost of the project by a ratio corresponding to the average value of each deviation when it has a shorter time; And when the total average value of the first inspection execution time has a longer time than the total average value of the second inspection execution time, increasing the inspection means cost of the project by a ratio corresponding to the average value of each deviation. It may include.

In some embodiments of the present invention, after the project inspection means for the plurality of inspectors is adjusted, the second inspection execution time recorded for the verification work result is averaged for each verification work result of the plurality of inspectors. It may further include the step of updating to the first inspection execution time that is the inspection execution.

The computer program according to another aspect of the present invention for solving the above-described problems is combined with a computer as hardware, and the project inspection means executes the control method using the verification work result of the crowdsourcing-based project, and a computer-readable recording medium Is stored in.

Other specific details of the present invention are included in the detailed description and drawings.

According to the present invention described above, the proficiency level of inspectors changes as the project progresses, and if the initially calculated proficiency level of the project is inappropriate as of the present time, the proficiency level of the changed inspectors can be adjusted in consideration of the proficiency level of the project. There is this.

In particular, not only the adjustment of the initially calculated project inspection cost, but also after the project inspection cost has been adjusted more than once, it is determined whether or not the proficiency of the inspectors has improved from the previous point based on this, and adjusts the project inspection cost based on this. I can.

The effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the following description.

1 is a conceptual diagram of a crowdsourcing service according to an embodiment of the present invention.
2 is a flowchart illustrating a crowdsourcing-based project progress process according to an embodiment of the present invention.
3 is a flowchart of a method for adjusting a project inspection means using a work result for verification of a crowdsourcing-based project according to an embodiment of the present invention.
4 and 5 are diagrams for explaining contents of generating a verification operation result.
6 is a view for explaining the calculation and comparison of the first and second inspection execution time for the verification work result.
7 is a diagram for explaining content of updating a second inspection execution time.
8 is a view for explaining a project inspection means adjusting device according to an embodiment of the present invention.

Advantages and features of the present invention, and a method of achieving them will become apparent with reference to the embodiments described below in detail together with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but may be implemented in a variety of different forms, only the present embodiments are intended to complete the disclosure of the present invention, It is provided to fully inform the technician of the scope of the present invention, and the present invention is only defined by the scope of the claims.

The terms used in the present specification are for describing exemplary embodiments and are not intended to limit the present invention. In this specification, the singular form also includes the plural form unless specifically stated in the phrase. As used in the specification, “comprises” and/or “comprising” do not exclude the presence or addition of one or more other elements other than the mentioned elements. Throughout the specification, the same reference numerals refer to the same elements, and “and/or” includes each and all combinations of one or more of the mentioned elements. Although "first", "second", and the like are used to describe various elements, it goes without saying that these elements are not limited by these terms. These terms are only used to distinguish one component from another component. Therefore, it goes without saying that the first component mentioned below may be the second component within the technical idea of the present invention.

Unless otherwise defined, all terms (including technical and scientific terms) used in the present specification may be used as meanings that can be commonly understood by those of ordinary skill in the art to which the present invention belongs. In addition, terms defined in a commonly used dictionary are not interpreted ideally or excessively unless explicitly defined specifically.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a conceptual diagram of a crowdsourcing service according to an embodiment of the present invention.

Referring to FIG. 1, a crowdsourcing service is performed by being composed of a client 10, a service provider 20, and a public 30.

The sponsor 10 refers to a company or individual who requests a crowdsourcing-based project (hereinafter, referred to as a project).

The requester 10 requests a project for the purpose of collecting source data or annotating data for generating artificial intelligence learning data. The data generated through the project can be used as learning data for arbitrary machine learning such as supervised learning, unsupervised learning, and reinforcement learning. Collection of source data refers to collecting raw data such as recorded voice collection and photo collection. Data annotation refers to inputting relevant annotation data into source data such as text, photos, and videos. For example, the data annotation may include finding an entity in a given fingerprint or finding a similar sentence, but is not limited thereto. Meanwhile, the types of the above-described projects are only one embodiment, and various projects may be handled in the present invention according to the design of the client.

The service provider 20 refers to a company that provides crowdsourcing services.

When the service provider 20 receives a request for a project for a product or service from the client 10, the service provider 20 allocates a task for the project to the general public 30 and receives the work result from the public 30. Thereafter, the final product extracted based on the work result is provided to the client 10.

At this time, the service provider 20 provides a crowdsourcing service to the client 10 and the public 30 through a crowdsourcing platform (hereinafter, the platform). That is, when the service provider 20 receives a request for a project from the client 10, the service provider 20 opens the project on the platform. Thereafter, when the work result for the open project is received from the public 30, the project is terminated on the platform, and the final product may be extracted and provided to the client 10.

The public 30 refers to the general public participating in the project open on the platform. Here, the public 30 may participate in a project opened on the platform through an application or website provided by the service provider 20.

The public 30 is composed of a worker 32 and an inspector 34.

The worker 32 decides to participate in a specific project among a plurality of projects open on the platform. Thereafter, the operator 32 performs an operation such as collection of source data or data annotation, and transmits it to the platform.

The inspector 34 decides to participate in a specific project among a plurality of projects open on the platform. Thereafter, the inspector 34 inspects the result of the work performed by the operator 32. As a result of performing the inspection, the inspector 34 may perform inspection pass processing or rejection processing, and input a rejection reason during rejection processing. In the case of passing the inspection, it is not necessary to rework and re-examination due to this, so passing inspection has the same meaning as completion of inspection.

2 is a flowchart illustrating a crowdsourcing-based project progress process according to an embodiment of the present invention.

First, the requester 10 requests one or more projects to the service provider 20 (S11).

Thereafter, the service provider 20 opens the requested project on the platform (S12). At this time, the service provider 20 may determine the grade in consideration of the difficulty of the project before opening the project. That is, depending on the difficulty level, it is possible to determine which level or higher public 30 to expose the project. Accordingly, it is possible to increase the reliability of the work result of the project.

Thereafter, the service provider 20 requests the work by assigning the work to the workers 32 of the corresponding level or higher according to the level of the project (S13).

Thereafter, the worker 32 performs the assigned task (S14). In this case, the worker 32 may input the reason for the inability to work without performing the work for a work in which the work itself is impossible for some reason.

Thereafter, the service provider 20 receives the work result from the worker 32 (S15), and assigns the work result to the inspector 34 to request the inspection (S16).

Similarly, according to an embodiment of the present invention, only a suitable project may be exposed to the inspector 34 out of the total projects being performed according to the level set according to the difficulty of the project or the qualification requirements of the inspector 34.

Thereafter, the inspector 34 performs the assigned inspection (S17). At this time, the inspector 34 determines that the inspection is completed when it is determined that the work has been properly performed, and if it is determined that the inspection operation is wrong, the inspector 34 rejects it. During rejection processing, the inspector 34 inputs the rejection reason for what reason the job was judged to be wrong.

Thereafter, the service provider 20 receives the inspection result from the inspector 34 (S18).

When the inspection result is the inspection completion, the service provider 20 uses the work result as valid data, and based on this, extracts the final product at the end of the project.

When the inspection result is rejection processing, the service provider 20 may internally perform the inspection again, or assign the work to the worker 32 again to perform the rework. When reworking, re-examination by an inspector is required.

Thereafter, when the project period ends or sufficient valid data is secured, the service provider 20 terminates the project (S19), calculates a final result based on the secured valid data, and provides it to the client 10 ( S20).

At this time, before the end of the project, the service provider 20 evaluates the performance results of the worker 32 and the inspector 34, calculates the work cost and the inspection cost according to the evaluation, to the worker 32 and the inspector 34. give.

In FIGS. 1 and 2, it is simply expressed as a requester 10, a service provider 20, an operator 32, and an inspector 34, but these are smartphones, tablets, PDAs, laptops, desktops, operated by each participant. It means a computer device such as a server or a telecommunication device.

Hereinafter, with reference to FIGS. 3 to 7, a method of adjusting a project inspection means using a work result for verification of a crowdsourcing-based project according to an embodiment of the present invention will be described.

3 is a flowchart of a method for adjusting a project inspection means using a work result for verification of a crowdsourcing-based project according to an embodiment of the present invention.

Meanwhile, the steps shown in FIG. 3 may be understood as being performed by a platform server (hereinafter, referred to as a server) operated by the service provider 20, but are not limited thereto.

In addition, a plurality of workers 32 or a plurality of inspectors 34 perform work using a predetermined terminal device. The terminal device of the operator 32 or the inspector 34 may be a computer device or a telecommunication device such as a smartphone, tablet, PDA, laptop, desktop, etc., but is not limited thereto.

First, referring to FIG. 3, the server assigns a plurality of tasks of a crowdsourcing-based project (hereinafter referred to as a project) to a plurality of workers 32 and requests the task to be performed (S105), and a plurality of workers 32 A plurality of work results are inputted from (S110).

Here, one project may contain multiple jobs, and the plurality of jobs may be grouped into a plurality of tasks according to their type, or each job may be a different type of job or the same job. , But is not limited thereto.

Next, the server assigns the plurality of work results to the plurality of inspectors 34 to request the inspection performance (S115), and receives a plurality of inspection results for the plurality of job results from the plurality of inspectors 34 (S120). ). At this time, the inspector 34 may input the inspection pass or rejection as a plurality of inspection results.

Next, the server extracts an inspection result that satisfies a predetermined condition from among the plurality of inspection results for the plurality of job results, and generates a plurality of verification job results (S125).

For reference, in the description of the present invention, the work result for the purpose of verifying the inspector 34 is referred to as the'verification work result', and in order to distinguish this, the inspector 34 actually performs the inspection and a task that should be used as a project result Let the result be called the'actual work result'.

In an embodiment, the server may extract an actual work result that satisfies a predetermined condition from among a plurality of inspection results for a plurality of actual work results during a predetermined period, and generate a verification work result.

This is to verify the plurality of inspectors 34 to perform the inspection in the second time period (t1) using the inspection result of the actual work result of the first inspector who performed the inspection in the first time period (t0). Likewise, a plurality of inspectors 34 to be inspected in the third time period t2 are verified using the inspection results of the actual work results of the second inspectors who performed the inspection in the second time period t1. Structure.

Hereinafter, with reference to FIGS. 4 and 5 to be described in detail with respect to the operation result for verification.

4 and 5 are diagrams for explaining contents of generating a verification operation result.

The verification work result is generated based on the inspection results of the inspectors 34 on the work actually performed by the plurality of workers 32 of the project.

Referring to FIG. 4, when an actual work result is input from a plurality of workers 32, the actual work result is assigned to an inspector 34, and the inspector 34 inspects the actual work result.

Then, when the inspection by the inspector 34 for the actual work result is completed, the server classifies the plurality of inspection results received from the inspectors 34 into an initial inspection pass, rejection, or re-review pass.

In other words, in one embodiment of the present invention, the server may generate the results of the first inspection, rejection, or re-examination as a result of a job that has been inspected by another inspector 34 among the plurality of inspection results. Extract it as an object.

In this process, the server uses the inspection result of the actual work as valid data to extract the final product at the end of the project, and in the case of the actual work result that has been rejected or re-returned, the task is assigned to the corresponding worker (32) again. Request to perform rework. At the same time, the server extracts the first pass or rejection of the inspection results, and the work results that have passed the re-review in order to generate the verification work results.

For reference, in the case of work results that have passed re-review, the inspector 34, who notified the first rejection, will normally perform re-examination by rework, but if the re-examination deadline is exceeded, the re-examination will be assigned to another inspector 34. It could be. In this case, it corresponds to re-examination of the rework result due to the rejection of the other examiner 34, and the rejection reason input by the other examiner 34 when rejected once is provided to the corresponding examiner 34.

Then, the server extracts an inspection result that satisfies a predetermined condition among the inspection results extracted through the above process. In this case, as an embodiment of the predetermined condition, based on the inspection time required by the plurality of inspectors 34 It may be that it satisfies a predetermined ranking.

That is, the server sorts the actual work results that have passed the inspection, rejected, and passed the re-review in the order in which the time required for the inspection is long, and extracts only the actual work results corresponding to a predetermined rank among them.

Here, the longer time spent on inspection means that the inspection is more likely to be faithfully performed compared to other inspectors (34), which means that there is less probability of error in the inspection result, so the time required for inspection is longer. It is desirable to extract the actual work results.

In this case, the predetermined ranking may be set in advance by the server, the administrator, or the requester 10, and the predetermined ranking may be extracted not only in units of number but also in units of time.

As such, the actual work result that satisfies the predetermined condition may be directly provided as the work result for verification.

However, since the inspector 34 who has inspected the actual work result may also make an error, in order to provide the inspector 34 with a more verified work result for verification, the other inspector 34 can check the actual work result. The result may be re-examined, and an actual work result having an inspection result that matches the re-examination result by another inspector 34 may be extracted as a target for the verification work result.

Referring to FIG. 5, when an actual work result that satisfies a predetermined priority is extracted, the actual work result is subjected to an additional inspection work by different inspectors 34, respectively.

For example, in FIG. 4,'actual work results 1 to 4 (P2)' were passed, rejected, passed and passed, respectively, by the first inspector's inspection. If the actual work results 1 to 3 (P3)' are extracted, the three actual work results (P3) are retested by the second and third inspectors different from the first inspector (P4). In this case, in the example of FIG. 5, it is shown that the re-examination is performed by two inspectors, but is not limited thereto.

As a result of the re-review by the 2nd and 3rd inspectors, the inspection results for'actual work results 1 and 2'out of the re-examination results by the 2nd inspector are consistent with the inspection results of the first inspector, and the third inspector If the inspection results for'actual work results 1 and 3'among the re-examination results match the inspection results of the first inspector,'actual work result 1 (P5), which has the same inspection results by all inspectors (34). 'Only can be created as a result of the verification work.

In this process, in generating the verification work result, the server matches the inspectors 34 who have inspected the actual work result corresponding to the verification work result to mutually match'actual work result-inspector' or'verification work result- Inspector' matching data can be created and stored. This is to prevent repeated assignments to the inspectors 34 who have already performed the inspection in allocating the results of the subsequent verification work to the plurality of inspectors 34. For example, if the inspector A and the actual work result a are matched and stored, in allocating the verification work results to the plurality of inspectors 34, the tester A may not be assigned the verification work result a for the actual work result a. do.

After the result of the verification work is generated, the pool of verification work results may be updated by checking whether the condition is satisfied in real time or every predetermined period with respect to the verification result of the actual work result.

On the other hand, there may be a state in which there is no actual work result that satisfies a predetermined condition, or all the verification work results are allotted to the inspector 34 so that the work results for verification can no longer be assigned. It goes without saying that one embodiment may be assigned to the inspector 34 through the verification work result prepared in advance by the client 10.

Referring to FIG. 3 again, the server allocates the verification result to a plurality of inspectors 34 and receives the inspection result (S130).

In one embodiment, the server allocates the verification work result to each inspector 34 based on the'actual work result-inspector' matching data. For example, when the inspector A matches the actual work result a, the server may allocate the verification work result a generated by the actual work result a to the plurality of inspectors 34 other than the inspector A.

If, after the project inspection cost is updated, the verification work result may be allocated to each inspector 34 based on the'verification work result-inspector' matching data. In other words, if the verification work result, which has already been presented to a specific inspector as the actual work result, is presented to the same specific inspector again, the wrong proficiency level may be judged. Not presented.

In addition, when the server allocates the work result for verification to the inspector 34, the actual work result is also allocated, and the inspector 34 cannot distinguish between the actual work result and the verification work result. That is, the inspector 34 performs the inspection by recognizing the inspected verification work result as an actual work result. Through this, it is possible to determine the proficiency of the inspectors 34 by the result of the verification work.

At this time, the verification work result can be requested by assigning the verification work result to the inspector 34 from the point when the work result of a predetermined ratio or more of the total work included in the project or the inspection result for the work result is submitted. have.

In other words, since proficient inspectors 34 rarely appear in the early stages of the project, allocating the verification work results to the inspectors 34 in advance may be a waste of resources, so the work results corresponding to the mid-to-late project are sufficiently submitted. It is desirable to allocate the results of the verification work from the point of time or from the point when the results of the verification of the work results are sufficiently submitted.

However, the timing of allocating the results of the verification work is not necessarily limited thereto, and of course, it may be allocated from the beginning of the project to periodically check the proficiency level of the inspector pool.

As the server receives the inspection results for the verification work results from the plurality of inspectors 34, the average inspection execution time of the inspectors 34 who have performed the inspection of the verification work results as shown in FIG. By calculating and comparing the recorded inspection execution time for the verification work result, it is possible to adjust the project inspection cost for the plurality of inspectors 34.

6 is a view for explaining the calculation and comparison of the first and second inspection execution time for the verification work result.

The server calculates the average inspection execution time (hereinafter referred to as "first inspection execution time") for each of the verification work results of the plurality of inspectors 34 (S135).

At this time, the average inspection execution time is the average of the inspection execution time calculated based on the start of inspection on the results of the verification tasks assigned to each inspector 34 and the time when the inspection performance is completed and the inspection results are entered. will be.

In one embodiment, the first inspection execution time for the verification work result of the inspector 34 whose number of inspections is less than n (where n is a natural number of 1 or more) among the plurality of inspectors 34 may be excluded.

As an example, the inspector 34 with fewer than n inspections may be a new inspector who is not yet familiar with the inspection, and if too many new inspectors' data are considered together, there is a problem in evaluating the pool of inspectors who are currently proficient. Can be, so can be excluded.

Next, the server compares the first inspection execution time with the inspection execution time (hereinafter referred to as "second inspection execution time") recorded for each verification task result before the calculation of the first inspection execution time ( S140), by using the comparison result, the project inspection means for the plurality of inspectors 34 is adjusted (S145).

Here, the second inspection execution time is the time recorded for the verification work result before the first inspection execution time is calculated, and the inspection performed by the inspector 34 who has inspected the actual work result corresponding to the verification work result. It can be time.

In addition, when the second inspection execution time is updated as the first inspection execution time, the second inspection execution time may be recorded as the average inspection execution time required to inspect the verification work results of the plurality of inspectors 34.

The server calculates deviations between the first inspection execution time and the second inspection execution time for the plurality of verification work results of the plurality of inspectors 34, and compares the first and second inspection execution times. And, based on the calculated average value of each deviation, the project inspection cost for the plurality of inspectors 34 is adjusted.

In an embodiment, the server may calculate an overall average value of the first inspection execution time for a plurality of verification work results and an overall average value of the second inspection execution time for the plurality of verification work results.

In addition, when the total average value of the first inspection execution time has a shorter time than the total average value of the second inspection execution time, the project inspection cost may be reduced by a ratio corresponding to the average value of each deviation.

For example, in FIG. 6, the total average value of the first inspection execution time for the verification work result ID 2001 to 2003 is 85/3 seconds ((30s+45s+10s)/3), and the second inspection execution time The total average value is 145/3 seconds ((50s+60s+35s)/3), and the total average value of the first inspection execution time has a shorter time than the total average value of the second inspection execution time. In this case, the server may reduce the project inspection cost by 20%, which is a ratio corresponding to the average value (-60/3=-20) of each deviation (-20, -15, -25).

In contrast, when the total average value of the first inspection execution time has a longer time than the total average value of the second inspection execution time, the project inspection cost may be increased by a ratio corresponding to the average value of each deviation.

In addition, in the present invention, there may be various embodiments in which the project inspection means price is adjusted based on the average value of each deviation of the first and second inspection execution times. In addition to applying a ratio to the average value of the deviation, the average of the deviation It is also possible to adjust the project inspection cost by classifying and applying values by grade (-11~-20=-10%, -21~30=-20%, etc.).

On the other hand, if the difficulty of the task corresponding to the task result for verification is too high, the deviation from the actual task result of the workers 32 may be large, and it may bring a large deviation to the test result of the inspectors 34 who inspect it. .

In the case of using the work results with large deviations as described above, a result different from the proficiency trend of the inspectors 34 may be reflected when the project inspection means is adjusted. In order to prevent this, an embodiment of the present invention adjusts the project inspection means. In doing so, it is possible to exclude data corresponding to the result of verification work that is too difficult.

In other words, the server calculates the rejection rate for a plurality of jobs corresponding to the results of the plurality of verification tasks based on the plurality of inspection results, and then the difficulty is considered to be too high in the case of a task whose calculated rejection rate is higher than the preset rejection rate. By judging, it is possible to exclude the time for performing the first inspection on the result of the verification work corresponding to the corresponding work.

For example, the rejection rate for a task corresponding to one verification task result is a multi-allocation method in which one actual task is assigned to a plurality of workers 32, worker A passes the initial inspection, and worker B is 1 If the inspection passes after returning twice, and the worker C is rejected twice and passes the inspection, the rejection rate can be calculated as (total of 3 returning/3 persons) = 100%. Or, if only the initial inspection result is considered (total of 2 times/three persons) = 66.7% can be calculated.

In this case, when the deviation between the first inspection execution time and the second inspection execution time is greater than or equal to a preset deviation, the server may exclude the first inspection execution time for the verification task result determined to have too high difficulty.

In other words, even if the rejection rate for the work corresponding to the work result for verification is high, the time of the reviewer 34 who has checked the actual work result for the work result for verification, and the current verification work result When there is little deviation in the average inspection time of the inspectors 34, it can be seen that there is little difference in proficiency between all inspectors 34. On the other hand, in the opposite case, since there is a high possibility that a large deviation exists between the inspectors 34 who have inspected the current verification work result, in this case, the first inspection execution time can be excluded.

In another embodiment, the present invention may classify each of the work results for verification having the work results that have passed the initial inspection, rejection and re-review, and individually manage and apply them.

For example, the verification work result in the present invention targets the work results that have passed the first inspection, rejection, and re-review among the plurality of inspection results, among which the first inspection pass and the rejected work result are the inspector 34 It corresponds to the first inspection of the case, and the result of the work that has passed the re-examination corresponds to the second inspection of the inspector (34).

Since the second inspection case, which passed the re-inspection case, is relatively difficult, such as taking more time to inspect than the first case, the inspector who inspected the actual work result (34) and the inspector who inspected the work result for verification (34) There may be a large deviation between the test results, or a deviation of the test execution time between each of the inspectors 34 who have inspected the result of the verification work.

If the deviation of the inspection time is large, the inspection performance time of the verification work result for the second inspection, which is the case of re-examination, is separated from the initial inspection, and the project inspection means excludes it from the data for adjustment, or It is also possible to adjust the project inspection cost by using it as separate data that is distinguished from the case of inspection.

After the project inspection cost is adjusted, the server may exclude previously used data from the index in order to grasp the trend of the inspectors 34 based on the current standard.

7 is a diagram for explaining content of updating a second inspection execution time.

After the project inspection cost for the plurality of inspectors 34 is adjusted, the server calculates the second inspection execution time recorded for the verification work result as the average inspection execution time for each verification work result of the plurality of inspectors 34 It can be updated with the time of the first inspection.

When the project inspection cost is adjusted for the first time, the second inspection execution time of the inspector (34) who has inspected the actual work result corresponding to the verification work result is excluded, and the second inspection execution time is the verification work result. It is updated with the first inspection execution time of a plurality of inspectors 34.

After the project inspection cost is adjusted again, the second inspection execution time recorded as the average inspection execution time of the plurality of inspectors (34) is excluded, and the first inspection, which is the average inspection execution time of the plurality of inspectors (34). The second inspection execution time is updated with time.

For example, if there are 20 inspectors and the results of verification tasks such as ID 2001 are assigned to 10 people in random order, and the project inspection cost increases from 2,000 won to 3,000 won based on the data, 10 The second inspection execution time is updated as the average inspection execution time for. In this case, the second inspection execution time, which is the average of the plurality of inspectors 34, is updated as the first inspection execution time of the inspector who inspects the actual work result for ID 2001 or the like.

After the project was further progressed, the results of verification work such as ID 2001 were assigned to the remaining 10 inspectors in random order, and as the inspectors (34) became more proficient in the latter half of the project, the cost of inspecting the project was lowered from 3,000 won. If it rises to 4,000 won, the average inspection time for the previous 10 people is excluded, and the second inspection time is updated with the average inspection time for the current 10 people. In this case, the second inspection execution time recorded and updated as the average inspection execution time for ID 2001 or the like is updated to the first inspection execution time, which is a new average by the plurality of inspectors 34.

Through such an update process, it is possible to more accurately determine whether the current inspectors 34 are more proficient than the existing ones.

Meanwhile, in the above description, steps S11 to S145 may be further divided into additional steps or may be combined into fewer steps, according to an embodiment of the present invention. In addition, some steps may be omitted as necessary, and the order between steps may be changed. In addition, even if other contents are omitted, the contents of FIG. 8 to be described later can also be applied to the method of adjusting the project inspection means using the verification work results of the crowdsourcing-based projects of FIGS. 1 to 7.

Hereinafter, a description will be given of a project inspection means control device 200 (hereinafter referred to as a project inspection means control device) using a work result for verification of a crowdsourcing-based project according to an embodiment of the present invention.

8 is a view for explaining a project inspection means adjustment device 200 according to an embodiment of the present invention.

Referring to FIG. 8, the device 200 for adjusting the project inspection means includes a communication module 210, a memory 220, and a processor 230.

The communication module 210 transmits a crowdsourcing-based work for one project to a plurality of workers 32 to request a task to be performed, and receives a work result from the plurality of workers 32. Further, the work results received from the plurality of workers 32 are transmitted to the plurality of inspectors 34 to request inspection, and the inspection results are received from the plurality of inspectors 34. In addition, the communication module 210 allocates the verification work results to the plurality of inspectors 34 to request the inspection performance, and receives the inspection results for the verification work results from the plurality of inspectors 34.

The memory 220 stores a program for adjusting the project inspection cost of the plurality of inspectors 34 based on the inspection execution time for the verification work result.

As the processor 230 executes the program stored in the memory 220, after generating the verification work result, assigns the verification work result to the plurality of inspectors 34, and performs an average inspection on the verification work result The inspection cost of the inspectors 34 is adjusted by comparing the time and the inspection execution time recorded in the verification work result.

The method for adjusting the project inspection means using the work result for verification of a crowdsourcing-based project according to an embodiment of the present invention described above is implemented as a program (or application) to be executed in combination with a computer, which is hardware, and stored in a medium. Can be.

The above-described program includes C, C++, JAVA, Ruby, which can be read by a processor (CPU) of the computer through the device interface of the computer, in order for the computer to read the program and execute the methods implemented as a program. It may include a code (Code) coded in a computer language such as machine language. Such code may include a functional code related to a function defining necessary functions for executing the methods, and a control code related to an execution procedure necessary for the processor of the computer to execute the functions according to a predetermined procedure. can do. In addition, such code may further include additional information required for the processor of the computer to execute the functions or code related to a memory reference to which location (address address) of the internal or external memory of the computer should be referenced. have. In addition, when the processor of the computer needs to communicate with any other computer or server in the remote in order to execute the functions, the code uses the communication module of the computer to determine how It may further include a communication-related code for whether to communicate, what kind of information or media should be transmitted and received during communication.

The stored medium is not a medium that stores data for a short moment, such as a register, cache, memory, etc., but a medium that stores data semi-permanently and can be read by a device. Specifically, examples of the storage medium include, but are not limited to, ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, and the like. That is, the program may be stored in various recording media on various servers to which the computer can access, or on various recording media on the user's computer. In addition, the medium may be distributed over a computer system connected through a network, and computer-readable codes may be stored in a distributed manner.

The above description of the present invention is for illustrative purposes only, and those of ordinary skill in the art to which the present invention pertains will be able to understand that it can be easily modified into other specific forms without changing the technical spirit or essential features of the present invention. will be. Therefore, it should be understood that the embodiments described above are illustrative in all respects and not limiting. For example, each component described as a single type may be implemented in a distributed manner, and similarly, components described as being distributed may also be implemented in a combined form.

The scope of the present invention is indicated by the claims to be described later rather than the detailed description, and all changes or modified forms derived from the meaning and scope of the claims and their equivalent concepts should be interpreted as being included in the scope of the present invention. do.

10: Client
20: service provider
30: public
32: worker
34: inspector
200: Project inspection means control device
210: communication module
220: memory
230: processor

Claims (11)

As a method performed by a crowdsourcing platform server computer (hereinafter, "server"),
The server allocating a plurality of actual tasks of a crowdsourcing-based project (hereinafter, "project") to a plurality of workers and requesting the task to be performed;
Receiving, by the server, a plurality of actual work results from the plurality of workers;
The server assigning the plurality of actual work results to a plurality of inspectors and requesting to perform inspection;
Receiving, by the server, an inspection pass or rejection as a plurality of inspection results for the plurality of actual work results from the plurality of inspectors;
Based on the plurality of inspection results for the plurality of actual work results during the first time period, the server uses the actual work results corresponding to the inspection results that satisfy a predetermined condition, and stores the plurality of verification work results. Generating;
Receiving, by the server, the plurality of verification work results for a second time period after the first time period by assigning the plurality of actual work results to a plurality of inspectors who perform the inspection;
Calculating, by the server, an average inspection execution time (hereinafter, "first inspection execution time") of the plurality of inspectors in relation to each verification operation result;
The first inspection execution time for each of the verification work results, and the inspection execution time recorded for each actual work result corresponding to the respective verification work results (hereinafter, "second inspection execution time") Comparing them; And
Including the step of adjusting, by the server, a project inspection cost for the plurality of inspectors using the comparison result,
The server determines the first inspection execution time for each verification work result and the inspection execution time recorded for each actual work result corresponding to each verification work result (hereinafter, "second inspection execution time"). The steps to compare are,
And calculating, by the server, a deviation between a first inspection execution time and a second inspection execution time for each of the plurality of verification work results,
The step of adjusting, by the server, a project inspection cost for the plurality of inspectors by using the comparison result,
Including the step of adjusting, by the server, project inspection means for the plurality of inspectors based on the average value of each deviation calculated for each of the plurality of verification work results,
A method of adjusting the cost of project inspection using the results of verification of crowdsourcing-based projects. The method of claim 1,
The predetermined conditions are:
It is a condition that satisfies a predetermined ranking based on the inspection execution time by the plurality of inspectors,
A method of adjusting the cost of project inspection using the results of verification of crowdsourcing-based projects. The method of claim 2,
The step of generating a plurality of verification work results by the server,
Requesting, by the server, at least one other inspector different from the inspector who has inspected the actual work result corresponding to the inspection result satisfying the predetermined condition; And
Including the step of generating the plurality of verification work results by using the actual work results corresponding to the inspection results that satisfy the predetermined condition and match the re-review results by the other inspectors,
A method of adjusting the cost of project inspection using the results of verification of crowdsourcing-based projects. The method of claim 1,
The step of generating the plurality of verification work results,
Based on the plurality of inspection results for the plurality of actual work results, to generate the work results that have passed the initial inspection, rejection or re-review by another inspector as the plurality of verification work results,
A method of adjusting the cost of project inspection using the results of verification of crowdsourcing-based projects. The method of claim 1,
The step of receiving, by the server, assigning the plurality of verification work results to a plurality of inspectors who perform the inspection by receiving the plurality of actual work results during a second time period after the first time period, and receiving the inspection results,
In relation to the tasks of a predetermined ratio or more among the total tasks included in the project, the results of the plurality of verification tasks are assigned and inspected after the actual task results or the inspection results for the actual task results are submitted. To request inspection by assigning to multiple inspectors performing
A method of adjusting the cost of project inspection using the results of verification of crowdsourcing-based projects. The method of claim 1,
The step of calculating, by the server, an average inspection execution time (hereinafter, "first inspection execution time") of the plurality of inspectors in relation to each of the verification work results,
Of the plurality of inspectors, the number of inspections for the actual work result is less than n (however, n is a natural number of 1 or more).
A method of adjusting the cost of project inspection using the results of verification of crowdsourcing-based projects. The method of claim 1,
The step of calculating, by the server, a deviation between the first inspection execution time and the second inspection execution time for each of the plurality of verification operation results,
Calculate rejection rates for the plurality of verification work results based on the plurality of inspection results,
The calculated rejection rate is to exclude and calculate the first inspection execution time for the verification work result that is more than a preset rejection rate,
A method of adjusting the cost of project inspection using the results of verification of crowdsourcing-based projects. The method of claim 7,
Excluding and calculating the first inspection execution time for the verification work result in which the calculated rejection rate is greater than or equal to a preset rejection rate,
The first inspection execution time for the verification work result in which the calculated rejection rate is equal to or greater than a preset rejection rate and the second inspection for an actual job corresponding to the verification work result in which the calculated rejection rate is equal to or greater than a preset rejection rate When the deviation of the execution time is greater than or equal to a preset deviation, the calculated rejection rate is calculated by excluding and calculating the first inspection execution time for the verification work result in which the calculated rejection rate is greater than or equal to the preset rejection rate,
A method of adjusting the cost of project inspection using the results of verification of crowdsourcing-based projects. The method of claim 1,
The step of adjusting, by the server, a project inspection cost for the plurality of inspectors by using the comparison result,
The server takes a shorter time than the total average value of the second inspection execution time for each actual work result corresponding to the verification work result. If so, reducing the cost of the inspection means for the plurality of inspectors by a ratio corresponding to the average value of the calculated deviations; And
The server determines that the total average value of the first inspection execution time for each verification work result is longer than the total average value of the second inspection execution time for each actual work result corresponding to each verification work result. If yes, comprising the step of increasing the inspection means cost of the project for the plurality of inspectors by a ratio corresponding to the average value of the calculated deviation,
A method of adjusting the cost of project inspection using the results of verification of crowdsourcing-based projects. The method of claim 1,
After the server has adjusted the project inspection means for the plurality of inspectors, the second inspection execution time for each actual work result corresponding to the verification work result is determined by the plurality of inspectors in relation to the respective verification work results. Further comprising the step of updating to the first inspection execution time, which is the average inspection execution time of,
A method of adjusting the cost of project inspection using the results of verification of crowdsourcing-based projects. A computer program combined with a computer and stored in a computer-readable recording medium in order to execute the control method by the project inspection means using the work result for verification of the crowdsourcing-based project of any one of claims 1 to 10.

Images