KR102155788B1 - Method and apparatus for arranging inspection order of work results according to the number of worker's return count - Google Patents

Abstract

Provided are a method and an apparatus for placing inspection sequences of a job result in accordance with the number of returns of a worker. The method includes the following steps of: granting a plurality of crowdsourcing-based jobs to a plurality of workers with respect to one single project; receiving a plurality of pieces of job result data from the plurality of workers; placing the plurality of pieces of job result data on an inspection standby list in a submission sequence and sequentially requesting an inspector for an inspection; detecting a first worker, of which the number of returns exceeds a preset number, of the plurality of workers as a result of the inspection; and transferring uninspected job result data of another worker, which is behind the next inspection turn of uninspected job result data of the first worker, to a previous position of the inspection turn on the inspection standby list. Therefore, the present invention is capable of preventing an increase in time required for an entire project.

Description

Method and arrangement of the order of inspection of work results according to the number of return of workers {METHOD AND APPARATUS FOR ARRANGING INSPECTION ORDER OF WORK RESULTS ACCORDING TO THE NUMBER OF WORKER'S RETURN COUNT}

The present invention relates to a method and apparatus for arranging the inspection order of work results according to the number of times the operator is rejected.

Recently, more and more companies collect and process large amounts of data on a crowdsourcing basis that engages the general public in some process of corporate activities. In other words, by opening a project and allowing the general public, that is, workers to participate in the project, necessary information is collected through the work results completed by the workers.

And, in order to collect more reliable information, the company assigns the results of work completed by the worker to the inspector to perform the inspection work.

On the other hand, there are various types of tasks provided to workers, and various work capabilities are required to perform each task. And, each worker naturally has a weak part and a superior part among various types of work for each individual.

In addition, a specific worker may be a defective worker whose work ability is significantly lower than that of other workers for various reasons.

1 is a view for explaining a case of sequentially inspecting work result data in the order of completion of work.

If the inspection is conducted in the order of submission of the work result data of all workers including the defective worker A, there are cases in which only the work result data of a specific worker who performed a large amount of work at a time is continuously inspected, so other workers You have to wait a long time.

At this time, if a specific worker is a defective worker A who continues to receive rejection results, the inspectors must continue to inspect only the work result data of the defective worker. In addition, since the work result data of the defective worker A will be returned continuously, only the inspection cost is continuously required, and the data required for the project cannot be collected because rework and re-review must be repeated.

In addition, on the inspection waiting list, the work result data of other workers, which are placed in a lower priority than the work result data of defective worker A, are not inspected by the inspector, and the inspection waiting state continues, resulting in a problem that the schedule of the entire project is delayed.

Unexamined Patent Publication No. 10-2014-0095956, 2014.08.04.

The problem to be solved by the present invention is to provide a method and apparatus for arranging the inspection order of work results according to the number of times the operator is returned.

The problems to be solved by the present invention are not limited to the problems mentioned above, and other problems that are not mentioned will be clearly understood by those skilled in the art from the following description.

As a solution to the above-described problem, the method according to the first aspect of the present invention comprises the steps of granting a plurality of crowdsourcing-based tasks for one project to a plurality of workers, and a plurality of work result data from the plurality of workers. Receiving the input, placing the plurality of work result data on the inspection waiting list according to the submission order and sequentially requesting inspection from the inspector, the first worker having a rejection number of more than a preset number of the plurality of workers as a result of inspection A step of detecting, and moving the unchecked work result data of another worker, which has a lower priority than the inspection sequence number, to a previous position of the next inspection sequence among the unchecked work result data of the first worker on the inspection waiting list. Include.

In addition, as a means of solving the above-described problem, the computer program according to the second aspect of the present invention is combined with a computer as hardware to execute the above inspection order arrangement method, and is stored in a computer-readable recording medium.

In addition, as a means of solving the above-described problem, the apparatus according to the third aspect of the present invention transmits a plurality of crowdsourcing-based tasks for one project to a plurality of workers, and a plurality of work results from the plurality of workers A communication module that receives data, transmits the plurality of work result data to a plurality of inspectors sequentially according to the arrangement order on the inspection waiting list, and receives inspection results from the plurality of inspectors, and the inspection result received from the communication module A memory in which a program for arranging the inspection order of the job result data is stored, and the plurality of job result data is placed on the inspection waiting list according to the submission order by executing the program stored in the memory, and , On the basis of the inspection result, a first worker having a rejection number of more than a preset number is detected among the plurality of workers, and a previous position of the next inspection sequence among unchecked work result data of the first worker on the inspection waiting list And a processor for moving the unchecked work result data of another worker, which has a lower priority than the inspection order number.

Other specific details of the present invention are included in the detailed description and drawings.

According to the present invention described above, by adjusting the inspection order based on the number of rejections of the operator, rather than simply sequentially inspecting the work result data received from a plurality of workers, the work result data of a normal worker is naturally and preferentially inspected. can do.

Accordingly, by imposing a penalty for defective workers, adjusting the inspection order to a lower priority, and inspecting the work result data of other normal workers first, the inspection time increases according to the high rejection rate, increasing the time required for the entire project. You can avoid problems.

As a result, the quality of the work result is improved by reducing the amount of defective work in the future due to the penalty that the inspection opportunity is given to the defective worker as a subordinate priority, and accordingly, the bottleneck of inspection can be prevented, and the inspector can perform the inspection. You can do it efficiently.

The effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the following description.

1 is a view for explaining a case of sequentially inspecting work result data in the order of completion of work.
2 is a conceptual diagram of a crowdsourcing service according to an embodiment of the present invention.
3 is a flowchart illustrating a crowdsourcing-based project progress process according to an embodiment of the present invention.
4 is a flowchart illustrating a method of arranging an order of inspection of work result data according to an embodiment of the present invention.
5 is a diagram for explaining content of detecting a first worker.
6 is a view for explaining the content of adjusting the inspection order of the first operator.
7A and 7B are diagrams for explaining an example of application of a method for arranging an inspection order according to an embodiment of the present invention.
8 is a diagram illustrating an apparatus for arranging an inspection order of work result data according to an embodiment of the present invention.

Advantages and features of the present invention, and a method of achieving them will become apparent with reference to the embodiments described below in detail together with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but may be implemented in a variety of different forms, only the present embodiments are intended to complete the disclosure of the present invention, It is provided to fully inform the technician of the scope of the present invention, and the present invention is only defined by the scope of the claims.

The terms used in the present specification are for describing exemplary embodiments and are not intended to limit the present invention. In this specification, the singular form also includes the plural form unless specifically stated in the phrase. As used in the specification, “comprises” and/or “comprising” do not exclude the presence or addition of one or more other elements other than the mentioned elements. Throughout the specification, the same reference numerals refer to the same elements, and “and/or” includes each and all combinations of one or more of the mentioned elements. Although "first", "second", and the like are used to describe various elements, it goes without saying that these elements are not limited by these terms. These terms are only used to distinguish one component from another component. Therefore, it goes without saying that the first component mentioned below may be the second component within the technical idea of the present invention.

Unless otherwise defined, all terms (including technical and scientific terms) used in the present specification may be used as meanings that can be commonly understood by those of ordinary skill in the art to which the present invention belongs. In addition, terms defined in a commonly used dictionary are not interpreted ideally or excessively unless explicitly defined specifically.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings.

2 is a conceptual diagram of a crowdsourcing service according to an embodiment of the present invention.

Referring to FIG. 2, the crowdsourcing service is performed by being composed of a client 10, a service provider 20, and a public 30.

The sponsor 10 refers to a company or individual who requests a crowdsourcing-based project (hereinafter, referred to as a project).

The requester 10 requests a project for the purpose of collecting source data or annotating data. Collection of source data refers to collecting raw data such as recorded voice collection and photo collection. Data annotation refers to inputting relevant annotation data into source data such as text, photos, and videos. For example, the data annotation may include finding an entity in a given fingerprint or finding a similar sentence, but is not limited thereto. Meanwhile, the types of the above-described projects are only one embodiment, and various projects may be handled in the present invention according to the design of the client.

The service provider 20 refers to a company that provides crowdsourcing services.

When the service provider 20 receives a request for a project for a product or service from the client 10, the service provider 20 allocates a task for the project to the general public 30 and receives the work result from the public 30. Thereafter, the final product extracted based on the work result is provided to the client 10.

At this time, the service provider 20 provides a crowdsourcing service to the client 10 and the public 30 through a crowdsourcing platform (hereinafter, the platform). That is, when the service provider 20 receives a request for a project from the client 10, the service provider 20 opens the project on the platform. Thereafter, when the work result for the open project is received from the public 30, the project is terminated on the platform, and the final product may be extracted and provided to the client 10.

The public 30 refers to the general public participating in the project open on the platform. Here, the public 30 may participate in a project opened on the platform through an application or website provided by the service provider 20.

The public 30 is composed of a worker 32 and an inspector 34.

The worker 32 decides to participate in a specific project among a plurality of projects open on the platform. Thereafter, the operator 32 performs an operation such as collection of source data or data annotation, and transmits it to the platform.

The inspector 34 decides to participate in a specific project among a plurality of projects open on the platform. Thereafter, the inspector 34 inspects the result of the work performed by the operator 32. As a result of performing the inspection, the inspector 34 may pass or reject the job, and may input a reason for rejection when returning the job.

3 is a flowchart illustrating a crowdsourcing-based project progress process according to an embodiment of the present invention.

First, the requester 10 requests one or more projects to the service provider 20 (S11).

Thereafter, the service provider 20 opens the requested project on the platform (S12). At this time, the service provider 20 may determine the grade in consideration of the difficulty of the project before opening the project. That is, depending on the difficulty level, it is possible to determine which level or higher public 30 to expose the project. Accordingly, it is possible to increase the reliability of the work result of the project.

Thereafter, the service provider 20 allocates a task to the workers 32 of the corresponding grade or higher according to the grade of the project (S13).

Thereafter, the worker 32 performs the assigned task (S14). In this case, the worker 32 may input the reason for the inability to work without performing the work for a work in which the work itself is impossible for some reason.

Thereafter, the service provider 20 receives the work result from the worker 32 (S15), and assigns the inspection work on the work result to the inspector 34 (S16).

Thereafter, the inspector 34 performs the assigned inspection (S17). At this time, the inspector 34 decides to pass the inspection (or complete the inspection) when it is determined that the work has been properly performed, and decides rejection when it is determined that the inspection work is wrong. When deciding to reject, the inspector 34 inputs the rejection reason for what reason the task was judged to be wrong.

In this case, in an embodiment of the present invention, the inspector does not sequentially inspect all the work result data received from a plurality of workers, but the number of rejections is more than a certain number, so that the work result data of the worker selected as a defective worker is higher than that. It is characterized by giving a penalty for adjusting the inspection order so that it is inspected at a lower priority than the work result data of other workers to prevent the bottleneck of inspection, and to allow the work result data of normal workers, not bad workers, to be inspected with priority. .

Thereafter, the service provider 20 receives the inspection result from the inspector 34 (S18).

If the inspection result is Pass, the service provider 20 uses the work result as valid data, and based on this, extracts the final product at the end of the project.

If the inspection result is rejected, the service provider 20 may internally perform the inspection again or assign the job again to the worker 32 to perform the job.

Thereafter, when the project period ends or sufficient valid data is secured, the service provider 20 terminates the project (S19), calculates a final result based on the secured valid data, and provides it to the client 10 ( S20).

At this time, before the end of the project, the service provider 20 evaluates the performance results of the worker 32 and the inspector 34, calculates the work cost and the inspection cost according to the evaluation, to the worker 32 and the inspector 34. give.

On the other hand, in Figures 2 and 3, although simply expressed as the client 10, the service provider 20, the worker 32, and the inspector 34, these are smartphones, tablets, PDAs, laptops, operated by each participant. It means a computer device or a telecommunication device such as a desktop or a server.

4 is a flowchart illustrating a method of arranging an order of inspection of work result data according to an embodiment of the present invention.

Meanwhile, the steps shown in FIG. 4 may be understood as being performed by a platform server (hereinafter, referred to as a server) operated by the service provider 20, but are not limited thereto.

Referring to FIG. 4, the server assigns a crowdsourcing-based work for one project to a plurality of workers 32 (S110), and receives work result data from the plurality of workers 32 (S120).

Here, one project may contain multiple jobs, and the plurality of jobs may be grouped into a plurality of tasks according to their type, or each job may be a different type of job or the same job. , But is not limited thereto.

In addition, a plurality of workers 32 or a plurality of inspectors 34 perform work or inspection using a predetermined terminal device. The terminal device of the worker 32 may be a computer device or a telecommunication device such as a smart phone, tablet, PDA, laptop, desktop, etc., but is not limited thereto.

Next, the server arranges the work result data in the order of completion of the work and sequentially allocates it to the inspector 34 (S130), and detects the first worker having a rejection number of more than a preset number of the plurality of workers 32 Do (S140).

The server arranges the work result data of a plurality of workers on the inspection waiting list according to the order in which each worker submits the work result data. The plurality of job result data on the inspection waiting list are sequentially assigned to the inspector 34 and inspected by the inspector 34.

Here, the preset number of rejections may be preset by the server according to the reliability. That is, the server may set the number of rejections lower as the level requested by the client 10 or the level required according to the nature of the project is higher.

5 is a diagram for explaining content of detecting a first worker.

When a plurality of workers A to C exist, work result data by the plurality of workers A to C on the inspection waiting list is allocated to the inspector 34 to be sequentially inspected. Accordingly, the first three work result data are inspected in the order of worker A, worker B, and worker C.

At this time, for example, when the preset number of rejections is set to one, as shown in FIG. 5, as the rejection of worker A occurs, worker A can be detected as a defective worker, that is, the first worker to adjust the inspection order. have.

Referring to FIG. 4 again, the server moves the unchecked work result data of another worker with a lower priority to the previous position of the next inspection sequence among the unchecked work result data of the first worker on the inspection waiting list (S150). Accordingly, the inspection order of the unchecked work result data of the first worker is changed to a lower priority.

6 is a view for explaining the content of adjusting the inspection order of the first operator.

After the first worker is detected, a predetermined number of inspection waiting areas are additionally created in the previous position of the next inspection sequence among the unchecked work result data of the first worker on the inspection waiting list. In addition, among the unchecked work result data of the first worker, the unchecked work result data of another worker positioned in a lower priority than the next inspection sequence number is moved to the inspection waiting area.

The number of additional inspection waiting areas may be set as a multiple of the number of rejections (e.g., when the number of rejections is X, the inspection waiting area is N*X generated, and N is a natural number of 1 or more). no.

As shown in FIG. 5, in the example in which the inspection order is sequentially determined and the number of rejections of the worker A meets the preset rejection number of 1, and is detected as the first worker, the case where the predetermined number of inspection waiting areas is set to two It will be described with reference to the example of FIG. 6.

On the inspection waiting list, after worker A's rejected work result data, 5 other worker's unchecked work result data is placed, and then in the next order, the next inspection order among the unchecked work result data of worker A (i.e. It is assumed that the fastest) work result data 41 is arranged, and after the unchecked work result data 41 of the worker A, unchecked work result data of other workers are arranged in order.

However, as worker A is selected as the first worker, two new inspection waiting areas are newly created in the previous position of the next number sequence number 41, and the work result data 41 of the next inspection sequence number of worker A in the inspection waiting area. The work result data 42 and 43 of the workers C and B located thereafter are moved, so that the work result data of the worker A is inspected in priority.

Unlike the examples of FIGS. 5 to 6, it is as described above that the number of rejections for determining as a defective worker may be set to two or more.

In this way, the work result data of worker A, which has a large number of rejections, is given a penalty so that it is inspected at a lower priority than other workers, and the work result data of normal workers B and C are inspected first, thereby improving the inspection speed. It has the advantage of shortening the overall lead time of the project. In addition, by improving the quality of work results so that worker A does not receive a penalty that the order of inspection is adjusted to a lower priority due to rejection behavior, in conclusion, the quality of work results of the entire project can be improved.

On the other hand, in an embodiment of the present invention, a predetermined number of inspection waiting areas generated in the previous position of the next inspection sequence of the first worker on the inspection waiting list is a predetermined number according to a penalty according to an increase in the number of rejections of the first worker. It can be weighted and created.

For example, if the preset number of rejections is set to one, if the work result data of the first worker is rejected once, the inspection order may be delayed by two areas in a lower order, and not reset again, but by 1 from the third time. A predetermined number of inspection waiting areas may be additionally generated according to various methods, such as being delayed in a lower order by one area each time it is returned (that is, three inspection waiting areas are additionally created).

According to an embodiment of the present invention, it is determined whether or not the number of rejections exceeding a preset number is determined for the unchecked work result data of another worker moved into the inspection waiting area created as above, and a second worker that satisfies the detection is detected. can do.

In other words, the unchecked work result data in the newly created inspection waiting area is only an advance in the inspection order, and since it does not specifically simplify inspection or prevent inspection, it must be treated the same as other general work result data.

If a second worker with a rejection number greater than or equal to a preset number of rejections is detected in the inspection waiting area generated by the repeated rejection process by the first worker, it is determined to the previous position of the next order among the result data of the second worker's unchecked work. An additional number of inspection waiting areas are created.

In addition, among the unchecked work result data of the second worker, the unchecked work result data of another worker, which is located in a lower priority than the next inspection sequence number, is moved to the inspection waiting area generated corresponding to the repeated rejection processing of the second worker.

At this time, the unchecked work result data of the other worker moving to the inspection waiting area also includes the unchecked work result data of the first worker who has been pushed down to the next level.

That is, the rejection of the work result data of the second worker occurring in the inspection waiting area generated due to the rejection of the work result data of the first worker is independently processed from the first worker who received the penalty earlier.

On the other hand, in this case, in consideration of the rejection of the first worker, the unchecked work result data of the first worker may be placed in a lower order than the unchecked work result data of the first worker and other third workers and moved.

On the other hand, differently, one embodiment of the present invention comprehensively considers the number of rejections, the total number of penalties, and whether or not the generated inspection waiting area was created by any worker, and the unchecked work result data in the inspection waiting area. You can also set the order of arrangement.

Hereinafter, an application example of the present invention will be described with reference to FIGS. 7A and 7B.

7A and 7B are diagrams for explaining an example of application of a method for arranging an inspection order according to an embodiment of the present invention.

7A and 7B are cases in which a total of 25 work result data from 1001 to 1025 by workers A and B C are in a state of waiting for inspection and then partially inspected and rejected. In this example, the preset number of rejections is set to 1, and the predetermined number of waiting areas for inspection is set to 2. The data ID is assigned in the order in which job result data is submitted.

Referring to FIG. 7A, worker A is in a state in which job result data from 1001 to 1006 are first submitted compared to other workers. At this time, when the inspection result of the work result data of No. 1002 is rejected, two inspection waiting areas are created before No. 1003, which is the next inspection order, among the unchecked work result data of worker A.

In addition, the unchecked work result data of the other workers located in the next inspection sequence is moved to the two inspection waiting areas, and as a result, the work result data of worker B's No. 1007 and No. 1008 are moved to the priority.

As another example, referring to FIG. 7B, if the work result data of worker A from 1001 to 1006 is inspected, and the result data of work 1006 is rejected, then, worker B and worker C from 1007 to 1013 The result data of the work is inspected without changing the order.

However, two inspection waiting areas are created before the next inspection sequence number 1014 among the untested work result data of worker A, and the inspection waiting area is the untested work results 1016 and 1017 by worker A and other worker B. Data is moved. As a result, the result data of worker B's work No. 1016 and No. 1017 are inspected first, and the work result data of worker A's work No. 1014 and No. 1015 are inspected in a lower order.

Meanwhile, in the above description, steps S110 to S150 may be further divided into additional steps or may be combined into fewer steps, according to an embodiment of the present invention. In addition, some steps may be omitted as necessary, and the order between steps may be changed. In addition, even if other contents are omitted, the contents of FIG. 8 to be described later may be applied to the method of arranging the inspection order of FIGS. 1 to 7B.

Hereinafter, with reference to FIG. 8, a description will be given of the apparatus 100 for placing the inspection order according to the number of rejections of the worker 32 according to an embodiment of the present invention.

8 is a diagram for explaining an apparatus 100 for arranging an inspection order of inspection result data according to an embodiment of the present invention.

Referring to FIG. 8, the inspection order arrangement apparatus 100 includes a communication module 110, a memory 120, and a processor 130.

The communication module 110 transmits a plurality of crowdsourced jobs for one project to a plurality of workers 32 and receives a plurality of job result data from the plurality of workers 32.

In addition, the communication module 110 transmits the work result data input from the plurality of workers 32 to the plurality of inspectors 34 sequentially according to the arrangement order on the inspection waiting list to be inspected, and the plurality of inspectors 34 The inspection result is received from.

The memory 120 stores a program for arranging the inspection order of work result data based on the inspection result received from the communication module 110.

The processor 130 executes a program stored in the memory 120. As the processor 130 executes the program stored in the memory 120, the plurality of job result data is placed on the inspection waiting list according to the order of submission, and based on the inspection result, the plurality of workers 32 Detects the first worker who has a rejection number of more than the set number, and in the previous position of the next inspection sequence among the unchecked job result data of the first worker on the inspection waiting list, the unchecked job result of another worker with a lower priority than the inspection sequence Let the data move.

The inspection order arrangement apparatus 100 described with reference to FIG. 8 may be provided as a component of the above-described server.

The method for arranging the inspection order according to an embodiment of the present invention described above may be implemented as a program (or application) and stored in a medium to be executed by being combined with a computer that is hardware.

The above-described program includes C, C++, JAVA, Ruby, which can be read by a processor (CPU) of the computer through the device interface of the computer, in order for the computer to read the program and execute the methods implemented as a program. It may include a code (Code) coded in a computer language such as machine language. Such code may include a functional code related to a function that defines necessary functions for executing the methods, and includes a control code related to an execution procedure necessary for the processor of the computer to execute the functions according to a predetermined procedure. can do. In addition, these codes may further include additional information required for the processor of the computer to execute the functions, or code related to a memory reference to which location (address address) of the internal or external memory of the computer should be referenced. have. In addition, when the processor of the computer needs to communicate with any other computer or server in the remote in order to execute the functions, the code uses the communication module of the computer to determine how It may further include a communication-related code for whether to communicate, what kind of information or media should be transmitted and received during communication.

The stored medium is not a medium that stores data for a short moment, such as a register, cache, memory, etc., but a medium that stores data semi-permanently and can be read by a device. Specifically, examples of the storage medium include, but are not limited to, ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, and the like. That is, the program may be stored in various recording media on various servers to which the computer can access, or on various recording media on the user's computer. In addition, the medium may be distributed over a computer system connected through a network, and computer-readable codes may be stored in a distributed manner.

The above description of the present invention is for illustrative purposes only, and those of ordinary skill in the art to which the present invention pertains can understand that it is possible to easily transform it into other specific forms without changing the technical spirit or essential features of the present invention. will be. Therefore, it should be understood that the embodiments described above are illustrative in all respects and not limiting. For example, each component described as a single type may be implemented in a distributed manner, and similarly, components described as being distributed may also be implemented in a combined form.

The scope of the present invention is indicated by the claims to be described later rather than the detailed description, and all changes or modified forms derived from the meaning and scope of the claims and their equivalent concepts should be interpreted as being included in the scope of the present invention. do.

10: Client
20: service provider
30: public
32: worker
34: inspector
100: inspection order arrangement device
110: communication module
120: memory
130: processor

Claims (10)

As a method performed by a computer,
Assigning a plurality of crowdsourcing-based tasks for one project to a plurality of workers;
Receiving a plurality of job result data from the plurality of workers;
Arranging the plurality of job result data on an inspection waiting list according to a submission order and sequentially requesting inspection from an inspector;
Detecting a first worker having a rejection number equal to or greater than a preset number of the plurality of workers as a result of the sequential inspection by the inspector; And
By imposing a penalty according to the number of rejections, the unchecked work result data of another worker with a lower priority than the inspection sequence is moved to the previous position of the next inspection sequence among the unchecked work result data of the first worker on the inspection waiting list. Including the step of,
The step of moving the unchecked work result data of the other worker,
Generating a predetermined number of first inspection waiting areas on the inspection waiting list at a previous position of the next inspection sequence among unchecked work result data of the first worker; And
Including the step of changing the inspection sequence number of the unchecked work result data of the first worker to a lower priority by moving the unchecked work result data of another worker having a lower priority than the inspection sequence in the predetermined number of first inspection waiting areas. and,
The step of creating a predetermined number of first inspection waiting areas at a previous position of the next inspection sequence,
The predetermined number is weighted and set according to a penalty according to an increase in the number of rejections of the first worker, and the predetermined number of first inspection waiting areas is generated. delete delete The method of claim 1,
The method further comprising the step of detecting a second worker having a rejection number of more than a preset number among the other workers as a result of the inspection of the unchecked work result data of another worker moved in the first inspection waiting area. The method of claim 4,
Generating the predetermined number of second inspection waiting areas on the inspection waiting list at a previous position of the next inspection sequence among the unchecked work result data of the second worker; And
Further comprising the step of changing the inspection sequence number of the unchecked work result data of the second worker to a lower priority by moving the unchecked work result data of another worker having a lower priority than the inspection order in the second inspection waiting area, How to arrange inspection order. The method of claim 5,
The unchecked work result data of another worker moved in the second inspection waiting area includes unchecked work result data of the first worker. The method of claim 6,
The unchecked work result data of the first worker is moved to a lower priority than the unchecked work result data of the first worker and another third worker. The method of claim 1,
The preset number of times is set according to the reliability required for the project. A computer program stored in a computer-readable recording medium for executing the inspection order arrangement method of any one of claims 1, 4 to 8, combined with a computer as hardware. Crowdsourcing for one project is sent to a plurality of workers, a plurality of work result data is received from the plurality of workers, and the plurality of work result data is sequentially arranged according to the order of arrangement on the inspection waiting list. A communication module for transmitting to a plurality of inspectors and receiving inspection results from the plurality of inspectors;
A memory storing a program for arranging an inspection order of the work result data based on the inspection result received from the communication module; And
As the program stored in the memory is executed, the plurality of job result data is placed on the inspection waiting list according to the submission order, and a preset number of times among the plurality of workers based on the sequential inspection results by the inspector Detects the first worker with more than the number of rejections, and imposes a penalty according to the number of rejections, and ranks in the previous position of the next inspection sequence among the untested work result data of the first worker on the inspection waiting list. Including a processor for moving the unchecked work result data of other workers,
Moving the unchecked work result data of the other worker,
On the inspection waiting list, a predetermined number of first inspection waiting areas are created at a previous position of the next inspection sequence among unchecked work result data of the first worker,
Including changing the inspection order of the unchecked work result data of the first worker to a lower priority by moving the unchecked work result data of another worker having a lower priority than the inspection order in the predetermined number of first inspection waiting areas, ,
Generating a predetermined number of first inspection waiting areas at the previous position of the next inspection sequence,
The predetermined number is weighted and set according to a penalty according to an increase in the number of rejections of the first operator, and the predetermined number of first inspection waiting areas is generated.

Images