KR102155879B1 - Method for managing inspection quality using feedback from workers of crowdsourcing based projects for artificial intelligence training data generation - Google Patents

Abstract

Provided is a method of managing the inspection quality using feedback from workers of crowd sourcing-based projects for artificial intelligence (AI) training data generation. The method generates a work guide and an inspection guide of the crowd sourcing-based projects, and provides the work guide and the inspection guide to the workers and the inspectors participating in the projects, respectively. When the workers and the inspectors check the work guide and the inspection guide, works of the projects are allocated to the workers and the work execution is requested to the worker, and then an input of the work results is received. Then, the work results are allocated to the inspectors, and the inspection execution is requested to the inspectors, and then an input of the inspection results is received. When the inspection results of some works are received as fail from the inspectors, re-works are allocated to the workers, and the re-work execution is requested to the workers. In response to the same, the re-work results or the objections to the inspection results are received from the workers. The method of managing the inspection quality using feedback from workers of crowd sourcing-based projects for AI training data generation is able to improve reliability of the inspection results.

Description

Inspection quality management method using feedback from workers in crowdsourcing-based projects for artificial intelligence learning data generation {METHOD FOR MANAGING INSPECTION QUALITY USING FEEDBACK FROM WORKERS OF CROWDSOURCING BASED PROJECTS FOR ARTIFICIAL INTELLIGENCE TRAINING DATA GENERATION}

The present invention relates to a quality control method for inspection using feedback from workers in a crowdsourcing-based project for generating artificial intelligence learning data.

Recently, more and more companies collect and process large amounts of data on a crowdsourcing basis that engages the general public in some process of corporate activities. In other words, by opening a project and allowing the general public, that is, workers to participate in the project, necessary information is collected through the work results completed by the workers.

At this time, in order to collect more reliable information, the company assigns the work result completed by the worker to the inspector to perform the inspection work.

Specifically, when one project is opened, a plurality of tasks are assigned to each of the plurality of workers. Each worker performs a plurality of tasks assigned to it, and provides the task results. Thereafter, a plurality of inspection tasks are assigned to each of the plurality of inspectors, and each inspector performs a plurality of inspection tasks assigned to them.

On the other hand, the evaluation of the work results submitted by the worker, that is, the inspection, is performed only by the inspector, and the judgment by the inspector is absolute and final. In particular, in the case of rejection of the work result, since the examiner's rejection decision is unilaterally transmitted to the worker, the worker must re-submit after reworking the rejected work.

In this case, even if the inspection judgment by the inspector is wrong, the operator must unconditionally perform the rework.

In this way, the degree to which the inspector affects the quality of work results is absolute, so the monitoring or verification method for inspectors is very important, but there is no way to verify or monitor inspectors within an appropriate cost structure.

Unexamined Patent Publication No. 10-2014-0095956, 2014.08.04.

The problem to be solved by the present invention is to provide an inspection quality control method utilizing feedback from workers of a crowdsourcing-based project that can improve the reliability of inspection results.

However, the problem to be solved by the present invention is not limited to the problem as described above, and other problems may exist.

In order to solve the above-described problems, the inspection quality management method using the feedback of the operator of a crowdsourcing-based project according to an aspect of the present invention includes the steps of generating a work guide and an inspection guide for a crowdsourcing-based project (hereinafter, the project), Providing the work guide to a worker participating in the project, when the worker checks the work guide, assigning the work of the project to the worker and requesting to perform the work, receiving a work result from the worker Step, Providing the inspection guide to the inspector participating in the project, When the inspector checks the inspection guide, assigning the work result to the inspector to request the inspection performance, The inspector to the work result Receiving the inspection result for the inspection pass or rejection, when receiving the inspection result as rejection from the inspector, assigning rework to the worker and requesting to perform rework, and (i) rework from the operator And (ii) receiving a request for an objection to the result of the inspection. At this time, the step of receiving an objection request for the inspection result from the operator includes the step of receiving an objection request from the operator, and the work guide and objection application tool to the operator who inputs the objection request. And receiving, from the operator, selection of an objection ground item on the work guide and a comment on the objection ground item by the objection application tool.

In some embodiments of the present invention, upon receiving the objection request from the worker, the objection request is assigned to the inspector, and comments on the objection ground item and the objection ground item on the work guide input from the worker It may further include the step of transmitting and requesting to perform a review on the appeal.

In some embodiments of the present invention, receiving the review result of the objection request from the inspector as a citation or rejection, and when receiving the review result as a citation from the inspector, the inspection result of the work result is passed. And canceling the assignment of the rework to the worker.

In some embodiments of the present invention, receiving the review result of the objection request from the inspector as citation or rejection, and when receiving the review result as rejection from the inspector, the inspection result of the work result is rejected. Maintaining and re-requesting the operator to perform the rework may be further included.

In some embodiments of the present invention, the step of receiving a review result of the objection application as rejection from the examiner includes receiving a rejection intention from the examiner, and the inspection guide to the examiner who inputs the rejection intention, and Providing the objection application tool, and receiving input from the inspector by the objection application tool to select a rejection basis item on the inspection guide and a comment on the rejection basis item.

In some embodiments of the present invention, the step of re-requesting the operator to perform the rework is, when receiving the review result as rejection from the inspector, maintaining the inspection result for the work result as a reject, and to the operator The rejection ground item and a comment on the rejection ground item on the inspection guide received from the inspector may be transmitted, and the rework may be requested again.

In some embodiments of the present invention, when the review result for the objection is received as rejection, receiving an adjustment request for the inspection result from the worker, and the error of the worker or the inspector according to the adjustment request It may further include determining as one of the faults of.

Some embodiments of the present invention may further include updating the work guide, and distributing at least a part of the updated work guide to all workers participating in the project if it is determined that the worker is at fault. .

Some embodiments of the present invention may further include updating the inspection guide, and distributing at least a part of the updated inspection guide to all inspectors participating in the project if it is determined that the inspector is at fault. .

Some embodiments of the present invention may further include the step of changing the inspection result for the work result to pass inspection, and canceling the assignment of the rework to the worker, if it is determined as the inspector's fault.

Some embodiments of the present invention may further include providing the inspector with the inspector training course for the project again, if it is determined that the inspector is at fault.

Some embodiments of the present invention may further include applying a result of the determination to the quality evaluation of the inspector, if it is determined as the inspector's fault.

Some embodiments of the present invention may further include the step of requesting a review on the work guide or the inspection guide if the objection to the inspection result is more than a reference number of times.

A computer program according to another aspect of the present invention for solving the above-described problems is combined with a computer that is hardware to execute a quality control method for inspection utilizing feedback from workers of the crowdsourcing-based project, and on a computer-readable recording medium. Is saved.

Other specific details of the present invention are included in the detailed description and drawings.

According to the present invention described above, even if there is a problem with the rejected inspection result, the operator is not forced to rework, but a request for reconsideration through an objection is made, so that the operator can monitor the inspector and the inspector There is an advantage in that it can further improve the quality of inspection.

In addition, if there is a mistake or mistake by the inspector, the quality of inspection of the inspector can be improved through retraining, and by providing a method to update the inspection guide even if there is a problem with the work guide or inspection guide, the wrong part It is possible to minimize the problem of incorrectly performing work and inspection due to ambiguity or.

In addition, there is an advantage of being able to automatically recognize whether or not a work guide or an inspection guide needs to be modified by counting the number of objections filed, and update it.

The effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the following description.

1 is a conceptual diagram of a crowdsourcing service according to an embodiment of the present invention.
2 is a flowchart illustrating a crowdsourcing-based project progress process according to an embodiment of the present invention.
3 is a flow chart of a quality control method for inspection using feedback from an operator of a crowdsourcing-based project according to an embodiment of the present invention.
4 is a diagram for explaining a process of requesting an inspector to review an objection request.
5 is a flow chart for explaining an objection review process.
6 is a diagram for explaining a process in which an examinee inputs a rejection as a result of reviewing an objection request.
7 is a flow chart for explaining the process of applying for an adjustment.
8 is a view for explaining the contents of updating the work guide and the inspection guide at the request of an operator or inspector.
9A to 9C are views for explaining an example of proceeding with an objection.
10 is a diagram for explaining a quality control apparatus for inspection using feedback from a worker in a crowdsourcing-based project according to an embodiment of the present invention.

Advantages and features of the present invention, and a method of achieving them will become apparent with reference to the embodiments described below in detail together with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but may be implemented in a variety of different forms, only the present embodiments are intended to complete the disclosure of the present invention, It is provided to fully inform the technician of the scope of the present invention, and the present invention is only defined by the scope of the claims.

The terms used in the present specification are for describing exemplary embodiments and are not intended to limit the present invention. In this specification, the singular form also includes the plural form unless specifically stated in the phrase. As used in the specification, “comprises” and/or “comprising” do not exclude the presence or addition of one or more other elements other than the mentioned elements. Throughout the specification, the same reference numerals refer to the same elements, and “and/or” includes each and all combinations of one or more of the mentioned elements. Although "first", "second", and the like are used to describe various elements, it goes without saying that these elements are not limited by these terms. These terms are only used to distinguish one component from another component. Therefore, it goes without saying that the first component mentioned below may be the second component within the technical idea of the present invention.

Unless otherwise defined, all terms (including technical and scientific terms) used in the present specification may be used as meanings that can be commonly understood by those of ordinary skill in the art to which the present invention belongs. In addition, terms defined in a commonly used dictionary are not interpreted ideally or excessively unless explicitly defined specifically.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a conceptual diagram of a crowdsourcing service according to an embodiment of the present invention.

Referring to FIG. 1, a crowdsourcing service is performed by being composed of a client 10, a service provider 20, and a public 30.

The sponsor 10 refers to a company or individual who requests a crowdsourcing-based project (hereinafter, referred to as a project).

The requester 10 requests a project for the purpose of collecting source data or annotating data for generating artificial intelligence learning data. The data generated through the project can be used as learning data for arbitrary machine learning such as supervised learning, unsupervised learning, and reinforcement learning. Collection of source data refers to collecting raw data such as recorded voice collection and photo collection. Data annotation refers to inputting relevant annotation data into source data such as text, photos, and videos. For example, the data annotation may include finding an entity in a given fingerprint or finding a similar sentence, but is not limited thereto. Meanwhile, the types of the above-described projects are only one embodiment, and various projects may be handled in the present invention according to the design of the client.

The service provider 20 refers to a company that provides crowdsourcing services.

When the service provider 20 receives a request for a project for a product or service from the client 10, the service provider 20 allocates a task for the project to the general public 30 and receives the work result from the public 30. Thereafter, the final product extracted based on the work result is provided to the client 10.

At this time, the service provider 20 provides a crowdsourcing service to the client 10 and the public 30 through a crowdsourcing platform (hereinafter, the platform). That is, when the service provider 20 receives a request for a project from the client 10, the service provider 20 opens the project on the platform. Thereafter, when the work result for the open project is received from the public 30, the project is terminated on the platform, and the final product may be extracted and provided to the client 10.

The public 30 refers to the general public participating in the project open on the platform. Here, the public 30 may participate in a project opened on the platform through an application or website provided by the service provider 20.

The public 30 is composed of a worker 32 and an inspector 34.

The worker 32 decides to participate in a specific project among a plurality of projects open on the platform. Thereafter, the operator 32 performs an operation such as collection of source data or data annotation, and transmits it to the platform.

The inspector 34 decides to participate in a specific project among a plurality of projects open on the platform. Thereafter, the inspector 34 inspects the result of the work performed by the operator 32. As a result of performing the inspection, the inspector 34 may perform inspection pass processing or rejection processing, and input a rejection reason during rejection processing. In the case of passing the inspection, it is not necessary to rework and re-examination due to this, so passing inspection has the same meaning as completion of inspection.

2 is a flowchart illustrating a crowdsourcing-based project progress process according to an embodiment of the present invention.

First, the requester 10 requests one or more projects to the service provider 20 (S11).

Thereafter, the service provider 20 opens the requested project on the platform (S12). At this time, the service provider 20 may determine the grade in consideration of the difficulty of the project before opening the project. That is, depending on the difficulty level, it is possible to determine which level or higher public 30 to expose the project. Accordingly, it is possible to increase the reliability of the work result of the project.

Thereafter, the service provider 20 requests the work by assigning the work to the workers 32 of the corresponding level or higher according to the level of the project (S13).

Thereafter, the worker 32 performs the assigned task (S14). In this case, the worker 32 may input the reason for the inability to work without performing the work for a work in which the work itself is impossible for some reason.

Thereafter, the service provider 20 receives the work result from the worker 32 (S15), and assigns the work result to the inspector 34 to request the inspection (S16).

Similarly, according to an embodiment of the present invention, only a suitable project may be exposed to the inspector 34 out of the total projects being performed according to the level set according to the difficulty of the project or the qualification requirements of the inspector 34.

Thereafter, the inspector 34 performs the assigned inspection (S17). At this time, the inspector 34 determines that the inspection is completed when it is determined that the work has been properly performed, and if it is determined that the inspection operation is wrong, the inspector 34 rejects it. During rejection processing, the inspector 34 inputs the rejection reason for what reason the job was judged to be wrong.

Thereafter, the service provider 20 receives the inspection result from the inspector 34 (S18).

When the inspection result is the inspection completion, the service provider 20 uses the work result as valid data, and based on this, extracts the final product at the end of the project.

When the inspection result is rejection processing, the service provider 20 may internally perform the inspection again, or assign the work to the worker 32 again to perform the rework. When reworking, re-examination by an inspector is required.

Thereafter, when the project period ends or sufficient valid data is secured, the service provider 20 terminates the project (S19), calculates a final result based on the secured valid data, and provides it to the client 10 ( S20).

At this time, before the end of the project, the service provider 20 evaluates the performance results of the worker 32 and the inspector 34, calculates the work cost and the inspection cost according to the evaluation, to the worker 32 and the inspector 34. give.

In FIGS. 1 and 2, it is simply expressed as a requester 10, a service provider 20, an operator 32, and an inspector 34, but these are smartphones, tablets, PDAs, laptops, desktops, operated by each participant. It means a computer device such as a server or a telecommunication device.

On the other hand, as described in FIG. 2, when the work result submitted by the worker 32 is rejected by the inspector 34, the service provider 20 allocates the work again to the worker 32 The work is performed, and the worker 32 must submit the rework result again to receive re-review by the inspector 34.

At this time, the worker 32 is rejected due to the wrong decision of the inspector 34, even though the worker 32 has properly performed the work according to the work guide provided by the service provider 20, or the inspector 34 provides If there is an error in the received inspection guide and the inspector 34 performs the inspection according to the inspection guide and is rejected, the result of work and rework will continue to be rejected despite the problem on the inspector 34 side, This in turn affects the quality of work and projects as well.

In order to solve this problem, an embodiment of the present invention provides a means for inputting feedback on the inspection result to the operator 32 for a work case that is rejected as a result of the inspection by the inspector 34. 34) It is possible to minimize the result of incorrect inspection.

In terms of cost, the cost of time and effort is incurred by performing the communication process, but it can be resolved through the mediation application function described below, which incurs costs due to excessive objection. It has the advantage of being able to derive quality work results through the process.

Hereinafter, with reference to FIGS. 3 to 9C, a method of quality control inspection using feedback from a worker of a crowdsourcing-based project according to an embodiment of the present invention will be described.

3 is a flowchart of an inspection quality control method using feedback from workers of a crowdsourcing-based project.

Meanwhile, the steps illustrated in FIGS. 3, 5 and 7 may be understood as being performed by a platform server (hereinafter, referred to as a server) operated by the service provider 20, but are not limited thereto.

In addition, a plurality of workers 32 or a plurality of inspectors 34 perform work using a predetermined terminal device. The terminal device of the operator 32 or the inspector 34 may be a computer device or a telecommunication device such as a smartphone, tablet, PDA, laptop, desktop, etc., but is not limited thereto.

First, referring to FIG. 3, the server generates a work guide and an inspection guide for a crowdsourcing-based project (hereinafter, referred to as a project) (S110).

At this time, in the work guide, items to be observed when the operator 32 performs the work are indicated, and the inspection guide is also indicated by the items to be observed when the inspector 34 performs the inspection. Items to be observed in performing work or inspection can be entered by the project manager. The server creates the corresponding items as work guides and inspection guides, respectively.

Meanwhile, for convenience, the work guide and the inspection guide may be generated to include the same item. Alternatively, the work guide may be included as part of the inspection guide. That is, the inspection guide may further include the contents of the work guide and additional items related to inspection. Alternatively, the work guide and the inspection guide may be generated to include at least one different item.

Next, a task guide is provided to the worker 32 participating in the project (S120), and when the worker 32 checks the task guide, the task of the project is assigned to the worker 32 and requests to perform the task (S130 ). And as the worker 32 completes the task, the task result is inputted from the worker 32 (S140).

For example, the work guide may be provided in an area of an online work space for performing work. In addition, a confirmation completion button may be provided at the bottom of the area where the work guide is provided, and whether or not the operator 32 checks the work guide may be checked through whether the confirmation completion button is clicked. Alternatively, it may be provided in the form of moving to a page for performing a task only when the confirmation complete button is clicked.

Here, the online work space means a space in which online work such as a web page or a page on an application or a specific URL provided by a server can be performed.

Next, the inspection guide is provided to the inspector 34 participating in the project (S150), and when the inspector 34 checks the inspection guide, the work result is assigned to the inspector 34 to request the inspection performance (S160). , The inspection result for the work result from the inspector 34 is input as the inspection pass or reject (S170).

Like the work guide, the inspection guide may be provided in an area of the online inspection space for performing inspection. A confirmation completion button may be provided at the bottom of the area where the inspection guide is provided, and whether or not the inspector 34 checks the inspection guide may be checked by clicking the confirmation completion button.

Here, the online inspection space means a space in which online inspection can be performed, such as a web page or a page on an application, or a specific URL provided by a server.

Next, when the inspection result is received as a rejection from the inspector 34, the rework is assigned to the worker 32 and the rework is requested (S180).

In the past, when the inspection result was rejected, the operator 32 had to input the rework result after performing the rework, but an embodiment of the present invention introduces a two-way communication function between the operator 32 and the inspector 34 This allows the worker 32 to raise an objection to the rejection process. That is, when the worker 32 accepts the rejection process, which is the result of the inspection, the rework result is inputted from the worker 32, and when the worker 32 refutes the inspection result, the objection to the inspection result is inputted from the worker 32. It becomes (S190).

4 is a diagram for explaining a process of requesting an inspector to review an objection request.

In an embodiment of the present invention, in order to receive an objection request for the inspection result from the operator 32, the objection intention is first input from the operator 32 (S191). Upon receiving the inspection result, the operator 32 may select the rework button to perform rework if the inspection result is accepted, and if the inspection result is not accepted, the operator 32 inputs an objection intention through the objection button selection. As will be converted, you can enter specific details for your appeal.

A work guide and an objection application tool are provided to the worker 32 who has entered the objection application (S192). The work guide was provided to the worker 32 before the work was assigned, and it is provided so that the worker 32 can check again, and when reviewing the objection by marking the objection grounds item on the work guide and entering a comment It is provided so that the inspector 34 can check the related contents.

The worker 32 selects an item on which the objection is based on the work guide through the objection application tool, and inputs a comment on the selected objection basis item (S193). That is, as shown in the example of FIG. 4,'[Basis 1]' may be selected, and a comment stating that'the task was performed in accordance with Basis 1'may be left.

The objection application tool is a tool that can mark and leave a comment on a specific part of the work guide, such as a speech bubble, a memo function, and a highlight function, and the operator 32 selects the items on which the objection is based among the contents of the work guide. Select and check, and write a comment in the corresponding part so that the inspector 34 can check it.

5 is a flow chart for explaining an objection review process. 6 is a diagram for explaining a process in which an examinee inputs a rejection as a result of reviewing an objection request.

In this way, upon receiving an objection request from the worker 32, the objection request is assigned to the inspector 34 (S210). In addition, the objection basis item on the work guide input from the worker 32 and the comment on the objection are transmitted to the inspector 34 to request review of the objection application (S220).

Upon receiving the objection request, the inspector 34 selects and inputs citation or rejection as a result of reviewing the objection application (S230).

At this time, when the inspector 34 accepts the objection and inputs the review result as a quotation, the inspection result for the work result is changed to the inspection pass, and the assignment of rework to the worker 32 is canceled (S240). . Here, the server can automatically change the inspection result for the work result to the inspection pass and cancel the rework assignment, and it is also possible to manually change the inspection pass and cancel the rework assignment by the inspector 34.

The result of the objection is communicated to the worker 32, and the worker 32 can perform the next task upon receiving the objection result, unless the objected task is the last task of the worker 32. do.

When the objection review result is entered as a citation, the server may activate the input window to request the examiner 34 to input the reason for the cancellation of the rejection. If there is an inspection mistake by one inspector, it is counted, and if such counting exceeds a predetermined number of times, as described later in k, retraining of the inspector 34 may be requested or other penalties may be imposed. If there is an error in the inspection guide that is not the inspection mistake of the inspector 34, the server receives the content of the error from the inspector 34 through the objection application tool and delivers it to the administrator. After reviewing, the inspection guide can be updated, and the updated inspection guide can be distributed to all inspectors 34. In this case, the entire contents of the updated inspection guide may be distributed or only the updated part may be distributed.

If the inspector 34 does not accept the objection and inputs the review result as rejection, the inspection result for the work result is maintained as a rejection, and re-requests the operator 32 to perform the rework (S250). Likewise, the server can automatically keep the inspection result for the work result as a reject and request rework to perform rework, or may not separately request rework execution. Of course, this is also possible manually by the inspector 34.

The result of the objection request is transmitted to the worker 32, and the worker 32 performs rework on the corresponding rejected work case as it is received.

At this time, in an embodiment of the present invention, the review result of the objection application may be input by the examiner 34 simply as citation or rejection, but in order for the examiner 34 to more reliably review the objection application, the examiner When the rejection intention is inputted from (34) (S231), an examination guide and an objection application tool are provided to the examiner 34 who inputs the rejection intention (S232).

The inspector 34 selects an item that is the basis for rejecting the objection on the inspection guide through the objection application tool, and inputs a comment on the selected rejection ground item (S233). That is, as shown in the example of FIG. 6,'[Evidence 1]' may be selected as the rejection basis item, and a comment indicating'rejected because it violates the basis 1'may be left.

These rejection ground items and comments are transmitted to the worker 32. That is, when the server receives the review result of the objection application as rejection from the inspector 34, the server maintains the inspection result of the work result of the worker 32 as rejection (S251). Then, the rejection basis item on the inspection guide input from the inspector 34 and a comment on the rejection basis are transmitted to the worker 32 (S252), and re-request to perform rework is performed (S253).

7 is a flow chart for explaining the process of applying for an adjustment.

On the other hand, the worker 32, who has received the result of the objection application, may accept the objection result, perform rework, and resubmit it. However, the worker 32 continues to be performed so that the result of his or her work is consistent with the work guide, and it may be determined that there is a problem with the result of the objection.

In this case, if the worker 32 makes an objection again, the inspector 34 will reject the objection again, which eventually slows the project progress, so coordination between the operator 32 and the inspector 34 is required. Do.

To this end, in an embodiment of the present invention, in a state in which the review result of the objection request is received as rejection from the inspector 34, when an adjustment request for the inspection result is input according to the selection of the worker 32 (S310), The server may determine one of the faults of the operator 32 and the inspector 34 according to the adjustment request (S320). As the request for adjustment is received, the manager of the project is provided with the objection basis items and comments indicated on the work guide of the worker 32, and the rejection basis items and comments indicated on the inspection guide of the inspector 34. In this case, it is determined whose fault is by reviewing it, and subsequent processing is performed by the server.

Meanwhile, in the embodiment of the present invention, it has been described that the application for adjustment is filed by the worker 32, but is not limited thereto. That is, if the worker 32 continues to file an objection even after reviewing the objection as rejection, the inspector 34 may not be able to proceed with the inspection for other work due to the objection of the worker 32 Therefore, according to the embodiment, the application for adjustment may also be filed by the inspector 34.

If the judgment result of the above adjustment application is determined as the error of the worker 32 (A), the work guide is improved and supplemented and updated (S331) to prevent the error or error of the worker 32 from recurring (S331), and the updated work At least a portion of the guide is distributed to all workers 32 participating in the project (S332). Since the result of the inspection on the result of the work is kept as a rejection, the worker 32 must perform rework on the rejected work again and submit it. At this time, you can refer to the updated work guide.

If the judgment on the above application for adjustment is determined as the error of the inspector 34 (B), the inspection guide is improved and supplemented to prevent recurrence of errors or errors of the inspector 34 (S341), and the updated inspection At least part of the guide is distributed to all inspectors 34 participating in the project (S342). The server changes the inspection result for the work result to pass inspection (S343), and cancels the assignment of the rework to the worker 32 (S344).

If, as a result of the judgment on the mediation application, the error or mistake of the examiner 34 is correct, but it is determined that it is the problem of one examiner, not the problem of the inspection guide, the inspection guide is not updated or redistributed.

On the other hand, in an embodiment of the present invention, when it is determined that the inspection by the inspector 34 is wrong as a result of the judgment on the application for adjustment, the updated inspection guide is distributed to the inspector 34, furthermore, the inspector training process of the project is provided to the inspector 34. Can provide again.

In principle, the inspectors 34 can be selected as inspectors 34 only after completing the inspector training course. As an example, the inspector training course may consist of a tutorial for inspecting work results in which the correct answer of the inspection result is already determined. This inspector training process is performed to prevent mistakes or mistakes of the inspector 34 in performing the inspection.

In the case of the training course of the inspector 34, if an application for adjustment is received due to the error or mistake of one inspector, and as a result, it is determined that the inspector 34 is wrong, the inspector 34 can be regarded as having misunderstood the inspector guide. Therefore, it is desirable to be provided to the corresponding inspector 34.

However, when the number of updates of the inspector guide exceeds a preset number or more, the inspector training course may be provided again for all inspectors 34.

According to an exemplary embodiment of the present invention, the number of times it is determined as a mistake of one inspector according to an adjustment request may be counted, and the determination result may be applied to the quality evaluation of the inspector 34. According to the quality evaluation of the inspector 34, a retraining of the inspector 34 may be requested, and other penalties may be given according to the counting frequency and quality evaluation.

On the other hand, in an embodiment of the present invention, the update of the work guide or the inspection guide may be performed according to the determination result by the request for adjustment of the operator 32 or the inspector 34, but may also be performed by request from the server.

In other words, when an objection from any one worker 32 for a specific job is more than the standard number of times, or when an objection from all workers 32 participating in the project for a specific job is more than the reference number, the server You may decide that the work guide or inspection guide related to the work is unclear or problematic, and you may request a review from the project manager.

In addition, it is also possible to update the contents of the work guide or the inspection guide as required by receiving a request from the operator 32 or the inspector 34.

8 is a view for explaining the contents of updating the work guide and the inspection guide at the request of an operator or inspector.

For example, as shown in (a) of FIG. 8, when it is determined by worker A that some of the basis items of the work guide need improvement, worker A is an objection application tool for updating the work guide even if it is not an objection application procedure. Can be provided, and through this, you can select the item in'[Evidence 1]' and write a comment saying'It needs to be modified to ooo' and send it to the manager.

Likewise, as shown in Fig.8(b), if it is determined by the inspector A that any of the basis items of the inspection guide need improvement, the inspector A will be provided with an objection application tool for updating the inspection guide even if it is not an objection application procedure. Through this, it is possible to select the item of'[Evidence 1]' and write a comment stating that'Evidence 1 is ambiguous and needs improvement' to be delivered to the manager.

Meanwhile, an update of the inspection guide may be received by an operator or a request for an update of the work guide may be received by the inspector.

Upon receiving such a request, the manager can more clearly identify the areas that need improvement, and if it is determined that the update is necessary after review, the content is reflected and updated, and then redistributed to all workers (32) or inspectors (34). have.

Hereinafter, an example in which an operator proceeds with an objection will be described with reference to FIGS. 9A to 9C.

9A to 9D are tasks of labeling the face data of the worker 32, and the worker 32 is presented with precautions or guidance for the task progress as a work guide.

9A shows a work guide for a task of dragging and bounding face data, and how to drag and bound the face of the operator 32, and cautions at this time, and how to use a mouse are provided.

9B is a step of selecting an expression when a face is photographed from among the work contents, and the worker 32 must select and input whether his or her expression corresponds to either an expressionless expression or a smile. At this time, even if the worker 32 is smiling finely, the'laugh' item must be selected.

9C is an example in which the worker 32 submits a job but is rejected by the inspector 34, and the worker 32 makes an objection. The worker 32 smiles finely according to the work guide. This is the case that the case was checked as'laugh' and submitted, but the inspector (34) judged it as'no expression' and rejected it.

In this case, the worker 32 selects the item'If you have a smile, please check with laughter' as the basis for objection in the part of the work guide that you want to object to, so that it is displayed as a highlight, and a speech bubble Through the memo, it is possible to describe'the choice was made with laughter based on the underlying item' as a comment.

Upon receiving this, the inspector 34 checks whether there is an error in the work guide or the inspection guide, whether there are different parts, or if there is no error, then cites or rejects the objection by checking whether his or her inspection method is wrong.

Meanwhile, in the above description, steps S11 to S344 may be further divided into additional steps or may be combined into fewer steps according to an embodiment of the present invention. In addition, some steps may be omitted as necessary, and the order between steps may be changed. In addition, even if other contents are omitted, the contents of FIG. 10 to be described later can also be applied to the inspection quality management method using feedback from workers of the crowdsourcing-based project of FIGS. 1 to 9C.

Hereinafter, an apparatus for quality control inspection using feedback from an operator of a crowdsourcing-based project according to an embodiment of the present invention will be described.

10 is a diagram for explaining a quality control apparatus for inspection using feedback from a worker in a crowdsourcing-based project according to an embodiment of the present invention.

Referring to FIG. 10, the inspection quality management apparatus 400 using the feedback of the operator of the crowdsourcing-based project according to the above includes a communication module 410, a memory 420, and a processor 430.

The communication module 410 transmits a crowdsourcing-based work for one project to a plurality of workers 32 to request a task to be performed, and receives work results from the plurality of workers 32. Further, the work results received from the plurality of workers 32 are transmitted to the plurality of inspectors 34 to request inspection, and the inspection results are received from the plurality of inspectors 34. At this time, the communication module 410 provides a work guide to the worker 32 participating in the project, and provides the inspection guide to the inspector 34 participating in the project.

The memory 420 stores a program for managing the quality of inspection using the feedback of the operator of the crowdsourcing-based project based on the data received from the communication module 410.

The processor 430 executes a program stored in the memory 420. As the processor 430 executes the program stored in the memory 420, a work guide and an inspection guide of the project are generated and provided to the operator 32 and the inspector 34, respectively, and the operator 32 and the inspector 34 When) checks the work guide and the inspection guide, respectively, the worker 32 requests the task to be performed after assignment, and the inspector 34 assigns the work result to request the inspection.

Thereafter, when the processor 430 receives the inspection result as a rejection from the inspector 34, it assigns the rework to the worker 32 and requests the rework to be performed, and the rework result is returned according to the operator 32's selection. You will receive an input or an objection to the inspection result.

At this time, when the processor 430 first receives an objection request from the operator 32, the operator 32 provides a work guide and an objection application tool, and the operator 32 provides the received operation guide and objection application tool. Through this, you can proceed with the objection by selecting the item of grounds for objection in the work guide and entering a comment on it.

The inspection quality management apparatus 100 using the feedback of the operator of the crowdsourcing-based project described with reference to FIG. 10 may be provided as a component of the above-described server.

The inspection quality control method using feedback from a worker of a crowdsourcing-based project according to an embodiment of the present invention described above may be implemented as a program (or application) to be executed in combination with a computer as hardware and stored in a medium. have.

The above-described program includes C, C++, JAVA, Ruby, which can be read by a processor (CPU) of the computer through the device interface of the computer, in order for the computer to read the program and execute the methods implemented as a program. It may include a code (Code) coded in a computer language such as machine language. Such code may include a functional code related to a function that defines necessary functions for executing the methods, and includes a control code related to an execution procedure necessary for the processor of the computer to execute the functions according to a predetermined procedure. can do. In addition, these codes may further include additional information required for the processor of the computer to execute the functions, or code related to a memory reference to which location (address address) of the internal or external memory of the computer should be referenced. have. In addition, when the processor of the computer needs to communicate with any other computer or server in the remote in order to execute the functions, the code uses the communication module of the computer to determine how It may further include a communication-related code for whether to communicate, what kind of information or media should be transmitted and received during communication.

The stored medium is not a medium that stores data for a short moment, such as a register, cache, memory, etc., but a medium that stores data semi-permanently and can be read by a device. Specifically, examples of the storage medium include, but are not limited to, ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, and the like. That is, the program may be stored in various recording media on various servers to which the computer can access, or on various recording media on the user's computer. In addition, the medium may be distributed over a computer system connected through a network, and computer-readable codes may be stored in a distributed manner.

The above description of the present invention is for illustrative purposes only, and those of ordinary skill in the art to which the present invention pertains can understand that it is possible to easily transform it into other specific forms without changing the technical spirit or essential features of the present invention. will be. Therefore, it should be understood that the embodiments described above are illustrative in all respects and not limiting. For example, each component described as a single type may be implemented in a distributed manner, and similarly, components described as being distributed may also be implemented in a combined form.

The scope of the present invention is indicated by the claims to be described later rather than the detailed description, and all changes or modified forms derived from the meaning and scope of the claims and their equivalent concepts should be interpreted as being included in the scope of the present invention. do.

10: Client
20: service provider
30: public
32: worker
34: inspector
100: inspection quality control device
110: communication module
120: memory
130: processor

Claims (14)

As a method performed by a computer,
Generating a work guide and an inspection guide for a crowdsourcing-based project (hereinafter, referred to as a project);
Providing the work guide to workers participating in the project;
When the worker checks the work guide, assigning a task of the project to the worker and requesting the task to be performed;
Receiving a work result from the worker;
Providing the inspection guide to inspectors participating in the project;
When the inspector checks the inspection guide, assigning the work result to the inspector and requesting to perform inspection;
Receiving an inspection result for the work result from the inspector as passing or rejecting the inspection;
Requesting the rework by assigning rework to the worker when receiving the test result as a rejection from the inspector;
Receiving an objection from the worker (i) a rework result when accepting the rejection or (ii) an objection to the inspection result when the rejection is not accepted;
Upon receiving the objection request from the worker, the objection request is assigned to the inspector, and a comment on the objection basis item and the objection basis item on the work guide received from the worker is delivered, and Requesting to perform a review;
Receiving a review result of the objection application as a citation or rejection from the inspector;
Upon receiving the review result as a quotation from the inspector, changing the inspection result for the work result to pass inspection, and canceling the assignment of the rework to the worker; And
When the review result is received as rejection from the inspector, the inspection result for the work result is maintained as rejection, and the rejection ground item on the inspection guide received from the inspector and a comment on the rejection ground item are transmitted, and the Including the step of re-requesting the operator to perform the rework,
The step of receiving an objection request for the inspection result from the worker,
Receiving an objection intention input from the worker, and
Providing the work guide and the objection application tool to the worker who has entered the objection application intention, and
Including the step of receiving input from the worker by the objection application tool for selection of the objection grounds item on the work guide and a comment on the objection grounds item,
The step of receiving the review result of the objection application as rejection from the inspector,
Receiving a rejection intention from the examiner; and
Providing the inspection guide and the objection application tool to the inspector who has input the rejection intention;
Including the step of receiving input from the inspector by the objection application tool for selection of the rejection ground item on the inspection guide and a comment on the rejection ground item,
The objection application tool is a tool for selecting a specific item on the work guide or the inspection guide and inputting a comment on the specific item,
Inspection quality management method using feedback from workers in crowdsourcing-based projects. delete delete delete delete delete The method of claim 1,
Receiving a request for adjustment of the inspection result from the operator when the review result of the objection request is received as rejection; And
Further comprising the step of determining one of the error of the operator or the error of the inspector according to the adjustment request,
Inspection quality management method using feedback from workers in crowdsourcing-based projects. The method of claim 7,
Updating the work guide if it is determined that the worker is at fault; And
Distributing at least a portion of the updated work guide to all workers participating in the project, further comprising,
Inspection quality management method using feedback from workers in crowdsourcing-based projects. The method of claim 7,
Updating the inspection guide if it is determined that the inspector is at fault; And
Further comprising distributing at least a part of the updated inspection guide to all inspectors participating in the project,
Inspection quality management method using feedback from workers in crowdsourcing-based projects. The method of claim 7,
If it is determined as the error of the inspector, further comprising the step of changing the inspection result for the work result to the inspection pass, and canceling the assignment of the rework to the worker,
Inspection quality management method using feedback from workers in crowdsourcing-based projects. The method of claim 7,
If it is determined that the inspector is wrong, further comprising the step of providing the inspector training course for the inspector of the project again to the inspector,
Inspection quality management method using feedback from workers in crowdsourcing-based projects. The method of claim 7,
If it is determined as the inspector's fault, further comprising the step of applying the determination result to the inspection quality evaluation of the inspector,
Inspection quality management method using feedback from workers in crowdsourcing-based projects. The method of claim 1,
If the objection to the inspection result is more than the reference number of times, further comprising the step of requesting a review of the work guide or the inspection guide to the project manager,
Inspection quality management method using feedback from workers in crowdsourcing-based projects. A computer program stored in a computer-readable recording medium to perform the inspection quality control method using the feedback of the operator of the crowdsourcing-based project according to any one of claims 1, 7 to 13 in combination with a computer.

Images