KR102155860B1 - Method and apparatus for monitoring inspector based on cause of return - Google Patents

Abstract

Provided are a method and an apparatus for monitoring an inspector based on a return reason. The method for monitoring an inspector based on a return reason includes the following steps of: granting a crowdsourcing-based job for one single project to a plurality of workers, and receiving job results from a plurality of workers; enabling a plurality of inspectors to inspect the job results inputted from the plurality of workers, while receiving a reason for a return from the plurality of inspectors with respect to a job which cannot pass the inspection; generally collecting the reason for the return inputted from the plurality of inspectors; extracting a plurality of return reason keywords from the collected reasons for return; selecting a main return reason keyword from among the plurality of extracted return reason keywords; and selecting an inspector with a failure depending on whether the main return reason keyword is included in a return reason of each of the inspectors. Therefore, the present invention is capable of increasing the accuracy of data collected for a project.

Description

Method and apparatus for monitoring inspectors based on the reason for the companion {METHOD AND APPARATUS FOR MONITORING INSPECTOR BASED ON CAUSE OF RETURN}

The present invention relates to a method and an apparatus for monitoring an examinee based on a reason for revocation.

Recently, more and more companies collect and process large amounts of data on a crowdsourcing basis that engages the general public in some process of corporate activities. In other words, by opening a project and allowing the general public, that is, workers to participate in the project, necessary information is collected through the work results completed by the workers.

At this time, in order to collect more reliable information, the company assigns the work result completed by the worker to the inspector to perform the inspection work.

Specifically, when the inspector determines that the work result cannot be passed during the inspection, the reason for rejection is input. The reason for rejection entered by the inspector can serve as a guide that can further increase the accuracy of the work performed by the operator in the future.

However, if the inspector does not properly perform the inspection work and enters a random reason for rejection that is not related to the work result, it may interfere with the subsequent operator's performance of the work, increasing only useless data, and inspection. There is a problem that the cost can be unreasonably increased.

Unexamined Patent Publication No. 10-2014-0095956, 2014.08.04.

The problem to be solved by the present invention is to provide a method and apparatus for monitoring a reviewer based on a reason for rejection by monitoring an examiner based on the reason for rejection input by the examiner and determining whether the examiner is a defective examiner or not.

The problems to be solved by the present invention are not limited to the problems mentioned above, and other problems that are not mentioned will be clearly understood by those skilled in the art from the following description.

In the method for monitoring a reviewer based on a reason for rejection according to an aspect of the present invention for solving the above-described problem, the step of assigning a crowd-sourcing-based work for one project to a plurality of workers and receiving work results from the plurality of workers. , Allowing a plurality of inspectors to inspect the work results input from the plurality of workers, but receiving a rejection reason for the rejected job due to failing to pass the inspection from the plurality of inspectors, the input from the plurality of inspectors Collecting the rejection reason as a whole, extracting a plurality of rejection reason keywords from the collected rejection reason keywords, selecting a major rejection reason keyword among the extracted rejection reason keywords, and the main reason for rejection of each examinee. And selecting a defective inspector according to whether or not the keyword for reason of rejection is included.

In some embodiments of the present invention, the step of extracting the plurality of reasons for reaction keywords may include extracting a word, which is a noun, from reasons for reaction consisting of sentences.

In some embodiments of the present invention, the selecting of the main reason for rejection keywords includes calculating an appearance frequency for each word, generating a list of reasons for rejection based on the frequency of occurrence, and predetermined from the list of reason for rejection keywords. It may include the step of selecting a word in the ranking as the main reason for rejection keyword.

In some embodiments of the present invention, the list of reasons for rejection may be updated in real time whenever a reason for rejection is input by an examiner.

In some embodiments of the present invention, the predetermined ranking may be determined according to reliability.

In some embodiments of the present invention, in the step of selecting the defective examiner, the number of rejection reasons including the main rejection reason keyword among the total rejection reasons of each examiner and the number of rejection reasons without the main rejection reason keyword is calculated. It can be.

In some embodiments of the present invention, the step of selecting the defective examiner includes calculating a ratio of the number of rejection reasons without the main rejection reason keyword calculated relative to the total rejection reason of each examinee, and the calculated ratio in advance. If the ratio is greater than or equal to the set ratio, the step of selecting the corresponding inspector as a defective inspector may be included.

In some embodiments of the present invention, the preset ratio may be set according to reliability.

In some embodiments of the present invention, a penalty including having at least a part of the work results inspected by the inspector selected as the defective inspector secondary inspection to another inspector, or warning the inspector selected as the defective inspector or suspending the inspection qualification Can be given.

In some embodiments of the present invention, the main rejection reason keyword is selected for each worker, and the step of selecting the defective inspector is defective according to whether the main rejection reason keyword for each worker is included in the rejection reason of each inspector for each worker. It may be to screen inspectors.

A computer program according to another aspect of the present invention for solving the above-described problem is combined with a computer that is hardware to execute the method for monitoring the examiner based on the reason for rejection, and is stored in a medium.

In order to solve the above-described problem, the reviewer monitoring device based on rejection reason according to another aspect of the present invention transmits a crowdsourcing-based work for one project to a plurality of workers, and receives work results from the plurality of workers. And, a communication unit that transmits the result of the work input from the plurality of workers to the plurality of inspectors so that the plurality of inspectors can inspect it, and receives the rejection reason for the rejected job due to failure to pass the inspection from the plurality of inspectors. , A storage unit that collectively collects and stores the rejection reason input from the plurality of examiners, extracts a plurality of rejection reason keywords from the collected rejection reason, and selects a major rejection reason keyword among the extracted rejection reason keywords And a determination unit that selects defective inspectors according to whether or not the main rejection reason keyword is included in the rejection reason of each examinee.

Other specific details of the present invention are included in the detailed description and drawings.

According to the reviewer monitoring method and apparatus based on the reason for rejection of the present invention as described above, the keywords for the reason for rejection are extracted from the entire reason for rejection collected for the same project, and the keywords for the reason for rejection input for each examiner are included. It is determined whether or not it is, and based on this, it is possible to select whether the corresponding inspector is a defective inspector.

In addition, the accuracy of the data collected for the project can be improved by having the inspector selected as a defective inspector perform a second inspection of the inspection results to another inspector, or giving a penalty such as warning or suspension of the inspection qualifications to the inspectors selected as defective inspectors. You can increase it.

The effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the following description.

1 is a conceptual diagram of a crowd sourcing service according to an embodiment of the present invention.
2 is a flowchart illustrating a crowdsourcing-based project progress process according to an embodiment of the present invention.
3 is a flowchart of a method for monitoring a reviewer based on a rejection reason according to an embodiment of the present invention.
4 is an exemplary view for explaining a reason data pool for rejection generated for the same project according to an embodiment of the present invention.
5 is an exemplary diagram for explaining a reason for rejection keyword and a main reason for rejection according to an embodiment of the present invention.
6 is an exemplary view for explaining the status of an inspector's inspection work according to an embodiment of the present invention.
FIG. 7 is an exemplary diagram for explaining whether or not a keyword for a main reason for rejection in a reason for rejection is included in a rejection task list of an examinee according to an embodiment of the present invention.
8 is an exemplary view for explaining a monitoring result for an inspector according to an embodiment of the present invention.
9 is a block diagram of an examiner monitoring apparatus based on a reason for rejection according to an embodiment of the present invention.

Advantages and features of the present invention, and a method of achieving them will become apparent with reference to the embodiments described below in detail together with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but may be implemented in a variety of different forms, only the present embodiments are intended to complete the disclosure of the present invention, It is provided to fully inform the technician of the scope of the present invention, and the present invention is only defined by the scope of the claims.

The terms used in the present specification are for describing exemplary embodiments and are not intended to limit the present invention. In this specification, the singular form also includes the plural form unless specifically stated in the phrase. As used in the specification, “comprises” and/or “comprising” do not exclude the presence or addition of one or more other elements other than the mentioned elements. Throughout the specification, the same reference numerals refer to the same elements, and “and/or” includes each and all combinations of one or more of the mentioned elements. Although "first", "second", and the like are used to describe various elements, it goes without saying that these elements are not limited by these terms. These terms are only used to distinguish one component from another component. Therefore, it goes without saying that the first component mentioned below may be the second component within the technical idea of the present invention.

Unless otherwise defined, all terms (including technical and scientific terms) used in the present specification may be used as meanings that can be commonly understood by those of ordinary skill in the art to which the present invention belongs. In addition, terms defined in a commonly used dictionary are not interpreted ideally or excessively unless explicitly defined specifically.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a conceptual diagram of a crowd sourcing service according to an embodiment of the present invention.

Referring to FIG. 1, a crowdsourcing service is performed by being composed of a client 10, a service provider 20, and a public 30.

The sponsor 10 refers to a company or individual who requests a crowdsourcing-based project (hereinafter, referred to as a project).

The requester 10 requests a project for the purpose of collecting source data or annotating data. Collection of source data refers to collecting raw data such as recorded voice collection and photo collection. Data annotation refers to inputting relevant annotation data into source data such as text, photos, and videos. For example, the data annotation may include finding an entity in a given fingerprint or finding a similar sentence, but is not limited thereto.

The service provider 20 refers to a company that provides crowdsourcing services.

When the service provider 20 receives a request for a project for a product or service from the client 10, the service provider 20 allocates a task for the project to the general public 30 and receives the work result from the public 30. Thereafter, the final product extracted based on the work result is provided to the client 10.

At this time, the service provider 20 provides a crowd sourcing service to the client 10 and the public 30 through a crowd sourcing platform (hereinafter, the platform). That is, when the service provider 20 receives a request for a project from the client 10, the service provider 20 opens the project on the platform. Thereafter, when the work result for the open project is received from the public 30, the project is terminated on the platform, and the final product may be extracted and provided to the client 10.

The public 30 refers to the general public participating in the project open on the platform. Here, the public 30 may participate in a project opened on the platform through an application or website provided by the service provider 20.

The public 30 is composed of a worker 32 and an inspector 34.

The worker 32 decides to participate in a specific project among a plurality of projects open on the platform. Thereafter, the operator 32 performs an operation such as collection of source data or data annotation, and transmits it to the platform.

The inspector 34 decides to participate in a specific project among a plurality of projects open on the platform. Thereafter, the inspector 34 inspects the result of the work performed by the operator 32. As a result of performing the inspection, the inspector 34 may pass or reject the job, and may input a reason for rejection when returning the job.

2 is a flowchart illustrating a crowdsourcing-based project progress process according to an embodiment of the present invention.

First, the client 10 requests a project to the service provider 20 (S41).

Thereafter, the service provider 20 opens the requested project on the platform (S42). At this time, the service provider 20 determines the grade in consideration of the difficulty level of the project before opening the project. In other words, depending on the level of difficulty, it is decided to expose the project to the public 30 or higher. Accordingly, it is possible to increase the reliability of the work result of the project.

Thereafter, the service provider 20 allocates a task to the worker 32 of the corresponding grade or higher according to the grade of the project (S43).

Thereafter, the worker 32 performs the assigned task (S44). In this case, the worker 32 may input the reason for the inability to work without performing the work for a work in which the work itself is impossible for some reason.

Thereafter, the service provider 20 receives the work result from the worker 32 (S45), and assigns the inspection work for the work result to the inspector 34 (S46).

Thereafter, the inspector 34 performs the assigned inspection (S47). At this time, the inspector 34 decides to pass the inspection when it is determined that the work has been properly performed, and determines rejection when it is determined that the inspection work is wrong. When deciding to reject, the inspector 34 inputs the rejection reason for what reason the task was judged to be wrong.

Thereafter, the service provider 20 receives the inspection result from the inspector 34 (S48).

If the inspection result is Pass, the service provider 20 uses the work result as valid data, and based on this, extracts the final product at the end of the project.

If the inspection result is rejected, the service provider 20 may internally perform the inspection again or assign the job again to the worker 32 to perform the job.

Thereafter, when the project period ends or sufficient valid data is secured, the service provider 20 terminates the project (S49), calculates a final result based on the secured valid data, and provides it to the client 10 ( S50).

At this time, before the end of the project, the service provider 20 evaluates the performance results of the worker 32 and the inspector 34, calculates the work cost and the inspection cost according to the evaluation, to the worker 32 and the inspector 34. give.

In Figs. 1 and 2, it is simply expressed as a client, service provider, worker, and inspector, but these refer to computer devices or telecommunication devices such as smartphones, tablets, PDAs, laptops, desktops, servers, etc. operated by each participant. .

3 is a flowchart of a method for monitoring a reviewer based on a reason for rejection according to an embodiment of the present invention, and FIG. 4 is an exemplary view for explaining a data pool for reason data generated for the same project according to an embodiment of the present invention. 5 is an exemplary diagram for explaining a reason for rejection keyword and a main reason for rejection keyword according to an embodiment of the present invention, and FIG. 6 is an exemplary view for explaining the status of the reviewer's inspection work according to an embodiment of the present invention 7 is an exemplary view for explaining whether or not the main reason for rejection keyword in the reason for rejection is included in the searcher's companion work list according to an embodiment of the present invention, and FIG. It is an exemplary diagram for explaining the monitoring result for the.

Steps S110 to S160 shown in FIG. 3 may be understood as being performed by a platform server (hereinafter, referred to as a server) operated by the service provider 20, but are not limited thereto.

Referring to FIG. 3, in step S110, the server receives work results from a plurality of workers 32.

The server is provided with the results of a plurality of tasks performed by a plurality of workers 32 to which the project is assigned.

Here, the plurality of workers 32 perform work using a predetermined terminal device. The terminal device of the worker 32 may be a computer device or a telecommunication device such as a smart phone, tablet, PDA, laptop, desktop, etc., but is not limited thereto.

Then, in step S120, the server causes the plurality of inspectors 34 to inspect the work result. At this time, the server prompts the examiner 34 to input a rejection reason for a job rejected because it did not pass the inspection.

Here, the plurality of inspectors 34 perform work using a predetermined terminal device. The terminal device of the inspector 34 may be a computer device or a telecommunication device such as a smart phone, tablet, PDA, laptop, desktop, etc., but is not limited thereto.

Subsequently, in step S130, the server collectively collects the rejection reason input from the plurality of inspectors 34.

In this case, the server may classify and store the collected rejection reasons for each inspector.

In one embodiment, the server may collect the rejection reason for each project and generate a rejection reason data pool for each project.

Referring to FIG. 4, a rejection reason data pool in which rejection reasons to which the same project ID is assigned are recorded one row at a time may be generated.

Subsequently, referring to FIG. 3 again, in step S140, the server extracts a plurality of rejection reason keywords from the collected rejection reason.

Specifically, the server extracts a word, which is a noun, from the reason for the reaction composed of sentences.

For example, referring to FIG. 4, from the reason for the rejection of the sentence "There is an echo", the noun "woolim" may be extracted as the keyword for the reason for rejection. In addition, the noun “noise” can be extracted as the keyword for the reason for rejection from the reason for rejection in the sentence "Noise is mixed."

Subsequently, referring to FIG. 3 again, in step S150, the server selects a major rejection reason keyword from among a plurality of extracted rejection reason keywords.

Specifically, the server calculates the frequency of occurrence of each word extracted as the reason for rejection keyword, and generates a list of reasons for rejection based on the frequency of appearance. In addition, a word within a predetermined ranking is selected as the main reason for the rejection keyword from the list of reasons for rejection.

For example, referring to FIG. 5, words such as ringing, noise, earphones, awkwardness, and blur are extracted as keywords for reasons of rejection, and are ranked according to the frequency of occurrence of each word and listed. At this time, the server may select words in the 3 rankings, ie, ringing, noise, and earphone as the main reason for returning keywords.

In an embodiment, a predetermined ranking selected as the main reason for rejection is a value previously set by the server according to the reliability. That is, the server may adjust the number of words selected as the main reason for rejection keyword according to the level requested by the client 10 or the level required according to the nature of the project.

In addition, in one embodiment, the list of reasons for rejection is updated in real time whenever the reason for rejection is input from the examiner 34.

That is, since the inspection of the work result of the worker 32 is performed in real time, the rejection reason input by the examiner 34 is collected in the rejection reason data pool in real time. The server extracts the keyword for the reason for the companion in real time from the reason for the response that is collected in the data pool for the reason for the companion. Accordingly, the frequency of occurrence of each keyword due to companion is updated in real time, and a list of reasoned keywords for companion, in which a plurality of reason keyword is listed in order of appearance frequency, is also updated in real time.

For example, referring to FIG. 5, as the reason for rejection is continuously input,'Ullim', which was the first priority, may be updated to the second priority, and the'Noise', which was the second priority, may be updated to the first priority.

As an embodiment, referring to FIG. 6, the server generates a list indicating the progress of the inspection for each inspector A, inspector B, inspector C, and inspector D participating in the same project.

The list shows the number of participating tasks by inspector, the number of normal jobs, the number of rejected jobs, and the total number of jobs.

The number of participating tasks represents the tasks that the inspector 34 is currently participating in among the total tasks assigned to the inspectors participating in the project. At this time, the total task is equally assigned to all the inspectors 34 participating in the project, and each task includes a plurality of tasks, so that the inspector 34 must complete the entire task included in one task. Will be completed.

The number of normal jobs refers to the number of jobs that have passed the inspection out of the total number of jobs that the inspector (34) completed inspection, and the number of rejected jobs is the number of jobs rejected out of the total number of jobs that the inspector (34) completed inspection. it means.

The total number of jobs means the number of jobs that have completed the inspection out of the total number of jobs assigned to the inspector 34.

For example, inspector A may be participating in a third task out of a total of five tasks, and both the previous first task and the second task have been completed. In addition, it can be seen that the inspector A has completed 5795 of the total 17502 jobs, and has decided to pass the inspection for 5573 of them, and has decided to reject 222 cases.

Subsequently, referring again to FIG. 3, in step S160, the server checks whether or not the main rejection reason keyword is included in each examinee's rejection reason, and selects the defective inspector according to whether or not the main rejection reason keyword is included.

That is, referring to FIG. 7, the server may generate a rejection task list by checking whether the main rejection reason keyword is included for each rejection of the examinee A.

For example, when the keywords for the main reason for the companionship are selected as ringing, noise, or earphone, it can be seen that at least one word of ringing, noise, and earphone is included in the reason for the companion marked with'O'.

Thereafter, based on the list indicating the presence or absence of the main reason for rejection, the server calculates the number of rejection reasons including the main reason for rejection keyword among the total rejection reasons of each inspector and the number of rejection reasons that do not include the main reason for rejection keyword. do.

In other words, the server calculates the number of cases with and without the main reason for rejection keywords of inspector A by displaying'O' and'X' in the list of rejection tasks of examiner A.

Specifically, the server calculates the ratio of the number of cases in which the main rejection reason keyword calculated relative to the total rejection reason of each examinee, and when the calculated ratio is greater than or equal to a preset ratio, the server determines the examinee as a defective examiner. At this time, the preset ratio is preset by the server according to the reliability. That is, the server may set the corresponding ratio lower as the reliability of the level requested by the client 10 or the level required according to the nature of the project is higher.

For example, referring to FIG. 8, it can be seen that the total number of rejection tasks of inspector A is 150, of which the number of cases in which the main rejection reason keyword is not included is 83. In other words, 55.3% of Inspector A's rejection cases do not include the main reason for rejection.

At this time, if the ratio set in advance by the server is 60%, inspector A is determined as a normal inspector as a result of monitoring.

On the other hand, if the ratio set in advance by the server is 50%, since the ratio of the number of cases not included in inspector A is 55.3%, inspector A is judged as a defective inspector as a result of monitoring.

After that, the server gives a penalty to the inspector who is determined to be a defective inspector.

For example, an inspector who is determined to be a defective inspector may have his or her qualifications for inspection suspended so that subsequent inspections cannot be performed, or another inspector may have a second inspector inspect at least some of the work results inspected by the inspector, Or, you can give your attention through a warning.

In one embodiment, the main reason for rejection keywords are selected for each worker, thereby increasing reliability when selecting a defect inspector. In this case, the defective inspectors can be selected according to whether or not the main rejection reason keyword for each worker is included in the reason for rejection of each inspector for each worker.

That is, since the worker 32 repeatedly makes the same mistake, the reasons why each worker's work is rejected are almost the same. On the other hand, the poor inspector 34 inputs a random reason for rejection regardless of the work content of the worker when rejecting. Therefore, when a plurality of inspectors inspect the work result of one operator, if a key reason for rejection is selected for each operator, it is possible to easily check which inspector is not properly performing the inspection.

9 is a block diagram of an examiner monitoring apparatus based on a reason for rejection according to an embodiment of the present invention.

Referring to FIG. 9, the reviewer monitoring device 200 (hereinafter, referred to as'tester monitoring device') based on a rejection reason includes a communication unit 210, a storage unit 220, an extraction unit 230, and a determination unit ( 240).

The communication unit 210 transmits a crowdsourcing-based work for one project to a plurality of workers, receives work results from the plurality of workers, and inspects the work results input from the plurality of workers by a plurality of inspectors. It transmits to the plurality of inspectors so that they can do so, and receives a rejection reason for a job rejected due to failure to pass the inspection from the plurality of inspectors.

The storage unit 220 serves to collectively collect and store the rejection reason input from a plurality of inspectors.

The extraction unit 230 serves to extract a plurality of reasons for revocation keywords from the collected reason for revocation, and to select a main keyword for reason of revocation among the extracted plurality of reason keywords.

The determination unit 240 serves to select defective inspectors according to whether the main rejection reason keyword is included in the rejection reason of each examinee.

The inspector monitoring apparatus 200 described with reference to FIG. 9 may be provided as a component of the above-described server.

The above-described method for monitoring an examiner based on a rejection reason according to an exemplary embodiment of the present invention may be implemented as a program (or application) and stored in a medium in order to be executed by being combined with a computer that is hardware.

The above-described program includes C, C++, JAVA, Ruby, which can be read by a processor (CPU) of the computer through the device interface of the computer, in order for the computer to read the program and execute the methods implemented as a program. It may include a code (Code) coded in a computer language such as machine language. Such code may include a functional code related to a function that defines necessary functions for executing the methods, and includes a control code related to an execution procedure necessary for the processor of the computer to execute the functions according to a predetermined procedure. can do. In addition, these codes may further include additional information required for the processor of the computer to execute the functions, or code related to a memory reference to which location (address address) of the internal or external memory of the computer should be referenced. have. In addition, when the processor of the computer needs to communicate with any other computer or server in the remote in order to execute the functions, the code uses the communication module of the computer to determine how It may further include a communication-related code for whether to communicate, what kind of information or media should be transmitted and received during communication.

The stored medium is not a medium that stores data for a short moment, such as a register, cache, memory, etc., but a medium that stores data semi-permanently and can be read by a device. Specifically, examples of the storage medium include, but are not limited to, ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, and the like. That is, the program may be stored in various recording media on various servers to which the computer can access, or on various recording media on the user's computer. In addition, the medium may be distributed over a computer system connected through a network, and computer-readable codes may be stored in a distributed manner.

In the above, embodiments of the present invention have been described with reference to the accompanying drawings, but those of ordinary skill in the art to which the present invention pertains can be implemented in other specific forms without changing the technical spirit or essential features. You can understand. Therefore, the embodiments described above are illustrative in all respects, and should be understood as non-limiting.

10: Client
20: service provider
30: public
32: worker
34: inspector
200: inspector monitoring device
210: communication department
220: storage unit
230: extraction unit
240: judgment unit

Claims (12)

As a method performed by a computer,
Granting a crowd-sourcing-based work for one project to a plurality of workers, and receiving work results from the plurality of workers;
Allowing a plurality of inspectors to inspect the work results input from the plurality of workers, but receiving a rejection reason for the rejected job due to failing to pass the inspection from the plurality of inspectors;
Collecting the rejection reason inputted from the plurality of inspectors as a whole;
Extracting a plurality of keyword reason for rejection from the collectively collected reason for rejection;
Selecting a main reason for rejection keyword from among the extracted reason keywords for rejection; And
Including the step of selecting a defective inspector according to whether or not the main rejection reason keyword is included in the rejection reason of each examinee,
The step of selecting the defective inspector,
Calculating the number of rejection reasons including the main rejection reason keyword and the number of rejection reasons not including the major rejection reason keyword among the total rejection reasons of each examinee;
Calculating a ratio of the number of rejection reasons that do not include the main rejection reason keyword calculated relative to the total rejection reason of each examinee; And
If the calculated ratio is greater than or equal to a preset ratio, the method of monitoring a rejection reason-based examiner comprising the step of determining that the examinee is inputting a reason for rejection and selecting a defect examiner. The method of claim 1,
The step of extracting the plurality of reasons for rejection keywords,
A method for monitoring a reviewer based on a reason for a companion, which extracts a word that is a noun from the reason for the companion composed of sentences. The method of claim 2,
The step of selecting the main reason for rejection keyword,
Calculating an appearance frequency for each word;
Generating a list of reasons for rejection based on the frequency of occurrence; And
Including the step of selecting a word within a predetermined ranking in the keyword reason for rejection as the main reason for rejection keyword, the reviewer monitoring method based on the reason for rejection. The method of claim 3,
The reason for rejection keyword list is updated in real time whenever the reason for rejection is input from the examiner. The method of claim 3,
The predetermined ranking is determined according to the reliability of the required level of the project, a rejection reason-based inspector monitoring method. delete delete The method of claim 1,
The preset ratio is set according to the reliability of the required level of the project, the reviewer monitoring method based on rejection reason. The method of claim 1,
Rejection reason-based, in which at least a part of the work results inspected by the inspector selected as the defective inspector is secondarily inspected by another inspector, or a penalty including warning or suspension of inspection qualification is given to the inspector selected as the defective inspector. How to monitor inspectors. The method of claim 1,
The keywords for the main reason for rejection are selected for each worker,
The step of selecting the defective inspector is to select the defective inspector according to whether the main rejection reason keyword for each worker is included in the rejection reason of each inspector for each worker, the inspection method based on rejection reason. A computer program that is combined with a computer that is hardware and stored in a medium to execute the method of monitoring the examiner based on the rejection reason of any one of claims 1 to 5, 8, 9 and 10. As an inspector monitoring device based on the reason for the companion,
Sending a crowdsourcing-based work for one project to a plurality of workers, receiving work results from the plurality of workers, and allowing a plurality of inspectors to inspect the work results input from the plurality of workers. A communication unit for transmitting a rejection reason for a job rejected due to failure to pass the inspection from the plurality of examiners;
A storage unit for collectively collecting and storing the reason for rejection input from the plurality of inspectors;
An extracting unit for extracting a plurality of reason for the companion keyword from the overall collected reason for the companion, and for selecting a keyword for the main reason for the reaction from among the extracted plurality of reason for the companion keyword; And
Including a determination unit that selects defective inspectors according to whether or not each inspector's rejection reason includes the above main rejection reason keyword,
The determination unit,
Among the total rejection reasons of each examinee, the number of rejection reasons including the main rejection reason keyword and the number of rejection reasons that do not include the main rejection reason keyword is calculated, and the above main rejection reason keywords calculated relative to the total rejection reasons of each examinee Calculates the ratio of the number of reasons for rejection that does not include a, and if the calculated ratio is greater than or equal to a preset ratio, the examiner is identified as entering a random reason for rejection and selects as a defective examiner. Monitoring device.

Images