KR102155866B1 - Method and apparatus for monitoring worker based on cause of return - Google Patents

Abstract

The present invention relates to a method and a device for worker monitoring based on a cause of return. The method includes: a step of granting crowdsourcing-based work with respect to one project to a plurality of workers and receiving the input of a work result from the plurality of workers; a step of causing a plurality of inspectors to inspect the work result input from the plurality of workers and receiving the input of a cause of return and a cause-of-return keyword with respect to work that the plurality of inspectors returned for non- pass of the inspection; a step of classifying the returned work of each worker by the cause-of-return keyword; and a step of sorting a substandard worker in accordance with the ratio of the returned work including one of the plurality of cause-of-return keywords to the total returned work of each worker. In the step of receiving the input of the cause of return and the cause-of-return keyword from the plurality of inspectors, each inspector can be provided with a cause-of-return keyword list including a main cause-of-return keyword and selectively input the cause-of-return keyword from the cause-of-return keyword list.

Description

Worker monitoring method and device based on the reason for refusal {METHOD AND APPARATUS FOR MONITORING WORKER BASED ON CAUSE OF RETURN}

The present invention relates to a method and apparatus for monitoring a worker based on a reason for rejection.

Recently, more and more companies collect and process large amounts of data on a crowdsourcing basis that engages the general public in some process of corporate activities. In other words, by opening a project and allowing the general public, that is, workers to participate in the project, necessary information is collected through the work results completed by the workers.

At this time, there are various types of tasks provided to the worker, and various work capabilities are required to perform each task. Therefore, each worker naturally has a weak part and a superior part among various types of work for each individual.

However, the current situation is that tasks are randomly assigned without taking into account the difference in capabilities for each worker. Therefore, if a task requiring the same ability is continuously assigned to a vulnerable worker in a specific part, there is a problem in that the task is rejected for the same cause, increasing only useless data and unreasonably increasing the cost of work.

In addition, in the crowdsourcing-based platform, there is a process of evaluating the work result and ability of the worker, but this is a method of evaluating the overall work result of the worker, and there is a problem that the vulnerable part of each worker cannot be identified.

Unexamined Patent Publication No. 10-2014-0095956, 2014.08.04.

The problem to be solved by the present invention is to identify the vulnerable part of each worker based on the rejection reason input by the inspector, and through this, monitoring the worker based on rejection reason that can select defective workers with low work ability for a specific project. It is to provide a method and apparatus.

The problems to be solved by the present invention are not limited to the problems mentioned above, and other problems that are not mentioned will be clearly understood by those skilled in the art from the following description.

In the method for monitoring a worker based on a reason for rejection according to an aspect of the present invention for solving the above-described problems, a crowd-sourcing-based task for one project is assigned to a plurality of workers, and the work results are inputted from the plurality of workers. , Having a plurality of inspectors inspect the work results input from the plurality of workers, but receiving the rejection reason and keywords for the rejection reason for the job rejected because the inspection failed from the plurality of inspectors, rejection of each worker Classifying a job by the reason for rejection keyword, and selecting a defective worker according to a ratio of the rejection operation including any one of the plurality of rejection reason keywords compared to the total rejection operation of each worker, In the step of receiving input of the rejection reason and the rejection reason keyword from the plurality of examiners, a list of rejection reason keywords including major rejection reason keywords is provided to each examiner, and each examiner selects the rejection reason keyword from the rejection reason keyword list. You can make it possible to input.

In some embodiments of the present invention, when the rejection reason keyword to be input is not included in the rejection reason keyword list, the examiner may directly input the rejection reason keyword.

In some embodiments of the present invention, the step of selecting the defective worker may include, if the ratio of the rejection operation including any one of the plurality of rejection reason keywords to the total rejection operation of each worker is greater than or equal to a preset ratio. , It may be to select the worker as a defective worker.

In some embodiments of the present invention, the preset ratio may be set according to reliability.

In some embodiments of the present invention, the method for monitoring a worker based on the reason for rejection allows a plurality of inspectors to inspect the work results input from the plurality of workers, but to the rejected task due to failure to pass the inspection from the plurality of inspectors. Receiving only the rejection reason for input, collecting the rejection reason inputted from the plurality of examiners as a whole, extracting a plurality of rejection reason keywords from the collected rejection reason, and among the plurality of extracted rejection reason keywords It may further include the step of selecting the main reason for the rejection keyword.

In some embodiments of the present invention, the step of extracting the plurality of rejection reason keywords may include extracting a word that is a noun from the rejection reason consisting of sentences.

In some embodiments of the present invention, the step of selecting the main reason for rejection keyword includes calculating an appearance frequency for each word and selecting a word within a predetermined ranking as the main reason for rejection keyword based on the appearance frequency. can do.

In some embodiments of the present invention, the predetermined ranking may be determined according to reliability.

In some embodiments of the present invention, the main reason for rejection keyword may be updated in real time whenever a reason for rejection is input by an examiner.

In some embodiments of the present invention, the default value of the list of reasons for rejection may include keywords for other projects similar to the project.

In some embodiments of the present invention, a penalty including warning or suspension of work qualification may be given to a worker selected as the defective worker.

A computer program according to another aspect of the present invention for solving the above-described problem is combined with a computer that is hardware to execute the worker monitoring method based on the reason for rejection, and is stored in a medium.

A worker monitoring device based on rejection reason according to another aspect of the present invention for solving the above-described problem transmits a crowdsourcing-based work for one project to a plurality of workers, and receives work results from the plurality of workers. And, the result of the work input from the plurality of workers is transmitted to the plurality of inspectors so that the plurality of inspectors can inspect it, and the rejection reason and the rejection reason keyword for the job rejected because the inspection did not pass from the plurality of inspectors. The communication unit that receives the and classifies the rejection job of each worker by the rejection reason keyword, and the defective worker according to the ratio of the rejection operation including any one of the plurality of rejection reason keywords compared to the total rejection operation of each worker Including a determination unit for selecting, wherein the communication unit provides a list of reasons for rejection keywords including the main reason keywords for rejection to each examiner, and allows each examiner to select and input the reason for rejection keyword from the list of reason for rejection keywords.

Other specific details of the present invention are included in the detailed description and drawings.

According to the method and apparatus for monitoring a worker based on the reason for rejection of the present invention as described above, keywords for rejection reason are extracted from all rejection reasons collected for the same project, and all rejection cases of each worker are classified by keyword reason for rejection, and each It is possible to calculate the ratio of the rejection cases including each rejection reason keyword to the total rejection of the worker, and based on this, whether the corresponding worker is a defective worker or not.

By providing a list of reasons for rejection including keywords for reasons of rejection to the examiner, it is possible to unify the expression of reasons for rejection among examiners for the same reason. Accordingly, a list of reasons for rejection keywords can be quickly and accurately completed.

In addition, it is possible to prevent the occurrence of a rejection case for the same reason by temporarily stopping the work that requires the corresponding capability by identifying the vulnerable part of each worker.

In addition, for the same reason, the amount of defective work is reduced by preventing the occurrence of the rejection case, thereby preventing the bottleneck of the inspection operation, and the inspector can efficiently perform the inspection.

The effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the following description.

1 is a conceptual diagram of a crowd sourcing service according to an embodiment of the present invention.
2 is a flowchart illustrating a crowdsourcing-based project progress process according to an embodiment of the present invention.
3 is a flowchart of a method for monitoring a reviewer based on a rejection reason according to an embodiment of the present invention.
4 is an exemplary diagram for explaining input of a reason for rejection and a keyword for reason of rejection according to an embodiment of the present invention.
5 is an exemplary view for explaining a monitoring result of a worker according to an embodiment of the present invention.
6 is a flowchart of a method of selecting a keyword for main reason for rejection using a reason for rejection according to an embodiment of the present invention.
7 is a diagram illustrating a timeline from project open to end according to an embodiment of the present invention.
8 is an exemplary view for explaining a reason data pool for rejection generated for the same project according to an embodiment of the present invention.
9 is an exemplary diagram for explaining a reason for rejection keyword and a main reason for rejection according to an embodiment of the present invention.
10 is a block diagram of an apparatus for monitoring a worker based on a rejection reason according to an embodiment of the present invention.

Advantages and features of the present invention, and a method of achieving them will become apparent with reference to the embodiments described below in detail together with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but may be implemented in a variety of different forms, only the present embodiments are intended to complete the disclosure of the present invention, It is provided to fully inform the technician of the scope of the present invention, and the present invention is only defined by the scope of the claims.

The terms used in the present specification are for describing exemplary embodiments and are not intended to limit the present invention. In this specification, the singular form also includes the plural form unless specifically stated in the phrase. As used in the specification, “comprises” and/or “comprising” do not exclude the presence or addition of one or more other elements other than the mentioned elements. Throughout the specification, the same reference numerals refer to the same elements, and “and/or” includes each and all combinations of one or more of the mentioned elements. Although "first", "second", and the like are used to describe various elements, it goes without saying that these elements are not limited by these terms. These terms are only used to distinguish one component from another component. Therefore, it goes without saying that the first component mentioned below may be the second component within the technical idea of the present invention.

Unless otherwise defined, all terms (including technical and scientific terms) used in the present specification may be used as meanings that can be commonly understood by those of ordinary skill in the art to which the present invention belongs. In addition, terms defined in a commonly used dictionary are not interpreted ideally or excessively unless explicitly defined specifically.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a conceptual diagram of a crowd sourcing service according to an embodiment of the present invention.

Referring to FIG. 1, a crowdsourcing service is performed by being composed of a client 10, a service provider 20, and a public 30.

The sponsor 10 refers to a company or individual who requests a crowdsourcing-based project (hereinafter, referred to as a project).

The requester 10 requests a project for the purpose of collecting source data or annotating data. Collection of source data refers to collecting raw data such as recorded voice collection and photo collection. Data annotation refers to inputting relevant annotation data into source data such as text, photos, and videos. For example, the data annotation may include finding an entity in a given fingerprint or finding a similar sentence, but is not limited thereto.

The service provider 20 refers to a company that provides crowdsourcing services.

When the service provider 20 receives a request for a project for a product or service from the client 10, the service provider 20 allocates a task for the project to the general public 30 and receives the work result from the public 30. Thereafter, the final product extracted based on the work result is provided to the client 10.

At this time, the service provider 20 provides a crowd sourcing service to the client 10 and the public 30 through a crowd sourcing platform (hereinafter, the platform). That is, when the service provider 20 receives a request for a project from the client 10, the service provider 20 opens the project on the platform. Thereafter, when the work result for the open project is received from the public 30, the project is terminated on the platform, and the final product may be extracted and provided to the client 10.

The public 30 refers to the general public participating in the project open on the platform. Here, the public 30 may participate in a project opened on the platform through an application or website provided by the service provider 20.

The public 30 is composed of a worker 32 and an inspector 34.

The worker 32 decides to participate in a specific project among a plurality of projects open on the platform. Thereafter, the operator 32 performs an operation such as collection of source data or data annotation, and transmits it to the platform.

The inspector 34 decides to participate in a specific project among a plurality of projects open on the platform. Thereafter, the inspector 34 inspects the result of the work performed by the operator 32. As a result of performing the inspection, the inspector 34 may pass or reject the job, and may input a reason for rejection when returning the job.

2 is a flowchart illustrating a crowdsourcing-based project progress process according to an embodiment of the present invention.

First, the client 10 requests a project to the service provider 20 (S41).

Thereafter, the service provider 20 opens the requested project on the platform (S42). At this time, the service provider 20 determines the grade in consideration of the difficulty level of the project before opening the project. In other words, depending on the level of difficulty, it is decided to expose the project to the public 30 or higher. Accordingly, it is possible to increase the reliability of the work result of the project.

Thereafter, the service provider 20 allocates a task to the worker 32 of the corresponding grade or higher according to the grade of the project (S43).

Thereafter, the worker 32 performs the assigned task (S44). In this case, the worker 32 may input the reason for the inability to work without performing the work for a work in which the work itself is impossible for some reason.

Thereafter, the service provider 20 receives the work result from the worker 32 (S45), and assigns the inspection work for the work result to the inspector 34 (S46).

Thereafter, the inspector 34 performs the assigned inspection (S47). At this time, the inspector 34 decides to pass the inspection when it is determined that the work has been properly performed, and determines rejection when it is determined that the inspection work is wrong. When deciding to reject, the inspector 34 inputs the rejection reason for what reason the task was judged to be wrong.

Thereafter, the service provider 20 receives the inspection result from the inspector 34 (S48).

If the inspection result is Pass, the service provider 20 uses the work result as valid data, and based on this, extracts the final product at the end of the project.

If the inspection result is rejected, the service provider 20 may internally perform the inspection again or assign the job again to the worker 32 to perform the job.

Thereafter, when the project period ends or sufficient valid data is secured, the service provider 20 terminates the project (S49), calculates a final result based on the secured valid data, and provides it to the client 10 ( S50).

At this time, before the end of the project, the service provider 20 evaluates the performance results of the worker 32 and the inspector 34, calculates the work cost and the inspection cost according to the evaluation, to the worker 32 and the inspector 34. give.

In Figs. 1 and 2, it is simply expressed as a client, service provider, worker, and inspector, but these refer to computer devices or telecommunication devices such as smartphones, tablets, PDAs, laptops, desktops, servers, etc. operated by each participant. .

3 is a flowchart of a method for monitoring a reviewer based on a reason for rejection according to an embodiment of the present invention, and FIG. 4 is an exemplary view for explaining inputting a reason for rejection and a keyword for reason of rejection according to an embodiment of the present invention, and FIG. 5 Is an exemplary view for explaining the monitoring result of a worker according to an embodiment of the present invention.

The steps illustrated in FIG. 3 may be understood as being performed by a platform server (hereinafter, referred to as a server) operated by the service provider 20, but are not limited thereto.

Referring to FIG. 3, in step S110, the server assigns a crowdsourcing-based work for one project to a plurality of workers 32, and receives work results from the plurality of workers 32.

The server is provided with results of a plurality of work performed by a plurality of workers 32 participating in the project. At this time, one project includes a plurality of tasks, and the plurality of tasks may be grouped into a plurality of tasks according to the type, or each task may be a different type of work or the same type of work. , But is not limited thereto.

Here, the plurality of workers 32 perform work using a predetermined terminal device. The terminal device of the worker 32 may be a computer device or a telecommunication device such as a smart phone, tablet, PDA, laptop, desktop, etc., but is not limited thereto.

Subsequently, in step S120, the server causes the plurality of inspectors 34 to inspect the work results input from the plurality of workers 32. At this time, the server prompts the plurality of inspectors 34 to input a rejection reason and a rejection reason keyword for a job rejected because it did not pass the inspection.

The server provides a screen for inputting the reason for rejection and the reason for rejection keyword to the examiner 34.

Referring to FIG. 4, the server provides a list of reasons for rejection keywords and an input window for reasons for rejection to the examiner 34. The examiner 34 selects one reason for rejection keyword from the list of reason for rejection keywords, and inputs a specific reason for rejection.

Here, the plurality of inspectors 34 perform work using a predetermined terminal device. The terminal device of the inspector 34 may be a computer device or a telecommunication device such as a smart phone, tablet, PDA, laptop, desktop, etc., but is not limited thereto.

In an embodiment, when a rejection reason to be input is included in the rejection reason keyword list, the examiner 34 selects a rejection reason keyword related to the rejection reason to be input from the rejection reason keyword list. In addition, the examiner 34 inputs a sentence containing the selected reason for rejection keyword or related to the selected reason for rejection keyword.

Referring to FIG. 4, when the inspector 34 selects "noise" from the list of reasons for rejection keywords, the reason for rejection is input. For example, you can also type "I hear noise" to include the keyword for the reason you selected (noise). Alternatively, you can enter "I hear a different sound" so that the keyword (noise) of the reason you chose is relevant.

In another embodiment, when the rejection reason to be entered in the rejection reason keyword list is not included, the examiner 34 selects "direct input" at the bottom of the rejection reason keyword list as shown in FIG. 4 and rejects A separate reason for rejection can be entered as a sentence in the reason input window.

The server stores the rejection reason and the rejection reason keyword input from the examiner 34 in the rejection reason data pool. The rejection reason data pool is continuously updated whenever the rejection reason and the rejection reason keyword are input from the plurality of inspectors 34.

Accordingly, the list of reasons for rejection keywords is also updated in real time. A detailed description of this will be described later.

Here, the reason for rejection keyword is used to monitor the worker 32, and the reason for rejection may be used as a guideline when requesting rework from the worker 32 later.

Subsequently, in step S130, the server classifies the rejected jobs of each worker 32 by rejection reason keywords.

That is, the work result of each worker 32 is rejected by the plurality of inspectors 34 for various reasons, at this time, the server performs the entire rejection of the worker 32 with a plurality of rejection reasons input from the plurality of inspectors 34 They are classified by keyword.

Referring to FIG. 5, if there are 120 jobs rejected by the inspector 34 among jobs completed by worker A, the server classifies 120 rejected jobs by rejection reason keywords.

Worker A said that out of a total of 120 rejection cases, 90 rejection actions containing the keyword “woolrim” for rejection reason, 20 rejection actions containing “shake”, and 10 rejection actions including “earphone”. I can confirm.

Subsequently, in step S140, the server selects a defective worker according to the ratio of the rejection job including any one of the plurality of rejection reason keywords compared to the total rejection work of each worker 32.

Specifically, if the ratio of the rejection job including any one of the plurality of rejection reason keywords to the total rejection work of each worker 32 is more than a preset ratio, the corresponding worker can be selected as a defective worker. have.

At this time, the preset ratio is preset by the server according to the reliability. That is, the server may set the corresponding ratio lower as the reliability of the level requested by the client 10 or the level required according to the nature of the project is higher.

For example, referring to FIG. 5, 120 cases of total rejection of worker A include 90 cases including the rejection reason keyword "woolim", 20 cases including the rejection reason keyword "shake", and the rejection reason keyword "earphone" It is classified into 10 included cases.

At this time, the ratio of the rejection job that includes the rejection reason keyword "woolim" compared to all rejection jobs is 75%, the ratio of the rejection job that includes the rejection reason keyword "quiver" is 17%, and the rejection reason keyword "earphone" The percentage of rejection work that includes this is 8%. In other words, inspector A is being rejected for the reason that it is "woolim" in a large proportion.

In this case, if the ratio preset by the server is 80%, the ratio of rejection work cases including each rejection reason keyword is 80% or less, so that worker A is determined to be a normal worker. That is, the worker A is determined to be a normal worker even though the specific work ability is significantly deteriorated or the environment is not suitable for performing the specific work, and continues to perform the same work. Accordingly, jobs that are rejected for the same reason continue to occur.

On the other hand, if the ratio preset by the server is 60%, the ratio of the rejection job that includes the rejection reason keyword "woolrim" is more than 60%, so the server repeats the rejection operation due to the reason that worker A is "woolim". It recognizes that it is generated and judges worker A as a defective worker. Accordingly, it is possible to block the occurrence of a rejected task for the same reason in advance.

Thereafter, the server gives a penalty to the worker 32 selected as a defective worker.

For example, the server may give attention to the worker 32 selected as a defective worker through a warning, or prevent the worker 32 selected as a defective worker from performing a task until the project is finished. have.

As the qualification of the worker 32 selected as a defective worker is stopped, it is possible to prevent occurrence of a rejection case for the same reason, thereby reducing useless data and increasing the speed and efficiency of the inspector's inspection work.

Until now, the server has the examiner 34 to select one of the rejection reason keyword list, and by using the selected keyword to create a rejection reason, based on the rejection reason keyword and the rejection reason input from the plurality of examiners 34 It has been described about monitoring workers and selecting defective workers.

In this case, the list of reasons for rejection provided to the examiner 34 is generated only when a sufficient amount of reason for rejection is input from the plurality of examiners 34. That is, the list of keywords for rejection reasons may be generated after the middle of the project in which the review work has been conducted to some extent. Accordingly, the inspectors 34 are provided with a list of reasons for rejection keywords after a certain amount of time elapses after the project is opened, and can select and input keywords for rejection reasons.

However, if a project similar to the corresponding project has been previously carried out, a list of reasons for rejection including keywords for the previous project can be used as a default value when the project is opened. That is, the server may provide the reviewers 34 with the list of reasons for rejection of the previous project at the beginning of the project, that is, until the list of reasons for rejection is generated.

Hereinafter, with reference to FIGS. 6 to 9, a list of reasons for rejection keywords is generated at the beginning of a project, and a list of reasons for rejection keywords is updated in the middle and late stages of the project will be described.

6 is a flow chart of a method for selecting a main reason for rejection keyword using a reason for rejection according to an embodiment of the present invention, and FIG. 7 is a view showing a timeline from project open to end according to an embodiment of the present invention. FIG. 8 is an exemplary diagram for explaining a reason for rejection data pool created for the same project according to an embodiment of the present invention, and FIG. 9 is a diagram illustrating a reason for rejection keyword and a main reason for rejection keyword according to an embodiment of the present invention It is an exemplary diagram for doing.

The steps shown in FIG. 6 may be understood as being performed by a platform server (hereinafter, referred to as a server) operated by the service provider 20, but are not limited thereto.

Referring to FIG. 6, in step S110, the server assigns a crowdsourcing-based work for one project to a plurality of workers, and receives work results from the plurality of workers. In other words, this step is a step that is carried out continuously throughout the early, mid, and late project, and the server receives work results from a plurality of workers until the project is opened and closed.

Subsequently, in step S220, the server causes the plurality of inspectors to inspect the work results input from the plurality of workers, but receives only the rejection reason for the rejected job due to failure to pass the inspection from the plurality of inspectors.

That is, at the beginning of the project, since there is not enough data to generate a list of reasons for rejection, the server receives the reason for rejection from the examiner 34 and collects sufficient data.

In one embodiment, in the case that a list of reasons for rejection including keywords of the previous project is provided by default at the beginning of the project, the examiner 34 selects one reason for rejection keyword from the list of reasons for rejection, and rejects Enter the reason. In this case, the server may generate a keyword list of reasons for rejection of the project by using only the reason for rejection.

In step S230, the server collectively collects rejection reasons input from the plurality of inspectors 34. At this time, the collected rejection reason is stored in the rejection reason data pool created in the server.

That is, referring to FIG. 7, at the beginning of the project, the server collectively collects rejection reasons input from a plurality of inspectors 34 to build a rejection reason data pool.

Referring to FIG. 8, a rejection reason data pool in which rejection reason to which the same project ID is assigned is recorded one row at a time may be constructed.

For an ongoing (opened) project, the server receives the work result from the operator in real time and the inspection result from the inspector in real time. Therefore, the rejection reason data pool for each project is updated by collecting the rejection reason in real time.

Thereafter, referring to FIG. 7, after the project is opened and a certain period of time has elapsed, the server generates a keyword list of reasons for rejection if it determines that the reason for rejection is sufficiently collected.

Specifically, referring again to FIG. 6, in step S240, the server extracts a plurality of keywords for rejection reason from the collected entire rejection reason. Specifically, the server extracts a word, which is a noun, from the reason for the reaction composed of sentences.

For example, referring to FIG. 8, the noun “woolim” may be extracted as a keyword for the reason for rejection from the reason for rejection of the sentence "There is an echo". In addition, the noun “noise” can be extracted as the keyword for the reason for rejection from the reason for rejection in the sentence "Noise is mixed."

Subsequently, referring to FIG. 6 again, in step S250, the server selects a major rejection reason keyword from among a plurality of extracted rejection reason keywords.

Specifically, the server calculates the frequency of appearance of each word extracted as the reason for rejection keyword, and selects a word within a predetermined ranking as the main reason for rejection based on the frequency of appearance.

For example, referring to FIG. 9, words such as ringing, noise, earphones, awkwardness, and blurring are extracted from all the reasons for the rejection, and are ranked according to the frequency of occurrence of each word. At this time, the server may select words in the 3 rankings, ie, ringing, noise, and earphone as the main reason for returning keywords.

The predetermined ranking selected as the main reason for returning keywords is a value previously set by the server according to the reliability. That is, the server may adjust the number of words selected as the main reason for rejection keyword according to the level requested by the client 10 or the level required according to the nature of the project.

Thereafter, the server provides the reviewer 34 with a list of reasons for rejection including the selected keywords for reason of rejection. The examiner 34 may select one reason for rejection keyword using the provided reason for rejection keyword list, and input the reason for rejection by using the selected reason for rejection keyword.

Referring to FIG. 7, after a list of reasons for rejection is generated and initially provided to the examiner 34, the list of reasons for rejection is updated every predetermined period or in real time.

From the middle of the project, the server collects the rejection reason keywords and rejection reason input from the plurality of inspectors 34 in real time in the rejection reason data pool. The server updates the list of reasons for rejection keywords collected in real-time in the reason for rejection data pool by aggregating the keyword reason for rejection every predetermined time or in real time.

In this case, the server allows the examiner 34 to directly input the rejection reason as well as the rejection reason keyword, or not select the rejection reason keyword.

That is, when only the keyword for reason of rejection is selected and input, the server cannot extract a new keyword for reason of rejection any more. In other words, since only the existing selected main reason for rejection keyword is continuously selected, the reason for rejection of new content cannot be entered, and accordingly, the accuracy of operator monitoring is degraded.

Therefore, the server prompts the examiner to select one reason for the rejection keyword from the list of reason for the rejection keyword, and inputs the reason for the rejection in the form of a sentence. Or, have them directly enter the reason for rejection without selecting a keyword for rejection reason.

Accordingly, a new keyword for the reason for rejection is extracted, the main keyword for reason for rejection is updated in real time, and the examiner is also provided with a list of the reason for the reason for rejection updated at a predetermined time or in real time, so that the reason for rejection can be selectively input. This allows the server to more accurately monitor workers.

For example, referring to FIG. 9, as the keyword for rejection reason and the reason for rejection are input in real time, “noise”, which was the second priority, may be updated to the first priority. In addition, a new keyword for reason of reaction, called "shake", may be extracted and ranked in the rankings.

10 is a block diagram of an apparatus for monitoring a worker based on a rejection reason according to an embodiment of the present invention.

Referring to FIG. 10, a worker monitoring device 300 (hereinafter, a worker monitoring device) based on a rejection reason includes a communication unit 310 and a determination unit 320.

The communication unit 310 transmits a crowdsourcing-based work for one project to a plurality of workers, receives work results from the plurality of workers, and inspects the work results input from the plurality of workers by a plurality of inspectors. It transmits to the plurality of inspectors so that they can be performed, and receives a rejection reason and a rejection reason keyword for a job rejected because it fails to pass the inspection from the plurality of inspectors.

In addition, the communication unit 310 provides a list of reasons for rejection including keywords for reasons of rejection to each examiner, and allows each examiner to select and input the reason for rejection keywords from the list of reasons for rejection.

The determination unit 320 categorizes the rejection work of each worker by the rejection reason keyword, and is defective according to the ratio of rejection work including any one of the plurality of rejection reason keywords relative to the total rejection work of each worker. It plays the role of selecting workers.

The worker monitoring apparatus 300 described with reference to FIG. 10 may be provided as a component of the above-described server.

The above-described method for monitoring a worker based on a reason for rejection according to an embodiment of the present invention may be implemented as a program (or application) and stored in a medium to be executed by being combined with a computer that is hardware.

The above-described program includes C, C++, JAVA, Ruby, which can be read by a processor (CPU) of the computer through the device interface of the computer, in order for the computer to read the program and execute the methods implemented as a program. It may include a code (Code) coded in a computer language such as machine language. Such code may include a functional code related to a function that defines necessary functions for executing the methods, and includes a control code related to an execution procedure necessary for the processor of the computer to execute the functions according to a predetermined procedure. can do. In addition, these codes may further include additional information required for the processor of the computer to execute the functions, or code related to a memory reference to which location (address address) of the internal or external memory of the computer should be referenced. have. In addition, when the processor of the computer needs to communicate with any other computer or server in the remote in order to execute the functions, the code uses the communication module of the computer to determine how It may further include a communication-related code for whether to communicate, what kind of information or media should be transmitted and received during communication.

The stored medium is not a medium that stores data for a short moment, such as a register, cache, memory, etc., but a medium that stores data semi-permanently and can be read by a device. Specifically, examples of the storage medium include, but are not limited to, ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, and the like. That is, the program may be stored in various recording media on various servers to which the computer can access, or on various recording media on the user's computer. In addition, the medium may be distributed over a computer system connected through a network, and computer-readable codes may be stored in a distributed manner.

In the above, embodiments of the present invention have been described with reference to the accompanying drawings, but those of ordinary skill in the art to which the present invention pertains can be implemented in other specific forms without changing the technical spirit or essential features. You can understand. Therefore, the embodiments described above are illustrative in all respects, and should be understood as non-limiting.

10: Client
20: service provider
30: public
32: worker
34: inspector
300: worker monitoring device
310: Communication Department
320: judgment unit

Claims (13)

As a method of monitoring workers based on the reason for rejection,
Granting a crowd-sourcing-based work for one project to a plurality of workers, and receiving work results from the plurality of workers;
Allowing a plurality of inspectors to inspect the work results input from the plurality of workers, and receiving a rejection reason and a rejection reason keyword from the plurality of inspectors for a job rejected because the inspection failed to pass;
Classifying each worker's rejection job by the rejection reason keyword; And
Including the step of selecting a defective worker according to the ratio of the rejection operation including any one of the plurality of rejection reason keywords relative to the total rejection work of each worker,
The step of receiving a rejection reason and a rejection reason keyword input from the plurality of inspectors,
Provide each examiner with a list of reasons for rejection including keywords for reasons of rejection, and allow each examiner to select and input the reason for rejection keywords from the list of reasons for rejection,
The step of selecting the defective worker,
If the ratio of the rejection job including any one of the plurality of rejection reason keywords to the total rejection of each worker is more than a preset ratio, it is identified as a vulnerable part of the worker and selects the worker as a defective worker. That is, the reason-based worker monitoring method. The method of claim 1,
When the rejection reason keyword to be entered is not included in the rejection reason keyword list, the examiner can directly input the rejection reason keyword, a method of monitoring a worker based on rejection reason. delete The method of claim 1,
The preset ratio is set according to the reliability of the required level of the project, rejection reason-based worker monitoring method. The method of claim 1,
Allowing a plurality of inspectors to inspect the work results input from the plurality of workers, but receiving only a rejection reason for the rejected job due to failure to pass the inspection from the plurality of inspectors;
Collecting the rejection reason inputted from the plurality of inspectors as a whole;
Extracting a plurality of rejection reason keywords from the collected rejection reason; And
A method for monitoring a worker based on a reason for rejection, further comprising the step of selecting the main reason keyword for rejection among the extracted reason keywords for rejection. The method of claim 5,
The step of extracting the plurality of reasons for rejection keywords,
A method of monitoring a worker based on a reason for rejection, which is to extract a word that is a noun from the reason for the companionship composed of sentences. The method of claim 6,
The step of selecting the main reason for rejection keyword,
Calculating an appearance frequency for each word; And
And selecting a word within a predetermined ranking as the main reason for rejection based on the frequency of occurrence of the word, based on the reason for rejection. The method of claim 7,
The predetermined ranking is determined according to the reliability of the required level of the project, a rejection reason-based worker monitoring method. The method of claim 1,
The main reason for rejection keyword is updated in real time whenever the reason for rejection is input from an examiner. The method of claim 1,
The default value of the rejection reason keyword list includes a key rejection reason keyword of another project similar to the project. The method of claim 1,
A method of monitoring a worker based on a reason for rejection, in which a penalty including warning or suspension of work qualification is given to the worker selected as the defective worker. A computer program that is combined with a computer as hardware and stored in a medium to execute the method of monitoring a worker based on the reason for rejection of any one of claims 1, 2 and 4 to 11. As a worker monitoring device based on the reason for the refusal,
Sending a crowdsourcing-based work for one project to a plurality of workers, receiving work results from the plurality of workers, and allowing a plurality of inspectors to inspect the work results input from the plurality of workers. A communication unit that transmits the rejection reason and the rejection reason keyword to the rejection reason and the rejection reason keyword for the job rejected because it fails to pass the examination from the plurality of examiners; And
A judgment unit for classifying each worker's rejection job by the rejection reason keyword, and selecting the defective worker according to the ratio of the rejection job including any one of the plurality of rejection reason keywords compared to the total rejection reason keyword of each worker Include,
The communication unit,
Provide each examiner with a list of reasons for rejection including keywords for reasons of rejection, and allow each examiner to select and input the reason for rejection keywords from the list of reasons for rejection,
The determination unit,
If the ratio of the rejection job including any one of the plurality of rejection reason keywords to the total rejection of each worker is more than a preset ratio, it is identified as a vulnerable part of the worker and selects the worker as a defective worker. , The reason-based worker monitoring device.

Images