KR102195953B1 - Method for reassignment of withdrawn work based on reasons for rejection in a crowdsourcing based projects for artificial intelligence training data generation - Google Patents

Abstract

The present invention relates to a method for reassigning a withdrawn work based on the reason for rejection in a crowdsourcing-based project. The method includes: a step of assigning project work to workers and making a work execution request; a step of receiving the input of work results from the workers; a step of assigning the work results to inspectors and making an inspection execution request; a step of receiving the input of inspection results with respect to the work results from the inspectors as inspection pass or reject; a step of reassigning the rejected work to the corresponding worker and making a re-work execution request; a step of setting the point in time when a preset predetermined period has elapsed since the re-work execution request as a re-work limit deadline; and a step of canceling the assignment to the worker and reassigning the withdrawn work to another worker in the event of non-fulfillment of the rejected work requested to be re-executed by the re-work limit deadline.

Description

{METHOD FOR REASSIGNMENT OF WITHDRAWN WORK BASED ON REASONS FOR REJECTION IN A CROWDSOURCING BASED PROJECTS FOR ARTIFICIAL INTELLIGENCE TRAINING DATA GENERATION}

The present invention relates to a method for reassigning a task that has been withdrawn based on a reason for rejection in a crowdsourcing-based project.

Recently, more and more companies collect and process large amounts of data on a crowdsourcing basis that engages the general public in some process of corporate activities. In other words, by opening a project and allowing the general public, that is, workers to participate in the project, necessary information is collected through the work results completed by the workers.

At this time, in order to collect more reliable information, the company assigns the work result completed by the worker to the inspector to perform the inspection work.

Specifically, when one project is opened, a plurality of tasks are assigned to each of the plurality of workers. Each worker performs a plurality of tasks assigned to it, and provides the task results. Thereafter, a plurality of inspection tasks are assigned to each of the plurality of inspectors, and each inspector performs a plurality of inspection tasks assigned to them.

On the other hand, if there is a rejected work result among the work results submitted by the worker, the worker must perform rework and submit. At this time, if the worker does not perform rework, the task is deleted and assigned to another worker.

In this case, if a task that has been deleted is reassigned to another worker without a separate standard, it is often rejected for the same rejection reason as the rejection reason at the time of initial rejection, which increases the inspection burden of the inspector. It leads to waste.

As described above, when a task for a rejected task result is deleted and reassigned to another worker, there is a need for a method to prevent repeated rejection due to the same rejection reason.

Unexamined Patent Publication No. 10-2014-0095956, 2014.08.04.

The problem to be solved by the present invention is, if the worker who is requested to perform the rework exceeds the rework limit and the assignment of the task is withdrawn, the experience passed by performing the rework of the rejected task result for the same rejection reason This is to provide a method of reassigning the withdrawn work based on the reason for rejection in a crowdsourcing-based project that prevents repeated rejection due to the same rejection reason by selecting other workers who are present and reassigning the withdrawn tasks.

However, the problem to be solved by the present invention is not limited to the problem as described above, and other problems may exist.

In order to solve the above-described problem, the reassignment method of the withdrawn task based on the reason for rejection in the crowdsourcing-based project according to an aspect of the present invention is to perform a plurality of tasks of a crowdsourcing-based project (hereinafter, "project"). Assigning to a worker and requesting to perform a task; Receiving a plurality of job results from the plurality of workers; Assigning the plurality of work results to a plurality of inspectors and requesting to perform inspection; Receiving an inspection pass or rejection as a plurality of inspection results for the plurality of work results from the plurality of inspectors; Reassigning the rejected work to a corresponding worker to request rework; Setting a time limit for limiting rework to a time point when a predetermined period of time has elapsed from the time when the rework performance is requested; And reassigning the assignment to the worker and reassigning the withdrawn task to another worker, if the rejected task requested to perform the rework has not progressed until the rework limit deadline. In this case, the step of reassigning the withdrawn task to another worker may include: selecting a rejected task result based on the plurality of inspection results for the same reason as the revocational task; Selecting another worker who has passed the inspection by performing rework on the result of the rejected work for the same reason; And reassigning the withdrawn task to the selected other worker.

In some embodiments of the present invention, the step of selecting a rejected task result with the same rejection reason as the withdrawn task based on the plurality of inspection results may include rejection reasons for rejected tasks in the plurality of inspection results. Among them, a task result including the keyword for the same reason as the main reason for rejection of the withdrawn task may be selected as the result of the task rejected for the same reason.

In some embodiments of the present invention, the step of receiving a test pass or rejection input from the plurality of inspectors as a plurality of inspection results for the plurality of work results includes a rejection reason keyword including a main rejection reason keyword from the plurality of examiners. Providing a list; And receiving a selection and input of a rejection reason keyword from the rejection reason keyword list and the rejection reason keyword list for the rejected task from the plurality of examiners, wherein the rejection reason keyword list is used to input the rejection reason keyword by the plurality of examiners. Can be updated by

In some embodiments of the present invention, the step of selecting another worker who has passed the inspection by performing the rework of the job result rejected for the same rejection reason is performed by another worker who reworked the rejected operation result due to the same rejection reason. In the case of multiple, any one other worker may be selected based on the inclusion of the main rejection reason keywords included in the rejected work results of the plurality of other workers.

In some embodiments of the present invention, the step of selecting another worker who has passed the inspection by performing the rework of the job result rejected for the same rejection reason is performed by another worker who reworked the rejected operation result due to the same rejection reason. In the case of multiple, one of the plurality of other workers may be selected based on the number of rework for the rejected job result including the main rejection reason keyword.

In some embodiments of the present invention, the step of selecting another worker who has passed the inspection by performing rework on the result of the job rejected for the same rejection reason includes the key rejection reason keyword among the plurality of other workers. Any other worker can be selected based on the number of cases that have passed the inspection by reworking the work results.

In some embodiments of the present invention, the step of reassigning the withdrawn task to another worker may include, if there is no result of the rejected task due to the same reason as the withdrawn task from the plurality of inspection results, the task Calculating work proficiency of the plurality of workers based on results and inspection results; And; It may include the step of reassigning the withdrawal of the assignment to the other worker with the highest job proficiency among the plurality of workers.

In some embodiments of the present invention, the step of reassigning the withdrawn task to another worker may include withdrawing the assignment to the other worker when the reassigned task reassigned to the other worker has not progressed until the rework limit deadline. And, it may further include the step of reassigning the assignment withdrawn work to another worker whose job proficiency is the next priority among the plurality of workers.

In some embodiments of the present invention, calculating the work proficiency of the plurality of workers based on the work result and the inspection result may include an average work execution time based on the work results of the plurality of workers and a rejection rate using the inspection result. The work proficiency may be calculated based on at least one of.

A computer program according to another aspect of the present invention for solving the above-described problems is combined with a computer that is hardware to execute a method of reassignment of the withdrawn task based on the reason for rejection in the crowdsourcing-based project, and computer-readable recording It is stored on the medium.

Other specific details of the present invention are included in the detailed description and drawings.

According to the present invention described above, by canceling rework assignments to workers who have not faithfully performed rework among workers participating in the project and assigning them to new workers, the bottleneck of performing rework is quickly solved and finally Project delays can be avoided.

In particular, when re-assigning work to a new worker, the case of being rejected for the same reason can be reduced by reassigning the work result that has already been rejected for the same reason for rejection to a worker who has passed the inspection after rework. It can minimize the inspection fatigue of the inspector and at the same time further prevent the schedule delay of the project.

The effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the following description.

1 is a conceptual diagram of a crowdsourcing service according to an embodiment of the present invention.
2 is a flowchart illustrating a crowdsourcing-based project progress process according to an embodiment of the present invention.
3 is a flowchart of a method for reassigning a task that has been withdrawn based on a rejection reason in a crowdsourcing-based project according to an embodiment of the present invention.
4 is an exemplary diagram for explaining input of a companion reason and a companion reason keyword.
5 is an exemplary diagram for explaining a companion reason data pool created in a project.
6 is an exemplary diagram for explaining a companion reason keyword and main companion reason keywords.
7 is a view for explaining a limit period for rework in an embodiment of the present invention.
8 is a flow chart illustrating a process of reassigning a task that has been withdrawn to another worker in an embodiment of the present invention.
9 is a diagram for explaining an example of reassigning a task that has been withdrawn to another worker.
10 is a view for explaining a work reassignment apparatus according to an embodiment of the present invention.

Advantages and features of the present invention, and a method of achieving them will become apparent with reference to the embodiments described below in detail together with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but may be implemented in a variety of different forms, only the present embodiments are intended to complete the disclosure of the present invention, It is provided to fully inform the technician of the scope of the present invention, and the present invention is only defined by the scope of the claims.

The terms used in the present specification are for describing exemplary embodiments and are not intended to limit the present invention. In this specification, the singular form also includes the plural form unless specifically stated in the phrase. As used in the specification, “comprises” and/or “comprising” do not exclude the presence or addition of one or more other elements other than the mentioned elements. Throughout the specification, the same reference numerals refer to the same elements, and “and/or” includes each and all combinations of one or more of the mentioned elements. Although "first", "second", and the like are used to describe various elements, it goes without saying that these elements are not limited by these terms. These terms are only used to distinguish one component from another component. Therefore, it goes without saying that the first component mentioned below may be the second component within the technical idea of the present invention.

Unless otherwise defined, all terms (including technical and scientific terms) used in the present specification may be used as meanings that can be commonly understood by those of ordinary skill in the art to which the present invention belongs. In addition, terms defined in a commonly used dictionary are not interpreted ideally or excessively unless explicitly defined specifically.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a conceptual diagram of a crowdsourcing service according to an embodiment of the present invention.

Referring to FIG. 1, a crowdsourcing service is performed by being composed of a client 10, a service provider 20, and a public 30.

The sponsor 10 refers to a company or individual who requests a crowdsourcing-based project (hereinafter, referred to as a project).

The requester 10 requests a project for the purpose of collecting source data or annotating data for generating artificial intelligence learning data. The data generated through the project can be used as learning data for arbitrary machine learning such as supervised learning, unsupervised learning, and reinforcement learning. Collection of source data refers to collecting raw data such as recorded voice collection and photo collection. Data annotation refers to inputting relevant annotation data into source data such as text, photos, and videos. For example, the data annotation may include finding an entity in a given fingerprint or finding a similar sentence, but is not limited thereto. Meanwhile, the types of the above-described projects are only one embodiment, and various projects may be handled in the present invention according to the design of the client.

The service provider 20 refers to a company that provides crowdsourcing services.

When the service provider 20 receives a request for a project for a product or service from the client 10, the service provider 20 allocates a task for the project to the general public 30 and receives the work result from the public 30. Thereafter, the final product extracted based on the work result is provided to the client 10.

At this time, the service provider 20 provides a crowdsourcing service to the client 10 and the public 30 through a crowdsourcing platform (hereinafter, the platform). That is, when the service provider 20 receives a request for a project from the client 10, the service provider 20 opens the project on the platform. Thereafter, when the work result for the open project is received from the public 30, the project is terminated on the platform, and the final product may be extracted and provided to the client 10.

The public 30 refers to the general public participating in the project open on the platform. Here, the public 30 may participate in a project opened on the platform through an application or website provided by the service provider 20.

The public 30 is composed of a worker 32 and an inspector 34.

The worker 32 decides to participate in a specific project among a plurality of projects open on the platform. Thereafter, the operator 32 performs an operation such as collection of source data or data annotation, and transmits it to the platform.

The inspector 34 decides to participate in a specific project among a plurality of projects open on the platform. Thereafter, the inspector 34 inspects the result of the work performed by the operator 32. As a result of performing the inspection, the inspector 34 may perform inspection pass processing or rejection processing, and input a rejection reason during rejection processing. In the case of passing the inspection, it is not necessary to rework and re-examination due to this, so passing inspection has the same meaning as completion of inspection.

2 is a flowchart illustrating a crowdsourcing-based project progress process according to an embodiment of the present invention.

First, the requester 10 requests one or more projects to the service provider 20 (S11).

Thereafter, the service provider 20 opens the requested project on the platform (S12). At this time, the service provider 20 may determine the grade in consideration of the difficulty of the project before opening the project. That is, depending on the difficulty level, it is possible to determine which level or higher public 30 to expose the project. Accordingly, it is possible to increase the reliability of the work result of the project.

Thereafter, the service provider 20 requests the work by assigning the work to the workers 32 of the corresponding level or higher according to the level of the project (S13).

Thereafter, the worker 32 performs the assigned task (S14). In this case, the worker 32 may input the reason for the inability to work without performing the work for a work in which the work itself is impossible for some reason.

Thereafter, the service provider 20 receives the work result from the worker 32 (S15), and assigns the work result to the inspector 34 to request the inspection (S16).

Similarly, according to an embodiment of the present invention, only a suitable project may be exposed to the inspector 34 out of the total projects being performed according to the level set according to the difficulty of the project or the qualification requirements of the inspector 34.

Thereafter, the inspector 34 performs the assigned inspection (S17). At this time, the inspector 34 decides to complete the inspection when it is determined that the work has been properly performed, and if it is determined that the inspection work is wrong, it rejects the inspection. During rejection processing, the inspector 34 inputs the rejection reason for what reason it is determined that the job is wrong.

Thereafter, the service provider 20 receives the inspection result from the inspector 34 (S18).

When the inspection result is the inspection completion, the service provider 20 uses the work result as valid data, and based on this, extracts the final product at the end of the project.

When the inspection result is rejection processing, the service provider 20 may internally perform the inspection again, or assign the work to the worker 32 again to perform the rework. When reworking, re-examination by an inspector is required.

Thereafter, when the project period ends or sufficient valid data is secured, the service provider 20 terminates the project (S19), calculates a final result based on the secured valid data, and provides it to the client 10 ( S20).

At this time, before the end of the project, the service provider 20 evaluates the performance results of the worker 32 and the inspector 34, calculates the work cost and the inspection cost according to the evaluation, to the worker 32 and the inspector 34. give.

In FIGS. 1 and 2, it is simply expressed as a requester 10, a service provider 20, an operator 32, and an inspector 34, but these are smartphones, tablets, PDAs, laptops, desktops, operated by each participant. It means a computer device such as a server or a telecommunication device.

Hereinafter, a method of reassigning a task that has been withdrawn based on a rejection reason in a crowdsourcing-based project according to an embodiment of the present invention will be described with reference to FIGS. 3 to 9.

3 is a flowchart of a method for reassigning a task that has been withdrawn based on a rejection reason in a crowdsourcing-based project according to an embodiment of the present invention.

Meanwhile, the steps shown in FIG. 3 may be understood as being performed by a platform server (hereinafter, referred to as a server) operated by the service provider 20, but are not limited thereto.

In addition, a plurality of workers 32 or a plurality of inspectors 34 perform work using a predetermined terminal device. The terminal device of the operator 32 or the inspector 34 may be a computer device or a telecommunication device such as a smartphone, tablet, PDA, laptop, desktop, etc., but is not limited thereto.

First, referring to FIG. 3, the server assigns a plurality of tasks of a crowdsourcing-based project (hereinafter referred to as a project) to a plurality of workers 32 and requests the task to be performed (S110), and a plurality of workers 32 A plurality of work results are inputted from (S120).

Here, one project may contain multiple jobs, and the plurality of jobs may be grouped into a plurality of tasks according to their type, or each job may be a different type of job or the same job. , But is not limited thereto.

Next, the server assigns the plurality of work results to the plurality of inspectors 34 to request the inspection performance (S130), and passes or rejects the plurality of inspection results for the plurality of job results from the plurality of inspectors 34 It is input as (S140).

In one embodiment, the server provides a list of reasons for refusal including keywords for main reasons for refusal to a plurality of inspectors 34, and for tasks that failed to pass the examination to the plurality of inspectors 34, together with the reason for rejection. , Enter the keyword for the reason for the reaction from the keyword list for the reason for reaction

In the case of reassigning the task to another worker after the assignment of the task to the worker 32 who has exceeded the rework limit in the future, it is determined whether it corresponds to the result of the rejected task for the same reason for rejection. It is used to do so, and the withdrawn task is assigned to another worker who is judged to have the rejected task result for the same reason.

4 is an exemplary diagram for explaining input of a companion reason and a companion reason keyword.

Referring to FIG. 4, the server provides a list of keywords for a friend reason and a window for inputting a reason for the reaction to the examiner 34. The examiner 34 selects one major companion reason keyword from the provided companion reason keyword list, and inputs a specific companion reason.

For example, when the companion reason to be entered is included in the companion reason keyword list, the examiner 34 selects a main companion reason keyword related to the companion reason to be input from the companion reason keyword list. In addition, the reviewer 34 inputs the reason for rejection of the content including or related to the selected reason for rejection keyword.

As an example, when the examiner 34 selects "noise" from the list of reasons for rejection keywords, the reason for rejection is inputted. For example, the inspector 34 may input "Noise is heard" so that the keyword (noise) of the reason for the selection is included. Alternatively, you can enter "I hear a different sound" so that the keyword (noise) of the reason you choose is related.

As another example, if the reason for input to be entered in the keyword list for the reason for rejection is not included, the examiner 34 selects "direct input" at the bottom of the keyword list for the reason for rejection, as shown in FIG. In the window, a separate reason for reply can be entered as a sentence.

In this way, the server stores the rejection reason and the rejection reason keyword list inputted from the examiner 34 in the rejection reason data pool. At this time, the reply reason data pool is continuously updated every time a reply reason and keywords are inputted from the plurality of inspectors 34.

Accordingly, the companion reason keyword list may also be updated in real time or periodically by input of the companion reason keyword by the plurality of examiners 34.

5 is an exemplary diagram for explaining a companion reason data pool created in a project. 6 is an exemplary diagram for explaining a companion reason keyword and main companion reason keywords.

In an embodiment, a list of reasons for a friendliness keyword list including keywords for main reasons for a previous project may be provided as a default at the beginning of the project. In this case, the examiner 34 selects one main reason for the rejection keyword from the list of reasons for rejection and inputs the reason for rejection. At this time, the server may generate a list of keywords for the reason for rejection of the project by using only the reason for rejection.

Thereafter, the server collectively collects the rejection reason input from the plurality of inspectors 34, and the collected rejection reason is stored in the companion reason data pool generated in the server. At this time, the rejection reason may be recorded one row after each job ID is assigned as shown in FIG. 5 and stored in the rejection reason data pool.

And as a certain time elapses, when the server determines that the reason for rejection has been sufficiently collected, the server creates a keyword list for the reason for rejection.

Then, the server extracts a plurality of keywords for the reason for the companionship from the collected reason for the companionship. As an example, the server may extract a word, which is a noun, from a companion reason composed of sentences.

For example, the server may extract the noun "woolim" as a companion reason keyword from the companion reason in the sentence "There is an echo" in FIG. 5. In addition, the server can extract the noun "noise" from the friend reason in the sentence "Noise is mixed" as the friend reason keyword.

Subsequently, the server selects a major companion reason keyword from among the extracted plurality of companion reason keywords.

For example, the server may calculate the appearance frequency of each word extracted as the companion reason keyword, and select a word within a predetermined ranking based on the appearance frequency as the main job impossibility reason keyword.

For example, referring to FIG. 6, words such as ringing, noise, earphones, awkwardness, and blurring from the entire companion reason are extracted as companion reason keywords, and are ranked according to the frequency of appearance of each word and listed. In this case, the server may select a word within a certain ranking (for example, 3 ranking), that is, ringing, noise, and earphone as the main reason for the companion keyword.

In this case, a predetermined ranking selected as a keyword for the main reason for the companion may be set in advance according to the target reliability required by the project. That is, the server may adjust the number of words selected as the main reasoning keyword according to the level requested by the client 10 or the level required according to the nature of the project.

Thereafter, the server provides the reviewer 34 with a list of reasoning keywords including the selected main reasoning keyword. The examiner 34 may select one companion reason keyword using the provided companion reason keyword list, and input the companion reason using the selected companion reason keyword.

After the companion reason keyword list is generated and initially provided to the examiner 34, the companion reason keyword list may be updated every predetermined period or in real time.

In addition, the server collects in real time the companion reason keywords and the companion reason inputted from the plurality of inspectors 34 in the companion reason data pool. In addition, the server updates the companion reason keyword list by aggregating the companion reason keywords collected in real time in the companion reason data pool every predetermined time or in real time.

In this case, the server allows the examiner 34 to directly input the reason for the refusal together with the keyword for the reason, or not select the keyword for the reason for the response.

In other words, if only the keyword for the reason for the companion is selectively input, the server can no longer extract the keyword for the reason for the companion. In other words, since only the existing keywords for the selected main reason for the reaction are continuously selected, the reason for the new content cannot be entered.

Therefore, the server allows the examiner 34 to select one companion reason keyword from the companion reason keyword list, and inputs the companion reason related thereto in the form of a sentence. Or, let them directly enter the reason for the response without selecting the keyword for the reason.

Accordingly, a new companion reason keyword is extracted so that the main companion reason keyword is updated in real time, and the examiner 34 is also provided with a companion reason keyword list updated at a predetermined time or in real time to select and input the reason for the response.

For example, referring to FIG. 6, as the companion reason keyword and the companion reason are input in real time, “noise”, which was the second priority, may be updated to the first priority. In addition, a new keyword for reason of reaction, called "shake", may be extracted and ranked in the rankings.

Referring back to FIG. 3, the server reassigns the rejected work to the corresponding worker 32 to request rework (S150), and rework it to a time when a predetermined period has elapsed from the time when rework is requested. A work limit period is set (S160).

Then, the server withdraws the task assigned to the worker 32 when the rejected task requested to perform the rework is not progressed until the rework limit deadline, and reassigns the reworked task to another worker (S170) ).

7 is a view for explaining a limit period for rework in an embodiment of the present invention. 8 is a flow chart illustrating a process of reassigning a task that has been withdrawn to another worker in an embodiment of the present invention. 9 is a diagram for explaining an example of reassigning a task that has been withdrawn to another worker.

After the inspection result is inputted by the inspector 34, the server requests the corresponding worker 32 for a corresponding rework for the rejected job. In this case, if a number of rework is assigned while the worker 32 does not perform the rework quickly, a bottleneck for performing the rework occurs from the worker 32, which leads to a delay in the entire project. .

In addition, when a number of rework is assigned, if the worker 32 is not guaranteed a sufficient time for performing rework for each rework, it may cause the worker 32 to deviate from the rework while performing the rework. Occurs.

In order to solve this problem, an embodiment of the present invention may be operated by setting a rework limit period for the rejected work result.

Specifically, the rework is assigned to the worker 32 on the condition that the rework begins to be performed within the rework limit period. If the worker 32 does not start the rework assigned by the rework limit deadline, the server cancels the rework assignment to the worker 32, assigns the rework to another worker, and requests the rework to be performed. .

This gives a penalty for canceling the rework assignment to the worker 32 who has not started rework, and quickly assigns the rework to a new worker, solving the bottleneck caused by delay in rework, and finally, the entire project. This is to prevent delays in schedule.

Referring to the example of FIG. 7, the worker 32 was assigned the Nth (N is a natural number) rework on July 13, 2020 at 09:00 and the N+1th rework at 10:00. This is because they were rejected as a result of the inspection by the inspector 34.

At this time, when the limit time for rework is set to 6 hours, the worker 32 must start and complete the Nth rework by 15:00 and the N+1th rework by 16:00, respectively. If the worker 32 does not start performing the N-th rework by 15:00, the N-th rework of the worker 32 is canceled and assigned to another worker and requested to perform the rework. In this case, P1 in FIG. 7 denotes a rework limit for the N-th rework, and P2 denotes a rework limit for the N+1-th rework.

Meanwhile, in the example of FIG. 7, the limit for rework set to a predetermined time (6 hours) may be set according to the difficulty of the project, the remaining period of the project, and the number of workers who participated in the project.

For example, for a project with relatively high difficulty, a predetermined time may be set to 12 hours, and for a project with relatively low difficulty, the predetermined time may be set to 6 hours.

In addition, when the remaining period of the project remains relatively large, a predetermined time may be set shorter than when the remaining period of the project remains relatively small, and in the case of a project with a large number of participating workers 32, the participating workers ( A predetermined time can be set shorter than in the case of a project with a small number of 32).

When the limit for rework is set in this way, the worker 32 must start within the limit for each rework. In this process, an embodiment of the present invention allows the worker 32 to reach the Nth limit for rework. If the N-th rework is initiated and the N-th rework is being performed, even if the worker 32 does not start performing the N+Mth rework by the N+Mth (M is a natural number) rework limit, N+M The assignment of the second rework to the worker 32 is not canceled.

For example, there may be a case where the time interval between the Nth rework limit and the N+Mth rework limit is not large, that is, rework is concentrated. This is the case where the work result of one worker 32 is inspected and rejected by several inspectors 34. In this case, canceling the allocation for the N+Mth rework because the N+Mth rework limit has expired during the Nth rework is an excessive and excessive penalty.

Therefore, when the worker 32 begins to perform the Nth rework until the Nth rework limit (P1) and is performing the Nth rework, each rework is performed until the N+Mth rework limit following this. Even if you don't start on, the corresponding rework assignment is not cancelled.

As an example, if the worker 32 currently needs to perform the 10th to 15th rework, the worker 32 starts to perform the 10th rework until the 10th rework limit deadline and is performing the rework, Even if the worker's 11th to 15th rework performance cannot be initiated until the corresponding rework limit deadline, respectively, due to the 10th rework performance, the penalty for rework assignment to the worker 32 is not given.

The penalty for the above-described worker 32 is a sanctioning means for the worker 32 delaying the completion of the project by not commencing rework execution until the rework limit deadline, causing a bottleneck for rework, and Because it is not.

In addition, in the process of faithfully performing rework, if a penalty is given until the limit for the next rework assigned to the worker 32 has expired, the ongoing rework will be hastily and the quality of the rework may be degraded. Because there is.

If the assignment of the rework is withdrawn because the work of the worker 32 is not performed during the rework limit period set as described above, the work is reassigned to another worker.

At this time, one embodiment of the present invention is a task that is rejected for the same reason as a task that has been withdrawn based on a plurality of inspection results in order to minimize the rate at which the rework result of another worker who has reassigned a task that has been withdrawn The result is selected (S171).

Specifically, the server selects the job results that contain the same keyword for the same reason as the main reason for the rejection of the rejected task from among the rejection reasons for the rejected task in the plurality of inspection results, and rejects the same. It can be selected as a result of the work rejected for reasons.

For example, as shown in (a) of Fig. 9, the work result of worker B including the keyword'woolim' and the keyword'woolim' that is the same as the main reason for rejection of the job withdrawn by worker A is the same reason for rejection It can be selected as the result of the returned work.

Next, the server performs rework on the job result returned for the same rejection reason, selects another worker that has passed the inspection (S173), and reassigns the job that has been withdrawn to the selected other worker (S175).

If there are multiple other workers who have reworked the rejected job results for the same rejection reason, the server will be based on the inclusion of the main rejection reason keywords included in the rejected job results of the multiple other workers. Can be selected.

For example, as shown in (b) of FIG. 9, the result of a job rejected with the same reason for rejection including the keyword'woolim' and the keyword'woolim' that is the main reason for rejection of the job withdrawn by worker A There may be multiple reworked workers (workers B, C).

At this time, worker B's reason for rejection includes the main reason keyword'woolim' twice, and worker C's reason for rejection includes the key reason keyword'woolim' 3 times. Among the reasons for refusal, a worker C that has passed the inspection can be selected by reworking the reason for the refusal that includes more of the keyword'woolim' as the main reason for refusing.

Along with this or separately, the server is based on the number of reworking of the rejected job results containing the main rejection reason keyword among the plurality of other workers when there are multiple other workers who reworked the rejected job results for the same rejection reason. Thus, any one other worker can be selected.

For example, there are multiple workers who reworked the result of the rejected task with the same rejection reason, which includes the keyword'woolim' and the keyword'woolim', which is the same as the keyword'woolim' for the rejection reason for the job withdrawn by worker A. There may be (workers B, C).

At this time, worker B reworked the rejected work results with a rejection reason including the main rejection reason keyword'woolim', and worker C has a rejection reason including the main rejection reason keyword'woolim'. If the number of reworked results of the rejected work is 5, the server may select worker C with a larger number of reworked work results for the same reason.

Furthermore, the server may select other workers by performing rework on the rejected work result including the main rejection reason keyword among the plurality of other workers, further considering the number of cases that have passed the inspection.

That is, even if the number of reworking of worker C in the above example is greater, if the number of reworking and passing inspection is greater than that of worker B, the server may select worker B.

On the other hand, due to reasons such as the initial stage of the project or the reason that the main reason keyword has not been extracted, among the plurality of inspection results, there may not be a result of the rejected task with the same reason as the withdrawn task.

In this case, according to an embodiment of the present invention, a task that has been withdrawn from assignment may be reassigned by selecting another worker based on work proficiency.

That is, the server may calculate job proficiency for the plurality of workers 32 based on the job result and the inspection result, select another worker with the highest job proficiency, and reassign the job withdrawn.

In one embodiment, the work proficiency for the plurality of workers 32 may be calculated based on the average work execution time based on the work results of the plurality of workers 32.

The average work execution time means the average of the time the worker 32 performs work or rework. Specifically, the average work execution time is the average of the time between each of the times after the start of work for a plurality of tasks assigned to the worker 32 and the time at which work results are entered after completing work are measured. It can be calculated through

In this case, the average task execution time may be calculated by considering only the execution time related to rework among a plurality of tasks assigned to the worker 32. That is, since the present invention is to perform re-assignment of the reworked task, only the execution time related to the rework can be considered when calculating the average task execution time.

In addition, work proficiency for the plurality of workers 32 may be calculated based on the rejection rate using the inspection results of the plurality of workers 32.

When the server receives a plurality of inspection results for a plurality of work results or rework results from a plurality of inspectors 34 as inspection pass or rejection, it can calculate the rejection rate of each worker 32 based on the result. have.

When the average work execution time and the rejection rate are calculated as described above, they are sorted by ranking, so that other workers with the shortest average work execution time and the lowest rejection rate can be selected.

In general, a task that has been withdrawn because the limit for rework has been exceeded is a case where the worker (32) initially performed the task and gave up the task despite the fact that if the reason for rejection is resolved, work costs may be received. In most cases. In addition, this work corresponds to a work that is delaying the schedule of the project, which has exceeded the limit of rework and has to be resumed by another worker.

Accordingly, an embodiment of the present invention selects and assigns the worker with the lowest rejection rate and the shortest average time to perform the task as the worker who has the highest possibility of completing the task of high difficulty and can perform the rework the fastest You can reassign the withdrawn work

If the job reassigned to the first priority other worker selected according to the job proficiency in consideration of the average job execution time and rejection rate is not progressed by the rework limit again, the server withdraws the assignment to the other worker. In addition, it is possible to reassign tasks that have been withdrawn from assignment to other workers whose job proficiency is the next highest among multiple workers.

Meanwhile, in the above description, steps S11 to S173 may be further divided into additional steps or may be combined into fewer steps, according to an embodiment of the present invention. In addition, some steps may be omitted as necessary, and the order between steps may be changed. In addition, even if other contents are omitted, the contents of FIG. 10 to be described later may also be applied to a method of reassigning a task that has been withdrawn based on the reason for rejection in the crowdsourcing-based project of FIGS. 1 to 9.

Hereinafter, a description will be given of a reassignment device 200 (hereinafter, a task reassignment device) of a task that has been withdrawn based on a reason for rejection in a crowdsourcing-based project according to an embodiment of the present invention.

10 is a view for explaining a job reassignment apparatus 200 according to an embodiment of the present invention.

Referring to FIG. 10, a task reassignment apparatus 200 according to an embodiment of the present invention includes a communication module 210, a memory 220, and a processor 230.

The communication module 210 transmits a crowdsourcing-based work for one project to a plurality of workers 32 to request a task to be performed, and receives a work result from the plurality of workers 32. In addition, an estimated rework time for the rejected work result is provided to the inspector 34, and a selection input of whether to abandon the inspection of the rejected work result is received from the inspector 34.

In the memory 220, a program for reassigning a task that has been withdrawn after a rework limit has elapsed is stored to another worker.

The processor 230 executes a program stored in the memory 220. As the processor 230 executes the program stored in the memory 220, the rejected task is reassigned to the corresponding worker 32 to request rework, and the rejected task requested to perform the rework is reworked. If it is not progressed by the limit, the task assigned to the worker 32 is withdrawn, and the task that has been withdrawn is reassigned to another worker.

At this time, the processor 230 selects the rejected task result for the same rejection reason as the withdrawn task based on the plurality of inspection results, and performs rework of the rejected task result for the same rejection reason, and passes the inspection. After selecting, reassign tasks that have been withdrawn to other selected workers.

The job reassignment apparatus 200 described with reference to FIG. 10 may be provided as a component of the above-described server.

In the crowdsourcing-based project according to an embodiment of the present invention described above, the reassignment method of the withdrawn task based on the reason for rejection is implemented as a program (or application) to be executed in combination with a computer that is hardware. Can be saved.

The above-described program includes C, C++, JAVA, Ruby, which can be read by a processor (CPU) of the computer through the device interface of the computer, in order for the computer to read the program and execute the methods implemented as a program. It may include a code (Code) coded in a computer language such as machine language. Such code may include a functional code related to a function defining necessary functions for executing the methods, and a control code related to an execution procedure necessary for the processor of the computer to execute the functions according to a predetermined procedure. can do. In addition, such code may further include additional information required for the processor of the computer to execute the functions or code related to a memory reference to which location (address address) of the internal or external memory of the computer should be referenced. have. In addition, when the processor of the computer needs to communicate with any other computer or server in the remote in order to execute the functions, the code uses the communication module of the computer to determine how It may further include a communication-related code for whether to communicate or what information or media to transmit and receive during communication.

The stored medium is not a medium that stores data for a short moment, such as a register, cache, memory, etc., but a medium that stores data semi-permanently and can be read by a device. Specifically, examples of the storage medium include, but are not limited to, ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, and the like. That is, the program may be stored in various recording media on various servers to which the computer can access, or on various recording media on the user's computer. In addition, the medium may be distributed over a computer system connected through a network, and computer-readable codes may be stored in a distributed manner.

The above description of the present invention is for illustrative purposes only, and those of ordinary skill in the art to which the present invention pertains will be able to understand that it can be easily modified into other specific forms without changing the technical spirit or essential features of the present invention. will be. Therefore, it should be understood that the embodiments described above are illustrative in all respects and not limiting. For example, each component described as a single type may be implemented in a distributed manner, and similarly, components described as being distributed may also be implemented in a combined form.

The scope of the present invention is indicated by the claims to be described later rather than the detailed description, and all changes or modified forms derived from the meaning and scope of the claims and their equivalent concepts should be interpreted as being included in the scope of the present invention. do.

10: Client
20: service provider
30: public
32: worker
34: inspector
200: work reassignment device
210: communication module
220: memory
230: processor

Claims (10)

As a method performed by a computer,
Allocating a plurality of tasks of a crowdsourcing-based project (hereinafter, “project”) to a plurality of workers and requesting the task to be performed;
Receiving a plurality of work results from the plurality of workers;
Assigning the plurality of work results to a plurality of inspectors and requesting to perform inspection;
Receiving an inspection pass or rejection as a plurality of inspection results for the plurality of work results from the plurality of inspectors;
Reassigning the rejected work to a corresponding worker to request rework;
Setting a time limit for limiting rework to a time point when a predetermined period of time has elapsed from the time when the rework performance is requested; And
If the rejected task requested to perform the rework has not progressed until the rework limit deadline, withdrawing the assignment to the worker, and reassigning the withdrawn task to another worker,
The step of reassigning the withdrawn task to another worker,
Selecting a rejected task result for the same rejection reason as the withdrawn task based on the plurality of inspection results;
Selecting another worker who has passed the inspection by performing rework on the result of the rejected work for the same reason; And
Including the step of reassigning the assignment withdrawn work to the selected other worker,
The step of selecting a rejected task result for the same rejection reason as the withdrawn task based on the plurality of inspection results,
Among the rejection reasons for the rejected task in the plurality of inspection results, a task result containing the same rejection reason keyword as the main rejection reason keyword of the rejection reason of the withdrawn task is returned as the same rejection reason. Is to be selected as,
The step of receiving an inspection pass or rejection as a plurality of inspection results for the plurality of work results from the plurality of inspectors,
Providing a list of reasons for refusal including keywords for main reasons for refusing to the plurality of inspectors; And
Including the step of selectively inputting the rejection reason keyword from the rejection reason keyword list and the rejection reason for the rejected task from the plurality of inspectors,
Wherein the list of reasons for the refusal is updated by at least one of the reason for the refusal input by the plurality of examiners and the reason for the keyword input,
A method of reassigning deassigned tasks based on reasons for rejection in crowdsourced projects.
delete delete The method of claim 1,
The step of selecting another worker who has passed the inspection by performing rework of the result of the rejected operation for the same reason for rejection,
When there are a plurality of other workers who rework the rejected work results for the same rejection reason, one other worker is selected based on the inclusion of the main rejection reason keywords included in the rejected work results of the plurality of other workers. To select,
A method of reassigning deassigned tasks based on the reason for rejection in crowdsourcing-based projects.
The method of claim 1,
The step of selecting another worker who has passed the inspection by performing rework of the result of the rejected operation for the same reason for rejection,
If there are multiple other workers who have reworked the rejected work result for the same rejection reason, one of the plurality of other workers is based on the number of reworking of the rejected operation result including the main rejection reason keyword. To select a worker,
A method of reassigning deassigned tasks based on the reason for rejection in crowdsourcing-based projects.
The method of claim 5,
The step of selecting another worker who has passed the inspection by performing rework of the result of the rejected operation for the same reason for rejection,
To select any one of the other workers based on the number of inspections passed by performing rework of the rejected job result including the main rejection reason keyword among the plurality of other workers,
A method of reassigning deassigned tasks based on the reason for rejection in crowdsourcing-based projects.
The method of claim 1,
The step of reassigning the withdrawn task to another worker,
In the case that there is no result of the rejected task due to the same rejection reason as the withdrawn task from the plurality of inspection results,
Calculating work proficiency of the plurality of workers based on the work result and the inspection result; And;
Including the step of reassigning the assignment withdrawn work to the other worker with the highest work proficiency among the plurality of workers,
A method of reassigning deassigned tasks based on the reason for rejection in crowdsourcing-based projects.
The method of claim 7,
The step of reassigning the withdrawn task to another worker,
If the reassigned job reassigned to the other worker has not progressed until the rework limit deadline, the assignment to the other worker is withdrawn, and the assignment withdrawn to another worker whose job proficiency is the next highest among the plurality of workers. Further comprising the step of reassigning,
A method of reassigning deassigned tasks based on the reason for rejection in crowdsourcing-based projects.
The method of claim 7,
Calculating the work proficiency of the plurality of workers based on the work result and the inspection result,
To calculate the job proficiency based on at least one of an average job execution time based on the job results of the plurality of workers and a rejection rate using the inspection result,
A method of reassigning deassigned tasks based on the reason for rejection in crowdsourcing-based projects.
A computer combined with a computer and stored on a computer-readable recording medium to execute the reassignment method of the withdrawn task based on the reason for rejection in the crowdsourcing-based project of any one of claims 1 and 4 to 9. program.

Images