KR102156582B1 - Method for selecting sincere worker and prioritizing inspection of crowdsourcing based projects for artificial intelligence training data generation - Google Patents

Abstract

Provided is a method of selecting sincere workers and prioritizing inspection of crowd sourcing-based projects for artificial intelligence (AI) training data generation. The method comprises the steps of: allocating a plurality of works of crowd sourcing-based projects to a plurality of workers and requesting the workers to perform the work; receiving an input of a plurality of work results from the plurality of workers; allocating the plurality of work results to a plurality of inspectors and requesting the inspectors to perform the inspection; receiving an input of the plurality of inspection results on the plurality of work results from the plurality of inspectors as Pass or Fail; evaluating the work sincerity of each worker based on the point of time related to the work execution and the inspection execution; and granting the high priority for inspection to the sincere workers whose work sincerity is a certain first reference value or higher among the plurality of workers. The method of selecting sincere workers and prioritizing inspection of crowd sourcing-based projects for AI training data generation is able to minimize the unnecessary time spent and reduce the period of performing the whole projects.

Description

{METHOD FOR SELECTING SINCERE WORKER AND PRIORITIZING INSPECTION OF CROWDSOURCING BASED PROJECTS FOR ARTIFICIAL INTELLIGENCE TRAINING DATA GENERATION}

The present invention relates to a method for selecting and prioritizing inspection and selection of conscientious workers for crowdsourcing-based projects for generating artificial intelligence learning data.

Recently, more and more companies collect and process large amounts of data on a crowdsourcing basis that engages the general public in some process of corporate activities. In other words, by opening a project and allowing the general public, that is, workers to participate in the project, necessary information is collected through the work results completed by the workers.

At this time, in order to collect more reliable information, the company assigns the work result completed by the worker to the inspector to perform the inspection work.

Specifically, when one project is opened, a plurality of tasks are assigned to each of the plurality of workers. Each worker performs a plurality of tasks assigned to it, and provides the task results. Thereafter, a plurality of inspection tasks are assigned to each of the plurality of inspectors, and each inspector performs a plurality of inspection tasks assigned to them.

On the other hand, in the progress of the project, if the work is rejected by the inspector, the worker must perform rework and resubmit it to receive re-review by the inspector. However, if an unscrupulous inspector is assigned to a worker who wants to work sincerely, the work that can be completed soon cannot be completed and the worker's motivation is lowered.

Likewise, if an unscrupulous worker is assigned to an inspector who wants to perform the inspection sincerely, the motivation of the inspector is also broken because rework is not submitted even though a reason for rejection occurred and the inspection was rejected.

This eventually leads to a problem of income of workers and inspectors, which leads to the departure of the project by conscientious workers and conscientious inspectors, resulting in a problem of lowering the progress and quality of the entire project.

Unexamined Patent Publication No. 10-2014-0095956, 2014.08.04.

The problem to be solved by the present invention is to evaluate the reliability of the operator and the inspector based on the time associated with the operator or inspector's work performance and inspection performance, and classify it as a conscientious operator and a conscientious inspector to give incentives. It is to provide a method of selecting and prioritizing inspection and selection of conscientious workers for crowdsourcing-based projects that can minimize delay and shorten the duration of the entire project.

However, the problem to be solved by the present invention is not limited to the problem as described above, and other problems may exist.

According to an aspect of the present invention for solving the above-described problems, a method for selecting and prioritizing inspection of sincere workers for a crowdsourcing-based project is a method of assigning a plurality of tasks of a crowdsourcing-based project to a plurality of workers and requesting the performance of the task. step; Receiving a plurality of job results from the plurality of workers; Assigning the plurality of work results to a plurality of inspectors and requesting to perform inspection; Receiving a plurality of inspection results for the plurality of work results from the plurality of inspectors as passing or rejecting the inspection; Evaluating the work integrity of each worker based on the time associated with the work performance and the inspection performance; And giving inspection priority to a sincerity worker whose work integrity is equal to or greater than a predetermined first reference value among the plurality of workers.

In some embodiments of the present invention, the step of evaluating the work integrity of each worker comprises: measuring the first time at which the inspection result is input by completing the inspection for each job result assigned to each inspector. Step to do; Measuring a second time of access to an online workspace for performing rework for each task assigned to each worker and inputted as a rejection of the inspection result; Measuring a work reaction time associated with each rework case based on the first time and the second time; And it may include the step of evaluating the work integrity of each worker based on the work reaction time.

In some embodiments of the present invention, in the step of evaluating the work integrity of each worker, the work response times associated with each rework case may be summed and averaged to evaluate the work integrity of each worker.

In some embodiments of the present invention, the step of assigning the inspection priority to the sincerity worker includes sequentially determining the inspection order of the plurality of work results according to the order in which the plurality of work results are input, It is possible to advance the inspection order of one or more work results by the sincere worker than one or more work results.

Some embodiments of the present invention include the steps of evaluating the inspection integrity of each inspector based on the time associated with performing the work and performing the inspection; And requesting to perform the inspection by assigning the work result of the sincerity worker to a sincerity inspector whose inspection integrity is equal to or greater than a predetermined second reference value among the plurality of inspectors.

In some embodiments of the present invention, the step of evaluating the inspection integrity of each inspector comprises: measuring a third time at which the task result is input by completing the task execution for each task assigned to each worker. step; Measuring a fourth time of access to the online inspection space for performing the inspection for each task assigned to each inspector; Measuring a first inspection reaction time associated with each inspection case based on the third time and the fourth time; And evaluating the inspection integrity of each inspector based on the first inspection reaction time.

In some embodiments of the present invention, in the step of evaluating the inspection integrity of each inspector, the rework result is completed by completing the rework for each job assigned to each worker and the inspection result input as a rejection. Measuring the input fifth time; Measuring a sixth time of access to the online inspection space for performing re-review for each task assigned to each inspector and the inspection result is input as a rejection; Measuring a second inspection reaction time associated with each re-examination case based on the fifth time and the fifth time; And evaluating the inspection integrity of each inspector further based on the second inspection reaction time.

In some embodiments of the present invention, in the step of evaluating the inspection integrity of each inspector, the inspection integrity of each inspector may be evaluated by measuring and averaging the first inspection reaction time associated with each inspection case. .

In some embodiments of the present invention, the step of evaluating the work integrity of each worker includes determining the sincerity of each worker at a point in time when the number of tasks that each worker has completed performing the task is more than a predetermined minimum number. Can be evaluated.

A computer program according to another aspect of the present invention for solving the above-described problems is combined with a computer that is hardware to execute a method of selecting and prioritizing the integrity workers of the crowdsourcing-based project, and stored in a computer-readable recording medium. do.

Other specific details of the present invention are included in the detailed description and drawings.

According to the present invention described above, the operator who faithfully performs the work and the inspector who faithfully performs the inspection are matched with each other, so that the work is completed quickly, so that it is possible to prevent departure of the sincere worker and the sincere inspector.

Accordingly, there is an advantage of improving the quality and progress of the entire project.

The effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the following description.

1 is a conceptual diagram of a crowdsourcing service according to an embodiment of the present invention.
2 is a flowchart illustrating a crowdsourcing-based project progress process according to an embodiment of the present invention.
3 is a flow chart of a method for selecting and prioritizing inspection of conscientious workers for a crowdsourcing-based project according to an embodiment of the present invention.
4 is a flow chart of the steps of evaluating the operator's work integrity.
5 is a flowchart of a step of evaluating inspection integrity based on a first inspection reaction time.
6 is a flow chart of the step of evaluating the inspection integrity by further considering the second inspection reaction time.
7 is a diagram for explaining the reaction time of work and the reaction time of inspection.
FIG. 8 is a diagram for explaining an apparatus for selecting sincere workers for a crowdsourcing-based project and prioritizing inspection according to an embodiment of the present invention.

Advantages and features of the present invention, and a method of achieving them will become apparent with reference to the embodiments described below in detail together with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but may be implemented in a variety of different forms, only the present embodiments are intended to complete the disclosure of the present invention, It is provided to fully inform the technician of the scope of the present invention, and the present invention is only defined by the scope of the claims.

The terms used in the present specification are for describing exemplary embodiments and are not intended to limit the present invention. In this specification, the singular form also includes the plural form unless specifically stated in the phrase. As used in the specification, “comprises” and/or “comprising” do not exclude the presence or addition of one or more other elements other than the mentioned elements. Throughout the specification, the same reference numerals refer to the same elements, and “and/or” includes each and all combinations of one or more of the mentioned elements. Although "first", "second", and the like are used to describe various elements, it goes without saying that these elements are not limited by these terms. These terms are only used to distinguish one component from another component. Therefore, it goes without saying that the first component mentioned below may be the second component within the technical idea of the present invention.

Unless otherwise defined, all terms (including technical and scientific terms) used in the present specification may be used as meanings that can be commonly understood by those of ordinary skill in the art to which the present invention belongs. In addition, terms defined in a commonly used dictionary are not interpreted ideally or excessively unless explicitly defined specifically.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a conceptual diagram of a crowdsourcing service according to an embodiment of the present invention.

Referring to FIG. 1, a crowdsourcing service is performed by being composed of a client 10, a service provider 20, and a public 30.

The sponsor 10 refers to a company or individual who requests a crowdsourcing-based project (hereinafter, referred to as a project).

The requester 10 requests a project for the purpose of collecting source data or annotating data for generating artificial intelligence learning data. The data generated through the project can be used as learning data for arbitrary machine learning such as supervised learning, unsupervised learning, and reinforcement learning. Collection of source data refers to collecting raw data such as recorded voice collection and photo collection. Data annotation refers to inputting relevant annotation data into source data such as text, photos, and videos. For example, the data annotation may include finding an entity in a given fingerprint or finding a similar sentence, but is not limited thereto. Meanwhile, the types of the above-described projects are only one embodiment, and various projects may be handled in the present invention according to the design of the client.

The service provider 20 refers to a company that provides crowdsourcing services.

When the service provider 20 receives a request for a project for a product or service from the client 10, the service provider 20 allocates a task for the project to the general public 30 and receives the work result from the public 30. Thereafter, the final product extracted based on the work result is provided to the client 10.

At this time, the service provider 20 provides a crowdsourcing service to the client 10 and the public 30 through a crowdsourcing platform (hereinafter, the platform). That is, when the service provider 20 receives a request for a project from the client 10, the service provider 20 opens the project on the platform. Thereafter, when the work result for the open project is received from the public 30, the project is terminated on the platform, and the final product may be extracted and provided to the client 10.

The public 30 refers to the general public participating in the project open on the platform. Here, the public 30 may participate in a project opened on the platform through an application or website provided by the service provider 20.

The public 30 is composed of a worker 32 and an inspector 34.

The worker 32 decides to participate in a specific project among a plurality of projects open on the platform. Thereafter, the operator 32 performs an operation such as collection of source data or data annotation, and transmits it to the platform.

The inspector 34 decides to participate in a specific project among a plurality of projects open on the platform. Thereafter, the inspector 34 inspects the result of the work performed by the operator 32. As a result of performing the inspection, the inspector 34 may perform inspection pass processing or rejection processing, and input a rejection reason during rejection processing. In the case of passing the inspection, it is not necessary to rework and re-examination due to this, so passing inspection has the same meaning as completion of inspection.

2 is a flowchart illustrating a crowdsourcing-based project progress process according to an embodiment of the present invention.

First, the requester 10 requests one or more projects to the service provider 20 (S11).

Thereafter, the service provider 20 opens the requested project on the platform (S12). At this time, the service provider 20 may determine the grade in consideration of the difficulty of the project before opening the project. That is, depending on the difficulty level, it is possible to determine which level or higher public 30 to expose the project. Accordingly, it is possible to increase the reliability of the work result of the project.

Thereafter, the service provider 20 requests the work by assigning the work to the workers 32 of the corresponding level or higher according to the level of the project (S13).

Thereafter, the worker 32 performs the assigned task (S14). In this case, the worker 32 may input the reason for the inability to work without performing the work for a work in which the work itself is impossible for some reason.

Thereafter, the service provider 20 receives the work result from the worker 32 (S15), and assigns the work result to the inspector 34 to request the inspection (S16).

Similarly, according to an embodiment of the present invention, only a suitable project may be exposed to the inspector 34 out of the total projects being performed according to the level set according to the difficulty of the project or the qualification requirements of the inspector 34.

Thereafter, the inspector 34 performs the assigned inspection (S17). At this time, the inspector 34 determines that the inspection is completed when it is determined that the work has been properly performed, and if it is determined that the inspection operation is wrong, the inspector 34 rejects it. During rejection processing, the inspector 34 inputs the rejection reason for what reason the job was judged to be wrong.

Thereafter, the service provider 20 receives the inspection result from the inspector 34 (S18).

When the inspection result is the inspection completion, the service provider 20 uses the work result as valid data, and based on this, extracts the final product at the end of the project.

When the inspection result is rejection processing, the service provider 20 may internally perform the inspection again, or assign the work to the worker 32 again to perform the rework. When reworking, re-examination by an inspector is required.

Thereafter, when the project period ends or sufficient valid data is secured, the service provider 20 terminates the project (S19), calculates a final result based on the secured valid data, and provides it to the client 10 ( S20).

At this time, before the end of the project, the service provider 20 evaluates the performance results of the worker 32 and the inspector 34, calculates the work cost and the inspection cost according to the evaluation, to the worker 32 and the inspector 34. give.

In FIGS. 1 and 2, it is simply expressed as a requester 10, a service provider 20, an operator 32, and an inspector 34, but these are smartphones, tablets, PDAs, laptops, desktops, operated by each participant. It means a computer device such as a server or a telecommunication device.

On the other hand, in carrying out a crowdsourcing-based project, the worker 32 must rework the work that was rejected by the inspector 34, but despite receiving the rejection notification, it delays a lot of time without starting the rework immediately. If so, the inspector 34 has to wait to inspect the results of the work or rework, which in turn delays the entire project period.

In the same way, the inspector 34 must immediately inspect the work results or rework results submitted from the worker 32. Even though the work or rework results have been submitted, the inspection should not be initiated immediately, and the rejection should be notified after delaying time. If so, it takes a lot of time to finally complete the work, and there is a problem that the duration of the project is lengthened.

In order to solve this problem, an embodiment of the present invention evaluates the reliability of the operator 32 and the inspector 34 based on the time associated with the operator 32 or the inspector 34 performing work and performing the inspection. By classifying them into conscientious workers and conscientious inspectors and giving them incentives, unnecessary time delays can be minimized and the entire project can be shortened.

Hereinafter, with reference to FIGS. 3 to 7, a method of selecting a conscientious worker for a crowdsourcing-based project and assigning priority to inspection will be described with reference to FIGS. 3 to 7.

3 is a flow chart of a method for selecting and prioritizing inspection of conscientious workers for a crowdsourcing-based project according to an embodiment of the present invention. 4 is a flow chart of the steps of evaluating the work integrity of the worker 32. 5 is a flowchart of a step of evaluating inspection integrity based on a first inspection reaction time. 6 is a flow chart of the step of evaluating the inspection integrity by further considering the second inspection reaction time. 7 is a diagram for explaining the reaction time of work and the reaction time of inspection.

Meanwhile, the steps shown in FIGS. 3 to 6 may be understood as being performed by a platform server (hereinafter, referred to as a server) operated by the service provider 20, but are not limited thereto.

In addition, a plurality of workers 32 or a plurality of inspectors 34 perform work using a predetermined terminal device. The terminal device of the operator 32 or the inspector 34 may be a computer device or a telecommunication device such as a smartphone, tablet, PDA, laptop, desktop, etc., but is not limited thereto.

First, referring to FIG. 3, the server assigns a plurality of tasks of a crowdsourcing-based project (hereinafter referred to as a project) to a plurality of workers 32 to request the task to be performed (S110), and from the plurality of workers 32 A plurality of work results are input (S120).

Here, one project may contain multiple jobs, and the plurality of jobs may be grouped into a plurality of tasks according to their type, or each job may be a different type of job or the same job. , But is not limited thereto.

Next, a plurality of work results are assigned to a plurality of inspectors 34 to request inspection (S130), and a plurality of inspection results for a plurality of work results from the plurality of inspectors 34 are entered as inspection pass or rejection Receive (S140).

Next, the job integrity of each worker 32 is evaluated based on the time associated with the job performance and inspection performance (S150).

In one embodiment, the work integrity of the worker 32 is evaluated based on the work reaction time related to the reworked case as it is rejected among the inspection results.

In general, reaction time refers to a time from a time point when an input is presented to a certain system or computer execution unit to a time point at which a reaction corresponding to the input occurs. Response time is also referred to as response time. The work reaction time refers to the time from the time when the inspection result by the inspector 34 is inputted to the time when the worker 32 accesses the online work space to perform rework, that is, when the rework begins. Here, the online work space means a space in which online work, such as a web page or a page on an application, or a specific URL provided by a server, can be performed.

Referring to Figure 4, the evaluation of the work integrity for each worker 32, first, as the inspection is completed for each work result assigned to each inspector 34, each inspector 34 The first time at which the inspection result is input is measured (S151).

Then, in order to perform rework for each task assigned to each worker 32 and the inspection result is input as a rejection, the second time when the worker 32 accesses the online workspace is measured (S152).

And, based on the first time and the second time, the work reaction time associated with each rework case is measured (S153). At this time, rework includes not only the initial rework but also subsequent rework such as the 2nd and 3rd.

Referring to FIG. 7, when the worker 32 submits the work result, the inspector 34 initiates the inspection. Thereafter, 15:16:10, which is the time at which the inspector 34 completes the inspection and inputs the inspection result, is measured as the first time, and the time when the operator 32 accesses the online work space for the rejection case. 19:20:00 is measured as the second time. Accordingly, the work reaction time (W1) of the worker 32 to the rework case is measured as 4 hours and 4 minutes (14,640 sec).

Similarly, in the case of re-returning after rework for a rejected work case, as the worker 32 submits the rework result, the inspector 34 initiates the re-review, completes the re-review, and enters the re-review result. One time, 23:21:25, is measured as the first time, and the second time, 23:35:45, which is the time when the worker 32 accesses the online work space for rework of the re-rejection case. It is measured as Accordingly, the work reaction time (W2) of the worker 32 to the rework case is measured as 14 minutes 25 seconds (865 sec).

When the measurement of the work reaction time is completed in this way, the work integrity of each worker 32 is evaluated based on the work reaction time (S154).

In one embodiment, the work integrity of each worker 32 may be evaluated by summing and averaging the work reaction times associated with each rework case.

For example, when the work response time (W1) for the initial rework case of the worker 32 in FIG. 7 and the work response time (W2) for the rework case after re-return are summed and averaged, 7,752 sec is the corresponding worker It is calculated as the average reaction time of, and based on the calculated result, the work integrity of the worker 32 can be evaluated.

Meanwhile, the job integrity may be evaluated at a point in time when the number of jobs that each worker 32 has completed performing the job is equal to or greater than a predetermined minimum number. In other words, if the task integrity is evaluated for too few cases, it is the same as a relative penalty is given until one or two times are delayed due to the circumstances of the worker 32, thereby lowering the worker 32's willingness to work. Since a problem occurs and the reliability of the measured work integrity is lowered, the sincerity of the worker 32 is evaluated from a point in time when a certain number or more is satisfied.

The work reaction time for evaluating such work integrity is measured on an individual worker basis.

In this way, when evaluating work integrity for individual workers, it is possible to evaluate integrity without being affected by other workers. In this case, if a person enters the rework late due to individual circumstances, the probability of receiving a general inspection increases. Conversely, if he enters the rework earlier than usual, the probability of receiving the inspection first increases.

Next, the inspection priority is given to a sincerity worker whose work integrity is equal to or greater than a predetermined first reference value among the plurality of workers 32 (S160).

In this case, the predetermined first reference value, which is a comparison criterion of the work integrity, may be a specific priority when the work integrity of the plurality of workers 32 is sorted in descending order. Alternatively, it may be provided as a specific value (numerical value). For example, the first reference value may be an average value of the work integrity of the workers 32 participating in the corresponding project. That is, the workers 32 having a specific rank or higher or a work integrity of a specific value or higher are selected as faithful workers.

Incentives are given to conscientious workers whose work integrity is equal to or greater than a predetermined first reference value.

That is, one embodiment of the present invention is based on determining the inspection order of the plurality of work results sequentially according to the order in which the plurality of work results are input, but one by a sincere worker rather than one or more work results by a general worker. The order of inspection of the above work results can be adjusted in advance.

In this case, when there are a plurality of sincerity workers, the inspection ranking may be adjusted according to the comparison of the work integrity between the sincerity inspectors.

On the other hand, an embodiment of the present invention may provide incentives to the selected integrity inspectors by separately selecting not only the sincerity workers but also the sincerity inspectors.

Specifically, when the work or rework submission of the worker 32 is completed, the inspection integrity of each inspector 34 is evaluated based on the time associated with the job performance and inspection performance (S170).

In one embodiment, the inspection integrity of the inspector 34 is evaluated based on the inspection reaction time until undertaking the inspection related to the work or rework of the operator 32.

The inspection reaction time refers to the time from the time when the work result by the operator 32 is input to the time when the inspector 34 accesses the online inspection space for inspection, that is, the time at which the inspection begins. Here, the online inspection space means a space in which online inspection tasks such as a web page or a page on an application or a specific URL provided by a server can be performed.

At this time, the inspection reaction time is the first inspection reaction time, which is the time from when the inspector 34 accesses the online inspection space to initiate the inspection after the initial work of the worker 32 is submitted, and the inspection result is rejected. After input, rework is submitted, and the second inspection reaction time, which is the time until the time when the inspector 34 accesses the online inspection space, may be calculated in order to re-examine it.

Referring to FIG. 5, as the task is completed for each task assigned to each worker 32, first, for the evaluation of the inspection integrity of the inspector 34 based on the first inspection reaction time, each worker (32) The third time when they input the work result is measured (S171).

Then, for each task assigned to each inspector 34, the fourth time when the inspector 34 accesses the online inspection space to perform the inspection is measured (S172).

Then, based on the third time and the fourth time, the first inspection reaction time associated with each inspection case is measured (S173), and the inspection integrity of each inspector 34 is evaluated based on the first inspection reaction time. (S174).

Referring to FIG. 7, the third time when the worker 32 submits the work result is 15:10:10, and the fourth time when the inspector 34 accesses the online inspection space for the work case is 15:15. It is measured as :20. Accordingly, the first inspection reaction time (C1) of the inspector 34 for the inspection case is measured as 5 minutes 10 seconds (310sec).

In one embodiment, the inspection integrity of the inspector 34 may be measured and evaluated by measuring and averaging the first inspection reaction time associated with each inspection case.

If the operator 32 passes without rejecting the initial work case, the inspection integrity of the inspector 34 is evaluated only by the first inspection reaction time. On the other hand, when the first work case of the worker 32 is rejected more than once, the inspection integrity of the inspector 34 should be evaluated by considering the second inspection reaction time together.

Referring to FIG. 6, in order to consider the second inspection reaction time together, when the operator 32 completes the rework performance for each task assigned to each operator 32 and the inspection result input as a rejection, each The workers 32 measure the fifth time when they input the rework result (S175).

Then, for each task assigned to each inspector 34 and the inspection result is input as rejection, a sixth time is measured when the inspector 34 accesses the online inspection space for re-examination (S176).

And, based on the fifth time and the sixth time, the second inspection reaction time of the inspector related to each re-examination case is measured (S177), and the second inspection reaction time is further reflected in the first inspection reaction time to each inspector. The integrity of the inspection of (34) is evaluated (S178). At this time, the re-examination includes not only the initial re-examination, but also subsequent re-examinations such as the 2nd and 3rd.

Referring to the example of FIG. 7, the fifth time when the rework result is submitted after the worker 32 is rejected is 19:20:40, and the inspector 34 accesses the online inspection space for the rework case. The fifth time is measured as 23:20:45. Accordingly, the second inspection reaction time (C2) of the inspector 34 for the re-examination case is measured as 4 hours and 5 seconds (14,405 seconds).

Accordingly, the inspection integrity of the inspector 34 may be measured and evaluated by adding up and averaging the second inspection reaction time by further reflecting the first inspection reaction time associated with each inspection case.

That is, in the example of FIG. 7, if the second inspection reaction time is further reflected in the first inspection reaction time, the average reaction time for the inspection and re-examination of the inspector 34 is calculated as 7,357 sec, and the calculated result is inspected. It can be applied with integrity.

The inspection integrity for each inspector 34 can also be evaluated at a point in time when the number of inspections or re-reviews completed by each inspector 34 is equal to or greater than a predetermined minimum number.

In this way, when the evaluation of the inspection integrity of the inspector 34 is completed, the inspection performance is requested by assigning the work result of the sincerity worker to a sincerity inspector whose inspection integrity is equal to or higher than a predetermined second reference value among the plurality of inspectors 34. (S180).

In other words, just as an incentive to adjust the inspection ranking ahead of the work results of a general worker is provided to a sincere worker, the sincerity inspector can be requested to perform the inspection by assigning the job result of the sincere worker.

In other words, in one embodiment of the present invention, the minimum work result and the inspection result are inputted for a specific period, and from that point on, the inspection priority is given to the sincere worker, and the sincere worker's work can be assigned to the sincerity inspector. .

At this time, in the case of a conscientious worker, in addition to adjusting the inspection priority to a priority, there is an effect of receiving additional incentives according to matching with the integrity inspector.

The second predetermined reference value for discriminating the integrity inspector may be a specific ranking when the inspection integrity of the inspector 34 is sorted in descending order. Alternatively, it may be provided as a specific value (numerical value). For example, it may be an average value of the inspection integrity of the inspectors 34 assigned to the project. In other words, the inspector 34 having an inspection integrity of a certain rank or higher or a specific value or higher is selected as the integrity inspector.

Meanwhile, in the above description, steps S11 to S180 may be further divided into additional steps or may be combined into fewer steps, according to an embodiment of the present invention. In addition, some steps may be omitted as necessary, and the order between steps may be changed. In addition, even if other contents are omitted, the contents of FIG. 8 to be described later may also be applied to the method of selecting and prioritizing inspections of conscientious workers for crowdsourcing-based projects of FIGS. 1 to 7.

Hereinafter, a description will be given of the apparatus 100 for selecting and prioritizing inspectors for a crowdsourcing-based project according to an embodiment of the present invention.

FIG. 8 is a diagram for explaining the apparatus 100 for selecting and prioritizing conscientious workers for a crowdsourcing-based project according to an embodiment of the present invention.

Referring to FIG. 8, the apparatus 100 for selecting and prioritizing conscientious workers for a crowdsourcing-based project accordingly includes a communication module 110, a memory 120, and a processor 130.

The communication module 110 transmits a crowdsourcing-based work for one project to a plurality of workers 32 to request a task to be performed, and receives work results from the plurality of workers 32. Further, the work results received from the plurality of workers 32 are transmitted to the plurality of inspectors 34 to request inspection, and the inspection results are received from the plurality of inspectors 34.

The memory 120 stores a program for selecting faithful workers of a crowdsourcing-based project and giving inspection priority based on data received from the communication module 110.

The processor 130 executes a program stored in the memory 120. As the processor 130 executes the program stored in the memory 120, the processor 130 evaluates the job integrity of each worker based on the time associated with the job performance and inspection, and the job integrity level among the plurality of workers is Priority of inspection is given to conscientious workers above the first standard value.

The apparatus 100 for selecting and prioritizing conscientious workers for a crowdsourcing-based project described with reference to FIG. 8 may be provided as a component of the above-described server.

The method for selecting and prioritizing conscientious workers for a crowdsourcing-based project according to an embodiment of the present invention described above may be implemented as a program (or application) to be executed in combination with a computer, which is hardware, and stored in a medium. have.

The above-described program includes C, C++, JAVA, Ruby, which can be read by a processor (CPU) of the computer through the device interface of the computer, in order for the computer to read the program and execute the methods implemented as a program. It may include a code (Code) coded in a computer language such as machine language. Such code may include a functional code related to a function defining necessary functions for executing the methods, and a control code related to an execution procedure necessary for the processor of the computer to execute the functions according to a predetermined procedure. can do. In addition, these codes may further include additional information required for the processor of the computer to execute the functions, or code related to a memory reference to which location (address address) of the internal or external memory of the computer should be referenced. have. In addition, when the processor of the computer needs to communicate with any other computer or server in the remote in order to execute the functions, the code uses the communication module of the computer to determine how It may further include a communication-related code for whether to communicate, what kind of information or media should be transmitted and received during communication.

The stored medium is not a medium that stores data for a short moment, such as a register, cache, memory, etc., but a medium that stores data semi-permanently and can be read by a device. Specifically, examples of the storage medium include, but are not limited to, ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, and the like. That is, the program may be stored in various recording media on various servers to which the computer can access, or on various recording media on the user's computer. In addition, the medium may be distributed over a computer system connected through a network, and computer-readable codes may be stored in a distributed manner.

The above description of the present invention is for illustrative purposes only, and those of ordinary skill in the art to which the present invention pertains will be able to understand that it can be easily modified into other specific forms without changing the technical spirit or essential features of the present invention. will be. Therefore, it should be understood that the embodiments described above are illustrative in all respects and not limiting. For example, each component described as a single type may be implemented in a distributed manner, and similarly, components described as being distributed may also be implemented in a combined form.

The scope of the present invention is indicated by the claims to be described later rather than the detailed description, and all changes or modified forms derived from the meaning and scope of the claims and their equivalent concepts should be interpreted as being included in the scope of the present invention. do.

10: Client
20: service provider
30: public
32: worker
34: inspector
100: Device for selecting sincere workers and giving priority to inspection
110: communication module
120: memory
130: processor

Claims (10)

As a method performed by a computer,
Assigning a plurality of tasks of a crowdsourcing-based project (hereinafter, referred to as a project) to a plurality of workers and requesting the task to be performed;
Receiving a plurality of job results from the plurality of workers;
Assigning the plurality of work results to a plurality of inspectors and requesting to perform inspection;
Receiving a plurality of inspection results for the plurality of work results from the plurality of inspectors as passing or rejecting the inspection;
Evaluating the work integrity of each worker based on the time associated with the work performance and the inspection performance; And
Including the step of giving inspection priority to a sincerity worker whose work integrity is equal to or greater than a predetermined first reference value among the plurality of workers,
The step of evaluating the work integrity of each worker,
Measuring a first time at which the inspection result is input by completing the inspection for each job result assigned to each inspector;
Measuring a second time of access to an online workspace for performing rework for each task assigned to each worker and inputted as a rejection of the inspection result;
Measuring a work reaction time associated with each rework gun corresponding to a time from the first time to the second time; And
Including the step of evaluating the work integrity of each worker based on the work reaction time,
How to prioritize inspection and selection of conscientious workers for crowdsourcing-based projects. delete The method of claim 1,
The step of evaluating the work integrity of each worker,
To evaluate the work integrity of each worker by summing and averaging the work reaction times associated with each of the rework cases,
How to prioritize inspection and selection of conscientious workers for crowdsourcing-based projects. The method of claim 1,
The step of giving inspection priority to the sincerity worker,
In order to sequentially determine the inspection order of the plurality of work results according to the order in which the plurality of work results are input, but adjust the inspection order of one or more work results by the sincerity worker earlier than the one or more work results by a general worker. ,
How to prioritize inspection and selection of conscientious workers for crowdsourcing-based projects. The method of claim 1,
Evaluating the inspection integrity of each inspector based on the time associated with performing the work and performing the inspection; And
The step of requesting to perform the inspection by assigning the work result of the sincerity worker to a sincerity inspector whose inspection integrity is equal to or greater than a predetermined second reference value among the plurality of inspectors,
How to prioritize inspection and selection of conscientious workers for crowdsourcing-based projects. The method of claim 5,
The step of evaluating the inspection integrity of each inspector,
Measuring a third time for inputting the task result by completing the task execution for each task assigned to each worker;
Measuring a fourth time of access to the online inspection space for performing the inspection for each task assigned to each inspector;
Measuring a first inspection reaction time associated with each inspection case based on the third time and the fourth time; And
Including the step of evaluating the inspection integrity of each inspector based on the first inspection reaction time,
How to prioritize inspection and selection of conscientious workers for crowdsourcing-based projects. The method of claim 6,
The step of evaluating the inspection integrity of each inspector,
Measuring a fifth time at which the rework result is input by completing rework for each task assigned to each worker and inputting the inspection result as a rejection;
Measuring a sixth time of access to an online inspection space for performing re-examination for each task assigned to each inspector and the inspection result is input as a rejection;
Measuring a second inspection reaction time associated with each re-examination case based on the fifth time and the fifth time; And
Including the step of evaluating the inspection integrity of each inspector further based on the second inspection reaction time,
How to prioritize inspection and selection of conscientious workers for crowdsourcing-based projects. The method of claim 6,
The step of evaluating the inspection integrity of each inspector,
To evaluate the inspection integrity of each inspector by measuring and averaging the first inspection reaction time associated with each inspection case,
How to prioritize inspection and selection of conscientious workers for crowdsourcing-based projects. The method of claim 1,
The step of evaluating the work integrity of each worker,
Evaluating the sincerity of each worker at a point in time when the number of tasks each worker has completed performing the task is more than a predetermined minimum number,
How to prioritize inspection and selection of conscientious workers for crowdsourcing-based projects. A computer program stored in a computer-readable recording medium in order to perform the method of selecting and prioritizing inspectors for a crowdsourcing-based project according to any one of claims 1, 3 to 9 in combination with a computer.

Images