KR102155881B1 - Method for trading inspector by inspector ability test of crowdsourcing based project for artificial intelligence training data generation - Google Patents

Abstract

The present invention relates to a method for trading an inspector through an inspector ability test of a crowdsourcing-based project for artificial intelligence learning data generation. The method includes: a step of assigning a plurality of works of a crowdsourcing-based project (hereinafter, project) to a plurality of workers and requesting work execution; a step of receiving the input of a plurality of work results from the plurality of workers; a step of assigning the plurality of work results to a plurality of inspectors and requesting inspection execution; a step of receiving the input of a plurality of inspection results with respect to the plurality of work results from the plurality of inspectors as either inspection pass or return; a step of selecting at least one trade target out of the plurality of inspectors based on the plurality of inspection results; a step of testing the trade target′s inspection ability for each of a plurality of functional elements of the project; a step of selecting a matching project in which the trade target is predicted to be proficient based on the result of the test; and a step of performing trade in which the trade target is released from the project and put into the matching project.

Description

METHOD FOR TRADING INSPECTOR BY INSPECTOR ABILITY TEST OF CROWDSOURCING BASED PROJECT FOR ARTIFICIAL INTELLIGENCE TRAINING DATA GENERATION}

The present invention relates to a method for trading inspectors through the evaluation of inspectors ability of a crowdsourcing-based project for generating artificial intelligence learning data.

Recently, more and more companies collect and process large amounts of data on a crowdsourcing basis that engages the general public in some process of corporate activities. In other words, by opening a project and allowing the general public, that is, workers to participate in the project, necessary information is collected through the work results completed by the workers.

Specifically, when one project is opened, a plurality of tasks are assigned to each of the plurality of workers. Each worker performs a plurality of tasks assigned to it, and provides the task results. Thereafter, a plurality of inspection tasks are assigned to each of the plurality of inspectors, and each inspector performs a plurality of inspection tasks assigned to them.

At this time, it is judged that the inspection performance is slow or the inspector with a high rejection rate is inferior in inspection ability and is excluded from the currently participating project to prevent further inspection.

When removing inspectors from the project, new inspectors must be recruited, which requires additional resources because the manager must review the qualifications of multiple inspectors and select an appropriate inspector among them.

In addition, since the inspector excluded from the project cannot perform the inspection until the project is completed, the problem that the inspector cannot properly utilize the inspection resources even though the inspector is always less than the operator due to the nature of the crowdsourcing platform occurs. do.

Unexamined Patent Publication No. 10-2014-0095956, 2014.08.04.

The problem to be solved by the present invention is to provide a method of trading inspectors through the evaluation of inspectors ability of a crowdsourcing-based project for generating artificial intelligence learning data.

However, the problem to be solved by the present invention is not limited to the problem as described above, and other problems may exist.

In order to solve the above-described problems, the method of trade of inspectors through evaluation of inspector ability of a crowdsourcing-based project according to an aspect of the present invention is performed by assigning a plurality of tasks of a crowdsourcing-based project (hereinafter, a project) to a plurality of workers. Requesting execution, receiving a plurality of work results from the plurality of workers, assigning the plurality of work results to a plurality of inspectors and requesting to perform the inspection, and the plurality of work results from the plurality of inspectors Receiving a plurality of inspection results for the inspection as passing or rejecting the inspection, selecting one or more trade targets from among a plurality of inspectors based on the inspection results, and the inspection ability of the trade target for each functional element of the project Evaluating, selecting a matching project predicted to be proficient by the trade target based on the evaluation result, and performing a trade of releasing the trade target from the project and putting it into the matching project. .

In some embodiments of the present invention, the step of selecting one or more trade targets from among the plurality of inspectors based on the plurality of inspection results, calculates the total cumulative number of inspections and the inspection pass rate for each inspector based on the plurality of inspection results. And selecting a predetermined inspector as a trade target from among the plurality of inspectors based on the total cumulative inspection number and inspection pass rate, wherein the inspection pass rate is the ratio of the final inspection pass to the total cumulative inspection number of the inspector Can be

In some embodiments of the present invention, the step of evaluating the inspection ability of the trade subject for each of the plurality of functional elements of the project includes at least one first functional element in which the trade subject has excellent inspection ability and the trade subject is inferior inspection Extracting one or more second functional elements showing capabilities, and selecting a matching project predicted to be proficient in the trade target based on the evaluation result is currently in progress except for the project in which the trade target is participating Among a plurality of projects in progress, a project including the first functional element and not including the second functional element may be selected as the matching project.

In some embodiments of the present invention, the step of evaluating the inspection ability of the trade target for each of the plurality of functional elements of the project includes calculating an average inspection time for each functional element of the project for all inspectors, the Comparing the overall average inspection time for each functional element of all inspectors and the average inspection time for each functional element of the trade target, and at least one first functional element indicating the inspection ability of the trade target based on the comparison result, and one It may include the step of extracting the above second functional element.

In some embodiments of the present invention, the extracting of the at least one first functional element and at least one second functional element indicating the inspection ability of the trade target based on the comparison result may include: Extracting the first functional element whose average inspection time is shorter than the total average inspection time from among functional elements by a first value or more, and the average inspection time among the plurality of functional elements based on the comparison result It may include the step of extracting the second functional element longer than the second value longer than the time.

In some embodiments of the present invention, when a plurality of projects including a first functional element and not including a second functional element are extracted among a plurality of projects currently in progress, the project with the highest ratio of the first functional element is matched with the project. Can be selected as a project.

In some embodiments of the present invention, extracting the first functional element whose average inspection time is shorter than the total average inspection time by a first value from among the plurality of functional elements based on the comparison result, comprises: When there are a plurality of functional elements whose time is shorter than the total average inspection time by a first value or more, the functional element having the smallest value obtained by subtracting the total average inspection time from the average inspection time may be extracted as the first functional element.

In some embodiments of the present invention, the step of extracting the second functional element having the average inspection time longer than the total average inspection time by a second value or more from among the plurality of functional elements based on the comparison result, the average inspection time When there are a plurality of functional elements whose time is longer than the second value of the total average inspection time, the functional element having the largest value obtained by subtracting the total average inspection time from the average inspection time may be extracted as the second functional element.

In some embodiments of the present invention, the functional element may be determined based on a work tool for executing a project.

The computer program according to another aspect of the present invention for solving the above-described problems is combined with a computer as hardware to execute the inspector trade method through the inspector ability evaluation of the crowdsourcing-based project, and is stored in a computer-readable recording medium. .

Other specific details of the present invention are included in the detailed description and drawings.

According to the present invention described above, the quality of the inspectors of a specific project can be improved by selecting a lower inspector who cannot properly perform the inspection among a plurality of inspectors participating in a specific project as a target of trade and discharging them from a specific project.

In addition, by evaluating the inspection ability of the inspector selected as a trade target in a specific project for each functional element of a specific project, the functional element showing the excellent inspection ability of the trade subject is extracted, and the functional element including the functional element among other projects currently in progress is evaluated. By selecting the project as the matching project of the person to be traded, and by injecting the person to be traded into the matching project, the person to be traded can properly demonstrate his or her inspection ability.

As a result, for each of the multiple projects currently in progress, based on the functional elements of the project, the trade targets who cannot properly perform the inspection are released, and the trade targets released from other projects that are expected to perform the inspection properly By proceeding with the input process, it is possible to efficiently utilize the inspector resources because the appropriate inspectors can be matched for each project.

The effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the following description.

1 is a conceptual diagram of a crowdsourcing service according to an embodiment of the present invention.
2 is a flowchart illustrating a crowdsourcing-based project progress process according to an embodiment of the present invention.
3 is a flowchart of a method for trading an inspector through evaluation of inspector ability of a crowdsourcing-based project according to an embodiment of the present invention.
4 is a flowchart illustrating a method of selecting a trade target according to an embodiment of the present invention.
5 is an exemplary view for explaining the arrangement of a plurality of inspectors based on the inspection pass rate according to an embodiment of the present invention.
6 is an exemplary diagram for explaining selecting a predetermined examiner as a trade target among a plurality of arranged examiners according to an embodiment of the present invention.
7 is a flow chart illustrating a method of evaluating the inspection ability of a trade target according to an embodiment of the present invention.
8 is an exemplary diagram for explaining functional elements of a project according to an embodiment of the present invention.
9 is an exemplary view for explaining calculating an average inspection time for each functional element for all inspectors according to an embodiment of the present invention.
10A and 10B are exemplary diagrams for explaining extraction of functional elements having excellent inspection ability and inferior functional elements of a trade subject according to an embodiment of the present invention.
11A and 11B are exemplary diagrams for explaining selecting a matching project of a trade target according to an embodiment of the present invention.
12 is a diagram for explaining an apparatus for trading inspectors through evaluation of inspectors ability of a crowdsourcing-based project according to an embodiment of the present invention.

Advantages and features of the present invention, and a method of achieving them will become apparent with reference to the embodiments described below in detail together with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but may be implemented in a variety of different forms, only the present embodiments are intended to complete the disclosure of the present invention, It is provided to fully inform the technician of the scope of the present invention, and the present invention is only defined by the scope of the claims.

The terms used in the present specification are for describing exemplary embodiments and are not intended to limit the present invention. In this specification, the singular form also includes the plural form unless specifically stated in the phrase. As used in the specification, “comprises” and/or “comprising” do not exclude the presence or addition of one or more other elements other than the mentioned elements. Throughout the specification, the same reference numerals refer to the same elements, and “and/or” includes each and all combinations of one or more of the mentioned elements. Although "first", "second", and the like are used to describe various elements, it goes without saying that these elements are not limited by these terms. These terms are only used to distinguish one component from another component. Therefore, it goes without saying that the first component mentioned below may be the second component within the technical idea of the present invention.

Unless otherwise defined, all terms (including technical and scientific terms) used in the present specification may be used as meanings that can be commonly understood by those of ordinary skill in the art to which the present invention belongs. In addition, terms defined in a commonly used dictionary are not interpreted ideally or excessively unless explicitly defined specifically.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a conceptual diagram of a crowdsourcing service according to an embodiment of the present invention.

Referring to FIG. 1, a crowdsourcing service is performed by being composed of a client 10, a service provider 20, and a public 30.

The sponsor 10 refers to a company or individual who requests a crowdsourcing-based project (hereinafter, referred to as a project).

The requester 10 requests a project for the purpose of collecting source data or annotating data for generating artificial intelligence learning data. The data generated through the project can be used as learning data for arbitrary machine learning such as supervised learning, unsupervised learning, and reinforcement learning. Collection of source data refers to collecting raw data such as recorded voice collection and photo collection. Data annotation refers to inputting relevant annotation data into source data such as text, photos, and videos. For example, the data annotation may include finding an entity in a given fingerprint or finding a similar sentence, but is not limited thereto. Meanwhile, the types of the above-described projects are only one embodiment, and various projects may be handled in the present invention according to the design of the client.

The service provider 20 refers to a company that provides crowdsourcing services.

When the service provider 20 receives a request for a project for a product or service from the client 10, the service provider 20 allocates a task for the project to the general public 30 and receives the work result from the public 30. Thereafter, the final product extracted based on the work result is provided to the client 10.

At this time, the service provider 20 provides a crowdsourcing service to the client 10 and the public 30 through a crowdsourcing platform (hereinafter, the platform). That is, when the service provider 20 receives a request for a project from the client 10, the service provider 20 opens the project on the platform. Thereafter, when the work result for the open project is received from the public 30, the project is terminated on the platform, and the final product may be extracted and provided to the client 10.

The public 30 refers to the general public participating in the project open on the platform. Here, the public 30 may participate in a project opened on the platform through an application or website provided by the service provider 20.

The public 30 is composed of a worker 32 and an inspector 34.

The worker 32 decides to participate in a specific project among a plurality of projects open on the platform. Thereafter, the operator 32 performs an operation such as collection of source data or data annotation, and transmits it to the platform.

The inspector 34 decides to participate in a specific project among a plurality of projects open on the platform. Thereafter, the inspector 34 inspects the result of the work performed by the operator 32. As a result of performing the inspection, the inspector 34 may perform inspection pass processing or rejection processing, and input a rejection reason during rejection processing. In the case of passing the inspection, it is not necessary to rework and re-examination due to this, so passing inspection has the same meaning as completion of inspection.

2 is a flowchart illustrating a crowdsourcing-based project progress process according to an embodiment of the present invention.

First, the requester 10 requests one or more projects to the service provider 20 (S11).

Thereafter, the service provider 20 opens the requested project on the platform (S12). At this time, the service provider 20 may determine the grade in consideration of the difficulty of the project before opening the project. That is, depending on the difficulty level, it is possible to determine which level or higher public 30 to expose the project. Accordingly, it is possible to increase the reliability of the work result of the project.

Thereafter, the service provider 20 requests the work by assigning the work to the workers 32 of the corresponding level or higher according to the level of the project (S13).

Thereafter, the worker 32 performs the assigned task (S14). In this case, the worker 32 may input the reason for the inability to work without performing the work for a work in which the work itself is impossible for some reason.

Thereafter, the service provider 20 receives the work result from the worker 32 (S15), and assigns the work result to the inspector 34 to request the inspection (S16).

Thereafter, the inspector 34 performs the assigned inspection (S17). At this time, the inspector 34 decides to pass the inspection when it is determined that the work has been properly performed, and if it is determined that the inspection work is wrong, the inspector 34 rejects it. During rejection processing, the inspector 34 inputs the rejection reason for what reason the job was judged to be wrong.

Thereafter, the service provider 20 receives the inspection result from the inspector 34 (S18).

If the inspection result passes the inspection, the service provider 20 uses the work result as valid data, and based on this, extracts the final product at the end of the project.

When the inspection result is rejection processing, the service provider 20 may internally perform the inspection again, or assign the work to the worker 32 again to perform the rework. When reworking, re-examination by an inspector is required.

Thereafter, when the project period ends or sufficient valid data is secured, the service provider 20 terminates the project (S19), calculates a final result based on the secured valid data, and provides it to the client 10 ( S20).

At this time, before the end of the project, the service provider 20 evaluates the performance results of the worker 32 and the inspector 34, calculates the work cost and the inspection cost according to the evaluation, to the worker 32 and the inspector 34. give.

In FIGS. 1 and 2, it is simply expressed as a requester 10, a service provider 20, an operator 32, and an inspector 34, but these are smartphones, tablets, PDAs, laptops, desktops, operated by each participant. It means a computer device such as a server or a telecommunication device.

3 is a flowchart of a method for trading an inspector through evaluation of inspector ability of a crowdsourcing-based project according to an embodiment of the present invention. 4 is a flowchart illustrating a method of selecting a trade target according to an embodiment of the present invention. 5 is an exemplary view for explaining the arrangement of a plurality of inspectors based on the inspection pass rate according to an embodiment of the present invention. 6 is an exemplary diagram for explaining selecting a predetermined examiner as a trade target among a plurality of arranged examiners according to an embodiment of the present invention. 7 is a flow chart illustrating a method of evaluating the inspection ability of a trade target according to an embodiment of the present invention. 8 is an exemplary diagram for explaining functional elements of a project according to an embodiment of the present invention. 9 is an exemplary view for explaining calculating an average inspection time for each functional element for all inspectors according to an embodiment of the present invention. 10A and 10B are exemplary diagrams for explaining extraction of functional elements having excellent inspection ability and inferior functional elements of a trade subject according to an embodiment of the present invention. 11A and 11B are exemplary diagrams for explaining selecting a matching project of a trade target according to an embodiment of the present invention.

Meanwhile, the steps shown in FIGS. 3, 4 and 7 may be understood as being performed by a platform server (hereinafter, referred to as a server) operated by the service provider 20, but are not limited thereto.

In addition, a plurality of workers 32 or a plurality of inspectors 34 perform work using a predetermined terminal device. The terminal device of the operator 32 or the inspector 34 may be a computer device or a telecommunication device such as a smartphone, tablet, PDA, laptop, desktop, etc., but is not limited thereto.

First, referring to FIG. 3, the server assigns a plurality of tasks of a crowdsourcing-based project (hereinafter, a project) to a plurality of workers 32 to request the task execution (S110), and a plurality of tasks from the plurality of workers 32 Receives the results of the work (S120), assigns a plurality of work results to a plurality of inspectors 34 to request inspection performance (S130), and a plurality of inspection results for the plurality of work results from the plurality of inspectors 34 It receives the input to pass or reject the inspection (S140).

The project contains multiple functional elements. The server can classify the project type based on a plurality of functional elements included in the project.

The functional elements are determined based on the work tool for carrying out the project.

For example, if you need a Bounding Box Tool for bounding work, an Input Box Tool for transfer work, and a Make Step Tool for sorting work to perform a project, The project can be classified as including a bounding functional element, a transcription functional element, and a classification functional element.

A plurality of workers 32 participating in the project perform tasks for each functional element included in the project.

Specifically, the plurality of workers 32 perform a bounding operation of selecting a specific object from an image using a bounding box tool, perform a transfer operation of inputting the name of a specific object selected using the input box tool, and make Each task is completed by performing a classification task that classifies the type of a specific object assigned a name using the step tool.

In addition, a plurality of inspectors 34 participating in the project perform inspection for each functional element included in the project.

Specifically, the plurality of inspectors 34 judged whether the bounding operation using the bounding box tool was properly performed, whether the transfer operation using the input box tool was properly performed, and whether the classification operation using the make step tool was properly performed, and passed the inspection. Or, by entering the inspection result as a rejection, one inspection is completed for each.

On the other hand, a plurality of functional elements may be assigned different weights in the project according to their importance. In other words, in a project that includes a bounding function element, a transcription function element, and a classification function element, if the importance of the bounding operation is high for reasons such as high difficulty, the bounding function element may be assigned a greater weight than the transfer function element and the classification function element. have. For example, a bounding functional element may occupy 50%, and a transfer functional element and a classification functional element may each account for 25%.

Subsequently, the server selects one or more trade targets among the plurality of inspectors based on the plurality of inspection results (S150).

The server selects an inspector 34 who is not properly performing the inspection among the plurality of inspectors 34, based on the inspection results of each of the inspectors 34 participating in the project, as a target for trade.

The overall inspection process of the project is slowed down due to the inspector who is not performing the inspection properly, resulting in an inspection bottleneck. In addition, the new inspector 34 cannot enter the project due to the inspector who is not properly performing the inspection (because the inspector who does not properly perform the inspection does not leave the project and is maintaining participation).

Therefore, while the project is in progress, by extracting the inspectors who are not properly performing the inspection based on the inspection results of the plurality of inspectors (34) and performing a trade to another project, the quality of the inspector of the project is guaranteed to speed the inspection progress. Is to be properly maintained and a new inspector (34) can enter the project.

Specifically, referring to FIG. 4, the server calculates the total cumulative number of inspections and the inspection pass rate for each inspector based on a plurality of inspection results (S151).

Here, the total cumulative number of inspections refers to the cumulative number of inspections performed by the inspector 34 as passing or rejecting the work result. At this time, the number of initial inspections for the initial work case and the number of re-reviews for the rework case may be divided and counted.

In addition, the inspection pass rate represents the ratio of the final inspection pass to the total cumulative inspection number of the inspector 34. In other words, it means the ratio of the number of cases in which the inspection result passed the inspection out of the total cumulative inspection number including both the number of initial inspection and re-examination.

The server sorts the plurality of inspectors 34 based on the inspection pass rate of each inspector 34 calculated at the present time.

In addition, the inspectors 34 having the same inspection pass rate are sorted based on the total cumulative inspection number.

Referring to FIG. 5, assuming that there are a total of 10 inspectors 34 participating in the project at the present time, the server calculates the total cumulative number of inspections and the inspection pass rate of each inspector at the present time. Then, 10 inspectors are sorted in the order of the highest inspection pass rate. It can be seen that Inspector 5 has the highest passing rate and was ranked first, and Inspector 1 has the lowest passing rate, so it can be seen that it was ranked in 10th.

At this time, the inspection pass rate of inspector 8 and inspector 4 is the same as 80%. Therefore, the server compares the total cumulative number of inspections of inspector 8 and inspector 4, and sorts the inspectors in the order in which the total cumulative inspection number is large. Since the total cumulative number of inspections of inspector 8 is 35 and the cumulative inspection of inspector 4 is 15, it can be seen that inspector 8 is ranked in 8th place and inspector 4 is in 9th place.

Referring back to FIG. 4, the server selects a predetermined inspector as a trade target from among a plurality of inspectors arranged based on the total cumulative inspection number and inspection pass rate (S152).

In other words, the server selects a predetermined inspector sorted below from among the sorted inspectors as a trade target.

Inspectors sorted below are judged to be unsuitable for the project because they do not meet the appropriate inspection speed of the project and cause an inspection bottleneck (as a result, the number of inspection passes is relatively small). It will be selected as a target for the trade.

Referring to FIG. 6, it can be seen that the server has selected two inspectors (reviewer 4 and inspector 1) arranged at the lower level among 10 inspectors participating in the project as targets for trade. Here, the description has been made on the assumption that the predetermined number is 2, but the description is not limited thereto, and the predetermined number may be set to an appropriate number to maintain an appropriate inspection speed of the project.

Referring back to FIG. 3, the server evaluates the inspection ability of the trade target for each of a plurality of functional elements of the project (S160).

As described above, the project includes a plurality of functional elements, and the inspector performs inspection for each functional element.

As shown in FIG. 8, when a trade target is currently participating in a project A including a functional element a, a functional element b, a functional element c, and a functional element d, the server Based on the time at which the inspection was performed, it is possible to evaluate the inspection ability of the trade target.

In addition, the server evaluates the inspection ability of the trade target, among a plurality of functional elements included in the project, one or more first functional elements in which the trade target shows excellent inspection ability and one or more second functions in which the trade target shows inferior inspection ability. Extract the elements.

Specifically, referring to FIG. 7, the server calculates the average inspection time for each functional element of the project targeting all inspectors (S161).

When 10 inspectors participate in the project A including functional element a, functional element b, functional element c, and functional element d to perform each inspection, the server will target 10 people and perform the entire inspection by each inspector (34). The average inspection time for each functional element is calculated for the accumulated number of cases.

That is, if a specific inspector participated in Project A and performed three inspections up to the present time, the server can calculate the average inspection time for each functional element for the three inspections by the specific inspector.

For example, when a specific inspector performs the first case, it took 10 seconds for the functional element a, 10 seconds for the functional element b, 5 seconds for the functional element c, 5 seconds for the functional element d, and the second case. , 10 seconds for functional element a, 20 seconds for functional element b, 10 seconds for functional element c, and 15 seconds for functional element d.When performing the third inspection, 10 seconds for functional element a and 25 for functional element b If seconds, 5 seconds for functional element c, and 10 seconds for functional element d, the server determines the average inspection time of the specific inspector's functional element a (10+10+10)/3=10 seconds, and the average inspection of functional element b Time is (10+20+25)/3=18 seconds, the average inspection time of functional element c is (5+10+5)/3=7 seconds, and the average inspection time of functional element d is (5+15+10) )/3=10 seconds.

Referring to Fig. 9, as 10 inspectors participate in the project A including functional element a, functional element b, functional element c, and functional element d to perform each inspection, the server performs an average inspection for each functional element of each inspector. Time is calculated, and the total average inspection time of all 10 inspectors can be calculated through the average inspection time for each functional element of each inspector.

Referring to FIG. 7 again, the server compares the total average inspection time for each functional element of all inspectors and the average inspection time for each functional element of the trade target (S162), and indicates the inspection ability of the trade target based on the comparison result. One or more first functional elements and one or more second functional elements are extracted (S163).

Specifically, the server extracts the first functional element whose average inspection time is shorter than the total average inspection time by a first value or more from among the plurality of functional elements based on the comparison result, and based on the comparison result, the average inspection time among the plurality of functional elements The second functional element is extracted, which is longer than a second value than the total average inspection time.

Here, the first value and the second value are set to predetermined values by the server.

As an embodiment, the first value and the second value may be set to the same value.

For example, when the first value and the second value are set to 0, the server simply determines whether the average inspection time is shorter or longer than the total average inspection time, and extracts the first and second functional elements.

Referring to FIG. 10A, when the first value and the second value are set to 0, the server has a shorter average inspection time of inspector 1 selected as a trade target among four functional elements than the total average inspection time of all inspectors (difference value This negative number) functional element a and functional element b are extracted as the first functional element (functional element with excellent inspection ability), and the average inspection time of inspector 1 is longer than the total average inspection time of all inspectors (the difference is positive) The functional element c can be extracted as a second functional element (a functional element having an inferior inspection ability).

In this case, the server does not extract the functional element d as the first functional element and the second functional element in which the average inspection time of inspector 1 selected as a trade target and the total average inspection time of all inspectors are the same.

As another example, when the first value and the second value are set to 5, the server determines whether the average inspection time is shorter or longer than the total average inspection time by 5 seconds or longer, and extracts the first and second functional elements. .

Referring to FIG. 10B, when the first value and the second value are set to 5, the server makes the average inspection time of inspector 1 selected as a trade target among the four functional elements 5 seconds or more than the total average inspection time of all inspectors. The short functional element a (with a difference of less than -5) is extracted as the first functional element (a functional element with excellent inspection ability), and the average inspection time of inspector 1 is longer than the total average inspection time of all inspectors by 5 seconds or more. The functional element c (with a difference value greater than 5) can be extracted as a second functional element (a functional element having an inferior inspection ability).

As another embodiment, the first value and the second value may be set to different values. Even when the first value and the second value are set to different values, since the method of extracting the first and second functional elements is the same as described above, a detailed description will be omitted.

As another embodiment, when there are multiple functional elements whose average inspection time is shorter than the first value or more than the total average inspection time, the server represents the functional element whose average inspection time minus the total average inspection time is the smallest. Can be extracted as an element. Here, the representative first functional element refers to a representative functional element judged to be the most proficient among a plurality of functional elements representing the competent inspection ability of the trade target.

On the other hand, if there are multiple functional elements whose average inspection time is longer than the second value than the total average inspection time, the server extracts the functional element with the largest value obtained by subtracting the total average inspection time from the average inspection time as the representative second functional element. I can. Here, the representative second functional element refers to a representative functional element judged to be the most inexperienced among a plurality of functional elements representing the unskilled inspection ability of the trade target.

Hereinafter, it is assumed that the first value and the second value are set to five.

When two functional elements (functional element a and functional element b) that are shorter than the average inspection time of the inspector who are traded by 5 seconds or more are extracted, the server averages the average inspection time from the average inspection time among the two functional elements. The functional element whose value minus the inspection time is smaller is extracted as the representative first functional element.

For example, if the difference value obtained by subtracting the total average inspection time from the average inspection time is -10 for the functional element a and -7 for the functional element b, the server represents the functional element a with the smaller difference value. Can be extracted with

Although the average inspection time of not only functional element a but also functional element b was calculated shorter than the first value of 5 or more than the overall average inspection time, the average inspection time of functional element a is shorter than that of functional element b. The functional element a is extracted as the representative first functional element by judging that it represents the most proficient inspection ability for a.

On the other hand, if two functional elements (functional element c and functional element d) are extracted, the average inspection time of the inspector who is the trade target is 5 seconds or longer than the overall average inspection time, the server will be calculated from the average inspection time among the two functional elements. The functional element whose value minus the total average inspection time is larger is extracted as the representative second functional element.

For example, if the difference value obtained by subtracting the total average inspection time from the average inspection time is 15 for the functional element c and 10 for the functional element d, the server extracts the functional element c with the larger difference as the representative second functional element. can do.

Although the average inspection time of not only functional element c but also functional element d is calculated longer than the overall average inspection time, the second value of 5 or more, the average inspection time of functional element c is longer than that of functional element d. The functional element d is extracted as the representative second functional element by judging that it represents the most inexperienced inspection ability for c.

Referring back to FIG. 3, the server selects a matching project predicted to be proficient in the trade target based on the evaluation result (S170).

Specifically, the server selects, as a matching project, a project including a first functional element and not including a second functional element among a plurality of projects currently in progress except for a project in which the trade target is participating.

For example, if the trade target is participating in Project A, and Project B, Project C, Project D, and Project E are currently in progress, the server evaluates the trade target's inspection ability, and the trade target is the functional element a in Project A. If it is judged that the inspection for the functional element c was performed excellently and the inspection for the functional element c was performed inferiorly, the functional element a is included and the functional element c is not included among projects B, C, D, and E in progress. Unexpected projects can be extracted and selected as a matching project for the trade target.

As an example, when a plurality of projects including the first functional element and not including the second functional element are extracted among a plurality of projects in progress, the server selects the project with the highest proportion of the first functional element as the matching project. I can. As described above, the proportion of functional elements is allocated by the server according to the importance of each functional element in the project.

Referring to FIG. 11A, when two projects (Project B and Project D) that include a functional element a and do not contain a functional element c are extracted among the other four projects currently in progress, the server has a proportion of the functional element a. The higher project B can be selected as the matching project of the trade target. Since the subject of the trade can perform excellent inspection on the functional element a, it is to match the project with a higher proportion of the functional element a so that the subject of the trade can maximize his or her ability.

As another embodiment, when a plurality of first functional elements are extracted and a plurality of projects including the first functional elements are extracted, a project having the highest weight may be selected.

In detail, a project in which the sum of the weights of the plurality of extracted first functional elements is the largest may be selected.

Referring to FIG. 11B, when functional element a and functional element b are extracted as a first functional element, and two projects including both functional element a and functional element b (project B and project D) are extracted, the server is In each project, a project with a larger sum of the weight of functional element a and the weight of functional element b can be selected as the matching project.

Since the sum of the weight of functional element a and the weight of functional element b of project B is 70%, and the sum of the weight of functional element a and functional element b of project D is 50%, the server is finally project B Is selected as a matching project.

Alternatively, after dividing the extracted plurality of first functional elements according to grades, a project with a high proportion of the highest grade functional elements may be selected as a matching project.

The explanation is assuming that the difference between the average inspection time of the trade targets and the total average inspection time of all inspectors is C grade, if it is more than 15 seconds, it is grade B, and if it is more than 20 seconds, it is grade A.

When the first functional element was extracted as functional element a and functional element b, the average inspection time of the functional element a of the traded person was calculated to be 25 seconds earlier than the overall average inspection time, and the average inspection time of the functional element b was the overall average. Assuming that it is calculated to be 10 seconds earlier than the inspection time, functional element a is matched to A grade and functional element b is matched to C grade. Accordingly, the server selects a project with a high proportion of the functional element a matched with the A grade among the currently ongoing projects as the matching project.

Alternatively, a weight is applied according to the grade, and the project with the highest calculated weight by applying the weight for each grade of the functional element may be selected as the matching project.

For example, it is assumed that a weight of 2 times is applied to grade A, a weight of 1.5 times is applied to grade B, and a weight of 1 times is applied to grade C.

If the first functional element is extracted as functional element a and functional element b, and functional element a is matched with A grade and functional element b with C grade, the proportion of functional element a is 40% and the proportion of functional element b is 30 When the percentage of project B and functional element a of 30% and functional element b of 30% are extracted, the server applies weights according to the grade for each project to determine the proportions of functional elements a and b. You can calculate and select a matching project. The weighted project B's combined weight is calculated as (40x2)+(30x1)=110%, and the weighted project D's combined weight is calculated as (30x2)+(30x1)=90%, and the server finally Project B is selected as the matching project.

Referring again to FIG. 3, the server performs a trade in which the target of trade is released from the project and put into the matching project (S180).

In other words, the server releases the trade target from the project the trade target is currently participating in, and newly inputs the trade target to the matching project matched with the trade target's inspection ability, so that the trade target can perform the check in the matching project. do.

Investigators who are unable to perform the inspection efficiency in a specific project are selected as the target of the trade and released from a specific project, and the target of the trade is inspected by themselves by putting the target of the trade into a matching project that is predicted to be able to perform the inspection skillfully. It can make it possible to perform the inspection more efficiently in the project suitable for the ability. Through this, it is possible to simultaneously manage the quality of inspectors of a plurality of projects currently in progress, saving operational resources, and enabling efficient operation of a plurality of projects.

Meanwhile, in the above description, steps S11 to S180 may be further divided into additional steps or may be combined into fewer steps, according to an embodiment of the present invention. In addition, some steps may be omitted as necessary, and the order between steps may be changed. In addition, even if other contents are omitted, the contents of FIG. 12 to be described later may also be applied to the method of trade of inspectors through the evaluation of inspector ability of the crowdsourcing-based project of FIGS. 1 to 11.

Hereinafter, with reference to FIG. 12, a description will be given of an inspector trade device 200 through an inspector ability evaluation of a crowdsourcing-based project according to an embodiment of the present invention.

12 is a diagram for explaining an apparatus for trading inspectors through evaluation of inspectors ability of a crowdsourcing-based project according to an embodiment of the present invention.

Referring to FIG. 12, an inspector trade device 200 (hereinafter, an inspector trade device) through an evaluation of inspector ability of a crowdsourcing-based project includes a communication module 210, a memory 220, and a processor 230.

The communication module 210 transmits a crowdsourcing-based work for one project to a plurality of workers 32 to request a task to be performed, and receives a work result from the plurality of workers 32. The work results received from the plurality of workers 32 are transmitted to the plurality of inspectors 34 to request inspection, and the inspection results are received from the plurality of inspectors 34.

The memory 220 stores a program for selecting a person to be traded in a specific project based on data received from the communication module 210 and evaluating the ability to inspect for each functional element of the person to be traded.

The processor 230 executes a program stored in the memory 220. As the processor 230 executes the program stored in the memory 220, the processor 230 selects a person to be traded from among a plurality of inspectors based on the total cumulative number of inspections and the inspection pass rate, and based on the average inspection time for each functional element of the trade object. Therefore, functional elements with excellent inspection ability and inferior functional elements are extracted, and a project that contains excellent functional elements and does not contain inferior functional elements among other projects except for the specific project in which the trade target is participating is selected as the matching project of the trade target. In addition, the target of trade can be released from a specific project currently participating and put into a matching project.

The inspector trade device 200 described with reference to FIG. 12 may be provided as a component of the server described above.

The method for trading inspectors through evaluation of inspectors ability of a crowdsourcing-based project according to an embodiment of the present invention described above may be implemented as a program (or application) and stored in a medium to be executed in combination with a computer that is hardware. .

The above-described program includes C, C++, JAVA, Ruby, which can be read by a processor (CPU) of the computer through the device interface of the computer, in order for the computer to read the program and execute the methods implemented as a program. It may include a code (Code) coded in a computer language such as machine language. Such code may include a functional code related to a function that defines necessary functions for executing the methods, and includes a control code related to an execution procedure necessary for the processor of the computer to execute the functions according to a predetermined procedure. can do. In addition, these codes may further include additional information required for the processor of the computer to execute the functions, or code related to a memory reference to which location (address address) of the internal or external memory of the computer should be referenced. have. In addition, when the processor of the computer needs to communicate with any other computer or server in the remote in order to execute the functions, the code uses the communication module of the computer to determine how It may further include a communication-related code for whether to communicate, what kind of information or media should be transmitted and received during communication.

The stored medium is not a medium that stores data for a short moment, such as a register, cache, memory, etc., but a medium that stores data semi-permanently and can be read by a device. Specifically, examples of the storage medium include, but are not limited to, ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, and the like. That is, the program may be stored in various recording media on various servers to which the computer can access, or on various recording media on the user's computer. In addition, the medium may be distributed over a computer system connected through a network, and computer-readable codes may be stored in a distributed manner.

The above description of the present invention is for illustrative purposes only, and those of ordinary skill in the art to which the present invention pertains can understand that it is possible to easily transform it into other specific forms without changing the technical spirit or essential features of the present invention. will be. Therefore, it should be understood that the embodiments described above are illustrative in all respects and not limiting. For example, each component described as a single type may be implemented in a distributed manner, and similarly, components described as being distributed may also be implemented in a combined form.

The scope of the present invention is indicated by the claims to be described later rather than the detailed description, and all changes or modified forms derived from the meaning and scope of the claims and their equivalent concepts should be interpreted as being included in the scope of the present invention. do.

10: Client
20: service provider
30: public
32: worker
34: inspector
200: Inspector trade device
210: communication module
220: memory
230: processor

Claims (10)

As a method performed by a computer,
Assigning a plurality of tasks of a crowdsourcing-based project (hereinafter, referred to as a project) to a plurality of workers and requesting the task to be performed;
Receiving a plurality of job results from the plurality of workers;
Assigning the plurality of work results to a plurality of inspectors and requesting to perform inspection;
Receiving a plurality of inspection results for the plurality of work results from the plurality of inspectors as passing or rejecting the inspection;
Selecting one or more trade targets from among a plurality of inspectors based on the plurality of inspection results;
Evaluating the inspection ability of the trade target for each of the plurality of functional elements of the project;
Selecting a matching project predicted to be proficient in the trade target based on the evaluation result; And
Comprising the step of performing a trade of releasing the trade target from the project and putting it into the matching project,
Evaluating the inspection ability of the trade target for each of the plurality of functional elements of the project,
Extracting one or more first functional elements, wherein the trade subject has excellent inspection ability, and one or more second functional elements, wherein the trade subject has inferior inspection ability,
The step of selecting a matching project predicted to be proficient in the trade target based on the evaluation result,
Selecting a project including the first functional element and not including the second functional element among a plurality of projects currently in progress except for the project in which the trade target is participating, as the matching project,
When a plurality of projects including the first functional element and not including the second functional element are extracted among a plurality of projects currently in progress, the project with the largest proportion of the first functional element is selected as the matching project,
When a plurality of projects including the first functional element and not including the second functional element are extracted among a plurality of projects currently in progress, and a plurality of the first functional elements are extracted, the plurality of first functional elements Select the project with the largest combined weight as the matching project,
The functional element is determined based on a work tool for performing a project, and the work tool is provided in a project and is a tool used by an operator to perform a task required by the project,
How to trade inspectors through the evaluation of inspectors ability of crowdsourcing-based projects. The method of claim 1,
Selecting one or more trade targets from among a plurality of inspectors based on the plurality of inspection results,
Calculating a total cumulative number of inspections and an inspection pass rate for each inspector based on the plurality of inspection results; And
Including the step of selecting a predetermined inspector as a trade target among the plurality of inspectors based on the total cumulative inspection number and inspection pass rate,
The inspection pass rate is the ratio of the final inspection pass to the total cumulative inspection number of the inspector,
How to trade inspectors through the evaluation of inspectors ability of crowdsourcing-based projects. delete The method of claim 1,
Evaluating the inspection ability of the trade target for each of the plurality of functional elements of the project,
Calculating an average inspection time for each functional element of the project for all inspectors;
Comparing the average inspection time for each functional element of the total inspectors and the average inspection time for each functional element of the trade target; And
Including the step of extracting one or more first functional elements and one or more second functional elements representing the inspection ability of the trade target based on the comparison result,
How to trade inspectors through the evaluation of inspectors ability of crowdsourcing-based projects. The method of claim 4,
The step of extracting at least one first functional element and at least one second functional element representing the inspection ability of the trade target based on the comparison result,
Extracting the first functional element whose average inspection time is shorter than the total average inspection time by a first value or more from the plurality of functional elements based on the comparison result; And
Including the step of extracting the second functional element from the plurality of functional elements based on the comparison result, the average inspection time is longer than a second value longer than the total average inspection time,
How to trade inspectors through the evaluation of inspectors ability of crowdsourcing-based projects. delete The method of claim 5,
The step of extracting the first functional element whose average inspection time is shorter than the total average inspection time by a first value or more from among the plurality of functional elements based on the comparison result,
When there are a plurality of functional elements whose average inspection time is shorter than the first value or more than the total average inspection time, the functional element having the smallest value obtained by subtracting the total average inspection time from the average inspection time is extracted as the first functional element. ,
How to trade inspectors through the evaluation of inspectors ability of crowdsourcing-based projects. The method of claim 5,
Extracting the second functional element from among the plurality of functional elements based on the comparison result that the average inspection time is longer than the total average inspection time by a second value or more,
When there are a plurality of functional elements whose average inspection time is longer than a second value than the total average inspection time, the functional element having the largest value obtained by subtracting the total average inspection time from the average inspection time is extracted as the second functional element. ,
How to trade inspectors through the evaluation of inspectors ability of crowdsourcing-based projects. delete In order to perform the method of trading inspectors through the evaluation of inspector ability of the crowdsourcing-based project in any one of paragraphs 1, 2, 4, 5, 7 or 8 in combination with a computer A computer program stored on a readable recording medium.

Images