KR102215416B1 - Method for periodically updating and optimizing groups of workers and inspectors of crowdsourcing based projects for artificial intelligence training data generation - Google Patents

Abstract

Provided is a method for periodically updating and optimizing a worker group of a crowdsourcing-based project for generating artificial intelligence training data. The method comprises the steps of: setting qualifications for participation in a crowdsourcing-based project (hereinafter, referred to as a project) when the project is opened; extracting a plurality of workers satisfying the qualifications for participation from among all workers registered in a crowdsourcing platform (hereinafter, referred to as a platform), and generating a worker group including the extracted workers; assigning a plurality of tasks to the workers included in the worker group, and requesting the workers to perform the tasks; receiving a plurality of task results from the workers included in the worker group; and automatically updating the worker group repeatedly at every predetermined period until the project is terminated. According to the present invention, the quality of a project can be further improved.

Description

{METHOD FOR PERIODICALLY UPDATING AND OPTIMIZING GROUPS OF WORKERS AND INSPECTORS OF CROWDSOURCING BASED PROJECTS FOR ARTIFICIAL INTELLIGENCE TRAINING DATA GENERATION}

The present invention relates to a method for periodically updating and optimizing groups of workers and inspectors of a crowdsourcing-based project for generating artificial intelligence learning data.

Recently, more and more companies collect and process large amounts of data on a crowdsourcing basis that engages the general public in some process of corporate activities. In other words, by opening a project and allowing the general public, that is, workers to participate in the project, necessary information is collected through the work results completed by the workers.

At this time, in order to collect more reliable information, the company assigns the work result completed by the worker to the inspector to perform the inspection work.

Specifically, when one project is opened, a plurality of tasks are assigned to each of the plurality of workers. Each worker performs a plurality of tasks assigned to it, and provides the task results. Thereafter, a plurality of inspection tasks are assigned to each of the plurality of inspectors, and each inspector performs a plurality of inspection tasks assigned to them.

On the other hand, in order to improve the convenience of workers and inspectors, and to improve the quality of the project, a plan is being considered to match workers or inspectors suitable for a specific project, associate the matched workers or inspectors with a specific project, and manage them as a group. A specific project may be provided as a list of possible projects to the workers or inspectors included in the group.

However, in the above case, once an operator or inspector is matched, there is a disadvantage that the matched group is maintained throughout the project period, making it impossible to compensate for inappropriate matching.

In addition, there is no guarantee that workers or inspectors matched according to participation qualifications will necessarily participate in the project.

In addition, since matching with the project is based on the past project history, new members who have recently joined (or registered) cannot be matched continuously due to the lack of project history. There is a problem that it acts as a barrier to entry when participating in the project as it deprives the opportunity to be evaluated for quality.

However, indiscriminate participation of new members in the project also has a problem that may result in excessive defect results due to unskilled workers or inspectors.

Unexamined Patent Publication No. 10-2014-0095956, 2014.08.04.

The problem to be solved by the present invention is to provide a method for periodically updating and optimizing a group of workers in a crowdsourcing-based project so that the group of workers participating in the project is updated every week and optimized for the project.

Another problem to be solved by the present invention is to provide a method of periodically updating and optimizing a group of inspectors participating in a project so that the group of inspectors participating in the project is updated every week and optimized for the project.

However, the problem to be solved by the present invention is not limited to the problem as described above, and other problems may exist.

The method of periodically updating and optimizing a worker group of a crowdsourcing-based project according to the first aspect of the present invention for solving the above-described problem is the step of setting a participation qualification for the project when a crowdsourcing-based project (hereinafter, a project) is opened. ; Extracting a plurality of workers satisfying the participation qualification from among all workers registered in a crowdsourcing platform (hereinafter, the platform), and generating a worker group including the extracted plurality of workers; Assigning a plurality of tasks to a plurality of workers included in the worker group and requesting to perform a task; Receiving a plurality of job results from a plurality of workers included in the worker group; And automatically updating the worker group repeatedly at a predetermined period until the end of the project. In this case, the step of automatically updating the worker group includes newly extracting a first worker of a first scale that meets the participation qualification from among all workers registered on the platform, and the remaining workers who do not meet the participation qualification Including the step of newly randomly extracting a second worker of a second scale from among, and automatically updating the worker group to include the first worker of the first scale and the second worker of the second scale. do.

In some embodiments of the present invention, the step of creating the worker group includes extracting the first worker of the first scale that satisfies the participation qualification from all workers registered on the platform, and the participation qualification Randomly extracting the second worker of the second scale from among the remaining workers that are not satisfied, and generating the worker group including the first worker of the first scale and the second worker of the second scale It may include the step of.

In some embodiments of the present invention, the step of newly randomly extracting a second worker of the second scale includes extracting a worker whose activity index is greater than or equal to a reference value from among the remaining workers who do not meet the participation qualification, and the activity It may include the step of newly randomly extracting a second worker of the second scale from among workers whose index is equal to or greater than the reference value.

In some embodiments of the present invention, the second scale may be smaller than the first scale and may correspond to a predetermined ratio of the first scale.

In some embodiments of the present invention, the cycle of automatically updating the worker group may be determined according to the difficulty level of the project.

Some embodiments of the present invention may further include analyzing a result of automatically updating the worker group, and analyzing an attribute of a worker transferring to the first worker among the second workers.

In addition, a method for periodically updating and optimizing a group of inspectors of a crowdsourcing-based project according to a second aspect of the present invention includes: setting a participation qualification for the crowdsourcing-based project (hereinafter, referred to as a project) when the project is opened; Extracting a plurality of inspectors satisfying the participation qualification from among all inspectors registered in a crowdsourcing platform (hereinafter, the platform), and generating an inspector group including the extracted plurality of inspectors; Allocating a plurality of tasks of the project to a plurality of workers and requesting the task to be performed; Receiving a plurality of job results from the plurality of workers; Allocating the plurality of work results to a plurality of inspectors included in the inspector group and requesting to perform inspection; Receiving a plurality of inspection results for the plurality of work results from a plurality of inspectors included in the inspector group; And automatically updating the examiner group repeatedly at predetermined periods until the end of the project. At this time, the step of automatically updating the examiner group includes the steps of newly extracting a first examiner of a first scale who meets the participation qualification from among all examiners registered on the platform, and the remaining examiners who do not meet the participation qualification. Including the step of newly randomly extracting a second inspector of a second scale from among, and automatically updating the inspector group to include the first inspector of the first scale and the second inspector of the second scale. do.

In some embodiments of the present invention, the step of generating the group of inspectors includes the steps of extracting the first inspectors of the first scale who meet the qualification for participation from among all inspectors registered on the platform, and the qualification for participation Randomly extracting the second inspector of the second scale from among the remaining inspectors that are not satisfied, and generating the inspector group including the first inspector of the first scale and the second inspector of the second scale It may include the step of.

In some embodiments of the present invention, the step of newly randomly extracting the second examiner of the second scale comprises: extracting a examiner whose activity index is greater than or equal to a reference value from the remaining examiners who do not meet the participation qualification, and the activity It may include the step of newly randomly extracting a second inspector of the second scale from among inspectors whose index is equal to or greater than the reference value.

In some embodiments of the present invention, the second scale may be smaller than the first scale and may correspond to a predetermined ratio of the first scale.

In some embodiments of the present invention, the cycle of automatically updating the examiner group may be determined according to the difficulty level of the project.

Some embodiments of the present invention may further include analyzing a result of automatically updating the examinee group, and analyzing an attribute of an examiner who transfers to the first examiner among the second examiners.

A computer program according to another aspect of the present invention for solving the above-described problems is combined with a computer that is hardware to execute a method for periodically updating and optimizing the worker group of the crowdsourcing-based project, and is stored in a computer-readable recording medium. .

A computer program according to another aspect of the present invention for solving the above-described problem is combined with a computer that is hardware to execute a method of periodically updating and optimizing the examiner group of the crowdsourcing-based project, and is stored in a computer-readable recording medium. .

Other specific details of the present invention are included in the detailed description and drawings.

According to the present invention described above, there is an advantage in that it is possible to gradually match workers and inspectors optimized for a project as work of a project progresses or as a number of projects progress.

In addition, the quality of the project can be further improved by discovering workers and inspectors with high quality among members who have recently joined with a small project history or those who have not participated in a specific project.

The effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the following description.

1 is a conceptual diagram of a crowdsourcing service according to an embodiment of the present invention.
2 is a flowchart illustrating a crowdsourcing-based project progress process according to an embodiment of the present invention.
3 is a flowchart of a method for periodically updating and optimizing a worker group of a crowdsourcing-based project.
4 is a flowchart of an embodiment of creating a worker group.
5 is a flowchart of an embodiment of automatically updating a worker group.
6 is a flowchart of a method for periodically updating and optimizing a group of inspectors in a crowdsourcing-based project.
7 is a flowchart of an embodiment of creating a group of inspectors.
8 is a flowchart of an embodiment of automatically updating a group of inspectors.
9 is a diagram for describing a device for periodically updating and optimizing a group of workers and inspectors of a crowdsourcing-based project according to an embodiment of the present invention.

Advantages and features of the present invention, and a method of achieving them will become apparent with reference to the embodiments described below in detail together with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but may be implemented in a variety of different forms, only the present embodiments are intended to complete the disclosure of the present invention, It is provided to fully inform the technician of the scope of the present invention, and the present invention is only defined by the scope of the claims.

The terms used in the present specification are for describing exemplary embodiments and are not intended to limit the present invention. In this specification, the singular form also includes the plural form unless specifically stated in the phrase. As used in the specification, “comprises” and/or “comprising” do not exclude the presence or addition of one or more other elements other than the mentioned elements. Throughout the specification, the same reference numerals refer to the same elements, and “and/or” includes each and all combinations of one or more of the mentioned elements. Although "first", "second", and the like are used to describe various elements, it goes without saying that these elements are not limited by these terms. These terms are only used to distinguish one component from another component. Therefore, it goes without saying that the first component mentioned below may be the second component within the technical idea of the present invention.

Unless otherwise defined, all terms (including technical and scientific terms) used in the present specification may be used as meanings that can be commonly understood by those of ordinary skill in the art to which the present invention belongs. In addition, terms defined in a commonly used dictionary are not interpreted ideally or excessively unless explicitly defined specifically.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a conceptual diagram of a crowdsourcing service according to an embodiment of the present invention.

Referring to FIG. 1, a crowdsourcing service is performed by being composed of a client 10, a service provider 20, and a public 30.

The sponsor 10 refers to a company or individual who requests a crowdsourcing-based project (hereinafter, referred to as a project).

The requester 10 requests a project for the purpose of collecting source data or annotating data for generating artificial intelligence learning data. The data generated through the project can be used as learning data for arbitrary machine learning such as supervised learning, unsupervised learning, and reinforcement learning. Collection of source data refers to collecting raw data such as recorded voice collection and photo collection. Data annotation refers to inputting relevant annotation data into source data such as text, photos, and videos. For example, the data annotation may include finding an entity in a given fingerprint or finding a similar sentence, but is not limited thereto. Meanwhile, the types of the above-described projects are only one embodiment, and various projects may be handled in the present invention according to the design of the client.

The service provider 20 refers to a company that provides crowdsourcing services.

When the service provider 20 receives a request for a project for a product or service from the client 10, the service provider 20 allocates a task for the project to the general public 30 and receives the work result from the public 30. Thereafter, the final product extracted based on the work result is provided to the client 10.

At this time, the service provider 20 provides a crowdsourcing service to the client 10 and the public 30 through a crowdsourcing platform (hereinafter, the platform). That is, when the service provider 20 receives a request for a project from the client 10, the service provider 20 opens the project on the platform. Thereafter, when the work result for the open project is received from the public 30, the project is terminated on the platform, and the final product may be extracted and provided to the client 10.

The public 30 refers to the general public participating in the project open on the platform. Here, the public 30 may participate in a project opened on the platform through an application or website provided by the service provider 20. Among the public (30), a person who can join (or register) the platform and participate in the project is called a member (or user). Also, the members who are participating in the project are called participants.

Participants are composed of a worker 32 and an inspector 34.

The worker 32 decides to participate in a specific project among a plurality of projects open on the platform. Thereafter, the operator 32 performs an operation such as collection of source data or data annotation, and transmits it to the platform.

The inspector 34 decides to participate in a specific project among a plurality of projects open on the platform. Thereafter, the inspector 34 inspects the result of the work performed by the operator 32. As a result of performing the inspection, the inspector 34 may perform inspection pass processing or rejection processing, and input a rejection reason during rejection processing. In the case of passing the inspection, it is not necessary to rework and re-examination due to this, so passing inspection has the same meaning as completion of inspection.

2 is a flowchart illustrating a crowdsourcing-based project progress process according to an embodiment of the present invention.

First, the requester 10 requests one or more projects to the service provider 20 (S11).

Thereafter, the service provider 20 opens the requested project on the platform (S12). At this time, the service provider 20 may determine the grade in consideration of the difficulty of the project before opening the project. In other words, depending on the level of difficulty, you can determine which level or higher members will be exposed to the project. Accordingly, it is possible to increase the reliability of the work result of the project.

In addition, the service provider 20 may determine participation qualifications including the above grades for projects requiring predetermined conditions. In addition, by exposing only suitable projects to individual members among the entire projects according to the qualifications of the worker 32, it is possible to improve reliability and to facilitate the selection of projects for members to participate as workers.

For example, as conditions constituting the participation qualification of the worker 32, nationality, age, gender, place of birth, place of residence, foreign language ability, grade (e.g., starter, bronze, silver, gold, etc.), project history (project type, etc.) , Number of participation, participation time, rejection rate, etc.) can be used. These elements can be stored and managed as member information for comparison purposes.

For example, the qualifications for participation of the worker 32 in a specific project may be specified under conditions such as male, 20s, Korea, starter grade, past participation in type A project two or more times.

Thereafter, the service provider 20 allocates a task to the worker 32 that satisfies the level or participation qualification of the project and requests the task (S13).

Thereafter, the worker 32 performs the assigned task (S14). In this case, the worker 32 may input the reason for the inability to work without performing the work for a work in which the work itself is impossible for some reason.

Thereafter, the service provider 20 receives the work result from the worker 32 (S15), and assigns the work result to the inspector 34 to request the inspection (S16).

Likewise, the service provider 20 can improve reliability and facilitate the selection of a project that members will participate as an inspector by exposing only suitable projects to individual members out of the total project according to the qualification of the inspector 34.

For example, the conditions constituting the qualification for participation of the examiner 34 are nationality, age, gender, place of birth, place of residence, foreign language ability, grade (e.g., starter, bronze, silver, gold, etc.), and project history (project type). , Number of participation, participation time, rejection rate, etc.) can be used.

Thereafter, the inspector 34 performs the assigned inspection (S17). At this time, the inspector 34 determines that the inspection is completed when it is determined that the work has been properly performed, and if it is determined that the inspection operation is wrong, the inspector 34 rejects it. During rejection processing, the inspector 34 inputs the rejection reason for what reason the job was judged to be wrong.

Thereafter, the service provider 20 receives the inspection result from the inspector 34 (S18).

When the inspection result is the inspection completion, the service provider 20 uses the work result as valid data, and based on this, extracts the final product at the end of the project.

When the inspection result is rejection processing, the service provider 20 may internally perform the inspection again, or assign the work to the worker 32 again to perform the rework. When reworking, re-examination by an inspector is required.

Thereafter, when the project period ends or sufficient valid data is secured, the service provider 20 terminates the project (S19), calculates a final result based on the secured valid data, and provides it to the client 10 ( S20).

At this time, before the end of the project, the service provider 20 evaluates the performance results of the worker 32 and the inspector 34, calculates the work cost and the inspection cost according to the evaluation, to the worker 32 and the inspector 34. give.

In FIGS. 1 and 2, it is simply expressed as a requester 10, a service provider 20, an operator 32, and an inspector 34, but these are smartphones, tablets, PDAs, laptops, desktops, operated by each participant. It means a computer device such as a server or a telecommunication device.

Hereinafter, a method of periodically updating and optimizing a worker group of a crowdsourcing-based project according to an embodiment of the present invention will be described with reference to FIGS. 3 to 5.

3 is a flowchart of a method for periodically updating and optimizing a worker group of a crowdsourcing-based project. 4 is a flowchart of an embodiment of creating a worker group. 5 is a flowchart of an embodiment of automatically updating a worker group.

Meanwhile, the steps illustrated in FIGS. 3 to 8 to be described later may be understood as being performed by a platform server (hereinafter, referred to as a server) operated by the service provider 20, but are not limited thereto.

In addition, a plurality of workers 32 or a plurality of inspectors 34 perform work using a predetermined terminal device. The terminal device of the operator 32 or the inspector 34 may be a computer device or a telecommunication device such as a smartphone, tablet, PDA, laptop, desktop, etc., but is not limited thereto.

Detailed descriptions of contents overlapping with those described with reference to FIGS. 1 to 2 will be omitted.

First, the server sets the project participation qualification for the worker 32 when the crowdsourcing-based project is opened (S110).

In one embodiment, the participation qualification for the worker 32 may be granted when the matching score is greater than or equal to a reference value. The matching score can be calculated by evaluating and summing at least one of nationality, age, gender, place of birth, place of residence, foreign language ability, grade, and project history required for the project.

Next, the server extracts a plurality of workers 32 that meet the participation qualifications among all workers 32 registered in the crowdsourcing platform, and creates a worker group including the extracted plurality of workers 32 (S120 ).

Referring to FIG. 4, in the process of creating a worker group, first, a first worker of a first scale who satisfies the participation qualification from among all workers 32 registered on the platform is extracted (S121). Here, the first scale is not limited as long as the eligibility for participation is met. In other words, it may be any N of all K workers registered on the platform that satisfy the participation qualification.

Next, a second worker of the second scale is randomly extracted from the remaining workers 32 who do not meet the participation qualification (S122). In an embodiment, the second scale may be smaller than the first scale and may be set to correspond to a predetermined ratio of the first scale. For example, the second scale may be 0.1*N people, which is 10% of the first scale N people. Here, the ratio of 10% is only an example, and it goes without saying that various modifications are possible according to embodiments.

Thereafter, a worker group including a first worker of a first scale and a second worker of a second scale is created (S123). Here, the first worker refers to a worker extracted from among workers who meet the participation qualification, and is only used to distinguish it from the second worker extracted from among workers who do not meet the participation qualification, and is not limited to other meanings.

After the worker group is created, a plurality of tasks are assigned to a plurality of workers 32 included in the worker group to request the task performance (S130), and a plurality of work results from the plurality of workers 32 included in the worker group It receives the input (S140). Details on this are duplicated as described above, so detailed descriptions are omitted.

The number of workers in the worker group and the number of workers involved in the actual project are strictly different. The project is included in the list of projects that can be participated by the worker belonging to the worker group, so that the worker can only participate in the project, and whether or not the worker actually participates is entirely up to the worker.

Next, the worker group is automatically updated repeatedly every predetermined period until the project is finished (S150). If the worker group is not updated, a new worker is not added or an existing worker is deleted in the worker group. That is, a worker who can participate in the project is fixed as a worker included in the worker group initially created in step S120.

In an embodiment, a predetermined period of automatically updating the worker group may be determined according to the difficulty of the project. That is, when the difficulty level is high, it takes a lot of time for the worker 32 to become proficient in the work of the project, and thus the predetermined cycle also increases.

In another embodiment, the predetermined cycle of automatically updating the worker group is determined in consideration of the task execution speed, average value, and the total number of tasks in the project, or the determined cycle is adjusted. It could be.

Referring to FIG. 5, as a process of updating a worker group, first, a first worker of a first scale who satisfies the participation qualification from among all workers 32 registered on the platform is newly extracted (S151).

In an embodiment, in the process of updating a worker group, a history of participating in a target project or another project during a predetermined period may be reflected. Project history is considered eligible for participation as described above. Depending on the embodiment, the project history may have the largest weight among various items related to participation qualification. Accordingly, if the updated project history is reflected in the participation qualification, as the project continues, there is an advantage that a worker group matching the worker 32 who can gradually improve the progress speed and quality of the project is created.

In this case, the first size may be any N number of persons who satisfy the participation qualification when the worker group is created, but when the worker group is updated, it has a fixed value of N when the worker group is created. That is, even if the number that satisfies the participation qualification is L, only N of them are newly extracted as first workers.

Therefore, even if the matching score of the worker 32 is greater than or equal to the reference value and satisfies the participation qualification when creating the first worker group, it may not be included in the first group when updating. That is, since the first scale has a fixed value, matching scores for the workers 32 are sorted in descending order and then selected by limiting them to a predetermined rank, and the first scale can be newly extracted according to these conditions.

Then, a second worker of the second scale is newly randomly extracted from the remaining workers 32 who do not meet the participation qualification (S152). In this case, the second scale may be smaller than the first scale as described above, and may be set to correspond to a predetermined ratio of the first scale.

Then, the worker group is automatically updated to include the first worker of the first scale and the second worker of the second scale (S153).

In one embodiment, in order to randomly extract a second worker of a second scale, the worker 32 whose activity index is equal to or greater than a reference value is extracted from the remaining workers 32 who do not meet the participation qualification. In addition, a second worker of a second scale is newly randomly extracted from the workers 32 whose activity index is equal to or greater than the reference value.

Here, the second worker randomly extracts regardless of the participation qualification, but the activity index is checked. This is because including workers 32 who are not participating in the project but registered on the platform in the worker group does not affect the efficiency of the project.

The activity index can mean the degree of participation in any project registered on the platform. In an embodiment, the activity index may be calculated from the number of jobs processed during a certain period, for example, may be calculated as the number of jobs processed within the last week. It goes without saying that depending on the embodiment, the number of rework for the rejection of the inspector 34 may be included or excluded when calculating the activity index.

The embodiment in FIGS. 3 to 5 will be described in more detail as follows.

First, in order to create a worker group, 100 first workers, which is the first scale that satisfies the participation qualification, are extracted from the 1000 workers 32 registered on the platform.

In addition, a second worker of a second scale is extracted from the remaining 900 workers who do not meet the participation qualification. Here, when the predetermined ratio for determining the second scale is 10%, the second scale is 10 people, which is 10% of the first scale.

Ten second workers of the second scale are extracted from 900 people.If the workers 32 with an activity index equal to or greater than the reference value are extracted as 100 out of 900 as a standard for extraction, 10 out of 100 are randomly extracted. Thus, it can be determined as a second worker.

Accordingly, a worker group may be created to include 110 first workers of the first scale and 110 second workers of the second scale.

Then, as the first cycle arrives, the worker group is updated again.

Assume that 100 new workers are registered on the platform. In order to update the worker group, 100 first workers, which are the first size of the 1100 workers 32 who meet the participation qualification, are newly extracted.

In addition, 10 second workers of the second scale are newly extracted from the remaining 1000 workers who do not meet the participation qualification.

Accordingly, as workers composed of 110 are newly recruited, the worker group is updated.

This process is repeated every predetermined cycle until the end of the project.

In this process, an exemplary embodiment of the present invention may analyze the result of automatically updating the worker group, and analyze the attribute of the worker 32 transferring from the second worker to the first worker.

For example, as the first cycle arrives, when five workers 32 among the second workers who participated in the project are eligible to participate and are selected as the first workers, the attributes of the workers are analyzed. Through this, there is a possibility of discovering an improvement element of the matching system used in the recruitment process of workers. In other words, it is possible to find the attributes (or features) of workers with high qualities and use them when matching with a project. In other words, the participation qualification can be updated in consideration of the attributes.

In this way, the second workers function similarly to mutations in the genetic algorithm, and as the worker group is updated every cycle, the optimization effect through evolution is exerted.

On the other hand, in the case of a genetic algorithm, a process of finding an optimal solution through mutation after selecting a random candidate solution, but in the case of an embodiment of the present invention, a mutation effect is only introduced by borrowing the concept of a genetic algorithm, and is strictly a different concept. Or it corresponds to an algorithm.

Since the project is provided as a project that can participate in such a worker group, it can be easy to select and participate in the project. In addition, from the perspective of the project, as opportunities are given so that the second workers can qualify for participation, they are converted to the first workers, which helps to improve the speed and quality of the project.

Hereinafter, a method for periodically updating and optimizing a group of inspectors of a crowdsourcing-based project according to an embodiment of the present invention will be described with reference to FIGS. 6 to 8.

On the other hand, the content of creating and updating the inspector group shares the content of creating and updating the worker group described in FIGS. 3 to 5 and the technical characteristics thereof, and thus overlapping content will be omitted.

6 is a flowchart of a method for periodically updating and optimizing a group of inspectors in a crowdsourcing-based project. 7 is a flowchart of an embodiment of creating a group of inspectors. 8 is a flowchart of an embodiment of automatically updating a group of inspectors.

First, when the crowdsourcing-based project is opened, the server sets a project participation qualification for the inspector 34 (S210). That is, an embodiment of the present invention sets and manages the participation qualification for the worker 32 and the participation qualification for the inspector 34 differently from each other.

In one embodiment, the qualification to participate in the examiner 34 may be granted when the matching score is equal to or greater than a reference value. The matching score can be calculated by evaluating and summing at least one of nationality, age, gender, place of birth, place of residence, foreign language ability, grade, and project history required for the project.

Next, the server extracts a plurality of inspectors 34 that satisfy participation qualifications among all inspectors 34 registered in the crowdsourcing platform, and generates an inspector group including the extracted plurality of inspectors 34 (S220). ).

Referring to FIG. 7, in a process of creating a group of inspectors, first, a first inspector of a first scale who satisfies the participation qualification from among all inspectors 34 registered on the platform is extracted (S221). Here, the first scale is not limited as long as the eligibility for participation is met. That is, it may be any N of all K examiners registered on the platform that satisfy the participation qualification. Here, K and N may be different values from K and N of the contents for creating and updating worker groups described in FIGS. 3 to 5, respectively. It is only reused for convenience of explanation.

Then, a second inspector of a second scale is randomly extracted from the remaining inspectors 34 who do not meet the participation qualification (S222). In an embodiment, the second scale may be smaller than the first scale and may be set to correspond to a predetermined ratio of the first scale. For example, the second scale may be 0.1*N people, which is 10% of the first scale N people. Here, the ratio of 10% is only an example, and it goes without saying that various modifications are possible according to embodiments.

Thereafter, an inspector group including a first inspector of a first scale and a second inspector of a second scale is created (S223). Here, the first inspector means an inspector extracted from among inspectors who meet the participation qualification, and is only used to distinguish it from the second inspector extracted from among inspectors who do not meet the participation qualification, and is not limited to other meanings.

Next, a plurality of tasks of the project are assigned to the worker 32 to request task performance (S230), and a plurality of task results are input from the plurality of workers 32 (S240).

Next, a plurality of work results are assigned to a plurality of inspectors 34 included in the inspector group to request inspection (S250), and a plurality of inspection results are input from the inspectors 34 included in the inspector group. (S260). Details on this are duplicated as described above, so detailed descriptions are omitted.

The number of inspectors belonging to the inspector group and the number of inspectors who participated in the actual project are strictly different. Since the project is included in the list of projects that can be participated by the inspector belonging to the inspector group, the inspector can only participate in the project, and the actual participation is entirely up to the inspector.

Next, until the end of the project, the group of inspectors is automatically updated repeatedly every predetermined period (S270). If the examiner group is not updated, a new examiner 34 is added or the existing examiner 34 is not deleted in the examiner group. That is, an inspector who can participate in the project is fixed as an inspector included in the inspector group initially created in step S220.

In one embodiment, a predetermined period of automatically updating the group of inspectors may be determined according to the difficulty of the project. That is, when the difficulty level is high, it takes a lot of time for the inspector 34 to become proficient in inspecting the work result of the project, and thus the predetermined period is also increased.

In another embodiment, the predetermined period of automatically updating the inspector group is determined in consideration of the inspection performance speed, average value, and the total number of work results in the project of the inspectors 34 included in the inspector group, or the determined period is It can also be adjusted.

Referring to FIG. 8, in the process of updating the group of inspectors, first, a first inspector of a first scale who satisfies the participation qualification among all inspectors 34 registered on the platform is newly extracted (S271).

In an embodiment, in the process of updating a group of inspectors, a history of participating in a target project or another project during a predetermined period may be reflected. Project history is considered eligible for participation as described above. Depending on the embodiment, the project history may have the largest weight among various items related to participation qualification. Accordingly, if the updated project history is reflected in the participation qualification, as the project continues, there is an advantage that a group of inspectors matching the inspectors 34 that can gradually improve the progress speed and quality of the project is created.

At this time, the first size may be any N number that satisfies the participation qualification when the examiner group is created, but when the examiner group is updated, it has a fixed value of N when the examiner group is created. That is, even if the number that satisfies the participation qualification is L, only N of them are newly extracted as the first examiner.

Therefore, even if the match score of the examiner 34 is greater than or equal to the reference value and satisfies the participation qualification when the first examiner group is created, it may not be included in the first group upon update. That is, since the first scale has a fixed value, matching scores for the examiner 34 are sorted in descending order and then selected by limiting to a predetermined ranking, and the first scale can be newly extracted according to these conditions.

Then, a second inspector of the second scale is newly randomly extracted from the remaining inspectors 34 who do not meet the participation qualification (S272). In this case, the second scale may be smaller than the first scale as described above and may be set to correspond to a predetermined ratio of the first scale.

Then, the group of inspectors is automatically updated to include the first inspectors of the first scale and the second inspectors of the second scale (S273).

In one embodiment, in order to randomly extract the second examiner of the second scale, the examiner 34 whose activity index is greater than or equal to a reference value is extracted from the remaining examiners 34 who do not meet the participation qualification. In addition, a second inspector of a second scale is newly randomly extracted from among inspectors 34 whose activity index is equal to or greater than the reference value.

Here, the second inspector randomly extracts regardless of the participation qualification, but the activity index is checked. This is because the inclusion of the inspectors 34 who are registered on the platform without participating in the project in the inspector group does not affect the efficiency of the project.

The activity index can mean the degree of participation in any project registered on the platform. In one embodiment, the activity index may be calculated from the number of inspections processed during a certain period. It goes without saying that according to an embodiment, the number of re-reviews for rework of the worker 32 may be included or excluded when calculating the activity index.

The processes shown in FIGS. 6 to 8 are repeated every predetermined period until the end of the project.

In this process, an exemplary embodiment of the present invention may analyze the result of automatically updating the examinee group, and analyze the attribute of the examiner 34 that transfers from the second examiner to the first examiner.

For example, with the arrival of the first cycle, if five inspectors 34 among the second inspectors who participated in the project are eligible to participate and are selected as the first inspectors, the attributes of the inspectors are analyzed. Through this, there is a possibility of discovering an improvement element of the matching system used in the process of recruiting inspectors. In other words, it can be used when matching with a project by discovering the attributes (or features) of inspectors with high qualities. In other words, the participation qualification can be updated in consideration of the attributes.

In this way, the second inspectors function similarly to mutations in the genetic algorithm, and as the inspectors group is updated every cycle, the optimization effect through evolution is exerted.

Since the project is provided as a project that can participate in such a group of inspectors, selection and participation of the project can be facilitated. In addition, from the perspective of the project, as opportunities are given so that the second inspectors can qualify for participation, they are converted to the first inspectors, helping to improve the speed and quality of the project.

Meanwhile, in the above description, steps S11 to S273 may be further divided into additional steps or may be combined into fewer steps, according to an embodiment of the present invention. In addition, some steps may be omitted as necessary, and the order between steps may be changed. In addition, even if other contents are omitted, the contents of FIG. 9 to be described later can also be applied to a method of periodically updating and optimizing a group of workers and inspectors of the crowdsourcing-based project of FIGS. 1 to 8.

Hereinafter, a device 300 for periodically updating and optimizing a group of workers and inspectors of a crowdsourcing-based project according to an embodiment of the present invention (hereinafter, a device for periodically updating and optimizing a group of workers and inspectors) will be described.

9 is a diagram for explaining a device 300 for periodically updating and optimizing a group of workers and inspectors of a crowdsourcing-based project according to an embodiment of the present invention.

Referring to FIG. 9, an apparatus 300 for periodically updating and optimizing a group of workers and inspectors according to the present invention includes a communication module 310, a memory 320, and a processor 330.

The communication module 310 transmits a crowdsourcing-based work for one project to a plurality of workers 32 to request a task to be performed, and receives work results from the plurality of workers 32. Further, the work results received from the plurality of workers 32 are transmitted to the plurality of inspectors 34 to request inspection, and the inspection results are received from the plurality of inspectors 34. In addition, the communication module 310 receives condition information related to qualifications from the operator 32 or the inspector 34 in order to set the participation qualification according to the necessary time, such as at the time of subscription or connection.

The memory 320 stores a program for periodically updating and optimizing a group of workers and inspectors who are participants in a crowdsourcing-based project based on the data received from the communication module 310.

The processor 330 executes a program stored in the memory 320. As the processor 330 executes the program stored in the memory 320, it sets the participation qualification for the participant when the project is opened, and extracts the plurality of workers 32 and the inspector 34 that meet the participation qualification. , It is created as a worker group and an inspector group, respectively.

Thereafter, when the work results and inspection results are received from the plurality of workers 32 and the inspectors 34 in the operator and inspector group, the operator group and inspector group are automatically updated repeatedly every predetermined period until the end of the project.

The apparatus 300 for periodic updating and optimizing the group of workers and inspectors described with reference to FIG. 9 may be provided as a component of the above-described server.

On the other hand, it goes without saying that the device for periodic update and optimization of the worker group and the device for periodic update and optimization of the inspector group may be separately provided.

In addition, it goes without saying that a method of periodically updating and optimizing a group of workers and a method of updating and optimizing a group of inspectors may be performed simultaneously.

The above-described method for periodically updating and optimizing a worker group for a crowdsourcing-based project or a method for periodically updating and optimizing a group of inspectors according to an embodiment of the present invention described above is a program (or application) to be executed in combination with a computer as hardware Can be implemented and stored in a medium.

The above-described program includes C, C++, JAVA, Ruby, which can be read by a processor (CPU) of the computer through the device interface of the computer, in order for the computer to read the program and execute the methods implemented as a program. It may include a code (Code) coded in a computer language such as machine language. Such code may include a functional code related to a function defining necessary functions for executing the methods, and a control code related to an execution procedure necessary for the processor of the computer to execute the functions according to a predetermined procedure. can do. In addition, such code may further include additional information required for the processor of the computer to execute the functions or code related to a memory reference to which location (address address) of the internal or external memory of the computer should be referenced. have. In addition, when the processor of the computer needs to communicate with any other computer or server in the remote in order to execute the functions, the code uses the communication module of the computer to determine how It may further include a communication-related code for whether to communicate or what information or media to transmit and receive during communication.

The stored medium is not a medium that stores data for a short moment, such as a register, cache, memory, etc., but a medium that stores data semi-permanently and can be read by a device. Specifically, examples of the storage medium include, but are not limited to, ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, and the like. That is, the program may be stored in various recording media on various servers to which the computer can access, or on various recording media on the user's computer. Further, the medium may be distributed over a computer system connected through a network, and computer-readable codes may be stored in a distributed manner.

The above description of the present invention is for illustrative purposes only, and those of ordinary skill in the art to which the present invention pertains will be able to understand that it can be easily modified into other specific forms without changing the technical spirit or essential features of the present invention. will be. Therefore, it should be understood that the embodiments described above are illustrative in all respects and not limiting. For example, each component described as a single type may be implemented in a distributed manner, and similarly, components described as being distributed may also be implemented in a combined form.

The scope of the present invention is indicated by the claims to be described later rather than the detailed description, and all changes or modified forms derived from the meaning and scope of the claims and their equivalent concepts should be interpreted as being included in the scope of the present invention. do.

10: Client
20: service provider
30: public
32: worker
34: inspector
100: Periodic update and optimization device of the operator and inspector group
110: communication module
120: memory
130: processor

Claims (14)

As a method performed by a computer,
Setting a participation qualification for a worker of the project upon opening a crowdsourcing-based project (hereinafter, referred to as a project);
Extracting a plurality of workers satisfying the participation qualification for the worker from among all workers registered in the crowdsourcing platform (hereinafter, the platform), and generating a worker group including the extracted plurality of workers;
Assigning a plurality of tasks to a plurality of workers included in the worker group and requesting to perform a task;
Receiving a plurality of job results from a plurality of workers included in the worker group; And
It includes the step of automatically updating the worker group repeatedly at a predetermined period until the end of the project,
The step of automatically updating the worker group,
A step of newly extracting a first worker of a first scale that satisfies the participation qualification from among all workers registered on the platform in the current cycle; and
A step of newly randomly extracting a second worker of a second size from among the remaining workers who do not meet the participation qualification except for the first worker registered on the platform in the current cycle,
And automatically updating the worker group to include the first worker of the first scale and the second worker of the second scale,
The step of creating the worker group,
Extracting the first worker of the first scale who meets the participation qualification for the worker from among all workers registered on the platform,
Randomly extracting the second worker of the second scale from among the remaining workers who do not meet the participation qualification for the worker,
Including the step of creating the worker group including the first worker of the first scale and the second worker of the second scale,
How to periodically update and optimize worker groups for crowdsourced projects. delete The method of claim 1,
The step of newly randomly extracting a second worker of the second scale,
Extracting a worker whose activity index is greater than or equal to a reference value from among the remaining workers who do not meet the participation qualification;
Including the step of newly randomly extracting a second worker of the second scale from among workers whose activity index is equal to or greater than a reference value,
How to periodically update and optimize worker groups for crowdsourced projects. The method of claim 1,
The second scale is smaller than the first scale, and corresponds to a predetermined ratio of the first scale,
How to periodically update and optimize worker groups for crowdsourced projects. The method of claim 1,
The cycle of automatically updating the worker group is determined according to the difficulty of the project,
How to periodically update and optimize worker groups for crowdsourced projects. The method of claim 1,
Analyzing the result of automatically updating the worker group, further comprising the step of analyzing the attribute of the worker transferring to the first worker among the second worker,
How to periodically update and optimize worker groups for crowdsourced projects. A computer program combined with a computer and stored on a computer-readable recording medium for executing the method of periodically updating and optimizing a group of workers in a crowdsourcing-based project according to any one of claims 1, 3 to 6. As a method performed by a computer,
Setting a participation qualification for an inspector of the project upon opening a crowdsourcing-based project (hereinafter, referred to as a project);
Extracting a plurality of inspectors satisfying participation qualifications for the inspectors from among all inspectors registered in a crowdsourcing platform (hereinafter, the platform), and generating an inspector group including the extracted plurality of inspectors;
Allocating a plurality of tasks of the project to a plurality of workers and requesting the task to be performed;
Receiving a plurality of job results from the plurality of workers;
Allocating the plurality of work results to a plurality of inspectors included in the inspector group and requesting to perform inspection;
Receiving a plurality of inspection results for the plurality of work results from a plurality of inspectors included in the inspector group; And
And automatically updating the inspector group repeatedly at a predetermined period until the end of the project,
The step of automatically updating the examiner group,
A step of newly extracting a first inspector of a first scale satisfying the participation qualification from among all inspectors registered on the platform in the current cycle; and
A step of newly randomly extracting a second inspector of a second scale from among the remaining inspectors who do not meet the participation qualification except for the first inspectors registered on the platform in the current cycle; and
Automatically updating the group of inspectors to include the first inspectors of the first scale and the second inspectors of the second scale,
The step of creating the inspector group,
Extracting the first examiner of the first scale who satisfies the participation qualification for the examiner from among all the examiners registered on the platform,
Randomly extracting the second inspector of the second scale from among the remaining inspectors who do not meet the participation qualification for the inspector; and
Generating the group of inspectors comprising the first inspectors of the first scale and the second inspectors of the second scale,
How to periodically update and optimize a group of inspectors for crowdsourced projects. delete The method of claim 8,
The step of newly randomly extracting a second inspector of the second scale,
Extracting an examiner whose activity index is greater than or equal to a reference value from among the remaining examiners who do not meet the participation qualification;
Comprising the step of newly randomly extracting a second inspector of the second scale from among inspectors whose activity index is equal to or greater than a reference value,
How to periodically update and optimize a group of inspectors for crowdsourced projects. The method of claim 8,
The second scale is smaller than the first scale, and corresponds to a predetermined ratio of the first scale,
How to periodically update and optimize a group of inspectors for crowdsourced projects. The method of claim 8,
The cycle of automatically updating the examiner group is determined according to the difficulty level of the project,
How to periodically update and optimize a group of inspectors for crowdsourced projects. The method of claim 8,
Analyzing the result of automatically updating the examiner group, further comprising the step of analyzing the attributes of the examiner transferring to the first examiner among the second examiners,
How to periodically update and optimize a group of inspectors for crowdsourced projects. A computer program combined with a computer and stored on a computer-readable recording medium for executing the method of periodically updating and optimizing a group of inspectors of a crowdsourcing-based project according to any one of claims 8 and 10 to 13.

Images