JP6306400B2 - Skill evaluation apparatus, program and method for evaluating worker skill in crowdsourcing - Google Patents

Description

  The present invention relates to a technique for evaluating a worker's skill.

  Computers have evolved by performing human tasks as “tasks”. 2. Description of the Related Art Conventionally, there is a technique for automatically detecting and clearly indicating a portion with a possibility of error, such as spell check, for a sentence written by a person (see, for example, Patent Document 1). There is also a technique for automatically recognizing a person's face image in a video by creating a template for each person's orientation (see, for example, Patent Document 2). These technologies are expected to have a certain level of accuracy due to the application of effective algorithms and the enhancement of databases.

  On the other hand, there is a technique for automatically evaluating the skill (Skill, work ability) of each individual who is a main subject of work (see, for example, Patent Document 3). According to this technique, questions with correct answers are given to each evaluation target person, and the skill of the evaluation target person is quantitatively evaluated based on the answer result. For example, in the case of a sentence proofreading operation, the determination is made based on whether or not an operator can find a typographical error for a plurality of grammatical sentences whose typographical positions are known. By evaluating each person's skill, it is possible to ask a highly skilled worker for the work.

  Also, according to the work of image recognition, when sufficient recognition accuracy cannot be obtained with a single recognition algorithm, high recognition accuracy can be achieved by integrating the calculation results of a plurality of recognition algorithms (for example, taking a majority vote). There is also a technique to obtain (see, for example, Patent Document 4).

JP 2005-267663 A JP 2012-238111 A JP 2009-222906 A JP 2009-140369 A

  In the case of sentence proofreading work and image recognition work, human recognition and judgment are required. For example, as in the techniques described in Patent Documents 1 and 2, sufficient performance can be obtained only by applying an effective algorithm and enhancing the database. It is difficult. In other words, it is difficult to make the machine completely dependent on work that requires human recognition and judgment.

  On the other hand, “crowd sourcing” in which an unspecified number of people (workers) are requested to work via the Internet has recently attracted attention. For work that requires human recognition and judgment, as a result, a task with higher accuracy can be obtained by performing the work as a human work. Even in that sense, crowdsourcing with a large number of human resources has a beneficial effect.

  However, the skill of the worker varies greatly depending on the work content. It also varies greatly depending on the specialized field of the worker. Specifically, even in the proofreading work, depending on the specialized field described in the sentence, the skill of the operator may be lowered. In the case of crowdsourcing, a large number of human powers can be used, but it is difficult to ensure a certain level of work quality.

  Further, for example, as in the technique described in Patent Document 3, it is necessary to cause all the target workers to answer in advance a problem whose correct answer is known. However, work other than the original work is generated, and there is a problem in cost.

  Furthermore, according to the technique described in Patent Document 4, it is assumed that the calculation result of each algorithm is correct to some extent (for example, a majority decision of 50% or more). However, when there are a large number of unspecified workers, a small number of answers is not always an incorrect answer. Only a few highly skilled people may output the correct task as a result. Like crowdsourcing, it is more difficult to correctly evaluate the skills of an unspecified number of workers.

  Therefore, an object of the present invention is to provide a skill evaluation apparatus, program, and method capable of selecting a highly skilled worker among a large number of unspecified workers such as crowdsourcing.

According to the present invention, a skill evaluation apparatus for evaluating each worker's skill with respect to an operation of adding “work task information” determined by the worker himself to “target content”,
Work task information collection means for collecting work task information of each worker;
Learning task information estimation means for outputting “learning task information” mechanically estimated according to work for the target content;
Work task information selection means for selecting a plurality of work task information whose similarity to learning task information is equal to or greater than a predetermined threshold;
Learning task information extracting means for generating new learning task information obtained by merging the selected plurality of work task information by logical product, logical sum or majority, and feeding back the learned task information to the work task information selecting means;
Finally, there is provided worker skill evaluation means for sorting a plurality of work task information in descending order of similarity to the learning task information and evaluating the rank as a skill for each worker.

According to another embodiment of the skill evaluation apparatus of the present invention,
The target content is “sentence”, and the work task information is error information for the sentence.
The target content is “image”, and the work task information is instruction information for the image, or
It is also preferable that the target content is “speech” and the work task information is text information of transcription for the sound.

According to another embodiment of the skill evaluation apparatus of the present invention,
Regarding the work task information selection means, the similarity in the comparison between the learning task information and the work task information is preferably an F value (F-measure) based on precision and recall.

According to another embodiment of the skill evaluation apparatus of the present invention,
It is also preferable to function as a crowdsourcing server in a crowdsourcing service.

According to the present invention, a program for causing a computer mounted on an apparatus for evaluating each worker's skill to function for the operation of adding “work task information” determined by the worker himself to “target content”,
Work task information collection means for collecting work task information of each worker;
Learning task information estimation means for outputting “learning task information” mechanically estimated according to work for the target content;
Work task information selection means for selecting a plurality of work task information whose similarity to learning task information is equal to or greater than a predetermined threshold;
Learning task information extracting means for generating new learning task information obtained by merging the selected plurality of work task information by logical product, logical sum or majority, and feeding back the learned task information to the work task information selecting means;
Finally, a plurality of work task information is sorted in descending order of similarity to the learning task information, and the computer functions as worker skill evaluation means for evaluating the rank as a skill for each worker.

According to the present invention, there is provided a skill evaluation method for an apparatus that performs an operation to evaluate each worker's skill with respect to an operation of adding "work task information" determined by the worker himself to "target content",
A first step of collecting work task information for each worker;
A second step of outputting “learning task information” mechanically estimated according to work for the target content;
A third step of selecting a plurality of work task information whose similarity to the learning task information is equal to or greater than a predetermined threshold;
A fourth step of generating new learning task information obtained by merging the selected plurality of work task information by logical product, logical sum or majority vote, and feeding back the learned task information to the work task information selecting means;
Finally, a plurality of work task information items are sorted in descending order of similarity to the learning task information, and a fifth step of evaluating the rank as a skill for each worker is provided.

  According to the skill evaluation apparatus, program, and method of the present invention, it is possible to select a highly skilled worker among an unspecified number and various workers such as crowdsourcing.

It is a system configuration diagram in the present invention. It is a functional block diagram in the skill evaluation apparatus of this invention. It is explanatory drawing showing the work task information of each worker with respect to object content. It is explanatory drawing showing the comparison with the learning task information and work task information in a text proofreading work. It is explanatory drawing which derived | led-out the learning task information which merged the some work task. It is explanatory drawing showing the comparison with new learning task information and work task information. It is explanatory drawing showing average skill evaluation at the time of making each operator work a some text proofreading work. It is explanatory drawing showing a comparison with learning task information and work task information in image recognition work.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

  FIG. 1 is a system configuration diagram according to the present invention.

  According to FIG. 1, a skill evaluation apparatus 1 and a number of workers' terminals 2 are connected via a network. Specifically, this system is applied as a “crowd sourcing service”. The work requester transmits “target content” as a work target to the skill evaluation apparatus 1 using his / her terminal 3. The skill evaluation apparatus 1 provides an API (Application Programming Interface) for crowdsourcing as viewed from the work requester's terminal 3. The skill evaluation apparatus 1 transmits this “target content” to the terminals 2 of a number of workers, and uses the “work task information” returned from each worker after the work. , Work ability). The skill evaluation apparatus 1 can evaluate each worker's skill by finally sorting the workers in descending order of skill using the target content.

  The worker works on the requested “target content” and adds “work task information” determined by himself / herself. Here, “work” is such that “work task information” is added to “target content”. In other words, the “work” targeted by the present invention is not something that creatively creates content for the worker, but is based on the operator's recognition and judgment on the original “target content”. It is like adding “work task information”.

When the requested work is a “sentence proofreading work” in which an error is found from the original sentence, for example, an underline (work task information) is added to the erroneous part of the original sentence (target content). is there.
Further, when the requested work is “image recognition work” for finding a specific part from the original image, for example, a surrounding mark (work task information) is placed on the recognized face part with respect to the original image (target content). It is like adding.
The present invention can also be applied to work that adds various work task information to various target contents.

According to FIG. 1, it is assumed that the work requester requests, for example, a proofreading work for the following text (target content).
“In recent years, attention has been focused on crowdsourcing, in which an unspecified large number of people request work from the Internet through Internets. It is difficult for computers to obtain sufficient performance, such as document welfare and image recognition. It will be useful in devising tasks that require judgment. "
On the other hand, each worker underlines a part that seems to be erroneously written as a spell check.

  FIG. 2 is a functional configuration diagram of the skill evaluation apparatus according to the present invention.

  According to FIG. 2, the skill evaluation device 1 includes a work task information collection unit 11, a learning task information estimation unit 12, a work task information selection unit 13, a learning task information extraction unit 14, and a worker skill evaluation unit 15. And have. These functional components are realized by executing a computer mounted on the apparatus. Further, the flow of processing of these functional components can be understood as a skill evaluation method using an apparatus.

[Work Task Information Collection Unit 11]
The work task information collection unit 11 transmits “target content” to each worker, and collects “work task information” worked by each worker.

  FIG. 3 is an explanatory diagram showing work task information (pointing out a misprint, spelling check) of each worker for the target content (sentence). According to FIG. 3, for each worker, the portion of the content of the target content that is determined to be erroneously written is underlined.

[Learning task information estimation unit 12]
The learning task information estimation unit 12 outputs “learning task information” that is mechanically estimated according to work for the target content. The learning task information estimation unit 12 outputs learning task information in various machine learning according to work.
Specifically, the automatic sentence composition function described in Patent Document 1 described above may be used for the sentence proofreading work.
For the image recognition work, the automatic image recognition function described in Patent Document 2 described above may be used.

  The learning task information estimation unit 12 outputs learning task information that can be estimated to be generally correct as a computer for the target content. That is, the learning task information is not completely correct because human recognition and judgment cannot be completely estimated.

  FIG. 4 is an explanatory diagram showing a comparison between learning task information and work task information in a sentence proofreading work.

  FIG. 4A shows the learning task information output from the learning task information estimation unit 12. This learning task information is compared with work task information from each worker.

[Work Task Information Selection Unit 13]
The work task information selection unit 13 extracts a plurality of high skill work task information whose similarity to the learning task information is equal to or greater than a predetermined threshold. It can be estimated that a worker having a higher degree of similarity is more likely to have a work skill that is at least equal to the learning task information. Note that the predetermined threshold value to be determined may be set to a different value depending on the work content.

According to FIG. 4B, the work task information proofread by each worker is compared with the learning task information. For example, the predetermined threshold is set to 60%. That is, a worker with a similarity of 60% or more is certified as a high skill worker.
Worker A-> 4 out of 5 matches: 80% similarity
○ "Internets""Documentwelfare""Difficult""Take"
× "Crowding sourcing"
Worker B-> 1 out of 5 matches: 20% similarity
○ “Crowdsourcing”
× "Internets""Documentwelfare""Difficult""Take"
Worker C-> 4 out of 5 matches: 80% similarity
○ "Internets""Documentwelfare""Difficult""Take"
× "Crowding sourcing"
Worker D-> 0 out of 5 matches: similarity 0%
× "Internets""Crowdsourcing""DocumentWelfare""Difficult""Take"
In this case, worker A and worker C have a similarity of 60% or more, and are selected as high skill workers. The work task information selection unit 13 selects, as candidates, a worker who has a high skill level from among a large number of workers, using learning task information estimated to be almost correct by the computer.

The work task information selection unit 13 may use an “F value (F-measure)” based on the recall rate or the matching rate as another embodiment in the comparison between the learning task information and the work task information. .
“Accuracy (precision)” is an index of accuracy representing the ratio of the correct part of the learning task information in the set of parts of the work task information.
Relevance rate P = number of matching parts / number of work task information parts "Recall rate" is the comprehensiveness indicating the percentage of work task information parts in the set of learning task information parts It is an indicator.
Reproducibility R = number of matching locations / number of locations of learning task information F value = 1/2 × (1 / R + 1 / P)
Here, there is a trade-off relationship that if the precision is increased, the recall is decreased, and if the precision is increased, the precision is decreased. Therefore, it is also preferable to evaluate the search performance using the F value that is a harmonic average of the precision and recall. The higher the F value, the closer the work task information of the worker is to the learning task information. Only work task information whose F value is equal to or greater than a predetermined threshold value can be selected.

[Learning task information extraction unit 14]
The learning task information extraction unit 14 generates new learning task information obtained by merging a plurality of high skill work task information by logical product, logical sum, or majority vote. In other words, it is possible to assume that a part having the same work result among a plurality of high-skill work task information is a highly accurate and correct result.

  FIG. 5 is an explanatory diagram in which learning task information obtained by merging a plurality of work tasks is derived.

According to FIG. 5, the work task information of the selected high skill workers A and C is merged by logical product.
(Worker A) (Worker C) (New learning task information)
"Internets""Internets""Internets"
「が」 × ×
Document welfare Document welfare Document welfare
"Body""body""body"
"It's difficult""It'sdifficult""It'sdifficult"
"Take""take""take"
× “Push” ×
The learned task information is fed back to the work task information selection unit 13. As a result, the work task information selection unit 13 uses the new learning task information to newly calculate the similarity to the work task information.

  FIG. 6 is an explanatory diagram showing a comparison between new learning task information and work task information.

According to FIG. 6, the work task information proofread by each worker is compared with new learning task information.
Worker A-> 5 out of 5 matches: 100% similarity (score improved from before update)
Worker B-> 0 out of 5 matches: similarity 0% (score lower than before update)
Worker C-> 5 out of 5 matches: 100% similarity (score improved from before update)
Worker D-> 0 out of 5 locations: 0% similarity (no change from before update)

  By repeating the feedback of the learning task information at least once by the work task information selection unit 13 and the learning task information extraction unit 14, the skill evaluation of the worker becomes highly accurate. The number of times of feedback may be set as a predetermined number of times, or may be terminated when there is no change in the learning task information.

[Worker skill evaluation unit 15]
The worker skill evaluation unit 15 finally sorts the plurality of work task information in the order of similarity to the learning task information. The sorting order of each worker is evaluated as the skill of that worker. The learning task information finally obtained is output as a work product.

  The skill evaluation of each worker may be registered in a database and used for various purposes. Moreover, skill evaluation of each worker can be collected for each work, and the average value can be used as the skill evaluation of the worker.

  FIG. 7 is an explanatory diagram showing an average skill evaluation when each worker performs a plurality of sentence proofreading operations. Thereby, the average skill for each worker is evaluated.

According to the present invention, it is difficult to obtain sufficient performance with only a computer, and the present invention can be applied to work that requires human recognition and judgment. For example, it can be applied to the following work.
(1) The target content is “sentence”, and the work task information is error information for the sentence.
(2) The target content is “image”, and the work task information is instruction information for the image.
(3) The target content is “voice”, and the work task information is text information (transcription) for the voice.
That is, the present invention can be applied to various operations such as adding some task information to the target content.

  FIG. 8 is an explanatory diagram showing a comparison between learning task information and work task information in image recognition work.

  According to FIG. 8, as an example of the image recognition work, a work of marking a part including a human face in an image as target content is shown. Such work may be necessary for moving image analysis. As the work task information and the learning task information, the position and area of the mark can be compared. Then, work task information whose position and area are within a predetermined threshold (similarity is equal to or higher than a predetermined threshold) is selected for the learning task information mark. New learning task information is created by merging the positions and areas of the selected high-skill work task information. The new learning task information is used to compare with the work task information again. As a result, the work task information finally obtained and having higher similarity to the learning task information can be evaluated as being worked by a highly skilled worker.

  As described above in detail, according to the skill evaluation apparatus, program, and method of the present invention, it is possible to select a highly skilled worker among a large number of unspecified workers such as crowdsourcing. it can. Further, according to the present invention, it is possible to obtain a work product of the task simultaneously with the skill evaluation of the worker. In particular, the operator can be evaluated without having to solve a problem for which the correct answer is known, so that cost reduction is expected.

  Finally, specific effects obtained by the present invention will be described. For example, assume that 50 high-skilled workers are allowed to work on 50 target contents. Here, it is assumed that 10 workers are registered in advance.

According to the prior art, it is necessary to separately work 20 target contents whose correct answers are known for skill evaluation. Therefore, the following steps are required.
(S1: Skill evaluation) First, 10 workers are allowed to work on 20 target contents.
(S2: High skill work) Next, five persons evaluated as high skill among them are made to work on 50 target contents.
Man-hours = (20 cases x 10 people) + (50 cases x 5 people) = 450 cases

On the other hand, according to the present invention, the following steps are required.
(S1: Skill evaluation) First, 10 workers are allowed to work on 20 target contents.
(S2: High-skill work) Next, the five persons evaluated as high-skill among them are allowed to work on the remaining 30 target contents.
Man-hours = (20 cases x 10 people) + (30 cases x 5 people) = 350 cases

That is, according to the present invention, it is possible to evaluate the skill of the worker with respect to S1 and obtain the deliverable. Further, there is no need to separately answer a problem for which a correct answer for skill evaluation is known. As a result, the following costs can be reduced.
Cost = 350 (present invention) / 450 (prior art) = 78%
As a result, the cost is reduced by 22% (= 100−78).

  Various changes, modifications, and omissions of the above-described various embodiments of the present invention can be easily made by those skilled in the art. The above description is merely an example, and is not intended to be restrictive. The invention is limited only as defined in the following claims and the equivalents thereto.

DESCRIPTION OF SYMBOLS 1 Skill evaluation apparatus 11 Work task information collection part 12 Learning task information estimation part 13 Work task information selection part 14 Learning task information extraction part 15 Worker skill evaluation part 2 Terminal for workers 3 Terminal for work requesters

Claims (6)

A skill evaluation device that evaluates each worker's skill (skill, work ability) for the task of adding “work task information” determined by the worker himself to “target content”,
Work task information collection means for collecting work task information of each worker;
Learning task information estimation means for outputting “learning task information” mechanically estimated according to work for the target content;
Work task information selection means for selecting a plurality of work task information having a similarity to the learning task information equal to or greater than a predetermined threshold;
Learning task information extracting means for generating new learning task information obtained by merging the selected plurality of work task information by logical product, logical sum or majority, and feeding back the learning task information to the work task information selecting means; ,
Finally, a skill evaluation apparatus comprising worker skill evaluation means for sorting a plurality of work task information in descending order of similarity to the learning task information and evaluating the rank as a skill for each worker . The target content is “sentence”, and the work task information is error information for the sentence.
The target content is “image”, and the work task information is instruction information for the image, or
The skill evaluation apparatus according to claim 1, wherein the target content is “voice”, and the work task information is text information of transcription for the voice.   Regarding the work task information selection means, the similarity in the comparison between the learning task information and the work task information is an F value (F-measure) based on precision and recall. The skill evaluation apparatus according to claim 1 or 2.   The skill evaluation apparatus according to any one of claims 1 to 3, wherein the skill evaluation apparatus functions as a crowdsourcing server in a crowdsourcing service. A program that causes a computer installed in a device that evaluates each worker's skill to function for adding work task information determined by the worker to the target content.
Work task information collection means for collecting work task information of each worker;
Learning task information estimation means for outputting “learning task information” mechanically estimated according to work for the target content;
Work task information selection means for selecting a plurality of work task information having a similarity to the learning task information equal to or greater than a predetermined threshold;
Learning task information extracting means for generating new learning task information obtained by merging the selected plurality of work task information by logical product, logical sum or majority, and feeding back the learning task information to the work task information selecting means; ,
Finally, a plurality of work task information is sorted in descending order of similarity to the learning task information, and the computer functions as a worker skill evaluation means for evaluating the rank as a skill for each worker. A skill evaluation method for a device that performs an operation to add the “work task information” determined by the worker to “target content” so as to evaluate each worker's skill,
A first step of collecting work task information for each worker;
A second step of outputting “learning task information” mechanically estimated according to work for the target content;
A third step of selecting a plurality of work task information having a similarity to the learning task information equal to or greater than a predetermined threshold;
A fourth step of generating new learning task information obtained by merging the selected plurality of work task information by logical product, logical sum or majority vote, and feeding back the learned task information to the work task information selecting means;
Finally, a fifth step of sorting a plurality of work task information in descending order of similarity to the learning task information and evaluating the rank as a skill for each worker is provided. Method.

Images